Creating custom system monitoring dashboards with watch and tmux

System administrators and developers often need to monitor multiple aspects of their servers simultaneously. While there are sophisticated monitoring tools available, sometimes you need a quick, customizable solution that doesn’t require installation of additional software. This is where the combination of watch and tmux becomes powerful.

Understanding the Fundamentals

What is watch?

The watch command is a simple yet powerful utility that runs a specified command repeatedly at regular intervals and displays the output. By default, it updates every 2 seconds, allowing you to monitor changes in real-time.

Basic syntax:

watch [options] command

Common options:

• -n – Set the update interval in seconds (e.g., -n 5 for 5 seconds)
• -d – Highlight differences between successive updates
• -t – Turn off the header showing the command and interval
• -c – Interpret ANSI color sequences

What is tmux?

Tmux (terminal multiplexer) allows you to split your terminal into multiple panes and windows. It’s like having several terminal windows within a single screen, making it perfect for creating dashboard layouts.

Key concepts:

• Session: A collection of windows that can be detached and reattached
• Window: Similar to a tab in a browser, contains one or more panes
• Pane: A split section within a window showing a terminal

Essential tmux commands:

• tmux new -s dashboard – Create a new session named “dashboard”
• Ctrl+b % – Split pane vertically
• Ctrl+b " – Split pane horizontally
• Ctrl+b arrow-key – Navigate between panes
• Ctrl+b d – Detach from session (keeps running in background)
• tmux attach -t dashboard – Reattach to a session

Basic Example: Your First Dashboard

Let’s create a simple two-pane dashboard to monitor CPU usage and disk space:

# Start a new tmux session
tmux new -s monitoring

# In the first pane, watch CPU information
watch -n 1 'top -bn1 | head -20'

# Split the window horizontally (Ctrl+b ")
# In the new pane, watch disk usage
watch -n 5 'df -h'

This creates a simple dashboard where the top pane shows real-time CPU and process information, while the bottom pane displays disk space usage.

Use Case Scenarios

Scenario 1: Web Server Monitoring

Perfect for monitoring a web server’s health, showing active connections, error logs, and resource usage.

Dashboard layout: 4 panes in a 2×2 grid

# Start tmux session
tmux new -s webserver

# Top-left: Active Apache/Nginx connections
watch -n 2 'netstat -an | grep :80 | wc -l'

# Split vertically (Ctrl+b %)
# Top-right: Memory usage
watch -n 3 'free -h'

# Move to top-left (Ctrl+b arrow keys) and split horizontally (Ctrl+b ")
# Bottom-left: Latest error log entries
watch -n 5 'tail -20 /var/log/apache2/error.log'

# Move to top-right and split horizontally
# Bottom-right: Disk I/O statistics
watch -n 2 'iostat -x 1 1'

Scenario 2: Database Performance Dashboard

Monitor database connections, queries, and system resources for database servers.

# Start tmux session
tmux new -s database

# Top pane: MySQL process list
watch -n 3 'mysql -e "SHOW PROCESSLIST"'

# Split horizontally
# Middle pane: Connection count
watch -n 2 'mysql -e "SHOW STATUS LIKE \"Threads_connected\""'

# Split horizontally again
# Bottom pane: Table lock status
watch -n 5 'mysql -e "SHOW OPEN TABLES WHERE In_use > 0"'

Scenario 3: Application Deployment Monitor

Track deployment progress, application logs, and server metrics during deployments.

# Start tmux session
tmux new -s deployment

# Left pane: Application logs (taller)
tail -f /var/log/application.log

# Split vertically
# Top-right: Git deployment status
watch -n 5 'cd /var/www/app && git status'

# Move to right pane and split horizontally
# Middle-right: Running processes
watch -n 3 'ps aux | grep node'

# Split horizontally again
# Bottom-right: Network connections
watch -n 2 'netstat -tulpn | grep LISTEN'

Scenario 4: Docker Container Monitoring

Keep track of container status, resource usage, and logs.

# Start tmux session
tmux new -s docker

# Top-left: Container list
watch -n 2 'docker ps'

# Split vertically
# Top-right: Docker stats
watch -n 3 'docker stats --no-stream'

# Move to left and split horizontally
# Bottom-left: Specific container logs
docker logs -f container_name

# Move to right and split horizontally
# Bottom-right: Disk usage
watch -n 10 'docker system df'

Scenario 5: Network Traffic Monitor

Monitor bandwidth usage, active connections, and network interface statistics.

# Start tmux session
tmux new -s network

# Top pane: Network bandwidth (requires iftop)
watch -n 1 'iftop -t -s 1 -L 10'

# Split horizontally
# Middle pane: Active connections
watch -n 2 'netstat -natp | grep ESTABLISHED | wc -l'

# Split horizontally
# Bottom pane: Network interface statistics
watch -n 3 'ifconfig eth0'

Advanced Tips and Techniques

Creating Reusable Dashboard Scripts

Instead of manually creating your dashboard layout each time, you can create a shell script:

#!/bin/bash
# save as dashboard.sh

SESSION="monitoring"

# Create session
tmux new-session -d -s $SESSION

# First window with 4 panes
tmux split-window -h
tmux split-window -v
tmux select-pane -t 0
tmux split-window -v

# Send commands to each pane
tmux select-pane -t 0
tmux send-keys "watch -n 1 'uptime'" C-m

tmux select-pane -t 1
tmux send-keys "watch -n 2 'free -h'" C-m

tmux select-pane -t 2
tmux send-keys "watch -n 3 'df -h'" C-m

tmux select-pane -t 3
tmux send-keys "watch -n 2 'who'" C-m

# Attach to session
tmux attach-session -t $SESSION

Make it executable and run:

chmod +x dashboard.sh
./dashboard.sh

Customizing Colors and Highlighting

Make important information stand out:

# Highlight differences in watch output
watch -d -n 2 'netstat -an | grep :80 | wc -l'

# Use colored output (when supported)
watch -c -n 1 'ls -l --color=always /var/log'

# Combine with grep for highlighting
watch -n 2 'ps aux | grep --color=always "apache2"'

Adjusting Pane Sizes

You can resize panes to emphasize important information:

• Ctrl+b Ctrl+arrow-key – Resize pane in small increments
• Ctrl+b Alt+arrow-key – Resize pane in larger increments
• Ctrl+b z – Toggle pane zoom (fullscreen current pane)

Saving and Restoring Layouts

You can save your tmux layout and restore it later:

# Save current layout
tmux list-windows

# This shows something like: layout: 2ae1,238x60,0,0{119x60,0,0,0,118x60,120,0,1}

# In your script, use:
tmux select-layout "2ae1,238x60,0,0{119x60,0,0,0,118x60,120,0,1}"

Best Practices

Update Intervals

• Fast-changing data (CPU, network): 1-2 seconds
• Moderate data (processes, connections): 3-5 seconds
• Slow-changing data (disk space, logs): 10-30 seconds

Using appropriate intervals prevents unnecessary system load and makes the dashboard easier to read.

Resource Considerations

• Be mindful of commands that consume significant resources (e.g., full process lists)
• Avoid running too many resource-intensive commands simultaneously
• Use filtering (grep, head, tail) to limit output
• Consider the impact on system performance, especially on production servers

Organizing Multiple Dashboards

Create different tmux sessions for different purposes:

# Different sessions for different needs
tmux new -s web-monitoring
tmux new -s database-monitoring
tmux new -s app-monitoring

# List all sessions
tmux ls

# Switch between sessions
tmux switch-client -t web-monitoring

Troubleshooting Common Issues

Command Output Too Large

If output overflows the pane, limit it:

watch -n 2 'command | head -n 30'

Watch Not Updating

Some commands might not work well with watch. Try:

# Use explicit bash execution
watch -n 2 'bash -c "your command here"'

Tmux Panes Too Small

Resize your terminal window or adjust the tmux layout to accommodate the content better.

Session Disconnected

Tmux sessions persist even if you disconnect. Reconnect with:

tmux attach -t session-name

Conclusion

Combining watch and tmux provides a lightweight, flexible solution for creating custom monitoring dashboards. While it may not replace comprehensive monitoring solutions like Grafana or Prometheus for large-scale operations, it’s perfect for:

• Quick troubleshooting sessions
• Development and testing environments
• Personal servers and small deployments
• Situations where you need immediate visibility without setup time

The beauty of this approach is its simplicity and customizability. You can monitor exactly what you need, how you need it, with tools that are already installed on virtually every Linux system. Start simple, experiment with different layouts, and create dashboard scripts that match your specific monitoring needs.