Troubleshooting Linux Systems with Real-World Log Examples

Troubleshooting is one of the most critical responsibilities of a Linux system administrator. Regardless of whether you’re managing a small virtual private server (VPS) or a comprehensive cluster of enterprise servers, encountering system issues such as failed SSH logins, service crashes, or disk-related problems is an inevitability. The good news is that Linux provides an extensive logging system—primarily managed through rsyslog—to assist in tracking down the root cause of these problems.

However, knowing where to look, what to look for, and how to interpret logs can be the deciding factor between resolving an issue in minutes or spending hours chasing shadows. In this article, we will go through real-world examples of common Linux system issues, explore which log files to inspect, and demonstrate how to effectively use tools like grep, awk, and rsyslog to troubleshoot efficiently. By understanding how to harness these powerful tools, you’ll be able to address issues in a timely manner, leading to a more stable and reliable Linux environment.

Understanding the Linux Logging System

What is rsyslog?

rsyslog stands for “Rocket-fast System for Log Processing” and is the default logging service on most modern Linux distributions. It collects logs from multiple sources, ensuring that administrators have a comprehensive view of system activities.

This logging system collects logs from various components, including:

• Kernel: Messages generated by the Linux kernel regarding system operations and hardware status.
• System Services: Logs from services managed by init, systemd, or any similar service manager.
• Applications: Application-specific log entries generated by different programs and user applications.
• Authentication Mechanisms: Logs detailing authentication attempts, successes, and failures.
• Scheduled Tasks: Entries related to cron jobs and other automated tasks that run on a schedule.

All these logs are typically stored in plain text files, with most of them residing within the /var/log directory. Rsyslog employs a modular configuration tool that allows administrators to customize where logs are sent, how they are filtered, and how they are formatted or rotated for management purposes.

Common Log File Locations

As you troubleshoot various issues, you will frequently refer to specific log files. The following are key log files that we’ll discuss throughout the article:

Log File	Description
/var/log/syslog	General system logs (Debian/Ubuntu)
/var/log/messages	General system logs (RHEL/CentOS)
/var/log/auth.log	Authentication logs (SSH, sudo, login attempts)
/var/log/kern.log	Kernel logs
/var/log/dmesg	Boot and hardware messages
/var/log/cron.log	Logs from scheduled cron jobs
/var/log/nginx/error.log	NGINX-specific error log
/var/log/mysql/error.log	MySQL error log

1. Diagnosing SSH Login Failures

Problem:

You’re trying to SSH into a server, but the authentication fails, or the connection hangs indefinitely.

Log to Check:

• /var/log/auth.log (Debian/Ubuntu)
• /var/log/secure (RHEL/CentOS)

Example Command:

sudo grep sshd /var/log/auth.log

Expected Output:

You might see something like the following:

Jul 5 11:02:43 server sshd[1234]: Failed password for invalid user admin from 192.168.0.101 port 55888 ssh2
Jul 5 11:03:10 server sshd[1234]: Accepted password for root from 192.168.0.105 port 57844 ssh2

Possible Fixes:

• For a Failed Password: Ensure that the username exists and confirm that the password used is correct.
• For a Permission Denied (public key): Check that the ~/.ssh/authorized_keys file has the correct public key and verify permissions on the SSH configuration file sshd_config.
• For Too Many Failed Attempts: Look for rate-limiting tools like fail2ban that may be blocking the IP address.

2. Investigating High Disk Usage

Problem:

Your system has run out of space, as indicated by errors such as “No space left on device.”

Log to Check:

While disk usage itself may not be logged directly, you might see side effects in:

• /var/log/syslog
• /var/log/messages

To check which logs are growing rapidly, you can use:

sudo du -sh /var/log/*

Look for Errors:

You can search for relevant messages using:

grep -i "disk full" /var/log/syslog
grep -i "no space" /var/log/syslog

Possible Fixes:

• Use logrotate to rotate large logs, ensuring they do not overwhelm your disk space.
• Clear unnecessary files in directories such as /tmp or /var/cache.
• Use df -h and du -sh to identify the largest disk users and analyze disk usage trends.

3. Service Crashes and Failures

Problem:

A service, such as NGINX, Apache, or MySQL, crashes or refuses to start.

Logs to Check:

General service logs include:

• /var/log/syslog
• /var/log/messages

Service-specific logs may include:

• /var/log/nginx/error.log
• /var/log/mysql/error.log

Example Command:

grep nginx /var/log/syslog

Expected Output:

You might find entries similar to this:

Jul 5 14:20:01 server nginx[4782]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

Possible Fixes:

• Address Already in Use: Another service is likely using port 80. Use lsof -i :80 to identify which service is running on that port.
• Missing Configuration File: Verify the configuration using nginx -t or apachectl configtest to check for syntax errors in the configuration file.
• Out-of-Memory Kills: Check /var/log/syslog for Out-Of-Memory (OOM) messages that may indicate that the service was killed due to resource constraints.

4. Identifying Boot Issues or Kernel Panics

Problem:

The system fails to boot properly or encounters kernel panics during startup.

Logs to Check:

• /var/log/kern.log
• /var/log/dmesg
• /var/log/syslog

Use dmesg to View Kernel Boot Messages:

dmesg | less

Look for:

• Hardware errors (related to disk, RAM, GPU)
• Kernel panics
• Missing kernel modules or drivers necessary for booting

Expected Output Example:

[    1.234567] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Possible Fixes:

• Boot from a live CD or USB and check disk health using tools like fsck.
• Reinstall GRUB or update initramfs to fix bootloader issues.
• Check the fstab file and ensure that UUIDs and paths are correctly specified to avoid filesystem mounting issues.

5. Troubleshooting Cron Jobs

Problem:

Scheduled cron tasks are not executing, or their output isn’t visible as expected.

Logs to Check:

• /var/log/cron.log
• /var/log/syslog

Example Command:

grep CRON /var/log/syslog

Expected Output:

You might see entries like this:

Jul 5 07:15:01 server CRON[1212]: (root) CMD (/usr/local/bin/backup.sh)
Jul 5 07:15:01 server CRON[1213]: (root) CMDOUT (Permission denied)

Possible Fixes:

• Permission Issues: Ensure that the script being run is executable by using chmod +x /usr/local/bin/backup.sh.
• Environment Variables: Remember that cron runs in a minimal shell environment. You may need to set the PATH variable inside the script to ensure it can find all necessary commands.
• Redirect Output: Implement logging within the script to capture errors. You can add redirection within the cron entry itself (e.g., /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1) to log output and errors for easier troubleshooting.

6. Detecting Unauthorized Access or Attacks

Problem:

You suspect brute force attacks or unauthorized access attempts are taking place against your system.

Logs to Check:

• /var/log/auth.log
• /var/log/fail2ban.log
• /var/log/messages

Example Command:

grep "Failed password" /var/log/auth.log | wc -l

Expected Output:

You might see something like:

Jul 5 11:02:43 server sshd[1234]: Failed password for invalid user test from 192.168.0.105
Jul 5 11:02:45 server sshd[1234]: Failed password for root from 192.168.0.105

Possible Fixes:

• Utilize fail2ban or similar tools to automatically block IP addresses that make too many failed login attempts.
• Change the default SSH port to prevent bots from targeting the standard port (22).
• Disable root login via SSH for better security practices.
• Implement public key authentication rather than password-based authentication to enhance security.

7. Application-Level Issues

Problem:

Your custom application crashes, returns HTTP 500 errors, or behaves in an unexpected manner.

Logs to Check:

If you’ve configured rsyslog to capture logs for your application, look in:

• /var/log/myapp.log

Alternatively, you can view general logs at /var/log/syslog.

Example Rule in /etc/rsyslog.d/myapp.conf:

if $programname == 'myapp' then /var/log/myapp.log
& stop

Example Usage:

grep "Exception" /var/log/myapp.log

Possible Fixes:

• Add appropriate logging in your application to capture errors effectively and trace issues when they arise.
• During testing, monitor logs in real-time using tail -f to catch issues as they occur.
• Set up alerts using log monitoring tools to be notified of critical errors or failures.

Tools to Enhance Log Troubleshooting

Utilizing various tools can significantly improve your efficiency in log management and troubleshooting:

• tail -f: Monitor logs in real-time to see new entries as they are made.

  tail -f /var/log/syslog

• journalctl: For systems using systemd, you can view service-specific logs:

  journalctl -u nginx.service --since "1 hour ago"

• logrotate: Prevent logs from becoming excessively large and consuming disk space with log rotation:

  sudo logrotate -d /etc/logrotate.conf

• watch: Run log checks at regular intervals:

  watch -n 2 tail -n 10 /var/log/syslog

Best Practices for Log Troubleshooting

To enhance your effectiveness in troubleshooting through logs, consider adopting these best practices:

• Know Your Logs: Familiarize yourself with which services write to which log files so you can efficiently navigate through logs based on the issue at hand.
• Use Time Filters: Limit your search to recent entries using time filters with commands like grep to focus on the most relevant data.
• Automate Alerts: Utilize tools such as logwatch or third-party logging dashboards to follow the statuses of critical logs and receive alerts about issues proactively.
• Centralize Logging: Employ rsyslog or the ELK stack (Elasticsearch, Logstash, Kibana) to collect, analyze, and visualize logs from multiple systems in a centralized manner.
• Secure Your Logs: Protect log files with proper permissions and user access controls to prevent unauthorized access and tampering.
• Test in Staging: Always conduct tests of cron jobs, services, and log configurations in non-production environments before rolling out changes to production systems.

Conclusion

Troubleshooting in Linux can be significantly streamlined when you understand how to read and interpret logs effectively. With tools like rsyslog, you have access to a robust logging framework that tracks everything happening on your system—from failed SSH attempts to service crashes and kernel errors. By practicing log inspection regularly and establishing a clean log management strategy, you can identify issues faster, maintain system health, and ensure your infrastructure runs smoothly.

Whether you are a seasoned system administrator or just starting your journey into the world of Linux, mastering log management is an essential step in becoming a competent troubleshooter. By leveraging the real-world examples and insights presented in this article, you will be better prepared to tackle issues as they arise, optimizing both your productivity and the performance of your Linux systems.