How to Analyze and Rotate Log Files

Log files are the lifeblood of system troubleshooting, performance monitoring, and security auditing in any Linux environment. These files record system events, application activity, user logins, hardware errors, security alerts, and more. Over time, log files can grow very large, making them difficult to manage and potentially consuming valuable disk space. If left unchecked, uncontrolled log growth can even cause system failures. That’s why analyzing log files to extract useful information and rotating them to manage their size are both essential tasks for administrators and power users.

This guide explains how to analyze log files effectively and how to configure log rotation to keep your Linux system healthy, secure, and organized. You will learn about the main types of logs, the tools for searching and filtering, and practical steps for setting up and testing log rotation.

Prerequisites

• Access to a Linux system (desktop, server, or cloud VM)
• Terminal or shell access
• Sudo privileges for advanced log management or editing system files

Understanding Log Files in Linux

Where Are Log Files Stored?

On most Linux distributions, system and application logs are stored in the /var/log/ directory. Each file or subdirectory within it typically relates to a specific service or component. Common examples include:

• /var/log/syslog or /var/log/messages: Records general system events and messages.
• /var/log/auth.log: Logs authentication attempts and security events.
• /var/log/kern.log: Contains kernel-generated messages.
• /var/log/dmesg: Hardware and boot information from the kernel ring buffer.
• /var/log/apache2/, /var/log/nginx/: Web server logs.
• /var/log/journal/: Persistent log storage for systems using systemd.

Note: The exact log file names and locations may vary between distributions (e.g., Ubuntu uses /var/log/syslog; CentOS uses /var/log/messages).

How to Analyze Log Files

Viewing Logs with Basic Tools

Use cat, less, and more

To read a log file:

$ cat /var/log/syslog
$ less /var/log/syslog
$ more /var/log/syslog

less is especially handy for large files, as you can scroll and search.

Use tail for Recent Entries

To see the last ten lines of a log:

$ tail /var/log/syslog

To watch a log update in real time:

$ tail -f /var/log/syslog

Exit with Ctrl+C.

Use head for the Start of a Log

To see the first ten lines:

$ head /var/log/syslog

Searching and Filtering Log Content

Use grep for Keyword Searches

Find all lines containing “error” (case-insensitive):

$ grep -i error /var/log/syslog

Combine with tail to search recent entries:

$ tail -n 1000 /var/log/syslog | grep -i warning

Use awk and cut for Advanced Filtering

Print the date and message fields from a log:

$ awk '{print $1, $2, $3, $5, $6, $7}' /var/log/syslog

Use sed to Extract Patterns

Remove all lines that don’t mention “fail”:

$ grep fail /var/log/auth.log | sed -n 's/.*fail.*//p'

Analyzing Systemd Journal Logs

Most modern distributions use systemd for service management, which stores logs in a binary format viewable with journalctl.

Basic journalctl Usage

• Show all logs:

  $ journalctl

• Show logs from current boot:

  $ journalctl -b

• Show logs for a specific service (e.g., SSH):

  $ journalctl -u ssh

• Filter by severity (e.g., errors only):

  $ journalctl -p err

• Show logs from a specific time:

  $ journalctl --since "2024-07-01 08:00" --until "2024-07-01 12:00"

Persisting Systemd Journals

Make sure logs survive reboots by enabling persistent storage:

$ sudo mkdir -p /var/log/journal
$ sudo systemctl restart systemd-journald

Using GUI Tools for Log Analysis

Many desktop environments offer graphical tools. For example:

• GNOME Logs: User-friendly interface for browsing and filtering logs.
• KSystemLog: KDE’s graphical log file viewer.

These tools are helpful for users less comfortable with the command line.

Why Rotate Log Files?

Unrotated logs can grow indefinitely, risking disk space exhaustion and making it hard to find relevant events. Log rotation automatically archives, compresses, and deletes old log files on a regular schedule, keeping your system manageable and healthy.

How Log Rotation Works with logrotate

Introduction to logrotate

logrotate is the standard tool for managing log rotation on Linux. It is typically run daily via a cron or systemd timer.

Configuration Files

• Global config: /etc/logrotate.conf
• Per-service configs: /etc/logrotate.d/

Each configuration file specifies which logs to rotate, how often, how many old logs to keep, whether to compress them, and what actions to take before/after rotation.

Example logrotate Configuration

Suppose you run a custom application that writes logs to /var/log/myapp/*.log. Here’s how you can set up rotation:

1. Create a configuration file:

   $ sudo nano /etc/logrotate.d/myapp

2. Add the following:

   /var/log/myapp/*.log {
       daily
       rotate 14
       compress
       delaycompress
       missingok
       notifempty
       create 0640 root adm
       postrotate
       systemctl reload myapp
       endscript
   }
   

• daily: Rotate every day.
• rotate 14: Keep 14 old log files.
• compress: Gzip old logs.
• delaycompress: Compress the file one day after rotation.
• missingok: Don’t error if the log is missing.
• notifempty: Don’t rotate empty logs.
• create 0640 root adm: Set permissions and ownership for new logs.
• postrotate/endscript: Run a command after rotation.

Forcing Manual Log Rotation

You can test or force log rotation at any time:

$ sudo logrotate -f /etc/logrotate.conf

Check the status and last rotation:

$ cat /var/lib/logrotate/status

Rotating systemd Journal Logs

If you rely on the systemd journal, configure retention and rotation in /etc/systemd/journald.conf. Common options:

• SystemMaxUse=500M: Maximum disk space for all journals.
• SystemMaxFileSize=100M: Maximum size per journal file.
• MaxRetentionSec=2week: Maximum retention time.

After editing, reload the daemon:

$ sudo systemctl restart systemd-journald

Best Practices for Log Management

• Review logs regularly for signs of errors, warnings, or security incidents.
• Set up automated alerts for critical log events using tools like logwatch, fail2ban, or rsyslog.
• Archive important logs to an external drive or remote server if you need long-term retention for compliance.
• Monitor disk space to avoid outages caused by log file growth.
• Secure log files by setting restrictive permissions, especially for sensitive logs.
• Document custom logrotate rules for troubleshooting and audits.

Conclusion

Being able to analyze and rotate log files is a vital skill for any Linux user or administrator. Use command-line tools like tail, less, grep, and journalctl to quickly spot problems or monitor system health. Rely on logrotate to keep logs manageable and your storage under control. With these best practices, you ensure your system stays responsive, secure, and easier to maintain over the long term.