How to Manage and Analyze Server Logs Effectively

Something breaks on your server at 2 a.m. The site is down, users are complaining, and you have no idea what happened. You SSH in, start poking around, and the first place you go is the logs. Because logs are the only record of what the server was actually doing when things went wrong.

That works fine when the logs are clean, well-organized, and not taking up half your disk. It becomes a problem when logs are scattered across a dozen files, rotated inconsistently, or so bloated with noise that finding the one relevant error means scrolling through thousands of irrelevant lines.

Managing logs well is not glamorous work. But it is the kind of thing that saves hours when something goes wrong – and something always goes wrong eventually. This article covers where logs live, how to read them, how to keep them from becoming a mess, and how to get useful information out of them quickly.

Prerequisites

• You manage Linux servers or work with them regularly.
• You are comfortable with the command line.
• No prior experience with log management tools is needed.

Where Logs Live on Linux

Most system and application logs on Linux end up in /var/log. Inside that directory, different services write to their own files or subdirectories.

A few you will see on almost every server:

• /var/log/syslog or /var/log/messages – general system messages from the kernel and various services. The catch-all for things that do not have their own log file.
• /var/log/auth.log – authentication events. Login attempts, sudo usage, SSH connections. If someone is trying to get into your server, the evidence usually shows up here.
• /var/log/kern.log – kernel messages. Hardware errors, driver issues, out-of-memory events.
• /var/log/nginx/access.log and /var/log/nginx/error.log – web server request logs and errors.
• /var/log/mysql/error.log – database errors and warnings.
• /var/log/dpkg.log – package installation and removal history on Debian-based systems.

Application logs vary. PHP-FPM writes to its own log. Custom applications often write wherever they are configured to. Check your application’s configuration if you cannot find where it is logging.

Reading Logs From the Command Line

Viewing Recent Entries

tail is the most-used command for looking at logs. By default, it shows the last ten lines. The -f flag follows the file in real time – useful when you are watching for errors as they happen.

$ tail -f /var/log/nginx/error.log

For the last 100 lines instead of 10:

$ tail -n 100 /var/log/syslog

head works the opposite way – shows the beginning of a file. Less useful for troubleshooting, more useful when you want to know when logging started or what format the file uses.

Searching Through Logs

grep is how you filter signal from noise. Instead of reading thousands of lines, you pull out only the ones that match something relevant.

Find all errors in the Nginx log:

$ grep "error" /var/log/nginx/error.log

Search across multiple files:

$ grep "failed" /var/log/auth.log /var/log/syslog

Case-insensitive search:

$ grep -i "warning" /var/log/syslog

Show ten lines of context around each match – useful when you need to see what happened before and after an error:

$ grep -C 10 "out of memory" /var/log/kern.log

Combine with tail to search only recent entries:

$ tail -n 500 /var/log/syslog | grep "error"

Filtering by Time With journalctl

On servers using systemd, journalctl gives you structured access to logs from all systemd-managed services in one place. The time filtering alone makes it worth knowing.

Show logs from the last hour:

$ journalctl --since "1 hour ago"

Show logs for a specific service since midnight:

$ journalctl -u nginx --since today

Show logs between two specific times:

$ journalctl --since "2024-03-15 02:00:00" --until "2024-03-15 03:00:00"

When something went wrong at a specific time and you want to see exactly what the server was doing, journalctl with time filters gets you there faster than grepping through individual files.

Log Rotation

Logs grow. Left alone, they will eventually fill your disk. Log rotation is the process of regularly archiving old log data and starting fresh, keeping disk usage under control without losing history entirely.

How logrotate Works

logrotate is the standard tool on Linux for this. It runs on a schedule – usually daily via cron – and processes each log file according to rules you define.

A typical logrotate configuration looks like this:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        nginx -s reopen
    endscript 
}

Breaking that down:

• daily – rotate once per day.
• rotate 14 – keep 14 rotated copies before deleting old ones.
• compress – gzip old log files to save space.
• delaycompress – do not compress the most recently rotated file yet, in case the application is still writing to it.
• create 0640 www-data adm – create a new empty log file with these permissions after rotation.
• postrotate – run this command after rotating. Nginx needs to be told to reopen its log files after they are moved.

Configuration files for individual services live in /etc/logrotate.d/. Check what is already configured before adding your own – most packages install their own logrotate config when they are installed.

Checking Rotation Status

To see what logrotate last did and when:

$ cat /var/lib/logrotate/status

To run logrotate manually and see what it would do without actually doing it:

$ sudo logrotate -d /etc/logrotate.conf

Centralized Log Management

Managing logs on one server is manageable. Managing logs across ten servers means SSHing into each one separately when something goes wrong, which is slow and error-prone. Centralized log management sends logs from all your servers to one place where you can search and analyze everything together.

The ELK Stack

Elasticsearch, Logstash, and Kibana – usually referred to together as the ELK stack – is the most common self-hosted option.

Logstash or a lightweight shipper like Filebeat runs on each server and forwards log data to Elasticsearch, which stores and indexes it. Kibana provides a web interface for searching and visualizing the data.

The practical value of this setup is correlation. An error in your application might be caused by something that happened on the database server two seconds earlier. With logs from both servers searchable in one place, you can see that connection. Searching through separate log files on separate servers, you might never make it.

Managed Log Services

Running your own ELK stack is work. Managed services like Datadog, Logtail, Papertrail, and similar tools handle the infrastructure and give you a web interface with search, alerting, and dashboards out of the box.

For small teams or organizations that do not want to manage logging infrastructure, a managed service trades money for simplicity. The time saved on setup and maintenance usually justifies the cost once you are past a handful of servers.

Alerting on Log Patterns

Manually checking logs only catches problems after you notice something is wrong. Alerting flips this – the monitoring system watches logs continuously and notifies you when something worth caring about appears.

Most centralized log systems support alerting rules. Common things worth alerting on:

• A spike in HTTP 500 errors over a short time window.
• Any occurrence of “out of memory” in kernel logs.
• SSH login attempts from new IP addresses.
• Database connection errors appearing more than a few times per minute.
• Application errors that were not present in the previous hour.

The goal is not to alert on everything – alert fatigue is real and alerts that fire constantly get ignored. Alert on things that represent a meaningful change in behavior or a specific condition that always requires attention.

Practical Troubleshooting Patterns

When something is wrong and you are looking through logs, a few patterns speed up the process.

• • Start with what changed. If the problem started at a specific time, look at what happened in the logs just before that time – not just when the errors started appearing. The cause usually precedes the symptom by seconds or minutes.
  • Narrow by service first. If it is a database-related error, start with the database log. If requests are failing, start with the web server access and error logs. Reading the right log is faster than reading all of them.
  • Look for the first occurrence. When you see the same error repeated many times, scroll back to find where it first appeared. The first occurrence often has more context than the repeated ones.
  • Correlate timestamps across services. If a spike in application errors started at 14:32, look at what the system log, the database log, and the kernel log show around 14:31. An event in one service often has a corresponding event in another.

$ grep "14:3[12]" /var/log/syslog /var/log/mysql/error.log /var/log/nginx/error.log

Conclusion

Logs are the closest thing a server has to a black box recorder. They capture what happened, when it happened, and often why. The value of that record depends almost entirely on how well it is managed – logs that are missing, unreadable, or buried under gigabytes of noise are not much better than no logs at all.

Get rotation configured properly so logs do not fill your disk. Know where each service writes its logs and how to search through them quickly. Set up centralized logging once you are managing more than a handful of servers. Build a small set of meaningful alerts so problems surface before users report them.

None of this is complicated individually. Done consistently, it turns log management from a reactive scramble into something that actually helps you stay ahead of problems.