Build a Self-Healing Server Script That Restarts Dead Services

Your web server will crash at 3am at some point. Someone will notice, file a ticket, and someone will have to log in and restart it. Or you can build a script that does this automatically — monitors your services, restarts them when they die, and tells you what happened. This is that script.

How the monitoring loop works

The core of the script is a while loop that checks whether a service is listening on its port. If the port is open, the service is running. If it is not, the script counts up toward a retry limit, then triggers a restart.

Use netcat to check if a port is open:

nc -z 127.0.0.1 8080 && echo "UP" || echo "DOWN"

The -z flag makes netcat scan without sending data — it just checks whether the connection succeeds. The exit code is 0 on success, non-zero on failure. That exit code is what your shell script reads with && or ||.

The restart command

When the retry limit is reached, the script calls systemctl to restart the service:

systemctl restart your-service

For the restart to work without manual intervention, the script must run as root or as a user with sudo permissions for systemctl. Running as root is simpler for a monitoring script. If your service runs as a non-root user, you can target the restart more precisely with systemctl --user restart yourservice.

Retry limits and why they matter

A service might hiccup for a few seconds and recover on its own. If you restart it the moment the port goes down, you might be interrupting a clean startup sequence. The retry limit gives the service a window to recover.

The script checks the port every N seconds, up to a maximum number of attempts. Only when all attempts are exhausted does it trigger the restart. Between attempts it logs the failure count so you know how long the service was down.

Sending a Telegram alert on failure

The script sends you a message when it is about to restart a service and again when the restart succeeds or fails. This requires a Telegram bot — create one with BotFather, get the bot token, and get your chat ID by messaging the bot directly or using a tool like getids.me.

send_alert() {
    local message="$1"
    curl -s -X POST "https://api.telegram.org/bot${BOT_TOKEN}/sendMessage" \
        -d "chat_id=${CHAT_ID}" \
        -d "text=${message}" > /dev/null 2>&1
}

The > /dev/null 2>&1 at the end suppresses all output so the curl call does not pollute your log file. Telegram messages mean you see the alert on your phone without needing a separate monitoring dashboard.

The complete script

#!/bin/bash
set -euo pipefail

SERVICE_NAME="your-service"
SERVICE_PORT=8080
CHECK_HOST="127.0.0.1"
MAX_RETRIES=3
RETRY_DELAY=10
TELEGRAM_BOT_TOKEN="YOUR_BOT_TOKEN"
TELEGRAM_CHAT_ID="YOUR_CHAT_ID"
LOG_FILE="/var/log/self-heal.log"

send_alert() {
    local message="$1"
    curl -s -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
        -d "chat_id=${TELEGRAM_CHAT_ID}" \
        -d "text=${message}" > /dev/null 2>&1
}

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

is_service_up() {
    nc -z "$CHECK_HOST" "$SERVICE_PORT" 2>/dev/null
}

restart_service() {
    log "Restarting $SERVICE_NAME..."
    systemctl restart "$SERVICE_NAME" 2>&1 | tee -a "$LOG_FILE"
}

log "Starting self-healing monitor for $SERVICE_NAME on port $SERVICE_PORT"
retry_count=0

while true; do
    if is_service_up; then
        [ $retry_count -ne 0 ] && log "$SERVICE_NAME recovered" && retry_count=0
        retry_count=0
    else
        retry_count=$((retry_count + 1))
        log "$SERVICE_NAME is DOWN (attempt $retry_count/$MAX_RETRIES)"

        if [ $retry_count -ge $MAX_RETRIES ]; then
            log "MAX RETRIES reached. Attempting restart..."
            send_alert "ALERT: ${SERVICE_NAME} is down after ${MAX_RETRIES} checks. Restarting..."
            restart_service
            sleep 5

            if is_service_up; then
                log "Restart succeeded"
                send_alert "OK: ${SERVICE_NAME} restarted successfully"
                retry_count=0
            else
                log "Restart FAILED"
                send_alert "CRITICAL: ${SERVICE_NAME} restart failed"
                retry_count=0
            fi
        fi
    fi

    sleep "$RETRY_DELAY"
done

Save this as /usr/local/bin/self-heal.sh, make it executable, and create a systemd service for it so it starts on boot and runs in the background.

Running the script as a systemd service

Create a unit file at /etc/systemd/system/self-heal.service:

[Unit]
Description=Self-healing service monitor
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/self-heal.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable self-heal
sudo systemctl start self-heal

What this actually looks like in practice

Here is the log from a real test. A Node.js service on port 8889 was killed at 18:37:22. The monitor detected it at the next check, counted three failed attempts, triggered the restart at 18:37:42, and confirmed the port was back up at 18:37:48:

[2026-07-08 18:37:12] Starting self-healing monitor for test-service on port 8889
[2026-07-08 18:37:22] test-service is DOWN (attempt 1/3)
[2026-07-08 18:37:32] test-service is DOWN (attempt 2/3)
[2026-07-08 18:37:42] test-service is DOWN (attempt 3/3)
[2026-07-08 18:37:42] MAX RETRIES reached. Attempting restart...
[2026-07-08 18:37:43] Restarting test-service...
[2026-07-08 18:37:48] Restart succeeded

The Telegram alert arrived on the phone before the log finished writing. By the time the human saw the message, the service was already back up.

When to use a script versus systemd watchdog

systemd has a built-in watchdog mechanism. Set WatchdogSec=30 in your service unit and systemd sends the service a watchdog ping every 30 seconds. If the service does not respond, systemd restarts it. This is cleaner than a shell script because it is built into the service manager.

The shell script approach is better when you want one monitor watching multiple services, when you want Telegram alerts with your own formatting, or when you need custom restart logic (like restarting a database before the API server). For a single well-behaved service, systemd watchdog is the simpler choice.