Linux Administration & Bash Automation

This covers the defensive bash habits, set -euo pipefail, log rotation, health checks, and error trapping, that keep an unattended script or scheduled job from failing silently.

set -euo pipefail and why it matters

Every bash script I write starts with this line, right after the shebang:

#!/usr/bin/env bash
set -euo pipefail

Each flag closes a real hole:

• -e exits the script the instant a command fails. Without it, a failed cd /app/releases/current still lets the next line run in the wrong directory.
• -u treats an unset variable as an error. This catches the classic typo ($RELEASE_DIR misspelled as $RELEASE_DR) before it silently expands to an empty string and turns rm -rf "$RELEASE_DR/old" into rm -rf /old.
• pipefail makes a pipeline fail if any command in it fails, instead of only checking the last one's exit code. Without it, cat missing_file.txt | sort > sorted.txt reports success. cat fails, sort sorts the empty input and exits 0, and the script carries on with an empty file.

As soon as a script uses an ERR trap (more on that at the end), I switch it to set -Eeuo pipefail. The extra -E (errtrace) makes functions, command substitutions and subshells inherit the trap. Without it, a failure inside a function still exits the script, but your trap never runs.

A log-rotation script that doesn't take down the disk with it

The other incident I keep running into is an app (or a script) logging to a flat file forever, until one day /var/log, or worse /, hits 100% and takes the host down with it. logrotate handles this for most system services, but plenty of app logs, cron output and custom job logs never get wired into it. For apps that write a new log file per day or per run, I drop in this script as a stopgap:

#!/usr/bin/env bash
set -euo pipefail
 
LOG_DIR="/var/log/myapp"
RETENTION_DAYS=14
 
if [[ ! -d "$LOG_DIR" ]]; then
  echo "ERROR: log directory $LOG_DIR does not exist" >&2
  exit 1
fi
 
# Compress logs not modified in the last 24 hours (not yet compressed)
find "$LOG_DIR" -maxdepth 1 -type f -name "*.log" -mmin +1440 -exec gzip {} \;
 
# Delete compressed logs past retention (-mtime +14 means 15 days or older)
find "$LOG_DIR" -maxdepth 1 -type f -name "*.log.gz" -mtime "+${RETENTION_DAYS}" -print -delete
 
echo "Log cleanup complete: $(date -Iseconds)"

Watch the find time units. -mtime +1 doesn't mean "older than a day". find rounds ages down to whole days, so +1 only matches files at least 48 hours old. If you mean 24 hours, use -mmin +1440.

This script doesn't cover the case I opened with, which is a single log file the app keeps open and appends to forever. Its modification time is always fresh, so find -mtime never matches it. For that file, add a size-based logrotate rule with copytruncate. It copies the file and truncates it in place, so the app keeps writing to the same file descriptor:

# /etc/logrotate.d/myapp
/var/log/myapp/app.log {
    size 100M
    rotate 7
    compress
    missingok
    notifempty
    copytruncate
}

logrotate normally runs once a day, so a size rule is only checked daily. If the file can grow faster than that, run logrotate /etc/logrotate.d/myapp from a more frequent timer. copytruncate can lose the few lines written between the copy and the truncate. That's an acceptable trade for an app you can't signal to reopen its log.

A health-check script you can actually schedule

For anything client-facing, I want something running on a schedule that hits the real endpoint, rather than only checking systemctl status:

#!/usr/bin/env bash
set -euo pipefail
 
URL="https://api.example.com/healthz"
TIMEOUT=5
EXPECTED_CODE=200
 
status_code=$(curl --silent --output /dev/null --write-out "%{http_code}" \
  --max-time "$TIMEOUT" "$URL" || true)
 
if [[ "$status_code" -ne "$EXPECTED_CODE" ]]; then
  echo "HEALTH CHECK FAILED: $URL returned $status_code" >&2
  exit 1
fi
 
echo "OK: $URL returned $status_code"
exit 0

The || true keeps set -e from killing the script when curl itself fails (DNS error, timeout, refused connection). curl already prints 000 as the status code in that case, so don't echo a fallback as well, or you'll get 000000.

What matters here is the exit code: it's what lets cron, systemd, or a monitoring agent (Nagios, Prometheus's blackbox exporter) decide whether to page someone.

Cron vs systemd timers

For simple, single-host schedules, cron is still fine:

*/5 * * * * /usr/local/bin/healthcheck.sh >> /var/log/healthcheck.log 2>&1

I still see this line shipped without the >> ... 2>&1 half, which means any output, including the error message you need, vanishes.

But once a job needs dependency ordering, retries, or resource limits, I reach for a systemd timer instead. Same if you just want its logs in journalctl rather than a flat file you have to rotate yourself:

# /etc/systemd/system/healthcheck.service
[Unit]
Description=API health check
 
[Service]
Type=oneshot
ExecStart=/usr/local/bin/healthcheck.sh

# /etc/systemd/system/healthcheck.timer
[Unit]
Description=Run healthcheck.service every 5 minutes
 
[Timer]
OnCalendar=*:0/5
Persistent=true
 
[Install]
WantedBy=timers.target

OnCalendar=*:0/5 fires every five minutes on the wall clock (:00, :05, :10...). With Persistent=true, if the machine was off when a run was due, systemd runs it right after boot. Persistent= only works with OnCalendar= timers and is silently ignored on monotonic ones like OnUnitActiveSec=. You can check the expression with systemd-analyze calendar '*:0/5'.

Enable it with systemctl enable --now healthcheck.timer. From then on, journalctl -u healthcheck.service gives you every run, its output, and its exit code, and it survives a reboot without a stray @reboot cron entry.

Logging and trapping errors so failures don't stay silent

The last habit is what keeps a failure from staying silent for weeks: make failure loud on purpose.

#!/usr/bin/env bash
set -Eeuo pipefail
 
LOG_FILE="/var/log/myapp/etl.log"
 
log() {
  echo "$(date -Iseconds) $*" | tee -a "$LOG_FILE"
}
 
on_error() {
  local exit_code=$?
  local line_no=$1
  log "ERROR: script failed at line $line_no with exit code $exit_code"
  # send a Slack/PagerDuty alert here instead of hoping someone reads the log
  exit "$exit_code"
}
 
trap 'on_error $LINENO' ERR
 
log "Starting ETL run"
# ... actual work happens here ...
log "ETL run completed successfully"

trap ... ERR runs whenever a command fails (with -e active), and $LINENO tells you where. The -E in set -Eeuo pipefail makes it fire for failures inside functions too. With plain set -euo pipefail, a failing command inside a function exits the script without calling on_error, and you're back to the silent failure this section is trying to prevent. With the trap in place, you stop finding out weeks later that the job died somewhere. You get an alert at the moment it fails, saying it stopped at line 42 at 03:14.

They're six habits, applied every time. The difference is whether you hear about a failure from a monitoring alert within minutes, or from someone else weeks later.