The logs nobody reads
Most log lines are written once and read never. They cost disk, they cost egress if you ship them somewhere, and they make the lines that matter harder to find.
Three questions
Before adding a log line, I've started asking:
- What would I do differently if I saw this?
- Would I find it, given I'd be searching at 2am with a vague symptom?
- Does it contain anything I'd regret storing?
The third one is the one people skip. Access logs in particular have a habit of capturing things nobody intended: tokens passed as query parameters, session identifiers, password reset links. Once that's in a log aggregator with a ninety-day retention, it's in ninety days of backups too.
Query strings are the usual culprit
The default nginx combined format logs $request, which includes the full query string. If any API on that host accepts a credential as a parameter — and eventually one will — it lands in plaintext.
Logging $uri instead drops the query while keeping the path:
log_format noquery '$remote_addr - $remote_user [$time_local] '
'"$request_method $uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent"';Apply it per-server rather than globally, so you keep full detail where it's harmless.
Retention is a feature
The instinct is to keep everything in case it's useful. In practice, logs older than a couple of weeks are almost never consulted for debugging — only for audit, and audit has different requirements and usually a different system.
Short retention on application logs, long retention on a small number of deliberately chosen events. That split is worth more than any amount of clever querying.