Your logs are eating your budget. Cut the bill without going blind.

Observability bills keep climbing, and the easy fix is to cut retention. Then the 3 AM page comes and the evidence is gone.

Here's how to cut the bill without going blind, from the carousel I posted on Instagram and LinkedIn, plus the log-bill checklist people asked for.

One debug line, $1.1 million

Log volume is cost. In a QCon talk in March 2024 ("Million Dollar Lines of Code"), Erik Peterson of CloudZero described a well-intentioned debug line that echoed everything into CloudWatch. That single function cost $1.1 million in log writes since it was deployed. Debug logging left on in production is the quiet one.

Five ways to cut the bill

1. Sample the noise. Debug and info logs are most of the volume and rarely read. Sample at the source, and keep 100% of errors.
2. Tier your retention. Don't pay hot-storage prices for logs nobody opens after a week. Hot for 7 days, warm for 30, then a cheap archive.
3. Count with metrics, not logs. Searching logs to count requests is the priciest counter there is. Emit a metric; log only the exceptions.
4. Tail-sample your traces. Storing every happy-path trace buries the ones that matter. Keep slow and failed traces, sample the rest.
5. Watch label cardinality. One user_id label can turn one metric into millions of series. Bounded labels only; IDs go in traces.

The log-bill checklist

1. Find: see where the money goes

• Ingest per log group or service, top 5 first
• Split by level: how much is DEBUG in production?
• Cost by environment (prod vs the rest)
• 30-day trend: what grew, and which deploy?
• Who owns each big group? Put a name on it.

2. Cut: stop paying for noise

• Debug logging OFF in production by default
• Log a summary, not the whole request or event payload
• Sample high-volume success logs (keep all errors)
• Drop health-check and heartbeat lines
• Shorter retention for non-critical groups
• Never log secrets or personal data

3. Guard: keep it from creeping back

• A log-budget alert per service (ingest per day)
• Review any new log line in a hot path
• A runtime log-level switch and a kill switch for runaway logs
• A weekly 10-minute ingest review
• AI agents: log decisions and tool calls, not full prompts

Cut the noise. Never the evidence you need at 3 AM.

Sources

• "Million Dollar Lines of Code", Erik Peterson (CloudZero), QCon, March 2024, published on InfoQ.

All opinions are my own.