Logs in production: the invisible cost of noise

Thu, 21 May 2026 10:51:41 -0300

Another chapter in the GenServer the “rabbit hole” series. No GenServer this time — just logs. But the same principle: the abstraction looks simple, and almost no one realizes they’re using it wrong until the bill arrives.

Picture the scene: 2 a.m., a pod restarted and the alert woke someone up. They open Loki, filter by the app label, and the first browser screen looks like this:

12:45:33Z info [InboundWebhook] - Income message: %{"path" => "proxy/977671907166781440/example.cdn.com/v3/openlive/CA2585112_5_2", ...}
12:45:33Z info Sent 200 in 44ms
12:45:33Z warn [ExternalProvider EventWorker] - Event already processed, skipping: "869247060431936_2026-05-21..."
12:45:33Z warn [SetAuthToken] Resource ba8bf8f3-5ceb-4d7d-a318-f0a88e228f41 send token 010000000000003D and AuthToken is not found
12:45:33Z info Sent 200 in 12ms
12:45:33Z warn [ExternalProvider EventWorker] - Event already processed, skipping: "865478070223930_..."
12:45:33Z info Sent 200 in 39ms
12:45:33Z error Error creating resource: "network exception"
12:45:33Z info Sent 200 in 22ms
12:45:33Z warn [SetAuthToken] Resource 38d916e1-1ee7-4f91-9f96-db6ff14c0bd7 send token 0100F68F9DD034FB and AuthToken is not found
... (70+ more lines per second)

The window covers 1 second. To understand the restart, you need what happened in the last 5 minutes. Good luck.

Observability on DanielWS

Logs in production: the invisible cost of noise