Logs in production: the invisible cost of noise
Another chapter in the GenServer the “rabbit hole” series. No GenServer this time — just logs. But the same principle: the abstraction looks simple, and almost no one realizes they’re using it wrong until the bill arrives. Picture the scene: 2 a.m., a pod restarted and the alert woke someone up. They open Loki, filter by the app label, and the first browser screen looks like this: 12:45:33Z info [InboundWebhook] - Income message: %{"path" => "proxy/977671907166781440/example.cdn.com/v3/openlive/CA2585112_5_2", ...} 12:45:33Z info Sent 200 in 44ms 12:45:33Z warn [ExternalProvider EventWorker] - Event already processed, skipping: "869247060431936_2026-05-21..." 12:45:33Z warn [SetAuthToken] Resource ba8bf8f3-5ceb-4d7d-a318-f0a88e228f41 send token 010000000000003D and AuthToken is not found 12:45:33Z info Sent 200 in 12ms 12:45:33Z warn [ExternalProvider EventWorker] - Event already processed, skipping: "865478070223930_..." 12:45:33Z info Sent 200 in 39ms 12:45:33Z error Error creating resource: "network exception" 12:45:33Z info Sent 200 in 22ms 12:45:33Z warn [SetAuthToken] Resource 38d916e1-1ee7-4f91-9f96-db6ff14c0bd7 send token 0100F68F9DD034FB and AuthToken is not found ... (70+ more lines per second) The window covers 1 second. To understand the restart, you need what happened in the last 5 minutes. Good luck. ...