Elixir on DanielWS

Logs in production: the invisible cost of noise

Thu, 21 May 2026 10:51:41 -0300

Another chapter in the GenServer the “rabbit hole” series. No GenServer this time — just logs. But the same principle: the abstraction looks simple, and almost no one realizes they’re using it wrong until the bill arrives.

Picture the scene: 2 a.m., a pod restarted and the alert woke someone up. They open Loki, filter by the app label, and the first browser screen looks like this:

12:45:33Z info [InboundWebhook] - Income message: %{"path" => "proxy/977671907166781440/example.cdn.com/v3/openlive/CA2585112_5_2", ...}
12:45:33Z info Sent 200 in 44ms
12:45:33Z warn [ExternalProvider EventWorker] - Event already processed, skipping: "869247060431936_2026-05-21..."
12:45:33Z warn [SetAuthToken] Resource ba8bf8f3-5ceb-4d7d-a318-f0a88e228f41 send token 010000000000003D and AuthToken is not found
12:45:33Z info Sent 200 in 12ms
12:45:33Z warn [ExternalProvider EventWorker] - Event already processed, skipping: "865478070223930_..."
12:45:33Z info Sent 200 in 39ms
12:45:33Z error Error creating resource: "network exception"
12:45:33Z info Sent 200 in 22ms
12:45:33Z warn [SetAuthToken] Resource 38d916e1-1ee7-4f91-9f96-db6ff14c0bd7 send token 0100F68F9DD034FB and AuthToken is not found
... (70+ more lines per second)

The window covers 1 second. To understand the restart, you need what happened in the last 5 minutes. Good luck.

GenServer the "rabbit hole" — Part 2: The Silent Cascade

Sat, 16 May 2026 14:10:51 -0300

Continuation of GenServer the “rabbit hole”. If you haven’t read it yet, I recommend going there first — we’ll start from the same genserver_study project and the same concepts of mailbox, supervisor and max_restarts.

In the previous article we explored how a “misbehaving” GenServer can shut down the whole supervision tree when it bursts the supervisor’s max_restarts. We showed how messages are lost in the mailbox and how “Let it crash” isn’t exactly a blank check to let processes break randomly.

GenServer — the "rabbit hole"

Sat, 19 Jul 2025 13:36:37 -0300

When we explore Elixir’s strengths as a programming language, two stand out compared to other languages on the market:

🚀 3. Concurrency and Scalability

Uses actors (processes) for concurrency (each with its own memory and message queue).

Millions of processes can run concurrently with low overhead.

Built-in tools for distribution across multiple nodes.

🛠 4. Fault-Tolerance and Supervision Trees

“Let it crash” philosophy: Failures are expected and isolated.

Supervision trees: Automatically restart failing processes, making systems self-healing and resilient.

GenServer is an abstraction over the Erlang VM’s process model, shipped by the language core, that connects directly to Elixir’s concurrency and supervision tree features. Because it’s a common pillar in the architecture of Elixir systems, and because it’s covered “prematurely” in the official docs (Client-server communication, OTP Concurrency), newcomers tend to model their first Elixir solutions using the abstraction provided by the GenServer module. It takes a while before the official doc gets to its first disclaimer for the unaware: When (not) to use a GenServer.

Elixir on DanielWS

Logs in production: the invisible cost of noise

GenServer the "rabbit hole" — Part 2: The Silent Cascade

GenServer — the "rabbit hole"

🚀 3. Concurrency and Scalability

🛠 4. Fault-Tolerance and Supervision Trees