“We have failover.” That sounds reassuring. But when real failure hits… many systems still go down — hard. Why? Because failover is easy to configure — but extremely hard to make reliable at global scale. Here are the most common ways failover fails in production: RDS Multi-AZ enabled Kubernetes failover configured Looks good on paper. Reality: Takes minutes instead of seconds Gets stuc
Tbh I had no idea this was even a thing until recently. I've been working with Rails for a while now and somehow never came across it. So let me explain it the way I understood it. You know how we normally do associations in Rails, User has many Posts, Post belongs to User. Two different models, two different tables. Simple. But what if a model needs to reference itself? Like same table, same mode
Is your website throwing 502 errors whenever an external API starts lagging? It is a common engineering grind where slow dependencies choke your server and kill your response times. The fix is not adding more resources. It is about changing how you handle work. Stop making users wait for external processes to finish. Offload heavy tasks to background jobs and queues. Distinguish between workers
The Signal: The Legally Binding Hallucination The failure wasn't that the LLM hallucinated—it’s that it was allowed to speak directly to the customer and the database without a chaperone. When you give a non-deterministic guest unregulated access to your deterministic house, you are legally and financially responsible for the fire. We need to stop treating AI as an open-ended "chat" interface and
Comments
The on-call alert at 02:14 said auth_5xx_rate spiked from 0.01 to 31.4. Not a deploy window. Not a traffic spike. Just thirty-one percent of authenticated requests failing for ~four minutes, then back to baseline. The cause was a JWKS rotation on the issuer side. New keys came in. Old keys went out. Caches in our service didn't refresh fast enough. Tokens signed with the new key were rejected beca
Why I built another Ruby test runner inspired by Playwright Test Ruby already has great testing tools. If you are building Rails applications today, you probably use one of these combinations: RSpec + Capybara Minitest + Capybara Rails system tests Maybe Selenium, Cuprite, Ferrum, or Playwright through Ruby bindings These tools are mature, battle-tested, and widely used. So the natural question
Every team experiences incidents. The teams that grow stronger from them are the ones that take postmortems seriously — not as blame sessions, but as structured learning opportunities. Yet most postmortems end up as a wall of text nobody reads twice, filed away and forgotten until the same incident happens again six months later. This guide walks you through writing postmortems that genuinely chan