How we moved from a fragile loop-based payout system to a reliable, idempotent, and traceable architecture. On paper, payouts sound simple: Customer places an order Platform collects payment Platform pays the seller That's it. Until you try to do it at scale. In any marketplace or fintech system, money flows across multiple parties: Sellers / vendors Delivery partners Platform fees Discounts, vouc
I still remember where i was when the email came in. December 25th. Christmas morning. Phone in hand while having breakfast, and there is an email from our client's CTO. No greetings, Just "We're terminating the contract. Our legal team will be in touch" We lost a 120K a year contract. On a Christmas morning because of a date calculation bug that none of us, not a person on a team of 5 experienced
At 3:17 AM on a Tuesday in Q3 2024, our production Kotlin 2.0 microservice fleet hit a 92% memory utilization threshold across 140 nodes, traced to a silent coroutine leak in Ktor 2.2’s request pipeline that had been bleeding 12MB of heap per second for 72 hours. We lost $14k in SLO credits before we found the root cause. A Couple Million Lines of Haskell: Production Engineering at Mercury (78 p