Apache Stateful Functions is one of the quietly powerful frameworks in the Flink ecosystem - durable per-key state, exactly-once messaging, polyglot remote functions, all on top of Apache Flink. It's also been functionally dormant since October 2024, and it doesn't run on Flink 2.x. We needed it on Flink 2.x. So we maintained the continuation: kzmlabs/flink-statefun. This post is the why and the h
At 3:17 AM on a Tuesday in Q3 2024, our production Kotlin 2.0 microservice fleet hit a 92% memory utilization threshold across 140 nodes, traced to a silent coroutine leak in Ktor 2.2’s request pipeline that had been bleeding 12MB of heap per second for 72 hours. We lost $14k in SLO credits before we found the root cause. A Couple Million Lines of Haskell: Production Engineering at Mercury (78 p