Most cloud sustainability tools are built for sustainability officers. They pull three-month-old billing data, run it through a proprietary model, and produce a PDF that engineers never see. By the time you know your us-east-1 cluster emits twice as much as us-west-2 would have, it's been running for a quarter. The architecture is locked in. The carbon is already burnt. The only moment you can act
If you have ever run DESCRIBE HISTORY on a Delta table that receives streaming data every 60 seconds and watched it either hang for hours or crash with an OutOfMemoryError, you are not alone and you are not doing anything wrong. The problem is architectural, and once you understand the internals, the fix becomes a lot clearer. Here is what I learned after digging into why this happens and what you