If you have ever run DESCRIBE HISTORY on a Delta table that receives streaming data every 60 seconds and watched it either hang for hours or crash with an OutOfMemoryError, you are not alone and you are not doing anything wrong. The problem is architectural, and once you understand the internals, the fix becomes a lot clearer. Here is what I learned after digging into why this happens and what you
State that survives a docker compose down is one of those things you don't think about, until your test suite needs it, your local dev needs it, and your CI pipeline absolutely doesn't. LocalStack handles persistence with one switch (PERSISTENCE=1) and it's a Pro-only feature. Floci ships four storage modes, all free, all in core, with per-service overrides. Pick the right tradeoff for the job.