An opinionated list of Python frameworks, libraries, tools, and resources
If you have ever run DESCRIBE HISTORY on a Delta table that receives streaming data every 60 seconds and watched it either hang for hours or crash with an OutOfMemoryError, you are not alone and you are not doing anything wrong. The problem is architectural, and once you understand the internals, the fix becomes a lot clearer. Here is what I learned after digging into why this happens and what you
I needed to coordinate background scripts running across different machines. The obvious answer was Redis. Everyone uses Redis for this. The tutorials all use Redis. The Stack Overflow answers all say "just use Redis." So I looked at what deploying Redis would actually cost me: A running Redis server I had to maintain A broker to connect workers to it Celery or RQ on top of that Memory-based stora