top of page


Stop Scaling Your Spark Cluster: How File Layout Fixes Slow Lakehouse Pipelines
There is nothing quite like watching a tiny 200 MB batch update drag on for 50 painful minutes while your Spark cluster burns money doing practically zero real work. We turned that exact pipeline into a fast 5-minute run on half the nodes not by tweaking obscure JVM options, but by fixing the underlying file layout. A 200 MB Update Shouldn’t Take 50 Minutes: Data Lakehouse Optimization Techniques That Actually Work Here is a number that bothered me for weeks: 200 MB. That was
8 min read
bottom of page