1.4T-Event Spotify Dashboard Machine, Perfect Query Plans, Databricks Pro Playbook, and Agents for Data Analysts
Real-world experience to ace Databricks DE Pro, How Spotify handles their dashboards, resilient scraping tactics, why “optimal” query plans mislead, and AI agents for enterprise data.
⚙️ When PostgreSQL's "Perfect" Query Plans Aren't
Tomas Vondra shared a fascinating case study this week that every data engineer should bookmark. A PostgreSQL query planner was confidently choosing a 5-second index scan when a 2-second sequential scan would have won.
The details reveal why database optimization remains more art than science: Bitmap scans often outperform index scans by 10x for 1-5% selectivity ranges.
Cost estimates built on coarse statistics make "perfect" planning impossible. Hardware quirks like prefetching and cache warmth can shift the winner mid-execution. "The optimizer's best guess may only be 'good enough,'" as Vondra puts it, and this resonates with anyone who's spent time performance-tuning production queries.
What this means for data engineers is clear: even with decades of development, database optimizers operate with incomplete information by design. They're fast because they use simplified stats, but that simplification comes with blind spots.
The implication is that we should benchmark our critical queries ourselves. Don't assume the planner knows best, especially in cloud environments where the hardware abstraction adds even more complexity to the cost model.
🎓 How to Ace the Databricks Pro Exam?
We came across a cool Reddit thread this week from someone who absolutely crushed the Databricks Data Engineer Professional exam with a 95% score. What struck us wasn't just the impressive result, but their honest breakdown of what actually moved the needle versus what felt like busy work.
The conventional wisdom says to rely heavily on courses like Derar Alhussein's Udemy offering. This test-taker had a different take: skim it for breadth, but don't expect it to carry you across the finish line.
The real meat was in drilling down on Delta Lake, Spark Structured Streaming, and security concepts. These aren't just exam topics - they're the daily reality of most data engineering teams we encounter.
What we found most valuable was their emphasis on rotating through multiple practice exams. Not because repetition breeds success, but because it exposes the outdated "gotcha" syntax questions that still lurk in these certifications.
The data ecosystem moves fast, but exam content often lags behind.
🎧 Inside Spotify's 1.4 Trillion-Event Data Engine
Sometimes you encounter a number that makes you pause and reconsider your assumptions about scale. For us this week, it was Spotify's 1.4 trillion events per day. Not per month. Per day. What fascinated us wasn't just the raw volume, but how they've architected around it.
The stack reads like a greatest hits of modern data infrastructure: Pub/Sub feeding into Beam and Flink pipelines, orchestrated through 38,000 Flyte workflows, with data landing in BigQuery and GCS/HDFS to serve roughly 5,000 Looker and Tableau dashboards for 6,000 users.
The migration story from Luigi/Flo to Flyte particularly caught our attention. Fragmented orchestration and visibility gaps - these are the unglamorous problems that don't make conference talks but absolutely cripple teams at scale.
Spotify's solution was refreshingly standard: battle-tested GCP primitives plus a scheduler that actually works. When you're processing 1.4 trillion events daily, the temptation to over-engineer is immense. Instead, they've doubled down on observability and reliability over cleverness.
🕷️ Web Scraping at Scale: The Uncomfortable Truths
A community discussion this week perfectly captured why large-scale web scraping feels like an endless game.
The usual suspects were all there: rotating proxies, CAPTCHA solvers, the perpetual arms race between scrapers and anti-bot measures.
Rotating proxies and CAPTCHA solvers buy you time, but they still break. Alerting becomes mandatory because failure is inevitable, not just possible.
The wisdom in the thread gravitated toward two key insights: Hidden or internal APIs are absolute gold - always exhaust these options before building brittle DOM parsers.
And past a certain pain threshold, most teams simply buy the data instead of maintaining Selenium farms. This maps to a broader pattern we see in data engineering. We often treat scraping as a permanent ETL source when it should be viewed as a stopgap. The smart teams we know invest in robust monitoring and plan their exit strategy (paid APIs, direct partnerships) from day one.
🤖 Our Next Lakehouse Days: AI Agents for Enterprise Data
We're hosting another hands-on meetup in Bengaluru, and this time we're diving deep into something every data team is grappling with: AI agents that actually work at enterprise scale.
The agenda goes beyond the usual "ChatGPT + your database" demos. We're tackling the real problems: how do you handle messy schemas without requiring perfect catalogs? How do you avoid the join errors that plague most text-to-SQL attempts? And critically, how do you move from basic query-and-result to genuinely conversational, context-aware workflows?
Talks we're hosting:
Bharath Harish (our Head of Product) will break down how we built Text-to-SQL agents with 95%+ accuracy for enterprise scale. He'll cover knowledge graph-driven relationship discovery, SQL engine-like planning to reduce hallucinations, and performance techniques like partition key usage and optimized CTEs.
Harsh Sharma from Flipkart will share why his team chose Milvus (open-source vector database) over traditional approaches for recommendation systems, plus the indexing and retrieval strategies that make it work at e-commerce scale.
Register here: https://lu.ma/8ufzg6gi
Community & Events
