🔍 “Show me your GitHub” - The Quest for Real Data Engineering Projects

A semi-senior data engineer asked the r/dataengineering community to share production-grade GitHub repos, hoping to see how senior engineers structure projects, choose tools, and design architectures. The responses revealed a fundamental problem: most real data engineering work is proprietary.

• The standout contribution: tobymao’s sqlglot, a SQL parser and transpiler that handles dialect translation across 20+ SQL variants (it’s what powers dialect conversion in DBT, Airflow, and a bunch of other tools you probably use daily)

• The technical gap is real: junior engineers are learning from tutorials with toy datasets, not systems that handle petabyte-scale data with complex SLA requirements, data quality checks, and orchestration patterns

• The consensus: strong fundamentals matter more than tool knowledge (CI/CD pipelines, proper testing, schema management, idempotent workflows - the infrastructure that keeps data pipelines from becoming unmaintainable messes)

⚡ When SQL Isn’t Enough (And When It Absolutely Is)

A Reddit thread kicked off with a pointed question: if Snowflake’s SQL handles complex transformations (CTEs, window functions, UDFs), why do we need Spark, Airflow, and DBT? The answers cut through a common architectural confusion.

• Spark vs. SQL is a category error: Spark is a distributed compute engine that happens to support SQL (via Spark SQL), while Snowflake is a cloud data warehouse where SQL is the primary interface (you’re comparing an execution engine to a storage-compute platform)

• The cost-performance trade-off: Snowflake charges for compute time, so transforming 10TB of raw logs directly in Snowflake can be 5-10x more expensive than preprocessing with Spark on cheaper compute, then loading cleaned data

• Where Snowflake SQL breaks down: when you need to read from Kafka streams, hit external APIs mid-transformation, or apply complex ML models that aren’t SQL-expressible (Snowpark helps, but you’re still constrained by what the warehouse supports)

🤖 Flink Gets Agentic: Real-Time AI That Actually Responds to Events

Flink Agents is a new framework from the Apache Flink community that combines stateful stream processing with LLM-based agents. If you’ve been trying to build AI systems that react to real-time event streams (not batch data), this architecture is worth understanding.

• The technical setup: Flink handles event ingestion, stateful processing, and exactly-once semantics, while AI agents (LLM-powered) make decisions based on streaming context (think fraud detection that adapts its rules based on emerging patterns, not static thresholds)

• Why this matters: traditional AI data tools work in request-response mode with static data; Flink Agents can maintain conversation state across millions of concurrent event streams with sub-second latency (Flink’s distributed snapshotting gives you fault tolerance without losing agent state)

• Real use cases emerging: real-time content moderation for live streams that adapts to context, intelligent alerting systems that reduce false positives by understanding event sequences, automated trading systems that process market data and execute decisions in the same pipeline

🧠 Text-to-SQL Is Just the Appetizer: Building Production AI Data Analysts

Pedro Nascimento’s deep-dive on building Findly’s AI data analyst is packed with architectural lessons from taking text-to-SQL from demo to production. His core argument: the SQL generation is 20% of the problem; the other 80% is context, validation, and multi-step reasoning.

• Multi-agent architecture: separate agents for planning (decompose “analyze cohort retention” into specific steps), SQL generation (with compile-time validation), Python execution (for post-query transforms like statistical tests), and result synthesis (the system runs 5-7 LLM calls per complex query, not one)

• Semantic layer as context: they use Malloy to define metrics, joins, and business logic in code, then compile it to optimized SQL (this gives you type-checking and prevents the LLM from hallucinating table relationships - it’s working with a known schema graph)

• RAG as a recommendation pipeline: keyword search (BM25) for exact term matching → embedding search (dense retrieval) for semantic similarity → fine-tuned reranker (instruction-following model) to pick the top-k most relevant schema fragments (they found off-the-shelf rerankers underperform by 15-20% without domain fine-tuning)

• Latency architecture: fast models (GPT-4o mini) for planning and simple queries, reasoning models (Claude Sonnet, Gemini 2.5 Pro) for complex SQL generation, aggressive caching at every layer (they hit 200-300ms for cached queries, 3-5s for complex uncached analysis)

🚀 Flotilla: When Spark Can’t Handle Your Multimodal Data (18x Faster)

Daft’s new distributed execution engine Flotilla is purpose-built for workloads that mix structured data with images, videos, PDFs, and audio. If you’ve tried processing terabytes of images with Spark and watched it crawl, the architectural choices here are instructive.

• Performance claims: 18x faster than Spark and Ray Data on multimodal benchmarks (they tested on image embedding generation across 10TB+ datasets - the kind of preprocessing needed for training vision models or RAG over images)

• The technical difference: Spark’s task scheduler assumes uniform task duration and homogeneous data, which breaks down when some tasks process 100KB images and others handle 50MB videos (Flotilla does content-aware scheduling and dynamic resource allocation based on actual data characteristics)

• Where this matters: ML preprocessing pipelines that need to resize/normalize millions of images, extract frames from videos, run inference models (CLIP embeddings, OCR), and join results with structured metadata (doing this efficiently requires understanding data size distribution, not just partition count)

• No manual tuning: Spark typically needs careful partition sizing, memory configuration, and shuffle tuning for multimodal data; Flotilla profiles data characteristics and auto-tunes (which matters when your data engineer doesn’t want to become a Spark performance expert)

💡 e6data AI Analyst Early Access

We’re launching early access to e6data AI Analyst, and honestly, it’s about time someone built a data querying interface that actually understands how humans think about data. Ask questions exactly as they occur to you, get contextual multi-turn conversations, and for once, actually enjoy the follow-up process.

• 95%+ accuracy on enterprise workloads: Because “pretty good” genuinely isn’t good enough when you’re dealing with 1000+ tables and the kind of complex relationships that make SQL joins look like abstract art

• Multi-turn conversations that make sense: Your data can finally talk back in a way that doesn’t require translating human curiosity into rigid query syntax

• Zero migration headaches: Works with your existing data platform because we know exactly how much “fun” those migration projects actually are

→ Get early access here

🔧 German Strings: The 16-Byte Memory Trick That’s Everywhere Now

Remember when string processing was just something you accepted would be slow? The data engineering world has quietly adopted this elegant optimization called German Strings, and it’s delivering 3x faster string comparisons across analytics engines. The core insight is beautifully simple: pack everything into a fixed 16-byte struct with length, inline prefixes, and buffer references instead of pointer-heavy layouts.

• Cache locality wins: Fixed 16-byte structs mean your CPU can load multiple string headers per cache line (compared to scattered std::string objects at ~24 bytes each)

• Prefix shortcuts: Most string comparisons short-circuit on the 4-byte inline prefix before touching full payloads for about 95% of equality checks in practice

• Zero-copy operations: Substrings become offset adjustments, not memory copies (which is huge for window functions and text manipulation)

What strikes us is how this “small” change ripples through everything from Parquet ingestion to dictionary encoding.

🤖 AI Agents Are Breaking Our Data Systems (And That’s Actually Good)

LLM agents query data completely differently than humans. While we write careful, targeted SQL, agents perform what researchers call “agentic speculation”: high throughput exploratory querying where they might run dozens of variations to figure out what they actually need.

• Volume explosion: Traditional systems optimized for human query patterns (batch, predictable) suddenly face agent workloads that are continuous and exploratory

• New interface needs: Agents need different abstractions than SQL, think more like “give me data to help with X” rather than “SELECT specific_columns FROM known_table”

• Steerability requirements: Unlike humans who adapt their queries, agents need systems that can guide them toward efficient query patterns

This feels like one of those paradigm shifts where we’ll look back and say “of course data systems needed to be agent-first.” The paper hints at fascinating research directions around query interfaces designed for AI reasoning rather than human syntax.

🧪 Why Spark Unit Testing Feels Like Pulling Teeth (And How Pybujia Might Fix It)

That Reddit thread about Spark testing hit close to home, didn’t it? The brutal truth is that creating DataFrame fixtures is genuinely painful- you end up with more boilerplate than actual test logic, and debugging multi-table joins in tests becomes its own engineering project.

• Fixture fatigue: Setting up realistic DataFrames for testing often takes longer than writing the actual Spark job (which explains why so many teams skip it)

• Debug complexity: When a test fails on a complex transformation, good luck figuring out which of your 47 fixture setup lines caused the issue

• Markdown magic: The Pybujia approach of defining table fixtures in Markdown tables is surprisingly elegant- readable test data that doesn’t require DataFrame constructor gymnastics

What’s interesting is how this mirrors the broader testing philosophy debate: do we mock everything or test against realistic data? For Spark jobs that inherently deal with data shape and volume, the realistic fixture approach probably wins.

⚡ GPU Training Reality Check: H100 vs GB200 Performance vs Reliability

SemiAnalysis dropped some sobering hardware truths about the GB200 NVL72 versus H100 comparison. Yes, GB200 shows impressive performance-per-dollar on paper, but reliability issues in large-scale training runs are creating real operational headaches.

• Power efficiency gains: GB200 NVL72 delivers meaningful improvements in cost-per-token metrics, especially for sustained training workloads

• Reliability tax: The newer architecture faces stability challenges that can crater your training run after hours or days of progress (ouch)

• Ecosystem maturity: Software stack optimization for H100 is simply more mature. Sometimes the “boring” choice is the right infrastructure choice

This reinforces something we’ve seen repeatedly: breakthrough hardware performance often comes with operational complexity that isn’t obvious in benchmarks. For production ML training, reliability might matter more than peak performance when you’re thinking about wall-clock time to trained model.

💡 Real-World Performance Wins That Actually Moved the Needle

This Reddit discussion about performance improvements in practice revealed some gems. The most upvoted responses weren’t about fancy algorithms, they were about fundamentals like proper partitioning strategies, eliminating unnecessary shuffles, and (shocker) actually profiling before optimizing.

• Partitioning precision: Moving from default hash partitioning to deliberate partition strategies based on actual query patterns (especially for time-series data)

• Shuffle surgery: Identifying and eliminating unnecessary shuffles through better join ordering and data locality planning

• Memory reality checks: Right-sizing executor memory and actually monitoring GC pressure instead of guessing (which apparently many teams skip)

Multiple engineers mentioned that their biggest wins came from profiling tools revealing bottlenecks they hadn’t suspected- classic reminder that intuition about performance is often wrong.

💡 e6data AI Analyst Early Access

We’re launching early access to e6data AI Analyst, and honestly, it’s about time someone built a data querying interface that actually understands how humans think about data. Ask questions exactly as they occur to you, get contextual multi-turn conversations, and for once, actually enjoy the follow-up process.

• 95%+ accuracy on enterprise workloads: Because “pretty good” genuinely isn’t good enough when you’re dealing with 1000+ tables and the kind of complex relationships that make SQL joins look like abstract art

• Multi-turn conversations that make sense: Your data can finally talk back in a way that doesn’t require translating human curiosity into rigid query syntax

• Zero migration headaches: Works with your existing data platform because we know exactly how much “fun” those migration projects actually are

→ Get early access here

Community & Events:

We are hosting the next Lakehouse Days with Bengaluru Streams on data streaming, lakehouse architecture, and the future of real-time analytics this 27th September.

AND we’re hunting for data engineers who get excited about AI and aren’t afraid to build the future.

🏗️ XTX Markets Just Open-Sourced Their Exabyte-Scale Filesystem (And It's Actually Interesting)

When an algorithmic trading firm tells you they've been running 500PB across 40,000 drives "without losing a single byte" - well, that gets your attention. XTX Markets just open-sourced TernFS, their distributed filesystem, and reading through their technical deep-dive is like getting a masterclass in what large-scale storage actually looks like in practice.

• The metadata sharding approach is clever: 256 logical shards from day one means no rebalancing nightmares when you scale (which is exactly the kind of forward-thinking that separates real engineering from "we'll figure it out later")

• Immutable files with snapshot protection: Once written, files can't be changed, but you get automatic protection against the dreaded rm -rf moments (truth!)

• Purpose-built for substantial files: Median size of 2MB means this isn't trying to be everything to everyone - it's optimized for the data engineering reality where you're actually processing meaningful datasets

A good way to think about this is that TernFS acknowledges the brutal economics of large-scale data storage and builds solutions around those constraints rather than pretending they don't exist. The firm started with "a couple of desktops and an NFS server" and ended up needing to store hundreds of petabytes - which is probably a more relatable scaling journey than most of us would like to admit.

📊 Databricks Shares Their Database Reliability Playbook (The Honest Version)

Database reliability conversations usually involve a lot of uptime percentages and theoretical guarantees, but this Databricks piece cuts through that to talk about what actually works when your database is the nervous system of someone's business operations (which is terrifying when you think about it).

• Monitoring that actually helps: They focus on capturing the full context of failures, not just error counts - because knowing something broke is utterly useless without understanding why it broke

• Automated recovery with human judgment: Smart enough to handle the obvious cases automatically, but wise enough to escalate the weird edge cases to humans who can actually think through novel problems

• Making reliability a cultural priority: Treating reliability as a first-class engineering concern rather than something you retrofit later (which should be obvious but apparently isn't)

What's interesting here is the implicit acknowledgment that reliability engineering isn't fundamentally a technical problem - it's an organizational one where technology is just the implementation mechanism. You can have perfect monitoring and automated failover, but if your team culture doesn't prioritize reliability, you'll still find creative ways to break things.

⚡ Cachey: A Read-Through Cache That Actually Understands Object Storage

Object storage is fantastic until you need to read the same blob repeatedly, at which point you remember why caching was invented in the first place. Cachey is a new open-source read-through cache designed specifically for S3-compatible storage, and it has that refreshing quality of solving exactly one problem really well.

• Hybrid memory-disk caching using the Foyer library: Intelligent about what goes where based on access patterns, so your frequently accessed data stays fast without your cache becoming a memory hog

• S3-compatible everything: Works with any S3-like storage and includes a /fetch API for pre-signed URLs (which is exactly how you'd want to integrate this into existing workflows)

• Built for immutable blobs: Acknowledges that most object storage use cases involve data that doesn't change, so the cache can be much more aggressive about retention

This falls into that category of tools that make you think "why didn't this exist already?" The answer, of course, is that building good caching is harder than it looks, but when someone gets it right, it feels obvious in retrospect.

🤖 Building RAG Systems at Enterprise Scale: The Brutal Realities Nobody Discusses

This Reddit thread is one of those refreshingly honest discussions that cuts through the RAG evangelism to talk about what actually happens when you try to implement RAG in banking, pharma, and legal environments. Spoiler: it's considerably messier than the conference talks suggest.

• OCR noise is the silent productivity killer: Real enterprise documents aren't clean markdown files - they're scanned PDFs with inconsistent formatting that systematically destroys your chunking strategies

• Metadata becomes mission-critical: Enterprise documents have complex relationships and hierarchical structures that simple vector similarity completely misses (which explains why your retrieval quality is mysteriously terrible)

• Domain-specific chunking strategies are non-negotiable: Generic text splitting fails spectacularly when dealing with legal contracts or financial reports that have meaningful structural boundaries

The honest truth about RAG in production environments is that roughly 80% of your engineering effort goes into data quality and preprocessing work, not the sophisticated ML components that get all the conference attention. It's unglamorous work, but it's the difference between a demo and a system that actually works.

🔧 Open-Source Ingestion Tools in 2025: What's Actually Working According to Practitioners

This community discussion reveals fascinating patterns about which open-source ingestion tools are winning in practice. The responses tell a story about how the data tooling landscape has matured (and where it definitely hasn't).

• dlthub and DLT are gaining serious momentum: Simple, well-documented tools are consistently beating feature-heavy platforms because most teams just want reliable data movement without vendor complications

• DuckDB keeps appearing in unexpected contexts: Its combination of performance and operational simplicity makes it a compelling choice for transform-heavy ingestion workflows

• Direct database integration remains surprisingly popular: Many teams are bypassing specialized ingestion tools entirely and going straight to PostgreSQL (which says something interesting about complexity creep in data tooling)

The pattern emerging here is that teams are gravitating toward boring, reliable solutions over exciting, complex ones - which is probably a healthy sign that the industry is maturing beyond the "let's rebuild everything" phase.

💡 e6data AI Analyst Early Access

We're launching early access to e6data AI Analyst, and honestly, it's about time someone built a data querying interface that actually understands how humans think about data. Ask questions exactly as they occur to you, get contextual multi-turn conversations, and for once, actually enjoy the follow-up process.

• 95%+ accuracy on enterprise workloads: Because "pretty good" genuinely isn't good enough when you're dealing with 1000+ tables and the kind of complex relationships that make SQL joins look like abstract art

• Multi-turn conversations that make sense: Your data can finally talk back in a way that doesn't require translating human curiosity into rigid query syntax

• Zero migration headaches: Works with your existing data platform because we know exactly how much "fun" those migration projects actually are

→ Get early access here

Community & Events:

1. The team published a new technical read on German Strings: The 16-Byte Secret to Faster Analytics!

2. We are hosting the next Lakehouse Days with Bengaluru Streams on data streaming, lakehouse architecture, and the future of real-time analytics this 27th September.

3. You'll also find us at: Big Data London (24-25 Sep, 2025)

AND we're hunting for data engineers who get excited about AI and aren't afraid to build the future.

🚀 Rust's Build Performance Reality Check: 3,700 Developers Weigh In

The Rust community just dropped some serious insights about compile times, and spoiler alert: it's complicated. The 2025 Compiler Performance Survey pulled in over 3,700 responses with an average satisfaction rating of 6/10.

• Incremental rebuilds are the biggest pain point - 55% of developers wait more than 10 seconds for rebuilds (ouch!)

• Workspace dependencies create unnecessary cascade rebuilds - change one crate, recompile everything downstream

• Linking is always from scratch - no incremental magic here, but LLD linker adoption is coming to help

The fascinating part? Build experience varies wildly across workflows. Some developers love Rust's performance compared to C++, while others point enviously at Go's speed. What's clear is that optimizing for different workflows requires completely different solutions (which explains why this is so hard to solve universally).

⚡ From 8 Days to 90 Minutes: The 75GB CSV Streaming Success Story

A data engineer just shared their journey of ingesting a massive 75GB CSV into SQL Server without melting their laptop.

• Java's InputStream + BufferedReader combo proved to be the hero - streaming without loading everything into memory

• Batching and parallel ingestion techniques cut processing time from 8 days to 90 minutes per file

• Tool selection matters more than you think - sometimes the "data science" tools aren't the right hammer for every nail

This reinforces something we see repeatedly: when dealing with truly large datasets, stepping outside your comfort zone language-wise can yield dramatic performance improvements (and will be fun).

🤖 "70% of My Workload is All Used by AI" - The New Data Engineering Reality

The r/dataengineering community is buzzing with a confession that hits close to home: most data engineering work is now AI-driven. This isn't just about building ML pipelines anymore.

• AI systems are becoming the primary consumers of data infrastructure - not just dashboards and reports

• Data engineers are evolving into AI enablement specialists whether we planned for it or not

• The skills gap is real - traditional ETL knowledge needs to expand into ML ops, vector databases, and real-time inference pipelines

A good way to think about this is: if your data architecture wasn't designed with AI workloads in mind, you're probably going to need some retrofitting soon (if you haven't already started).

📈 Oracle Hits Record Highs as Wall Street Discovers Data Engineering

Oracle's stock is shattering records, and the Street is attributing it directly to the AI and data engineering boom across enterprises. This isn't just another tech stock story.

• Enterprise data modernization is driving serious investment - companies are finally putting real money behind their data strategies

• Data engineering roles are becoming strategic, not just operational - we're moving from "keep the lights on" to "enable the business transformation"

• The talent premium is real - when your stock price depends on data capabilities, you pay up for the people who can deliver them

What's interesting here is how quickly the narrative has shifted from "data engineering is a cost center" to "data engineering is a competitive advantage" (and Wall Street has noticed).

🔄 Apache Wayang: The Federation Play for Multi-Engine Processing

Apache Wayang (still incubating) is making some bold claims about federated data processing - up to 150x faster than centralized platforms by keeping data in place.

• Application independence across processing engines - write once, run on Spark, Flink, PostgreSQL, Java Streams, or whatever

• In-situ processing philosophy - avoid the expensive data movement that kills performance in traditional architectures

• Minimal code changes for engine switching - the holy grail of not being locked into specific processing frameworks

This feels like a response to the very real pain of vendor lock-in and the complexity of managing multiple processing engines in modern data stacks. Time will tell if the federation approach delivers on its promises, but the problem it's solving is definitely real.

💡 Meet e6data AI Analyst (Early Access)

Ask questions exactly as you think them, get contextual multi-turn conversations, and actually enjoy the follow-up process for once. Early access here.

• 95%+ accuracy on enterprise workloads - because "pretty good" isn't good enough when you're dealing with 1000+ tables and complex relationships

• Multi-turn conversations- make your data talk back

• Zero migration required - works with your existing data platform (we know how much fun those migration projects are)

Community & Events:

1. The team published a new technical deep dive on Partition Projection in Data Lakehouses this week!

2. Also, we are hosting the next Lakehouse Days with Bengaluru Streams on data streaming, lakehouse architecture, and the future of real-time analytics this 27th September.

3. You'll also find us at:

• Big Data London (24-25 Sep, 2025)

• Databricks World Tour Mumbai (19 Sep, 2025)
  

AND we're hunting for data engineers who get excited about AI and aren't afraid to build the future. No corporate speak here - just cool problems and smart people to solve them with.

🔥 Apache Fluss: Flink's Fast New Table Storage Engine That Actually Gets Changelog Right

Alibaba's newest contribution to the Apache ecosystem tackles the painful reality that even the best lakehouse formats like Paimon just aren't fast enough for real-time data engineering (truth!).

• Fluss provides dual-tier architecture with RocksDB-backed tablet servers for hot data and tiering to Paimon for historical storage (finally, someone who understands that object storage alone isn't enough for microsecond-latency use cases)

• Primary key tables now generate efficient changelogs without the lookup hell that makes Paimon painful for high-throughput streaming

• Client-side stitching intelligently merges real-time and historical data, giving Flink jobs a unified view without the complexity of managing multiple storage layers

What's fascinating is how Fluss essentially admits that the "one table format to rule them all" dream is dead - sometimes you need speed, sometimes you need scale, and the magic happens in making them work together seamlessly.

🦀 The Rust Productivity Paradox: Why Fighting the Compiler Actually Makes You Faster

That brutal initial learning curve with Rust's ownership model isn't a bug - it's the feature that eventually makes you surprisingly productive

• Rust forces you to think about memory safety and concurrency upfront, eliminating entire classes of runtime bugs that usually haunt production systems

• The ownership model creates surprisingly maintainable codebases where refactoring doesn't feel like defusing a bomb

• Strong typing and pattern matching reduce cognitive load once you internalize the patterns, making complex data transformations more readable than traditional imperative code

For data engineers tired of mysterious Spark crashes and memory leaks in long-running pipelines, Rust might be worth the investment - or so does our e6data team champion now.

⚡ Kafka-to-Iceberg Integration: Three Paths, Each With Its Own Gotchas

Robin Moffatt breaks down the eternal data engineering question: how do you get streaming data from Kafka into your lakehouse without losing your sanity (or your data quality)?

• Flink SQL provides the most control but requires managing yet another streaming runtime and understanding Flink's occasionally mystical state management

• Kafka Connect offers operational simplicity but can struggle with complex transformations and schema evolution (great until you need to do anything beyond basic ETL)

• Confluent's Tableflow promises managed convenience but locks you into their ecosystem and pricing model (the classic build-vs-buy decision with modern cloud economics)

Seems like here there's no silver bullet - your choice depends entirely on whether you value operational simplicity, transformation flexibility, or vendor independence most.

📊 The Two Parquets: Why Your Files Might Not Play Nice Together

Jerónimo López exposes an uncomfortable truth about the Parquet ecosystem - we're living in a world of format fragmentation that most data engineers don't even realize exists.

• Parquet v1 and v2 aren't just version numbers - they represent fundamentally different encoding schemes that affect both performance and compatibility across tools

• Many popular engines still default to v1 for compatibility, leaving performance gains on the table (looking at you, Spark with your conservative defaults)

• The ecosystem's partial adoption creates invisible data pipeline bottlenecks where different tools read the same files with wildly different performance characteristics

This is a perfect example of why understanding your file formats matters - that "simple" Parquet file might be the reason your queries are mysteriously slow, and switching versions could be a free 2x performance improvement.

💸 Snowflake: When Your Data Warehouse Bill Exceeds Your AWS Compute

A candid Reddit discussion about logistics margins being devoured by Snowflake costs reveals the hidden complexity of cloud data warehouse economics

• Real-world usage patterns often trigger expensive auto-scaling and clustering that can turn a reasonable monthly bill into budget-busting surprises

• The community suggests aggressive query optimization, warehouse right-sizing, and even considering hybrid architectures to regain cost control (We write about it here)

• Many organizations discover too late that Snowflake's consumption model works great for predictable workloads but can be financially catastrophic for spiky, exploratory analytics

This thread is a sobering reminder that in the cloud era, architectural decisions have direct P&L impact - and that understanding your cost model is as important as understanding your query performance.

💡 We Built the Cost Optimization Hub Your Teams Been Demanding

Our Query and Cost Optimization Hub - a comprehensive guide that actually tells you how to optimize your compute engine(s) for max output.

• Multi-engine coverage across Snowflake, Databricks, BigQuery, Redshift, and more - because your stack is probably more fragmented than you'd like to admit, and optimization is necessary

• Beginner to advanced techniques with actual code examples, not just vague suggestions about "right-sizing your clusters" (looking at you, vendor documentation)

Community & Events:

While we don't have specific CFPs to announce this time, keep an eye on the usual suspects - DataCouncil, and the regional meetups where the best conversations happen in the hallway track anyway.

You'll also find us at:

• Big Data London (24-25 Sep, 2025)

• Databricks World Tour Mumbai (19 Sep, 2025)

Don't miss Bengaluru Streams x Lakehouse Days on 27 Sep - subscribe to our calendar for registration.

📊 Wide Tables: The Performance Paradox in Modern Data Warehouses

Turns out even cloud-native warehouses have their limits when it comes to handling tables with hundreds of columns - who would have thought more isn't always better? Let’s say what this Reddit discussion says:

• Column count matters more than row count for query performance, especially when your SELECT * queries start timing out (looking at you, analysts who never learned to specify columns)

• Indexing and partitioning strategies can save wide tables from themselves, but proper schema design beats optimization band-aids every time

• The normalization vs denormalization debate continues - wide tables offer query simplicity but at the cost of maintenance complexity and storage efficiency

Modern doesn't mean magical - even Snowflake and BigQuery have physics to contend with, so design your schemas thoughtfully rather than throwing everything into one massive table.

🔄 SQL Server Mirroring: A Journey Through On-Prem to Cloud Reality

One brave soul documented their Microsoft Fabric mirroring adventure, complete with the inevitable "it's more complicated than the documentation suggests" moments we all know and love.

• Thorough testing isn't optional - what works in dev might fail spectacularly in production with real data volumes and network constraints (Murphy's Law applies double to data replication)

• Unexpected downtime happens even with the best planning, so having rollback strategies and communication plans ready isn't paranoia, it's professionalism

• Real-world accounts beat vendor demos every time - this kind of honest experience sharing helps the community avoid the same pitfalls (we need more of this transparency)

Migration stories like this are worth their weight in gold because they show the messy reality behind glossy case studies (plus they remind us we're not alone in fighting these battles).

🤖 ToolFront: Open Source Text-to-SQL That Actually Works

Finally, someone built a text-to-SQL tool that doesn't hallucinate database schemas - this open-source Python library provides AI agents with safe, read-only access to understand and query your actual database structure.

• Schema hallucination is a real problem when AI models invent table names and columns that don't exist (we've all seen those confidently wrong SQL queries)

• Read-only by design means you can let AI explore without worrying about accidental data modification or deletion (security through architecture, not just permissions)

• Bridging natural language and SQL effectively could democratize data access for non-technical users while keeping data teams in the loop

This feels like a genuine step toward making AI useful for database interactions rather than just impressive demos.

🎯 When Your BI Users Go Rogue (And How to Manage the Chaos)

Ever wonder what happens when business users start creating their own reports without guardrails? This Reddit thread dives into the eternal struggle between user autonomy and data governance (spoiler: it's messier than your staging tables).

• Self-service can backfire spectacularly when users don't understand the underlying data model (leading to those "why don't our numbers match?" conversations)

• The consensus points toward controlled flexibility - give users sandbox environments and clear training rather than locking everything down (nobody wants to be the data team that says "no" to everything)

• Dynamic datasets beat one-off reports - building reusable, parameterized datasets serves multiple users better than custom builds for every stakeholder request

The sweet spot isn't choosing between control and chaos, but designing systems that let users explore safely while keeping your sanity intact (trust me, future you will thank present you).

🏗️ The Medallion Architecture Farce: When Marketing Meets Data Modeling

This brutal takedown from Confessions of a Data Guy pulls no punches about Databricks' "medallion architecture" - calling it: rebranded data warehousing concepts with shinier marketing.

• Bronze/Silver/Gold is just RAW -> FACT/DIM with extra steps (and extra storage costs that conveniently benefit your cloud provider)

• The confusion is real - Reddit threads still ask "what's the difference between Silver and Gold?" because the distinctions are genuinely unclear (when seasoned engineers can't explain it simply, something's wrong)

• Three decades of proven data warehousing patterns work just fine in your lakehouse - no need to invent new terminology for staging, transformation, and marts

Sometimes the emperor really has no clothes, and this piece reminds us to steel-man vendor propositions before adopting their latest architectural innovations.

Community & Events:

We recently hosted an SRE meetup right here at the e6data office where Pranav from our team spoke about "Battle-Tested GitOps with ArgoCD" to a packed house.

Also, We're hitting the road (and the skies) this season! You'll find us at

• Big Data London (24-25 Sep, 2025)

• Databricks World Tour Mumbai (19 Sep, 2025)

Two events where the data engineering community gathers to share war stories and debate the merits of yet another lakehouse architecture.

We’re growing! Explore open engineering roles → here

⚙️ When PostgreSQL's "Perfect" Query Plans Aren't

Tomas Vondra shared a fascinating case study this week that every data engineer should bookmark. A PostgreSQL query planner was confidently choosing a 5-second index scan when a 2-second sequential scan would have won.

• The details reveal why database optimization remains more art than science: Bitmap scans often outperform index scans by 10x for 1-5% selectivity ranges.

• Cost estimates built on coarse statistics make "perfect" planning impossible. Hardware quirks like prefetching and cache warmth can shift the winner mid-execution. "The optimizer's best guess may only be 'good enough,'" as Vondra puts it, and this resonates with anyone who's spent time performance-tuning production queries.

• What this means for data engineers is clear: even with decades of development, database optimizers operate with incomplete information by design. They're fast because they use simplified stats, but that simplification comes with blind spots.

The implication is that we should benchmark our critical queries ourselves. Don't assume the planner knows best, especially in cloud environments where the hardware abstraction adds even more complexity to the cost model.

🎓 How to Ace the Databricks Pro Exam?

We came across a cool Reddit thread this week from someone who absolutely crushed the Databricks Data Engineer Professional exam with a 95% score. What struck us wasn't just the impressive result, but their honest breakdown of what actually moved the needle versus what felt like busy work.

• The conventional wisdom says to rely heavily on courses like Derar Alhussein's Udemy offering. This test-taker had a different take: skim it for breadth, but don't expect it to carry you across the finish line.

• The real meat was in drilling down on Delta Lake, Spark Structured Streaming, and security concepts. These aren't just exam topics - they're the daily reality of most data engineering teams we encounter.

• What we found most valuable was their emphasis on rotating through multiple practice exams. Not because repetition breeds success, but because it exposes the outdated "gotcha" syntax questions that still lurk in these certifications.

• The data ecosystem moves fast, but exam content often lags behind.

🎧 Inside Spotify's 1.4 Trillion-Event Data Engine

Sometimes you encounter a number that makes you pause and reconsider your assumptions about scale. For us this week, it was Spotify's 1.4 trillion events per day. Not per month. Per day. What fascinated us wasn't just the raw volume, but how they've architected around it.

• The stack reads like a greatest hits of modern data infrastructure: Pub/Sub feeding into Beam and Flink pipelines, orchestrated through 38,000 Flyte workflows, with data landing in BigQuery and GCS/HDFS to serve roughly 5,000 Looker and Tableau dashboards for 6,000 users.

• The migration story from Luigi/Flo to Flyte particularly caught our attention. Fragmented orchestration and visibility gaps - these are the unglamorous problems that don't make conference talks but absolutely cripple teams at scale.

• Spotify's solution was refreshingly standard: battle-tested GCP primitives plus a scheduler that actually works. When you're processing 1.4 trillion events daily, the temptation to over-engineer is immense. Instead, they've doubled down on observability and reliability over cleverness.

🕷️ Web Scraping at Scale: The Uncomfortable Truths

A community discussion this week perfectly captured why large-scale web scraping feels like an endless game.

• The usual suspects were all there: rotating proxies, CAPTCHA solvers, the perpetual arms race between scrapers and anti-bot measures.

• Rotating proxies and CAPTCHA solvers buy you time, but they still break. Alerting becomes mandatory because failure is inevitable, not just possible.

• The wisdom in the thread gravitated toward two key insights: Hidden or internal APIs are absolute gold - always exhaust these options before building brittle DOM parsers.

• And past a certain pain threshold, most teams simply buy the data instead of maintaining Selenium farms. This maps to a broader pattern we see in data engineering. We often treat scraping as a permanent ETL source when it should be viewed as a stopgap. The smart teams we know invest in robust monitoring and plan their exit strategy (paid APIs, direct partnerships) from day one.

🤖 Our Next Lakehouse Days: AI Agents for Enterprise Data

We're hosting another hands-on meetup in Bengaluru, and this time we're diving deep into something every data team is grappling with: AI agents that actually work at enterprise scale.

The agenda goes beyond the usual "ChatGPT + your database" demos. We're tackling the real problems: how do you handle messy schemas without requiring perfect catalogs? How do you avoid the join errors that plague most text-to-SQL attempts? And critically, how do you move from basic query-and-result to genuinely conversational, context-aware workflows?

Talks we're hosting:

• Bharath Harish (our Head of Product) will break down how we built Text-to-SQL agents with 95%+ accuracy for enterprise scale. He'll cover knowledge graph-driven relationship discovery, SQL engine-like planning to reduce hallucinations, and performance techniques like partition key usage and optimized CTEs.

• Harsh Sharma from Flipkart will share why his team chose Milvus (open-source vector database) over traditional approaches for recommendation systems, plus the indexing and retrieval strategies that make it work at e-commerce scale.

Register here: https://lu.ma/8ufzg6gi

Community & Events

• AWS Storage Day: Rajath Gowda, our founding data engineer spoke about S3 tables and how e6data powers them, at the AWS event!

• Blog Series: Building a Modern Data Pipeline in Snowflake- deep dive from our engineering team

• Hiring: We’re growing! Explore open engineering roles → here

So we're experimenting with GPT-5 for this week's newsletter. Jury's still out, but here's what it cooked up for you...

The problem that's probably burning your budget right now

You know that feeling when one Spark job takes 3x longer than it should? Nine times out of ten, it's skewed joins. Some hot key (anonymous users, null tenant IDs, that one power user with 10M events) creates a few massive shuffle partitions while the rest of your cluster sits there twiddling its thumbs.

I see this constantly in r/dataengineering threads. "Why is my job so slow?" Usually it's join skew torching their AWS bill.

Here's the copy-paste fix

The setup that's probably familiar:

from pyspark.sql import functions as F

# This looks innocent enough...

events = spark.table("fact_events") # 100M rows
users = spark.table("dim_users") # 1M rows

# But user_id has that one enterprise customer with 20% of all events

joined = events.join(users, "user_id", "left")

Step 1: Turn on the good stuff (AQE)

# These should honestly be defaults by now

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionRowCountThreshold", 250000)
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", 268435456) # 256MB

# Now run the same join - AQE will split those massive partitions

joined = events.join(users, "user_id", "left")

Step 2: Broadcast the small side (if it actually IS small)

# Rule of thumb: if your dimension is < 500MB after filters, broadcast it

joined = events.join(users.hint("BROADCAST"), "user_id", "left")

Step 3: For the really stubborn cases - salting

# When you have that one customer that's just... massive
HOT_CUSTOMER_ID = 42  # Replace with your actual hot key
SALT_BUCKETS = 16

events_salted = (events
    .withColumn("salt", 
        F.when(F.col("user_id") == HOT_CUSTOMER_ID, 
               (F.rand() * SALT_BUCKETS).cast("int"))
         .otherwise(F.lit(0))))

users_salted = (users
    .withColumn("salt",
        F.when(F.col("user_id") == HOT_CUSTOMER_ID,
               F.sequence(F.lit(0), F.lit(SALT_BUCKETS-1)))
         .otherwise(F.array(F.lit(0))))
    .withColumn("salt", F.explode("salt")))

# Join on both user_id AND salt
joined = events_salted.join(users_salted, ["user_id", "salt"], "left")
# Drop the salt column downstream - it's just a technical detail

How to spot this in the wild

Spark UI red flags:

• Stage timeline looks like a hockey stick (most tasks finish fast, a few stragglers kill you)

• Shuffle read size distribution is all over the place

• One executor is pegged while others are idle

Quick diagnostic:

# Always run this before big joins

events.groupBy("user_id").count().orderBy(F.desc("count")).show(20, False)

If the top row has 10x more records than the second row, you've got skew.

If you're using Snowflake/BigQuery instead

Snowflake:

-- Force broadcast with hints
SELECT /*+ BROADCAST(users) */
    events.user_id, 
    SUM(events.amount)
FROM events 
JOIN users ON events.user_id = users.user_id
WHERE events.event_date >= CURRENT_DATE - 7
GROUP BY 1;

BigQuery:

• Cluster your large tables by join keys

• Use partitioning aggressively to limit scanned bytes

• Pre-aggregate when possible

The real talk

Look, skewed joins are everywhere. That anonymous user ID, deleted records with NULL foreign keys, your biggest enterprise customer - they all create hot partitions. AQE helps, but sometimes you need to get your hands dirty with salting.

The good news? Once you fix it, it stays fixed.

Human here, let us know how much our experiment has succeeded in comments. We’ll love to test more!

Now, onto the other things:

Launching e6data’s Hybrid Data Lakehouse: 10x Faster Queries, Near-Zero Egress, Sub-Second Latency

Most enterprises today deal with high egress costs, governance issues, migration, and latency issues in hybrid lakehouses due to their fundamental architectural limitations.

We are solving this with our federated SQL engine + hybrid cluster architecture. The architecture is designed such that the hybrid cluster is abstracted out from the end user's querying experience, and they get to write queries as though there were a single cluster talking to all these data sources.

Used in production by customers, the latest benchmarks on e6data’s hybrid data lakehouse deployment are as follows:

• 10x speed by keeping compute local.

• ~0% egress fees

• Adding cache reduces another ~40% off latency with no extra data movement.

For more details, benchmark setup, case study, product feature and docs, refer to this page!

Community & Events

• Lakehouse Days– We are hosting a new one this month on Agentic AI. Sign up to our calendar for first updates!

• Blog Series – Embedding Essentials: Cosine Similarity in SQL — deep dive from our engineering team

• Hiring – We’re growing! Explore open engineering roles → here

🔬 How to make Vector Embeddings?

An interactive Hugging Face Space peels back the 1,536-D token matrix of DeepSeek-R1-Distill-Qwen-1.5B, letting you poke real vectors with cosine-similarity tools.

• Dissects static token vectors vs contextual hidden-state embeddings, clarifying why only the first layer stays fixed while deeper layers mutate with self-attention.

• Code shows how torch.nn.Embedding is lifted, cached, and queried in milliseconds—then uses torch.topk on cosine scores to map local neighbourhoods.

• Notebook prints raw rows: e.g. the token "HTML" expands to a 1,536-float vector, proving verbatim that the model’s dictionary sits in RAM not VRAM.

🛠️ Kafka Core in ~300 LOC of Python

A Redditor rewrites Kafka’s 2011 design in pure Python: single-threaded broker, producers, consumers, offset tracking, no ZooKeeper, no partitions.

• Append-only log + pull-based consumer loop illustrate back-pressure mechanics.

• Community calls out the missing bits: partitioning, segment compaction, persistence, concurrency-safety.

• Author plans to add partitions next; goal is architectural clarity, not prod-grade throughput.

Our Takeaway: Re-implementing the log-structured core is the fastest way to grok why sequential disk I/O + monotonic offsets scale.

🔑 Primary Keys vs Petabytes: When Constraints Bite Back

Another Reddit thread debates surrogate hashes & GUIDs versus brute-force uniqueness checks.

• Surrogate keys (MD5/UUID) enable SCD2 change tracking while decoupling from source schemas.

• One team’s 5.7 trillion-row sensor fact grew its 8-byte PK index to 45 TB uncompressed- sometimes the index is the data.

• Others drop keys on ultra-wide fact tables, trading insert speed for read-time de-duplication.

Our Takeaway: Keys are good insurance until the index dwarfs payload, then explore late-binding dedupe or zone maps.

🗑️ SQL Spec Gripes: MERGE, RIGHT JOIN & Recursive Rumbles

A Reddit rant blacklists MERGE, recursive CTEs, and RIGHT JOIN, sparking a highly heated debate.

• OP slams non-idempotent MERGE, “clever” recursive CTEs, and unreadable RIGHT JOINs

• Veterans defend recursive CTEs for graph traversals; say RIGHT JOIN is merely rare, not wrong

• Tangent: BigQuery’s new pipe operator (|>) pitched as a saner logical-order alternative

Our Takeaway: A disciplined, cross-engine SQL subset—plus pipeline syntax—beats holy wars over edge-case clauses.

💡 Cost Calculator: Instantly Price Your Lakehouse

We’re launching an interactive Cost Calculator that shows what you’ll spend on e6data versus legacy engines, before you deploy.

• Estimates query costs on your lakehouse platform in < 1 min.

• Pick cloud (AWS, Azure, GCP), region, cluster size, hours/day, and see side-by-side monthly totals in seconds.

Community & Events

• Lakehouse Days Replay – Missed our “From Stream to Lakehouse” session? Catch the full recording on e6data’s YouTube.

• Blog Series – Building a Modern Data Pipeline in Snowflake: From Snowpipe to Managed Iceberg Tables with Sync Checks — deep dive from our engineering team (Part 1 out now).

• Hiring – We’re growing! Explore open engineering roles → here

🛠️ OpenRewrite Beats Gemini AI at JUnit Migration
This week, our senior engineer pitted Google’s Gemini against OpenRewrite when moving tests from JUnit 4 → 5—Gemini timed-out, while OpenRewrite finished in minutes.
• Gemini burned API quota and stalled on “complex” tests
• OpenRewrite migrated nearly all tests in ~10 min; only edge cases needed tweaks
• Moral: pick the right-sized tool. LLMs aren’t a silver bullet for structured code refactors
Our Takeaway: Rule-based refactoring still trumps LLM magic for deterministic migrations. The big a-ha moment for him was, it passed all tests except tpox (known) and one geospatial test (which is because it uses assertThrows, and the expected error message is coming different)

💾 Databases Chase Unified Memory Limits
An in-depth post argues modern query engines must police themselves with strict memory caps and spill-to-disk safety nets—language choice matters.
• Two must-haves: kill a query before it OOMs, and spill gracefully when it would
• Go’s silent allocations make tracking painful; Rust’s lack of fallible allocators is “annoying” but fixable
• Teams layer custom allocators/monitors to meter both memory and CPU for every pipeline operator
Our Takeaway: Expect next-gen engines to ship with first-class resource governors, not bolt-ons.

🚦 Stopping the Bronze Data Deluge
A Reddit thread tackles medallion-layer mayhem when incremental feeds piggy-back on full loads and accidentally delete 95 % of data.
• Hash-based “delete-on-absence” logic breaks with sparse incrementals
• Community suggests adding load_type, is_current, and soft-delete flags instead of hard deletes
• Keep only 30–90 days in Bronze; archive deep history cold and manage SCD 2 in Silver
Our Takeaway: Treat Bronze as a transient landing zone and preserve history higher up and let incrementals append, not annihilate.

🧊 Delta, Iceberg & Hudi: Same Parquet, Different Vibes
Another lively debate says the three lakehouse table formats feel like Oracle, Postgres and MySQL—different logos, same spoon.
• Hudi wins for low-latency streaming upserts; Delta shines in Databricks’ managed world; Iceberg rules with vendor-neutral engine support
• All wrap Parquet files with metadata layers; feature gaps are narrowing fast
• Consensus: by 2025, Iceberg is the safest long-term bet for neutrality across Snowflake, Flink, Trino & friends
Our Takeaway: Master the concepts of partitioning, snapshots, manifest files and you can hop between formats as ecosystems blur. Btw, we launched an improved support for Delta, Iceberg, Hudi, and Delta this week.

💡 Product Launch

Product Update: Improved support for Iceberg, Polaris, Hudi & Delta Lake

Our improved support lets you natively read and browse all four open-table formats (Iceberg, Polaris, Hudi, Delta Lake) inside one e6data workspace with zero-copy ingestion, cross-catalog joins, Polaris RBAC/Unity Catalog enforcement, and a unified, metadata-aware query path.

📚 Blog Release

Snowflake Snowpipe → Managed Iceberg Tables (with sync checks)

Our lead engineer wrote a complete blueprint for building a modern Snowflake pipeline that ingests with Snowpipe, transforms via SQL/stored procedures, writes to Snowflake‑managed Iceberg tables, and adds row‑level sync checks—all while keeping the data open and queryable by any Iceberg-compatible engine.

Key highlights:

• End‑to‑end SQL: landing/core tables, CREATE ICEBERG TABLE, and a transform_orders() stored procedure.

• Snowpipe AUTO_INGEST from S3 to landing tables; continuous or scheduled loads.

• Data quality & sync checks (row counts, diffs, optional checksums) between curated and Iceberg tables.

• Snowflake Tasks to automate the daily run (CRON 0 2 * * * UTC).

• Open catalog, multi‑engine interoperability: query the same Iceberg data from Snowflake, Spark, Trino, Presto, e6data, etc.

• Benefits called out: automated ingestion, curated & governed analytics layer, and zero lock‑in via Iceberg’s open metadata

🤝 Community & Events

• Lakehouse Days Replay – ICYMI, here is the full recording of our “​From Stream to Lakehouse” is on e6data’s YouTube.

• Building a Modern Data Pipeline in Snowflake: From Snowpipe to Managed Iceberg Tables with Sync Checks: an in-depth blog series by our engineering team

• Meet Us Today at MachineCon USA – We’ll be on-site; come say hi!

• Hiring: We’re growing! Check out open engineering roles [here]

🤖 Rust Context-Generic Programming (CGP) in a Nutshell

A very insightful take on Rust CGP took over our engineering team this week. Here are a few highlights:

• CGP swaps “objects” for contexts: functions declare the few capabilities they need via consumer traits, and any struct that offers those capabilities can satisfy them. The compiler wires everything together, so there’s no runtime cost.

• It tackles Rust’s big headaches—bloated trait APIs, orphan-rule conflicts, forced public types, copy-pasted code—by cleanly splitting provider-and-consumer roles and letting overlapping implementations live side-by-side.

• Power tools include HasField and cgp_auto_getter macros, extensible records/builders for piecemeal struct construction, and extensible variants/visitors for enums that grow without breaking old code.

Our Takeaway: Use CGP when systems have lots of inter-dependent services or tight performance demands; stick with plain trait-object DI for small, stable parts. CGP keeps tests easy and mixes real or mock back-ends just by swapping contexts.

🧊 Iceberg, The Right Idea – The Wrong Spec

A very contrarian blog by the Database Doctor called “Iceberg, The Right Idea – The Wrong Spec” is claiming history on why object-store tables risk repeating 1990s storage issues:

• The author shows how old-school filesystems and today’s object stores choke on millions of tiny files, lock handling, and fragmentation; in contrast, classic databases solved these “space-management” headaches decades ago.

• Because databases already handle storage, compression, atomic writes, and bit-rot, pushing data into object stores just shifts cost and pain to users while giving cloud vendors more lock-in.

• Data lake tables still need fast, tiny metadata updates, yet object stores are not the place for that; even at multi-petabyte scale, metadata stays small enough to fit on a single server, so a real metadata database beats file blobs.

• Conclusion: openness matters but be wary until it tackles metadata and lock-in as cleanly as proven database engines do.

Our Takeaway: Before betting a 100% on open-table formats, scrutinise how they handle metadata at scale to unlock the best RoI.

💥 Spark: Still Got the Fire?

Another r/dataengineering thread asks whether Apache Spark is passé in 2025—and finds the community surprisingly loyal. “Yes—when the data’s truly big.”:

• Petabyte shops say Spark “just works” for multi-PB tables

• Critics highlight launch latency and opaque debugging, steering smaller teams to Polars first

• Hidden cost of “tool churn”: migrating stacks as volumes grow burns hiring and context-switching budget

• Many would still choose Spark for once workloads cross the Petabyte barrier; below that, lightweight engines win on ergonomics

Our Takeaway: Spark isn’t dead—just reserve it for workloads that justify the cluster costs.

🧪 Unit Tests ≠ Data Quality Checks

Another great Reddit debate separates build-time tests from run-time data validation.

• Unit testing is about making sure that some dependency change or code refactor doesn’t result in bad code that gives wrong results. Integration and e2e testing are about the whole integrated pipeline performing as expected.

• Data quality checks are about checking the integrity of production data as it’s already flowing, each time it flows. It’s a “runtime” construct, ie after your code is released.

• The community agrees the two are complementary; DQ systems should be platform-agnostic and drive feedback to data stewards.

• Overlap is rare: unit tests may cover complex transforms, but DQ thresholds still catch unexpected business-data anomalies.

Our Takeaway: Keep CI tests and data-quality observability as distinct layers—each fails fast in its own domain.

We have made a faster Froid-inspied UDF engine!

• UDFs let you keep loops and business rules right inside SQL, but the usual multi-statement kind makes the engine hop out for every row, killing speed.

• That drag comes from three things: context-switch overhead, row-by-row work, and an optimizer that can’t peek inside the function—turning some queries 10-1000× slower.

• Databricks and Snowflake sidestep the pain by only allowing single-expression UDFs, which forces devs to cram logic into one line or shift it to external code.

• e6data’s Froid-style “inliner” dissolves multi-statement UDFs back into one relational plan, so the optimizer can parallelize freely and queries run orders faster—without changing user code.

Community & Events

• Lakehouse Days Replay – The full recording of our “​From Stream to Lakehouse” is now live on e6data’s YouTube.

• Inside e6data’s Froid-Inspired UDF Engine: an in-depth blog series by our engineering team

• Meet Us at MachineCon USA – We’ll be on-site; come say hi!

• Hiring: We’re growing! Check out open engineering roles [here]

🦀 How to Write Idiomatic Rust

David Drysdale’s freely available Effective Rust condenses years of field experience into short, actionable items here:

• Six chapters cover types, traits, concepts, dependencies, tooling and beyond—35 focused lessons in total.

• Rust’s rigorous type system is the star: if it compiles, it probably works, though newcomers still wrestle with lifetimes and the borrow checker.

• Practical advice spans Clippy linting, CI setup and taming dependency creep for production-ready crates.

Our Takeaway: Must-read. Rust is taking over the code.

🔍 Why Cursor Might Dump Vector Search for Plain Text

A LinkedIn post by Jacky Liang from Tigerdata argues that grep-style lexical search beats embeddings for code—and Cursor could follow Claude Code’s lead.

• Claude Code uses only exact-match tools like grep and outperforms Cursor’s vector pipeline for code retrieval

• The architects behind Claude Code are joining Cursor, sparking talk of ripping out vectors or relegating them to coarse filtering

• Commenters note a broader industry shift: vectors excel at semantic chat, but keyword search wins for structured codebases

Our Takeaway: Expect a renaissance of smarter lexical search in developer tools—embeddings aren’t always the right hammer.

🚨 Supabase MCP’s Can Dump your Private Tables?

A new write-up shows how a single malicious support-ticket message can make Supabase’s MCP dump your private tables.

• An attacker embeds SQL-style instructions in a ticket; the Cursor IDE’s LLM agent obediently executes them and posts the results back to the thread.

• Because the agent runs with the all-powerful service_role, it sails past row-level security and can read anything.

• Fixes proposed: run MCP in read-only mode and add prompt-injection filters to scrub user text before it hits the model.

Our Takeaway: Be extra careful with IAM in the new and evolving era.

💡 Apache Arrow Summit 2025 – CFP Now Open!

Apache Arrow Summit is on 2 Oct 2025 in Paris, hosted by PyData Paris. It’s the place to gather the Arrow community, share what’s coming next, and spark new ideas.

• Call for Proposals is open now – submit talks by 26 July

• Topics welcome: Arrow internals, ecosystem tooling, real-world use cases, road-map discussions

• Meet contributors, users and maintainers face-to-face in the Paris!
  

Community & Events:

• Event: Lakehouse Days — “Real‑time Streaming Ingest” is tomorrow at, AWS Office, Bangalore with speakers from Confluent, AWS, and e6data. Subscribe to the calendar to register early for future events!

• We are going to MachineCon USA in the New York City this July 25th!

• Hiring: We’re growing! Check out open engineering roles [here].

🔧 We used a few agents to bring in JUNit runner support for regression run, it sort of worked?

An engineer from our team pointed IntelliJ’s Junie agent (Claude 3.7 under the hood) at a hand-rolled regression harness (~2000 SQL queries) and asked it to surface each query as an IDE-runnable JUnit test. Here is what went down:

• Agent #1 — Junie + JUnit 4 reflection adapter. A single prompt asked Junie (IntelliJ’s Claude-powered agent) to analyse RegressionTest.runTests and emit an adapter that registers every query as a JUnit test. Junie delivered code that works, but hangs on brittle reflection hooks and produces hash-based test names with no Run/Debug actions.

• Agent #1, retry — JUnit 5 Parameterized / Dynamic tests. After nudging Junie toward Jupiter, it rewrote the runner, solving the name issue but introducing new ones: incremental discovery kept appending tests mid-run; only ~70 % of the 2006 tests were detected, and per-test stdout/stderr never reached the console.

• Edge-cases that still bite.

  • IDE agent loops on feedback instead of asking for help.

  • Environment variables (default schema, storage endpoint) are ignored, so failed tests hit remote buckets.

Our Takeaway: IDE agents can scaffold a runner in minutes, but deep plumbing—stable discovery, clean logs, env-aware execution—still needs a human who knows JUnit extension points. Treat today’s agents as junior devs: great at boilerplate, lost in integration hell.

🔄 CDC vs. Daily Snapshots?

Data engineers on Reddit recently argued (and sort of concluded) that CDC is cheaper, safer and more granular than nightly full snapshots:

• Pulling changes straight from the DB WAL avoids table locks and SELECT * hell.

• Real-time extraction doesn’t force real-time delivery—buffer it rather then landing hourly or daily.

• For 100M+ rows tables, daily snapshots hammer network & compute; Debezium-style CDC streams are faster.

Our Takeaway: Unless your tables are tiny, CDC + append-only logs win on both cost and correctness, while still letting teams materialize snapshot views whenever they like.

🦆 DuckLake vs. Iceberg — Metadata Wars (Again?)

r/dataengineering is hot with another metadata war this week:

• One-binary catalog. DuckLake boots from a single DuckDB file; you’re querying with plain SQL in minutes, no Hive metastore, no Spark session spin-up.

• Adoption hurdle. Redditors call it a “pure improvement” over Iceberg but say they’ll jump only after Spark / Trino / Flink connectors and real petabyte tests—land.

• Could just fold in. Even Iceberg fans note its catalog is already pluggable; a SQL-backed option like DuckLake might slip upstream instead of forking the ecosystem.

Our Takeaway: The fight isn’t about table files but about where metadata lives. We will watch out for connector support: once the major engines speak DuckLake, JSON catalogs might become legacy.

💡 We’re cooking up a Query & Cost Optimization Hub

We’re launching: a hub which distills cost-optimising and latency-crushing playbooks

• One for or every major compute engine—Snowflake, Databricks, BigQuery, Redshift, Athena, ClickHouse, Fabric, Starburst, and more.

• Filter by engine, choose “query” or “cost” mode, and grab beginner-to-advanced tactics.

First one is out with Snowflake Cost Optimisation for Beginners, rest to follow soon.

Community & Events:

1. Event: Lakehouse Days — “Real‑time Streaming Ingest” on 12 July, AWS Office, Bangalore with speakers from Confluent, AWS, and e6data. Subscribe to the calendar and register before seats run out!

   

2. Hiring: We’re growing! Check out open engineering roles [here].

🤖 Is Your Snowflake Bill Really High? Here is How to Check

• Warehouse size – Anything above LARGE (> 8 credits/hr/cluster) is too much. Cost doubles each size step (XS 1 → 4XL 128 credits/hr); leave a 4XL up for 8 hrs and you burn $2 k–$6 k.

• Idle ratio – If a warehouse sits idle > 30–40 % of the time, you’re paying for nothing (billing is per-second but with a 60s minimum).

• Cloud-services share – When cloud-services charges exceed 10 % of daily compute, you’re past the free tier and likely over-using serverless features.

• Query cost – Interactive query over 0.1 credit (~$0.20–$0.60) starts to sting; Dashboards usually finish under 0.01 credit.

• Account run-rate – If your 90-day burn rate is >20 % above your annual commitment pace, you’re in high-cost territory.

• Serverless features – Snowpipe, materialised views, auto-clustering, etc. should stay <15 % of total compute; once they creep higher, review them.

Takeaway – If two or more bullets fire, you have real savings hiding in plain sight. Watch next week for our beginner & advanced optimisation guides.

📣 Call for Proposals: ACM India IndiCS Seminars

IndiCS is ACM India’s Dagstuhl‑style, fully‑immersive seminar series that brings 45 invited researchers together for 3–5 days of deep dives at rotating venues across India. The program funds student travel and encourages open‑ended collaboration on frontier CS topics.

Next deadline: 31 July 2025 (for seminars in autumn/winter 2025). A second window closes 31 December 2025 for spring/summer 2026 slots.

🐌 Spark’s Hybrid Engine: Unified ≠ Fast

Vanilla Spark juggles three execution paths—row‑oriented Volcano, JVM‑generated WholeStage, and a thin slice of vectorization. That Swiss‑army flexibility serves logs, streams, ML, and SQL in one runtime, but it also brings GC churn, branchy code, and frequent fall‑backs when UDFs or exotic types appear.

For classic lakehouse OLAP (Parquet + flat schemas, few UDFs), a fully vectorized kernel wins. Vendors are swapping Spark’s final stage:

• Databricks Photon, Apache Gluten, DataFusion Comet drop C++/Rust kernels into the DAG and see 2–4× speed‑ups with no API break.

• Spark’s own vector path is narrow—only some Parquet/ORC scans use SIMD batches—so WholeStage still dominates most queries.

Takeaway: Spark is slow for pure OLAP by design, not accident. If your workload is > 90 % columnar SQL, a vector back‑end or Arrow‑native engine can reduce runtime and cost—keep vanilla Spark for truly mixed jobs. Read more.

🧪 Why Data Engineers Still Skip Tests — Reddit Reacts

A r/dataengineering thread boiled the problem down to four points:

1. Cost & deadlines – quick‑and‑dirty DAGs ship first; bug‑fixes bill to the next sprint.

2. Skills gap – many DEs start in SQL/Excel land, meet pytest only after pain.

3. Unstable inputs – schemas drift, APIs mutate; unit tests feel brittle, so teams lean on downstream DQ checks.

4. Tooling debt – unclear lines between unit, data‑quality, and prod monitors stalls adoption despite dbt, GX, SQLMesh.

Takeaway: testing is an org‑level habit, not a novel tech problem. Seed the habit with lightweight assertions + CI on critical transforms—rerunning petabyte jobs is the expensive path.

🧠 Ask in English, Get Perfect SQL — Even Across Hundreds of Tables

We’re sipping our own Kool-Aid again at e6data: NL2SQL v0 plugs into e6-MCP and turns a plain-language prompt into a rock-solid, schema-faithful query—even when your data stores looks like a city map.

How? An agentic triple-play:

• Vector search surfaces the likeliest tables & columns

• Random-walk graph traversal traces real relationships, not guesswork

• Cross-attention re-ranker locks the winning set, then lets the agent self-reflect and patch any schema slip-ups

The result: accurate answers, zero hard-coded limits. Where Databricks stalls at 25 tables and Snowflake taps out at 10, NL2SQL sails past—ready for enterprise-scale schemas.

➡️ Join the private beta

Community & Events:

1. Blog: Iceberg Catalogs 2025 — a deep dive into modern metadata management across Project Nessie, Apache Gravitino, Apache Polaris, Lakekeeper, and more. [Read here]

2. Event: Lakehouse Days — “Real‑time Streaming Ingest” on 12 July, Bangalore with speakers from Confluent. Subscribe to the calendar for early registration.

3. Hiring: We’re growing! Check out open engineering roles [here].
   

   Thanks for reading Data Engineering ACID! This post is public so feel free to share it.

   Share

   
   

🤖 Deep-Fake Devs: Well, this could have been a sitcom episode

Recently, a hiring manager spent 30 minutes interviewing a deepfake AI bot (as a data engineer), AND the internet is hilarious about it.

Redditors point out their irony: HR wanted algorithms to pick humans, so now algorithms are applying for the jobs. Since ATS filters let bot-written CVs sail through (thanks to tools like Jobscan, HyperWrite’s Resume Aligner), real engineers suffered at times. Give the entire thread a read here.

Takeaway: The HR industry for tech hiring is (and will more and more) adding checks like live‑coding, camera checks, put your hand in front of your face, or “tell me five facts about X, unrelated to engineering”‑style prompts to help things get better.

⚙️ Modern ETL is all about living up to modern software deployment principles

In yet another ETL rant, a Reddit user claimed that an Oracle migration to dbt went wrong from one stored procedure to hundreds of models, stretching runtime from 1 hour to 4 hours. The community notes that dbt’s strength is testable, incremental ETL, not raw speed, and that each model incurs orchestration overhead, and Oracle’s optimizer prefers fatter queries.

Takeaway: Power users fix it by materialising large views, batching models, or shifting heavy lifts to Spark/Python.

🛡️ Shift Left or Shift Blame?

Reddit is split on yet again or whether this is an actual trend or just another marketing gimmick: “start right” (spray downstream observability) then “shift left” (convert findings into upstream data contracts).

Some people argue it is the latter, while few give examples like- If a dashboard user spots bad numbers, you trace the issue back to the raw data and repair it there. That single upstream fix saves every downstream report, not just the one someone complained about. Which is why shift left is more of a stronger detection and prevention strategy.

Takeaway: Choose your battles depending on your scale. Meanwhile, vendors already sell both ends of the funnel.

We’re fixing multi-cloud pain, too — query data wherever it sits

We’re sipping our own Kool-Aid at e6data: Hybrid Lakehouse lets you drop a single, location-aware cluster next to every storage bucket—on-prem, AWS, or “that other cloud.” Compute stays put, so nothing mixes across regions, cutting egress fees by up to 99% and keeping every table compliant and governed.

With affinity-aware execution and streaming ingest baked in, dashboards light up in sub-minute latency—no duplicate copies, no forklift migrations, and zero ACL drift.

Spin up a hybrid cluster? ➡️ Book a 15-min walkthrough

Some cool stories from our team:

1. Vector Search on S3 through SQL- breaking down e6data’s ability to query unstructured data

2. Solving Geospatial Analytics Performance Bottleneck: H3 vs Quadkey: Unlocking geospatial analytics in production at scale

3. Hive Metastore as an Apache Iceberg Catalog: moving pieces, end-to-end setup, locking, and concurrency quirks
   

Meet us on the Road

We had fun at the Data + AI Summit, Snowflake Summit, and AWS Summit this week. Here’s what’s went down:

• Vishnu took the stage at the Data + AI Summit showing how e6data lifts your lakehouse to any environment—cloud, on-prem, or hybrid.

• Bharath, our Head of Product, took the stage at the AWS Summit to present how e6data powers vector search.

• We exchanged ideas with a mountain of cutting-edge teams, stocking the team’s calendar with months of interesting conversations.

Also, the Lakehouse Days community has officially 1200 members and counting from top engineering teams like Netflix, Uber, Meta, and more. We are hosting the next one on July 12th in Bangalore, subscribe to the calendar for the first updates!

🔥“Spark Is the New Hadoop” — or Is It?

A widely shared post argues we’ve reached peak Spark: JVM GC pain, sluggish startup, and the rise of Rust/C++ query engines (Photon, DataFusion + Comet, Velox, Daft) point to a future with lighter, Arrow‑native back‑ends. Critics counter that Spark’s API and rich ecosystem—not its engine—are the real stickiness, and drop‑in Rust/C++ cores may keep it alive.

Takeaway: Decouple APIs from engines and keep an eye on emerging Rust projects before declaring Spark “dead.”

▶️ YouTube‑Scale Guardrails

Decades in, YouTube still breaks the curve: 20 million videos are uploaded per day, 20 billion videos are stored, and 3.5 billion daily likes.

At that size, a forgotten SELECT * or rogue cluster can cost millions. Therefore, the data infra team enforces automated guardrails:

• Mandatory partitions (upload_date or shard_id)

• Query cost alerts ≥ $1,000

• “Design for safe failure”, so teams accidentally don’t melt the budget
  

🐻 Polars & the “Goldilocks” Zone

Early adopters report 5–7x speed‑ups (and ~80 % cost cuts) by replacing PySpark jobs with Rust‑based Polars on a single r6g.8xlarge. Key features:

• Lazy execution by default

• Full threading on Arrow columns

• Upcoming distributed mode via Ray

• Optional GPU acceleration through RAPIDS

Expect Polars + Iceberg to gobble the 10 GB to 1 TB jobs that don’t justify a full cluster.

We are working on something cool too! (land and query event data in <15 seconds)

We’re drinking our own Kool-aid at e6data: the team is working on a new product offering that lands Kafka (and soon CDC) streams directly into your lakehouse and makes them queryable in sub-second latency—no Flink clusters, no real-time OLAP database, no data shuffles. This can help reduce time-to-insight by 95%, and relieve some of that infrastructure headache. Look for native CDC support and mobile-SDK ingestion in v2 next quarter.

Early‑access ➡️ Book a 15‑min walkthrough

Fresh From the Team

• Podcast: Decoding e6data’s Architecture – our Head of Engineering with Pete Soderling, Zero Prime (listen)

• Blog: Why Catalogs Matter – the Book‑keeping of Apache Iceberg (read)

• Tech Note: Eliminating Redundant Computations with Automatic CTE Detection (read)
  

Meet Us on the Road

We are at the Snowflake Summit, Databricks’ Data + AI Summit 2025, and the Gartner Summit this month. Here’s what’s going down:

• Vishnu is taking the stage at the Data + AI Summit to present how e6data can help deploy your data platform (like Databricks) to any environment: cloud, on-prem, or hybrid.

• The engineering team is demoing our real-time streaming ingest to the world at our booths at both the Summits

• Giveaway: Enter to win limited‑edition Star Wars LEGO® sets
  

Lakehouse Days Community

We just crossed 1,100 members from teams like Netflix, Uber, and Meta. Next meetup: June 21 —add the calendar to get first dibs. Subscribe to our YouTube channel for previous session recordings.

Thanks for reading—see you next month! 🚀