1. Data Lakehouse — Introduction
What a lakehouse actually is, why the industry converged on it, and the Iceberg concepts you need before any of the GCP patterns make sense.
Why the lakehouse exists
The warehouse and the lake, and why clients ended up with both
Warehouses gave transactions, governance, and fast SQL but locked data into a proprietary format. Lakes gave open, cheap, engine-agnostic storage but no table semantics. Most clients run both and pay for both.
The copy problem
The same facts land in GCS, get loaded into BigQuery, get extracted again for the ML team, and get exported once more for a partner. Every copy is a governance boundary, a freshness lag, and a reconciliation job.
What "lakehouse" claims to fix
Warehouse table semantics — ACID commits, schema evolution, time travel — applied directly to open files in object storage, so one physical copy serves every engine.
What it does not fix
Query latency, small-file problems, governance across engines, and the operational work of maintaining the files. The lakehouse moves these problems; it does not remove them.
The separation that makes it work
Storage — open files in object storage
Parquet (usually) in GCS, S3, or ADLS, owned by the customer, readable by anything. Object storage is cheap, durable, and has no compute attached.
Table format — the metadata layer that turns files into a table
A specification that records which files belong to the table right now, what the schema is, and how it is partitioned. Without it, a directory of Parquet files is a directory, not a table.
Catalog — who owns the pointer to the current table state
The single authoritative answer to "what is the current metadata file for this table". This is the component that makes concurrent writes safe, and the one most teams underestimate.
Compute — interchangeable engines
BigQuery, Spark, Trino, Flink, DuckDB. Any engine that can talk to the catalog and read the files can query the table. This is the payoff of the whole design.
Why "separation of storage, catalog and compute" is the actual pitch
Each layer can be scaled, priced, replaced, and governed independently — and each layer becomes something you now have to own.
Apache Iceberg
The metadata tree
Catalog pointer → metadata JSON → manifest list → manifest files → data files. Walking this chain once makes every later behaviour (snapshots, pruning, stale metadata) obvious.
Snapshots and atomic commits
Every write produces a new immutable metadata file and a new snapshot; the commit is the atomic swap of the catalog pointer. Readers never see a partial write.
Optimistic concurrency
Two writers can prepare commits simultaneously; the loser detects the pointer moved and retries. This is why the catalog must be the only authority.
Hidden partitioning and partition evolution
Partition transforms are recorded in metadata rather than in directory paths, so consumers never write WHERE partition_col = ... and the partition scheme can change without rewriting data.
Schema evolution by field ID
Columns are identified by a stable ID rather than by name or position, which is what makes schemas portable across engines and safe to rename.
Row-level deletes: copy-on-write vs merge-on-read
The choice that determines whether your write cost or your read cost goes up. Copy-on-write (the default for delete, update, and merge) rewrites affected files and suits read-heavy tables; merge-on-read records deletes separately and suits frequent small changes. In v2 that means position or equality delete files; in v3, binary deletion vectors — at most one per data file per snapshot, which keeps reads bounded. Get this wrong and the table is either unwritable or unreadable at scale.
Format versions — v2, v3, and v4
Checked on 2026-09-25 against the Iceberg spec. v2 is still the default for new tables. v3 is complete and adopted: deletion vectors, row lineage, default column values, the variant, geospatial, and nanosecond-timestamp types, multi-argument transforms, and table encryption keys. Engine support is uneven — BigQuery reads v3 external tables in Preview, the Lakehouse runtime catalog supports v3 in Preview but BigQuery cannot write v3 tables and the new types are unsupported, Trino's v3 support is experimental, and Databricks supports most of it on DBR 18. v4 is under development and not adopted. Default to v2 for a multi-engine GCP table unless a v3 feature is a real requirement and every engine that touches the table supports it.
What Iceberg does not give you
Not a database, not an engine, not a governance model, and not a maintenance service. No clustering, no automatic statistics, no query acceleration.
Iceberg, Delta Lake, and Hudi
The short orientation
All three solve the same problem with a metadata layer over Parquet. Iceberg has the broadest multi-vendor write support and the only vendor-neutral catalog protocol (REST). Delta is Databricks- and Spark-centric; other engines mostly reach it through UniForm's Iceberg metadata, which is read-only. Hudi is strongest at record-level upserts and CDC but has the narrowest engine support. With v3 adding deletion vectors, row lineage, and variant, the feature gap between Delta and Iceberg has largely closed — the difference that remains is who can write.
Why this knowledge base anchors on Iceberg
Google's managed lakehouse products, the client demand we see, and cross-cloud portability all point the same way. Every catalog in this knowledge base — Google's, Unity Catalog, Snowflake Horizon, Glue — now speaks Iceberg REST, which is what makes Chapter 4's federation possible.
Lakehouse vs data lake vs data warehouse
Side-by-side comparison
Storage ownership, table semantics, governance model, engine choice, operational burden, and cost shape across the three.
The honest version of the comparison
A managed warehouse still wins on performance, integration, and total operations for single-engine workloads. The research benchmark in this knowledge base measured native BigQuery as fastest and cheapest in slot terms.
When a lakehouse is the wrong answer
One engine, one team, no portability requirement
If BigQuery is the only consumer and will stay that way, a lakehouse adds catalog operations and maintenance jobs in exchange for benefits nobody will use.
No owner for table maintenance
Compaction, snapshot expiry, and orphan cleanup are recurring work. Without a named owner, lakehouse tables degrade silently.
"Open format" as a procurement checkbox
Portability that is never exercised is a cost, not a hedge. Ask what specifically would be migrated, to where, and by when.
Terms used in this knowledge base
Table format, catalog, metadata file, manifest, snapshot, commit, credential vending, compaction, snapshot expiry, orphan file, time travel, hidden partitioning — defined inline here and reused across all modules.