Data Lakehouse on GCP
An internal reference for Aliz data engineers on data lakehouse architecture, anchored on Apache Iceberg and Google Cloud. Use this to build the mental model before a client conversation, to choose between the GCP implementation patterns with evidence rather than preference, and as LLM context when drafting architecture recommendations.
This module is in progress. Every page currently holds a section-level outline rather than finished content — see _PROJECT_PLAN.md in the repo for scope, task sequence, and open questions.
How to use this knowledge base
"I need the mental model" — Read Chapter 1. If you only read those two modules, you should still be able to hold a credible lakehouse conversation with a client.
"I need to pick an architecture" — Read Chapter 2. Each pattern module ends with decision gates; the benchmark evidence and its caveats live in modules 03 and 04.
"I need to pick an engine" — Read Chapter 3. The four modules share a structure so they can be read side by side, and module 09 closes with the cross-engine comparison.
"The client has two catalogs, or data in another cloud" — Read Chapter 4. Module 10 covers federating catalogs and when a shared Iceberg REST catalog is the better answer; module 11 covers querying AWS-, Databricks-, or Snowflake-hosted Iceberg tables from GCP without migrating them.
Learning path
Four chapters. Each module is self-contained — terms are defined inline, and cross-links are "go deeper" pointers rather than prerequisites.
Ch 1 — Foundations
| Module | What it covers |
|---|---|
| 01-introduction | Why the lakehouse exists, the storage/catalog/compute separation, Iceberg's metadata tree and snapshot model, and when a lakehouse is the wrong answer |
| 02-basic-architecture | The component map, read and write walkthroughs, the commit protocol, table maintenance as a workload, governance across a split architecture |
Ch 2 — GCP Implementation Patterns
| Module | What it covers |
|---|---|
| 03-gcp-bigquery-managed-iceberg | Storage in GCS, catalog and engine in BigQuery. Full DML and automatic maintenance, but only column-level governance and no external writers; also covers BigLake external Parquet tables |
| 04-gcp-lakehouse-runtime-catalog | A managed Iceberg REST catalog outside BigQuery, reachable by any compatible engine. Credential vending, the governance gap, and the metadata-pinned external table workaround |
| 05-gcp-dataproc-metastore | Managed Hive Metastore as the catalog — the older setup, its narrowing support, when it still fits, and the migration tooling off it |
Ch 3 — Transformation Engines
| Module | What it covers |
|---|---|
| 06-bigquery-as-transformation-engine | Serverless SQL over lakehouse tables, catalog compatibility per pattern, cost shape, and the richest governance surface of the three |
| 07-spark-as-transformation-engine | The Iceberg reference engine — broadest write support and the owner of table maintenance |
| 08-trino-as-transformation-engine | Federated interactive SQL and its GCP deployment problem |
| 09-databricks-as-transformation-engine | Managed Spark with its own catalog — Delta versus Iceberg, UniForm, Unity Catalog alongside the GCP catalog, and the cross-engine comparison |
Ch 4 — Multi-Catalog and Cross-Cloud
| Module | What it covers |
|---|---|
| 10-catalog-federation | Exposing one catalog's tables through another — how it differs from query federation and catalog sync, write ownership, metadata freshness, the Lakehouse runtime catalog's federated catalogs, and why governance does not federate |
| 11-cross-cloud-data-access | Querying Iceberg tables in another cloud through a federated catalog — transport, caching, egress cost, residency and CMEK constraints, and when to federate, replicate, or migrate |
Research inputs
The _research/ folder in this module holds the raw material behind the performance and tradeoff claims — benchmark scripts, results, and the native-versus-Iceberg-versus-Parquet analyses. It is underscore-prefixed, so it is not published to the site; read it in the repo.
| File | What it is |
|---|---|
benchmark_results.md | Measured comparison of six BigQuery table implementations on a two-table join |
biglake_perf_benchmark.sh | The benchmark harness — creates all six table types and runs the comparison |
iceberg_authorized_view_simulation.sh | Proof of the authorized-view-over-pinned-metadata workaround, including the catalog metadata resolution |
gcp_managed_catalog_iceberg_vs_native_and_aws_migration.md | Native BigQuery versus managed-catalog Iceberg, plus the AWS migration path |
parquet_external_table_tradeoffs.md | Native BigQuery versus BigLake external Parquet tables |
ods_parquet_external_table_datalake_design_summary.md | A worked design summary with verified Singapore list pricing |
All measurements are from three repetitions of a single two-table join on one file layout. They are directional, not conclusive, and the direct-catalog path has an unexplained 2x bytes-processed reading that must be investigated before it is used in any cost estimate. Never quote a number from here to a client without its caveat.
GCP's lakehouse products move quickly. On 2026-04-20 BigLake became Lakehouse and BigLake metastore became the Lakehouse runtime catalog (APIs, CLI, and IAM still say biglake); Dataproc is now Managed Service for Apache Spark and Dataplex is Knowledge Catalog. The research inputs were verified 2026-08-07 and 2026-08-14; product claims across all modules were re-checked against current documentation on 2026-09-25, and many of them are Preview or Beta. Re-verify any product limitation, price, or feature claim against current documentation and date-stamp it before it goes into a client deliverable.