9. Transformation Engine — Databricks
A managed Spark platform with its own catalog and its own native table format. On GCP it shows up as the client's existing platform far more often than as Aliz's recommendation.
Claims below were checked against the Databricks on Google Cloud documentation (the /gcp/ docs, not AWS or Azure — feature and region availability differ by cloud) and Google's Lakehouse documentation. Several integrations are Beta or Preview; each is marked. No Aliz hands-on test exists yet.
The role Databricks plays
Spark, plus a platform
The same engine as Module 07, packaged with managed compute, notebooks, jobs, SQL warehouses, and Unity Catalog. The engine is the familiar part; the platform around it is what the client is really buying.
Why it appears in GCP engagements
Usually inherited: a multi-cloud client standardised on Databricks elsewhere, or a data science team that brought it in. Treat "keep Databricks" as a constraint to design around, not a preference to argue with.
Databricks on Google Cloud
The control plane runs in Databricks' account. Classic compute runs on Compute Engine in the client's own project and VPC (it moved off GKE); serverless compute runs in a Databricks-hosted plane in the same region, available in roughly 14 GCP regions. Billing is Databricks units plus, for classic compute, the underlying GCP infrastructure. Check region support early — several features below are region-limited on GCP.
Delta, Iceberg, and UniForm
Delta is the native format
Databricks is Delta-first. Iceberg support is real but is the interoperability path, not the default. This is the one engine in the knowledge base where the anchor format is not the engine's own.
Iceberg interoperability options
Two options, both available on GCP. Unity Catalog managed Iceberg tables (GA) — Iceberg all the way down, writable by external engines. Delta with Iceberg reads (UniForm, DBR 14.3 LTS+) — Delta tables that also publish Iceberg metadata, read-only to Iceberg clients.
Unity Catalog's Iceberg REST endpoint
Unity Catalog exposes an Iceberg REST catalog at /api/2.1/unity-catalog/iceberg-rest. External engines can read and write managed Iceberg tables, and only read UniForm and foreign tables. It needs external data access enabled on the metastore, the EXTERNAL USE SCHEMA privilege, and external locations — credential vending does not work on workspaces using default storage.
What "Iceberg-compatible" actually means here
Readable as Iceberg by other engines is not the same as writable by them. Establish the direction of every table before promising interoperability.
Catalog compatibility
Read this against the same diagram in the other Chapter 3 modules. Thick arrow: first-class read and write. Thin: works, with caveats. Dotted: read-only or degraded.
Databricks is first-class only against its own catalog, which is none of the three GCP patterns — every GCP-owned table is read-only to it. That is the defining fact of this module, and the reason Module 10 exists.
Unity Catalog
The native path. Full write, governance, and maintenance — but it is a catalog alongside the GCP one, not one of the three patterns. The reverse direction also works: GCP engines can read Unity-owned tables through a Lakehouse federated catalog (Preview since 2026-04-22 — see Module 10).
Lakehouse runtime catalog (Pattern 2) and BigQuery-managed Iceberg (Pattern 1)
One integration covers both: Unity Catalog's Google Cloud Lakehouse catalog federation (Beta on GCP), a GOOGLE_CLOUD_LAKEHOUSE connection authenticated with a service-account key. Read-only, needs DBR 18 LTS+ or a pro/serverless SQL warehouse, does not support dedicated clusters, and skips tables with TIME columns.
Dataproc Metastore (Pattern 3)
Unity Catalog federates external Hive metastores read-only — useful as a migration bridge, not a destination. Only Databricks' own internal Hive metastore is federated read-write.
Query federation is a different thing
Lakehouse Federation's JDBC connection to BigQuery pushes the query down to BigQuery, which executes and bills it. Useful for ad-hoc joins; not the same as Databricks reading the table.
The write path
What Databricks supports
Full DML on Delta and on Unity Catalog managed Iceberg. Foreign Iceberg tables — including every table owned by a GCP catalog — are read-only.
Commit semantics
Optimistic concurrency, as with every engine here — but a table has one authoritative catalog. The read-only federation rules make the one-writer design the default: Databricks writes what Unity owns, GCP engines write what the GCP catalog owns.
Maintenance ownership
Predictive optimisation (OPTIMIZE, VACUUM, ANALYZE) covers Unity Catalog managed Delta and Iceberg tables only — not external or foreign tables. It needs the Premium plan and is region-limited on GCP. Tables owned by the GCP catalog still need a maintenance owner on the GCP side (Modules 04 and 07).
Performance and cost shape
The cost model
Databricks units on top of GCP compute, with separate rates for jobs, SQL warehouses, and serverless. Compare against BigQuery slots and Managed Service for Apache Spark on the same workload, not on list price.
Where Databricks wins and loses
Wins on notebook-to-production workflow, ML integration, and teams already fluent in it. Loses on being a second platform and a second bill inside a GCP estate.
No Aliz benchmark exists for Databricks against these catalogs. Do not make comparative performance or cost claims to a client without running one.
Governance
Unity Catalog's model
Grants, ABAC row filter and column mask policies, lineage, and audit — comparable in depth to BigQuery's, but a separate implementation.
Governance for external readers
Normally a table with row filters or column masks cannot be read through the Iceberg REST endpoint at all. Cross-engine ABAC (Beta on GCP) lets external Spark read it by filtering through Databricks serverless compute — reads only, recent Iceberg-Spark or Delta-Spark versions only, no dynamic views.
The integration problem
BigQuery's policies do not apply to a Databricks reader, and Unity Catalog's do not apply to a BigQuery reader of a federated table. Two rich governance surfaces over the same tables is worse than one, not better.
Development experience and orchestration
Client surface
Notebooks, Databricks SQL, Python/Scala, and dbt. The notebook-centric workflow is usually why the data science side of the client wants to keep it.
Orchestration
Databricks Workflows versus Cloud Composer, and where the handoff between them sits when both exist.
When Databricks is the right transformation engine
Decision gates
The client already runs Databricks and the platform team supports it; ML and data engineering share one workspace by design; or multi-cloud consistency outweighs GCP-native integration.
Anti-patterns
Introducing Databricks into a GCP-only estate for SQL transformation BigQuery already covers; expecting Databricks to write GCP-owned tables; building on Beta federation without a fallback; and treating UniForm as two-way interoperability.
Choosing between the four engines
Where most engagements land once all four engine modules have been read:
Each engine doing what it is first-class at, over one set of tables. Trino and Databricks are dotted for the same reason: each costs an extra governance implementation, and Databricks also brings an extra catalog.
The comparison
BigQuery, Spark, Trino, and Databricks side by side on catalog support, write maturity, maintenance capability, governance depth, cost model, and operational burden.
The common landing point
Spark for ingestion and maintenance, BigQuery for transformation and governed serving, Trino only when federation or engine neutrality is a real requirement, and Databricks when the client already runs it — writing the tables Unity owns and reading the ones GCP owns.