5. GCP Pattern 3 — Dataproc Metastore
A managed Hive Metastore as the catalog. The older setup — still the right answer for some estates, and a liability for others.
No benchmark exists for this pattern. Product status, compatibility, and migration tooling below were checked on 2026-09-25 against the Dataproc Metastore release notes and the Lakehouse migration guide. Dataproc is now Managed Service for Apache Spark; the metastore kept its name.
The shape of the pattern
The same frame once more. Note which engine now gets the dotted line — that inversion is the pattern in one glance.
In Pattern 1 the Hadoop engines were dotted; here BigQuery is. That is the whole decision in a picture — this pattern serves a Spark-centric estate, not a BigQuery-centric one.
Component placement
Data files in GCS. Catalog is a managed Hive Metastore (HMS) service. Engines are Spark, Hive, and Trino on Managed Service for Apache Spark clusters. BigQuery has no current documented way to query DPMS-registered tables; Google's answer is migrating to the Lakehouse runtime catalog.
Hive Metastore in one paragraph
A relational database holding databases, tables, columns, partitions, and storage locations, exposed over the Thrift protocol. Predates Iceberg by a decade and was designed for directory-partitioned Hive tables.
HMS with Iceberg tables
HMS can serve as an Iceberg catalog through Iceberg's HiveCatalog: Spark reads and writes, Hive only reads. It works, but leaves you with Iceberg semantics on a catalog not designed for them — and it does not work on Spark image 3.0.
Why it still exists
The Hadoop migration path
Clients arriving from on-premises Hadoop already have an HMS with thousands of table definitions, and lift-and-shift to Dataproc keeps them working unchanged.
Ecosystem breadth
Almost every engine in the open data ecosystem can read HMS. Iceberg REST support is newer and less universal.
Existing tooling assumptions
Legacy jobs, schedulers, and governance tools that address tables through HMS.
Setting it up
Creating the service
Dataproc Metastore 2 (horizontal scaling, regional HA) versus the tiered DPMS 1; Hive 3.1.2 (default) or 2.3.6; and the gRPC endpoint (recommended, IAM-authenticated, required for federation, irreversible) versus the legacy Thrift endpoint. New multi-region services can no longer be created (checked on 2026-09-25).
Attaching Dataproc clusters
Cluster configuration, and why a shared metastore across ephemeral clusters is the point of the managed service. Only image 2.x clusters — image 3.0 (Spark 4.1) is incompatible with Dataproc Metastore.
Reaching it from BigQuery
There is no current documented path. DPMS "metadata federation" runs the other way — it exposes BigQuery datasets and DPMS instances to Hive and Spark through one endpoint. For BigQuery access, migrate.
What HMS gives you and what it lacks
Compared with an Iceberg REST catalog
No credential vending, no standard REST interface, weaker multi-tenancy, Thrift and network configuration to manage, and a relational database whose availability sits in your query path.
Governance model
Coarser than both Pattern 1 and Pattern 2. Fine-grained control depends on the engine and on bucket IAM.
Operational profile
A managed service, but one with version upgrades, capacity tiers, backup and restore, and network reachability to manage.
Migration paths
Three routes off an existing metastore, selected by what the client is actually trying to do:
Never greenfield. Every route here starts from an estate that already exists.
HMS to the Lakehouse runtime catalog
The realistic modernization route. Two Preview tools: the built-in gcloud beta metastore services migrations start (Hive tables to a Lakehouse Hive catalog, Iceberg tables to an Iceberg REST catalog, with a dry-run report), and the BigQuery Migration Service for Hive Metastore or Iceberg REST sources (Parquet only, up to 10,000 tables, no Iceberg v3). Both are one-time backfills that do not migrate permissions — plan the consumer cutover and the grants separately.
HMS to BigQuery-managed Iceberg
Appropriate when the Hadoop workload is being retired and BigQuery is becoming the primary engine.
Staying put
A legitimate outcome when the Spark estate is large, stable, and not driving any new requirement.
Roadmap caution
Verify product status before recommending it
As of 2026-09-25 Dataproc Metastore is GA with no end-of-sale notice, but it is being narrowed: multi-region services deprecated (2026-01), old Hive versions deprecated (2026-03), and the newest Spark image unsupported. Google's investment is in the Lakehouse runtime catalog. Re-check before putting it into a client architecture.
What to tell a client already on it
Not urgent to leave, but moving to Spark image 3.0 forces the question. Do not build new capability on it without a deliberate reason.
When to choose Pattern 3
Decision gates
An existing HMS estate is being migrated; the engine mix is Hadoop-ecosystem rather than BigQuery-centric; ecosystem compatibility with older tools matters more than credential vending and REST access.
When not to
Greenfield. For a new lakehouse on GCP, Pattern 1 or Pattern 2 is the answer.