Architecture Patterns Reference
Pattern catalog for anonymization architectures — zone models, pipeline patterns, key management, and AI/LLM patterns. Restructured from Modules 06, 07, and 08 for quick lookup during architecture design and client recommendations.
Abbreviations used throughout: SDP = Sensitive Data Protection; HMAC = keyed-hash message authentication code; FPE = format-preserving encryption; DDM = dynamic data masking (BigQuery policy tags); ARX = open-source structural anonymization tool; MSISDN = subscriber phone number; CDR = Call Detail Record.
For per-zone techniques, see the techniques reference. For GCP tool configurations, see tooling. For operational monitoring of these architectures, see governance and ops.
Decision tree: which pattern do I need?
For AI/LLM workloads, see the AI/LLM pattern selection flowchart below.
Zone model
Three-zone model (default)
| Zone | Content | Access | Retention |
|---|---|---|---|
| Raw | Unmodified source data, full PII | Ingestion pipelines + break-glass only | Days to weeks (reprocessing only) |
| Pseudonymized | Direct IDs replaced (FPE, det. enc., HMAC); QIs at original granularity | Engineers (full), analysts (DDM-masked), ML pipelines | Months to years per classification |
| Anonymized | Structurally anonymized (k-anonymity) and/or DP aggregates | Broad — business users, partners, dashboards, external | Years or indefinite (outside GDPR if truly anonymous) |
Four-zone extension
Split the anonymized zone when data leaves the organization or record-level + aggregate outputs coexist:
| Zone | Content | Access | Threat model |
|---|---|---|---|
| Zone 2: Structurally anonymized | k-anonymous record-level data | Internal broad — analysts, BI | Bounded adversary (known QIs) |
| Zone 3: DP-guaranteed | DP aggregates, DP-trained synthetic data | Widest — partners, regulators, monetization | Unbounded adversary (arbitrary auxiliary data). Non-DP synthetic data (e.g., SDV without DP-SGD) does not belong in Zone 3 — it requires separate memorization validation and does not provide formal guarantees against unbounded adversaries. For DP synthetic data, DPSynth provides marginal-based DP synthesis with an Apache Beam pipeline mode suitable for Dataflow-scale processing. |
When to use 4 zones: Data leaves the organization, record-level + aggregate outputs have different consumers, multiple releases of overlapping data.
When to stay with 3: All anonymized data serves the same consumers, or only aggregate DP outputs are produced.
Critical detail: DP aggregation sources from Zone 1 (pseudonymized), NOT Zone 2 — DP needs full-granularity input data.
Zone model by vertical
| Dimension | Telco | Retail | Financial services | Energy |
|---|---|---|---|---|
| Zone 0 retention | Days (ePrivacy) | Days to weeks | Years (AML mandates) | Days (operational) |
| Zone 1 anchor | MSISDN | Canonical customer ID | Account number | Meter ID |
| Zone 1 technique | HMAC; FPE if format needed | Deterministic encryption | Tokenization (PCI); HMAC | HMAC |
| Zone 2 approach | k-anonymity (k ≥ 50) | k-anonymity + sequence aggregation | k-anonymity on summaries | Temporal aggregation |
| Zone 3 use | Mobility analytics, monetization | Marketing, partner sharing | Risk modeling, regulatory | Grid planning, forecasting |
Pipeline patterns
Streaming: anonymize at ingestion
The default for greenfield platforms and real-time data. Source → Pub/Sub → Dataflow (streaming) with SDP per-record de-identification → BigQuery Storage Write API (Zone 1). Reduces blast radius and simplifies access control — PII never lands in storage at original granularity. The tradeoff is reprocessing rigidity: if transform logic changes, you can't re-derive from stored data. Schema evolution and debugging are harder when the raw zone has short retention.
Batch pipeline
For workloads where minutes-to-hours freshness is acceptable. Source (GCS / BigQuery / JDBC) → Dataflow batch or Cloud Run → SDP → BigQuery load. Simpler architecture, cost-optimized (autoscale-to-zero), but higher latency. Most large-dataset anonymization uses this path.
Structural anonymization (batch-only)
Produces Zone 2 outputs. Zone 1 → ARX (Cloud Run job) → Zone 2 for k-anonymity. Cannot run inline in streaming — structural anonymization requires the full dataset for equivalence class computation.
DP aggregation (batch recommended)
Produces Zone 3 outputs. Zone 1 → Privacy on Beam (batch pipeline with budget tracking) or BigQuery DIFFERENTIAL_PRIVACY (SQL clause) → Zone 3. BigQuery DP technically supports interactive queries, but materialized batch outputs are recommended for budget control — standalone DIFFERENTIAL_PRIVACY queries have no cumulative tracking across invocations. BigQuery Data Clean Rooms can enforce cumulative privacy budgets via analysis rules for controlled sharing scenarios.
Query-time masking (DDM)
For different users needing different views of the same dataset within a zone, when data stays in BigQuery. Uses Knowledge Catalog taxonomy → policy tags → DDM masking rules → IAM (fineGrainedReader / maskedReader). No separate copies, declarative, Terraform-manageable. Only works for BigQuery-routed queries — bypassed if data is exported or read directly by Spark/Trino. Masking is access control, NOT anonymization. Data crossing organizational boundaries needs pipeline transforms.
Late-arriving data strategies
| Strategy | Mechanism | Tradeoff |
|---|---|---|
| Append and regenerate | Late records to pseudo zone; anon zone regenerated next batch | Simple, eventually consistent |
| Micro-batch re-anonymization | Re-anonymize affected partition on arrival | Better consistency, higher cost |
| Cutoff and drop | Records after threshold logged but not processed | Simple but lossy |
| Separate late-arrival partition | Accumulate in staging, merge on schedule | Balanced |
Key management patterns
Transform selection
| Transform | Reversibility | Key architecture | Join compatible | Best for |
|---|---|---|---|---|
| Redaction | No (data destroyed) | No keys | N/A | Fields not needed downstream |
| HMAC | No (one-way) | Shared wrapped key in SDP template | Yes (deterministic) | Default for identifier pseudonymization |
| FPE | Yes | DEKs required; per-subject DEKs if crypto-shredding (requires custom Tink — SDP templates take a single fixed key) | Yes (deterministic) | Legacy format compatibility |
| Deterministic encryption (AES-SIV) | Yes | DEKs required; per-subject DEKs if crypto-shredding (requires custom Tink — SDP templates take a single fixed key) | Yes (deterministic) | Reversibility without format constraints |
Cloud KMS configuration
Cloud KMS
├── Key ring: anonymization-keys (regional)
│ ├── KEK: pseudonymization-master (HSM protection level)
│ │ ├── Version 2 (active for new pseudonymization)
│ │ └── Version 1 (retained for decryption of historical data)
│ └── KEK: partner-export-key (HSM protection)
└── Wrapped DEKs (in pipeline config / Secret Manager)
├── DEK for MSISDN FPE transform
└── DEK for IMSI deterministic encryption
Protection levels
| Level | Key material location | When to use |
|---|---|---|
| SOFTWARE | Google software infrastructure | Dev, non-regulated workloads |
| HSM | FIPS 140-2 Level 3 HSMs | Production default — protects against software infra breaches. One-line Terraform change. |
| EXTERNAL / EXTERNAL_VPC | On-premises or third-party HSM via Cloud EKM | Regulations requiring provider inaccessibility (Swiss FADP + FINMA/SBA) |
Key rotation
KEK rotation — routine, non-disruptive. New version wraps new operations; old versions still unwrap existing DEKs. 90 days (CIS default) or per compliance policy. Never disable/destroy old versions that still wrap active DEKs.
DEK rotation — disruptive, changes pseudonyms:
| Transform | Effect of DEK rotation | Handling |
|---|---|---|
| FPE / det. encryption | New DEK → different ciphertexts; pre/post records can't join | Pin DEK for defined period. When rotating, re-encrypt entire pseudo zone in batch migration. |
| HMAC | New key → different hashes; irreversible, can't re-key existing | "Pseudonym reset" — permanent continuity break. Use as deliberate lifecycle event only. |
When to rotate DEK: Key compromise, personnel change in custodian role, end of data sharing agreement, legal hold release, major migration.
Vault architecture
| Architecture | How it works | Throughput | When to use |
|---|---|---|---|
| Stateless (crypto-derived) | Pseudonym computed from input + key; no lookup table | Very high | Default. FPE and det. enc. via SDP are stateless by design. |
| Stateful (lookup table) | Random token in mapping table | Limited by DB | PCI-DSS tokenization, cross-org sharing, audit-enumerable mappings |
Crypto-shredding
Two approaches for implementing GDPR right-to-erasure via key destruction:
| Dimension | Approach A: Same-table keysets (BigQuery AEAD) | Approach B: Separate DEK table |
|---|---|---|
| Crypto-shredding | Multi-row update (NULL keyset column) | Single-row delete |
| Architecture complexity | Simpler — no separate store | More complex — DEK table + coordinator |
| Re-identification | Query-time AEAD.DECRYPT_* — pure SQL | Compute-then-filter (lookup DEK → compute → filter → decrypt) |
| Residual groupability | No grouping possible (AEAD is randomized) | Deterministic pseudonyms remain groupable |
| Best for | Privacy advantage; simpler architecture | Atomic erasure; deterministic pseudonyms needed for joins |
DEK store options (Approach B):
| Store | Best for |
|---|---|
| BigQuery | Most engagements — same governance perimeter |
| Firestore | Strict physical key-data separation |
| Bigtable | Very high new-subject rates (ad-tech, IoT). Atomic CheckAndMutateRow. |
| Redis (Memorystore) | Coordination layer in front of durable store |
Coordinator for new subjects (race condition prevention):
| Approach | Best for |
|---|---|
| Cloud Run + Firestore transactions | Stable populations, low new-subject rate (telco) |
| Redis SETNX | Higher new-subject rates (e-commerce). Sub-ms. |
| Batch pre-generation | Batch-only ingestion |
AI/LLM architecture patterns
Impact on LLM workloads
| LLM task | Anonymization impact | Approach |
|---|---|---|
| RAG retrieval | High — placeholders disrupt embeddings | Detect-then-process with realistic substitutes |
| Prompt grounding | Moderate–high — placeholder tokens give LLM nothing to reason about | Prompt-level data minimization |
| Agent tool calls | Low–moderate — operates on column values | Standard DDM / pipeline transforms |
| Conversational analytics | Low — LLM generates SQL; DDM masks results | BigQuery Conversational Analytics |
| Summarization | Low–moderate — quality degrades only if context removed | Model Armor or detect-then-process |
| Customer-facing generation | High — placeholder tokens unacceptable in output | Detect-then-process with re-substitution |
Six patterns (defense-in-depth stack)
| # | Pattern | When to use | GCP service | Key limitation |
|---|---|---|---|---|
| 1 | VPC-SC Perimeter | Always — baseline for all LLM workloads with sensitive data | VPC Service Controls | Prevents exfiltration but doesn't control what goes into prompts |
| 2 | Prompt-level data minimization | Always — defense in depth | Cloud Run/GKE application logic | Requires per-task analysis of needed fields |
| 3 | Detect-then-process pipeline (custom) | RAG, semantic search, context-preserving pseudonymization | SDP + Presidio NER → custom Beam → Agent Platform | Custom pipeline code, substitution map management |
| 4 | Model Armor (managed) | Classification, summarization, tasks tolerant of standard SDP masking | Model Armor on Agent Platform | No context-preserving pseudonymization (standard SDP transforms) |
| 5 | Gemini Enterprise (enterprise search) | AI assistant over organizational data | Gemini Enterprise + DLP filtering | Not an anonymization tool — indexes data as-is. Safety net only. |
| 6 | BigQuery Conversational Analytics (DDM) | Natural language queries over structured data | BigQuery Conversational Analytics | Privacy only as strong as IAM config |