5. Tooling Landscape
This module covers the tooling that implements the techniques (Module 03) and privacy models (Module 04) in production. Rather than cataloging products, it follows the sequence of problems you solve when building an anonymization pipeline: find the PII, transform it, control who sees what, manage the keys, apply formal privacy models, produce DP outputs, and run it all at scale.
Each section presents the problem first, then the GCP-managed and open-source options side by side. GCP is the primary lens — the platform most Aliz engagements target — but every section covers the OSS alternative for on-premises, multi-cloud, or capability gaps. Module 06 shows how these tools combine into end-to-end architecture patterns.
Finding PII — detection and classification
Before you can anonymize data, you need to know where the sensitive fields are. In a greenfield project with a well-documented schema, this is obvious — the msisdn column contains phone numbers. In practice, most engagements involve inherited data lakes, undocumented tables, free-text fields with embedded PII, and schemas where column names like field_7 or cust_ref reveal nothing about content. Detection tooling automates the discovery: scan data, classify what's sensitive, and feed the results into the transformation layer.
GCP Sensitive Data Protection — inspection
GCP Sensitive Data Protection (formerly Cloud DLP) provides the inspection engine. It scans content for sensitive data using InfoTypes — pattern definitions that describe what to look for. GCP ships 150+ built-in InfoTypes:
| Category | Example InfoTypes | What they detect |
|---|---|---|
| Personal identifiers | PERSON_NAME, EMAIL_ADDRESS, PHONE_NUMBER | Names, emails, phone numbers in various formats |
| National/government IDs | US_SOCIAL_SECURITY_NUMBER, SPAIN_DNI_NUMBER, HUNGARY_TAX_IDENTIFICATION_NUMBER | Country-specific ID formats |
| Financial | CREDIT_CARD_NUMBER, IBAN_CODE | Payment card and bank account numbers |
| Network/device | IP_ADDRESS, MAC_ADDRESS | IPv4, IPv6 addresses, MAC addresses |
Built-in InfoTypes cover standard PII well, but telco data contains identifiers with no built-in detector — MSISDNs (international phone numbers in E.164 format), IMSIs (15-digit subscriber identity numbers), and IMEIs (device identifiers). For these, you define custom InfoTypes:
- Regex-based — pattern matching for structured identifiers. An MSISDN custom InfoType might match
\+?[1-9]\d{6,14}with context rules to distinguish phone numbers from other numeric sequences. - Dictionary-based — exact or substring matching against a list. Useful for detecting known subscriber names, internal system identifiers, or company-specific codes in free-text fields.
- Stored InfoTypes — dictionaries in Cloud Storage for large or frequently updated lists, referenced by the inspection template rather than inlined.
Custom InfoTypes live in inspection templates — JSON/Terraform-manageable configurations that specify which InfoTypes to scan for, their likelihood thresholds, and context rules. Templates are reusable across pipelines and environments.
Likelihood scoring. Each detection includes a score from VERY_UNLIKELY to VERY_LIKELY. Setting a minimum threshold controls the sensitivity-specificity trade-off: lower thresholds catch more PII but produce more false positives. For structured data (database columns), use LIKELY or higher. For unstructured text (support tickets, notes), start at POSSIBLE and tune based on false positive rates.
Sensitive Data Protection can scan in three modes: inline via the content.deidentify API (synchronous, real-time), as async inspection jobs over Cloud Storage / BigQuery / Datastore (discovery and classification), and through pre-built Dataflow templates (pipeline-integrated).
SDP discovery service and Knowledge Catalog integration
Beyond one-off inspection jobs, Sensitive Data Protection offers a discovery service (formerly "Automatic DLP") that continuously profiles data across an organization, folder, or project. As long as a discovery scan configuration is active, SDP automatically profiles new and modified resources on a configurable schedule (daily, weekly, monthly — default 30 days).
What gets profiled:
| Resource type | Profile granularity |
|---|---|
| BigQuery (standard tables, snapshots, Lakehouse for Apache Iceberg tables) | Project → table → column level |
| Cloud SQL databases | Project → table → column level |
| Cloud Storage buckets | Bucket level with file clustering |
| Vertex AI training data | Dataset level |
| Amazon S3 buckets, Azure Blob Storage | Bucket/container level |
Each profile captures detected InfoTypes, calculated sensitivity level (HIGH, MODERATE, LOW) and data risk level, plus metrics about data characteristics. This is the automated version of the "PII discovery pass" — instead of running it once at the start of an engagement, it runs continuously and catches new tables or schema changes.
The key connection: Knowledge Catalog. Discovery results flow into Knowledge Catalog (GCP's data catalog and governance product, formerly Dataplex Universal Catalog) as aspects — structured metadata attached to catalog entries. When you enable the "Send to Knowledge Catalog as aspects" action on a discovery scan, SDP creates a Sensitive Data Protection profile aspect on the catalog entry for each profiled resource. The aspect contains the sensitivity level, risk level, detected InfoTypes (including per-column), and a link to the full profile.
This makes sensitivity metadata searchable and governable from the catalog:
aspect:sensitive-data-protection-profile— find all resources with SDP dataaspect:sensitive-data-protection-profile.sensitivity=HIGH— find high-sensitivity resources- Governance teams can browse the catalog to see which tables contain what types of PII without running separate scans
Discovery also supports actions beyond Knowledge Catalog — auto-tagging resources with sensitivity labels, publishing to Security Command Center, saving profiles to BigQuery, and event-driven workflows via Pub/Sub. These are operational controls covered in Module 10.
Presidio — NLP-powered PII detection
Microsoft Presidio (Apache 2.0) is the primary open-source alternative. Its detection engine combines regex, rule-based logic, and NLP-based named entity recognition (spaCy NER), giving it an edge over Sensitive Data Protection for unstructured text — customer support tickets, free-text notes, legal documents.
| Capability | Sensitive Data Protection | Presidio |
|---|---|---|
| Text PII detection | Good — regex + context rules | Strong — regex + NLP (spaCy NER) + context |
| Structured data scanning | BigQuery, GCS, Datastore | DataFrames, CSVs via Presidio-Structured |
| Custom entity recognizers | Regex, dictionary, stored | Regex, deny lists, NER model fine-tuning |
| Deployment | Managed API | Self-hosted (Docker, Kubernetes, embedded Python) |
| Cost model | Per-content-unit pricing | Compute cost only |
When to reach for Presidio over or alongside Sensitive Data Protection:
- On-premises or multi-cloud environments where GCP APIs are unavailable
- Heavy text processing where NLP-based entity recognition outperforms regex
- PySpark pipelines — Presidio-Structured integrates directly with PySpark DataFrames
- Cost optimization — for high-volume scanning where Sensitive Data Protection API costs are prohibitive
Start every engagement with a PII discovery pass — run Sensitive Data Protection inspection jobs across the data lake to build a PII map. For ongoing operations, set up the SDP discovery service so new tables and schema changes are profiled automatically, with results flowing into Knowledge Catalog as aspects. This gives governance teams a searchable, always-current view of where sensitive data lives. For clients with significant unstructured text (support logs, notes), add Presidio for its NLP detection. Module 10 covers how to wire discovery results into operational workflows — auto-tagging, conditional IAM, and sensitivity-driven access control.
Transforming records — per-record de-identification
Once PII is located, the next step is transforming it. This is per-record work: each record passes through the pipeline and comes out with its sensitive fields replaced, encrypted, hashed, or generalized. The techniques from Module 03 — format-preserving encryption (FPE), deterministic encryption, keyed hashing, masking, bucketing, redaction — are implemented here.
Sensitive Data Protection transforms
Sensitive Data Protection combines detection and transformation in a single API call. Its de-identification transforms map directly to Module 03 techniques:
| Transform | Technique (Module 03) | Reversible | Format-preserving | Use case |
|---|---|---|---|---|
CryptoReplaceFfxFpeConfig | Format-preserving encryption | Yes (with key) | Yes | MSISDNs, IMSIs in systems requiring format compatibility |
CryptoDeterministicConfig | Deterministic encryption (AES-SIV) | Yes (with key) | No | Consistent pseudonyms across datasets; enables joins |
CryptoHashConfig | Keyed hashing (SHA-256) | No | No | One-way pseudonymization where re-identification is never needed |
CharacterMaskingConfig | Masking | No | Yes | Partial masking for display — show last 4 digits |
DateShiftConfig | Date shifting | No (without key) | Yes | Shift dates by a random number of days within a range |
BucketingConfig | Generalization (bucketing) | No | No | Age → age band, duration → duration bucket |
ReplaceWithInfoTypeConfig | Suppression/redaction | No | No | Replace detected PII with its type label — "John" → [PERSON_NAME] |
RedactConfig | Suppression | No | No | Remove detected PII entirely |
Choosing the right crypto transform
The three cryptographic transforms — FPE, deterministic encryption, keyed hashing — cover different requirements. The choice comes down to three questions:
- Does the downstream system require the output to look like the input? (Legacy billing systems that validate phone number format, schema constraints on field length) →
CryptoReplaceFfxFpeConfig(FPE) - Do you need consistent pseudonyms across tables for joins, with the option to re-identify later? →
CryptoDeterministicConfig(AES-SIV) - Is this strictly one-way — no re-identification ever? →
CryptoHashConfig
All three use Cloud KMS keys in production (unwrapped keys are available for testing). A single de-identification template can apply different transforms to different InfoTypes — FPE for MSISDN, bucketing for age, redaction for free-text names — expressing the entire per-record transformation policy in one configuration. Templates are Terraform-manageable.
Per-record transformation is the first pipeline stage after detection. On GCP, Sensitive Data Protection handles both detection and transformation in one API call — this is the default choice. Presidio can also detect and transform, but its transform set is narrower (no FPE, no deterministic encryption, no bucketing) — it is strongest as a detection engine paired with custom transformation logic. Neither SDP nor Presidio implements structural anonymization (k-anonymity) or differential privacy — those are dataset-level operations covered in the sections below.
Controlling access — dynamic masking
Transforming data in a pipeline produces a new dataset. But many use cases require the same dataset to appear differently to different users — data engineers see raw values, business analysts see hashed pseudonyms, external dashboards see nullified fields. Dynamic data masking (DDM) solves this at query time without creating separate physical copies.
BigQuery column-level security
BigQuery implements DDM through a policy tag system with three components:
- Taxonomy — a named classification scheme (e.g.,
PII_Classification), scoped to a project and region. - Policy tags — categories within the taxonomy (e.g.,
High,Medium,Low), optionally hierarchical. - Column bindings — policy tags attached to specific columns. A column tagged
Highinherits the masking and access rules defined on that tag.
Each policy tag can have a masking rule that determines what masked readers see:
| Masking rule | Output for masked reader | Typical use |
|---|---|---|
| SHA-256 hash | Deterministic hash of the value | Joins on pseudonymized columns without seeing raw values |
| Nullify | NULL | Complete suppression — field exists but value is hidden |
| Default value | 0, "", or type-appropriate default | Preserve schema validity while hiding content |
| Date year mask | Only the year; month/day set to Jan 1 | Temporal analysis at annual granularity |
| Last four characters | ****1234 | Partial visibility for verification |
Access is controlled by two IAM roles:
roles/datacatalog.categoryFineGrainedReader— sees raw, unmasked data.roles/bigquerydatapolicy.maskedReader— sees the masked output defined by the masking rule.- Neither role — query fails with a permission error if the tagged column is accessed.
The result is a three-tier access model on a single table: data engineers with Fine-Grained Reader see raw values, business analysts with Masked Reader see hashes or nulls, and principals with neither role are denied access entirely. One table, one schema, multiple views — enforced by IAM, not pipeline logic. Any tool that queries BigQuery (Looker, connected sheets, dbt, Dataform) automatically respects the masking without application changes. This extends to Iceberg tables managed through Lakehouse for Apache Iceberg — the same policy tags and masking rules apply, enforced via the BigQuery Storage API (Module 06's query-time masking section covers this pattern and its limitations). Taxonomies and bindings are fully Terraform-manageable (google_data_catalog_taxonomy, google_data_catalog_policy_tag, google_bigquery_datapolicy_data_policy).
Connection to SDP discovery. The SDP discovery service (previous section) identifies which columns contain what types of sensitive data. That classification feeds directly into the policy tag design: columns SDP flags as containing direct identifiers get tagged High, quasi-identifiers get Medium, and so on. The SDP auto-tagging action can apply Resource Manager sensitivity tags automatically; applying BigQuery policy tags based on SDP findings currently requires a manual or scripted step, but the SDP profile data (available in BigQuery or Knowledge Catalog) provides the input for automating that mapping.
Apache Ranger — the on-prem equivalent
For Hadoop/Spark ecosystems, Apache Ranger provides the equivalent capability: centralized, tag-based, role-aware column masking for Hive, Spark SQL, Presto, and other query engines. Masking types include redact, hash, nullify, show first/last N characters, and custom expressions. If the client runs a Hive/HDFS CDR warehouse, Ranger provides the DDM layer without a full migration to BigQuery.
Dynamic masking is not anonymization — the raw values still exist in storage. It is an access control mechanism that governs what each user sees at query time. Use it in the pseudonymized and analytics zones where different teams need different access levels to the same tables. When data leaves the controlled environment (partner exports, external sharing), static transformation or differential privacy is required — DDM alone is insufficient.
Managing keys — cryptographic infrastructure
The crypto transforms from the previous sections — FPE, deterministic encryption, HMAC keyed hashing — all depend on cryptographic keys. Key management is not an anonymization technique, but it is a critical infrastructure dependency: a lost key means pseudonymized data becomes unlinkable (potentially good for right-to-erasure, catastrophic for operational continuity), and a leaked key means every pseudonym derived from it is compromised.
Cloud KMS — key hierarchy and rotation
Cloud KMS provides the key lifecycle for GCP anonymization architectures (Module 07 covers the full key management design, including vault architecture, rotation strategy, and crypto-shredding). A production setup typically uses a two-tier hierarchy:
| Level | Key type | Stored in | Purpose |
|---|---|---|---|
| Master key (KEK) | Symmetric AES-256 | Cloud KMS (HSM-backed) | Wraps/unwraps data-level keys |
| Data key (DEK) | Symmetric AES-256 or HMAC | Encrypted in application storage | Used directly in transforms |
Key rotation has specific implications for anonymization:
- For HMAC and keyed hashing: a new key produces different pseudonyms for the same input. This supports right-to-erasure by key destruction — destroy the key, and all pseudonyms become unlinkable. But it also means datasets produced before and after rotation cannot be joined. Design rotation around the data lifecycle: annually or on legal hold release, not daily.
- For FPE and deterministic encryption: rotation re-encrypts with the new key version. Sensitive Data Protection can use a specific key version or the primary (latest). For join continuity across datasets, all must use the same key version.
Envelope encryption pattern
The envelope encryption pattern separates the key that touches data (DEK) from the key that protects it (KEK):
- Generate a DEK locally (e.g., a 256-bit AES key for FPE transforms).
- Encrypt the DEK with the KEK in Cloud KMS.
- Store the encrypted DEK alongside the pipeline configuration.
- At runtime, decrypt the DEK via KMS, use it for transforms, discard the plaintext DEK from memory.
The plaintext DEK never persists outside the pipeline's runtime, and KMS audit logs track every decryption. For Sensitive Data Protection, this is built in — you provide a wrapped key in the de-identification template, and the service handles the unwrap-transform-discard cycle internally. For on-premises or multi-cloud deployments, HashiCorp Vault provides equivalent key lifecycle management.
Structural anonymization — dataset-level privacy models
The previous sections cover per-record operations: find PII, transform it, mask it, manage the keys. But per-record transforms alone do not achieve anonymization. A dataset where every MSISDN is hashed but cell towers, timestamps, and call durations are untouched is still vulnerable to re-identification via behavioral fingerprinting (Module 12 covers the attack patterns).
Structural anonymization operates on the dataset as a whole — grouping records, generalizing quasi-identifiers, suppressing outliers — to satisfy formal privacy models like k-anonymity, l-diversity, and t-closeness (Module 04). This is fundamentally a batch operation: you need the entire dataset (or a representative partition) to compute equivalence classes and assess group sizes.
ARX — the primary tool
ARX is the most complete open-source tool for structural anonymization (Apache 2.0, Java library + GUI):
- Privacy models: k-anonymity, l-diversity (distinct, entropy, recursive), t-closeness, δ-presence, and DP — supports combining models (e.g., k=50 with entropy l-diversity l=3).
- Generalization hierarchies: Define how each quasi-identifier generalizes at successive levels (cell tower → cluster → district → region →
*). ARX explores the space of possible combinations to find the one achieving the target model while minimizing information loss. - Utility metrics: Quantifies information loss across configurations. Compare "cell towers to district + timestamps to hourly" vs "cell towers to cluster + timestamps to 15-minute" on a formal utility scale.
- Risk analysis: Measures re-identification risk before and after anonymization — prosecutor risk, journalist risk, marketer risk. Useful for GDPR accountability documentation.
- Suppression control: Configurable maximum suppression rate (percentage of records removable to achieve the target model). Typically 1–5%.
There is no GCP-managed equivalent for k-anonymity. Sensitive Data Protection handles per-record transforms; BigQuery handles dynamic masking and DP aggregation; but no GCP service implements structural anonymization. ARX fills this gap.
Integration: ARX is a batch tool. In a GCP architecture, it fits as a scheduled batch job — read from BigQuery (pseudonymized zone), apply structural anonymization, write results to the anonymized zone. Containerize the Java library for Cloud Run jobs or Dataflow batch pipelines. Niche alternatives exist — sdcMicro (R) for formal risk quantification reports that European DPAs recognize, Amnesia (web GUI) for workshops with non-technical users — but neither is designed for pipeline integration.
Structural anonymization is the step that transforms pseudonymized data into formally anonymized data — where, if the anonymization is robust against re-identification by any means reasonably likely to be used (GDPR Recital 26), the output may qualify as anonymous data outside GDPR scope. It's also the step with no managed GCP service. Plan for ARX (or sdcMicro) as a batch component in the architecture, typically running on a schedule between the pseudonymized and anonymized data zones.
Differentially private outputs — aggregate guarantees
Structural anonymization (k-anonymity family) protects record-level datasets. But when the output is aggregate statistics — counts, sums, averages, histograms, mobility reports — differential privacy (Module 04) provides a stronger, more composable guarantee. The tooling for DP aggregate outputs ranges from a SQL clause in BigQuery to full pipeline frameworks with budget tracking.
BigQuery DIFFERENTIAL_PRIVACY — the simplest path
BigQuery's DIFFERENTIAL_PRIVACY clause adds calibrated noise to aggregate queries with no external libraries or pipeline changes:
SELECT WITH DIFFERENTIAL_PRIVACY
OPTIONS (
epsilon = 1.0,
delta = 1e-7,
max_groups_contributed = 3,
privacy_unit_column = subscriber_id
)
district,
COUNT(*) AS call_count,
SUM(duration_seconds) AS total_duration
FROM pseudonymized_cdrs
WHERE date = '2025-01-15'
GROUP BY district
Key parameters:
privacy_unit_column— the column identifying individuals. For telco CDRs, typicallysubscriber_idormsisdn_hash.epsilon— privacy budget for this query. Start at 1.0 for moderate privacy.delta— failure probability.1e-7is a safe default for million-record datasets.max_groups_contributed— bounds how many output groups a single privacy unit can influence. This enforces bounded sensitivity — the technical requirement for noise calibration.
Small groups may be automatically suppressed (replaced with NULL) if they contain too few privacy units.
The critical limitation: BigQuery does not track cumulative budget across queries. Each query consumes epsilon independently. An analyst running 20 exploratory queries at ε=1.0 each has consumed ε=20 cumulatively — at that level, the DP guarantee is meaningless. This makes BigQuery DP suited for materialized outputs (scheduled reports, partner export tables, dashboard source tables), not interactive exploration. Materialize the DP results once, then let consumers query the materialized table freely — that's post-processing immunity.
Privacy on Beam — pipeline-grade DP with budget tracking
When DP outputs need cumulative budget tracking, complex multi-aggregation pipelines, or Dataflow/Spark execution, Privacy on Beam is the step up from BigQuery DP. It wraps Beam's aggregation primitives (Count, Sum, Mean, Quantiles) with DP-aware versions that automatically add calibrated noise:
Read CDRs
→ Key by privacy_unit (subscriber_id)
→ Group by analysis dimensions (district, hour_bucket)
→ DP Count / DP Sum / DP Mean
(noise calibrated by PrivacySpec: ε=1.0, δ=1e-7)
→ Write aggregated results
Advantages over BigQuery DP:
- Budget tracking. Privacy on Beam tracks cumulative epsilon across all aggregations in a pipeline. Five aggregations over the same data are automatically composed.
- Partition selection. Automatically suppresses output partitions with too few privacy units — explicit and configurable.
- Complex pipelines. Supports multi-stage DP pipelines with end-to-end composition tracking.
Privacy on Beam runs on Dataflow (GCP), Apache Flink, or Spark via Beam's runner abstraction. The DP guarantees are identical regardless of runner.
Alternatives: PipelineDP (Python/JVM) provides the same DP primitives with native Spark integration — for clients with existing Spark infrastructure who don't want to adopt Beam. OpenDP (MIT, Harvard Privacy Tools project) offers composable DP primitives with formal verification — each operation carries a machine-checkable proof. Reach for it when the client requires auditable, formally verified DP guarantees or needs fine-grained control over composition.
Privacy Loss Distribution tool — budget planning
The PLD tool computes tight composition bounds by tracking the full privacy-loss distribution across queries. When a pipeline runs multiple DP queries over the same data, naive budget accounting (sum of all epsilons) overstates the loss — PLD computes a tighter bound, which can often unlock significantly more queries for the same total epsilon (typically 20–40%, depending on the mechanisms and parameters used).
Use PLD during architecture design to answer: "Given our target total epsilon and the number of reports we need, what per-query epsilon should we use?" This is a design-time planning tool, not a runtime component.
Match the DP tool to the use case. BigQuery DP for simple materialized outputs (one query, noise added, result table consumed freely). Privacy on Beam for pipelines that compute multiple aggregations with budget tracking. OpenDP when formal verification is a requirement. Use PLD during architecture design to plan budget allocation — the answer to "how many reports can we safely generate?" comes from PLD, not from guessing epsilon values.
Running at scale — pipeline integration
The tools above handle individual capabilities — detection, transformation, masking, privacy models. In production, they combine inside data pipelines that process records at CDR scale (millions to billions of records). Apache Beam on Dataflow is the standard GCP execution engine for this integration.
Anatomy of an anonymization pipeline
The canonical pattern layers the tools covered in this module:
Source (Pub/Sub, GCS, JDBC)
→ Read / Parse
→ Per-record de-identification (Sensitive Data Protection)
├── CryptoDeterministicConfig for identifiers (MSISDN, IMSI)
├── CryptoHashConfig for device IDs (IMEI)
└── BucketingConfig for quasi-identifiers (age → band, timestamp → hour)
→ Custom transforms
├── Cell-ID generalization (lookup table → cluster assignment)
└── Business logic validation
→ Write to BigQuery (pseudonymized table with policy tags)
└── DDM rules applied at query time (previous section)
[Batch — scheduled separately]
→ Structural anonymization (ARX: k-anonymity on pseudonymized table)
→ DP aggregation (Privacy on Beam: aggregate outputs for external sharing)
→ Write to BigQuery (anonymized zone, DP output tables)
The pipeline splits into two stages: streaming or batch per-record processing (pseudonymization) and batch dataset-level operations (structural anonymization, DP aggregation). This split is fundamental — per-record transforms can run in real time, but k-anonymity and DP require the full dataset.
Integrating Sensitive Data Protection in Beam
Two approaches:
Direct API calls within Beam transforms. Each element passes through a DoFn that calls content.deidentify. Full control over per-record configuration, but adds API call latency. Batch the calls — Sensitive Data Protection supports batch de-identification. Record-by-record API calls at CDR scale will hit rate limits.
Google-provided Dataflow templates. Pre-built templates that handle batching, error handling, and retry. Use when the standard inspect-and-transform pattern fits; switch to custom Beam code for non-standard transforms (cell-ID generalization, cross-record operations).
Streaming vs batch
| Dimension | Streaming | Batch |
|---|---|---|
| Source | Pub/Sub topic | Cloud Storage, BigQuery, JDBC |
| Latency target | Seconds to minutes | Minutes to hours |
| Exactly-once | Depends on write method (Storage Write API in exactly-once mode) | Requires idempotent write design (WRITE_TRUNCATE, dedup keys) |
| SDP throughput | Batch API calls per window; tune maxBatchSize | Batch API calls per bundle; scales with workers |
| Error handling | Dead-letter to Pub/Sub topic for retry | Failed records to error table |
| Cost model | Always-on workers (or Dataflow Prime flex) | Autoscaled, pay per batch |
Streaming is the right pattern when data must be queryable shortly after generation — real-time CDR analytics, fraud detection on pseudonymized streams, live monitoring. The pipeline applies per-record pseudonymization in the streaming path and leaves structural anonymization and DP to scheduled batch jobs on the materialized data.
Batch fits historical data migration, periodic partner exports, and all structural anonymization (k-anonymity is inherently a batch operation).
Don't try to run k-anonymity or DP in a streaming pipeline — they need the full dataset. The standard pattern is: streaming pseudonymization (per-record transforms in Dataflow) → materialized pseudonymized table in BigQuery → batch structural anonymization and DP aggregation on a schedule. Module 06 covers the full architecture in detail.
Choosing tools — decision framework
The choice between GCP-managed and self-hosted tools is not a platform decision — it is a per-capability decision. Most architectures combine both.
Managed vs self-hosted
| Factor | Managed (GCP) | Self-hosted (OSS) |
|---|---|---|
| Operational overhead | Low — no infra to manage | Medium to high — deployment, scaling, patching |
| Cost at scale | Per-API-call pricing; expensive at CDR volumes | Compute cost only; cheaper for very high throughput |
| Technique coverage | Per-record transforms, DP, masking. No k-anonymity | Full k-anonymity/l-diversity/t-closeness (ARX). DP (OpenDP, PipelineDP) |
| Integration | Native GCP (BigQuery, Dataflow, IAM, KMS) | Custom integration; containerization for GCP |
| Compliance posture | SOC 2, ISO 27001, HIPAA BAA inherited | Self-assessed; audit burden on operator |
| Data residency | Regional endpoints | Full control — deploy anywhere |
| Vendor lock-in | Moderate — SDP APIs are GCP-specific | None — portable across clouds and on-prem |
Tool-to-task quick reference
| Task | GCP tool | OSS alternative |
|---|---|---|
| PII detection (structured) | Sensitive Data Protection | Presidio-Structured |
| PII detection (text) | Sensitive Data Protection | Presidio (NLP — stronger for unstructured text) |
| Pseudonymization (FPE, hashing, det. enc.) | Sensitive Data Protection | Custom HMAC/FPE libraries |
| Dynamic data masking | BigQuery column-level security | Apache Ranger (Hadoop/Spark) |
| k-anonymity, l-diversity, t-closeness | — | ARX |
| Differential privacy (SQL) | BigQuery DIFFERENTIAL_PRIVACY | — |
| Differential privacy (pipelines) | Privacy on Beam | PipelineDP (Spark), OpenDP |
| Key management | Cloud KMS | HashiCorp Vault |
| Risk assessment / re-identification risk | — | ARX, sdcMicro |
| Synthetic data generation | — | SDV (no DP), DPSynth (with DP) |
Use GCP managed services for the capabilities they cover well — PII detection, per-record transforms, dynamic masking, key management, DP aggregation. Fill the gaps with OSS where GCP has no equivalent — primarily k-anonymity (ARX) and NLP-heavy text detection (Presidio). The architecture question isn't "GCP or OSS" — it's "which tool for which layer."
Next: Module 06 covers architecture patterns — how these tools combine into end-to-end anonymization pipelines across data zones, with reference architectures for streaming and batch processing.
Created by Claude Opus with ❤️ for the Aliz Data Engineering team.