Tooling Reference

GCP and OSS tool cards for anonymization pipelines. Each card covers purpose, capabilities, pricing, limitations, and integration patterns. Restructured from Module 05 with key management details from Module 07 for quick lookup during pipeline design.

Abbreviations used throughout: SDP = Sensitive Data Protection; HMAC = keyed-hash message authentication code; FPE = format-preserving encryption; DDM = dynamic data masking (BigQuery policy tags); ARX = open-source structural anonymization tool.

For the techniques each tool implements, see the techniques reference. For pipeline architecture integration, see architecture patterns.

Tool-to-task quick reference

Task	GCP tool	OSS alternative
PII detection (structured)	SDP Inspection	Presidio-Structured
PII detection (text)	SDP Inspection	Presidio (NLP — stronger for unstructured text)
Pseudonymization (FPE, hash, det. enc.)	SDP De-identification	Custom HMAC/FPE libraries
Dynamic data masking	BigQuery DDM	Apache Ranger (Hadoop/Spark)
k-anonymity, l-diversity, t-closeness	—	ARX (no GCP equivalent)
Differential privacy (SQL)	BigQuery DIFFERENTIAL_PRIVACY	—
Differential privacy (pipelines)	Privacy on Beam	PipelineDP (Spark), OpenDP
Key management	Cloud KMS	HashiCorp Vault
Risk assessment / audit	—	ARX, sdcMicro
Synthetic data (no DP)	—	SDV
Synthetic data (with DP)	—	DPSynth, Gretel (SaaS)

---

GCP tools

SDP — Sensitive Data Protection

Inspection

Field	Detail
Purpose	Find PII in structured and unstructured data
Key capabilities	150+ built-in InfoTypes; custom regex/dictionary/stored InfoTypes; likelihood scoring (VERY_UNLIKELY to VERY_LIKELY); three scan modes (inline API, async jobs, Dataflow templates)
Discovery service	Continuous profiling of BigQuery, Cloud SQL, GCS, Vertex AI, S3, Azure Blob. Outputs to Knowledge Catalog (aspects), Security Command Center, BigQuery, Pub/Sub.
Limitations	Regex-based, not NLP — weaker on unstructured text than Presidio
When to use	Default for GCP; structured data scanning; continuous discovery

De-identification

Field	Detail
Purpose	Per-record transformation
Key capabilities	8 transform types: FPE (CryptoReplaceFfxFpeConfig), AES-SIV deterministic encryption (CryptoDeterministicConfig), keyed hash (CryptoHashConfig), character masking, date shift, bucketing, replace with InfoType, redact. All configurable in reusable templates; Terraform-manageable.
Limitations	Per-record only — no structural anonymization (k-anonymity) or DP. Single fixed crypto_key per template — no per-subject keying.
When to use	Default for per-record transforms on GCP; combine inspection + de-identification in single API call

Pricing

Verify before quoting to clients

Pricing and quotas below are a snapshot — verify against the official SDP pricing page and quotas page before including in any client-facing material.

Operation	First 1GB free	1GB–50TB	50TB+
Content inspection	Yes	$3.00/GB	$2.00/GB
Content transformation	Yes	$2.00/GB	$1.00/GB
Storage inspection	Yes	$1.00/GB	$0.75/GB
Storage transformation	Yes	$1.00/GB	$0.75/GB
Discovery profiling	—	$0.03/GB	—

Cost optimization: Use storage jobs ($1/GB) over content API ($3/GB) when data is already in BigQuery/GCS. RedactConfig + ReplaceWithInfoType are free when combined with inspection.

Quotas

Quota structure is multi-layered — verify against the official SDP quotas page for current limits.

Limit	Value
Requests/min per region (global endpoint)	600
Requests/min per region (regional endpoint)	100
Max payload per request	0.5 MB / 50k values

---

BigQuery DDM (Dynamic Data Masking)

Field	Detail
Purpose	Dynamic masking — different users see different values at query time
Setup	Knowledge Catalog taxonomy → policy tags → column bindings → masking rules → IAM roles
Masking rules	SHA-256, Nullify, Default Value, Date Year Mask, Last 4 Characters
Access tiers	fineGrainedReader (see real values), maskedReader (see masked values), denied (no access)
Strengths	No separate copies; declarative; Terraform-manageable; works on Iceberg tables
Limitations	Only for BigQuery-routed queries — bypassed on data export or Spark/Trino direct reads
Critical caveat	Access control, NOT anonymization. Never use as sole measure for data crossing organizational boundaries.
When to use	Within-zone access control where different teams need different views

---

BigQuery Differential Privacy

Field	Detail
Purpose	DP aggregate queries with calibrated noise
Syntax	SELECT WITH DIFFERENTIAL_PRIVACY OPTIONS(epsilon=, delta=, max_groups_contributed=, privacy_unit_column=)
Capabilities	Automatic noise injection on aggregate functions; automatic small-group suppression
Limitations	Standalone DIFFERENTIAL_PRIVACY queries have no cumulative budget tracking — each query independent. BigQuery Data Clean Rooms can enforce cumulative budgets via analysis rules. Standalone use is suited for materialized outputs, not interactive exploration.
When to use	Simple materialized DP outputs (one query → result table consumed freely)

---

Cloud KMS

Field	Detail
Purpose	Key lifecycle management for anonymization transforms
Architecture	Two-tier hierarchy: KEK (in KMS) wraps DEK (in pipeline config / Secret Manager)
Protection levels	SOFTWARE (dev), HSM (production default — FIPS 140-2 Level 3), EXTERNAL/EXTERNAL_VPC (regulated — Swiss FADP/FINMA)
Rotation	Automatic rotation for ENCRYPT_DECRYPT keys only. MAC keys require manual version creation (Cloud KMS API constraint). For pseudonymization, HMAC rotation should be event-driven regardless — a new key version produces different pseudonyms, breaking join continuity.
Key features	Envelope encryption pattern; audit logging; IAM separation of key admin vs. key use
When to use	Always — production key management for pseudonymization

Key rotation schedules

Key type	Typical rotation	Automation	Impact
KEK	90 days (CIS default)	Cloud KMS automatic	Non-disruptive; old versions still unwrap existing DEKs
DEK (pseudonymization)	6–12 months	Coordinated migration	Disruptive — breaks joins until re-pseudonymized
HMAC key	Rarely	Manual	Permanent continuity break; deliberate lifecycle event only

---

Dataflow (Apache Beam)

Field	Detail
Purpose	Pipeline execution engine for anonymization at scale
Key capabilities	Streaming + batch; integrates SDP via direct API calls or pre-built templates; ARX / Privacy on Beam as batch stages
When to use	Default execution engine for GCP anonymization pipelines

---

OSS tools

ARX Data Anonymization Tool

Field	Detail
Purpose	Structural anonymization — the primary production tool for k-anonymity, l-diversity, t-closeness
Key capabilities	Privacy models (k-anon, l-diversity variants, t-closeness, δ-presence, DP); generalization hierarchy exploration; utility metrics; risk analysis (prosecutor/journalist/marketer); suppression control
Language	Java (library + GUI)
Integration	Containerize for Cloud Run jobs or Dataflow batch
Limitations	Batch only; no streaming
When to use	Structural anonymization (Zone 1 → Zone 2); risk assessment for audit evidence
When NOT to use	Per-record transforms (use SDP)
Key fact	No GCP equivalent exists — ARX fills the k-anonymity gap

Privacy on Beam

Field	Detail
Purpose	Pipeline-grade DP with composition budget tracking
Key capabilities	DP-aware Beam aggregations (Count, Sum, Mean, Quantiles); automatic composition tracking; partition selection (suppress small groups)
Integration	Runs on Dataflow, Flink, Spark via Beam runners
When to use	DP pipelines needing cumulative budget tracking, multiple aggregations, complex multi-stage composition

PipelineDP

Field	Detail
Purpose	DP primitives with native Spark integration
Language	Python / JVM
When to use	Clients with existing Spark infrastructure who don't want to adopt Beam

Microsoft Presidio

Field	Detail
Purpose	NLP-powered PII detection
Key capabilities	Regex + NLP (spaCy NER) + context rules; custom entity recognizers; Presidio-Structured for PySpark DataFrames
Limitations	Transform set narrower than SDP (no FPE, no det. enc., no bucketing) — strongest as detection engine
When to use	On-prem / multi-cloud; heavy unstructured text; PySpark pipelines; cost optimization at high volume
When NOT to use	As sole transform engine — pair with SDP or custom transforms

OpenDP

Field	Detail
Purpose	Composable DP with formal verification (MIT / Harvard)
Language	Rust core / Python bindings
Key capabilities	Machine-checkable proofs per operation; fine-grained composition control
When to use	When auditable, formally verified DP guarantees are required

SDV (Synthetic Data Vault)

Field	Detail
Purpose	Synthetic data generation (tabular, relational, time-series)
Key capabilities	Multiple synthesizers; evaluation metrics for disclosure risk and attribute inference. Full membership inference testing requires a separate train/holdout attack model (see Module 12).
Limitations	No native DP mode — requires separate disclosure-risk evaluation; memorization risk on small datasets
When to use	ML training data, dev/test environments, record-level external sharing
When NOT to use	When formal DP guarantees needed without separate validation — use DPSynth or Gretel DP mode

DPSynth

Field	Detail
Purpose	Differentially private synthetic data generation (google/dpsynth)
Language	Python (NumPy/JAX in-memory; Apache Beam pipeline mode)
Key capabilities	DP-guaranteed synthesis for tabular, relational, nested, sequence, and text data. Mechanisms include AIM (adaptive iterative marginals), MST (maximum spanning tree), and SWIFT (scalable high-dimensional). In-memory mode for datasets up to ~100M rows; pipeline mode on Apache Beam for distributed processing on Dataflow. Post-processing mode for pre-computed noisy marginals.
Limitations	Under active development; not an officially supported Google product. Tabular mode is most mature; other modalities are experimental.
When to use	Zone 3 synthetic data that requires formal DP guarantees — partner sharing, ML training data, external analytics. The Beam pipeline mode maps naturally onto Dataflow for large-scale GCP deployments.
When NOT to use	When DP is not required and SDV's richer evaluation metrics suffice; when a fully managed SaaS solution is preferred (Gretel)

Google DP Libraries

Field	Detail
Purpose	Low-level DP primitives
Languages	C++, Go, Java
Key capabilities	Count, Sum, Mean, Quantiles with calibrated noise
When to use	Building custom DP applications; typically accessed via Privacy on Beam

sdcMicro

Field	Detail
Purpose	Statistical disclosure control metrics
Language	R
Key capabilities	Risk reports recognized by European DPAs
When to use	Formal risk quantification for audit / regulatory evidence
When NOT to use	Pipeline integration (not designed for it)

Amnesia

Field	Detail
Purpose	k-anonymity / t-closeness with web GUI
When to use	Workshops with non-technical users; quick demonstrations
When NOT to use	Production pipelines

---

GCP vs. OSS selection guide

Factor	Managed (GCP)	Self-hosted (OSS)
Operational overhead	Low	Medium–high
Cost at scale	Per-GB-processed; expensive at CDR volumes	Compute only; cheaper for very high throughput
Technique coverage	Per-record, DP, masking. No k-anonymity.	Full k-anon/l-div/t-close (ARX). DP (OpenDP, PipelineDP).
Integration	Native GCP (BigQuery, Dataflow, IAM, KMS)	Custom; containerization
Compliance posture	SOC 2, ISO 27001, HIPAA inherited	Self-assessed
Data residency	Regional endpoints	Full control
Vendor lock-in	Moderate — SDP APIs are GCP-specific	None

Typical production stack: SDP (per-record transforms) + BigQuery DDM (access control) + Cloud KMS (key management) + ARX (structural anonymization) + Privacy on Beam or BigQuery DP (differential privacy). GCP handles per-record and access control; OSS fills the structural anonymization and advanced DP gaps.