Scalable, safe execution environments for AI agents
An earlier version of this article described a prototype that gave each chat its own disposable sandbox and deleted it after an idle period. The second iteration keeps that isolation but lets work survive a break: each engineer works in a resumable workspace with its own sandbox, and an idle workspace is saved, including its running processes, and restored when the engineer returns. This version adds what we learned about saving and restoring agent workspaces safely, and it lists what the prototype does not cover yet.
A chatbot can safely answer questions without ever touching a shell. A useful engineering agent cannot. It needs to clone repositories, inspect files, run tests, query live systems, and sometimes prepare changes. That ability creates the value, but it also turns the agent into an untrusted workload with credentials and network access.
This article explains the architecture we chose for a data-engineering customer on Google Cloud: an existing web application and agent orchestration layer, an embedded coding-agent harness, and a dedicated GKE Agent Sandbox for each workspace. It also captures what our proof of concept taught us about isolation, identity, networking, persistence, latency, and operations. The design is validated in a prototype, not yet in production.
The customer need
The customer wanted data engineers to use AI without installing and governing agent harnesses on every engineer's workstation. The team had no direct subscription with Anthropic, OpenAI, or another model provider: Google Cloud had to remain the inference and billing boundary.
The customer had built a web-based prototype using Gemini on Vertex AI. Service accounts connected it to Azure DevOps, BigQuery, Dataform, Cloud Composer, Power BI, and a curated knowledge base in Vertex AI Search. The prototype used separate agents for jobs such as Composer development, architecture, and user-story creation.
The prototype was useful, but like many function-calling assistants, its architecture stopped at question → function call → answer. That shape has four limitations:
- Documentation became stale. The agents depended on manually maintained descriptions rather than inspecting the current repository and live environment.
- Tool selection was unreliable. A growing flat list of functions gave the model more choices, but not a robust work method.
- There was no correction loop. The agent could not make a change, run a test, inspect the failure, and try again.
- There was nowhere safe to execute. Giving the web application a general-purpose shell would have mixed users, credentials, files, and long-running processes in one trust boundary.
The missing capability was a controlled execution plane in which an agent could explore and validate its work, while continuing to use models through Google Cloud.
The limitations also pointed to a simpler agent structure, which mainly addresses the second one. Most roles did not need independent agents with duplicated function lists. They needed one capable execution loop with skills loaded for the task. A Composer skill can encode repository conventions and validation; a story-refinement skill can define its workflow and output; skills stored in a repository change together with its code. Restricted modes, such as a read-only planning mode, can limit tools or identity where a real boundary is required. Separate agents remain useful for different identities or lifecycles, not merely different instructions.
Constraints that shape the design
A production execution environment should be designed from the threat model, not from the demo. Our key constraints were:
- Treat generated commands as untrusted. Prompt injection, a malicious repository, or a model mistake can all produce dangerous shell activity.
- Isolate concurrent work. One workspace's repository, generated files, process state, and credentials must not be visible to another.
- Use short-lived, scoped identity. Avoid downloaded service-account keys. A task should receive only the APIs and repositories it needs.
- Control both ingress and egress. Isolation is incomplete if a sandbox can scan internal services or send source code to arbitrary destinations.
- Keep EU processing available. Compute location and model endpoint are separate decisions; both must meet the workload's residency requirements.
- Preserve a human approval boundary. The target output is evidence plus a pull request, not an autonomous production change or merge.
- Start quickly, scale down cheaply, and let engineers return. Interactive users will not wait minutes for every chat. Idle environments should not run forever, but an engineer who comes back after lunch should not lose a half-finished investigation.
- Reuse the current product. The customer already had the UI, connectors, and domain logic. Replacing them would add risk without solving the execution problem.
The main options
The comparison reflects the options as we assessed them in October 2026.
| Option | Strength | Limitation | Best fit |
|---|---|---|---|
| Shell in the application container | Fastest prototype | Shared blast radius; weak session and credential isolation | Local experiments only |
| Container, Cloud Run job, or VM per task | Familiar primitives and flexible images | More lifecycle code; cold starts or idle cost; isolation is still our responsibility | Short jobs or long-lived developer workspaces |
| Provider-managed cloud coding sessions (Claude Code on the web, Codex cloud, GitHub Copilot coding agent) | Harness, compute, and scaling arrive together | Usually tied to a provider subscription, provider-hosted inference, and GitHub-centric workflows; less control over residency and networking | Teams already standardized on that provider and source-control platform |
| Agent Runtime + Agent Platform Code Execution sandbox | Managed GCP agent runtime, sessions, identity, scaling, and isolated stateful code execution | The standard code sandbox has a limited filesystem, no network, and fixed libraries; it is better for analysis than cloning repositories and running real build toolchains | Agent Development Kit (ADK) agents executing bounded Python or data-analysis tasks |
| Third-party managed sandbox | Fast integration and good cold starts | Another vendor boundary; verify identity, egress, image, and residency controls | Teams optimizing for speed to market |
| GKE Agent Sandbox per workspace | Custom images, gVisor isolation, policy control, warm pools, Pod snapshots, and native GCP identity | Kubernetes complexity and cluster cost | Governed, interactive engineering execution |
The provider-managed options were poor fits because the customer had no model-provider subscription and wanted inference to remain in GCP. Agent Runtime was strategically attractive, but its standard Code Execution sandbox did not provide the networked, customizable engineering workspace needed to clone Azure Repos and run project toolchains. Custom containers can narrow that gap, and Agent Runtime remains an option as the managed control plane matures. A VM per engineer also gives long-lived workspaces, but it keeps paying for idle machines and leaves isolation, images, and lifecycle to us. Pod snapshots let us keep sandbox isolation while a suspended workspace costs only its snapshot storage.
For the proof of concept we chose GKE Agent Sandbox. It combines Kubernetes scheduling and policy with a gVisor-isolated runtime designed for agent-generated code, while allowing our own harness image and Vertex AI inference. It is not a complete security system by itself: IAM, network policy, secret delivery, image security, quotas, and audit logging still matter.
Separate the control plane from the execution plane
The key design decision is to keep orchestration outside the sandbox and the agent's work inside it.
The control plane authenticates the user, applies policy, creates and resumes workspaces, and records lifecycle events. It should not run model-generated commands. The sandbox image and the harness configuration define which models, tools, and permissions are available; the harness loads skills itself. A SandboxClaim is the GKE Agent Sandbox resource that requests a sandbox; when a pre-started sandbox is free in the warm pool, the claim binds to it at once. In the prototype, the gateway serves opencode's own web UI from the sandbox.
The execution plane contains the coding-agent harness and workspace. It can use file, shell, and repository tools, but only inside a sandbox with bounded identity, network access, CPU, memory, time, and storage.
The execution environment and the harness are separate choices. We compared harnesses that could sit behind the existing UI:
| Harness | How it embeds | Model path | GCP-hosted inference | Source availability |
|---|---|---|---|---|
| opencode | HTTP server, event stream, SDK, and a native web UI | Gemini, Claude, and others | Yes, including Gemini on Vertex AI | Full harness: MIT |
| GitHub Copilot | MIT SDK controlling the Copilot CLI | Copilot service, or BYOK through an OpenAI-compatible endpoint | Conditional: use a compatible Vertex AI endpoint and validate model/tool support | SDK: MIT; CLI/runtime: proprietary |
| Claude Code / Agent SDK | Headless JSON stream and Agent SDK | Claude native or Claude on Vertex AI | Yes, with Claude on Vertex AI | Agent SDK: MIT; CLI/runtime: proprietary |
| Antigravity (Google) | Python SDK controlling a compiled runtime distributed in its wheel | Gemini on Vertex AI and other models | Yes | SDK: Apache-2.0; runtime: compiled |
The Copilot SDK's bring-your-own-key (BYOK) mode does not require GitHub authentication or a Copilot subscription and can use arbitrary OpenAI-compatible endpoints and renewable bearer tokens. This makes it different from the provider-managed Copilot coding agent discussed above, and a credible option when the selected Vertex model exposes a compatible endpoint.
We selected opencode because the complete harness was open source, built to embed as a server, and had a directly validated Gemini-on-Vertex path without a separate provider subscription. It already provided repository exploration, file and shell tools, a multi-step work-check-correct loop, event streaming, and permissions. In the second iteration of our prototype, we replaced our own chat UI with opencode's native web interface, which already provides session, file, diff, and terminal views. A chat alone is not enough: engineers need to inspect, edit, and comment on what the agent changed before they continue the conversation. ADK and Agent Runtime remain valuable for managed orchestration, identity, sessions, governance, and publishing. They were not the shortest path for this coding use case because a general framework would still require us to build and maintain the coding tools and loop that the harness already supplied.
One workspace, one sandbox
A shared sandbox is acceptable for proving that a model can clone and inspect a repository. It is not a multi-user architecture. A Kubernetes Service in front of a pool is also insufficient: normal load balancing may send consecutive requests to different filesystems.
Our first iteration bound each chat thread to its own sandbox and deleted it after an idle timeout. That model was safe, but it lost context: every new conversation started from an empty disk, and engineers often wanted a second conversation about the same checkout. The second iteration therefore makes the workspace the isolation boundary:
- A workspace owns one stable
SandboxClaim, one gVisor Pod at a time, its/workspacefiles, and its snapshots. - A session is an opencode conversation inside a workspace. Several sessions share the workspace's files and processes, which is what engineers expect when they discuss the same repository. Sessions are not isolated from each other: two agents changing the same checkout at the same time can conflict, as two engineers in one clone would. Use separate workspaces for independent parallel changes.
- Creating a new workspace is the way to get a new isolation boundary.
The workspace gateway keeps one durable record per workspace (in our prototype, a Kubernetes ConfigMap). The record holds the claim, the snapshot group, the lifecycle state, and the last activity time. A gateway restart or deployment therefore does not lose or delete workspaces. Each workspace is served on its own browser origin (w-<id>.localhost in the prototype), because opencode's web UI selects its server by origin and separate origins let two workspaces stay open side by side. The gateway resolves the workspace's current Pod address on every request instead of caching it, because a restored Pod usually gets a new IP address.
A warm pool keeps a few sandboxes started in advance, so a new workspace does not wait for a new node, which can take minutes. When a workspace is closed, its sandbox and snapshots are deleted rather than returned for reuse, and the warm pool provisions a fresh replacement. Filesystem and process state therefore do not cross workspace boundaries.
Resumable workspaces with Pod snapshots
GKE Pod snapshots capture a whole Pod: its filesystem and the memory of its running processes. That is the difference from a persistent volume. A volume keeps files, but the agent harness would restart cold. A Pod snapshot brings back the running opencode server, including its sessions, so the engineer sees the same conversation, files, and diffs after a resume.
After about 15 minutes without user or agent activity, the gateway suspends the workspace. Suspension must never capture a workspace in the middle of a change, so the gateway follows a strict order:
- Block new writes to the workspace and let requests in progress finish.
- Ask opencode, for every project directory it has opened in the workspace, whether an agent turn or terminal is active. If either is active, the workspace stays running.
- Take the snapshot and accept it only after GKE reports it as complete.
- Suspend the sandbox and wait until the original Pod is gone before the workspace is marked as suspended.
A resume is accepted only when a different Pod reports that it restored the recorded snapshot. A Pod that starts without that confirmation is not accepted as a restore, even if Kubernetes reports it as ready.
This verification proved necessary. In our October 2026 tests, on newly provisioned Autopilot nodes, some restores started an empty Pod instead of the saved state, with the error the storage backend may not be available. The saved snapshots were intact and restored correctly once the nodes were initialized. We did not establish the root cause. The gateway now keeps the workspace blocked, removes the empty Pod without taking a snapshot of it, and retries the same snapshot once. If that also fails, the engineer sees a Retry restore action instead of a silently empty workspace.
Other practical lessons:
- Snapshots need supported hardware. Whole-Pod snapshots do not support the E2 machine family, so sandboxes are pinned to N2 nodes through a ComputeClass.
- Pin images per workspace. A snapshot can only restore into a compatible image. New sandbox images go to a new, versioned warm pool, and existing workspaces stay on their original image until they are closed.
- Group snapshots per sandbox. Snapshot grouping by the sandbox's identity label prevents one workspace from restoring another workspace's state. We keep at most three snapshots per workspace to bound storage.
- Know your recovery point. The latest completed snapshot is the recovery point. Work done after it can be lost if a Pod or node fails unexpectedly. A snapshot taken on suspension is not a continuous backup.
- Suspend and resume take seconds. On initialized nodes, suspend took about 16 seconds and resume about 18 seconds in a single measured run.
At the time of our tests, Google's suspend/resume tutorial used a manually installed Agent Sandbox controller. In our tests, the managed GKE add-on completed snapshot, suspend, and resume, so we kept the manual controller only as a documented fallback.
Security is a stack of boundaries
The sandbox is only one layer. The prototype implements gVisor isolation, Workload Identity Federation, a non-root runtime with resource limits, images pinned by digest, and a dedicated, access-restricted snapshot bucket. The rest of this list is our recommended production baseline; the gaps are listed further below.
- Identity: use Workload Identity Federation instead of key files. Grant permissions by task, separate read from write, and use delegated user access where downstream row- or column-level authorization must follow the human.
- Network: default-deny ingress and egress. Use Kubernetes
NetworkPolicyfor pod and IP/CIDR boundaries, and route external HTTPS through an egress proxy or firewall with an FQDN-aware allowlist for model, source-control, and Google API endpoints. Add narrow rules separately for DNS and the identity metadata server. Test the policy: blocking DNS or metadata breaks the workload, while broad HTTPS egress defeats data-loss controls. - Runtime: run a minimal, approved image as a numeric non-root user; enforce CPU, memory, time, and storage limits; delete the workspace and its snapshots when the user closes it.
- Secrets: inject short-lived credentials only when needed. Keep them out of prompts, logs, repositories, image layers, and persistent volumes.
- Snapshots: a whole-Pod snapshot contains process memory, which can include short-lived tokens and repository content. Protect the snapshot bucket like the workspace itself: a dedicated bucket, narrow IAM, an encryption-key policy that meets your requirements, a retention limit, and deletion on close.
- Actions: allow low-impact reads and tests, but require confirmation for writes. Keep merges, production mutation, IAM changes, and schema migrations outside the agent's authority. Cap retries and spend.
- Supply chain and audit: pin, scan, and sign images. Record the image, identity, policy, tool calls, resource use, and resulting artifacts without logging sensitive content by default.
What the proof of concept validated
The complete prototype is available in the aliz-ai/gke-agent-sandbox-proto repository.
We built the design in increments so each phase answered one uncertainty. The first iteration covered phases 1–4; the second added phases 5 and 6.
- Local container: opencode called Gemini on Vertex AI, cloned a public repository, inspected it, and returned a summary.
- GKE sandbox: the same image ran non-root under gVisor and authenticated through Workload Identity Federation. Egress was limited to DNS, the metadata server, and public HTTPS, with cluster-internal and private ranges blocked.
- Web integration: a small UI built with CopilotKit, an open-source chat UI framework, drove the real harness through an AG-UI (agent-user interaction protocol) adapter and streamed progress back to the browser.
- Per-chat isolation (first iteration): two browser chats created two
SandboxClaimresources and two gVisor pods. A repository cloned in chat A was absent from chat B's/workspace. - Native UI and workspaces: opencode's own web interface ran through the workspace gateway. Two workspaces ran in two Pods with separate files, while two sessions inside one workspace saw the same files.
- Suspend and resume: after a suspend removed the Pod, the resumed workspace showed the same uncommitted file, session history, and diff. This worked for manual suspension and for automatic suspension after 15 idle minutes. Workspace records and states survived a gateway redeployment, and a model request after resume read the restored file correctly.
The phased approach exposed operational details early:
- gVisor pods required the sandbox runtime selector, toleration, and numeric user ID; Git was reliable after forcing HTTP/1.1.
- Network policy needed explicit DNS and metadata-server rules besides HTTPS; without them, name resolution and Workload Identity failed. Aggressive readiness probes interrupted long streams.
- Sandbox Pods cannot be reached with
kubectl port-forward. The gateway runs in-cluster and reaches sandboxes by Pod address; production needs the Agent Sandbox router, which routes traffic to claimed sandboxes, or an authenticated gateway. - Model availability and residency are separate. The public-repository prototype used a global endpoint; EU workloads must select an EU-supported endpoint.
What the prototype does not cover yet
The prototype is a feasibility test for one operator. Before production, these gaps need their own design and testing:
- Authentication and ownership. The prototype has no user sign-in or workspace ownership. It is reachable only through a local port-forward to the gateway.
- Egress allowlisting. The prototype blocks private ranges but allows public HTTPS. Production needs the FQDN-aware egress control described above.
- A tighter recovery point. Only orderly suspend and resume are verified. Protecting work done since the last snapshot needs periodic checkpoints or another design, which we have not tested.
- Action and spend controls. Write confirmations, spend caps, and image scanning and signing are recommendations; the prototype does not enforce them.
- Scale. We tested a small number of concurrent workspaces. Warm-pool sizing, node scale-up, and restore behavior under realistic concurrency are still open.
A production checklist
Before production, verify four things end to end:
- Isolation: one authenticated user's workspace maps to one sandbox, with tested deny-by-default networking and workspace-scoped authorization.
- Lifecycle: idle workspaces suspend only when no work is active; restores are verified and failed restores stay blocked; closed workspaces remove their claims, snapshots, and credentials; the recovery point is agreed with users.
- Governance: compute, inference, logs, snapshots, and artifacts meet one residency policy; images are pinned, scanned, and signed; resource use and spend have hard limits.
- Approval: the durable output is a reviewable artifact with test evidence. Production mutation and merge remain human-controlled.
The broader lesson
Agent safety is not achieved by asking the model to be careful. It comes from making the environment safe when the model is wrong.
The pattern is therefore: reuse a capable harness, run each workspace in its own sandbox, suspend it safely when idle, grant the minimum identity and network access, collect evidence, and stop at a human approval gate. This architecture lets agents inspect the current code and systems instead of depending on stale documentation, while keeping their execution bounded enough to earn trust incrementally.