Lesson 12: Sandbox Deployment Options — From Cloud Providers to Self-Hosted Kubernetes

This lesson covers the four main options for deploying agent sandboxes: cloud-hosted providers, custom Kubernetes backends, self-hosted NemoClaw/OpenShell on Kubernetes, and hybrid approaches. We compare security features, credential handling, and when to use each option.

Primary sources: LangChain DeepAgents Sandboxes | OpenShell Kubernetes Setup | OpenShell Helm Chart

1. The Four Deployment Options

Four Deployment Options Overview

graph TB subgraph OPTION_A["Option A: Cloud Sandbox Providers"] A1["E2B, Modal, Daytona, Vercel, LangSmith"] A2["External cloud infrastructure"] A3["Provider-managed security"] end subgraph OPTION_B["Option B: Custom K8s Backend"] B1["Your Kubernetes Cluster"] B2["Custom SandboxBackend implementation"] B3["DIY security with K8s primitives"] end subgraph OPTION_C["Option C: Self-Hosted OpenShell on K8s"] C1["Your Kubernetes Cluster"] C2["OpenShell Gateway + Helm Chart"] C3["Full OpenShell security stack"] end subgraph OPTION_D["Option D: Hybrid Approach"] D1["OpenShell locally or on K8s"] D2["DeepAgents orchestration"] D3["Best of both worlds"] end style OPTION_A fill:#dbeafe,stroke:#3b82f6 style OPTION_B fill:#dcfce7,stroke:#16a34a style OPTION_C fill:#dcfce7,stroke:#16a34a style OPTION_D fill:#fef3c7,stroke:#f59e0b

2. Option A: Cloud Sandbox Providers (LangChain Native)

Cloud-Hosted Sandboxes External

LangChain DeepAgents integrates with multiple cloud sandbox providers:

Architecture

Cloud Provider Architecture

graph TB subgraph YOUR_INFRA["Your Infrastructure"] APP["Your Application"] AGENT["DeepAgents Code"] end subgraph CLOUD["Cloud Provider - E2B, Modal, etc."] SB["Sandbox Container"] FS["Filesystem"] SHELL["Shell Execution"] end APP --> AGENT AGENT -->|"API calls"| SB SB --> FS SB --> SHELL style YOUR_INFRA fill:#e0e7ff,stroke:#6366f1 style CLOUD fill:#dbeafe,stroke:#3b82f6

Code Example

# Using LangSmith Sandbox with DeepAgents
from deepagents import create_deep_agent
from deepagents.backends import LangSmithSandbox
from langsmith.sandbox import SandboxClient

client = SandboxClient()
ls_sandbox = client.create_sandbox()
backend = LangSmithSandbox(sandbox=ls_sandbox)

agent = create_deep_agent(
    model="anthropic:claude-sonnet-4-20250514",
    system_prompt="You are a coding assistant with sandbox access.",
    backend=backend,
)

try:
    result = agent.invoke({
        "messages": [{"role": "user", "content": "Create and run a Python script"}]
    })
finally:
    client.delete_sandbox(ls_sandbox.name)

What You Get

FeatureStatusNotes
Filesystem isolationYesAgent can't read your local files
Shell executionYesVia execute() method
Process isolationYesContainer-level
File upload/downloadYesSeed and retrieve artifacts
Binary-scoped network policyNoCan't say "only git can reach github.com"
Credential injection at runtimeNoMust inject into sandbox (security risk)
Kernel-level enforcementNoContainer-level only
OPA policy languageNoProvider-specific configuration
Security Warning (from LangChain docs):
"Never put secrets inside a sandbox. API keys, tokens, database credentials... can be read and exfiltrated by a context-injected agent."

When to Use

3. Option B: Custom Kubernetes Backend

DIY Kubernetes Sandbox Self-Hosted

Implement your own SandboxBackendProtocol to run agent commands in Kubernetes pods you control.

Architecture

Custom K8s Backend Architecture

graph TB subgraph K8S_CLUSTER["Your Kubernetes Cluster"] APP["Your Application Pod"] AGENT["DeepAgents Code"] subgraph SANDBOX_POD["Sandbox Pod"] SB["Agent Container"] SECCOMP["Seccomp Profile"] NETPOL["NetworkPolicy"] end end APP --> AGENT AGENT -->|"kubectl exec"| SB SB --> SECCOMP SB --> NETPOL style K8S_CLUSTER fill:#dcfce7,stroke:#16a34a style SANDBOX_POD fill:#bbf7d0,stroke:#22c55e

Implementation

from deepagents.backends.sandbox import BaseSandbox, ExecuteResult
import kubernetes

class KubernetesSandbox(BaseSandbox):
    """Custom sandbox backend that runs commands in Kubernetes pods."""

    def __init__(self, pod_name: str, namespace: str = "default"):
        self.pod_name = pod_name
        self.namespace = namespace
        self.k8s_client = kubernetes.client.CoreV1Api()

    def execute(self, command: str, timeout: int = 300) -> ExecuteResult:
        """Execute command in the K8s pod via kubectl exec."""
        response = kubernetes.stream.stream(
            self.k8s_client.connect_get_namespaced_pod_exec,
            self.pod_name,
            self.namespace,
            command=["/bin/bash", "-c", command],
            stdout=True,
            stderr=True,
            stdin=False,
            tty=False,
        )
        return ExecuteResult(
            output=response,
            exit_code=0,  # Parse from response
        )

Pod Template with Security

# sandbox-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: agent-sandbox-${SESSION_ID}
  namespace: sandboxes
spec:
  securityContext:
    # Seccomp profile for syscall filtering
    seccompProfile:
      type: Localhost
      localhostProfile: agent-sandbox-seccomp.json

  containers:
  - name: sandbox
    image: python:3.12-slim
    securityContext:
      # Drop dangerous capabilities
      capabilities:
        drop:
        - SYS_ADMIN
        - SYS_PTRACE
        - NET_RAW
        - SYS_MODULE
        - MKNOD
        - SETUID
        - SETGID
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      runAsUser: 1000

    volumeMounts:
    - name: workspace
      mountPath: /sandbox
    - name: tmp
      mountPath: /tmp

  volumes:
  - name: workspace
    emptyDir:
      sizeLimit: 10Gi
  - name: tmp
    emptyDir:
      sizeLimit: 1Gi

NetworkPolicy for Egress Control

# network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: sandbox-egress
  namespace: sandboxes
spec:
  podSelector:
    matchLabels:
      app: agent-sandbox
  policyTypes:
  - Egress
  egress:
  # Allow only specific destinations
  - to:
    - ipBlock:
        cidr: 10.0.0.0/8  # Internal services only
    ports:
    - protocol: TCP
      port: 5432  # PostgreSQL

What You Get

FeatureStatusNotes
Filesystem isolationYesPod-level isolation
Shell executionYesVia kubectl exec
Process isolationYesContainer + seccomp
Capability dropsYesVia securityContext
Network policyPartialPer-pod, not per-binary
Binary-scoped networkNoK8s NetworkPolicy is pod-level
Credential injectionDIYBuild with sidecar proxy
Landlock LSMDIYRequires kernel 5.13+

When to Use

4. Option C: Self-Hosted OpenShell on Kubernetes

Full OpenShell Stack Self-Hosted

Deploy NVIDIA OpenShell's complete security stack on your own Kubernetes cluster using the official Helm chart.

Key Discovery: OpenShell publishes an official Helm chart at oci://ghcr.io/nvidia/openshell/helm-chart and runs a K3s cluster inside a single Docker container. The gateway image is available for linux/amd64 and linux/arm64.

Architecture

Self-Hosted OpenShell on Kubernetes

graph TB subgraph K8S_CLUSTER["Your Kubernetes Cluster"] subgraph CONTROL_PLANE["OpenShell Control Plane - nemoclaw namespace"] GW["Gateway Service
ghcr.io/nvidia/openshell/gateway"] OPA["Policy Engine
OPA/Rego"] IR["Inference Router
inference.local handler"] PKI["PKI Init Job
mTLS certificates"] end subgraph SANDBOX_NS["Sandbox Pods - sandboxes namespace"] SB1["Sandbox Pod 1"] SB2["Sandbox Pod 2"] SBN["Sandbox Pod N"] subgraph INSIDE_POD["Inside Each Pod"] SUP["Supervisor"] AGENT["Agent Process"] PROXY["Policy Proxy Sidecar"] end end subgraph YOUR_APP["Your Application"] APP["Your Services"] CB["Context Builder"] PG["Policy Gateway"] end end APP --> CB CB --> GW GW --> SB1 GW --> SB2 GW --> SBN SUP --> AGENT AGENT --> PROXY PROXY --> OPA AGENT -->|"inference.local"| IR SB1 --> PG PG --> APP style CONTROL_PLANE fill:#dcfce7,stroke:#16a34a style GW fill:#4ade80 style OPA fill:#4ade80 style IR fill:#4ade80

OpenShell Security Layers

From the OpenShell documentation:

LayerComponentFunction
ProcessSupervisorDrops privileges, applies identity rules, disables escalation paths, spawns agent as restricted child
FilesystemLandlock LSMPre-start policy application; undeclared paths blocked (kernel-enforced)
NetworkPolicy ProxyAll egress through proxy; evaluates destination, port, binary identity, L7 rules
CredentialsGatewayControlled delivery via policy paths or proxy rules
InferenceInference RouterIntercepts inference.local, hides provider credentials
ObservabilitySupervisorSecurity/lifecycle logs, relay endpoints

Helm Installation

# 1. Install Agent Sandbox CRDs (prerequisite)
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/manifest.yaml

# 2. Create namespace
kubectl create namespace openshell

# 3. Install OpenShell from OCI registry
helm upgrade --install openshell \
  oci://ghcr.io/nvidia/openshell/helm-chart \
  --version "0.1.0" \
  --namespace openshell \
  --set server.sandboxNamespace=sandboxes \
  --set server.oidc.issuer="https://your-idp.example.com/realms/openshell"

# 4. Wait for rollout
kubectl -n openshell rollout status statefulset/openshell

# 5. Port-forward for local access
kubectl -n openshell port-forward svc/openshell 8080:8080

Key Helm Values

# values.yaml for production
replicaCount: 2  # HA requires external PostgreSQL

workload:
  kind: deployment  # Use deployment with external DB

server:
  logLevel: "info"
  sandboxNamespace: "sandboxes"
  sandboxImage: "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"

  # External PostgreSQL for multi-replica
  externalDbSecret: "openshell-pg-credentials"

  # OIDC authentication
  oidc:
    issuer: "https://your-idp.example.com/realms/openshell"
    audience: "openshell-cli"
    adminRole: "openshell-admin"
    userRole: "openshell-user"

  # AppArmor profile for sandbox containers
  appArmorProfile: "runtime/default"

# Supervisor topology
supervisor:
  topology: "sidecar"  # or "combined"
  sidecar:
    processBinaryAwareNetworkPolicy: true
    proxyUid: 1337

# Network policy for sandbox pods
networkPolicy:
  enabled: true

Compute Drivers

OpenShell supports four compute drivers for provisioning sandboxes:

DriverWhen to UseSecurity Features
Docker Local development, single-machine gateways Container isolation, mount restrictions, GPU via CDI
Podman Rootless container requirements on Linux Rootless isolation, same mount restrictions as Docker
MicroVM VM-level isolation, enhanced security, GPU workloads Full VM boundary, nftables per TAP interface, immutable root disk
Kubernetes Shared clusters, remote compute, GPU scheduling RBAC, AppArmor, ServiceAccount tokens, PVC isolation, Landlock

Supervisor Topologies

Supervisor Topologies Comparison

graph LR subgraph COMBINED["Combined Topology - default"] C_AGENT["Agent Container"] C_CAP["Linux Capabilities
for network, Landlock, etc."] C_AGENT --> C_CAP end subgraph SIDECAR["Sidecar Topology"] S_INIT["Init Container
nftables setup"] S_AGENT["Agent Container
runs as sandbox UID"] S_SIDECAR["Sidecar
binary-aware policy"] S_INIT --> S_AGENT S_AGENT --> S_SIDECAR end style COMBINED fill:#e8f5e9,stroke:#76b900 style SIDECAR fill:#e3f2fd,stroke:#1a5fb4

Combined: Agent container has Linux capabilities for network namespace, Landlock, privilege changes.

Sidecar: Agent runs with no added capabilities; root init container does nftables; sidecar handles binary-aware network policy.

What You Get

FeatureStatusNotes
Filesystem isolationYesLandlock LSM (kernel-enforced)
Binary-scoped networkYesCONNECT proxy + OPA policies
Credential injectionYesinference.local pattern
Capability dropsYes15 capabilities dropped by supervisor
Syscall filteringYesSeccomp-bpf via supervisor
OPA policy languageYesRego policies for all layers
Blueprint versioningYesSHA-256 verified, immutable
Multi-tenantPartialNamespace isolation; shared nodes need care
Experimental Status: "The OpenShell Helm chart is experimental and under active development... Do not use it in production." — OpenShell Docs

When to Use

5. Option D: Hybrid — OpenShell + DeepAgents

Best of Both Worlds Hybrid

Use DeepAgents for orchestration with OpenShell as the sandbox backend. This is the official integration path.

Official Integration: LangChain provides openshell-deepagent — "A general-purpose coding agent that runs inside an NVIDIA OpenShell sandbox, orchestrated by Deep Agents and powered by NVIDIA Nemotron."

Architecture

Hybrid Architecture: DeepAgents + OpenShell

graph TB subgraph YOUR_INFRA["Your Infrastructure"] subgraph DEEPAGENTS["DeepAgents Orchestration"] HARNESS["Harness Layer
AGENTS.md, Skills, Tools"] CB["Context Builder"] PG["Policy Gateway
Deterministic"] end subgraph OPENSHELL["OpenShell Runtime"] GW["OpenShell Gateway"] SUP["Supervisor"] PROXY["Policy Proxy"] IR["Inference Router"] end subgraph SANDBOX["Sandbox"] AGENT["Agent Process"] end end HARNESS --> CB CB --> GW GW --> SUP SUP --> AGENT AGENT --> PROXY AGENT -->|"inference.local"| IR AGENT -->|"structured output"| PG PG --> HARNESS style DEEPAGENTS fill:#fef3c7,stroke:#f59e0b style OPENSHELL fill:#dcfce7,stroke:#16a34a style GW fill:#4ade80

Code Example

# Using OpenShell as DeepAgents backend
import openshell
from deepagents import create_deep_agent
from langchain_nvidia_openshell import OpenShellSandbox

# Create OpenShell sandbox
with openshell.Sandbox(delete_on_exit=True) as sandbox:
    backend = OpenShellSandbox(sandbox=sandbox)

    # Create DeepAgent with OpenShell backend
    agent = create_deep_agent(
        model="anthropic:claude-sonnet-4-20250514",
        system_prompt=load_skill("coding.md") + load_agents_md("client_a"),
        backend=backend,
    )

    result = agent.invoke({
        "messages": [{"role": "user", "content": "..."}]
    })

    # Policy Gateway (outside sandbox)
    decision = policy_gateway(result, client_rules)

Why This Combination Works

From the LangChain blog:

"Deep agents connect to real data sources (Linear, Slack, Salesforce). You want the agent to learn new things on the fly, but you don't want to rely on prompt instructions alone to prevent misuse — something external to the agent has to enforce security, which is what OpenShell does."
ComponentFrom DeepAgentsFrom OpenShell
OrchestrationYes
AGENTS.md / SkillsYes
LangSmith tracingYes
Human-in-the-loopYes
Memory (SQLite)Yes
Kernel-level isolationYes
Binary-scoped networkYes
Credential injectionYes
OPA policiesYes

6. Comprehensive Comparison

Feature Cloud Providers
(E2B, Modal)
Custom K8s
Backend
Self-Hosted
OpenShell
Hybrid
(DeepAgents+OpenShell)
Infrastructure External Your K8s Your K8s Your K8s
Setup complexity Low Medium High High
Filesystem isolation Container Container Landlock (kernel) Landlock (kernel)
Network policy All or nothing Per-pod Per-binary + L7 Per-binary + L7
Credential isolation Must inject DIY sidecar inference.local inference.local
Syscall filtering No Basic seccomp Full seccomp-bpf Full seccomp-bpf
Capability drops Provider default Configurable 15 dropped 15 dropped
Policy language Provider-specific K8s YAML OPA/Rego OPA/Rego
Harness tuning DIY DIY DIY Built-in
HITL support Basic DIY Gateway level Full
Observability Provider logs K8s logs Supervisor logs LangSmith + Supervisor
GPU support Some providers K8s GPU plugin CDI / nvidia.com/gpu CDI / nvidia.com/gpu
Linux kernel req N/A Any 5.13+ (Landlock) 5.13+ (Landlock)

7. Decision Framework

Decision Flowchart: Choosing a Deployment Option

flowchart TD START["Start"] --> Q1{"Need binary-scoped
network policy?"} Q1 -->|"No"| Q2{"Need credential
isolation?"} Q1 -->|"Yes"| OPENSHELL["Option C or D:
Self-Hosted OpenShell"] Q2 -->|"No"| Q3{"Want to manage
infrastructure?"} Q2 -->|"Yes"| Q4{"Can build
sidecar proxy?"} Q3 -->|"No"| CLOUD["Option A:
Cloud Providers"] Q3 -->|"Yes"| CUSTOM["Option B:
Custom K8s Backend"] Q4 -->|"No"| OPENSHELL Q4 -->|"Yes"| CUSTOM OPENSHELL --> Q5{"Need DeepAgents
orchestration?"} Q5 -->|"Yes"| HYBRID["Option D:
Hybrid Approach"] Q5 -->|"No"| PURE["Option C:
Pure OpenShell"] style CLOUD fill:#dbeafe,stroke:#3b82f6 style CUSTOM fill:#dcfce7,stroke:#16a34a style PURE fill:#dcfce7,stroke:#16a34a style HYBRID fill:#fef3c7,stroke:#f59e0b

Quick Decision Table

If you need...Choose...
Quick setup, no infra managementOption A: Cloud Providers
K8s control, basic securityOption B: Custom K8s Backend
Binary-scoped network, credential isolationOption C: Self-Hosted OpenShell
Full security + harness tuning + HITLOption D: Hybrid
Financial/compliance workloadsOption C or D (kernel-level enforcement)

8. Kubernetes-Specific Considerations

Node Requirements for OpenShell

# Label nodes that support Landlock (kernel 5.13+)
kubectl label nodes node-1 openshell.nvidia.com/landlock=true

# Check kernel version
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kernelVersion}{"\n"}{end}'

Multi-Tenancy

Multi-Tenant Considerations:

Production Checklist

Before deploying OpenShell to production:
  1. Verify nodes have Linux kernel 5.13+ (for Landlock)
  2. Install Agent Sandbox CRDs
  3. Configure external PostgreSQL for HA (not SQLite)
  4. Set up OIDC provider for authentication
  5. Configure TLS certificates (or use cert-manager)
  6. Define OPA policies for network/filesystem rules
  7. Test supervisor topology (combined vs sidecar)
  8. Monitor: gateway logs, supervisor logs, OPA decisions

Knowledge Check

Question 1: What is the key security limitation of cloud sandbox providers (E2B, Modal, etc.)?

Correct Answer: B - Cloud sandbox providers offer container-level isolation but cannot provide binary-scoped network policies ("only git can reach github.com") or credential injection at runtime (credentials must be injected into the sandbox, creating a security risk).

Question 2: What Linux kernel version is required for Landlock LSM support in OpenShell?

Correct Answer: C - Landlock LSM requires Linux kernel 5.13 or newer. Compatible systems include Ubuntu 20.04+, RHEL 8+, Rocky Linux 8+, Amazon Linux 2023+, and Fedora 32+.

Question 3: What is the purpose of the inference.local endpoint in OpenShell?

Correct Answer: B - The inference.local endpoint routes model traffic to configured backends while keeping provider credentials outside the sandbox. The supervisor intercepts requests, the inference router injects credentials and forwards to providers, and the agent never sees the raw API keys.

Question 4: What prerequisite must be installed before the OpenShell Helm chart?

Correct Answer: B - OpenShell uses the Agent Sandbox Kubernetes SIG project to provision sandbox pods. You must install the Agent Sandbox controller and its CRDs before installing the OpenShell Helm chart.

Question 5: When should you choose the Hybrid approach (DeepAgents + OpenShell)?

Correct Answer: C - The Hybrid approach combines DeepAgents' orchestration capabilities (AGENTS.md, Skills, HITL, LangSmith tracing) with OpenShell's kernel-level security (Landlock, binary-scoped network, credential isolation). This is ideal for production workloads requiring both flexibility and strong security.

Summary

Key Takeaways

Related: Lesson 11: Harness Tuning & Policy Gateways | Lesson 10: Five Protection Layers

Questions? Ask your teacher for clarification on any concept. Deployment architecture decisions have long-term implications — it's worth understanding the trade-offs.

Sources