Lesson 10: Five Protection Layers - Technical Deep Dive
Understanding the kernel-level implementation, upstream/downstream connections, and customization of each security layer
Five Layers: Upstream to Downstream Flow
(network, file, syscall)"] end subgraph LAYER5["Layer 5: INFERENCE"] INF["inference.local interception
Credential injection"] end subgraph LAYER4["Layer 4: GATEWAY AUTH"] GW["Device authentication
Session validation"] end subgraph LAYER3["Layer 3: PROCESS"] PROC["Capability checks
Seccomp filters"] end subgraph LAYER2["Layer 2: FILESYSTEM"] FS["Landlock LSM
Path validation"] end subgraph LAYER1["Layer 1: NETWORK"] NET["Policy Proxy
Binary + destination check"] end subgraph EXTERNAL["External Resources"] EXT["Model Provider
Network Endpoint
Filesystem"] end A --> LAYER5 A --> LAYER3 A --> LAYER2 A --> LAYER1 LAYER5 --> LAYER4 LAYER4 --> LAYER1 LAYER1 --> EXT LAYER3 -->|"Syscall allowed"| LAYER2 LAYER2 -->|"Path allowed"| EXT style LAYER1 fill:#e3f2fd,stroke:#1a5fb4,stroke-width:2px style LAYER2 fill:#e8f5e9,stroke:#76b900,stroke-width:2px style LAYER3 fill:#fff3e0,stroke:#f57c00,stroke-width:2px style LAYER4 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px style LAYER5 fill:#fce4ec,stroke:#c2185b,stroke-width:2px
Layers don't run sequentially for every request - different request types hit different layers. Network requests go through Layer 1; file operations through Layer 2+3; inference through Layer 5→4→1.
What It Does (High Level)
Intercepts ALL outbound network traffic from the sandbox and decides: ALLOW, ROUTE (for inference), or DENY.
Technical Implementation
Core Technology: CONNECT Proxy
All sandbox egress flows through OpenShell's CONNECT proxy. This isn't iptables NAT - it's a userspace proxy that the agent must use (enforced by network namespace).
Network Layer Architecture
(forced by namespace)"] end subgraph HOST["Host (Outside Sandbox)"] subgraph POLICY_PROXY["Policy Proxy"] INTERCEPT["Traffic Interceptor"] BINARY_ID["Binary Identifier
/proc/pid/exe + SHA256"] RULE_EVAL["Rule Evaluator
(OPA)"] TLS_TERM["TLS Terminator
(for L7 inspection)"] end end subgraph EXTERNAL["External"] DEST["Destination
(github.com, pypi.org, etc.)"] end AGENT -->|"connect()"| PROXY_CLIENT PROXY_CLIENT -->|"CONNECT request"| INTERCEPT INTERCEPT --> BINARY_ID BINARY_ID --> RULE_EVAL RULE_EVAL -->|"ALLOW"| DEST RULE_EVAL -->|"DENY"| AGENT RULE_EVAL -->|"L7 needed"| TLS_TERM TLS_TERM --> DEST style POLICY_PROXY fill:#e3f2fd,stroke:#1a5fb4
How Binary Identification Works
The Policy Engine needs to know WHICH binary is making the request (so "git" can access github.com but "curl" can't).
# Binary identification process: # 1. Get the binary path from kernel (trusted) binary_path = readlink("/proc/<pid>/exe") # Returns: /usr/bin/git # 2. Walk process tree for ancestor binaries ancestors = walk_proc_tree(pid) # Returns: [/usr/bin/git, /bin/bash, /usr/bin/dcode] # 3. Compute SHA256 hash on first use (cached) binary_hash = sha256(read_file(binary_path)) # Returns: a1b2c3d4e5f6... # 4. Match against policy rules rule_match = evaluate_policy(binary_path, binary_hash, destination)
Two Inspection Modes
| Mode | What It Checks | Performance | Use Case |
|---|---|---|---|
| L4-only | Host, port, binary | Fast (no TLS termination) | Simple allow/deny by destination |
| L7 inspection | HTTP method, path, headers, body | Slower (TLS termination required) | Fine-grained API access control |
TLS Termination for L7
For L7 inspection, the proxy terminates TLS using the sandbox's ephemeral CA. The agent sees a valid certificate (signed by the sandbox CA which is in its trust store), but the proxy can inspect the plaintext HTTP.
Network Policy Configuration
# network_policy.yaml version: "1" default_action: deny # Deny-by-default rules: # Rule 1: Git can access GitHub (L4-only) - name: "git-github" binaries: - /usr/bin/git destinations: - host: "github.com" - host: "*.github.com" action: allow # Rule 2: npm can access registry (L4-only) - name: "npm-registry" binaries: - /usr/bin/npm - /usr/bin/node destinations: - host: "registry.npmjs.org" action: allow # Rule 3: Python with L7 inspection (restrict to specific API paths) - name: "python-stripe-api" binaries: - /usr/bin/python3 destinations: - host: "api.stripe.com" protocol: rest # Enable L7 inspection rules: - methods: [GET, POST] paths: - "/v1/charges/**" - "/v1/customers/**" # Deny: /v1/transfers/**, /v1/payouts/** action: allow # Rule 4: Wildcard patterns - name: "google-ai-apis" binaries: - "*" # Any binary destinations: # Single label wildcard - host: "*.googleapis.com" # Recursive wildcard - host: "**.google.com" # Intra-label wildcard - host: "*-aiplatform.googleapis.com" action: allow
Upstream / Downstream Connections
- Agent network syscalls (connect, sendto)
- Layer 5 (inference requests routed here)
- Gateway (policy updates, hot-reloadable)
- External endpoints (if allowed)
- Audit log (all decisions logged)
- Agent (DENY responses)
Creating Custom Network Rules
# Apply custom network policy openshell policy set my-sandbox --network-policy ./my-network-policy.yaml # View current policy openshell policy get my-sandbox --format yaml # Hot-reload (no sandbox restart needed) openshell policy reload my-sandbox
What It Does (High Level)
Controls which filesystem paths the agent can read, write, or execute. Enforced at the kernel level - cannot be bypassed from userspace.
Technical Implementation
Core Technology: Landlock LSM
Landlock is a Linux Security Module (LSM) available in kernel 5.13+ (Ubuntu 22.04+). Unlike AppArmor or SELinux, Landlock is stackable and can be used by unprivileged processes to sandbox themselves.
Landlock Enforcement Flow
(applied Landlock rules at startup)"] end subgraph KERNEL["Kernel Space"] VFS["VFS Layer
(Virtual File System)"] LANDLOCK["Landlock LSM Hook"] RULES["Landlock Ruleset
(compiled at sandbox creation)"] FS["Actual Filesystem"] end AGENT -->|"syscall"| VFS VFS --> LANDLOCK LANDLOCK -->|"check"| RULES RULES -->|"ALLOW"| FS RULES -->|"DENY"| AGENT SUPERVISOR -.->|"landlock_add_rule()
landlock_restrict_self()"| RULES style LANDLOCK fill:#e8f5e9,stroke:#76b900,stroke-width:2px style RULES fill:#c8e6c9,stroke:#76b900
How Paths Are Created and Managed
The Supervisor creates the Landlock ruleset at sandbox startup. Once landlock_restrict_self() is called, the rules are immutable for that process tree.
// Simplified Landlock setup (what Supervisor does internally) // 1. Create a new ruleset struct landlock_ruleset_attr ruleset_attr = { .handled_access_fs = LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_WRITE_FILE | LANDLOCK_ACCESS_FS_EXECUTE | LANDLOCK_ACCESS_FS_MAKE_REG | ... }; int ruleset_fd = landlock_create_ruleset(&ruleset_attr, sizeof(ruleset_attr), 0); // 2. Add rules for read-only paths struct landlock_path_beneath_attr path_attr = { .allowed_access = LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR, .parent_fd = open("/usr", O_PATH) }; landlock_add_rule(ruleset_fd, LANDLOCK_RULE_PATH_BENEATH, &path_attr, 0); // 3. Add rules for read-write paths path_attr.allowed_access = LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_WRITE_FILE | LANDLOCK_ACCESS_FS_MAKE_REG; path_attr.parent_fd = open("/sandbox", O_PATH); landlock_add_rule(ruleset_fd, LANDLOCK_RULE_PATH_BENEATH, &path_attr, 0); // 4. Enforce the ruleset (irreversible!) prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); // Required first landlock_restrict_self(ruleset_fd, 0); // Apply rules // After this point, the process (and all children) are restricted // Rules CANNOT be removed or modified
Default Filesystem Policy
# filesystem_policy.yaml (default) version: "1" read_only: # System binaries and libraries - /usr - /lib - /lib64 # System configuration (read but not write) - /etc # Proc filesystem (process info) - /proc # Device nodes (limited) - /dev/urandom - /dev/null - /dev/zero # Application code (mounted from host) - /app # Logs (read for debugging) - /var/log read_write: # Agent workspace - this is where the project lives - /sandbox # Temporary files - /tmp # PTY devices (for terminal) - /dev/pts no_access: # Sensitive paths (hidden via mount namespace before Landlock) - /root - /home - /var/run/secrets
Creating Custom Filesystem Paths
# custom_filesystem.yaml version: "1" # Inherit defaults extends: "default" # Add custom read-only paths read_only: # Include defaults plus: - /opt/mycompany/libs # Company shared libraries - /data/models # ML models (read-only) - /etc/myapp # App configuration # Add custom read-write paths read_write: # Include defaults plus: - /sandbox - /tmp - /var/cache/myapp # Application cache - /data/outputs # Where agent writes results # Explicitly deny (even if parent is allowed) deny: - /etc/shadow # Never readable - /etc/sudoers # Never readable
# Apply in Blueprint spec: policy: filesystem: config_file: "./custom_filesystem.yaml" # Or inline: read_write: - /sandbox - /tmp - /data/outputs
Important: Filesystem Policy is NOT Hot-Reloadable
Unlike network policy, filesystem policy is compiled into the Landlock ruleset at sandbox creation. Changing it requires destroying and recreating the sandbox.
Inode-Type Tailoring
Landlock rules are adjusted based on whether paths are directories or files:
- Directory: Gets
READ_DIR,REMOVE_DIR,MAKE_*permissions - File: Gets
READ_FILE,WRITE_FILE,EXECUTEpermissions
The Supervisor detects this automatically when building the ruleset.
Upstream / Downstream Connections
- Layer 3 (syscall must pass seccomp first)
- Agent file operations (open, read, write, unlink)
- Actual filesystem (if allowed)
- Agent (EACCES error if denied)
Source: Linux Kernel Landlock Documentation, NVIDIA OpenShell Security
What It Does (High Level)
Controls what the agent process CAN DO at the system level: which syscalls it can make, which capabilities it has, and how many resources it can consume.
Technical Implementation
Process Security Stack
(e.g., socket(), ptrace())"] end subgraph CHECKS["Security Checks (in order)"] CAP["1. Capability Check
Does process have required cap?"] SECCOMP["2. Seccomp-BPF Filter
Is syscall in allowlist?"] LSM["3. LSM Hooks
(Landlock, etc.)"] end subgraph KERNEL["Kernel"] EXEC["Execute Syscall"] end SYSCALL --> CAP CAP -->|"Has cap"| SECCOMP CAP -->|"Missing cap"| DENY1["EPERM"] SECCOMP -->|"Allowed"| LSM SECCOMP -->|"Blocked"| DENY2["SIGSYS / EPERM"] LSM -->|"Allowed"| EXEC LSM -->|"Blocked"| DENY3["EACCES"] style CAP fill:#fff3e0,stroke:#f57c00 style SECCOMP fill:#ffecb3,stroke:#f57c00 style LSM fill:#ffe0b2,stroke:#f57c00
3.1 Capability Dropping
Linux capabilities break root into ~40 distinct privileges. OpenShell drops dangerous ones:
# Capabilities dropped at entrypoint (via capsh) # Dangerous admin capabilities CAP_SYS_ADMIN # Mount, namespace, many admin ops - THE most dangerous CAP_SYS_PTRACE # Debug/trace other processes - can steal secrets CAP_SYS_CHROOT # Change root directory - escape sandbox # Network capabilities CAP_NET_RAW # Raw sockets - bypass proxy, packet sniffing CAP_NET_BIND_SERVICE # Bind to ports < 1024 # File capabilities CAP_DAC_OVERRIDE # Bypass file permission checks - read anything CAP_FOWNER # Bypass ownership checks CAP_FSETID # Set SUID/SGID bits CAP_SETFCAP # Set file capabilities # Other dangerous caps CAP_MKNOD # Create device nodes CAP_AUDIT_WRITE # Write to audit log # Dropped during privilege step-down CAP_SETUID # Change UID CAP_SETGID # Change GID CAP_CHOWN # Change file ownership CAP_KILL # Send signals to other processes
Fail-Closed Behavior
Supervisor retains CAP_SETPCAP only to clear the bounding set. Spawn aborts unless the bounding set ends up empty. Set NEMOCLAW_REQUIRE_CAP_DROP=1 for strict mode.
3.2 Seccomp-BPF Filters
Seccomp (Secure Computing) filters syscalls using BPF programs:
# Simplified seccomp filter (what OpenShell applies) # First, set no-new-privileges (required for seccomp) prctl(PR_SET_NO_NEW_PRIVS, 1); # Then apply seccomp filter struct sock_filter filter[] = { /* Load syscall number */ BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)), /* Block dangerous syscalls */ BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_ptrace, 0, 1), BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ERRNO | EPERM), BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_socket, 0, 3), /* For socket(), check domain - block AF_PACKET, AF_NETLINK */ BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, args[0])), BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, AF_PACKET, 0, 1), BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ERRNO | EPERM), /* Allow everything else */ BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW), }; seccomp(SECCOMP_SET_MODE_FILTER, 0, &prog);
3.3 Resource Limits (ulimit / cgroups)
# Resource limits applied to sandbox # Process limit - prevents fork bombs ulimit -u 512 # Max 512 processes # File descriptor limit ulimit -n 65536 # Max 65536 open files # Docker equivalent docker run \ --ulimit nproc=512:512 \ --ulimit nofile=65536:65536 \ --memory=4g \ --cpus=2 \ ... # In Blueprint spec: resources: limits: processes: 512 file_descriptors: 65536 memory: "4Gi" cpu: "2"
Application Order (Critical!)
# OpenShell applies security in this exact order: 1. Namespace entry # Enter isolated namespaces 2. Privilege drop # Drop capabilities, switch to unprivileged user 3. Landlock # Apply filesystem restrictions 4. Seccomp # Apply syscall filters # Why this order? # - Namespaces must be entered before dropping privs (some need CAP_SYS_ADMIN) # - Landlock before seccomp because Landlock setup needs syscalls # - Seccomp last because once applied, even setup syscalls are filtered
Upstream / Downstream Connections
- Agent syscalls (all system calls)
- Supervisor (applies rules at startup)
- Layer 2 (if file-related syscall allowed)
- Layer 1 (if network syscall allowed)
- Agent (EPERM/SIGSYS if blocked)
Source: Linux Capabilities, Seccomp
What It Does (High Level)
Authenticates connections to the control plane (Gateway). Ensures only authorized clients can manage sandboxes and access credentials.
Technical Implementation
Gateway Authentication Flow
Gateway Binding
# Gateway binds to loopback by default (security) NEMOCLAW_GATEWAY_BIND_ADDRESS=127.0.0.1 # This means: # - Only local processes can connect to Gateway # - Remote attackers cannot reach Gateway API # - Sandbox communicates via host network loopback
Auto-Pair Allowlist
# Only these clients can auto-pair (no manual approval needed) auto_pair_allowlist: - cli # openshell CLI - openclaw-cli # OpenClaw CLI - openclaw-control-ui # Control UI # CRITICAL: operator.admin scope NEVER auto-approved # Requires explicit manual approval for admin operations
Sandbox Token (JWT)
# From compute_driver.proto // Gateway-minted JWT identifying sandbox to gateway // Driver materializes via native secret mechanisms // NEVER echoed to public Sandbox proto string sandbox_token = 11 [(openshell.options.v1.secret) = true]; # Token contains: # - Sandbox ID # - Allowed scopes (which APIs sandbox can call) # - Expiration time # - Blueprint hash (for verification)
Permission Scopes
| Scope | Allows | Auto-Approve? |
|---|---|---|
sandbox.read |
Read sandbox state, logs | Yes |
sandbox.write |
Create, modify sandboxes | Yes (for allowlisted clients) |
inference.request |
Make inference requests | Yes (from sandbox) |
credential.read |
Access stored credentials | No (injected by Gateway only) |
operator.admin |
Admin operations | NEVER |
Upstream / Downstream Connections
- CLI/SDK (user commands)
- Supervisor (sandbox session)
- Layer 5 (inference requests)
- Credential Store (fetch secrets)
- Layer 1 (forward authenticated requests)
- Supervisor (policy updates)
Source: NVIDIA Security Best Practices
What It Does (High Level)
Intercepts all LLM inference requests, injects credentials from the host, and routes to the configured provider. Agent NEVER sees API keys.
Technical Implementation
Inference Routing Architecture
(Virtual Endpoint) participant PP as Policy Proxy
(on Host) participant GW as Gateway participant CS as Credential Store participant M as Model Provider
(Nemotron, OpenAI, etc.) A->>IL: POST /v1/chat/completions
(NO credentials) IL->>PP: Intercepted PP->>PP: Validate request format PP->>GW: Request credentials for provider GW->>CS: Fetch API key CS-->>GW: api_key=sk-xxx... GW-->>PP: Credentials PP->>PP: Inject Authorization header PP->>M: POST with credentials M-->>PP: Response PP->>PP: Strip any credential echoes PP-->>A: Response (clean) Note over A,M: Agent never sees sk-xxx...
How inference.local Works
# Inside the sandbox, there's a loopback metadata server # It responds to requests to "inference.local" # Agent code thinks it's calling a normal API: response = requests.post( "http://inference.local/v1/chat/completions", json={ "model": "nemotron-3-ultra", "messages": [{"role": "user", "content": "Hello"}] } # NO Authorization header needed! ) # What actually happens: # 1. DNS for inference.local resolves to 127.0.0.1 (loopback) # 2. Loopback server (part of Supervisor) receives request # 3. Request forwarded to Policy Proxy on host # 4. Policy Proxy contacts Gateway for credentials # 5. Credentials injected, request sent to real provider # 6. Response returned (credentials stripped)
Credential Placeholder Resolution
# Credentials are configured as placeholders on the host # ~/.openshell/credentials.yaml providers: nvidia: api_key: "${NVIDIA_API_KEY}" # Resolved from env endpoint: "https://api.nvidia.com/v1" openai: api_key: "${OPENAI_API_KEY}" endpoint: "https://api.openai.com/v1" anthropic: api_key: "${ANTHROPIC_API_KEY}" endpoint: "https://api.anthropic.com" # In Blueprint, specify which provider to use: spec: inference: provider: nvidia model: nemotron-3-ultra
Why This Architecture?
| Risk | How Inference Layer Prevents It |
|---|---|
| Agent leaks API key in logs | Agent never has the key |
| Agent sends key to attacker endpoint | Key only injected for allowed destinations |
| Agent stores key in memory/files | Key never in sandbox memory |
| Compromised agent exfiltrates key | Nothing to exfiltrate |
Configuring Inference Providers
# CLI commands # Set provider and model openshell inference set --provider nvidia --model nemotron-3-ultra # Configure credentials (stored on host, never in sandbox) openshell credentials set nvidia --api-key "$NVIDIA_API_KEY" # Use local inference (Ollama) openshell inference set --provider ollama --model llama3.3:70b --endpoint http://localhost:11434 # View current configuration openshell inference get
Upstream / Downstream Connections
- Agent inference requests (to inference.local)
- Supervisor (forwards to host)
- Layer 4 (Gateway for credentials)
- Layer 1 (Network policy for outbound)
- Model Provider (actual API call)
How All Layers Connect: Complete Request Flows
Flow 1: Agent Reads a File
Flow 2: Agent Makes Network Request
Flow 3: Agent Calls LLM
(with injected credentials) L1->>L1: Destination: api.nvidia.com L1->>L1: Policy: ROUTE (inference) L1->>M: API call M-->>L5: Response L5->>L5: Strip credential echoes L5-->>A: Clean response
Summary: Layer Responsibilities
| Layer | Technology | What It Controls | Hot-Reloadable? | Enforcement Level |
|---|---|---|---|---|
| 1. Network | CONNECT Proxy + OPA | Egress destinations, binary-scoped rules | Yes | Userspace (proxy) |
| 2. Filesystem | Landlock LSM | Path read/write/execute permissions | No | Kernel |
| 3. Process | Capabilities + Seccomp | Syscalls, privileges, resources | No | Kernel |
| 4. Gateway | JWT + Scopes | API access, credential access | Yes | Application |
| 5. Inference | Credential injection | LLM API access, key isolation | Yes | Application |
Knowledge Check
Primary Source Reading
Recommended Reading
-
Linux Landlock Documentation
https://docs.kernel.org/userspace-api/landlock.html
Official kernel documentation for Landlock LSM -
Linux Capabilities Manual
https://man7.org/linux/man-pages/man7/capabilities.7.html
All ~40 capabilities and what they allow -
Seccomp Manual
https://man7.org/linux/man-pages/man2/seccomp.2.html
Syscall filtering with BPF -
NVIDIA OpenShell Security
https://docs.nvidia.com/nemoclaw/user-guide/openclaw/security/
OpenShell-specific security implementation