Lesson 11: Harness Tuning & Policy Gateways for Domain-Specific Agents

This lesson covers how to improve agent accuracy through harness tuning (without changing model weights), how policy gateways provide deterministic enforcement, and how to design sandbox boundaries for multi-agent systems. We use a financial identity resolution system as our running example, grounded in NemoClaw's architecture.

Primary source: Tuning the Harness, Not the Model: A Nemotron 3 Ultra Playbook

1. What Is "The Harness"?

From the LangChain blog:

"The harness is the scaffolding around a model: system prompts, tool descriptions, and middleware around model/tool calls. Distinct from model weights."

The harness is everything EXCEPT the model weights. You improve accuracy by tuning these components, not by fine-tuning the model.

Why Tune the Harness Instead of Fine-Tuning?

2. The Three Harness Components

Component 1: AGENTS.md — Client/Domain-Specific Rules

From DeepAgents documentation:

"Configuration file for dcode containing user or project customizations to agent instructions. Located at ~/.deepagents/{agent}/AGENTS.md (user) or {project}/.deepagents/AGENTS.md (project)."

For a financial identity resolution system, the AGENTS.md contains client-specific rules:

# .deepagents/AGENTS.md (for Client A - Cash App)

## Identity Resolution Rules

### Field Extraction
- Customer name: Look in "Remitter Name" field, NOT "Account Holder"
- Bank account: 10 digits for this client (not 12)
- Invoice format: INV-XXXXX-YYYY (always this pattern)

### Matching Priority
1. Bank account + routing number = strongest signal
2. Corporate email domain (not gmail/yahoo/hotmail) = high confidence
3. Invoice reference exact match = high confidence
4. Name fuzzy match alone = low confidence, needs corroboration

### Client-Specific Aliases
- "ABC Corp" also appears as "ABC Corporation" and "A.B.C. Corp"
- Customer uses DBA names — check alias table

### Confidence Thresholds (for this client)
- Bank account exact match: 0.95
- Email + name match: 0.85
- Name fuzzy only (Jaro-Winkler > 0.9): 0.60
- Amount + date only: 0.40

### Escalation Triggers
- Amount exceeds $50,000
- Customer name contains "Trust" or "Estate"
- Multiple potential matches with similar confidence

This is injected into the agent's context. The LLM reads these rules and follows them. It's prompt engineering, but structured and versioned per client.

Component 2: Skills Library — Reusable Prompt-Behavior Modules

From the NemoClaw documentation:

"Skills Library: Prompt-behavior modules (field discovery, tool ordering, grounding)"

Skills are task-specific prompts that the Context Builder assembles:

Example skill file:

# skills/field_discovery.md

## Purpose
Extract structured fields from unstructured payment documents.

## Process
1. Identify document type (bank statement, remittance, invoice, check)
2. Extract fields based on document type:
   - Customer name (look for variations: payer, remitter, account holder)
   - Bank account number (may be partial, may include routing)
   - Email address (if present)
   - Reference numbers (invoice, PO, customer ref)
   - Amount and date

3. For each extracted field:
   - Quote the exact text from the document
   - Note the location (page, section, field name)
   - Flag uncertainty if ambiguous

## Output Format
Return extracted fields with evidence:
{
  "customer_name": {"value": "...", "source": "Field 'Remitter' on page 1"},
  "bank_account": {"value": "...", "source": "Account field, partial: last 4 digits"},
  ...
}

Key distinction: Skills define HOW to do something. AGENTS.md defines WHAT rules apply for this client.

Component 3: Tool Registry — What the Agent Can Call

From the NemoClaw documentation:

"Tool Registry: Read-only tools available to the planner & subagents (configuration)"

Tool descriptions guide the LLM on WHEN and HOW to use each tool:

@tool
def lookup_customer_by_account(
    account_number: str,
    routing_number: str | None = None
) -> list[CustomerMatch]:
    """
    Exact match on bank account number.

    Use this FIRST when you have a bank account number.
    Returns high-confidence matches if found.

    Args:
        account_number: Full or partial account number (min 4 digits)
        routing_number: Optional routing/sort code for disambiguation

    Returns:
        List of matching customers with confidence scores
    """
    ...

@tool
def fuzzy_match_customer_name(
    name: str,
    threshold: float = 0.8
) -> list[CustomerMatch]:
    """
    Fuzzy match on customer name using Jaro-Winkler similarity.

    Use ONLY when exact matches fail.
    Results below 0.8 similarity should be treated with caution.

    Args:
        name: Customer name to search
        threshold: Minimum similarity score (default 0.8)

    Returns:
        List of potential matches with similarity scores
    """
    ...

3. How the Context Builder Assembles Everything

The Context Builder creates a prompt like this:

[System] You are an identity resolution agent.

[Skill: field_discovery.md content]
- Extract fields from documents...
- Quote exact text as evidence...

[Skill: matching.md content]
- Use exact match tools FIRST...
- Fall back to fuzzy only when needed...

[Client Rules from AGENTS.md]
- Bank account is 10 digits for this client
- Invoice format: INV-XXXXX-YYYY
- "ABC Corp" aliases: "ABC Corporation", "A.B.C. Corp"
- Confidence threshold: 0.85 for auto-accept

[Tools available]
- lookup_customer_by_account: Use FIRST when you have account number
- fuzzy_match_customer_name: Use ONLY when exact fails

[Document to process]
{actual document content}

Result: The same base model behaves differently per client, per domain, without fine-tuning.

4. Two Types of Rules: Prompt-Level vs Enforcement-Level

This is a critical distinction.

Type A: Prompt-Level Rules (Inside the Harness)

These are in AGENTS.md. The LLM reads them and is ASKED to follow them.

# AGENTS.md
- If confidence is below 0.7, flag for human review
- Always cite evidence from the source document

Problem: The LLM might follow these. Or it might hallucinate. It's advisory.

Type B: Enforcement-Level Rules (Policy Gateway)

These are evaluated AFTER the LLM outputs, deterministically by code (not LLM).

# Policy Gateway (deterministic, no LLM)
def evaluate_policy(result: IdentityMatch) -> Decision:
    if result.confidence >= 0.85:
        return Decision.ACCEPT
    elif result.confidence >= 0.60:
        return Decision.HUMAN_REVIEW
    else:
        return Decision.REJECT

    # Additional rules
    if result.amount > 50000:
        return Decision.HUMAN_REVIEW

    if result.customer_id in watchlist:
        return Decision.REJECT

Key point: The LLM cannot bypass this. It doesn't matter what the LLM says — the Policy Gateway evaluates the structured output and makes the final decision.

Comparison

5. Why Both Exist: Defense in Depth

6. Mapping to NemoClaw's Five Protection Layers

NemoClaw provides runtime enforcement. Here's how the harness and policy gateway map to NemoClaw's architecture:

What Each Layer Protects Against

7. Policy Gateway Implementation Options

Option A: Plain Python

from dataclasses import dataclass
from enum import Enum

class Decision(Enum):
    ACCEPT = "accept"
    REVIEW = "review"
    REJECT = "reject"

@dataclass
class PolicyResult:
    decision: Decision
    reason: str
    escalation_queue: str | None = None

def policy_gateway(match_result: dict, client_rules: dict) -> PolicyResult:
    """Deterministic policy evaluation. No LLM."""

    confidence = match_result.get("confidence", 0)
    amount = match_result.get("amount", 0)
    customer_id = match_result.get("customer_id")

    # Rule 1: Confidence thresholds
    accept_threshold = client_rules.get("accept_threshold", 0.85)
    review_threshold = client_rules.get("review_threshold", 0.60)

    if confidence < review_threshold:
        return PolicyResult(Decision.REJECT, "Confidence below threshold")

    # Rule 2: Amount escalation
    amount_threshold = client_rules.get("amount_threshold", 50000)
    if amount > amount_threshold:
        return PolicyResult(
            Decision.REVIEW,
            f"Amount ${amount} exceeds threshold",
            escalation_queue="high_value"
        )

    # Rule 3: Watchlist
    if customer_id in load_watchlist():
        return PolicyResult(Decision.REJECT, "Watchlist match")

    # Rule 4: Evidence completeness
    if not match_result.get("evidence"):
        return PolicyResult(Decision.REVIEW, "Missing evidence citation")

    # Accept
    if confidence >= accept_threshold:
        return PolicyResult(Decision.ACCEPT, "High confidence match")

    # Review
    return PolicyResult(Decision.REVIEW, "Moderate confidence, needs review")

Option B: OPA (Open Policy Agent) — Same Pattern as OpenShell

OpenShell uses OPA for network/filesystem rules. You can use OPA for business logic too:

# business_policy.rego
package identity_resolution

default decision = "review"

decision = "accept" {
    input.confidence >= 0.85
    input.amount <= 50000
    not watchlist_check(input.customer_id)
}

decision = "reject" {
    input.confidence < 0.50
}

decision = "reject" {
    watchlist_check(input.customer_id)
}

watchlist_check(id) {
    data.watchlist[_] == id
}

The key point: whether Python or OPA, it's deterministic, not LLM-based.

8. Sandbox Boundary Decisions for Multi-Agent Systems

When you have multiple agents, you need to decide: shared sandbox or separate sandboxes?

Decision Framework

Case Study: Identity Resolution Pipeline

Why this split?

1. All LLM agents share a sandbox: They all work on the same document, same DB read access, same output format. No reason to isolate them from each other.
2. Policy Gateway outside sandbox: If an LLM is compromised via prompt injection, it cannot influence the final accept/reject decision. The gate is deterministic.

9. Nested DeepAgents and Policy Inheritance

From DeepAgents documentation:

"The harness includes a task tool enabling the main agent to spawn ephemeral subagents for isolated work."

DeepAgents CAN spawn subagents. The question is: what sandbox do they get?

Policy Inheritance Rules

# Parent agent with balanced policy
parent = create_deep_agent(
    blueprint="balanced",  # network: [pypi, github, npm]
    ...
)

# Subagent can only be same or stricter
child = parent.spawn_subagent(
    prompt="...",
    policy_override={
        "network": ["pypi.org"],  # OK: subset of parent
        # "network": ["pypi.org", "evil.com"]  # BLOCKED: exceeds parent
    }
)

10. Complete Implementation Flow

11. Practical Implementation Steps

Step 1: Create Bare-Bone Agents (Pre-Created, Not Spawned)

from deepagents import create_deep_agent

# Create agents at startup (not spawned dynamically)
field_discovery_agent = create_deep_agent(
    model="anthropic:claude-sonnet-4-20250514",
    tools=[],  # No tools, just extraction
    system_prompt=load_skill("field_discovery.md")
)

matching_agent = create_deep_agent(
    model="anthropic:claude-sonnet-4-20250514",
    tools=[
        lookup_customer_by_account,
        lookup_customer_by_email,
        fuzzy_match_customer_name,
    ],
    system_prompt=load_skill("matching.md")
)

Step 2: Build Context Builder

def build_context(client_id: str, document: str) -> str:
    """Context Builder: assemble prompt from components."""

    # Load client-specific rules
    agents_md = load_file(f"clients/{client_id}/AGENTS.md")

    # Load relevant skills
    skills = [
        load_skill("field_discovery.md"),
        load_skill("matching.md"),
        load_skill("confidence_scoring.md"),
    ]

    # Assemble context
    return f"""
{chr(10).join(skills)}

## Client-Specific Rules
{agents_md}

## Document to Process
{document}
"""

Step 3: Add Policy Gateway

(See Section 7 for full implementation)

Step 4: Compose the Pipeline

async def process_identity_resolution(
    client_id: str,
    document: str
) -> dict:
    """Main pipeline: agents → policy gateway."""

    # Load client config
    client_rules = load_client_rules(client_id)

    # Build context with client-specific AGENTS.md
    context = build_context(client_id, document)

    # Agent 1: Field Discovery
    fields = await field_discovery_agent.ainvoke({
        "messages": [{"role": "user", "content": context}]
    })

    # Agent 2: Matching (with extracted fields)
    match_result = await matching_agent.ainvoke({
        "messages": [{"role": "user", "content": f"""
Given these extracted fields:
{fields}

Find the matching customer using the available tools.
Return: customer_id, evidence (with citations), confidence score.
"""}]
    })

    # Policy Gateway: Deterministic decision
    policy_result = policy_gateway(match_result, client_rules)

    return {
        "fields": fields,
        "match": match_result,
        "decision": policy_result.decision.value,
        "reason": policy_result.reason,
        "queue": policy_result.escalation_queue,
    }

Step 5: Later — Wrap with NemoClaw/OpenShell

# blueprint.yaml
name: identity-resolution-agent
version: "1.0"

sandbox:
  type: container
  image: python:3.12-slim

filesystem:
  permissions:
    - operations: ["read"]
      paths: ["clients/*/AGENTS.md", "skills/*.md"]
      mode: allow
    - operations: ["read", "write"]
      paths: ["/workspace/output/**"]
      mode: allow

network:
  allowed_hosts:
    - "database.internal:5432"  # Your single database
  blocked_hosts:
    - "*"  # Block everything else

inference:
  providers:
    - anthropic
  credential_injection: true  # API keys stay on host

The agents stay the same. OpenShell adds:

• Network isolation (only database access)
• Filesystem isolation (only config and output)
• Credential isolation (API keys injected, not in sandbox)

Knowledge Check

Summary