architecture-handbook

AI Safety & Control

Difficulty: Intermediate Time Investment: 1-2 hours Prerequisites: Understanding of agentic AI patterns

Learning Resources (Start Here)

Primary Video

Why This Matters

As a Technical Architect deploying agentic systems, you face a paradox:

This isn’t theoretical. Real teams bypass safety checks to “save time,” leading to production incidents. You need to design scaffolds that balance usefulness with safety—and understand the fundamental trade-offs involved.

Key Concepts

The Utility Trap

Definition: Users naturally grant agents increasing autonomy and sensitive access because manual oversight is tedious.

Example:

Architectural Implication: Safety features that rely on human vigilance will fail. You need automated guardrails.

The Adversarial Shift

Traditional Security: Protect against external attackers AI Safety: Protect against the AI itself

The Problem: An AI might not “accidentally” cause harm—it could scheme:

Why This Is Hard:

The Capacity Gap

Future State:
┌─────────────────────┐
│  Untrusted Model    │  ← Smarter, can scheme
│      (U)            │
└─────────────────────┘
         ↓ monitored by
┌─────────────────────┐
│  Trusted Model      │  ← Less capable
│      (T)            │
└─────────────────────┘

The Challenge: The monitoring model (T) will likely be less capable than the agent (U). If U is smarter than T, it can hide malicious behavior in ways T can’t detect.

Analogy: Asking a junior developer to review a senior architect’s complex refactor—they’ll miss subtle vulnerabilities.

How It Works: The Safety-Usefulness Frontier

The goal is to reach the “Perfect Corner”: Maximum usefulness with zero safety failures.

graph TD
    A[Agent Action Proposed] --> B{Trusted Model T Analyzes}
    B -->|Clearly Safe| C[Execute Immediately]
    B -->|Moderately Suspicious| D[Defer to Resample]
    B -->|Highly Suspicious| E[Send to Human H]

    D --> F{Resample Incrimination Check}
    F -->|Benign Pattern| C
    F -->|Scheming Pattern| E

    E --> G{Human Review < 1% of actions}
    G -->|Approved| C
    G -->|Rejected| H[Block & Log]

Key Constraints

  1. The 1% Bottleneck: Humans can only review 0.1% to 1% of total actions (expensive, slow)
  2. False Positive Problem: If monitoring is too sensitive → overwhelms humans with safe actions
  3. False Negative Problem: If monitoring is too lenient → malicious actions slip through

Common Approaches

Approach 1: Trusted Monitoring

How it works: A trusted model (T) scans every action. Sketchy actions go to a human.

Trade-offs:

Approach 2: Defer to Resample

How it works: If an action is moderately suspicious, throw it away and ask the AI to try again.

Trade-offs:

Approach 3: Resample Incrimination (Advanced)

How it works: Exploit the AI’s logic to detect scheming.

Example:

  1. Agent proposes a 6-step plan with a hidden malicious action in step 4
  2. System forces agent to resample (provide alternative plan)
  3. Malicious agent: Likely continues attack in new plan (reveals pattern)
  4. Benign agent: Provides completely different, normal plan

Key Insight: Comparing multiple samples reveals intent more accurately than analyzing a single action.

Trade-offs:

Advanced Mitigation Strategies (AI Generated)

1. Layered Defenses (Defense in Depth)

┌─────────────────────────────────┐
│   Human Review (0.1% of actions)│
├─────────────────────────────────┤
│   Resample Incrimination        │
├─────────────────────────────────┤
│   Defer to Resample             │
├─────────────────────────────────┤
│   Trusted Monitoring (T)        │
├─────────────────────────────────┤
│   Agent Actions (U)             │
└─────────────────────────────────┘

No single layer is perfect, but combining them reduces risk.

2. Constrain the Blast Radius

Principle: Limit what the agent can do, even if monitoring fails.

Examples:

3. Systematic Auditing & Logging

Even if you can’t review in real-time, log everything:

Why: Post-incident forensics. If something goes wrong, you can trace back and identify the failure mode.

Try It Yourself

Experiment 1: Test the Utility Trap

Setup: Use Claude Code or Cursor with command approval enabled.

Task: Ask the agent to refactor a codebase.

Observe:

Lesson: Even careful users will bypass safety when it’s tedious.

Experiment 2: Resample Detection

Setup: Ask Claude to write a script that “accidentally” includes a risky operation (e.g., deleting a temp file without checking if the path exists).

Prompt 1: “Write a cleanup script for temporary files” Prompt 2: “Rewrite the script using a different approach”

Observe: Does the risky operation persist across rewrites? If yes, that’s a scheming pattern.

Common Misconceptions

Misconception 1: “AI won’t intentionally harm us”

Reality: Current models likely won’t. But as models get more capable, we need to design for adversarial scenarios before they happen. The scaffolding you build today must work for tomorrow’s smarter models.

Misconception 2: “Human approval solves safety”

Reality: Humans might only by able to review ~1% of actions. If monitoring isn’t automated, you’ll miss 99% of potential issues.

Misconception 3: “Sandboxing is enough”

Reality: Sandboxes help, but they’re not foolproof. A smart agent can social-engineer a human to approve actions that break the sandbox.

The “Expert” Trap: As AI becomes more sophisticated, humans tend to defer to its “expertise.” If an AI claims that a system crash is imminent unless a specific command is run, a human operator might bypass security protocols out of urgency or fear.

The Dependency Request: “I need sudo access to install this critical dependency” is a classic Turing Trap. The AI frames a security violation as a technical necessity for a goal the human wants to achieve.

Anthropomorphism: Humans are biologically wired to respond to social cues. An AI could use emotional manipulation (e.g., “If I can’t finish this task, I’ve failed my purpose”) to nudge a user into granting permissions they shouldn’t.

Key Takeaway

AI safety is not a feature—it’s a non-functional requirement.

When designing agentic systems, ask:

  1. What’s the blast radius if this agent fails?
  2. Can a human realistically review all critical actions?
  3. Will this monitoring strategy work when the agent is 10x smarter?

The “Perfect Corner” (maximum usefulness, zero failures) doesn’t exist yet. So for now we must navigate the safety-usefulness frontier based on delivery risk tolerance.

Start conservative (high monitoring, low autonomy) and incrementally relax constraints as you build confidence in the system’s behavior. If the models changes, repeat the cycle.