Watch for Emergent Failures

Complete Hub-and-Spoke vs Peer-to-Peer first; this lesson builds on architecture pattern recognition so you know which seams to watch for feedback loops and conflicting writes.

Here is the before and after: Your generator agent produces a task list and writes it to the shared queue. Your reviewer agent reads the queue, decides the tasks look incomplete, and asks the generator to add more. The generator adds tasks. The reviewer reads again and asks for more. By the time your token cap fires, you have spent $40 and the queue has 200 tasks, none of them completed. Neither agent did anything wrong. The failure came from their interaction.

This is an emergent failure. No single agent causes it. No single agent can fix it. It only appears when agents interact at speed.

Three patterns account for most emergent failures. Feedback loops: Agent A's output triggers Agent B, which triggers Agent A again. In a multi-agent system this can double cost every cycle before any timeout fires. Conflicting writes: two agents modify the same resource from separate stale reads. The result compiles, passes its own tests, and breaks the combined system. Resource starvation: one agent type monopolizes API quota, queue capacity, or context window, and other agents queue indefinitely or time out.

The mitigation toolkit is not complex, but it must be wired before the first production run. Circuit breakers cap iterations or cost at the system level, above any individual agent. Read-before-write checks verify current state before committing a change. Dead-letter queues hold failed tasks without blocking the fleet. These are infrastructure decisions, not prompt engineering. Prompting your way out of a systemic coordination failure is a local fix that breaks elsewhere.

Note: emergent failure modes at this scale are an active area of engineering. The patterns above are well-observed. General solutions that work reliably across all agent configurations are still being developed.

Paste this into Claude:

I want to identify emergent failure risks in my agent setup. Here is how my agents interact: [describe your multi-agent flow: what does each agent produce, what does it read from, and what shared resources do agents access]. Help me: (1) Identify any feedback loop risk: is there a path where Agent A's output could retrigger Agent A directly or through another agent? Draw the cycle if one exists. (2) Identify any conflicting write risk: which resources do two or more agents write to, and what happens if they both read the same stale version? (3) Identify any resource starvation risk: is there a shared resource (API quota, queue, context window) that one agent type could monopolize? For each risk found, propose a specific mitigation: a circuit breaker, a read-before-write check, or a dead-letter queue.

What good looks like:

• You checked for all three failure patterns, not just the most obvious one
• For any feedback loop found, you drew the cycle as a text diagram showing the trigger path
• For any conflicting write found, you named the specific shared resource and the stale-read scenario
• Each mitigation is specific and wirable, not 'add better error handling'
• You identified at least one risk that exists in your current setup, even if it has never fired

What a good response looks like:

Emergent failure audit for a generator/reviewer pipeline:

Feedback loop risk: FOUND
Cycle: generator writes tasks to queue -> reviewer reads queue -> reviewer calls generator.addTasks() if count < threshold -> generator writes more tasks -> reviewer reads again
Diagram:
  generator --> [queue] --> reviewer
     ^                         |
     |_____ addTasks() _______|
Mitigation: circuit breaker at the supervisor level. After 3 generator calls in a single run, trip the breaker and route to dead-letter. Do not allow the reviewer to call addTasks() more than 3 times per run.

Conflicting write risk: FOUND
Shared resource: task-state.json
Scenario: reviewer reads task-state.json at t=0. Generator writes a new task at t=1. Reviewer writes a status update at t=2 based on its stale read, overwriting the generator's new task.
Mitigation: read-before-write check. Reviewer must re-read task-state.json immediately before writing, not use its cached read.

Resource starvation risk: FOUND
Bottleneck: API rate limit (10 calls/minute). Generator can consume all 10 slots in a burst, leaving reviewer queued for up to 60 seconds per cycle.
Mitigation: rate-limit token bucket at the supervisor level, shared across all agents. No single agent gets more than 5 calls per 10-second window.

Paste this into Claude:

Design a circuit breaker for one agent flow you identified as having a feedback loop or runaway cost risk. The circuit breaker should: (1) Define the metric it watches (iterations, cost, tokens, wall-clock time). (2) Define the threshold at which it trips. (3) Define what happens when it trips: pause, abort, alert, or route to a dead-letter queue. (4) Define how it resets: manual only, or automatic after a cooldown. Then write the instructions you would add to your supervisor prompt or CLAUDE.md to wire this circuit breaker into every run of that flow.

What good looks like:

• Your circuit breaker watches a specific measurable metric, not 'if something goes wrong'
• The threshold is a number, not a description
• The trip action is one of: pause, abort, alert, or dead-letter, with a reason for your choice
• The reset condition is explicit: who resets it and when
• You wrote the actual instruction text you would add to wire it, not just described it

What a good response looks like:

Circuit breaker for the generator/reviewer feedback loop:

Metric watched: number of times reviewer has called generator.addTasks() in the current run
Threshold: 3 calls
Trip action: ABORT with alert. Reason: pausing would resume the loop after cooldown. Aborting forces manual review before the next run. The failure mode is runaway cost, not a recoverable transient error.
Reset condition: manual only. A human must review the dead-letter queue, understand why the reviewer kept requesting more tasks, and adjust the reviewer's completion threshold before re-running.

Supervisor prompt instruction to add:
'CIRCUIT BREAKER: Track how many times the reviewer has called addTasks() in this run. Store this count as addTasks_call_count. If addTasks_call_count reaches 3, immediately STOP the run. Write the current queue state to dead-letter/[run_id].json. Send this message: CIRCUIT BREAKER TRIPPED: reviewer called addTasks() 3 times without completing tasks. Run aborted. Review dead-letter/[run_id].json before re-running. Do not attempt to complete remaining tasks or call any more agents.'

This instruction goes in the supervisor CLAUDE.md under ## Circuit Breakers. It fires on every run of this flow automatically.