Building Resilient LLM Chains in Production

Everyone talks about chaining LLMs. Few talk about what happens when chain link #3 hallucinates a JSON key that crashes link #4, at 3 AM, with 10,000 requests in the queue. This is what production LLM infrastructure actually looks like.

The Failure Taxonomy

After running multi-agent LLM systems in production for 8 months, I've categorized failures into 4 types. Understanding these shapes your entire defensive architecture:

Defense Layer 1: Structural Validation

Never trust an LLM's output structure. Every chain link should validate its input and output against a strict schema. We use Pydantic models with fallback defaults:

from pydantic import BaseModel, Field, validator
from typing import Optional

class AnalysisResult(BaseModel):
    severity: str = Field(default="unknown")
    root_cause: str = Field(default="unidentified")
    confidence: float = Field(default=0.0, ge=0.0, le=1.0)
    suggested_action: Optional[str] = None

    @validator('severity')
    def validate_severity(cls, v):
        allowed = ['critical', 'high', 'medium', 'low', 'unknown']
        return v.lower() if v.lower() in allowed else 'unknown'

def safe_parse(raw_output: str) -> AnalysisResult:
    """Parse LLM output with graceful degradation."""
    try:
        return AnalysisResult.model_validate_json(raw_output)
    except Exception:
        # Extract what we can, default the rest
        return AnalysisResult()

Defense Layer 2: Circuit Breakers

Borrowed from microservices architecture, circuit breakers prevent cascade failures. If an LLM endpoint fails 3 times in 60 seconds, we trip the breaker and route to a fallback (smaller model, cached response, or human escalation).

class LLMCircuitBreaker:
    def __init__(self, failure_threshold=3, reset_timeout=60):
        self.failures = 0
        self.threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.last_failure = 0
        self.state = "CLOSED"  # CLOSED -> OPEN -> HALF_OPEN

    async def call(self, func, *args, **kwargs):
        if self.state == "OPEN":
            if time.time() - self.last_failure > self.reset_timeout:
                self.state = "HALF_OPEN"
            else:
                return await self.fallback(*args, **kwargs)

        try:
            result = await asyncio.wait_for(func(*args, **kwargs), timeout=10)
            self.failures = 0
            self.state = "CLOSED"
            return result
        except (asyncio.TimeoutError, Exception):
            self.failures += 1
            self.last_failure = time.time()
            if self.failures >= self.threshold:
                self.state = "OPEN"
            return await self.fallback(*args, **kwargs)

Defense Layer 3: Context Windowing

Instead of passing the entire chain history forward, we implemented a sliding context window with semantic compression. At each chain step, we summarize the previous context to fit within 30% of the model's token budget, preserving the remaining 70% for the current task.

The compression itself is done by a fast, small model (Haiku) — it costs pennies and saves dollars by preventing the expensive model from receiving degraded prompts.

Results After 6 Months

The key insight: treat LLM chains like distributed microservices, not like function calls. Every link can fail. Design for it.