Architecture Center
Failure handling and recovery
Give operators a model for recovering from denials, retryable and permanent effect failures, and pending receipts, using the audit trail.
previewoperationsContract >=0.8 <1.0Reviewed 2026-07-20
Problem statement
Long-running, effectful workflows fail in distinct ways. Treating every failure the same causes double-applies or stuck work. Recovery depends on distinguishing transient from permanent.
When to use
- You run effectful workflows in production.
- You need to recover failed effects without support intervention.
When not to use
- Purely internal computation with no external effect or receipt.
Flow
Failure detectedClassify: denial / retryable / permanent / pendingSatisfy condition or retry with keyDead-letter if permanentVerify receipt + audit
Resource inventory
- Policy denial
- An auditable refusal with the unmet condition.
- Effect status
- Distinguishes retryable from permanent failure.
- Receipt
- Confirms final outcome; pending until finalized.
Failure modes
- Policy denied
- Read the unmet condition, satisfy it, retry via a legal path.
- Effect failed (retryable)
- Retry with the same idempotency key.
- Effect failed (permanent)
- Route to dead-letter; correct the cause; issue a new command.
- Receipt pending
- Wait for finalization; do not re-issue without the same key.
Security controls
- Denials and retries are auditable.
- Irreversible effects never shown complete before their receipt.
Deployment checklist
- Alerting on permanent failures and dead-letter growth.
- Runbooks for each failure class.
- Idempotency keys on all retried commands.
- Receipt and audit dashboards in place.
Related quickstart: Execute an approved effect →