Resilience, Disaster Recovery and Multi-Region Design

Start with two numbers

These come from the business, not from engineering. Tighter targets cost much more, so define them per workload, not for everything.

Failure scopes

Include bad deployments, configuration mistakes, deleted data, and dependency failures in your planning alongside AZ and regional disruptions. Multi-AZ does not recover logically corrupted data.

The four DR strategies

Building blocks by layer

Compute: everything as code (CloudFormation, CDK, Terraform) and container images replicated with ECR cross-region replication; AMIs copied to the DR region.

Data is the hard part:

Traffic: Route 53 health checks with failover or latency routing, or Global Accelerator for fast failover without waiting for DNS TTLs. For control of failover decisions during major events, use Application Recovery Controller routing controls.

Designing for failure inside a region

• Spread across at least 2 AZs; size so the loss of one AZ still handles load (N+1).
• Make services stateless; put state in managed, replicated stores.
• Prefer static stability: a system that keeps working without calling the control plane at failure time. Pre-provision capacity rather than depending on scaling during an AZ failure.
• Use timeouts, retries with exponential backoff and jitter, circuit breakers, and bulkheads so one slow dependency does not take down everything.
• Apply load shedding and graceful degradation: serve cached or reduced features rather than errors.
• Use health checks that reflect real readiness, but beware shallow checks that mark everything healthy and deep checks that cascade failures.

delay = random(0, min(cap, base * 2^attempt))      # exponential backoff with full jitter

Backups that survive disasters

• Use AWS Backup with plans for RDS, DynamoDB, EFS, EBS and S3.
• Copy backups to a separate account and region.
• Turn on Backup Vault Lock (WORM) to protect against ransomware and deletion.
• Enable S3 versioning, MFA delete where appropriate, and Object Lock for compliance data.

Test, do not assume

1. Game days: rehearse an AZ loss and a region failover.
2. AWS Fault Injection Service injects failures (stop instances, throttle APIs, add latency) in a controlled way.
3. Restore backups into a clean environment and time it against your RTO.
4. Document runbooks and keep them close to the on-call team; automate steps wherever possible.
5. Use the AWS Resilience Hub to assess an application against RTO and RPO targets.

Also run failback plans: returning to the primary region is another migration.

Recovery drill: prove a 30-minute target

Suppose the shop sets an RTO of 30 minutes and RPO of 5 minutes for orders. These are example business requirements, not service guarantees. Break recovery into detection, decision, data preparation, compute readiness, traffic cutover, and validation. The sum must fit the target.

A backup every 24 hours cannot normally satisfy five minutes of acceptable data loss by itself. A rapid traffic switch also cannot compensate for missing recent orders.

Prevent two writable primaries

Before promoting a recovery writer, understand how you will prevent the old writer from accepting conflicting traffic if it becomes reachable again. DNS cutover alone does not stop clients with cached addresses or long-lived connections. Define fencing, write ownership, and reconciliation deliberately.

For global tables, choose the intended consistency mode rather than assuming all configurations share the same conflict behavior. DynamoDB supports eventual and strong multi-Region consistency modes with different characteristics; see global-table consistency.

Restore versus replication

Replication can copy a mistaken deletion. Preserve a recovery path for logical corruption separately from infrastructure failure. Verify encryption keys, account permissions, images, configuration, and external integrations in the recovery environment.

Exercise: Rehearse restoration using sample data in an isolated environment. Record actual elapsed time and the latest recoverable record. Then plan failback: synchronize accepted writes, change ownership safely, validate again, and retire temporary capacity. A DR diagram without timed evidence remains a hypothesis.

Remember this

• Set RTO and RPO per workload; pick the cheapest DR strategy that meets them.
• Multi-AZ solves most availability needs; multi-region is for rare and critical cases.
• Data replication is the hardest part; use native cross-region features.
• Protect backups in a separate account with vault lock, and test restores and failovers regularly.