RFC: AWS Resiliency Plugin

[nirmal84]

Is this related to an existing feature request or issue?

No response

Summary

The AWS Resiliency Plugin gives your AI agent a failure-mode-first approach to architecture reviews. Share your IaC or application code, and the agent reviews it across 7 resiliency domains — surfacing single points of failure, missing failover configurations, and application-level gaps that cause production incidents.

Use case

The AWS Resiliency Plugin gives your AI agent a failure-mode-first approach to architecture reviews. Share your IaC or application code, and the agent reviews it across 7 resiliency domains — surfacing single points of failure, missing failover configurations, and application-level gaps that cause production incidents.

Target users: Cloud Engineers, SREs, Platform Engineers, and Cloud Architects preparing production deployments, operational readiness reviews (ORRs), game-day exercises, or Well-Architected Reviews (WARs).

Key use cases:

Pre-production hardening: Review a CDK/Terraform stack before it goes to production — find the missing Multi-AZ, the absent DLQ, the Lambda timeout that will silently drop requests
ORR preparation: Structured review across all seven resiliency domains to validate readiness criteria before launch
Incident post-mortem: Share the IaC of a service that just had an outage — the agent finds the architectural gap that caused it
Game-day planning: Identify blast radius and failure modes to design meaningful chaos experiments
WAR prep: Map findings to Well-Architected REL pillar questions with evidence from the actual code

###Before / After user experience:

Before (without the plugin):

User: Is this CDK stack resilient?

Agent: Here are some general resiliency best practices:
1. Use Multi-AZ deployments
2. Implement retry logic
3. Consider disaster recovery
[...10 generic bullet points with no reference to the actual code...]
After (with aws-resiliency):

User: Is this CDK stack resilient?
[pastes CDK TypeScript stack]

Agent: Two-layer resiliency review complete across 7 domains.

   | Domain      | Finding                              | Severity | Blast Radius              | RTO Impact  | Fix                                    |
   |-------------|--------------------------------------|----------|---------------------------|-------------|----------------------------------------|
   | COMPUTE     | ECS desiredCount=1, no multi-AZ      | Critical | Full service outage        | ~5 min      | desiredCount≥2 + AZ placement strategy |
   | DATA        | RDS missing multiAz                  | Critical | Read+write outage          | 10–30 min   | multiAz: true                          |
   | COMPUTE     | Lambda timeout=3s on payments fn     | High     | Silent 504s to ALB         | Per-request | timeout: 29 (ALB max)                  |
   | MESSAGING   | SQS consumer missing DLQ             | High     | Message loss on failure    | N/A         | deadLetterQueue, maxReceiveCount: 3    |
   | NETWORKING  | Single NAT Gateway (one AZ)          | Medium   | Outbound loss if AZ fails  | ~5 min      | One NAT GW per AZ                      |

Layer 2 (application code): RDS client is correctly initialised outside the handler , but connection pool maxConnections=10 will exhaust during the RDS failover reconnection storm. Recommend maxConnections=2 with exponential backoff on connect. Top risk: RDS Multi-AZ + ECS single-task together mean any AZ failure causes a full outage with manual recovery. Fix both before production. REL 6 (REL_FAULT_ISOLATION), REL 9 (REL_MITIGATE_BULK_FAILURES).

Proposal

Plugin structure

plugins/aws-resiliency/
aws-resiliency-plugin/
├── .claude-plugin/
│   └── plugin.json          
├── skills/
│   ├── SKILL.md             ← orchestrator (loads references on demand)
│   └── references/
│       ├── compute-resiliency.md
│       ├── data-resiliency.md
│       ├── networking-resiliency.md
│       ├── storage-resiliency.md
│       ├── messaging-resiliency.md
│       ├── observability-resiliency.md
│       ├── multi-region-dr.md
│       ├── service-failure-modes.md
│       └── well-architected-reliability.md
├── mcp.json                 ← 6 AWS MCP server configs
└── README.md

MCP server dependencies

Server	Type	Purpose	Required?
awslabs.aws-documentation-mcp-server	stdio	Well-Architected REL pillar references, AWS service documentation, service limits, SLA definitions	Required
awslabs.aws-iac-mcp-server	stdio	CDK + CloudFormation resource schema lookups, cfn-lint validation, cfn-guard compliance checks, IaC best-practice patterns	When CDK/CFn detected
awslabs.terraform-mcp-server	stdio	Terraform provider registry, module validation, HCL resource schema checks	When Terraform HCL detected
awslabs.aws-knowledge-mcp-server	stdio	Architecture best practices, cross-service integration patterns, service-specific failure behaviour FAQs	On-demand for service behaviour lookups

<style> </style>

awslabs.aws-documentation-mcp-server is the only required server — it provides Well-Architected REL pillar grounding for all findings. The IaC and Terraform servers are conditionally invoked based on detected input format. The knowledge server is invoked when the agent needs to confirm specific service behaviours (e.g., exact RDS DNS failover propagation window, DynamoDB eventual consistency semantics under partition).

Defaults

Setting	Default	How to Override
IaC languages	CDK (TypeScript, Python, Java, Go), Terraform HCL, CloudFormation YAML/JSON, SAM YAML	Auto-detected from syntax and file extension
Severity: Critical	Single point of failure, no automated recovery, estimated RTO > 4 hours	State: "my RTO requirement is X"
Severity: High	Significant reliability gap, degraded service, RTO 1–4 hours
Severity: Medium	Well-Architected best-practice violation, limited blast radius, RTO < 1 hour
Severity: Low	Improvement opportunity, no immediate blast radius, affects future scale
Default RTO threshold	4 hours (Critical if exceeded)	State target RTO explicitly
Default RPO threshold	1 hour (Critical if exceeded)	State target RPO explicitly
Review scope	All IaC files in the current directory + application code shared in the conversation	Specify: "review only the data layer"
Output format	Markdown findings table (Domain / Finding / Severity / Blast Radius / RTO Impact / Fix / WAF Ref) + Layer 2 narrative + top-risk summary
Fix format	Corrected code/config snippet in the same IaC language as the input
WAF references	Mapped to Well-Architected REL pillar question titles (stable across versions)

---

Dependencies and Integrations

MCP dependencies (all from AWS Labs):

• awslabs.aws-documentation-mcp-server — Well-Architected REL pillar, service docs, SLAs
• awslabs.aws-iac-mcp-server — CDK + CloudFormation validation, cfn-lint, cfn-guard
• awslabs.terraform-mcp-server — Terraform provider/module validation
• awslabs.aws-knowledge-mcp-server — Architecture patterns, service failure behaviour

Integration with existing plugins:

• Complements deploy-on-aws: deploy-on-aws generates dev-sized IaC; aws-resiliency reviews it for production hardening. Recommended workflow: generate → review → harden → redeploy.
• Complements aws-observability (RFC RFC: Add AWS Observability plugin #67): Resiliency review surfaces missing CloudWatch alarms, absent X-Ray tracing, and health check gaps — findings that feed directly into aws-observability workflows.

Reference implementation: A working version of the skill and reference files is available at https://github.com/nirmal84/aws-resiliency-plugin

 

Potential Challenges

• IaC language detection in monorepos: Multi-file CDK projects with multiple stacks require heuristic detection. Mitigation: SKILL.md instructs the agent to ask the user to identify the main stack file if auto-detection is ambiguous.
• Two-layer correlation: Connecting an IaC finding (RDS Multi-AZ enabled) with an application finding (connection pool not handling the 60s DNS failover window) requires both layers to be present. Mitigation: SKILL.md instructs the agent to flag missing layers and deliver a partial review with explicit caveats.
• WAF question ID currency: REL pillar question IDs change across framework versions. Mitigation: References use stable question titles rather than volatile IDs; aws-documentation-mcp-server fetches current content when available.
• Reference file size: Domain reference files covering IaC checks, application code checks, and failure modes will approach the 100-line guideline. Mitigation: Files load only when the relevant domain is detected — a Lambda-only review never loads multi-region-dr.md. SKILL.md stays under 200 lines.

[krokoko]

Thank you for this submission @nirmal84 ! A few questions/requests:

Could you please:

1. Write distinct Summary and Use Case sections (don't duplicate). Include a before/after user experience.
2. Propose a skill structure - name the skill(s), write draft trigger phrases, propose a directory tree with reference files for the 7 domains.
3. Define defaults - IaC language support, severity criteria, output format, RTO/RPO thresholds, scope.
4. List MCP dependencies - does this need awsknowledge for Well-Architected references? Any other MCP servers?
5. Define "Out of scope" properly - draw the line against security, operational, and cost concerns. Address overlap with deploy-on-aws.
6. Add error scenarios - what happens with unrecognized IaC, no resiliency-relevant resources, etc.
7. Model after the Observability RFC (RFC: Add AWS Observability plugin #67) - that RFC is the highest quality example in the repo, with a complete directory structure, MCP server table, skill design, and user experience description.

Thanks !

[nirmal84]

Hi @krokoko I have updated the original submission. Thanks for the feedback.

[krokoko]

Thanks @nirmal84 ! My 2 last questions/comments:

1. Regarding what is out of scope, I think it is important to define it because the boundaries are ambiguous:

• Security vs. Resiliency: Is "missing encryption at rest" a resiliency finding or a security finding? Is "no WAF on the ALB" resiliency or security? The RFC's 7 domains don't explicitly address this boundary.
• Cost vs. Resiliency: Multi-AZ everything is resilient but expensive. Does the plugin consider cost trade-offs, or does it always recommend the most resilient option regardless of cost?
• Observability: The RFC mentions "Observability" as one of the 7 domains, but there's a separate aws-observability plugin (RFC RFC: Add AWS Observability plugin #67). Where does one end and the other begin?

For instance, a clean boundary would be: "This plugin identifies resiliency gaps like missing redundancy, absent failover, unhandled failure modes. It does not assess security posture (encryption, IAM, network ACLs), estimate cost impact of remediation, or configure monitoring/alerting. It surfaces missing alarms as a resiliency finding but does not create them."

2. The 2 layers review is great. Layer 1 seems easier, layer 2 is much harder. The agent will need to correlate infra and app code findings and have some deep knowledge about aws services failure behavior. I think you are encoding the knowledge in service-failure-modes.md, but if this is incomplete or wrong, the layers 2 findings will be wrong. Should we add some confidence levels for findings of layers 2 ?

[nirmal84]

Thanks again @krokoko , for point 2 above yes we can add confidence levels for sure.

For the Out of scope question

Scope will be determined by : Does this affect whether the system recovers from failure? Security, cost, and performance are out of scope unless they directly impact recovery

Security vs. Resiliency:
The plugin treats security findings only where they have a direct resiliency impact:

Missing encryption at rest → out of scope (data confidentiality, not recovery)
No WAF on the ALB → out of scope (attack surface, not failure recovery)
Backups in the same AWS account as production → in scope (account compromise = no recovery path)No termination protection on RDS → in scope (accidental deletion = data loss)
The boundary: security controls that limit blast radius or protect recovery paths are resiliency findings. Everything else (encryption algorithms, IAM policies, network ACLs, WAF rules) belongs in a security plugin.

Cost vs. Resiliency:
The plugin does not optimize for cost. It surfaces the resiliency gap and, for DR patterns, discloses cost multipliers (Backup & Restore = 1x, Pilot Light = 1.5-2x, Warm Standby = 2-3x, Active-Active = 2-4x) so the human user can make the trade-off. It will never say "skip Multi-AZ to save money" — that's a business decision, not a resiliency finding. It flags the gap; the architect decides whether to accept the risk.

Observability boundary with aws-observability plugin:
The resiliency plugin treats observability as a detection RTO problem: "If you can't detect a failure within your detection RTO, your actual RTO is unbounded." It checks for the existence of alarms on critical metrics, correct TreatMissingData configuration, and whether alarms route to an on-call system. It does not create dashboards, configure log aggregation pipelines, set up tracing instrumentation, or tune alert thresholds — that's the observability plugin's domain.

Concretely: this plugin surfaces "no CloudWatch alarm on RDS CPU — detection RTO is unbounded" as a resiliency finding. The observability plugin would then help you build the dashboard, set the threshold, and configure the notification channel.

Happy to add an explicit "Scope" section to the README that codifies this. Something like:

In scope: Missing redundancy, absent failover, unhandled failure modes, recovery path gaps, blast-radius-limiting security controls, detection RTO (alarm existence).

Out of scope: Security posture (encryption, IAM, WAF, network ACLs), cost optimization of remediation, monitoring/alerting configuration (threshold tuning, dashboard creation, log pipelines), performance tuning (except where it directly impacts failure behaviour).