Your alarms catch outages, not slowness: test gray failures with AWS FIS

AWS update: Nov 2025 · By Anmol Handa · The reel is 30 seconds. This is the long version.

Tumhare alarms outage pakadte hain, slowness nahi. Most real incidents don't start with "down". They start with "slow for some people".

What's new

On November 12, 2025, AWS added two scenarios to the AWS Fault Injection Service (FIS) scenario library for partial failures:

AZ: Application Slowdown adds latency and degrades performance for resources, dependencies and connections within a single Availability Zone. It works for both single-AZ and multi-AZ applications.
Cross-AZ: Traffic Slowdown disrupts traffic between Availability Zones, for multi-AZ applications.

Both can target a portion of your application's traffic, which is how real partial failures look, such as packet loss on some flows. They're available in all Regions where FIS is, including AWS GovCloud (US).

Why it matters

This is a gray failure: the system is up, health checks pass, but it's slow or flaky for part of the traffic. An alarm on "is it up?" stays green. Customers see spinners.

Seedhi baatGray failure matlab system zinda hai, par dheere chal raha hai. Aur tumhara dashboard bolega sab theek hai.

AWS positions these scenarios for three things: validating observability, tuning alarm thresholds, and practising AZ evacuation decisions, meaning shifting traffic away from a sick zone before it becomes an outage.

Steps

1. Write down what should catch it

Before you inject anything, list the alarms you expect to fire: latency (p99, not just average), error rate, and ideally per-AZ views. Check what's already in alarm so you have a clean baseline:

aws cloudwatch describe-alarms --state-value ALARM

2. Create an experiment template from the scenario

In the AWS FIS console, open Scenario library, choose AZ: Application Slowdown (or Cross-AZ: Traffic Slowdown), and create an experiment template from it. Set your targets, the AZ, the duration, and stop conditions (CloudWatch alarms that halt the experiment).

3. Run and watch

aws fis list-experiment-templates
aws fis start-experiment --experiment-template-id <TEMPLATE_ID>

# check progress
aws fis get-experiment --id <EXPERIMENT_ID>

# the stop button
aws fis stop-experiment --id <EXPERIMENT_ID>

4. Score yourself

Did an alarm fire? How long after the slowdown started?
Did it point at the right AZ?
Did someone make the evacuation call, and did shifting traffic actually help?

Caveats

These inject real latency and traffic disruption. Partial doesn't mean harmless.
Each scenario supports specific resource types and needs an IAM role for FIS. Read the scenario description in the console before you build on it.
If nothing fires, that's a finding, not a failed test. It's exactly what this scenario exists to show.
Averages hide gray failures. If your only latency alarm is on the mean, expect to miss it.

Try it safely

Staging first, one AZ, short duration.
Add at least one stop-condition alarm, and keep stop-experiment ready in a terminal.
Tell on-call it's a drill, but don't tell them which AZ. See whether the alarms find it.
Write down time to detect and time to decide. Fix the slowest one first.
Re-run after you tune thresholds.