Turning weekly production incidents into a rare event

Context

The production platform was experiencing serious incidents approximately every week. The problem was not one failing component. It was a combination of incomplete signals, gaps in alerting, traffic sensitivity, manual delivery risk, and application behavior that only became visible under load.

The goal was not to install more monitoring products. It was to make failures easier to detect, diagnose, correct, and prevent.

Constraints

Reliability work happened while the existing product and release schedule continued. Some failure modes were visible in application monitoring; others—such as specific 502 and 503 responses—were not reaching the team through the primary alerts. Scaling also had to account for the behavior of a legacy PHP application rather than assume the application was already cloud-native.

The response needed to improve both the platform and the way developers shipped into it.

1. 01Signals

   APM · metrics · targeted alerts

2. 02Diagnosis

   Application and infrastructure together

3. 03Correction

   Scaling · delivery · code path

4. 04Verification

   Observe · learn · adjust

Decisions

Build complementary signals

New Relic remained the primary application and server monitoring tool. Prometheus and Grafana provided complementary infrastructure visibility. Targeted alerting covered important HTTP errors that were not being surfaced reliably through the existing path.

The purpose of the additional signals was not dashboard volume. Each alert needed a clear reason to exist and a path to investigation.

Add capacity without adding unnecessary operations

Application servers moved behind load balancing and into an Auto Scaling Group. Scaling behavior was tied to real resource pressure so traffic increases no longer depended on a manual server response.

A multi-region design was considered and rejected because its operating cost and complexity were disproportionate to the team’s needs. Reliability improved more by strengthening the existing region, delivery path, and observability than by multiplying failure domains the team would have to support.

Treat deployment reliability as SRE work

CI/CD automated the application build, frontend build, database migration steps, object-storage sync, and CDN invalidation. Fewer manual release steps meant fewer opportunities for environment drift and partial deployments.

Implementation

The work followed a feedback loop rather than a single infrastructure project:

1. Identify the incidents and signals the team could not see early enough.
2. Add targeted monitoring and alerting around those failure modes.
3. Trace failures across application code, storage access, compute, and delivery.
4. Correct the highest-leverage code or infrastructure issue.
5. Improve the deployment or scaling path that allowed it to recur.
6. Watch production behavior and adjust the assumptions.

One difficult example involved S3 signed URLs. The implementation behaved acceptably under normal traffic but caused severe slowdowns when traffic increased. Finding it required crossing the infrastructure–application boundary: the storage service itself was not the root cause; the application’s signed-URL path was doing too much work at the wrong point in the request flow.

Results

Serious production incidents fell from approximately weekly to nearly zero after sustained work across monitoring, alerting, autoscaling, troubleshooting, and deployment reliability. The change came from improving the operating system around production—not from one tool or isolated fix.

Lessons

Observability is useful only when it shortens the path from symptom to decision. A dashboard that does not change the response is decoration; a targeted alert that exposes a blind spot can prevent the next serious incident.

Reliability also rarely belongs to infrastructure alone. The signed-URL issue reinforced the value of being able to read the application, understand the production environment, and correct the boundary between them.