How I Approach Production Incident Investigation

Users report increased errors. The worst first move is to restart everything and hope.

Incident investigation needs structure. The goal is to protect users, understand the system, and improve it after the immediate pressure is gone.

1. Establish Impact

Start with scope.

• What is failing?
• Which users, services, or regions are affected?
• When did it begin?
• Is impact increasing?
• Is there a safe mitigation?

This prevents the team from chasing interesting but irrelevant signals.

2. Check Recent Changes

Recent changes matter: deployment, configuration, database migration, feature flag, dependency change, infrastructure change, or traffic pattern.

But the latest deployment is not always the cause. It is a lead, not a verdict.

3. Use Observability

Logs, metrics, and traces answer different questions.

Logs explain events. Metrics show trends, saturation, and rates. Traces connect work across services.

In a distributed workflow, I want correlation:

Request ID
-> API
-> service
-> Kafka event
-> worker
-> downstream call

Without correlation IDs or trace context, teams waste time stitching together fragments manually.

4. Protect Users First

Root cause analysis can continue after stabilization.

Mitigation may include rollback, disabling a feature flag, rerouting traffic, reducing load, increasing capacity, or using a circuit breaker to stop hammering a failing dependency.

The right mitigation is the one that safely reduces user impact without making recovery harder.

5. Separate Symptom from Root Cause

“Database CPU was high” is a symptom.

Why was it high?

Possibilities include a new query, missing index, retry storm, increased workload, lock contention, migration, or connection pool issue.

Stopping at the symptom leads to weak follow-up work.

6. Follow Up

After mitigation, the work becomes learning and prevention.

Useful follow-up includes blameless RCA, corrective actions, monitoring improvements, regression tests, runbooks, alerts, and architectural changes.

The goal is not blame. The goal is a system and team that handle the next failure better.

Production Principle

Good incident response is calm, evidence-driven, and user-focused. It treats production as a system to understand, not a place to improvise under pressure.