Lesson 10 of 10Track checkpoint

20 min · theory and practice · Advanced

Final incident: diagnosis, recovery and handoff

Turn technical work and a clear report into a complete incident investigation.

Take a case from the first symptom to a verified outcome and explain why the chosen action helped.

The course finale values controlled investigation over rapid command guessing. Establish impact, facts, priority, a justified change and verification. Communication helps preserve that structure under pressure.

Define impact and recovery criteria

Name the affected user, operation and impact boundary. A reachable document does not negate broken payment or internal orders. Choose a verification that corresponds to the user’s actual recovery.

Separate immediate stabilization from resolving the cause. Record any accepted compromise such as reduced capacity, delayed processing or limited functionality if a temporary mode is chosen.

Keep a short hypothesis log

For each hypothesis, record supporting evidence, a next probe and a disconfirming result. Keep useful negative findings; they explain why a layer was ruled out.

Tie the log to time and components. In a cascade, connection growth can follow increased waiting and retries. Two elevated metrics alone do not establish causal direction.

Choose a bounded action

Compare a change’s impact, expected effect and rollback path. If the cause is a leak or excessive client timeout, repeatedly increasing a limit can amplify memory use and waiting.

Before acting, verify the current configuration version, intended PID and correct replica. Preserve the diff and application procedure afterward.

Test positive and negative scenarios

A positive test confirms the restored operation. A negative test verifies a required restriction: an invalid secret is rejected, TLS is validated, or an unready backend does not receive traffic.

Add a bounded stability check: repeat requests, replica state, resource growth or the next scheduled run. Do not describe a short test as a full day of observation.

Hand the result to the next person

A short report includes symptom, impact boundary, causal evidence, change, verification and remaining risk. “Restarted it, now works” is not enough to assess durability.

Choose one cause-related prevention step: restore testing, coordinated asset publication, a useful health contract or bounded retries. It should follow from the failure rather than from a generic checklist.

From symptom to check

ObservationNext checkHow to interpret it
Local improvement, user still failingOriginal operation and full routeThe recovery criterion was too narrow.
Waiting and connections rose togetherTimeline, timeouts, retry countEstablish the causal chain instead of raising every limit.
Works only with a bypassRequired constraints and a negative testRecovery must preserve the required contract.

Common mistakes

  • Using a technical status as the only recovery criterion.
  • Claiming a cause confidently without a disprovable hypothesis.
  • Promising durability that was never observed.

How to prove recovery

  1. The symptom, cause and change are connected clearly.
  2. User-facing and required negative tests support the result.
  3. The next person understands remaining limitations and the next action.

MINI INCIDENT

Faster, but ready to close?

Reducing a client timeout decreased errors, but database connections are still increasing. Write an accurate status and choose two next checks.

Evidence to investigate
error rate: lower
request time: lower
database connections: still increasing
Explore the diagnosis

Status: partial stabilization; durability remains unverified. Compare retry counts at each layer and client connection release, then check whether growth stops. Repeat the real operation and state the condition required before calling the repair complete.

WORKSHOP

Workshop: explain the evidence and choose an action

Write a final status from this short timeline. Separate cause, stabilization and follow-up.

Sample evidence
14:00 errors begin after credentials rotation
14:08 gateway v2, receiver v1 confirmed
14:12 receiver updated; valid request 200
14:13 invalid credentials still 401
  1. Which causal relationship is supported?

    Compare your reasoning

    A credential-version mismatch accompanied the failure; configuration comparison and the targeted change’s result support the conclusion. Name those observations explicitly.

  2. Which two verification types were completed?

    Compare your reasoning

    The positive request succeeds and the negative request remains denied. This is stronger than one 200 because the access boundary is preserved.

  3. Which follow-up addresses the cause?

    Compare your reasoning

    A verifiable rotation and compatibility procedure for sender and receiver, with observation and rollback. “Monitor everything” does not address this specific incompatibility.

CHECK YOUR UNDERSTANDING

How does a useful report differ from a command list?

Show explanation

It explains decisions: which fact narrowed the investigation, why the change addresses the cause and what confirmed the outcome. Another person can assess the conclusion without repeating a scattered search.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Choose a linked case and write three diagnostic checks before changing configuration. After the repair, repeat the original user operation.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.