Lesson 9 of 10Diagnostics

12 min · theory and practice · Advanced

Time, NTP and token errors

Check the clock when tokens and certificates suddenly become “not yet valid”.

Distinguish an incorrect clock from a timezone issue and verify ongoing synchronization.

Tokens, TLS, cron and log ordering depend on time. If independent systems reject fresh data simultaneously, check their shared assumption: the server clock. Changing the timezone alters its representation, not an incorrect UTC instant.

Record the time and a reference

Compare date -u with a trusted reference in your environment and inspect timedatectl. The RTC provides another local clock, but can also be wrong. Agreement between two local clocks does not prove accuracy.

Commands
date -u
date +%s
timedatectl

Log timestamps may be local or UTC. Normalize the timezone before correlating events, preserving original timestamps and the suspected offset.

Inspect synchronization state

For chrony, tracking describes clock state and sources lists time sources and the selected source. A running daemon does not guarantee synchronization: its sources may be unreachable or unsuitable. The case uses these diagnostic commands.

Commands
systemctl status chrony --no-pager
chronyc tracking
chronyc sources -v
Example output
Leap status     : Not synchronised

This indicates a synchronization problem. Next check whether an acceptable source exists, is reachable and is allowed to adjust the clock.

Relate the failure to its validity window

Certificates have notBefore and notAfter; tokens may have iat, nbf and exp. A slow server clock makes a fresh token appear to come from the future. Do not disable validity checks: they reject invalid credentials.

Commands
sudo journalctl -u orders-api -n 40 --no-pager

A service token may be an opaque string rather than a JWT. Investigate time based on the symptoms and validation method, not merely the word “token”.

Repair the mechanism, not one reading

A one-off date adjustment leaves future drift uncorrected. Inspect chrony configuration, sources and the correction policy after VM pauses. In a live system, coordinate a clock step with application requirements; the training case isolates that experiment.

Commands
sudo journalctl -u chrony -n 50 --no-pager
grep -Ev "^[[:space:]]*(#|$)" /etc/chrony/chrony.conf

Do not copy an arbitrary public NTP address into an isolated lab; use the reachable source specified for that environment.

Recheck dependent operations

After restoring the source, recheck synchronization, then user login and TLS. Consider how existing logs and schedules will be interpreted across the clock change. Record this in the incident review so a clock transition is not mistaken for another outage.

From symptom to check

ObservationNext checkHow to interpret it
token used before issuedUTC and a trusted referenceThe server may lag behind the token issuer.
chrony active but Not synchronisedsources, tracking and configurationA live daemon does not establish a usable selected source.
Local display differs by a timezone offsetUTC instant independently of timezoneFirst separate representation from an actual clock error.

Common mistakes

  • Expanding token leeway instead of inspecting the clock.
  • Changing timezone and declaring synchronization recovered.
  • Setting the date once while leaving NTP broken.

How to prove recovery

  1. Compare UTC against a trusted reference.
  2. Inspect tracking and the selected synchronization source.
  3. Repeat authentication and TLS validation.
  4. Verify synchronization continues over an observation interval.

MINI INCIDENT

A token “from the future”

The server shows 10:00 UTC and a trusted reference shows 10:05. A token issued at 10:05 is rejected as used before issued. Should you change token lifetime?

Evidence to investigate
server UTC: 10:00
trusted UTC: 10:05
chrony: Not synchronised
Explore the diagnosis

First restore correct time and ongoing synchronization, then obtain a fresh token and recheck. A longer token lifetime does not fix clock lag and may hide failures in other dependencies.

WORKSHOP

Workshop: explain the evidence and choose an action

The service rejects a token as issued in the future. Compare clocks and time constraints before rotating a secret.

Sample evidence
trusted reference UTC: 12:00:00
server UTC:            11:48:00
token issued-at:       11:59:55
  1. Which mismatch explains the refusal?

    Compare your reasoning

    The validating server is nearly 12 minutes behind, making a valid issued-at look future-dated. This supports a clock hypothesis, not a wrong-secret conclusion.

  2. What should be checked beyond date?

    Compare your reasoning

    Check synchronization state, selected source and its reachability. Correcting the displayed time once does not establish that drift accumulation is resolved.

  3. How should recovery be tied to the original error?

    Compare your reasoning

    Repeat the operation with expiry validation enabled, compare UTC with the reference and inspect synchronization stability. Do not remove token time checks to declare recovery.

CHECK YOUR UNDERSTANDING

Will selecting Europe/Moscow fix a UTC clock that is an hour slow?

Show explanation

No. It changes the displayed timezone. The system needs the correct instant and working synchronization.

Documentation

Guest progress is stored separately in this browser.

TAKE IT TO THE TERMINAL

Practice on an incident

Use validity windows to investigate login and TLS failures; in the clock case, verify automatic time maintenance.

DID IT MAKE SENSE?

Was this article helpful?

Rate the article or tell us what needed more explanation. Your feedback helps improve the lessons.

Helpful 0 Not helpful 0

Sign in to rate this article and leave a comment.

Discussion

0

Join the discussion →

No comments yet. Share your first impression or ask a question about this topic.