Learn · 30 lessonsUnderstand the cause. Fix with confidence.Command walkthroughs, common mistakes and preparation for your cases Prepare for an incident

On-call practice in a safe environment

Prod is down.
Your shift.

Incident training for SRE and DevOps

Fix broken servers in a real Linux console: get an alert, find the cause, restore the service, and review the incident like an experienced SRE.

Your first case, “Store unavailable”, is hands-on practice lasting 3–5 minutes: find the cause of the outage and bring the store back up.

  • 24live cases
  • 0installations
  • Linuxin your browser
  • Freefree of charge

History Leaderboard Log in Sign up

INCIDENT / SHOP-API LIVE
Error rate38.4%
p95 latency2.41s
Disk free0 MB
ALERT shop-api: 502 · disk 100% full
$ df -h /var/lib/shop
tmpfs       512M  512M     0 100% /var/lib/shop
$ sudo du -sh /var/lib/shop/* | sort -h
494M  /var/lib/shop/export-nightly.csv
$ sudo rm /var/lib/shop/export-nightly.csv
$ sudo systemctl restart shop-api
$ curl -sI localhost:8080/health
HTTP/1.1 200 OK
RESOLVED resolved in 3:41 · +100 points

How it works

  1. ALERT

    An incident arrives

    You receive the symptoms and context of recent changes.

  2. $ _

    Investigate in the console

    Explore real Linux directly in your browser.

  3. RESOLVED

    Check and debrief

    Restore the service and get an explanation of the solution.

Library · 24 incidents

Broken production. Hands-on practice.

Each case is a separate Linux server, real symptoms, and an automatic check of your fix.

Filters: difficulty and tags
Difficulty
Tags

Track 1 Basics

0/14 completed
LOW Case 0 · 5 min

Store unavailable

After an update, customers cannot open the store: the page returns 502 Bad Gateway. The server is reachable and nginx is running, but shop-api does not respond. Find the cause and restore the store, from your first check to a successful customer request.

practicelinuxsystemd
LOW Case 1 · 15 min

Application won’t start

Now shop-api won’t start. The log contains the reason: find it, free space for the application, and restore the store. One fault and one clear diagnostic path.

linuxlogsdisk
CRITICAL Case 2 · 30 min

Database unresponsive

PostgreSQL went down after a deployment. Users receive 500 errors.

postgresqldatabase
HIGH Case 3 · 25 min

High CPU usage

API servers use 95% CPU. Latency increased tenfold.

linuxperformance
CRITICAL Case 4 · 20 min

OOM killer terminates processes

Memory is exhausted and the system is killing critical services.

memoryoom
MEDIUM Case 5 · 15 min

Disk 100% full

Logs filled /var/log. The service cannot write new data.

disklogs
HIGH Case 6 · 35 min

Kubernetes pods in CrashLoopBackOff

All pods are crash-looping after a configuration update.

kubernetesconfig
CRITICAL Case 7 · 40 min

SSL certificate expired

Users cannot access the site; browsers block the connection.

sslnginx
HIGH Case 8 · 20 min

Site returns 502 Bad Gateway

After an application release, nginx returns 502 to customers even though nginx itself is running.

nginxsystemd
MEDIUM Case 9 · 20 min

No space left, but df shows free space

The API reports “No space left on device” although three quarters of the filesystem is free.

diskinode
HIGH Case 10 · 25 min

Payment gateway unreachable

The payment service cannot reach the gateway by name, even though the gateway is running.

dnsnetwork
LOW Case 11 · 15 min

Nightly backup missing

Store backups stopped appearing after an environment change. Restore automatic backups and verify the data.

linuxcronbackup
MEDIUM Case 12 · 20 min

HTTP 200, but a blank page

After an update, the site shows an empty page although monitoring reports 200 OK. Restore the interface and data loading.

nginxhttpfrontend
LOW New Case 13 · 20 min

Small files upload, larger files fail

Small product files are saved, while a normal photo gets 413. Disk space is available and media-api is running. Locate the limiting layer and restore uploads up to 4 MiB.

nginxhttpuploads

Track 2 Advanced

0/10 completed
CRITICAL Weekly on-call challenge Case 1 · 15 min

API exhausted PostgreSQL connections

Long connections overload PostgreSQL; fixing the timeout exposes a retry storm.

postgresqlcascading-failuresre
HIGH Case 2 · 25 min

Too many open files

The notification gateway stopped accepting connections. Its log says Too many open files.

linuxlimitssystemd
HIGH Case 3 · 25 min

Server clock skew

Users cannot log in: the service considers tokens “not yet valid”.

timentplinux
MEDIUM Case 4 · 20 min

The site works intermittently

After a store update, some requests fail. Find the cause of the instability and restore every application instance.

nginxload-balancingbackend
MEDIUM Case 5 · 25 min

Microservices no longer trust each other

The external API works, but requests to the internal service return 401. Restore trusted communication between services.

apimicroservicesauthentication
HIGH Case 6 · 25 min

Kubernetes Service cannot reach pods

After an update, pods are running but the application is unreachable through its Service. A Kubernetes training simulation with real HTTP processes and editable manifests.

kubernetesnetworkingsimulation
HIGH New Case 7 · 30 min

API hangs while PostgreSQL is running

The store opens, but checkout times out. PostgreSQL accepts connections and uses little CPU. Find what holds the working queries and restore checkout.

postgresqllockstransactions
HIGH New Case 8 · 30 min

Orders accepted, fulfillment stalled

The API returns 202, RabbitMQ is reachable and the worker is running, but new orders are not fulfilled. Trace messages through acknowledgement and drain the queue without losing orders.

rabbitmqqueuesdelivery
HIGH New Case 9 · 30 min

The cache mixed up customer carts

After a store optimization, customers see someone else’s cart or get rejected after signing in. The backend distinguishes users. Find where private responses became shared and restore session isolation.

nginxcachesessions
HIGH New Case 10 · 30 min

An incident is happening, but no alert arrives

The synthetic checkout probe fails, but the on-call engineer receives no message. Trace Prometheus → Alertmanager → pager and restore actual alert delivery.

prometheusalertmanagerobservability

Frequently asked questions

Is it free?

Yes, all cases are free. Case 0 requires no registration. For the others, create an account with just a username and password—no email or phone number.

Do I need to install anything?

No. The console opens in your browser; nothing is installed on your computer.

Is this real Linux?

Yes. Each player gets a separate, isolated Linux server with a broken service. Run any commands: your computer and other players are unaffected.

What if I get stuck?

Each case has three hints: direction → where to look → a specific command. Hints reduce your score, but the solution still counts. You can reset the lab and start over.

Can I use a phone?

For case 0, yes. The other cases work best on a computer: there are many commands, and typing them on a phone is awkward.

Who is this for?

For aspiring SREs, DevOps engineers, system administrators, on-call engineers, and anyone preparing for their first shift or an interview. You can start from scratch: case 0 introduces independent diagnosis of a simple incident.