On-call practice in a safe environment
Prod is down.
Your shift.
Incident training for SRE and DevOps
Fix broken servers in a real Linux console: get an alert, find the cause, restore the service, and review the incident like an experienced SRE.
Your first case, “Store unavailable”, is hands-on practice lasting 3–5 minutes: find the cause of the outage and bring the store back up.
- 24live cases
- 0installations
- Linuxin your browser
- Freefree of charge
ALERT shop-api: 502 · disk 100% full $ df -h /var/lib/shop tmpfs 512M 512M 0 100% /var/lib/shop $ sudo du -sh /var/lib/shop/* | sort -h 494M /var/lib/shop/export-nightly.csv $ sudo rm /var/lib/shop/export-nightly.csv $ sudo systemctl restart shop-api $ curl -sI localhost:8080/health HTTP/1.1 200 OK RESOLVED resolved in 3:41 · +100 points
How it works
-
ALERT
An incident arrives
You receive the symptoms and context of recent changes.
-
$ _
Investigate in the console
Explore real Linux directly in your browser.
-
RESOLVED
Check and debrief
Restore the service and get an explanation of the solution.
Library · 24 incidents
Broken production. Hands-on practice.
Each case is a separate Linux server, real symptoms, and an automatic check of your fix.
Filters: difficulty and tags
No cases match the selected filters.
Track 1 Basics
Store unavailable
After an update, customers cannot open the store: the page returns 502 Bad Gateway. The server is reachable and nginx is running, but shop-api does not respond. Find the cause and restore the store, from your first check to a successful customer request.
Application won’t start
Now shop-api won’t start. The log contains the reason: find it, free space for the application, and restore the store. One fault and one clear diagnostic path.
Database unresponsive
PostgreSQL went down after a deployment. Users receive 500 errors.
High CPU usage
API servers use 95% CPU. Latency increased tenfold.
OOM killer terminates processes
Memory is exhausted and the system is killing critical services.
Disk 100% full
Logs filled /var/log. The service cannot write new data.
Kubernetes pods in CrashLoopBackOff
All pods are crash-looping after a configuration update.
SSL certificate expired
Users cannot access the site; browsers block the connection.
Site returns 502 Bad Gateway
After an application release, nginx returns 502 to customers even though nginx itself is running.
No space left, but df shows free space
The API reports “No space left on device” although three quarters of the filesystem is free.
Payment gateway unreachable
The payment service cannot reach the gateway by name, even though the gateway is running.
Nightly backup missing
Store backups stopped appearing after an environment change. Restore automatic backups and verify the data.
HTTP 200, but a blank page
After an update, the site shows an empty page although monitoring reports 200 OK. Restore the interface and data loading.
Small files upload, larger files fail
Small product files are saved, while a normal photo gets 413. Disk space is available and media-api is running. Locate the limiting layer and restore uploads up to 4 MiB.
Track 2 Advanced
API exhausted PostgreSQL connections
Long connections overload PostgreSQL; fixing the timeout exposes a retry storm.
Too many open files
The notification gateway stopped accepting connections. Its log says Too many open files.
Server clock skew
Users cannot log in: the service considers tokens “not yet valid”.
The site works intermittently
After a store update, some requests fail. Find the cause of the instability and restore every application instance.
Microservices no longer trust each other
The external API works, but requests to the internal service return 401. Restore trusted communication between services.
Kubernetes Service cannot reach pods
After an update, pods are running but the application is unreachable through its Service. A Kubernetes training simulation with real HTTP processes and editable manifests.
API hangs while PostgreSQL is running
The store opens, but checkout times out. PostgreSQL accepts connections and uses little CPU. Find what holds the working queries and restore checkout.
Orders accepted, fulfillment stalled
The API returns 202, RabbitMQ is reachable and the worker is running, but new orders are not fulfilled. Trace messages through acknowledgement and drain the queue without losing orders.
The cache mixed up customer carts
After a store optimization, customers see someone else’s cart or get rejected after signing in. The backend distinguishes users. Find where private responses became shared and restore session isolation.
An incident is happening, but no alert arrives
The synthetic checkout probe fails, but the on-call engineer receives no message. Trace Prometheus → Alertmanager → pager and restore actual alert delivery.
Frequently asked questions
Is it free?
Yes, all cases are free. Case 0 requires no registration. For the others, create an account with just a username and password—no email or phone number.
Do I need to install anything?
No. The console opens in your browser; nothing is installed on your computer.
Is this real Linux?
Yes. Each player gets a separate, isolated Linux server with a broken service. Run any commands: your computer and other players are unaffected.
What if I get stuck?
Each case has three hints: direction → where to look → a specific command. Hints reduce your score, but the solution still counts. You can reset the lab and start over.
Can I use a phone?
For case 0, yes. The other cases work best on a computer: there are many commands, and typing them on a phone is awkward.
Who is this for?
For aspiring SREs, DevOps engineers, system administrators, on-call engineers, and anyone preparing for their first shift or an interview. You can start from scratch: case 0 introduces independent diagnosis of a simple incident.