Design a Job Scheduler

"Send this user a reminder in 24 hours." "Retry this failed upload in 5 minutes." "Build the sales report every day at 06:00 in Tokyo." Almost every product has work that must happen later, and every team that writes its own timer on one server eventually loses jobs when that server restarts. A shared job scheduler takes the job, stores it safely, and makes sure some worker runs it close to the right time.

Interviewers like this question because "run it later" hides three hard problems: finding the few jobs due right now among millions waiting, handing each one to exactly one worker when workers crash and pause, and being honest that "exactly once" is something you build out of retries and idempotent jobs. The steps follow A Framework for Any Design.

#Clarify the problem

• Functional: schedule a one-off job at a time or after a delay; create recurring schedules from a cron expression and time zone; cancel jobs; see a job's status and attempts; retry failures with backoff up to a limit, then park them for a human.
• Non-functional: no accepted job is lost (durable, replicated storage); jobs start within 10 seconds of due time under normal load; at-least-once execution; no single machine whose loss stops scheduling.
• Out of scope: running job code ourselves, job dependencies and workflows, priorities beyond a few named queues.

#Estimate the scale

• Job runs: 100 million ÷ 86,400 ≈ 1,200 a second on average.
• Writes: each run is created, enqueued, claimed and finished: about 4 writes, so about 4,800 a second. Too many for comfort on one database primary at peak; this points to sharding.
• Waiting jobs: suppose 10% of jobs are scheduled about a week ahead and the rest within the hour. 100 million × 0.1 × 7 days = 70 million waiting at once, at about 1 KB each, 70 GB. Keeping 7 days of finished jobs for debugging adds 100 million × 7 × 1 KB = 700 GB.
• The burst: if 20% of the million schedules say "every hour, on the hour", then 200,000 jobs are due in the same second at every :00.

Where 1,800 comes from: at 1,200 jobs a second and about 2 seconds per job, 1,200 × 2 = 2,400 jobs run at any moment. Workers run 10 jobs each, so 240 workers are fully busy; with 50% headroom you run 360, which can do 360 × 10 ÷ 2 = 1,800 jobs a second.

#Define the API

POST /jobs
  body: { "queue": "emails", "payload": {...}, "run_at": "2026-11-15T10:05:20Z",
          "max_attempts": 5, "timeout_s": 60, "idempotency_key": "remind-u42-d15"? }
  201 { "job_id": "j_a8f3c1" }
  400 run_at too far in the future, or payload over 64 KB
 
POST /schedules
  body: { "queue": "reports", "cron": "0 6 * * *", "time_zone": "Asia/Tokyo",
          "on_overlap": "skip", "on_missed": "run_once" }
  201 { "schedule_id": "s_311" }
 
DELETE /jobs/{id}      200 cancelled   409 already running
GET    /jobs/{id}      200 { "status": "running", "attempt": 2, "last_error": "..." }
 
Worker side:
POST /queues/{q}/lease            200 [ { "job_id", "attempt", "lease_until", "payload" } ]
POST /jobs/{id}/heartbeat         body: { "attempt": 2 }   200, or 409 lease lost
POST /jobs/{id}/complete          body: { "attempt": 2, "ok": true }   200, or 409 lease lost

The decision to argue is pull, not push. If the scheduler pushed jobs at workers, it would need to know each worker's capacity, and a slow team's workers would back up into the scheduler. When workers pull, each takes only what it can run, and a backlog simply waits in the queue. The attempt number in every worker call is what lets the scheduler reject a worker that has lost its lease.

#Model the data

jobs                            -- one row per job, found by id
  job_id         VARCHAR(64) PRIMARY KEY   -- random, or "s_311@2026-11-15T06:00" for a schedule run
  shard          SMALLINT       -- hash(job_id) % 64
  queue          VARCHAR(64)
  payload        BLOB           -- up to 64 KB; larger goes to blob storage
  run_at         TIMESTAMP
  status         VARCHAR(12)    -- scheduled | enqueued | running | succeeded | cancelled | dead
  attempt        INT
  lease_until    TIMESTAMP NULL -- indexed for running jobs, for the lease sweep
  schedule_id    VARCHAR(32) NULL
 
due                             -- the timer: which ids are due when
  shard          SMALLINT
  bucket         TIMESTAMP      -- run_at rounded down to the minute
  run_at         TIMESTAMP
  job_id         VARCHAR(64)
  PRIMARY KEY ((shard, bucket), run_at, job_id)
 
schedules
  schedule_id, cron, time_zone, queue, payload, on_overlap, on_missed, next_run_at

A job and its due rows hash to the same shard, so creating a job is one local transaction across both tables. The access patterns are: insert a job and its due row; read one (shard, bucket) partition of due in time order; update one job's status with a condition ("only if attempt is still 2"); drop whole old buckets. A retry or a new run time just adds another due row. The dispatcher checks the job's status before enqueuing, so a due row left over from a cancelled or rescheduled job is skipped. Each dispatcher also records, per shard, the last due row it handled, so after a restart or a takeover it resumes from there.

Any store with partitions, sorted keys inside a partition and per-row conditional writes fits: a sharded relational database with the due table partitioned by time, or a wide-column store such as Cassandra. I would pick the sharded relational database because claims and leases are conditional updates, which it does cheaply; the wide-column store wins if write volume grows far past this.

#High-level design

Step through one delayed job, including a worker that stalls halfway.

#Deep dives

#Finding due jobs

Seventy million jobs wait; about 1,200 become due each second. The scheduler must find those 1,200 cheaply, every second, forever.

With 64 shards, the 200,000-job burst at :00 lands as 200,000 ÷ 64 ≈ 3,125 rows per partition: one quick read each. A dispatcher that falls behind does not skip anything; it reads the older buckets it missed first, so a slow minute delays jobs but never loses them.

#Leases and at-least-once execution

A worker that claims a job gets a lease: the job is theirs until lease_until, extended by heartbeats. If the worker dies, the lease runs out and the job is handed out again. That is the whole recovery story, and it is why execution is at least once: a job is never stuck forever, but a worker that only looked dead may still finish its work.

Choosing the lease length is a trade-off. A short lease (30 seconds here) recovers quickly from a crash but re-runs jobs on any worker that pauses for longer. A long lease means fewer duplicates but a dead worker's job waits longer. Heartbeats let you keep the lease short for jobs that run for an hour.

Shard ownership uses the same idea one level up. Each dispatcher holds a lease on its shards from a coordination service built on consensus (see consensus and leader election). If a dispatcher stalls, another takes its shards when the lease expires. The stalled one does not know yet, so for a moment two dispatchers can scan one shard. That is safe because every step is conditional: only one of them can move a job from scheduled to enqueued, and a duplicate id on the queue finds the job already running when a worker tries to claim it, and is dropped.

#The exactly-once illusion

"Exactly once" in this system can only mean exactly-once effect. The tracer shows why: W3 sent the email, then paused, and nothing outside W3 could know whether the send happened. Any scheduler that re-runs jobs from dead workers must sometimes re-run jobs from live ones. Say this plainly in the interview, then give the tools:

• Idempotent handlers. Pass an idempotency key derived from the job (job id, or the caller's own key) to every downstream call, as in idempotency and retries.
• Fencing tokens. Every write the job makes carries its attempt number, and the target rejects older attempts. The store does this for job status; a job can do the same for its own outputs.
• Natural idempotency. "Set the report for 2026-11-15 to this file" can safely run twice; "append a row to the report" cannot.

#Cron schedules versus one-off jobs

Store a schedule as a definition, not as a million future jobs. When an occurrence is claimed, create the next one, with an id built from the schedule id and its scheduled time. Insert-if-absent then makes a repeat harmless: two dispatchers trying to create the 07:00 run produce one row. Three policies need a deliberate answer:

• Overlap: the 06:00 run is still going at 07:00. Skip, queue, or run both. Default to skip for reports.
• Missed runs: the scheduler was down from 02:00 to 05:00. Run once to catch up, run all three, or skip. Default to once.
• Time zones: compute the next time in the schedule's own zone. On a daylight-saving change, 02:30 can happen twice or not at all, so decide which, and test it.

For the :00 burst, let schedules say "hourly, any minute" and pick a stable minute from a hash of the schedule id. The 200,000 hourly jobs then arrive at about 200,000 ÷ 3,600 ≈ 56 a second instead of all at once, and the fleet you already have keeps the 10-second promise.

#Bottlenecks and failures

At 10x, a billion runs a day, about 12,000 a second: add shards to the store (64 to 512), split dispatcher ownership finer, and expect the burst to dominate. Ready queues are per job type, so one team's flood of jobs does not delay another team's; the team that floods its own queue scales its own workers.

#What a strong answer sounds like

#Follow-up questions

#How do you cancel a job that is due in a minute?

Set its status to cancelled with a conditional update that only succeeds while it is still scheduled or enqueued. A worker's claim checks status, so a cancelled id that is already in the queue is dropped. Once running, cancellation becomes a request the job may honour.

#How do retries work?

A failed attempt gets a new run time, now plus exponential backoff with jitter, and a new due row in the matching bucket. After max_attempts the job goes to a dead-letter status for a human, so one broken job cannot retry forever.

#One team schedules 10 million jobs for midnight. What protects everyone else?

Per-queue rate limits on dispatch and separate ready queues per team. Their jobs start late; everyone else's keep their 10-second promise.

#How would you add priorities?

Give each priority its own ready queue and let workers pull from high before low. Keep priority out of the store's key, so finding due jobs stays one partition read.

#How do you know the scheduler is healthy?

Track scheduling lag (start time minus due time) per queue, the age of the oldest due job not yet enqueued, lease expiries per minute and dead-letter counts. Lag is the number users feel; alert on it.