What is Job Queue?
Definition
A job queue is a mechanism that records work the user does not need to wait for, such as sending emails, generating PDFs or resizing images, as background jobs and hands them to separate worker processes. The web request returns quickly while the job runs in the background. Failed jobs are retried according to defined rules, and each job's state can be tracked from enqueue to completion.
Also known as: background job, background jobs, worker, task queue, background worker
Getting slow work out of the request
If a “download report” button triggers 40 seconds of work, doing it inside the HTTP request causes trouble: the browser or a proxy times out, request handlers stay tied up, and a page refresh starts the whole thing again. A job queue moves that work out of the request. The application records “generate this report” in the queue and immediately tells the user it is being prepared. A separate worker process picks the job up, finishes it and stores the result.
Typical background jobs include transactional emails, image resizing and conversion, export files, syncing data to third-party services, bulk notifications and slow analyses.
Job queue versus message queue
A message queue is general infrastructure for moving data between services. A job queue is an application-level abstraction built on top of one. A job is more than a message: it has a name, arguments, a state, an attempt count, its last error and a time it should run. Libraries such as BullMQ for Node.js, Sidekiq for Ruby and Celery for Python keep that state in Redis or a database and give you retries, delayed jobs, priorities and concurrency limits out of the box.
The life of a job
| State | Meaning |
|---|---|
| Waiting | Enqueued, waiting for a free worker |
| Delayed | Scheduled for later, or backing off before a retry |
| Active | Claimed and locked by a worker |
| Completed | Finished successfully, result stored |
| Failed | Out of attempts, waiting for someone to look at it |
If the worker holding an active job dies, the job must not stay active forever. Queue systems handle this with a lock timeout or heartbeat: when it expires, another worker can claim the job.
Retries and idempotency
Background jobs talk to networks, external APIs and other services, so transient failures are a fact of life. A sensible retry policy tells two kinds of failure apart:
- Transient: timeouts, 429 Too Many Requests, 503s. Retry with exponential backoff (for example 10 s, 1 min, 5 min) plus some random jitter.
- Permanent: invalid input, a deleted record, a permission error. Retrying will not change the outcome; fail the job straight away and raise an alert.
Retries mean a job can run more than once. If a worker sends an email and crashes before marking the job complete, the email goes out twice. Jobs therefore need to be idempotent, for instance by recording that the confirmation for an order has been sent, or by passing an idempotency key to the external service.
Workers and concurrency
More workers drain a queue faster, up to a point. Each one holds database connections and memory, and when jobs call an external API, that API's rate limits become the real ceiling. Splitting queues by type helps: short, critical jobs such as password reset emails get their own queue and concurrency setting, separate from long, heavy jobs such as bulk exports. A wave of heavy work then cannot delay the emails users are waiting for.
A minimal database-backed queue
Small and mid-sized projects can use a table in their existing PostgreSQL database as a job queue. The one thing to get right is that two workers never claim the same job. FOR UPDATE SKIP LOCKED does exactly that by skipping rows another worker has already locked:
BEGIN;
SELECT id, kind, args
FROM jobs
WHERE status = 'waiting' AND run_at <= now()
ORDER BY run_at
LIMIT 1
FOR UPDATE SKIP LOCKED;
-- run the job, then:
UPDATE jobs SET status = 'completed' WHERE id = $1;
COMMIT;Work that must recur at fixed times also needs a scheduler. The classic tool is a cron job, though many job queue libraries can define repeatable jobs themselves.