Queue & Background Job Engineering

A queue is the part of a Laravel application that is easiest to add and hardest to operate. dispatch() is one line, and everything that makes the difference between a queue and a reliable queue happens after that line.

The four questions a queue has to answer

What happens when the job fails? Not if. A third-party API times out, a record is deleted between dispatch and execution, a deploy restarts the worker mid-job. The answer has to be written down as a retry policy with a backoff, a maximum attempt count, and a destination for whatever is still failing at the end of it.

What happens when it runs twice? At-least-once delivery is the normal guarantee, which means every job must either be safe to repeat or be guarded by something that makes repeating harmless. An email sent twice is an embarrassment. A payment captured twice is a chargeback and a support thread.

Who finds out? failed_jobs filling up silently is the most common production fault we are called to look at, and it is never really a queue fault. The job failed, the framework recorded it exactly as designed, and nothing was attached to the recording.

What happens during a deploy? A worker holding a job when the code changes under it is either restarted gracefully or killed. Which one it is depends on configuration most teams inherit rather than choose.

What we do

Make jobs idempotent. An identifier the job carries, enforced where the effect actually lands - a unique constraint, an idempotency key at the payment provider, a status transition that can only happen once. Not a check-then-act in PHP, which two workers will pass simultaneously on the day it matters.

Set retry policies per job, not per application. A transient HTTP failure wants several attempts with an increasing delay. A validation failure wants zero - retrying it just burns the queue and delays everything behind it. Distinguishing them is a two-line change and it is almost never made.

Give failures somewhere to go. Failed jobs reported to whatever your team already watches, with enough context to act on. A dead letter path for what cannot be retried. An alert on the rate rather than the individual event, so it is signal and not noise.

Right-size the topology. Separate queues by latency requirement rather than by feature, so a nightly export cannot delay a password reset. Workers sized against the actual shape of the work. Horizon configured with supervisors that match that, and metrics that make a growing backlog visible before it is an incident.

Make long work survivable. Batched jobs with progress, chunked so a restart costs one chunk rather than the whole run, and a way to resume rather than start again.

Scheduled work, which has the same problems

The scheduler gets less attention than the queue and fails in the same ways. A task that overlaps itself because the previous run is still going. A task that silently stops because the cron entry was on a server that got replaced. A task whose failure is a log line nobody reads.

We treat scheduled tasks as jobs with a trigger: overlap protection where it matters, a heartbeat so that a task which stops running is noticed, and the same failure path as everything else.

How an engagement runs

Send the job classes and, if you keep them, a week of failed job records. What has already gone wrong says more about the design than the code does.

What comes back is a scope: every job with its failure behaviour, the queue topology, the monitoring, and what is in phase one. It has a price, and the contract refers to it rather than to a conversation.

The work lands as reviewable pull requests in your repository, with the retry and idempotency behaviour tested rather than described.

What you receive

A review of every job and scheduled task in the application with its failure behaviour documented, the fixes as reviewable pull requests, the queue topology and worker configuration as code, and the monitoring wired into whatever you already use.

The deliverable we care about is the one that is hardest to demo: a queue that nobody has thought about for a month, because it has not needed anyone to.

The whole design rests on one assumption: every job will run twice. Most queue bugs are a violation of it. The second most common is a worker still running last week's code, which is a deploy fault that reports itself as a queue fault. If the jobs are slow and not wrong, performance work is the place to start.