Home / User Guide / Durable Job Queue

Durable Job Queue

Why this exists

Async job state normally lives entirely in the API process's memory. This is fast and simple, but it means a crash — an unhandled exception, an OOM kill, a redeploy — loses every in-flight job silently. A client polling GET /jobs/{job_id} for a job that was mid-solve when the crash happened gets 404 Not Found on restart, with no record anywhere that the job ever existed.

The durable job queue closes that gap: job records survive a process restart, and jobs that were abandoned mid-solve are automatically recovered and resubmitted.

What changes for an API consumer

With JOB_PERSISTENCE_ENABLED=true, your side of the integration doesn't change at all. You still submit to /optimize/async or /optimize/replan/async, get a job_id back immediately, and poll GET /jobs/{job_id} on your own schedule until the status is terminal.

What's different is what happens behind the scenes if the server crashes while your job is queued or running: instead of your job simply disappearing, it's recovered and resubmitted automatically on the next startup, under the exact same job_id you already have. You don't need to know a crash happened, you don't need your own retry/resubmission logic, and you don't need to change how you poll. This is the point of the feature — durability is the server's problem to solve, not something pushed onto every client integrating with the API.

With JOB_PERSISTENCE_ENABLED unset (the default), behavior is unchanged from before this feature existed: job state is in-memory only, and a crash loses in-flight jobs exactly as it always has.

The heartbeat mechanism

A job sitting in queued is unambiguous: queued jobs never expire on their own, so on restart, one the process didn't just create is, by construction, leftover from before the crash.

A job showing running is trickier — it could genuinely still be in progress on a different, healthy server. Status alone can't tell "actively being worked on elsewhere" apart from "the process that owned this died." That's what the heartbeat is for: while a job is running, the server periodically signals that it's still alive by refreshing a timestamp on the job's record, every JOB_HEARTBEAT_INTERVAL_SECONDS (default 10 seconds). If the whole process dies, nothing updates that timestamp anymore, and it simply stops ticking.

On startup, the recovery process treats a running job as abandoned only once its heartbeat is older than JOB_HEARTBEAT_STALE_AFTER_SECONDS (default 30 seconds — three missed ticks at the default interval, to tolerate a single slow tick under load without a false positive). A healthy, still-running job on a still-alive server is never touched by another server's recovery sweep; only a genuinely stale heartbeat triggers recovery.

Auto-retry, capped

When a job is found abandoned at startup, it's automatically resubmitted through the normal launch path — same concurrency limits, same subprocess isolation, same heartbeat — under its original job_id. This is capped by JOB_MAX_AUTO_RETRIES (default 1): if a job's retry count has already reached that cap, it is not resubmitted again. Instead it's marked permanently failed, with an error identifying it as exceeding the abandoned-retry limit.

The cap exists specifically to prevent a single malformed or resource-exhausting request from crash-looping the entire deployment: without it, a request that reliably crashes the process on every attempt would be resubmitted on every subsequent restart, forever. Once a job hits the cap, GET /jobs/{job_id} returns a clean, terminal failed status — not a job stuck retrying indefinitely.

Async endpoints only

This feature covers /optimize/async and /optimize/replan/async only. The synchronous endpoints — /optimize/sync, /optimize/replan, /optimize/reoptimize — are intentionally out of scope.

The reasoning is structural, not an oversight: an async caller receives a job_id immediately and is expected to come back and poll for it later — there's always a "check back" step, and durable persistence has something useful to answer when that check happens, even across a restart. A sync caller holds one HTTP connection open for the life of the request and waits on it for the single response. If the server crashes mid-solve, that connection is simply gone: there's no id the caller was ever given, and no "later" for them to check back on. A durable record for a sync job would have no one left to read it back.

Configuration

curl example

# Submit an async job as usual -- no different from a non-persisted setup
curl -X POST https://api.fieldgenius.be/api/v1/optimize/async \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-adapter-key" \
  -H "X-Client-Key: your-client-key" \
  -d @request.json
# -> {"job_id": "864cbd2f-...", "status": "queued"}

# Even if the server restarts mid-solve, the same id keeps working
curl https://api.fieldgenius.be/api/v1/jobs/864cbd2f-... \
  -H "X-API-Key: your-adapter-key" \
  -H "X-Client-Key: your-client-key"
# -> status: "running" (recovered and resubmitted after the restart), then "completed"

Python example (httpx)

import time
import httpx

BASE = "https://api.fieldgenius.be/api/v1"
HEADERS = {
    "X-API-Key": "your-adapter-key",
    "X-Client-Key": "your-client-key",
}

with httpx.Client(timeout=30, headers=HEADERS) as client:
    submit = client.post(f"{BASE}/optimize/async", json=request_payload)
    submit.raise_for_status()
    job_id = submit.json()["job_id"]

    # No special handling needed even if the server restarts in between --
    # keep polling the same job_id exactly as you would without persistence.
    while True:
        job = client.get(f"{BASE}/jobs/{job_id}").json()
        if job["status"] in ("completed", "failed", "cancelled"):
            break
        time.sleep(3)

    print("final status:", job["status"])