Guide · Engineering

Crash-proof job queues: parallel work without losing jobs

Frameo turns a spreadsheet of prompts into AI videos, one job per row, rendered at the same time across several accounts. Each account has a limit on how many renders it may run at once. A render takes minutes, so a crash, a power cut or a closed laptop in the middle of a batch is not an edge case. It is Tuesday. This is how the queue is built so that no job is ever lost or run twice. Frameo has two queues, the original render queue and a newer scheduler that spreads work across several accounts. Both follow the same rules, and each excerpt below names the file it comes from.

By Syed Abdullah Shah · 23 September 2026 · C# / .NET 8, WPF, SQLite

1. The database is the only source of truth

Every job is a row, and its status is a column. The scheduler keeps nothing in memory that it cannot rebuild from the database, apart from live counters for the dashboard. That one rule is what makes everything else possible: if the process dies, nothing is lost, because nothing lived only in the process.

Statuses are stored as text, not numbers, so a database opened by hand still reads plainly, and adding a stage later cannot renumber existing rows. A job moves through Queued → Processing → Generating → Rendering → Downloading → Completed, with Failed and Cancelled kept apart, so the failure count stays a count of things that actually went wrong.

2. One loop hands out work, and it counts before it gives

Assignment happens in a single pump loop, never from several threads at once, so two passes can never both see the same free slot. The rule is plain arithmetic: free slots equal the account’s limit minus the jobs it is running now. No AI and no guessing is involved.

foreach (var account in EligibleAccounts())
{
    var free = account.Capacity - VideoJobRepository.LiveCountForAccount(account.Id);
    for (var i = 0; i < free; i++)
    {
        var job = VideoJobRepository.ClaimNextQueued(_userId, account.Id, account.Provider);
        if (job is null) break;                  // nothing left to hand out

        SyncActiveCount(account.Id);
        StartWorker(job, account);
    }
}

Accounts are visited in a fixed order, and only those that are connected, authenticated and backed by a configured provider are considered. Each account’s limit is set when the account is connected, to what that provider permits, and the loop never goes above it.

3. Claiming a job is atomic

Picking the next job and marking it taken happen in one transaction, and the update only succeeds if the row is still queued. If anything else got there first, the claim changes nothing and returns empty, so a job cannot be handed out twice.

UPDATE video_jobs
SET status = 'ASSIGNED', provider = $prov, provider_account_id = $acc,
    started_at = COALESCE(started_at, $now), updated_at = $now
WHERE id = $id AND status = 'QUEUED';

4. Retries have a ceiling

A failed render waits briefly, is released from its account, and goes back in the queue, up to three attempts in total. After that it is marked failed for good. An error that retrying cannot fix fails at once instead of wasting two more attempts. If the worker itself crashes, the job goes the same way, so an exception can never swallow a job.

if (job.AttemptCount >= MaxAttempts)
{
    Fail(job, error, terminal: true);
    return;
}

// RETRY_WAIT -> (delay) -> QUEUED, released from its account.
job.Status = JobState.RetryWait;
job.Error = error;
VideoJobRepository.Save(job);

try { await Task.Delay(RetryDelay); } catch { /* fall through to requeue */ }

job.Status = JobState.Queued;
job.ProviderAccountId = null;
job.ProviderJobId = null;
VideoJobRepository.Save(job);

5. Recovery on start-up, without doing work twice

When the app starts, anything still marked as in progress is by definition a leftover from a crash: nothing is running yet. The naive fix is to requeue all of it. That is wrong, because a job may have finished writing its video a moment before the crash, and requeueing it would render and bill it a second time. So recovery first marks as completed any job whose finished file is already on disk, and only then requeues the rest, all in one transaction:

-- A row that already has a finished file is finished, whatever its
-- status column says.
UPDATE jobs
SET status = 'Completed', progress = 1
WHERE status IN ('Processing', 'Generating', 'Rendering', 'Downloading')
  AND output_path IS NOT NULL
  AND file_size > 0
  AND finished_at IS NOT NULL;

UPDATE jobs
SET status = 'Queued', progress = 0, started_at = NULL
WHERE status IN ('Processing', 'Generating', 'Rendering', 'Downloading');

6. Keys never touch the database

Each account’s API key is encrypted with Windows DPAPI, tied to the signed-in Windows user, and kept in a separate encrypted file. The database stores only a reference to it. Copying the database, or the key file, to another machine yields nothing readable.

7. Proving it: a self-test that runs headless

A command-line flag runs the whole pipeline end to end against a mock provider that simulates renders, so the scheduling logic is tested without paying for real ones. It creates a throwaway user, runs four tests, writes a report, and deletes everything it made. The recorded run from 16 September 2026:

A mock proves the scheduling, recovery and retry logic. It cannot prove how a real provider behaves under load, which is what a production run is for.

What I would add next

The retry wait is fixed. Against a real provider’s rate limits, a wait that grows with each attempt and carries a little randomness spreads retries out, instead of bringing them back all at once. The queue also lives in SQLite on one machine, which is right for a desktop app. Running the same design across several machines would move the queue into a shared database, keeping the same atomic claim.

The rules, in short

• Keep every job’s state in the database, never only in memory.
• Assign work from one loop, counting free slots before handing any out.
• Claim jobs with a conditional update, so a job can only be taken once.
• Bound retries, and fail at once when retrying cannot help.
• On recovery, finish what already finished before requeueing the rest.
• Test the whole pipeline headless, against a mock, and keep the report.

Frameo is written up as a case study. For work like this, see business automation and API integration.