Async Media Jobs: Combining Webhooks, Polling, and State Machines

Design a recoverable job lifecycle that prevents duplicate billing, lost callbacks, and permanently pending work.

Key takeawayWebhooks provide low-latency notification, polling guarantees convergence, and a state machine prevents conflicts between them.

Long jobs should not pretend to be synchronous

Video generation can move through queueing, execution, moderation, upload, and transcoding. Holding one HTTP request open invites gateway timeouts and makes refresh behavior unclear. A cleaner API returns a request_id and pending state immediately, then advances through queries or callbacks.

Define a one-way state machine

Use a finite set such as pending, running, succeeded, failed, and canceled, with explicit allowed transitions. A terminal state cannot be overwritten by an older event; duplicate events are safe; and every transition retains its source, time, and provider job ID.

Submit with an idempotency key so network retries do not create another job.
Verify webhook signatures and timestamps, and retain an event digest.
Deduplicate before updating state, then trigger downstream work.
Validate file type, size, and expiry after downloading the result.

Webhooks and polling are complementary

Webhooks alone remain vulnerable to network, configuration, and consumer failures. Frequent polling alone wastes quota and increases load. ModelRush recommends webhooks as the primary path with exponential-backoff polling for reconciliation. After the expected duration, reduce frequency and enter alerting or manual review.

Handle late and duplicate events

Distributed events can repeat, arrive out of order, or be late. Use a version or conditional write for state changes. A succeeded job must not return to running because an old callback arrived, and a duplicate success must not publish twice.

Test reliability by injecting faults: drop a callback, repeat an event, delay it ten minutes, and expire a result URL. The workflow should still converge to the correct state.