Failed deliveries are retried with increasing delays and resumed after a restart. When attempts run out, the delivery becomes a dead letter that you can inspect and replay.
Stored deliveries
Each delivery has an id consisting of dlv_ and 24 hex digits. The digits encode
a millisecond timestamp and sequence, so ids sort by time. Each delivery is one
record in the app's __deliveries tree in logs.db. It contains a fixed 64-byte
header with kind, state, attempt, target, times, and last HTTP status, plus the
event, collection, record id, last error, and payload.
States are pending (waiting for its next attempt), delivered and dead.
After a restart, pending deliveries are tried again in creation order, so a
record's trigger runs retain their order.
A delivered record is kept for delivery.retain_days (default 7; 0 keeps none)
so it can be replayed; a sweep removes older ones each hour. A dead record is
kept until it is replayed.
Retry
After a failed attempt, the next attempt is delayed by base * 2^(attempt - 1)
milliseconds, up to the cap. Retries continue up to the attempt count. These are
app settings set with settings.set; values are whole numbers, and attempt counts
are at least 1:
| Setting | Default | Meaning |
|---|---|---|
webhooks.max_attempts |
5 | attempts in all, the first included |
webhooks.backoff_ms |
1000 | wait after the first failure |
triggers.max_attempts |
5 | the same, for trigger runs |
triggers.backoff_ms |
1000 | |
cron.max_attempts |
3 | the same, for scheduled runs (see schedules) |
cron.backoff_ms |
1000 | |
delivery.backoff_cap_ms |
3600000 | the longest wait, for all three |
delivery.retain_days |
7 | days a delivered record stays replayable |
A run fails when the function throws, traps, hits a limit, or returns a status of
400 or more, in JavaScript and in Rust. A direct call (a route, /fn/<name>,
function.invoke) returns its status and is not retried.
A failed trigger run delays later runs for the same record until it succeeds or becomes dead. A run refused because the executor is stopping does not count as an attempt. It is retained and tried again after the first delay.
Webhooks
webhook.list (GET /v1/_webhooks, MCP substrio_webhook_list) pages the app's
webhooks oldest first with limit and cursor, and returns next_cursor.
curl -s "$SUBSTRIO_URL/v1/_webhooks?limit=20" -H "Authorization: Bearer $SUBSTRIO_KEY"Every request carries <Brand>-Delivery-Id: dlv_..., the same id on each
attempt and on a replay, so a receiver can drop a duplicate. <Brand> is the
webhooks.brand setting, as for the <Brand>-Signature header.
webhook.replay (POST /v1/_webhooks/deliveries/{id}/replay, MCP
substrio_webhook_replay) sends a delivered or dead delivery again under its
same id and returns immediately. The replay is a new attempt chain with the full
attempt count. A pending delivery is refused (it is already going out); an
unknown id is 404; a delivery whose webhook was removed is refused.
Dead letters
A delivery out of attempts is one document in the app's _dead_letters
collection: delivery_id, kind (webhook, trigger or cron), target
(the webhook, trigger or schedule id), destination (the URL or the function
name), attempts, last_error, last_status, collection, record_id,
payload and failed_at. If the document cannot be written the delivery stays
dead in the store, unmirrored, and is written at the next start.