Retry, failure, and recovery

PR #23 replaces unlimited queue redelivery for handler failures with an explicit durable failure ledger in the application database. Pub/Sub remains the fallback only when that ledger cannot be written.

Failure handoff

The saved envelope includes extracted Shopify headers and identifiers, so replay does not depend on the original Pub/Sub message remaining available.

Failure state machine

Only pending rows are selected by retry_failed_events. Terminal failed rows are not retried automatically, and resolved rows are retained for audit.

Retry schedule

The retry command increments retry_count only after a failed replay. It then schedules 2 ** retry_count minutes, capped at 60 minutes.

Failed replay count	Next delay
1	2 minutes
2	4 minutes
3	8 minutes
4	16 minutes
5	Terminal with the default max_retries = 5
Higher custom limits	32 minutes, then 60-minute maximum

A successful first replay goes directly from pending to resolved and does not increment retry_count.

Running retries

Retry up to 50 due events:

python manage.py retry_failed_events

Use a smaller batch:

python manage.py retry_failed_events --limit 10

Target one due, pending failure row:

python manage.py retry_failed_events --event-id 42

The command prints Retried=<n> resolved=<n> failed=<n>.

Scheduling is external

The PR adds the command but does not schedule it. A production deployment must invoke it periodically. Ensure only one scheduler instance runs at a time because the query does not claim rows with SELECT FOR UPDATE or a lease.

Error handling matrix

Failure	Captured fields	Typical corrective action
Invalid JSON or non-object message	Raw decoded data where possible, exception, traceback	Correct publisher format; replay only after envelope is usable
Unsupported or malformed payload	Enriched envelope, ValueError, traceback	Correct data or parser and replay
Missing target configuration	Organization/shop context, StorageConfigurationError	Create Database and DatabaseCredential, provision, replay
Bad credentials or network outage	Full envelope and psycopg exception	Restore access, replay
Missing/incompatible column	Full envelope and PostgreSQL exception	Apply schema correction, replay
Deadlock or lock timeout	Full envelope and database exception	Let contention clear, replay
Code logic error	Full envelope and Python exception	Deploy fix, replay
Application failure table unavailable	Failure cannot be stored	Restore application DB; Pub/Sub redelivery is requested with nack

Worker restart backoff versus event retry

Two independent mechanisms exist:

Mechanism	Handles	Timing
Worker restart loop	Streaming pull crashes or connection-level worker failure	5, 10, 20, 40, then 60 seconds
SubscriberFailedEvent replay	Individual decode, parser, handler, and target database errors	2, 4, 8, 16, 32, then 60 minutes depending on retry count

The restart loop keeps the subscriber process alive. It does not replay an event already acknowledged after durable failure capture.

Operational queries

Pending and due workload:

SELECT id, source, event_type, shop_domain, retry_count, next_retry_at, error_type
FROM subscriber_subscriberfailedevent
WHERE status = 'pending'
  AND (next_retry_at IS NULL OR next_retry_at <= NOW())
ORDER BY created_at;

Terminal failures requiring intervention:

SELECT id, event_id, shop_domain, retry_count, error_type, error_message, updated_at
FROM subscriber_subscriberfailedevent
WHERE status = 'failed'
ORDER BY updated_at DESC;

Recent recovery rate:

SELECT status, COUNT(*)
FROM subscriber_subscriberfailedevent
WHERE created_at >= NOW() - INTERVAL '24 hours'
GROUP BY status;

Current reliability gaps

• Missing or duplicate Shopify connections and unknown handlers are logged and acknowledged without creating a failure row.
• Replay rows are not leased; overlapping retry processes can dispatch the same row concurrently.
• There is no uniqueness constraint preventing duplicate failure rows for the same event.
• There is no automatic operator alert for terminal failed status.
• There is no retention or redaction policy for payloads and tracebacks.
• Snapshot ordering is processing-time last-write-wins; out-of-order Shopify events are not compared using a source sequence or update timestamp.
• Schema creation handles absent tables, not incompatible schema evolution.

Target-table upserts limit the impact of duplicate replay, but these gaps should be addressed before high-volume multi-worker operation.