fix: retry S3 hiccups (signed for the host on every try), clear a backup error once the destination checks out, log backups and failed jobs #37

Merged

jfro merged 1 commit from backup-logging-retries into main

2026-10-17 17:01:01 +00:00

jfro commented

2026-10-17 16:59:28 +00:00

Owner

From trying backups in prod against Backblaze B2.

Retries, and a signing bug they exposed. B2 answers 500 now and then, which failed the whole backup until Oban retried the job. S3 requests now retry transient failures (408/429/5xx, dropped connections) up to 4 times with Req's backoff. Writing the test for that found that a retry re-runs every request step, so SigV4 signed the pinned IP as the host (the URL pin_address had left), sending Host: <ip> with a signature for the IP. Req's default already retried GETs this way in 1.1.x. Now an unpin_address step puts the host back first, so every try is signed for the hostname and sent to the checked address; the fake bucket asserts both on retries too.

The error after a good Test connection. Backups.check/1 now clears last_error when the destination checks out (and tells the page), since the error was about a destination that now works; the next backup reports anything still wrong.

Logging (there was next to none):

every S3 retry and final failure, with operation, key, bucket, endpoint, what S3 said, and x-amz-request-id/x-amz-id-2; endpoints refused by the network check;
each backup: start (destination, item count, forced), files already stored vs uploading, the snapshot made (counts, bytes, uploaded, duration), skips as unchanged, failures (as errors, with duration); prunes, unused files deleted, records forgotten; connection checks;
failed Oban jobs, app-wide (RmAnalyzer.JobLogger): nothing logged them before, so failing OCR/sorting/tagging/backup jobs were silent. A warning while they'll be retried, an error with the stacktrace once given up on. Successful jobs aren't logged.

Tests: a 500 retried twice then succeeding (with the request id in the log), a 503 that lasts (giving up, logged, backup failed as an error), a check clearing the error, and the job logger. 715 passed; checked live against B2 us-west-000.

From trying backups in prod against Backblaze B2. **Retries, and a signing bug they exposed.** B2 answers 500 now and then, which failed the whole backup until Oban retried the job. S3 requests now retry transient failures (408/429/5xx, dropped connections) up to 4 times with Req's backoff. Writing the test for that found that a retry re-runs every request step, so SigV4 signed the *pinned IP* as the host (the URL `pin_address` had left), sending `Host: <ip>` with a signature for the IP. Req's default already retried GETs this way in 1.1.x. Now an `unpin_address` step puts the host back first, so every try is signed for the hostname and sent to the checked address; the fake bucket asserts both on retries too. **The error after a good Test connection.** `Backups.check/1` now clears `last_error` when the destination checks out (and tells the page), since the error was about a destination that now works; the next backup reports anything still wrong. **Logging** (there was next to none): - every S3 retry and final failure, with operation, key, bucket, endpoint, what S3 said, and `x-amz-request-id`/`x-amz-id-2`; endpoints refused by the network check; - each backup: start (destination, item count, forced), files already stored vs uploading, the snapshot made (counts, bytes, uploaded, duration), skips as unchanged, failures (as errors, with duration); prunes, unused files deleted, records forgotten; connection checks; - **failed Oban jobs, app-wide** (`RmAnalyzer.JobLogger`): nothing logged them before, so failing OCR/sorting/tagging/backup jobs were silent. A warning while they'll be retried, an error with the stacktrace once given up on. Successful jobs aren't logged. Tests: a 500 retried twice then succeeding (with the request id in the log), a 503 that lasts (giving up, logged, backup failed as an error), a check clearing the error, and the job logger. 715 passed; checked live against B2 us-west-000.

jfro added 1 commit

2026-10-17 16:59:28 +00:00

fix: retry S3 hiccups (signed for the host on every try), clear a backup error once the destination checks out, log backups and failed jobs

CI / test (pull_request) Successful in 1m12s

Details

84f585731f