feat: back up libraries to a folder on the server or an S3 bucket, as deduplicated snapshots #29

Merged

jfro merged 5 commits from library-backups into main

2026-10-16 18:26:02 +00:00

jfro commented

2026-10-16 15:29:44 +00:00

Owner

Closes #26 (first version; restore is #28). Design notes in docs/backups.md.

What it does

New Settings → Backups (per user): back up to this server (a folder per user under BACKUP_DIR, only offered when it's set; priv/backups in dev) or an S3 bucket (AWS or any S3-compatible service: endpoint, region, bucket, optional folder, keys; the secret is stored encrypted and a blank one keeps it). Snapshots to keep: 3/7/14/30/90. Daily on/off. Test connection and Back up now.
Deduplicated snapshots: files go to blobs/<ab>/<sha256> once; each snapshot is a manifest at snapshots/<name>.json listing every item (synced and imported) and its files. Only new files are uploaded; the manifest is written last.
Retention: after each backup, snapshots beyond keep are deleted, then every blob none of the rest use. If a remaining manifest can't be read, no blobs are deleted.
When: BackupWorker (queue backups: 1, unique per user). Daily runs after a sync when the last run was a day ago (like the tag scan), and are skipped when the library's fingerprint matches the last snapshot. Back up now always makes one.
Getting things back: each snapshot opens to its notebooks/PDFs/EPUBs, each downloadable as an .rmdoc built from the backup's own files. Rmdoc.zip/2 was split out of Rmdoc.write/1 for this.
S3 uses Req's built-in SigV4 (no new deps), path-style URLs, ListObjectsV2 with paging.

Testing

16 new tests, including a fake signed S3 bucket (Req.Test) that pages listings 2 keys at a time. Full suite 673 passed; --warnings-as-errors (test and prod) and format --check-formatted are clean.
Manual: against a real S3 server (SeaweedFS in Docker; MinIO's images are no longer published) and the local folder, with a synced library of 37 items / 186 files / 20.5 MB. All 30 documents' .rmdocs built from the backup were identical to the library's. Wrong S3 keys give a clear signature error.

Deploying: set BACKUP_DIR to offer local backups (ideally another disk than /data); a migration adds backup_settings and backup_snapshots.

Closes #26 (first version; restore is #28). Design notes in `docs/backups.md`. **What it does** - New **Settings → Backups** (per user): back up to **this server** (a folder per user under `BACKUP_DIR`, only offered when it's set; `priv/backups` in dev) or an **S3 bucket** (AWS or any S3-compatible service: endpoint, region, bucket, optional folder, keys; the secret is stored encrypted and a blank one keeps it). **Snapshots to keep**: 3/7/14/30/90. **Daily** on/off. Test connection and Back up now. - **Deduplicated snapshots**: files go to `blobs/<ab>/<sha256>` once; each snapshot is a manifest at `snapshots/<name>.json` listing every item (synced and imported) and its files. Only new files are uploaded; the manifest is written last. - **Retention**: after each backup, snapshots beyond `keep` are deleted, then every blob none of the rest use. If a remaining manifest can't be read, no blobs are deleted. - **When**: `BackupWorker` (queue `backups: 1`, unique per user). Daily runs after a sync when the last run was a day ago (like the tag scan), and are skipped when the library's fingerprint matches the last snapshot. Back up now always makes one. - **Getting things back**: each snapshot opens to its notebooks/PDFs/EPUBs, each downloadable as an `.rmdoc` built from the backup's own files. `Rmdoc.zip/2` was split out of `Rmdoc.write/1` for this. - S3 uses Req's built-in SigV4 (no new deps), path-style URLs, ListObjectsV2 with paging. **Testing** - 16 new tests, including a fake signed S3 bucket (`Req.Test`) that pages listings 2 keys at a time. Full suite 673 passed; `--warnings-as-errors` (test and prod) and `format --check-formatted` are clean. - Manual: against a real S3 server (SeaweedFS in Docker; MinIO's images are no longer published) and the local folder, with a synced library of 37 items / 186 files / 20.5 MB. All 30 documents' `.rmdoc`s built from the backup were identical to the library's. Wrong S3 keys give a clear signature error. **Deploying**: set `BACKUP_DIR` to offer local backups (ideally another disk than `/data`); a migration adds `backup_settings` and `backup_snapshots`.

jfro added 1 commit

2026-10-16 15:29:44 +00:00

feat: back up libraries to a folder on the server or an S3 bucket, as deduplicated snapshots

CI / test (pull_request) Successful in 1m7s

Details

381a7e363d

jfro added 1 commit

2026-10-16 17:10:52 +00:00

feat: browse a backup snapshot by folder, with search across all of it

CI / test (pull_request) Successful in 1m7s

Details

207d816ceb