feat: ebook libraries, scanning, matching and downloads #17
Loading…
Reference in a new issue
No description provided.
Delete branch "jtk/ebook-management"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Merges Jeremy's
jtk/ebook-managementbranch for v0.6.0. Opened as-is: the branch merges cleanly with currentmain(after #2–#4), and on that mergemix compile --warnings-as-errors,mix format --check-formattedandmix testpass locally (409 tests, 4 skipped). No commits were added or rewritten.What it adds
manual,auto_watch(file_system watcher, debounced) andscheduled(cron).ebook_scanqueue (1 at a time) walks a library;ebook_processqueue (10 at a time) extracts metadata and covers per file. Scan progress shows on the library. Oban Web dashboard at/admin/oban(admin only).pdfinfo/pdftoppm;poppler-utilsadded to the Docker image), CBZ via a zstream ZIP reader.GET /ebooks/:id/download, login required)./adminscope now pipes through:require_admin, which closes #8 (import/export reachable by any logged-in user).Migrations (10)
Oban jobs table (+ upgrade to v14),
ebooks,libraries,ebooks.library_id, library scan progress, extended book metadata, and severalstring→textchanges on books and ebooks. They run on app start like the others.Config / env vars
No new env vars. New app config in
config/config.exs(:fuzzy_catalog, :ebooks: supported formats, 100 MB max file size, fuzzy matching on/threshold, watcher and scheduler toggles) and the Oban config. Libraries are set up in the admin UI.New deps: oban, oban_web, saxy, file_system, zstream, excoveralls (test only).
Known gaps / out of scope
Closes #8.
Review by Nye (PR review / audit) — REQUEST CHANGES
What's fine: download route is in the authenticated scope and the path comes only from the DB; admin LiveViews + Oban dashboard are behind
require_admin(closes #8); migrations are additive and ordered after main's latest; advisory lock around match/create is right; no conflicts with #18/#19.Blockers
B1. App can go down when a library is set to auto_watch in Docker. The runtime image has no
inotify-tools(only CI has it).file_systemthen returns:ignorefromFileSystem.start_link, andLibraryWatcher.start_watcheronly matches{:ok,_}/{:error,_}→CaseClauseError→ restart loop → app supervisor gives up. Fix: add inotify-tools to the runtime image; handle:ignoreand{:error,_}(log library id + path, keep running); consider not linking. Add a test for:ignore.B2. Scheduled scan mode never fires.
LibrarySchedulerstarts after Oban and only doesApplication.put_envon the Oban config; the running Cron plugin never re-reads it. Entries are{expr, {Mod, :fun, args}}, which isn't a valid Oban Cron entry. Fix: one static cron entry for aScheduledScanWorkerthat enqueuesScanWorkerfor each scheduled library whose schedule matches now, skipping ones already scanning.B3. A library can get stuck in 'scanning' forever, blocking Scan now and Delete.
handle_exhausted/handle_cancelledare Oban Pro hooks that OSS Oban never calls; there's no Lifeline plugin; timeouts kill the process before the rescue runs;File.stat!increate_ebook_recordraises if a file vanishes mid-scan. Fix: Lifeline plugin, handle the final attempt insideperform,File.statand skip, makescan_now/scanning?tolerate stale status (or reset at boot). Remove the deadhandle_*functions.Follow-ups (filed as issues, label ebook-branch)
F1 unmounted NAS share wipes ebook records (promoted to release blocker, fixed in this PR) · F2 non-transient processing errors retried · F3 BookMatcher candidate limit can miss the exact match · F4 deleting a library orphans ebooks · F5 Book.changeset casts internal fields (folded into #7) · F6 watcher drops events mid-scan · F7 symlinks in a library expose server files (with #15). #14 (zip bomb) and #15 still apply.
Pushed fixes for the review blockers (4 new commits, no history rewrite), plus the NAS-unmount wipe, which is now a release blocker:
ef40c16):inotify-toolsadded to the runtime image.LibraryWatcherhandles:ignore/{:error, _}/ anything else fromFileSystem.start_link, logs library id + path, records "Auto-watch unavailable: …" on the library, and traps exits so a dying watcher only drops its own library. Tests::ignore,{:error, _}, killed watcher.8f740fb): one static crontab entry{"* * * * *", ScheduledScanWorker}(ownebook_schedulequeue). It callsLibraryScheduler.run_due/1, which matches each scheduled library's cron (viaOban.Cron.Expression, UTC, judged at the job'sscheduled_at) and starts the due ones, skipping ones already scanning or with a missing directory. The GenServer and the runtimeput_envare gone.Library.changesetnow rejects invalid cron expressions. The UI says times are UTC.f7155c4): deadhandle_exhausted/handle_cancelledremoved.Oban.Plugins.Lifeline(rescue_after 30 min) added. A telemetry handler (Ebooks.JobFailureHandler) marks the library/ebook failed when a ScanWorker/ProcessWorker is discarded (timeouts on the last attempt included) or cancelled.Libraries.scanning?/1treats "scanning" as stale when there's no available/scheduled/executing/retryable ScanWorker for the library or ProcessWorker for its ebooks (2-minute grace after the last update), so Scan now and Delete work again.Libraries.start_scan/1marks + enqueues in one transaction and is used by Scan now, the watcher and the scheduler.File.stat!→File.statwith skip. Ebooks left pending/processing by a dead job are re-queued on the next scan.c7e0ec4): a scan that finds 0 files while the library has records aborts with "Is the share mounted? Kept the existing records." and marks the library failed; removal also re-checks that the directory still has entries. A missing directory already aborted before removal. Tests for empty mountpoint, missing dir, empty new library.mix precommit: 419 tests, 0 failures.Review by Nye (PR review / audit) — APPROVE @
c7e0ec4CI green, merges cleanly onto main (
5dd20bb). Verified against Oban 2.24.2 source: telemetry:statevalues include:discard/:cancelledso JobFailureHandler fires correctly;Oban.Plugins.Lifeline/Cronmap toOban.Lifeline/Oban.Croninternally so config is valid; Lifelinerescue_after30m > ScanWorker 10m timeout. B1 (inotify-tools in runtime image +:ignore/{:error}/EXIT handling), B2 (per-minute ScheduledScanWorker +valid_cron?validation, UTC noted in UI), B3 (Lifeline + JobFailureHandler + job-table stale detection inscanning?/start_scan) and F1 (abort when 0 ebook files found but records exist, plus re-check before removals) are all resolved.Non-blocking follow-ups (to be filed as one low issue after merge):
scanning_statuswhenscanning?/1detects it stale.lifeline: [rescue_after: ...],cron: [crontab: ...]) instead of the deprecatedOban.Plugins.*names.Oban.Cron.Expression, which is@moduledoc false(private API); recheck on Oban upgrades.start_scanis check-then-insert, so watcher + scheduler in the same minute can queue two scans (serialized by ebook_scan concurrency 1, harmless);unique: [keys: [:library_id], states: :incomplete]on ScanWorker would close it.