- Tag each Spotify track with its origin (liked / from_playlist / from_top,
combinable) so genuinely-saved tracks are distinguishable from ones only
surfaced by the top-tracks endpoint.
- Top tracks are no longer added by default (they inflated "missing" with
unowned listening stats). --include-top pulls them as full objects so they
carry an ISRC for real ID matching.
- On --spotify-full, prune Spotify rows that are neither liked nor in a
playlist (top-only / orphaned), cleaning an existing DB in one pass.
- Cached-ingest fallback honours include_top and prunes too.
- In-place SQLite migration adds the new columns (no DB rebuild needed).
- Ctrl+C exits cleanly (130) with a message instead of a traceback;
incremental progress is already committed, so re-run to continue.
- Surface origin breakdown in the fetch log, match summary, and GUI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The live fetch no longer re-pulls everything each run:
- playlists: re-fetch a playlist's tracks only when its snapshot_id changed
(new/deleted playlists handled); state in spotify_playlist table.
- liked songs: page newest-first, stop at the newest added_at already seen
(watermark in ingest_state).
- cover art: download only when the file is missing or the URL changed.
New src/ingest/spotify_fetch.py drives this against the low-level
SpotifyWebAPI client and upserts straight to the DB; fetch uses it when creds
are set (falls back to the cached-JSON ingest otherwise). `fetch --spotify-full`
forces a complete resync. Shared upsert helper extracted into spotify_raw.
Verified with a mocked API: unchanged re-run does 0 playlist-track fetches and
one liked page; a snapshot change re-fetches exactly that playlist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- spotify_raw: skip re-ingesting when the cache files are unchanged (signature
in ingest_state), matching the iTunes/local incremental behaviour — so a
re-run with no changes touches nothing.
- fetch --no-enrich: skip MusicBrainz enrichment and rely on MBIDs already on
tracks (e.g. local Picard tags).
Verified on the real library: 1st run ~9s, unchanged re-run ~0.3s with all
three sources skipping; with enrichment off, 1164 Spotify tracks still match
locally by ISRC/MBID from tags.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the incremental pipeline to real sources and expose two workflows.
Config (Env.py): three explicit paths — LOCAL_LIBRARY_PATH (scan source),
ITUNES_XML_PATH (manual export), DATA_DIR (storage root: DB + Spotify cache).
Real, incremental ingesters:
- local_scan: walks the library, skips files whose mtime/size are unchanged,
prunes deleted ones, and reads embedded ISRC + recording MBID from tags
(tagged files get real IDs with no network), via tags.py.
- itunes_xml: parse the iTunes Library.xml as a plist; skip if mtime unchanged.
- spotify_raw: ingest raw Spotify JSON (ISRC from external_ids); live fetch
wrapper for when credentials are set.
Workflows (src/workflows.py + main.py):
- `fetch` — pull + scan + parse + enrich + match, with readable step logs.
First run is slow (full scan + MB lookups); later runs only touch changes
(verified: ~71s first, ~0.6s second).
- `display` — launch the GUI.
Also: enrich uses ISRC-first again (local MBIDs now come from tags), fix
--enrich-limit 0, quiet musicbrainzngs XML warnings, ignore /data/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the JSON-file pipeline with a SQLite/SQLAlchemy store (Postgres via
DATABASE_URL) and an incremental, ID-based matcher whose goal is "which
Spotify/iTunes tracks am I missing locally".
- src/db: models (Track/MbCache/FileState/IngestState) + engine/session.
- src/ingest: upsert prefetched spotify/itunes/local (work_dir/latest-1) by
(source, source_id); pulls ISRC from raw Spotify; preserves enrichment.
- src/enrich: MusicBrainz recording-MBID resolver. Resolves via NORMALIZED
text search (so the same song across sources lands on one MBID + one cached
lookup), ISRC as exact fallback. Only touches unresolved tracks, caches
positive AND negative results, rate-limited and resumable.
- src/match: normalization + cross-source matcher with tiers ISRC > MBID >
normalized text; reports present/missing and the match method.
- main.py: CLI (ingest / enrich / match). Gui.py: missing-tracks viewer.
- README + requirements (SQLAlchemy, musicbrainzngs); ignore *.db.
Incremental everywhere: re-running updates only what changed; enrich continues
where it stopped. Live Spotify/local backends are left intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- fetch() uses a retrying session (connection resets, 5xx, 429 Retry-After)
with backoff, so a transient drop mid-fetch no longer aborts the run.
- tqdm progress bars on all paginated fetches (driven by paging `total`).
- Flow.fetch_spotify(): drop stray `self` from the @staticmethod.
- gui.sh runs the venv's streamlit from the script dir; add requirements.txt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
