Incremental live Spotify fetch (snapshot/watermark)
The live fetch no longer re-pulls everything each run: - playlists: re-fetch a playlist's tracks only when its snapshot_id changed (new/deleted playlists handled); state in spotify_playlist table. - liked songs: page newest-first, stop at the newest added_at already seen (watermark in ingest_state). - cover art: download only when the file is missing or the URL changed. New src/ingest/spotify_fetch.py drives this against the low-level SpotifyWebAPI client and upserts straight to the DB; fetch uses it when creds are set (falls back to the cached-JSON ingest otherwise). `fetch --spotify-full` forces a complete resync. Shared upsert helper extracted into spotify_raw. Verified with a mocked API: unchanged re-run does 0 playlist-track fetches and one liked page; a snapshot change re-fetches exactly that playlist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
ca1ca74244
commit
6920b634c4
6 changed files with 244 additions and 36 deletions
13
README.md
13
README.md
|
|
@ -78,8 +78,17 @@ python main.py display # open the GUI to see what you're missing
|
|||
**`fetch`** pulls Spotify, scans the local library, parses the iTunes export,
|
||||
resolves MBIDs, and matches — logging progress at each step. The **first run is
|
||||
slow** (full scan + MusicBrainz lookups at ~1 req/s); **later runs are quick**
|
||||
because only changed files, the iTunes export if it changed, and still-unresolved
|
||||
MBIDs are touched. Useful flags/helpers:
|
||||
because each source only touches what changed:
|
||||
|
||||
- **Spotify** — playlists are re-fetched only when their `snapshot_id` changes;
|
||||
liked songs are paged newest-first and stop at the newest one already stored;
|
||||
cover art downloads only when missing or its URL changed (`--spotify-full`
|
||||
forces a complete resync).
|
||||
- **Local** — only files whose mtime/size changed; deleted files are pruned.
|
||||
- **iTunes** — skipped entirely if the export's mtime is unchanged.
|
||||
- **MBID enrichment** — only still-unresolved tracks.
|
||||
|
||||
Useful flags/helpers:
|
||||
|
||||
```bash
|
||||
python main.py fetch --enrich-limit 500 # cap MBID lookups this run
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue