Incremental live Spotify fetch (snapshot/watermark)

The live fetch no longer re-pulls everything each run:
- playlists: re-fetch a playlist's tracks only when its snapshot_id changed
  (new/deleted playlists handled); state in spotify_playlist table.
- liked songs: page newest-first, stop at the newest added_at already seen
  (watermark in ingest_state).
- cover art: download only when the file is missing or the URL changed.

New src/ingest/spotify_fetch.py drives this against the low-level
SpotifyWebAPI client and upserts straight to the DB; fetch uses it when creds
are set (falls back to the cached-JSON ingest otherwise). `fetch --spotify-full`
forces a complete resync. Shared upsert helper extracted into spotify_raw.

Verified with a mocked API: unchanged re-run does 0 playlist-track fetches and
one liked page; a snapshot change re-fetches exactly that playlist.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Шурупов Илья Викторович 2026-06-15 09:26:09 +03:00
parent ca1ca74244
commit 6920b634c4
6 changed files with 244 additions and 36 deletions

View file

@ -78,8 +78,17 @@ python main.py display # open the GUI to see what you're missing
**`fetch`** pulls Spotify, scans the local library, parses the iTunes export,
resolves MBIDs, and matches — logging progress at each step. The **first run is
slow** (full scan + MusicBrainz lookups at ~1 req/s); **later runs are quick**
because only changed files, the iTunes export if it changed, and still-unresolved
MBIDs are touched. Useful flags/helpers:
because each source only touches what changed:
- **Spotify** — playlists are re-fetched only when their `snapshot_id` changes;
liked songs are paged newest-first and stop at the newest one already stored;
cover art downloads only when missing or its URL changed (`--spotify-full`
forces a complete resync).
- **Local** — only files whose mtime/size changed; deleted files are pruned.
- **iTunes** — skipped entirely if the export's mtime is unchanged.
- **MBID enrichment** — only still-unresolved tracks.
Useful flags/helpers:
```bash
python main.py fetch --enrich-limit 500 # cap MBID lookups this run