Incremental live Spotify fetch (snapshot/watermark)

The live fetch no longer re-pulls everything each run:
- playlists: re-fetch a playlist's tracks only when its snapshot_id changed
  (new/deleted playlists handled); state in spotify_playlist table.
- liked songs: page newest-first, stop at the newest added_at already seen
  (watermark in ingest_state).
- cover art: download only when the file is missing or the URL changed.

New src/ingest/spotify_fetch.py drives this against the low-level
SpotifyWebAPI client and upserts straight to the DB; fetch uses it when creds
are set (falls back to the cached-JSON ingest otherwise). `fetch --spotify-full`
forces a complete resync. Shared upsert helper extracted into spotify_raw.

Verified with a mocked API: unchanged re-run does 0 playlist-track fetches and
one liked page; a snapshot change re-fetches exactly that playlist.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Шурупов Илья Викторович 2026-06-15 09:26:09 +03:00
parent ca1ca74244
commit 6920b634c4
6 changed files with 244 additions and 36 deletions

View file

@ -19,7 +19,8 @@ def cmd_fetch(args):
_setup_logging()
from src.workflows import run_fetch
run_fetch(enrich_limit=args.enrich_limit, enrich=not args.no_enrich)
run_fetch(enrich_limit=args.enrich_limit, enrich=not args.no_enrich,
spotify_full=args.spotify_full)
def cmd_display(args):
@ -56,6 +57,8 @@ def build_parser():
help="cap MBID lookups this run (default: resolve all)")
fetch.add_argument("--no-enrich", action="store_true",
help="skip MusicBrainz enrichment; use MBIDs already on tracks")
fetch.add_argument("--spotify-full", action="store_true",
help="force a full Spotify resync (ignore snapshot/watermark)")
sub.add_parser("display", help="open the Streamlit GUI")