Incremental live Spotify fetch (snapshot/watermark)
The live fetch no longer re-pulls everything each run: - playlists: re-fetch a playlist's tracks only when its snapshot_id changed (new/deleted playlists handled); state in spotify_playlist table. - liked songs: page newest-first, stop at the newest added_at already seen (watermark in ingest_state). - cover art: download only when the file is missing or the URL changed. New src/ingest/spotify_fetch.py drives this against the low-level SpotifyWebAPI client and upserts straight to the DB; fetch uses it when creds are set (falls back to the cached-JSON ingest otherwise). `fetch --spotify-full` forces a complete resync. Shared upsert helper extracted into spotify_raw. Verified with a mocked API: unchanged re-run does 0 playlist-track fetches and one liked page; a snapshot change re-fetches exactly that playlist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
ca1ca74244
commit
6920b634c4
6 changed files with 244 additions and 36 deletions
5
main.py
5
main.py
|
|
@ -19,7 +19,8 @@ def cmd_fetch(args):
|
|||
_setup_logging()
|
||||
from src.workflows import run_fetch
|
||||
|
||||
run_fetch(enrich_limit=args.enrich_limit, enrich=not args.no_enrich)
|
||||
run_fetch(enrich_limit=args.enrich_limit, enrich=not args.no_enrich,
|
||||
spotify_full=args.spotify_full)
|
||||
|
||||
|
||||
def cmd_display(args):
|
||||
|
|
@ -56,6 +57,8 @@ def build_parser():
|
|||
help="cap MBID lookups this run (default: resolve all)")
|
||||
fetch.add_argument("--no-enrich", action="store_true",
|
||||
help="skip MusicBrainz enrichment; use MBIDs already on tracks")
|
||||
fetch.add_argument("--spotify-full", action="store_true",
|
||||
help="force a full Spotify resync (ignore snapshot/watermark)")
|
||||
|
||||
sub.add_parser("display", help="open the Streamlit GUI")
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue