MusicIndexer/README.md

MusicIndexer

Pull your Spotify library and listening stats via the Spotify Web API, index a local music library (audio tags + iTunes XML), and match the two so you can see which tracks you own, what's missing, and your top artists/tracks/playlists — in a Streamlit GUI.

It is database-backed and incremental: data lives in a DB (SQLite by default, Postgres optional) and every step UPSERTs, so re-running only updates what changed and nothing already fetched is lost. The headline use case is seeing which Spotify/iTunes tracks you don't have locally, decided by real recording identifiers (ISRC / MusicBrainz MBID) with a normalized-text fallback.

Pipeline

ingest  ->  enrich (MBIDs)  ->  match  ->  GUI

Step	Command	Incremental rule
ingest	python main.py ingest	upsert tracks by (source, source_id); never drops enrichment
enrich	python main.py enrich	query MusicBrainz only for tracks with no MBID yet; every lookup (incl. "not found") cached; rate-limited & resumable
match	python main.py match	classify each Spotify/iTunes track: present locally (ISRC > MBID > text) or missing
GUI	./gui.sh	browse missing/present with the match method shown

How matching works

Each track is resolved to a MusicBrainz recording MBID by searching with the normalized artist+title, so the same song from different sources lands on the same MBID (and shares one cached lookup). A Spotify/iTunes track counts as "have" if its ISRC, its MBID, or its normalized artist|title matches a local track — otherwise it's missing. The GUI labels each match isrc/mbid (ID-certain) or text (fuzzy).

Note: the local library here has no ISRC/MBID tags, so before enrich runs, matching is normalized-text only (still useful). enrich upgrades matches to ID-certain. Tagging local files with Picard/AcoustID (ISRC) would make the ID tiers exact end-to-end.

Setup

Requires Python 3.13+.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Configure (.env)

cp .env.example .env       # then edit and `source .env`

Three things you point at your own data:

Variable	What
LOCAL_LIBRARY_PATH	folder of audio files to scan (ISRC/MusicBrainz tags are read)
ITUNES_XML_PATH	the Library.xml you export from Music (File → Library → Export Library…)
DATA_DIR	storage root — the database and cached Spotify responses live here

SPOTIFY_CLIENT_ID / SPOTIFY_CLIENT_SECRET are only needed for live Spotify fetching (register http://127.0.0.1:8888 as a Redirect URI in your Spotify app). Set DATABASE_URL=postgresql+psycopg://… to use Postgres instead of SQLite — no code change.

Two workflows

python main.py fetch      # do all the work, incrementally, and report
python main.py display    # open the GUI to see what you're missing

fetch pulls Spotify, scans the local library, parses the iTunes export, resolves MBIDs, and matches — logging progress at each step. The first run is slow (full scan + MusicBrainz lookups at ~1 req/s); later runs are quick because each source only touches what changed:

• Spotify — playlists are re-fetched only when their snapshot_id changes; liked songs are paged newest-first and stop at the newest one already stored; cover art downloads only when missing or its URL changed (--spotify-full forces a complete resync).
  • Your library is liked songs + playlist tracks. The top-tracks endpoint is not pulled by default (those are just listening stats, not saved music, and would inflate "missing"). Add --include-top to pull them too — they're fetched as full Spotify objects, so they carry an ISRC for real ID matching like everything else.
  • Each Spotify track records its origin (liked / playlist / top, combinable). On a --spotify-full run, rows that are only there because the top endpoint listed them (neither liked nor in a playlist) are pruned — run python main.py fetch --spotify-full once to clean up an existing DB.
• Local — only files whose mtime/size changed; deleted files are pruned.
• iTunes — skipped entirely if the export's mtime is unchanged.
• MBID enrichment — only still-unresolved tracks.

Useful flags/helpers:

python main.py fetch --enrich-limit 500   # cap MBID lookups this run
python main.py enrich                      # resume MBID resolution only
python main.py match                       # print the present/missing summary

ISRC/MBID matching works as soon as fetch runs (local tags + Spotify ISRC). MusicBrainz enrichment fills in the rest (iTunes, untagged local files); it's cached and resumable, so you can stop/continue any time.

Layout

main.py                 CLI: fetch / display / enrich / match
Gui.py / gui.sh         Streamlit GUI (missing-tracks viewer)
Env.py                  config: LOCAL_LIBRARY_PATH / ITUNES_XML_PATH / DATA_DIR
src/
  workflows.py          run_fetch (the whole pipeline) + run_display
  db/                   SQLAlchemy models + engine/session
  ingest/
    spotify_raw.py      live fetch + ingest raw Spotify JSON (ISRC)
    local_scan.py       incremental file scan (reads ISRC/MBID tags)
    itunes_xml.py       parse the iTunes Library.xml (plist)
    tags.py             read ISRC / recording MBID / duration from a file
  enrich/               MusicBrainz MBID resolver (cached, resumable)
  match/                normalization + cross-source matcher
  backends/             lower-level Spotify Web API client / OAuth
$DATA_DIR/              database + cached Spotify responses (gitignored)

Notes

• The DB is the source of truth; everything is incremental and re-runnable.
• Local matching is real-ID where files are tagged (ISRC/MusicBrainz, e.g. via Picard); untagged files fall back to MusicBrainz text lookup, then text.
• Only currently-supported Spotify endpoints are used (no deprecated audio-features/recommendations/etc.).