MusicIndexer/README.md

# MusicIndexer

Pull your Spotify library and listening stats via the Spotify Web API, index a
local music library (audio tags + iTunes XML), and match the two so you can see
which tracks you own, what's missing, and your top artists/tracks/playlists — in
a Streamlit GUI.

It is **database-backed and incremental**: data lives in a DB (SQLite by
default, Postgres optional) and every step UPSERTs, so re-running only updates
what changed and nothing already fetched is lost. The headline use case is
seeing **which Spotify/iTunes tracks you don't have locally**, decided by real
recording identifiers (ISRC / MusicBrainz MBID) with a normalized-text fallback.

## Pipeline

```
ingest  ->  enrich (MBIDs)  ->  match  ->  GUI
```

| Step | Command | Incremental rule |
|------|---------|------------------|
| **ingest** | `python main.py ingest` | upsert tracks by `(source, source_id)`; never drops enrichment |
| **enrich** | `python main.py enrich` | query MusicBrainz only for tracks with no MBID yet; every lookup (incl. "not found") cached; rate-limited & resumable |
| **match** | `python main.py match` | classify each Spotify/iTunes track: present locally (ISRC > MBID > text) or missing |
| **GUI** | `./gui.sh` | browse missing/present with the match method shown |

### How matching works

Each track is resolved to a MusicBrainz **recording MBID** by searching with the
*normalized* artist+title, so the same song from different sources lands on the
same MBID (and shares one cached lookup). A Spotify/iTunes track counts as
"have" if its ISRC, its MBID, or its normalized artist|title matches a local
track — otherwise it's **missing**. The GUI labels each match `isrc`/`mbid`
(ID-certain) or `text` (fuzzy).

> Note: the local library here has no ISRC/MBID tags, so before `enrich` runs,
> matching is normalized-text only (still useful). `enrich` upgrades matches to
> ID-certain. Tagging local files with Picard/AcoustID (ISRC) would make the ID
> tiers exact end-to-end.

## Setup

Requires Python 3.13+.

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### Configure (`.env`)

```bash
cp .env.example .env       # then edit and `source .env`
```

Three things you point at your own data:

| Variable | What |
|----------|------|
| `LOCAL_LIBRARY_PATH` | folder of audio files to scan (ISRC/MusicBrainz tags are read) |
| `ITUNES_XML_PATH` | the `Library.xml` you export from Music (File → Library → Export Library…) |
| `DATA_DIR` | storage root — the database and cached Spotify responses live here |

`SPOTIFY_CLIENT_ID` / `SPOTIFY_CLIENT_SECRET` are only needed for live Spotify
fetching (register `http://127.0.0.1:8888` as a Redirect URI in your
[Spotify app](https://developer.spotify.com/dashboard)). Set
`DATABASE_URL=postgresql+psycopg://…` to use Postgres instead of SQLite — no
code change.

## Two workflows

```bash
python main.py fetch      # do all the work, incrementally, and report
python main.py display    # open the GUI to see what you're missing
```

**`fetch`** pulls Spotify, scans the local library, parses the iTunes export,
resolves MBIDs, and matches — logging progress at each step. The **first run is
slow** (full scan + MusicBrainz lookups at ~1 req/s); **later runs are quick**
because each source only touches what changed:

- **Spotify** — playlists are re-fetched only when their `snapshot_id` changes;
  liked songs are paged newest-first and stop at the newest one already stored;
  cover art downloads only when missing or its URL changed (`--spotify-full`
  forces a complete resync).
  - Your library is **liked songs + playlist tracks**. The *top-tracks* endpoint
    is **not** pulled by default (those are just listening stats, not saved
    music, and would inflate "missing"). Add `--include-top` to pull them too —
    they're fetched as full Spotify objects, so they carry an ISRC for real ID
    matching like everything else.
  - Each Spotify track records its **origin** (`liked` / `playlist` / `top`,
    combinable). On a `--spotify-full` run, rows that are *only* there because
    the top endpoint listed them (neither liked nor in a playlist) are pruned —
    run `python main.py fetch --spotify-full` once to clean up an existing DB.
- **Local** — only files whose mtime/size changed; deleted files are pruned.
- **iTunes** — skipped entirely if the export's mtime is unchanged.
- **MBID enrichment** — only still-unresolved tracks.

Useful flags/helpers:

```bash
python main.py fetch --enrich-limit 500   # cap MBID lookups this run
python main.py enrich                      # resume MBID resolution only
python main.py match                       # print the present/missing summary
```

> ISRC/MBID matching works as soon as `fetch` runs (local tags + Spotify ISRC).
> MusicBrainz enrichment fills in the rest (iTunes, untagged local files); it's
> cached and resumable, so you can stop/continue any time.

## Layout

```
main.py                 CLI: fetch / display / enrich / match
Gui.py / gui.sh         Streamlit GUI (missing-tracks viewer)
Env.py                  config: LOCAL_LIBRARY_PATH / ITUNES_XML_PATH / DATA_DIR
src/
  workflows.py          run_fetch (the whole pipeline) + run_display
  db/                   SQLAlchemy models + engine/session
  ingest/
    spotify_raw.py      live fetch + ingest raw Spotify JSON (ISRC)
    local_scan.py       incremental file scan (reads ISRC/MBID tags)
    itunes_xml.py       parse the iTunes Library.xml (plist)
    tags.py             read ISRC / recording MBID / duration from a file
  enrich/               MusicBrainz MBID resolver (cached, resumable)
  match/                normalization + cross-source matcher
  backends/             lower-level Spotify Web API client / OAuth
$DATA_DIR/              database + cached Spotify responses (gitignored)
```

## Notes

- The DB is the source of truth; everything is incremental and re-runnable.
- Local matching is real-ID where files are tagged (ISRC/MusicBrainz, e.g. via
  Picard); untagged files fall back to MusicBrainz text lookup, then text.
- Only currently-supported Spotify endpoints are used (no deprecated
  audio-features/recommendations/etc.).