0e64d439c4
Replace the JSON-file pipeline with a SQLite/SQLAlchemy store (Postgres via DATABASE_URL) and an incremental, ID-based matcher whose goal is "which Spotify/iTunes tracks am I missing locally". - src/db: models (Track/MbCache/FileState/IngestState) + engine/session. - src/ingest: upsert prefetched spotify/itunes/local (work_dir/latest-1) by (source, source_id); pulls ISRC from raw Spotify; preserves enrichment. - src/enrich: MusicBrainz recording-MBID resolver. Resolves via NORMALIZED text search (so the same song across sources lands on one MBID + one cached lookup), ISRC as exact fallback. Only touches unresolved tracks, caches positive AND negative results, rate-limited and resumable. - src/match: normalization + cross-source matcher with tiers ISRC > MBID > normalized text; reports present/missing and the match method. - main.py: CLI (ingest / enrich / match). Gui.py: missing-tracks viewer. - README + requirements (SQLAlchemy, musicbrainzngs); ignore *.db. Incremental everywhere: re-running updates only what changed; enrich continues where it stopped. Live Spotify/local backends are left intact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
115 lines
4.4 KiB
Markdown
115 lines
4.4 KiB
Markdown
# MusicIndexer
|
|
|
|
Pull your Spotify library and listening stats via the Spotify Web API, index a
|
|
local music library (audio tags + iTunes XML), and match the two so you can see
|
|
which tracks you own, what's missing, and your top artists/tracks/playlists — in
|
|
a Streamlit GUI.
|
|
|
|
It is **database-backed and incremental**: data lives in a DB (SQLite by
|
|
default, Postgres optional) and every step UPSERTs, so re-running only updates
|
|
what changed and nothing already fetched is lost. The headline use case is
|
|
seeing **which Spotify/iTunes tracks you don't have locally**, decided by real
|
|
recording identifiers (ISRC / MusicBrainz MBID) with a normalized-text fallback.
|
|
|
|
## Pipeline
|
|
|
|
```
|
|
ingest -> enrich (MBIDs) -> match -> GUI
|
|
```
|
|
|
|
| Step | Command | Incremental rule |
|
|
|------|---------|------------------|
|
|
| **ingest** | `python main.py ingest` | upsert tracks by `(source, source_id)`; never drops enrichment |
|
|
| **enrich** | `python main.py enrich` | query MusicBrainz only for tracks with no MBID yet; every lookup (incl. "not found") cached; rate-limited & resumable |
|
|
| **match** | `python main.py match` | classify each Spotify/iTunes track: present locally (ISRC > MBID > text) or missing |
|
|
| **GUI** | `./gui.sh` | browse missing/present with the match method shown |
|
|
|
|
### How matching works
|
|
|
|
Each track is resolved to a MusicBrainz **recording MBID** by searching with the
|
|
*normalized* artist+title, so the same song from different sources lands on the
|
|
same MBID (and shares one cached lookup). A Spotify/iTunes track counts as
|
|
"have" if its ISRC, its MBID, or its normalized artist|title matches a local
|
|
track — otherwise it's **missing**. The GUI labels each match `isrc`/`mbid`
|
|
(ID-certain) or `text` (fuzzy).
|
|
|
|
> Note: the local library here has no ISRC/MBID tags, so before `enrich` runs,
|
|
> matching is normalized-text only (still useful). `enrich` upgrades matches to
|
|
> ID-certain. Tagging local files with Picard/AcoustID (ISRC) would make the ID
|
|
> tiers exact end-to-end.
|
|
|
|
## Setup
|
|
|
|
Requires Python 3.13+.
|
|
|
|
```bash
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
### Credentials
|
|
|
|
Create an app at https://developer.spotify.com/dashboard and add
|
|
`http://127.0.0.1:8888` as a **Redirect URI**. Credentials are read from the
|
|
environment — never commit them.
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# edit .env with your client id/secret
|
|
source .env
|
|
```
|
|
|
|
`Env.py` reads `SPOTIFY_CLIENT_ID` / `SPOTIFY_CLIENT_SECRET` (only needed for
|
|
live fetching), plus optional `DATABASE_URL`, `PREFETCHED_DIR` and `MB_CONTACT`.
|
|
|
|
### Database
|
|
|
|
Defaults to a local SQLite file (`musicindexer.db`). To use Postgres instead,
|
|
set one env var — no code changes:
|
|
|
|
```bash
|
|
export DATABASE_URL=postgresql+psycopg://user:pass@localhost:5432/musicindexer
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python main.py ingest # load prefetched data (work_dir/latest-1) into the DB
|
|
python main.py enrich # resolve MusicBrainz MBIDs (resumable; ~1 req/s)
|
|
python main.py match # print present/missing summary
|
|
./gui.sh # browse what you're missing
|
|
```
|
|
|
|
`ingest` and the GUI work immediately (text-based matching). `enrich` queries
|
|
MusicBrainz only for tracks that don't have an MBID yet, caches every result,
|
|
and is **resumable** — if it's interrupted, just run it again to continue. Run
|
|
`python main.py enrich --limit 300` to do a quick partial pass, or
|
|
`--source local` to restrict to one source.
|
|
|
|
> Live fetching (`SpotifyWebAPI`, the local indexer) is unchanged and still
|
|
> available; this redesign reads prefetched JSON so the pipeline can be built
|
|
> and demoed without network access.
|
|
|
|
## Layout
|
|
|
|
```
|
|
main.py CLI: ingest / enrich / match
|
|
Gui.py / gui.sh Streamlit GUI (missing-tracks viewer)
|
|
Env.py config (DB URL, paths, credentials from env)
|
|
src/
|
|
db/ SQLAlchemy models + engine/session
|
|
ingest/ load prefetched JSON into the DB (upsert)
|
|
enrich/ MusicBrainz MBID resolver (cached, resumable)
|
|
match/ normalization + cross-source matcher
|
|
backends/ live fetchers/parsers (spotify, local, itunes, ...)
|
|
Library.py in-memory data model used by the backends
|
|
work_dir/latest-1/ prefetched JSON used by `ingest` (gitignored)
|
|
musicindexer.db the database (gitignored)
|
|
```
|
|
|
|
## Notes
|
|
|
|
- The DB is the source of truth; `work_dir/` JSON is only an ingest source.
|
|
- Only currently-supported Spotify endpoints are used (no deprecated
|
|
audio-features/recommendations/etc.).
|