ilusha/MusicIndexer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

DB-backed incremental redesign: find tracks missing locally

Browse source 
Replace the JSON-file pipeline with a SQLite/SQLAlchemy store (Postgres via
DATABASE_URL) and an incremental, ID-based matcher whose goal is "which
Spotify/iTunes tracks am I missing locally".

- src/db: models (Track/MbCache/FileState/IngestState) + engine/session.
- src/ingest: upsert prefetched spotify/itunes/local (work_dir/latest-1) by
  (source, source_id); pulls ISRC from raw Spotify; preserves enrichment.
- src/enrich: MusicBrainz recording-MBID resolver. Resolves via NORMALIZED
  text search (so the same song across sources lands on one MBID + one cached
  lookup), ISRC as exact fallback. Only touches unresolved tracks, caches
  positive AND negative results, rate-limited and resumable.
- src/match: normalization + cross-source matcher with tiers ISRC > MBID >
  normalized text; reports present/missing and the match method.
- main.py: CLI (ingest / enrich / match). Gui.py: missing-tracks viewer.
- README + requirements (SQLAlchemy, musicbrainzngs); ignore *.db.

Incremental everywhere: re-running updates only what changed; enrich continues
where it stopped. Live Spotify/local backends are left intact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Шурупов Илья Викторович 2026-06-15 01:25:21 +03:00
parent 83f782d125
commit 0e64d439c4
16 changed files with 784 additions and 274 deletions
Download patch file Download diff file Expand all files Collapse all files

121
README.md
View file

@ -5,19 +5,38 @@ local music library (audio tags + iTunes XML), and match the two so you can see
which tracks you own, what's missing, and your top artists/tracks/playlists — in
a Streamlit GUI.
## Features
It is **database-backed and incremental**: data lives in a DB (SQLite by
default, Postgres optional) and every step UPSERTs, so re-running only updates
what changed and nothing already fetched is lost. The headline use case is
seeing **which Spotify/iTunes tracks you don't have locally**, decided by real
recording identifiers (ISRC / MusicBrainz MBID) with a normalized-text fallback.
- **Spotify backend** — OAuth (Authorization Code) login, then fetch the user
profile, liked tracks, all playlists and their tracks, long-term top tracks
and top artists, and playlist/profile cover art. Resilient paginated fetching
(follows `next`, retries dropped connections, honours `429` rate limits) with
progress bars.
- **Local backend** — index a folder of audio files via `mutagen` tags.
- **iTunes backend** — parse an iTunes Library XML export.
- **Matching** — map local tracks to Spotify tracks (fuzzy matcher; ISRC/MBID
matching planned).
- **Stats & lyrics** — generate listening stats and fetch lyrics.
- **GUI** — browse libraries as artist → album → track tables.
## Pipeline
```
ingest -> enrich (MBIDs) -> match -> GUI
```
| Step | Command | Incremental rule |
|------|---------|------------------|
| **ingest** | `python main.py ingest` | upsert tracks by `(source, source_id)`; never drops enrichment |
| **enrich** | `python main.py enrich` | query MusicBrainz only for tracks with no MBID yet; every lookup (incl. "not found") cached; rate-limited & resumable |
| **match** | `python main.py match` | classify each Spotify/iTunes track: present locally (ISRC > MBID > text) or missing |
| **GUI** | `./gui.sh` | browse missing/present with the match method shown |
### How matching works
Each track is resolved to a MusicBrainz **recording MBID** by searching with the
*normalized* artist+title, so the same song from different sources lands on the
same MBID (and shares one cached lookup). A Spotify/iTunes track counts as
"have" if its ISRC, its MBID, or its normalized artist|title matches a local
track — otherwise it's **missing**. The GUI labels each match `isrc`/`mbid`
(ID-certain) or `text` (fuzzy).
> Note: the local library here has no ISRC/MBID tags, so before `enrich` runs,
> matching is normalized-text only (still useful). `enrich` upgrades matches to
> ID-certain. Tagging local files with Picard/AcoustID (ISRC) would make the ID
> tiers exact end-to-end.
## Setup
@ -41,60 +60,56 @@ cp .env.example .env
source .env
```
`Env.py` reads `SPOTIFY_CLIENT_ID` / `SPOTIFY_CLIENT_SECRET` (plus optional
`MUSIC_LIBRARY_PATH` and `WORK_DIR`) from the environment.
`Env.py` reads `SPOTIFY_CLIENT_ID` / `SPOTIFY_CLIENT_SECRET` (only needed for
live fetching), plus optional `DATABASE_URL`, `PREFETCHED_DIR` and `MB_CONTACT`.
### Database
Defaults to a local SQLite file (`musicindexer.db`). To use Postgres instead,
set one env var — no code changes:
```bash
export DATABASE_URL=postgresql+psycopg://user:pass@localhost:5432/musicindexer
```
## Usage
Edit `main.py` to uncomment the steps you want, then run:
```bash
python main.py
python main.py ingest # load prefetched data (work_dir/latest-1) into the DB
python main.py enrich # resolve MusicBrainz MBIDs (resumable; ~1 req/s)
python main.py match # print present/missing summary
./gui.sh # browse what you're missing
```
The flow steps (see `Flow.py`) include:
`ingest` and the GUI work immediately (text-based matching). `enrich` queries
MusicBrainz only for tracks that don't have an MBID yet, caches every result,
and is **resumable** — if it's interrupted, just run it again to continue. Run
`python main.py enrich --limit 300` to do a quick partial pass, or
`--source local` to restrict to one source.
| Step | What it does |
|------|--------------|
| `fetch_spotify()` | OAuth login + download all Spotify data into `work_dir/spotify/` |
| `fetch_local()` | index local audio files into `work_dir/local/` |
| `parse_spotify_library()` / `parse_itunes_library()` / `parse_local_library()` | build normalized libraries |
| `load_libraries()` / `save_libraries()` | (de)serialize libraries to `work_dir` |
| `map_local_to_spotify()` | match local tracks against the Spotify library |
| `fetch_and_save_lyrics()` | fetch lyrics for tracks |
| `log_stats()` | generate stats |
On first `fetch_spotify()` a browser opens for the Spotify login; a local server
on `127.0.0.1:8888` catches the redirect and exchanges the code for a token.
### GUI
```bash
./gui.sh # runs: .venv/bin/streamlit run Gui.py
```
> Live fetching (`SpotifyWebAPI`, the local indexer) is unchanged and still
> available; this redesign reads prefetched JSON so the pipeline can be built
> and demoed without network access.
## Layout
```
main.py entry point (CLI flow)
Gui.py / gui.sh Streamlit GUI
Flow.py orchestrates fetch / parse / map / stats
Env.py config (reads credentials from the environment)
main.py CLI: ingest / enrich / match
Gui.py / gui.sh Streamlit GUI (missing-tracks viewer)
Env.py config (DB URL, paths, credentials from env)
src/
Library.py data model (Library / Artist / Album / Track)
LibrarySerializer.py load/save libraries
backends/
spotify/ Web API client, OAuth, parser
local/ local-file indexer, parser, playlist generator
itunes/ iTunes XML parser
navidrome/ Navidrome backend
mappers/ local↔Spotify matching (fuzzy / search)
lyrics/ lyrics fetching
work_dir/ fetched data and serialized libraries (gitignored)
db/ SQLAlchemy models + engine/session
ingest/ load prefetched JSON into the DB (upsert)
enrich/ MusicBrainz MBID resolver (cached, resumable)
match/ normalization + cross-source matcher
backends/ live fetchers/parsers (spotify, local, itunes, ...)
Library.py in-memory data model used by the backends
work_dir/latest-1/ prefetched JSON used by `ingest` (gitignored)
musicindexer.db the database (gitignored)
```
## Notes
- Data is cached as JSON under `work_dir/` (gitignored); re-running reuses it.
- The deprecated Spotify endpoints (audio-features, recommendations, etc.) are
not used — only currently-supported endpoints.
- The DB is the source of truth; `work_dir/` JSON is only an ingest source.
- Only currently-supported Spotify endpoints are used (no deprecated
audio-features/recommendations/etc.).