[–] ferrarosimone 1 day ago
[–] abetusk 1 day ago
It would be good to have copyright information of the artists. If any provide their music under a libre/free license (CC0, CC-BY, CC-BY-SA) that would make it attractive to developers to use.
Some folks also want to search for artists to invite to shows, so being able to filter by location might be beneficial.
[–] ferrarosimone 1 day ago
Just to clarify, the CC0 part is about the Discogs data (credits, labels, releases), not the music itself. Dub Digger doesn't host any audio, the previews come from Apple.
Your points are really good and I agree it would be super useful finding a way to add those info. So I checked the dump straight away.
Long story short, there is a field for who owns the copyright, but nothing about licences.
CC licencing only shows up in a patchy way in free-text fields. It means it's not standardised in any way (no consistency, no structure...). That makes it impossible to filter on it decently.
Location is the same story: there's no location field for artists, only free text in some profiles.
Release country is there, the problem is that it's the country of release and it's not necessarily linked to where the artist is based.
I'm keeping on digging the issue, though!
[–] abetusk 16 hours ago
Soundcloud occasionally has copyright information tied to an individual track. Bandcamp has a structured field that lists the copyright.
A long time ago, I scraped Bandcamp metadata and put it in a SQLlite file [0]. From that you can filter on bands by copyright (among other things).
Maybe a stripped down LLM could handle the free form text and figure out licensing issues. It looks like your list is pretty reasonably sized, so that might not be such a big job.
[–] ferrarosimone 15 hours ago
On the LLM idea: I thought the same to be honest, but the problem on Discogs is less fetching and structuring the free text and more that the info is rarely there. I launched an LLM to check and Creative Commons is there on about 1 release in 10,000, so there isn't much to rely on.
[–] abetusk 12 hours ago
All major search engines index Bandcamp, so I'm not sure they really can justify shutting down everyone else.
From what I remember, about 1% to 5% of artists or songs on bandcamp had libre/free (CC0, CC-BY, CC-BY-SA) terms. I would guess that the most likely reason it's so high compared with your number is that Bandcamp has a dropdown option, essentially, so they make it easy for bands to provide alternate licensing terms. Any libre/free licensing is going to be pretty niche and the people that care about such things is also niche, so I hear you that it's a lot of work for a very small payoff.
Consider the data set size, I was, at one point, considering just downloading all libre/free data, including songs, to avoid bit rot. Maybe I still will.
Doing a quick search with that data set I linked to, I see a couple dozen dub techno bands that are libre/free licensed.
Anyway, keep up the good work. I personally enjoy dub techno very much. I don't think I've ever met anyone offline who also likes it so it's nice to see other people who are passionate about it.
I started digging for dubby, minimal and ambient electronic records many years ago, and as you may imagine I literally spent hours on Discogs.
For a crate digger like myself the most powerful thing is the credits: which label a release is on, who engineered, mastered or remixed something. The connections between these things is what allows you to discover new stuff.
Doing that on Discogs is definitely possible, but it comes with a lot of friction.
Dub Digger is a personal, side project and it's my attempt at a tool to remove that friction. It's essentially the tool I would use to dig out records.
You search an artist, label or release and follow the credits from one name to the next. I needed something focussed on a core of specific genres, so I built a curated database centered on techno, deep, ambient and experimental labels and artists, based on Discogs data. Each artist and label has a relevance grade based on how close each entity is to the core of the database, from very high down to extremely low.
On top of that I added a manual curation layer to mitigate the issues and inconsistencies the automated rules were showing. I am also constantly adding manual overrides, to promote or demote specific artists I think should sit at a different level of relevance from where the raw Discogs data places them.
It currently has about 305k artists, 109k labels and 1M releases.
One interesting thing is that I tried to avoid using user data as much as possible and I did my best to make it truly accessible.
How I built it may be of interest
Source: Discogs' monthly XML dumps (CC0). The releases file is over 100 GB unpacked. No calls to the Discogs API to keep it light and with as few dependencies as possible.
Ingest: a Node/TypeScript script streams the XML, keeps only the fields the tool needs and goes through about 19M releases. Two stages: Stage 1 - it takes records tagged dub techno, deep techno, dub, ambient or minimal, with some rules added on top (for example dub only if tagged along with electronic , otherwise you get the whole reggae catalogue). An artist is in the core only if at least 10% of their releases have those tags. Labels work in the same way. A label belongs to the core if at least half of the artists on it are core artists. Stage 2 - one step out: releases with a core artist tagged on them, and releases on core labels. It stops there, as I was afraid a further step would pull in most of techno.
Precomputing: relevance grades, counts, rankings and the search index are all calculated offline. The output is one read-only SQLite file (about 1.3 GB).
Serving: Astro with server-side rendering, on a small nginx VPS, reading the SQLite file directly. No database server and no live API calls from the server. Covers and 30-second previews come from Apple: I match them offline with the iTunes Search API, and the browser loads them straight from Apple (the previews only when the user press play).
Updates: I re-run the pipeline on a new monthly dump and reupload the file.
Curation: The rules worked well only to an extent. The data were patchy so I had to add my personal curation layer. As a starting point I defined nine "traditions/influences" in the config, among them roots dub, reggae, Detroit techno, afrobeat, dubstep and UK garage. Belonging to one of them raises an artist to a fixed grade. All of this is paired with manual overrides.
Client: no accounts, cookies or tracking. “My List” uses localStorage. Aims for WCAG 2.1 AA ( I checked it myself, no certifications from any third parties).
It’s a beta and a passion project. And of course is far from perfect. I’d love some (kind, please!) feedback on how it feels to navigate.
Cheers