points by NoMoreNicksLeft 1 day ago | 0 comments
I've been scraping Overdrive for the last few months, so that I can identify every book/magazine title they hold. About halfway done (13 million ids to check). I can slurp down full magazines from their preview function. Books I can only get through my local library's Overdrive service, and they only have a tiny subset of those.
Are you doing newspapers?
[–] qingcharles 1 day ago
[–] NoMoreNicksLeft 1 day ago
I'll probably download all the magazines. They keep releasing new issues (for the few titles still printing). They claim that they only keep the back issues for the last 3 years... but all they do is make those impossible to find with their search/browse features. If one has the ids for the individual issues, they seem to have 100% retention. So I'll be able to get those... there are 4 or 5 titles I can't get current issues of otherwise.
I think your estimate of magazine issues is high, but I commend your effort. I wouldn't mind hearing about which you've managed to complete. I keep finding that I can get all the historical issues, and I can get all the recent issues, but there's this hole in the middle from about 1985 through 2005 that is usually impossible. It's irritating.
I might test if I can pull audio books down too, though those will be limited to the ones appearing in my own local library (every municipal library seems to have access to the full magazines though, if you know the magic urls).
[–] qingcharles 17 hours ago
Fire me an email (on my profile).
[–] 1317 22 hours ago
note that doing this directly costs your library lots of money per book/mag
Overdrive has a pretty limited amount of back issues for magazines, which is why I generally ignore it, except if I want to quickly read something on my Kindle.
Do you have a plan for all the data once you have it?