Hacker News

points by blakesterz 1 day ago | 0 comments

What I still don't understand is who is doing it? I assume it's not the big companies like OpenAI or Google or Anthropic, right? So there are a bunch of small things out there doing the scraping and then they sell it to the big places? Or are these just smaller AI models doing the scraping and using it for themselves? Is there an underground ecosystem where people buy and sell this stuff? I understand why it's valuable to the AI companies I'm curious about how all this work, the economics of it all.

[–] qingcharles 1 day ago

Can't speak for everyone. I'm very gently scraping a lot of libraries for public domain periodical scans. The scans often exist nowhere else, or not in the same quality. Sometimes I find a certain library has perhaps the only known remaining intact copy of a periodical or a specific page.

The problem with the libraries is that all these scans are behind walls, so while the texts are public domain nobody can easily access them. I'm trying to liberate them and re-upload them outside the walls.

Until recently a lot of libraries would allow you to remotely sign up for a library card, but in the last 12 months almost all of them have reverted to having to bring ID in person to the library to validate your account, which puts the brakes on things.

[–] sriacha 1 day ago

I've been looking at historic newspapers recently, its bothersome and unfortunate that many (most?) are all behind newspapers.com (ancestry) paywall.

[–] qingcharles 1 day ago

Yeah, I'll be on ancestry.com and it says "We found this article about your great-grandfather!" and then you click it and it's newspapers.com and I have to close the tab.

What's the solution, though? All these newspapers won't scan themselves. I'm setting up a non-profit right now to deal with the magazines. The rights issues are both better and worse with magazines. "Better" in that most magazines have gone out of business and the rights are cheap. Worse in that most of them are orphans and the rights are very hard to track down.

[–] NoMoreNicksLeft 1 day ago

I've been scraping Overdrive for the last few months, so that I can identify every book/magazine title they hold. About halfway done (13 million ids to check). I can slurp down full magazines from their preview function. Books I can only get through my local library's Overdrive service, and they only have a tiny subset of those.

Are you doing newspapers?

[–] qingcharles 1 day ago

Definitely not newspapers. I would love to, but the scope creep would kill me. I estimate there are 100 million magazine issues published over the last 200 years across the world. That will keep me busy enough :)

Overdrive has a pretty limited amount of back issues for magazines, which is why I generally ignore it, except if I want to quickly read something on my Kindle.

Do you have a plan for all the data once you have it?

[–] NoMoreNicksLeft 1 day ago

I'll probably download all the magazines. They keep releasing new issues (for the few titles still printing). They claim that they only keep the back issues for the last 3 years... but all they do is make those impossible to find with their search/browse features. If one has the ids for the individual issues, they seem to have 100% retention. So I'll be able to get those... there are 4 or 5 titles I can't get current issues of otherwise.

I think your estimate of magazine issues is high, but I commend your effort. I wouldn't mind hearing about which you've managed to complete. I keep finding that I can get all the historical issues, and I can get all the recent issues, but there's this hole in the middle from about 1985 through 2005 that is usually impossible. It's irritating.

I might test if I can pull audio books down too, though those will be limited to the ones appearing in my own local library (every municipal library seems to have access to the full magazines though, if you know the magic urls).

[–] qingcharles 17 hours ago

Fire me an email (on my profile).

[–] 1317 22 hours ago

note that doing this directly costs your library lots of money per book/mag

[–] TazeTSchnitzel 1 day ago

We're currently in the middle of an AI goldrush. There are probably hundreds of companies trying to do model pretraining. Anthropic and OpenAI and so on probably care about optics and implementing their crawlers properly. But there will be many firms few of us have heard of. I would guess some of them are Chinese, for example.

[–] oasisbob 1 day ago

Some people making money off it, it's a whole value chain:

- Residential proxy providers

- Hosting providers with IP-rotation as a service.

- Hosting providers who chase AI training customers, DigitalOcean is a big one

- Clouds like AWS with headless browsing as a service

- Platforms like Replit whose value proposition starts to fall apart if clients can't make requests from their platform.

- Companies like meta, whose existing bots are now obviously completely unchained from their advertised responsibilities

[–] joe_the_user 1 day ago

As I understand, most large companies now are relying on a variety of contractors and the constractors try to vacuum up everything so they have the maximum to sell to different companies.

This is the way for companies these days to avoid legal responsibility for their actions - hire a contractor on a "don't ask, don't tell" basis.

Of course such things are extra-ironic in the context of US companies complaining about having their condensed by Chinese companies.

Edit: I read this in one article a while back. I can't links either corroborating or refuting it now and I think that's again because whatever the companies do, they want to keep it hush-hush.