Hacker News
points by EvanAnderson 1 day ago | 0 comments
I mind a public access website for a government entity and this is exactly the pattern I see. US residential IPs make a couple of requests then never come back.

The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.

I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HTML as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.

Aún no hay comentarios.
Hacker News