Hacker News

points by cpburns2009 3 days ago | 0 comments

This isn't a theoretical problem either. My employer's website was being hammered by residential proxies a month ago. It wasn't until we setup WAF captcha rules that they mostly subsided. It wasn't even the typical exploit scanner. I don't know what the purpose was but it was effectively a DDoS.

[–] cowlevel 3 days ago

Everyone's is. Far as anyone can tell, it's probably AI companies.

[–] cpburns2009 3 days ago

It wasn't any sensible AI scraper. It kept requesting the same pages a thousand times over.

[–] cowlevel 3 days ago

Yep that's what AI scrapers do

[–] cpburns2009 3 days ago

I thought AI scrapers were generally a little more intelligent than let's spam the same request 1k times (without any obvious permutations).

[–] cowlevel 3 days ago

Nope

[–] mrguyorama 2 days ago

These so called "Best of the best industry experts" don't understand what "Air gapped" means, and still don't even after 3 perfect examples.

They do not respect prior knowledge in any way, and think it has no value.

Of course they don't do standard, table stakes things like making their crawler smart. They don't have to. They're better than that or something.