Hacker News

points by eatonphil 1 day ago | 0 comments

I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.

I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.

[–] EvanAnderson 1 day ago

I mind a public access website for a government entity and this is exactly the pattern I see. US residential IPs make a couple of requests then never come back.

The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.

I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HTML as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.

[–] Bender 1 day ago

Bots give all the information required to block or rate limit them including those that try to look like a real browser (some things they just can't hide, even residential proxies via behavior.). Ask the LLM how to change your log format to gather all the fields your web server can see. Share the resulting updated logs with your favorite LLM. Ask it for generalized rules to block the bots whilst permitting real browsers. A tiny percentage of bots will still get through but the load becomes much more manageable. Tell your LLM you also wish to spot fake chrome via CH and bots via ssl curves. No need for JA3. Nginx can expose the important aspects of GREASE.

[–] eatonphil 1 day ago

I know you can't be super specific because you'd just be enabling the companies who get around this (who, funnily, I also index on The Consensus), but I have spent a few rounds on this over the months and haven't yet been able to cut them out.

Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.

I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.

All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.

edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.

edit 2:

> tiny percentage of bots will still get through but the load becomes much more manageable.

The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.

[–] twosdai 1 day ago

You can consider adding in a small amount of js that tracks if a person actually engages with the content before "counting" their view. Eg, did they stay on the page for more than 15 seconds.

Youtube wont count a view until you watch for some percentage of the video. Maybe consider something similar.

[–] snackbroken 1 day ago

Funnily enough, Youtube recently changed how they count views to be more in line with its competitors, so now they do count views as soon as the video starts playing.

[–] eatonphil 1 day ago

Yeah I do have the JavaScript side in place, was just loathe to start totally owning this data instead of Fathom.

[–] twosdai 1 day ago

You could still have fathom own it, but create a sort of small proxy that takes in the view requests and then forwards them on to your instance behind the scenes. That way you can be responsible for counting who is actually a view and fathom can own the data and do all the BI things.

Just my two cents, you'll figure out a good solution.

[–] Bender 1 day ago

Aside from the things I suggested asking LLM's (they will take one down some fun rabbit holes) there are more crude methods such as forcing brotli or zstd on specific URLs accepting that older versions of Safari pre-2026 will get gibberish if forcing zstd. It's rude and paternal but botters can be rude and childish so these are the games we play.

I love sharing methods with the botters so they step up their game, thus forcing me to step up my game. Everything I use does not require javascript, cookies or authentication. I save authentication as a last resort which means I lost that round.

Botters are welcome and encouraged to test their setup on my silly hobby sites. Here's one of them: [1]

[1] - https://altchan.org/sh/33

P.S. These are the simplest crude methods for the obvious bots, let your LLM to help spot the fakers. pre-2026 Safari will not work. Do not visit image boards with unpatched browsers.

[–] ljlolel 1 day ago

seems like open web is dead which is sad

i heard hcaptcha is good

[–] j45 1 day ago

Static web pages might be back for a bit while this gets figured out