Hacker News

points by mzajc 3 days ago | 0 comments

> benign

On a server that used to get below 1k HTTP requests a day, I've had several cases of mass scraping from these botnets exceeding 100req/s. The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.

How is DDOSing every site on the web benign?

[–] nucleardog 3 days ago

Relatively benign.

A compromised device inside the White House network could presumably be doing something _much_ worse than participating in a DDOS against random websites.

[–] mcv 3 days ago

And it's right there in the second line of the article:

> One of the proxy services is a Russian company whose founder’s web hosting business was ground zero for cyberattacks against election systems in Arizona and Illinois in 2016.

Surely a Russian company known for cyberattacks against the US having direct access to the White House network is far more serious than just DDOSing random websites. This is espionage at the highest level. (Again.)

[–] KaiserPro 3 days ago

The owner of the proxy is not great, but the fact that its an open door to snoop on everything else on the entire network is the realy fucker there.

[–] sandworm101 3 days ago

>> A compromised device inside the White House network

A whitehouse network. It isn't like everything done in that building run through a single wifi router. It is likely an unprotected network for personal use with zero connecting to anything important.

[–] nucleardog 3 days ago

If the progression of security over the last couple years has made anything clear, it's that letting the barbarians knock down a couple of walls because "well we have more walls" is not a winning defense strategy. The barbarians are very creative and will often exploit the inch you give in unexpected ways.

Great, they have some isolated network.

What are the chances it _never_ touches a device with a path to an important computer (e.g., one core router or switch at some point)? Good thing there haven't been any Cisco/Juniper/etc vulnerabilities ever.

What are the chances no important device _ever_ touches this isolated network? No important person ever gets annoyed at the security/filtering on the secured network and decides to connect to the public one instead? The secured one has some sort of issue, so their phone decides to connect to the guest network?

Are they going to compromise the military C&C network in the underground bunker? Probably not.

But it's probably more realistically damaging if they were to compromise some high level staffer's phone and using it as a listening device in a bunch of mid-level Whitehouse meetings.

[–] Spooky23 2 days ago

Remember all the senior people are engaged in blatant corruption and illegality.

So you're looking at not just national security stuff, but the ability to get personal data like selfies of fucking goats or screenshots of Maltese & Azerbaijan bank transfers, etc. Makes it easy for nefarious parties to manipulate standing with POTUS and blackmail people. The only escape for many of these people is pardons.

[–] BeetleB 3 days ago

> The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.

Indeed. I recently set up a bot on my local machine that I can interact with via Telegram. I gave it the ability to retrieve the contents of a URL via Playwright (with modifications). Basic things like getting the Readability view and getting the site as Markdown with links.

I was essentially mimicking the feature OpenAI etc offer where it can do a web search for you and get you the answer you seek. I just didn't want to pay a company for that feature.

It's not autonomous. It only acts when I tell it to. The actual search is via the Kagi API, but the URL contents is via Playwright. I also ensured it can't do lots of requests at a time. There's a mandatory (random) sleep of between 1 and 5 seconds between two requests.

In less than a week, I've been flagged. When I do a Google search (as a human), I have to prove I'm human. Many sites have simply blocked me - I can't buy stuff from various vendors[1], I cannot access several news sites, etc.

The thing is, I didn't really access that many sites with the bot! At most it's the equivalent of someone mildly addicted to the web. Yet they caught me quickly.

Obviously, I disabled the bot's capability to get site content. I wonder how long my IP will be in the blacklist.

I know some people will say "Good riddance" to the outcome and to what I did. I'd love to have a real, serious discussion on whether what I did was immoral. But whatever your thoughts on the matter, do ponder over the fact that the system has made it hard for average Joes like you and me to have this "feature", but if you pay a giant tech company a monthly fee, you can have it.

[1] Thankfully, Amazon didn't block me. In the past I'd avoid them, but now I'm in the dilemma where I don't want to buy stuff from sites that make it difficult for me to buy stuff from them!

[–] nostrademons 3 days ago

I think that major gateways like Google and Cloudflare and Akamai are doing things like TLS-fingerprinting and smearing.

I normally use a VPN to access the Internet. It's an ordinary consumer-grade VPN, and I use it for pretty innocuous things like watching the international versions of Netflix or avoiding being geographically-profiled and charged more when I book travel. The VPN itself, I'm sure, is being used for all sorts of naughty stuff: when I look it up in whatismyip, it comes up as VPN, "abuse detected", and "residential proxy detected". I expect to get blocked occasionally on it and just pause the VPN then.

What I found surprising is that pausing the VPN doesn't always clear the CAPTCHAs. The next request will also fail, sometimes. For a while I was wondering if I have a residential proxy on my network, but I went hunting with lsof and bandwidth monitoring and a list of all the open connections through my router, and it seems clear. Yet even with VPN disabled, I'll get CAPTCHAs from Google and Cloudflare.

I suspect they're using browser/TLS fingerprinting and then smearing the results of that across bad IPs. If you connect to them through a known-bad IP, they not only will CAPTCHA you, but they will record a fingerprint of any data they can. Then if you connect again with that fingerprint, regardless of the IP, they block that too.

Ironically, if you are actually malicious, you can get around this pretty easily. Because you'll know not to use a fingerprintable browser, or to rotate fingerprints. So you have one completely separate identity on a given IP (that is not yours), and if it gets blocked, you roll up another completely separate identity on a different IP. But this is too much work for an ordinary consumer who just wants to keep their IP private. So you end up with this excluded middle where the only rational strategies are to either do exactly what the website expects you to do (with no funny business, even if it's innocuous), or to go whole-hog as the aggressor where you're determined to skirt their policies no matter what, and if you're doing that you might as well amortize the effort by extracting everything of value you can from them.

[–] BeetleB 2 days ago

Yeah, I didn't want to put too many details as it would obscure my point. I was actually using a modified Playwright that's been designed to better act as a human. It works, in that I got fewer captchas etc than regular Playwright, but apparently not well enough.

That modified Playwright was based on Firefox, and that's what I use for my regular browsing. So that didn't help.

I can actually launch Chrome and not deal with this - but I'm not going to. In fact, what surprises me is that I can create a new Firefox profile and not get those captchas. I don't know how my using that modified Playwright as a bot got them hooked on to my Firefox profile. I suspect they had my FF profile as a "suspicious" entity even before I made the bot (I use a number of privacy/anti-cookie extensions), and the bot pushed the score over the edge.

I'll probably start fiddling with my profile - disabling extensions, until things seem to work. Thankfully, no one else in the household is impacted.

For now, I got around all this by just paying for the Kagi extract API. It's not as good as my modified Playwright had been, though. I thought of a VPN, but I suspect that won't last long (not a residential proxy, etc), and I don't want an ongoing monthly fee. I like Kagi's pay as you go option.

But as I said, it's a weird world where suddenly Amazon is coming out the "better" site/vendor. It's funny - in the old days I was pretty anti-Walmart, but a few years ago I started preferring them over Amazon (and even several local stores) for a number of reasons (better collection than local stores, not having to deal with fakes for online orders, etc).

I've not been liking the state of the web for years prior to the advent of LLMs, but this is feeling like a final straw where I accept "the web is dead". We need a fundamentally different protocol - the model of "Set up a server anyone can reach, put content out there, and then put up a lot of sophisticated technology so your server isn't hammered" is not tenable. The protocol itself needs to manage the load.

[–] proxyscore 3 days ago

Expect to remain blocked, that's how defense works, some services plenty use will have is bot or known proxy exit node set to true, and some websites use the same defense services, request velocity is just one factor, if you don't have random human like scrolling and jitter with a bot, it's a dead profile walking, metaphorically speaking

[–] 83 3 days ago

My residential network has been on Google's shit list for a few years now, I don't think it will ever go away. Like others on this thread if I see a Google recaptcha that site won't get my business as I have to exit since the captcha is an infinite loop.

[–] itishappy 3 days ago

I don't think this is an immoral application at all from your description, but would that be clear on the receiving end?

[–] hnthrow10282910 3 days ago

I just did this as well. I also added paywall bypass by default along with inline article chat. The internet is such trash these days.

I made my modifications at the network level and auto redirect a configurable set of sites to go to reader mode/paywall bypass by default with a MITM proxy and a custom CA cert I trust

[–] cowlevel 3 days ago

Just restart your home router to rotate your IP address.

[–] qlte 3 days ago

That is entirely ISP dependent and many don't actually work like that. Your cable modem gets a specific IP for as long as Comcast or whoever sees fit, restarting does nothing on the WAN side.

[–] consp 3 days ago

In my case a downtime of over 4 hours will switch it. Might just be lease time doing its job.

[–] Denatonium 2 days ago

On cable ISPs, oftentimes, changing the router's WAN interface's MAC address and then rebooting the modem will get you a fresh IP address, but rebooting the modem without first changing the router's WAN MAC address will get you the same IP address.

[–] drewnick 3 days ago

True, I've had the same IPv4 address since 2018 on Spectrum despite rebooting at least 10 times.

[–] prmoustache 3 days ago

This is simply called the dhcp lease time. Restarting has never forced a change of ip on a system using a dhcp client.

[–] theGeatZhopa 3 days ago

It depends on the DHCP part and your provider lease time. When you access the Internet through a modem and landline, it's an highly dynamical system, where it's difficult to guarantee every caller/user gets its own IP. Especially when IP sharing is used among the clients. Even more difficult it is with the mobile Internet. It's a constant connection loosing/reconnecting. Actually resetting and freeing up IPs for new devices, the whole time. With cable Internet (the TV cable) one is having a kind of stationary access point. The network provider can see it's topology and it never changes once a leafe has joined the network. I typically have the same access creds over months. Even when I restart my cable modem, it's still the same IP. When I was on modem, I could get a new IP every second of I wanted. Now I have to think twice, because I just can't get a new IP, even if is forcedly, not.

[–] proxyscore 3 days ago

Doesn't work like that, only on mobile phones you can toggle airplane mode to get a new ip assigned and cgnat makes sure mobile ips are "trusted" , but they fingerprint you beyond ip address

[–] TheDong 3 days ago

> I know some people will say "Good riddance" to the outcome and to what I did.

I will instead say "skill issue".

Using Playwright results in some very obvious and easy to detect fingerprints. I'd expect better of someone on hacker news.

Mine hasn't been detected as far as I can tell, so I know it's possible to do better.

[–] DANmode 3 days ago

Weird to hide basic stuff like random mouse-paths, mouse wiggle.

[–] lynx97 3 days ago

elitist nonesense. "Look I am soooo cool, and you are not as cool as me."

[–] franga2000 3 days ago

1) The proxy isn't the scraper. There are plenty of scrapers not using residential proxies.

2) They're clearly talking from the perspective of the user and network admin. A residential proxy virus is in fact quite benign for both.

3) No matter what, DoS is quite a low priority in terms of security. And it's not even really DDoS, you'll know a real DDoS when you're on the receiving end.

[–] account42 3 days ago

> A residential proxy virus is in fact quite benign for both.

That's only because it isn't being prosecuted appropriately.

[–] franga2000 3 days ago

Let's not get ahead of ourselves... If someone plants a bomb under your car, you drive to work, then it blows up while you're out for lunch, should you be prosecuted?

I'm all for creating consequences for people who negligently don't secure their systems and people get hurt because of it, but people who installed a mobile game that has a proxy library in it? Please, chill.

[–] cyanydeez 3 days ago

Compared to, russian intel scraping beyond demented word of mouth.

[–] miki123211 3 days ago

> The web is getting harder and harder to use

What are you doing to make this true?

I'm a screen reader user (which is as "weird and unusual" as it gets when fingerprinting mechanisms are concerned), and I am not really noticing much difference versus 2022 or so. Sure, there's little bits of anubis here or there, and a few sites (which I know for a fact are experiencing targeted DDoS attacks unrelated to AI scraping) having annoying Cloudflare verification screens, but that's about it.

My setup (other than the screen reader) is as vanilla as it gets. Safari, AdGuard for Ad blocking, no VPN except work VPN when necessary, no PiHole, no settings changed. If you're using something more esoteric, maybe try going mainstream and see if that improves your experience.

[–] coke12 3 days ago

What do you mean by “anubis” here?

[–] account42 3 days ago

It's a DRM middleware used to harass users of customized and privacy respecting browsers which is popular with many smaller websites.

[–] Aldqueath 3 days ago

I am not sure what you are doing to get "harassed" by anubis but I use privacy respecting browsers with ad blockers and vpns myself, and anubis is by far the least annoying, almost invisible even.

In comparison I get cloudflare recaptcha on every site that uses it, sometimes multiple times per session and sometimes not even letting me pass at all due to "error".

And if the website uses Google recaptcha then I close the tab because I know it will just end up in endless recaptcha hell.

[–] jacquesm 3 days ago

Google too. 'We are detecting unusual traffic from your address'. I don't know how often I've seen this and there isn't a byte that goes in or out on this connection without monitoring so if it really would be 'unusual' I would hope I would be on it before Google told me, especially not from a session that's been active for days or weeks.

[–] account42 3 days ago

If you need to compare the project to Google and Cloudflare in order to look less bad that's not exactly a good sign.

[–] andrepd 3 days ago

Come on... This thread has people saying their small website went from 1k requests a day to 100 requests per second. What would you have them do to mitigate this, I wonder? I would rather not have to have an anime girl picture stalling the website load for 1s, but I'd rather have the anime girl and see the website than no anime girl and no website.

[–] barbafant 3 days ago

Is that the one that shows an anime figure holding a magnifyig glass while it is checking your browser?

[–] notafox 2 days ago

You can remove the figure - here's uBlock Origin filter to block it from loading:

  ! Title: Hide Anubis Image
  */.within.website/x/cmd/anubis/static/img/*.webp$image

[–] VorpalWay 3 days ago

Yes, and at least in the past it did proof of work to verify that it wasn't a bot. Though apparently that is no longer effective against all bots.

[–] tempfile 3 days ago

Please substantiate this claim. What do you think makes it DRM? What kind of browser are you using that results in "harassment"? I use tor browser and other oddball firefox forks all the time, Anubis never gave me any trouble. I think it just requires you to compute some hashes.

[–] kasey_junk 3 days ago

It’s a software tool that specifically exists to allow content creators to enforce rules about how their content is used. It meets the dictionary definition of DRM.

It’s DRM that I think of as silly larping that doesn’t really bother me, but it’s clearly drm.

[–] tempfile 3 days ago

> It’s a software tool that specifically exists to allow content creators to enforce rules about how their content is used.

This is what I would like substantiated. I can go to a site behind Anubis, download the content on it, and do whatever I want with it. It doesn't do any kind of attestation of hardware or software. So - what makes it DRM?

[–] philipwhiuk 3 days ago

This is such a wild re-imagining of 'preventing uncountable AI crawlers perform DoS attacks on a whim to make line go up'

[–] furst-blumier 3 days ago

Not OP but most likely https://anubis.techaro.lol/

[–] kuschku 3 days ago

https://anubis.techaro.lol/

[–] NavinF 3 days ago

> 100req/s

While your server handles that, what does your CPU do with the other 99% of available cycles?

I run https://bunnystomp.com on an Always Free VM from a major cloud provider. It handles 1,930 req/s at 33 ms latency with no tuning/optimization. It's literally just python behind cloudflare.

[–] mzajc 3 days ago

You seem to know a lot about my server. From what I could see, all cores were pinned to 100%, but maybe my budget AMD CPU is hiding fantastical amounts of compute from the kernel. I'd love to know how to make use of it.

I have no idea what the site you linked does, but it doesn't look like a cgit instance to me; that's what got hammered on my end, and when hosting it, one very quickly finds out that many git operations are CPU-bound.

[–] NavinF 3 days ago

Funny. My site is a search engine and cgit is one of many tarpits that I block. Otherwise the index gets filled with garbage and the crawler hits TCP conn limits because the bytes trickle in too slowly.

[–] fn-mote 3 days ago

If you want credibility from the get-go, you need to be up front about your application. I certainly took it to be a generic host for an unpopular site in your first comment.

[–] yard2010 3 days ago

Nah this is hn, just trust people upfront