Hacker News

points by vanyle 3 days ago | 0 comments

Residential proxies are a very important part of the internet, they allow scraping for archiving, automation and can be used for extra anonymity.

While they have grown a bit too much, they are a natural response to the absurd amount of effort every website is pouring into anti-scraping tech.

[–] 10111001 3 days ago

> Residential proxies are a very important part of the internet, they allow scraping for archiving, automation and can be used for extra anonymity.

Aren't they almost always run unknowingly, by people tricked into running them?

[–] dabbz 3 days ago

There exist services that will pay you a fraction of money to host their residential proxy service on a computer you own. But I do think the majority of residential proxies are hijacked.

[–] kube-system 3 days ago

There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.

Ethically, if someone just wants to play a free game and doesn't care about it, is it okay? How knowledgable does a user have to be in order to have informed consent?

[–] 10111001 3 days ago

> There are providers who claim they are ethically sourced, and that users have consented. Although I guess many are users who don't know or care that it is happening.

Betcha that means there was "consent" language on page 55 of some click-through EULA.

[–] kube-system 3 days ago

The landscape spans the gamut of everything from malware to explicit consent: https://packetstream.io/share-bandwidth/

[–] xyzzy_plugh 3 days ago

Leveraging compromised devices is a very important part of the Internet??? This is an incredibly twisted perspective.

Help me understand. As far as I'm aware, these residential proxies are almost always malware.

[–] OptionX 3 days ago

And making DDoS-attacks nigh impossible to IP block.

Residential proxies are in many cases at least shady, both on the side of who uses them and for what and on providers that are often not above using dark pattern to hide them in some innocuous app for a smart-whatever that silently runs a proxy on a unaware user's connection. And in many places in the world if illegal stuff in done using your connection you are liable, regardless if you knew about it.

[–] georgel 3 days ago

I wouldn't call turning on CloudFlare's bot blocker an "absurd amount of effort" from the perspective of site owners.

But getting around it sure is annoying, sometimes I get flagged IP's from Comcast and can't visit sites like costco.com despite not doing any questionable stuff on my network.

[–] mcmcmc 3 days ago

Hard disagree. They are malware 99.9% of the time. The utility does not erase the fact they are abusive.

[–] sparkling 3 days ago

>absurd amount of effort every website is pouring into anti-scraping tech.

Maybe some people do not want to provide their entire website as free training material to [insert LLM lab here]?

[–] ethin 3 days ago

You... Do know that people scrape websites for non-AI reasons, right?

[–] anonreplier 3 days ago

Why should the site owners be interested in allowing them to do that?

[–] ethin 3 days ago

Because it's legal to scrape websites? And, oh, I don't know, maybe people like me might want to archive those websites for historical reasons?

[–] anonreplier 2 days ago

that's why you might want to do it, not why they would want to let you do it

[–] lakeviewterrace 3 days ago

"Malware is an important part of the Internet"

> and can be used for extra anonymity

Great, so you can conduct illegal transactions and have some unsuspecting grandma take the blame and have her front door kicked in?

[–] kube-system 3 days ago

Just like Tor, encryption, VPNs, etc, there are tons of legal uses for these proxies as well.

[–] lakeviewterrace 3 days ago

People who run Tor exit nodes generally do so by choice, and know the risks implications involved.

They didn't just connect a coffee pot to their WiFi one day.

[–] kube-system 3 days ago

Not all of these networks are built via malware.

e.g. https://packetstream.io/share-bandwidth/

[–] timacles 3 days ago

"Malware is an important part of the Internet

Technically correct statement though

[–] benjojo12 3 days ago

Did a residential proxy provider write this

[–] lakeviewterrace 3 days ago

Huh, and I wonder why I've been getting tons of "you might be a bot" CAPTCHA challenges from Google lately.

[–] mcmcmc 3 days ago

Frontier lab training data “collector”

[–] mikestew 3 days ago

Residential proxies are a very important part of the internet

So important that we have to trick people into using them. Remember when folks were running SETI@Home? Of course you do, because it was widely thought to be a good thing, and folks willingly signed up.

How many would buy the "box" if the advertising were honest about what's actually in the box?

[–] Bender 3 days ago

Most tools can be dual-use. Residential proxies are also used by malevolent actors to hack infrastructure or financial institutions and let some other poor soul get investigated for it. This is why we can't have nice things.

[–] kube-system 3 days ago

They're also used by the good guys who are going after those exact same bad guys.

[–] Bender 3 days ago

True, though unfortunately people can not be trusted to declare their intentions so the residential proxies will ultimately have to be sacrificed and given they are fairly easy to identify by behavior I could foresee the feds now having a reason to round 'em up faster. They already took out NetNut, IPIDEA, SocksEscort, Doppellganger and the "Kim Wolf" Offshoots.

[–] kube-system 3 days ago

Some of the more legit providers have KYC in place, and do literally require you to declare you intentions to open an account, and block certain traffic proactively unless you can prove a legitimate use.

[–] Bender 3 days ago

How do I identify which residential proxies have gone through KYC and which are wanting to pour hot sauce in my under garments?

[–] kube-system 3 days ago

The ones that do KYC list it pretty prominently on their website.

[–] Bender 3 days ago

They list the IP addresses of all the residential proxies? Is there a link to the netsets of all the IPv4 addresses they are using I can download once a day? (preferably over rsync but git over https is fine too)

[–] mzajc 3 days ago

s/Residential proxies/DDoS networks/, then re-read your comment.

[–] squidmaster 3 days ago

Virtually all of it is the product of malware. This is like you saying that robbery is very important because you like buying cheap tools at pawnshops.