I don't think it's a cloud. It's more likely a residential proxy network, which are typically created by installing malware on users' machines.
The operators of these proxy networks want to avoid detection by both the users whose bandwidth they're stealing, and by the companies whose data is being scraped. So they want to make the bandwidth very expensive. And that expensive bandwidth in turn means that their only clients are dodgy as well. Either people looking to scrape data without consent and monetize it, or outright criminals.
I use one. I run a bot on IRC that extracts the <title> of every link posted (or downloads the image/whatever and extracts Metadata) and announces that to the channel. It has become more and more pointless to run this on a vps. Google/YouTube block the IP range, a lot of websites return the cloudflare security check, Amazon works on some days and doesn't on others... Ever since I proxy via residential proxies it just works. I'm a smooth criminal. :>
To stop spammers, you implement measures before posting, not before viewing. Spam is just a minor technical nuisance. It's automated interaction that really makes their executives sweat and shiver.
install a fresh wordpress and you'll be blocking hundreds of Digital Ocean Ip addys within 24 hours..
Or you won't.
If you add a failed login detector you'll see DO, OVH Hetzer and similar are non-stop password guessing your logins everyday.
Maybe your admin password is so long they will never get in, maybe you don't care that your server gets slower responses for actual users as it serves pages and does DB lookups for all the bots trying to use a different password every minute.
Maybe the website owner doesn't know any of these and the open internet is actually legoland where everything is awesome all the time.
It's kind of surprising that a presumptively legitimate company (and YC-funded startup) would out themselves as buying black market residential proxy bandwidth, isn't it?
Their frontpage also advertises the ability to pass CAPTCHAs, whether by automation or more likely by delegating them to third-world CAPTCHA farms. If that's a major selling point for your automation service then your target market probably ranges from dubious (e.g. data scrapers trying to get around limits) to extremely dubious (e.g. ticket scalpers, spammers, click fraud, etc).
Just because something can be used for sketchy purposes doesn't mean that's the only purpose of it. there are thousands of situations where people are forced to interact with a shitty website 100x per day and the site won't provide an api. Imagine if your job was booking plane tickets all day. United could provide you an API key to do so via an API, but in practice they won't, only some enterprisey travel software company can get that kind of access, for a steep fee. You could build a tool which automatically puts together an itinerary based on rules and books it, through a tool like this. Perhaps a slightly contrived example but I believe things like this definitely happen.
> United could provide you an API key to do so via an API, but in practice they won't, only some enterprisey travel software company can get that kind of access, for a steep fee. You could build a tool which automatically puts together an itinerary based on rules and books it, through a tool like this. Perhaps a slightly contrived example but I believe things like this definitely happen.
And you think that's NOT sketchy?
I'm almost afraid to ask where you think the bar is...
And why is it? A company provides you an API for a "fee" and a free web-based interface, as long as you are agile enough to use it, with some limitations per ip/cookie. You choose the second path and automate it. What's wrong with that? Limits of the free access are the public contract. You're not obliged to play along with someone's "monetary spirit".
And in practice, APIs are often much more PITA than the actual interface, but you can't buy unlimited web automation. Few years ago one of my projects literally OCRed data from an android phone screen because receiving it via API took a couple minutes longer and involved email-like back and forth with polling and id matching after a convoluted authentication that fails every few requests.
I really wish I was a better programmer with more time, I would install the accursed "MyQ" garage door app on a dedicated Android, and bridge it into Home Assistant using an OCR type of strategy. (they are notorious for flipping the bird to the whole open home automation community by not integrating with anything)
A very common and pro-consumer use for residential proxies is price scraping and price comparisons.
Most businesses don't want to compete on price and are extremely unhappy if you tell people that their competition sells the same stuff but for less, that their "best deal of the month" is actually a price raise, or that they significantly raise toilet paper prices every time there's a natural disaster.
Agreed. Just for reference, one of our most popular use-cases is automating data entry into CRMs without APIs... No one wants to be doing this stuff manually, and automating it has some serious positive QoL impact
We get a lot of requests for bad usage (ie spinning up upvote rings on Reddit) but we don't want to support things like that
> one of our most popular use-cases is automating data entry into CRMs without APIs... No one wants to be doing this stuff manually, and automating it has some serious positive QoL impact
No-one would need captcha automation or residential proxies for a use case like that that's all on the level.
>one of our most popular use-cases is automating data entry into CRMs without APIs... No one wants to be doing this stuff manually, and automating it has some serious positive QoL impact
Good question! A lot of CRMs have CloudFlare enabled with location based blocking by default, so we needed a way to spoof a local location to be able to interact with the website
> Imagine if your job was booking plane tickets all day. United could provide you an API key to do so via an API, but in practice they won't, only some enterprisey travel software company can get that kind of access, for a steep fee.
Even your example sounds sketchy though. If you're not legitimate enough to use the enterprise software, why are purchasing tickets all day? And why do you need to proxy your bandwidth instead of just accessing the site directly. And why aren't you concerned about the fact that it's a crime to bypass the approved channels by hiding behind the proxy to do this?
Imagine a legitimate travel agency cannot book 100 United tickets a day via methods outlined in business contracts and need to resort to shady practice.
Dude, please provide some real solid evidence to back this up, and perhaps come up with another realistic scenario where bypassing captcha is justified.
> Imagine a legitimate travel agency cannot book 100 United tickets a day
That's the whole point, I never said travel agency, I was thinking a company with travelling consultants.
How TF is it "shady" to purchase and use airfare?
And again, bypassing captcha, say, to purchase tickets isn't evil either, if you are purchasing them for use and not for resale. It would just allow a person to book tickets for 50 people without wasting 6 hours to complete 25 CAPTCHAS and type in my information 25 times.
CAPTCHA is a blunt instrument deployed in an attempt to mitigate abuse, but it has a massive bad side effect that for every heavy user (not just evil users), it requires a human butt to be in a seat somewhere to do mindless busywork that could otherwise be automated. Working around that (sounds like OP agrees to do so on a case by case basis) is not inherently evil. It's as evil (or benign) as whatever you're using it for.
You ever see that video of the women paying a thousand dollars to skip to the front of the release day line to buy one of the first generation iPhones?
Then when she did and the employees told her they limited customers to buying one or two iPhones per person she becomes incredibly flustered. The guy who sold his spot in the line celebrates with a free phone.
What you’re describing is analogous and there’s a reason that went viral on the internet and was reported on in the mainstream, but I won’t spell it out for you.
It's almost never done with the full understanding of the person providing the proxy, doesn't matter if they get promised some change, their browser addons betray them or they install bundleware/adware.
I'd say it has about the same moral standing as a payday loan.
The look of surprise on their faces is almost universal when the feds knock on the door with a search warrant in relation to something that came from their IP address.
There's many reputable residential proxy networks too, usually there's a lot of vetting involved too as they don't want people running illegal activities though their network.
It's almost a necessity these days to have access to that due to how much datacenter ranges are blocked.
It’s not necessarily malware. There are services that are pretty upfront and pay cash money for residential US bandwidth. That said, naive people might be surprised when their IP starts getting blocked.
how does expensive bandwidth equate to dodgy clients? There are lot's of valid use cases for scraping data, and it's legal to scrape publicly available data, even if the websites hosting it try to block it (try a curl request to reddit, for example)
>>>and it's legal to scrape publicly available data, even if the websites hosting it try to block it
Is that something that's been fully decided? https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc. is the most relevant case I'm aware of, and it suggests it might actually be illegal (if you know you've been blocked, at least).
The operators of these proxy networks want to avoid detection by both the users whose bandwidth they're stealing, and by the companies whose data is being scraped. So they want to make the bandwidth very expensive. And that expensive bandwidth in turn means that their only clients are dodgy as well. Either people looking to scrape data without consent and monetize it, or outright criminals.