This isn't about blocking "misbehaving" AI bots. This is about blocking the competitors to the big boys like OpenAI and Anthropic.
I help administer a somewhat active (10-20 thousand hits/day) site that sits behind Cloudflare.
ChatGPTBot has been a menace - crawling several pages per second and going deep into site for years old content, which is polluting/diluting the cache. It also happens to be hitting a lot of pages that are very 'expensive' to generate. it also ignored a robots.txt file change for almost two full days.
Yet...I try to crawl my municipality's shitty website because there are announcements there that are made nowhere else and they're too lazy to figure out how to set up email announcements...and Cloudflare instantly blocked my change detection bot running on my home server. It hits one page every 24 hours, using a full headless version of Chrome. BZZZZT - cloudflare's bot detection smacks it upside the head.
If you think this is by chance or they don't know this is happening: bridge for sale etc.
This is just more collusion with other large tech firms, working to kill each other's competitors, small services and sites, and innovators. Really cute, given half of SV got where it is by "disrupting" things (ie breaking laws and regulations - it's cool bro, It's An App!)
Gmail will allow endless amounts of shit to stream into my inbox from "email marketing service" companies like mailchimp because I bought something 6 years ago from that company - but the second I need an email from a small community group mailing list that uses their own email server - a domain I've sent and received numerous emails to *and repeatedly clicked "Not spam" for - Gmail still keeps right on sending it to spam. I've checked. Their domain and IP range are both completely clean. It's simply Google saying "this wouldn't be happening if you were using Gmail for your domain's email."
We desperately need to claw the internet back from these corporations or it will only get worse. Remember when you could run a web server on dialup and nobody fucking cared? Now you even so much has have port 443 open for some self-hosted stuff only you know exists and your ISP bitches a fit. Remember when you could use any client you wanted for services like AIM, but now we have Slack and Discord and they'll ban you for using a non-official client?
I remember reading someone's shower thoughts that if the internet was completely safe there would be no need for Cloudflare, so it's in Cloudflare's best interest to keep the internet unsafe. It's an interesting thought even if a bit tinfoil hat-esque.
I help administer a somewhat active (10-20 thousand hits/day) site that sits behind Cloudflare.
ChatGPTBot has been a menace - crawling several pages per second and going deep into site for years old content, which is polluting/diluting the cache. It also happens to be hitting a lot of pages that are very 'expensive' to generate. it also ignored a robots.txt file change for almost two full days.
Yet...I try to crawl my municipality's shitty website because there are announcements there that are made nowhere else and they're too lazy to figure out how to set up email announcements...and Cloudflare instantly blocked my change detection bot running on my home server. It hits one page every 24 hours, using a full headless version of Chrome. BZZZZT - cloudflare's bot detection smacks it upside the head.
If you think this is by chance or they don't know this is happening: bridge for sale etc.
This is just more collusion with other large tech firms, working to kill each other's competitors, small services and sites, and innovators. Really cute, given half of SV got where it is by "disrupting" things (ie breaking laws and regulations - it's cool bro, It's An App!)
Gmail will allow endless amounts of shit to stream into my inbox from "email marketing service" companies like mailchimp because I bought something 6 years ago from that company - but the second I need an email from a small community group mailing list that uses their own email server - a domain I've sent and received numerous emails to *and repeatedly clicked "Not spam" for - Gmail still keeps right on sending it to spam. I've checked. Their domain and IP range are both completely clean. It's simply Google saying "this wouldn't be happening if you were using Gmail for your domain's email."
We desperately need to claw the internet back from these corporations or it will only get worse. Remember when you could run a web server on dialup and nobody fucking cared? Now you even so much has have port 443 open for some self-hosted stuff only you know exists and your ISP bitches a fit. Remember when you could use any client you wanted for services like AIM, but now we have Slack and Discord and they'll ban you for using a non-official client?