Why kind of data that isn’t public would be so valuable for AI training?

Seems like there’s a fuck ton. All of Wikipedia, GitHub for code, etc.

I can understand targeting certain sites like Reddit, etc. but not random websites

It's to rip off copyrighted content and profit from it instead of the original authors. It's like every other low rent and highly automated scam that finds it's way onto the internet.

If you look closely even Google does this. This is probably why many popular sites started getting down ranked in the last 2 years. Now they're below the fold and Google can present their content as their own through the AI box.

Please remember that Google only needs to be marginally better than the competition. And, of course, their primary biz is ads, not serving great results; that is a distant second priority.

Their biz is ads, but since search is winner takes all they need only be marginally better than the competition... twenty years ago.

> Their biz is ads,

Yea, but, the FTC doesn't want it to be.

Discord I guess would be quite valuable, even the de facto public servers.

Honestly it's hard to tell how much more value the LLM people are going to get out of another copy of the internet.

It feels a lot like they're stuck for improvements but management doesn't want to hear it.

I’m not sure what you mean by breakthrough, but if you’re talking about Deepseek, it’s more of an incremental improvement than a breakthrough.

Scraping social media is good data, even without ML. The fact that something is "happening" to people in a social space inherently has importance to people. The specter of law is more threatening to whether companies can get their hands on good data.