This feels like an impossible assessment - yes, a model probably can't give you the education that a advanced/expert book on a topic will, but implying that having a verifiable goal is somehow fake learning feels like an intractable problem.

What level of evidence would be sufficient for you to accept that a model may be able to teach a concept?

I'm happy to take on this challenge with a topic of your choosing, but I don't believe there will be an evidence base that satisfies you that the knowledge is earned or deep enough.