But that'd still be vulnerable.

The issue isn't that you can read whatever was in/got brought into cache, the issue is that you can tell which part of the cache got evicted and infer values based on that.