Yes I wish there was a way to opt out and just rely on phone support. At least with phone support the attackers would have to social engineer things at an individual records level versus just being able to brute force huge bulk data records out.
Most doctors offices are effectively overworked and understaffed small businesses. Shifting all that support to the phone would not be reasonable, from a workload perspective.
Just to expand on that, patients aren't the only ones with a legitimate reason to access that data. Insurance companies need copies as well, so they would still want some kind of mass-data portal even if customers don't use it. If you change doctors, they want copies of your old medical records. If you see a specialist, if you go to the hospital, etc, etc. Pharmacies might call and ask if they think there's something weird about a prescription.
Direct patient contacts are a vanishingly small percentage of records requests for a doctor. Doctors could likely handle those via phone, but it doesn't solve the issue of needing an EMR.