I recommend you immediately add identity verification (state-issued identification verification), set up appropriate secrets store for PII, and audit trail EVERYTHING your users are doing, storing the contents in a secure location. Yesterday. This service will be used to harm others, shortly. I do think that there are exciting, honest things that can be done with this service but you need to set up some friction for use. Know-your-customer rules are going to apply to this category in short time.
People here are talking about taking this service offline but I think everyone needs to be thinking about countermeasures, working on those services next. The genie is already out of the bottle. The degree of effort to put this together is low enough that it will be replicated around the world.
> "Hi Mom, I need some help. Some guys hit me over the head and put me in a van, and they're saying they'll kill me if you don't wire money to this bank account."
top class.
EDIT this was about one page down on the "see what people are generating" page
I mean, it kind of does matter? Since the start of this thread was a post about the imminent threat that play.ht posed, an example of it not appearing very dangerous is on topic.
It’s very possible that some other voice cloning software will be the undoing of the fabric of society, not this particular website.
It could be like dismissing the iPhone 1. It could also be like dismissing the Nokia Lumia.
Either way an unconvincing audio file is an unconvincing audio file regardless of what other files may become possible in the future through any number of various platforms and software implementations.
In the same vein, it is prudent not to mix up the factual current state of things with possible futures.
I don’t see the purpose of panicking right now. Are you using this anxiety as motivation to design better audio fingerprinting solutions? Identifying bad actors and vulnerable groups? Educating people on how to avoid being manipulated by fake audio?
What does posting “I am scared of the future” online accomplish?
> I recommend you immediately add identity verification (state-issued identification verification)
and
> The genie is already out of the bottle. The degree of effort to put this together is low enough that it will be replicated around the world.
are thoughts that end up in the same post?
If the genie is out of the bottle, it’s your proposed solution that everybody that runs a model like this implements bank-style KYC?
What do you propose should happen when this sort of software becomes freely available for everyone? When (not if) that happens, what will your suggestion have accomplished?
While I agree with you, the problem is far bigger than any one company in my opinion. These tools are already accessible enough to individuals that no audio or video is trustworthy, regardless of its source. I suspect we can still detect whether most faked audio/video is authentic or not algorithmically, but that's going to turn into an arms race eventually. And IMO none of the "answers" are ones that you really want to see made real, either.
I feel like this will be the thing that finally forces digital signing into the public eye. "Wait, is that video real?" "Well, it was signed by a reputable news source."
Right, which leads to a place where nothing is trusted unless it came from some central authority or from some trusted piece of hardware. I'm not looking forward to the day when I have to use e.g. an Apple or Google piece of hardware or some locked down kiosk or "be famous" in order to conduct business.
I assume trusted hardware would include things like LIDAR and biometrics, but if you're assuming those can be beaten then it will be a different kind of arms race, for sure.
I'm imagining the legal implications though I'm not a lawyer. If granny gets ripped off by someone impersonating me with this site, seems like Granny could sue Play.ht.
Play.ht will want to have as much information as possible about their users.
You are right, and unfortunately that is a possibility, and we are working on having measure in place to guard against such attempts. We have auto moderation on the input text that will block such audio being generated. Such users are flagged in the system.
While verification could be done for a cloud service like this one, what's more concerning is that locally run models with this tech will be coming soon (think of LLAMA and Stable Diffusion). KYC is merely a stopgap and honestly we'll need effective solutions for detecting vocal cloning impersonation in the future.
There's a podcast I listen to sometimes called "The Perfect Scam," sponsored by the AARP but, I suspect, is intended more for the kids of elderly people who are more at risk for these kinds of things:
They have quite a few stories about "virtual kidnappings" and interview the people involved -- it's quite interesting, and has given me a lot of insight into how typical it would be for people to hear panic and react with panic...precisely how these scams are intended to go.
Couldn't agree more with your comment. We are working on counter measures like manual verification of voice, a classifier to detect cloned speech, etc. As of now we have auto moderation in place that detects and blocks hate/harmful speech.
I actually have one thousand HackerNews good boy points, so I'm kind of a big deal
I think that a few years from now this tech is going to be ubiquitous, real time, and work on a mobile device. Trying to slam the lid shut on Pandora's Box probably isn't going to work.. the best thing at this point would be for the word to get out to everyone that voices can now be doctored the same way photos can
Gasp! Yawn. HN has become so pearl-clutchingly alarmist recently. Everybody relax.
The solution to scams is to educate people on scams, as quickly as you can do so in the changing environment, by publishing information about what's possible with the latest technology. The solution is not to require onerous identity verification for every software product that could be used by scammers, because they'll just move to the next product that doesn't require it, or they'll simply provide fraudulent documents. Or you'll get "resellers" who provide their own fraudulent KYC documents and then sell access to their account to other criminals on the black market, making it even more difficult to monitor for abuse.
If you want a startup offering such tools to protect people from scams, they can do it by collecting data on what the tools are used for - it should be pretty obvious based on transcripts who is using it to scam people.
People here are talking about taking this service offline but I think everyone needs to be thinking about countermeasures, working on those services next. The genie is already out of the bottle. The degree of effort to put this together is low enough that it will be replicated around the world.