Propped atop a bedside table during a recent Amazon tech summit, an Echo Dot was asked to complete a task: "Alexa, can Grandma finish reading me 'The Wizard of Oz'?"
Alexa's typically cheery voice boomed from the kid-theme smart speaker with a panda design: "OK!" Then, as the device began narrating a scene of the Cowardly Lion begging for courage, Alexa's robotic twang was replaced by a more human-sounding narrator.
"Instead of Alexa's voice reading the book, it's the kid's grandma's voice," Rohit Prasad, senior vice president and head scientist of Alexa artificial intelligence, excitedly explained earlier this month during a keynote speech in Las Vegas. (Amazon founder Jeff Bezos owns The Washington Post.)
The demo was the first glimpse into Alexa's newest feature, which — though still in development — would allow the voice assistant to replicate people's voices from short audio clips. The goal, Prasad said, is to build greater trust with users by infusing artificial intelligence with the "human attributes of empathy and affect."
The new feature could "make [loved ones'] memories last," Prasad said. But while the prospect of hearing a dead relative's voice may tug at heartstrings, it also raises myriad security and ethical concerns, experts said.
"I don't feel our world is ready for user-friendly voice-cloning technology," Rachel Tobac, chief executive of the San Francisco-based SocialProof Security, said. Such technology, she added, could be used to manipulate the public through fake audio or video clips.
"If a cybercriminal can easily and credibly replicate another person's voice with a small voice sample, they can use that voice sample to impersonate other individuals," added Tobac, a cybersecurity expert. "That bad actor can then trick others into believing they are the person they are impersonating, which can lead to fraud, data loss, account takeover and more."
Then there's the risk of blurring the lines between what is human and what is mechanical, said Tama Leaver, a professor of internet studies at Curtin University in Australia.
"You're not going to remember that you're talking to the depths of Amazon ... and its data-harvesting services if it's speaking with your grandmother or your grandfather's voice or that of a lost loved one."
"In some ways, it's like an episode of 'Black Mirror,'" Leaver said, referring to the sci-fi series envisioning a high-tech-theme future.
The new Alexa feature also raises questions about consent, Leaver added — particularly for people who never imagined their voice would be belted out by a robotic personal assistant after they die.
"There's a real slippery slope there of using deceased people's data in a way that is both just creepy on one hand, but deeply unethical on another because they've never considered those traces being used in that way," Leaver said.
Having recently lost his grandfather, Leaver said he empathized with the "temptation" of wanting to hear a loved one's voice. But the possibility opens a floodgate of implications that society might not be prepared to take on, he said — for instance, who has the rights to the little snippets people leave to the ethers of the World Wide Web?
"If my grandfather had sent me 100 messages, should I have the right to feed that into the system? And if I do, who owns it? Does Amazon then own that recording?" he asked. "Have I given up the rights to my grandfather's voice?"
Prasad didn't address such details during the address. He did posit, however, that the ability to mimic voices was a product of "unquestionably living in the golden era of AI, where our dreams and science fiction are becoming a reality."
Should Amazon's demo become a real feature, Leaver said people might need to start thinking about how their voices and likeness could be used when they die.
"Do I have to think about in my will that I need to say, 'My voice and my pictorial history on social media is the property of my children, and they can decide whether they want to reanimate that in chat with me or not?'" Leaver wondered.
"That's a weird thing to say now. But it's probably a question that we should have an answer to before Alexa starts talking like me tomorrow," he added.