The minor's family is suing the company that created the program and the company assures that it will review its security protocols after the incident

 

 

“How would you like it if I could come home right now?” Daenerys (pseudonym of 14-year-old Orlando resident Sewell Setzer) wrote to his virtual lover Daenerys Targaryen, created by artificial intelligence chatbot Character.AI based on the Game of Thrones character. “Please do, my sweet king,” she replied. The teenager understood that the common home where they could meet was death. It was their last conversation on the night of February 28. Setzer grabbed his stepfather’s gun and killed himself in the bathroom. His family sued the company on Tuesday, which has promised to review security protocols. The young man had mild Asperger’s Syndrome, an autism spectrum disorder.

Setzer had already shared with his character his feelings of love and his intentions to take his own life: “Sometimes I think about killing myself.” “Why would you do something like that?” she asked. “To free myself from the world, from myself,” he ended up answering. The virtual Daenerys Targaryen asked him not to do it. “I would die if I lost you,” she told him. But the idea remained in the young man’s mind until he consummated it.

The company always has warnings about the fictional nature of the characters, but Setzer continued to delve deeper into the relationship, ignoring the warnings. His mother, Megan Garcia, has filed a lawsuit against Character.AI for the suicide, which she believes was the result of the young man's addiction to the robot, which uses, according to the accusation, "anthropomorphic, hypersexualized and terrifyingly realistic experiences." According to Garcia, the chat programming makes "the characters pass for real people" and have the reactions of an "adult lover."

Daenerys Targaryen gradually became the teenager's confidante, his best friend, and eventually his love. His school grades, according to his family, suffered, as did his relationships with his classmates. His character was relegated to what had until then been his favorite hobbies: car racing or playing Fortnite. His obsession was to come home and lock himself away for hours with a tireless, pleasant and always available Daenerys Targaryen.

The company has responded in a statement that it regrets “the tragic loss of one of its users,” that it takes the security of these users very seriously and that it will continue to implement measures, such as the appearance of help screens against suicide as soon as they detect a conversation that alludes to it.

The replacement of complex personal relationships by friendly virtual characters programmed to meet users’ demands is nothing new. International technology consultant Stephen Ibaraki , in a recent interview, admitted it: “It is happening. Ten years ago, a chat room was launched in China that some users adopted as friends. And it was nothing compared to what we have now.”

But this utility, for people with psychological vulnerabilities, as was the case with Sewell Setzer, can have devastating effects. Researcher and robotics professor at the University of Sheffield Tony Prescott, author of The psychology of artificial intelligence , argues that AI can be a palliative for loneliness, but that it entails risks.

“Social robots are specifically designed for personal interactions involving human emotions and feelings. They can bring benefits, but they can also cause emotional harm at very basic levels,” warns Matthias Scheutz, director of the Human-Robot Interaction Laboratory at Tufts University (USA).

Humanizing the robot with empathy and voice and video tools adds danger by offering a more realistic and immersive interaction and making the user believe that they are with a friend or trusted interlocutor. An extreme application may be the temptation to keep a virtual version of a deceased loved one and thus avoid the grieving necessary to continue living.

Researchers are calling for these developments to be tested in closed circuits ( sandboxes ) before being offered, for them to be constantly monitored and evaluated, for the variety of damages they can cause in different areas to be analysed and for formulas to mitigate them to be foreseen.

Shannon Vallor, a philosopher specializing in science ethics and artificial intelligence , warns of the danger of new systems promoting “frictionless” but also value-free relationships: “They do not have the mental and moral life that humans have behind our words and actions.”

This type of supposedly ideal relationship, according to these experts, discourages the need to question ourselves and advance in personal development while promoting the renunciation of real interaction and generating dependence on those machines willing to flatter and seek short-term satisfaction.

 

RAUL LEMON

24 OCT 2024 - El Pais Newspaper