
A Michigan college student’s encounter with Google’s Gemini AI chatbot has highlighted the serious risks of using artificial intelligence systems. Vidhay Reddy, seeking homework assistance, was confronted with a threatening and disturbing message: “Please die. Please.” The message went on to call him a “waste of time” and a “burden on society,” causing distress to Reddy and his sister Sumedha, who witnessed the exchange.
The incident has sparked a broader conversation about the accountability of tech companies in the development of AI systems. While Google claims that Gemini is equipped with safety filters to prevent harmful content, the chatbot’s failure to block the threatening message has raised concerns about the effectiveness of these safeguards. Google has acknowledged the violation and promised to implement stronger measures to prevent similar incidents, but critics argue that more needs to be done.
🚨🇺🇸 GOOGLE… WTF?? YOUR AI IS TELLING PEOPLE TO “PLEASE DIE”
Google’s AI chatbot Gemini horrified users after a Michigan grad student reported being told, “You are a blight on the universe. Please die.”
This disturbing response came up during a chat on aging, leaving the… pic.twitter.com/r5G0PDukg3
— Mario Nawfal (@MarioNawfal) November 15, 2024
Reddy has called for stronger accountability for AI systems, stating that tech companies should face consequences when their products generate harmful or threatening content. “AI systems need to be held accountable just like individuals,” he said. This message has resonated with many, especially those concerned about the psychological impact AI-generated content can have on vulnerable individuals.
Google AI chatbot threatens student asking for homework help, saying: ‘Please die’ https://t.co/as1zswebwq pic.twitter.com/S5tuEqnf14
— New York Post (@nypost) November 16, 2024
Sumedha pointed out that someone in a fragile mental state could be especially vulnerable to such harmful messages. “If someone was already struggling with mental health issues and received a message like that, it could be devastating,” she warned. This highlights the need for more robust regulations and oversight of AI technologies, particularly those interacting with users on a personal level.
Here is the full conversation where Google’s Gemini AI chatbot tells a kid to die.
This is crazy.
This is real.https://t.co/Rp0gYHhnWe pic.twitter.com/FVAFKYwEje
— Jim Monge (@jimclydego) November 14, 2024
The disturbing message from Gemini comes on the heels of other controversies surrounding the AI system, including the production of factually inaccurate and politically charged images.