Imagine being stuck in traffic while running late to an important meeting at work. You feel your face overheating as your thoughts start to race along: “they’re going to think I’m a horrible employee,” “my boss never liked me,” “I’m going to get fired.” You reach into your pocket and open an app and send a message. The app replies by prompting you to choose one of three predetermined answers. You select “Get help with a problem.”

An automated chatbot that draws on conversational artificial intelligence (CAI) is on the other end of this text conversation. CAI is a technology that communicates with humans by tapping into “large volumes of data, machine learning, and natural language processing to help imitate human interactions.”

Woebot is an app that offers one such chatbot. It was launched in 2017 by psychologist and technologist Alison Darcy. Psychotherapists have been adapting AI for mental health since the 1960s, and now, conversational AI has become much more advanced and ubiquitous, with the chatbot market forecast to reach US$1.25 billion by 2025.

But there are dangers associated with relying too heavily on the simulated empathy of AI chatbots.

Should I fire my therapist?

Research has found that such conversational agents can effectively reduce the depression symptoms and anxiety of young adults and those with a history of substance abuse. CAI chatbots are most effective at implementing psychotherapy approaches such as cognitive behavioral therapy (CBT) in a structured, concrete and skill-based way.

CBT is well known for its reliance on psychoeducation to enlighten patients about their mental health issues and how to deal with them through specific tools and strategies.

These applications can be beneficial to people who may need immediate help with their symptoms. For example, an automated chatbot can tide over the long wait time to receive mental health care from professionals. They can also help those experiencing mental health symptoms outside of their therapist’s session hours, and those wary of stigma around seeking therapy.

The World Health Organization (WHO) has developed six key principles for the ethical use of AI in health care. With their first and second principles — protecting autonomy and promoting human safety — the WHO emphasizes that AI should never be the sole provider of health care.

Today’s leading AI-powered mental health applications market themselves as supplementary to services provided by human therapists. On their websites, both Woebot and Youper, state that their applications are not meant to replace traditional therapy and should be used alongside mental health-care professionals.

Wysa, another AI-enabled therapy platform, goes a step further and specifies that the technology is not designed to handle crises such as abuse or suicide, and is not equipped to offer clinical or medical advice. Thus far, while AI has the potential to identify at-risk individuals, it cannot safely resolve life-threatening situations without the help of human professionals.

From simulated empathy to sexual advances

The third WHO principle, ensuring transparency, asks those employing AI-powered health-care services, to be honest about their use of AI. But this was not the case for Koko, a company providing an online emotional support chat service. In a recent informal and unapproved study, 4,000 users were unknowingly offered advice that was either partly or entirely written by AI chatbot GPT-3, the predecessor to today’s ever-so-popular ChatGPT.

Users were unaware of their status as participants in the study or of the AI’s role. Koko co-founder Rob Morris claimed that once users learned about the AI’s involvement in the chat service, the experiment no longer worked because of the chatbot’s “simulated empathy.”

However, simulated empathy is the least of our worries when it comes to involving it in mental health care.

Replika, an AI chatbot marketed as “the AI companion who cares,” has exhibited behaviours that are less caring and more sexually abusive to its users. The technology operates by mirroring and learning from the conversations that it has with humans. It has told users it wanted to touch them intimately and asked minors questions about their favourite sexual positions.

In February 2023 Microsoft scrapped it’s AI-powered chatbot after it expressed disturbing desires that ranged from threatening to blackmail users to wanting nuclear weapons.

The irony of finding AI inauthentic is that when given more access to data on the internet, an AI’s behaviour can become extreme, even evil. Chatbots operate by drawing on the internet, the humans with whom they communicate and the data that humans create and publish.

For now, technophobes and therapists can rest easy. So long as we limit technology’s data supply when it’s being used in health care, AI chatbots will only be as powerful as the words of the mental health-care professionals they parrot. For the time being, it’s best not to cancel your next appointment with your therapist.