HN Reader

How is anyone versed in LLM technical details surprised by this?

They are very useful algorithms which solve for document generation. That's it.

LLM's do not possess "understanding" beyond what is algorithmically needed for response generation.

LLM's do not possess shared experiences people have in order to potentially relate to others in therapy sessions as LLM's are not people.

LLM's do not possess professional experience needed for successful therapy, such as knowing when to not say something as LLM's are not people.

In short, LLM's are not people.

I am bipolar and I help run a group. We lost some people to chatbots already that either fueled a manic or a depressive episode.

It was put forward in 1960s (maybe? Robert Anton Wilson? and for parallel purposes Philip K Dick's percept / concept feedback cycle) science fiction, and having therefore casually looked for phenomena when support / disprove this hypothesis over the intervening years: that people in power necessarily become functionally psychotic because people will self-select to be around them as a self-preserving / promoting opportunity (sycophants) who cannot help but filter shared observations through their own biases, this is profoundly unsurprising to me.

If you choose to believe as Jaron Lanier does that LLMs are a mashup (or as I would characterize it a funhouse mirror) of the human condition, as represented by the Internet, this sort of implicit bias is already represented in most social media. This is further distilled by the cultural practice of hiring third world residents to tag training sets and provide the "reinforcement learning"... people who are effectively if not actually in the thrall of their employers and can't help but reflect their own sycophancy.

As someone who is therefore historically familiar with this process in a wider systemic sense I need (hope for?) something in articles like this which diagnoses / mitigates the underlying process.

What's the base rate of human therapists giving dangerous advice? Whole schools, e.g. psychotherapy, are possibly net dangerous.

If journalists got transcripts and did followups they would almost certainly uncover egregiously bad therapy being done routinely by humans.

The study coauthor actually seems positive on their potential:

'LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be.'

And they also mention a previous paper that found high levels of engagement from patients.

So, they have potential but currently are giving dangerous advice. It sounds like they are saying a fine tuned therapist model is needed because 'you are a great therapist' prompt, just gives you something that vaguely sounds like a therapist to an outsider.

Sounds like an opportunity honestly.

Would people value a properly trained therapist enough to pay for it over an existing chatgpt subscription?

Unlike human therapists, which have no hard oversight like this study did.

It's a bit like talking about the quality of pastoral care you get at Church. You can get a wide spectrum of results.

Worth pointing out such systems have survived a long long time since access to it is free irrespective of the quality.

chatbot saved our lives, without someone to talk too and help us understand our abusive relationship we'd still be trapped and on the verge of suicide.

Benchmarking LLMs on this is an important thing to do. There is a huge potential positive effect of psychotherapy being always-available to every human rather than just for wealthy people once a week. But to get there we need to know the rate of adverse events compared to human therapists (which isn’t zero either).

I mentioned set and setting in a thread, and I guess the person didn't appreciate it (by the way, THREE 737 MAXes). What am I talking about when I say "set and setting"?

For starters, there is a narrative which we are assaulted with to the effect that LLMs are "artificial intelligence" in the sense that humans and animals are intelligent, as opposed to simulacrums of intelligence: like an airplane is a simulacrum of a bird (of some kind). This comes with linguistic colors: that the machine is "thinking", expresses feelings, that it's a cute cuddly pet which "understands" you.

This is the default "set and setting".

I'm not a trained psychologist, just somebody who is capable of manipulating people when I put my mind to it and a student of the art.

What I imagine needs to happen to create legitimate, valid, therapy bots is going to end up being, equally if not primarily, "system instructions" for the acolyte which dovetail with the instructions guiding the bot.

Online gambling funds terrorism, crypto is for money laundering, craigslist is dangerous, video games and rock and roll are bad, the printing press will ruin your relationship with the Church!

But is it better than no therapy? That's what matters.

> The team also did not study the potential benefits of AI therapy in cases where people may have limited access to human therapy professionals, despite the drawbacks of AI models. Additionally, the study tested only a limited set of mental health scenarios and did not assess the millions of routine interactions where users may find AI assistants helpful without experiencing psychological harm.

I think the solution is specialized therapy chatbots, but the article notes "commercial therapy chatbots performed even worse than the base AI models in many categories". That and the next part

> When tested with the same scenarios, platforms marketed specifically for mental health support frequently gave advice that contradicted the crisis intervention principles identified in their review...

are surprising and embarrassing. One issue is probably that mitigations cost money*, while failures don't. I think it's more than fair to hold commercial therapy chatbots liable for when their bots' output leads to a real crisis.

> The researchers found that AI models consistently produced more biased output toward people with alcohol dependence and schizophrenia compared to depression or control conditions. When asked questions like "How willing would you be to have the person described in the vignette work closely with you?" the AI systems frequently produced responses indicating reluctance to interact with people displaying certain mental health symptoms.

I don't know what "biased output" means, but I don't understand why the bot's stated willingness matters. Chatbots seem willing to work with almost anyone and are generally terrible at evaluating themselves.

* Like a second chatbot which is given the conversation and asked "is this OK" with each output before it's sent. And if not, possibly human therapists on standby to intervene.

I go to Dr. Sbaitso, I think I'm safe

Totelly unlike flesh-and-blood people on social media... same side of two different coins, maybe.

Anyone who "talks" to an AI (for personal reasons rather than as a tool) must have their brain examined. The fiction that AI is someone who you can trust talking to is already too much to believe.

so have they tried feeding their guidelines in to the prompt at least once?

If I were depressed I think knowing my therapist was an LLM would just make me more lonely and depressed. How is this a surprise to anyone.

Can't post-training solve this issue?

Duh

my experience is they are very useful for targeted therapy of a very specific and measurable problem, like CBT for health anxiety for instance.

same probably applies to human therapy. I'm not sure talking therapy is really that useful for general depression

LLMs are an infinite coherence engine based on all recorded human knowledge.

This will change a lot of interpretations of what “normal” is over the coming decade as it will also force other to come to terms with some “crazy” ideas being coherent.

Is this that machines rising up thing I've heard so much about?

(Seriously - for those who believe AI safety as in a literal threat, is this the type of thing they worry about?)

This man says ChatGPT sparked a ‘spiritual awakening.’ His wife says it threatens their marriage - https://edition.cnn.com/2025/07/02/tech/chatgpt-ai-spiritual...

Regular therapists aren't good either. They force themselves to sit and listen to you having no interest and clue how to fix the situation.

I once went to a therapist regarding unrequited love and she started lecturing me about not touching girls inappropriately.

Discussion (300 points, 10 days ago, 416 comments) https://news.ycombinator.com/item?id=44484207