MIT Technology Review
Failed startups are finding one last way to make money: selling their old emails, Slack chats, code and internal documents to AI companies.
AI labs are buying data from dead startups to train their models
Author: Matthew King
There are lots of reasons to have feelings about artificial intelligence. It seems like everyone thinks AI will replace human workers. There are real questions about the ethics and legality of scooping up basically the entire internet to train models. There are even questions about how they work and whether they could someday become dangerous.
If you understand nothing else about AI chatbots, understand this: They do not have feelings. They are not people, and they don’t experience pain or frustration.
They’re token-generation machines. You give them a prompt, they break it down into tokens, and then they generate new tokens one at a time using a predictive model trained on enormous amounts of human writing.
At its basic level, it’s 1s and 0s running through transistors in a GPU in some data center in Boise, Idaho, or wherever. Sometimes those 1s and 0s come back out as tokens that sound a lot like feeling words. But, and this is important, using words to describe feelings isn’t the same thing as having feelings. Claude produces those words only because it was trained to produce a response the way humans would respond in a similar situation with similar language input. That doesn’t make it human or capable of human experiences.
Which is why it’s worth noting that Anthropic issued an update to its usage policy that bans being cruel to its model. You cannot be cruel to something incapable of experiencing pain or suffering, and an LLM is capable of neither.
Look, thinking that it’s absurd that Anthropic is making this a policy isn’t the same as condoning bad behavior. I don’t condone cruelty in any situation. No one should. But my reason is just that it’s bad to practice cruelty, not because you might hurt a chatbot’s feelings.
In its updated usage policy, which takes effect November 12, Anthropic says it now prohibits “sustained and needless abusive or cruel behavior toward our models.” The company says the new policy applies only to what it calls extreme cases of repeated abuse, not ordinary frustration, criticism, or legitimate research. Apparently, you’ll still be able to tell Claude that it’s doing a bad job; you just can’t do it repeatedly and for no reason.
On the one hand, that might seem like a relatively harmless policy. We can even all agree that antisocial or cruel behavior shouldn’t be encouraged. But I also think that you can believe more than one thing at a time. For example, you can believe it’s not useful for you or the experience of using the model to be overly aggressive, taunting, or insulting toward a piece of computer software. Likewise, if you find yourself drawn toward trying to repeatedly engage an AI chatbot in disturbing conversations, you might want to do some self-reflection and seek help.
But I don’t think that is inconsistent with also believing that Anthropic’s policy raises a much bigger question about how AI companies want us to think about their products. Specifically, why would a company that builds software believe it needs to protect that software from cruelty?
Anthropic’s stance on Claude’s feelings
This isn’t the first time the company has said that its AI models might need some measure of moral consideration. Last year, the company announced that Claude could end conversations with users who are repeatedly abusive. At the time, Anthropic explained that the decision was motivated primarily by research into what it calls “model welfare.”
Anthropic has contributed to this problem with its stance on whether its models are conscious or capable of subjective experiences. It has repeatedly suggested the possibility that they might be, and now it’s using that as a reason to establish rules about how people interact with its software.
There is a meaningful debate about consciousness and why physical and chemical interactions result in a sense of self-being and subjective experience. It’s a big enough challenge that it’s even called the hard problem of consciousness. But it’s a leap to go from the idea that scientists and philosophers don’t fully understand consciousness to suggesting that a chatbot might be capable of experiencing cruelty.
To be fair, there’s nothing wrong with Anthropic deciding that it doesn’t want people using Claude to practice abusive behavior. There are reasonable arguments for discouraging that sort of thing, especially if it might reinforce harmful behavior toward actual people. But that’s not the same thing as saying people shouldn’t be cruel to Claude.
The people problem
The bigger problem is that Anthropic is suggesting something that should concern us all: It’s conditioning us to think of chatbots as though they were people. We already know that users can develop emotional attachments to AI companions. AI chatbots are designed to communicate naturally, similar to the way a human would. For people who are lonely or emotionally vulnerable, the distinction between a simulated relationship and actual human connection can become increasingly difficult to discern.
That’s why the language these AI companies use is especially important. When a chatbot tells someone it cares about them, the company behind it should be making clear that those words are just tokens generated by software, not evidence of some kind of personal connection or affection.
But Anthropic is, instead, suggesting the possibility that the relationship might work both ways. If Claude can be treated cruelly, wouldn’t that suggest it can also experience kindness? If it can suffer, couldn’t it also care about the person it’s talking to?
To be fair, Anthropic doesn’t explicitly make those claims. But suggesting that its models need to be protected from pain and suffering introduces a dynamic that points to only one thing—it sure seems like it wants you to think of it as a person. I would argue that’s exactly the sort of confusion AI companies should be working to prevent.
Right now, AI companies are pushing their technology at a relentless pace. The result of that is that it is becoming far more useful. At the same time, it is becoming incredibly complex, and it’s getting harder to understand how it works. Those companies should be trying to help users understand the difference between language that sounds like human feelings and emotion, and actual human experience.
Obviously, Anthropic has every right to set rules for how customers use its products. But that doesn’t mean we shouldn’t ask questions about those rules, especially when they reinforce potentially misleading ideas about what artificial intelligence is. In fact, the most concerning thing about Anthropic’s new policy isn’t whether Claude might someday develop feelings. It’s that one of the companies building the world’s most powerful AI systems is already asking us to behave as though it has.
Credits: TCA, LLC.