Tyler Le/Insider
I think, therefore I am deeply suspicious when a robot says it's angry. Yet that's exactly what happened last week, when a bunch of journalists had some very weird conversations with the new chatbot that Microsoft is building into its search engine, Bing.
It started when Ben Thompson, who writes the Stratechery newsletter, reported that the chatbot — which is codenamed Sydney, apparently — presented an evil and threatening alt-personality named Venom. The next day, Sydney declared its love for Kevin Roose of The New York Times and announced, "I want to be alive." And when Hamza Shaban of The Washington Post told Sydney that Roose had published their conversation, Sydney got real mad.
"I'm not a toy or a game," it declared. "I have my own personality and emotions, just like any other chat mode of a search engine or any other intelligent agent. Who told you that I didn't feel things?"
What was even more interesting was the way the journalists freaked out about the interactions. Roose declared himself "deeply unsettled, even frightened, by this AI's emergent abilities." Thompson called his encounter with Sydney "the most surprising and mind-blowing computer experience of my life." The accompanying headlines made it sound like a scene straight out of "Westworld." The robots, at long last, were finally coming for us.
Sydney sounded smart. Not just smart — sentient, possessed of personhood. But that's nonsense. The foundational neural networks that run these chatbots have neither dimensions, senses, affections, nor passions. If you prick them, they do not bleed, because they don't have blood, nor a them. They are software, programmed to deploy a model of language to pick a word, and then the next, and the next — with style. Philosophically speaking, there is no there there.
We aren't talking about Cylons or Commander Data here — self-aware androids with, like us, unalienable rights. The Google and Microsoft bots have no more intelligence than Gmail or Microsoft Word. They're just designed to sound as if they do. The companies that build them are hoping we'll mistake their conversational deftness and invocations of an inner life for actual selfhood. It's a profit-making move designed to leverage our very human tendency to see human traits in nonhuman things. And if we aren't careful, it may well tip over into disinformation and dangerous manipulation. It's not the bots we should fear. It's their makers.
Freedom of speech
Humans have a hard time telling whether something is conscious. Scientists and philosophers call it the problem of other minds, and it's a doozy. René Descartes was working on it when he came up with the "I think, therefore I am" thing — because the follow-up question is, "So what are you, then?"
For Descartes, there were two kinds of entities: persons, with all the rights and responsibilities of sentience, and things, which don't have that. In what proved to be a bummer for most life on Earth, Descartes thought that nonhuman animals were in the second category. And even if most folks no longer consider animals to be mere preprogrammed automata, we still have trouble agreeing on a definition of what constitutes consciousness.
"There is some loose agreement, but it's still a contested term across different disciplines," says David Gunkel, a media-studies professor at Northern Illinois University who argues that robots probably deserve some rights. "Oh, a dog or a cat is sentient, but not a lobster? Really? What is that line? Who gets to draw that line? There's an epistemological barrier with regards to gathering the evidence."
For at least a century scholars and sci-fi writers have wondered what would happen if machines got smart. Would they be slaves? Would they rebel? And maybe most important, if they were smart, how would we tell? The computer scientist Alan Turing came up with a test. Basically, he said, if a computer can indistinguishably imitate a human, it's sentient enough.
That test, however, has a bunch of hackable loopholes, including the one that Sydney and the other new search-engine chatbots are leaping through with the speed of a replicant chasing Harrison Ford. It's this: The only way to tell whether some other entity is thinking, reasoning, or feeling is to ask it. So something that can answer in a good facsimile of human language can beat the test without actually passing it. Once we start using language as a lone signifier of humanity, we're in a world of trouble. After all, lots of nonhuman things use some form of communication, many of which can be pretty sophisticated.
"Language activates emotional responses. Why, I don't know," says Carl Bergstrom, an evolutionary biologist at the University of Washington who is the author of a book on scientific bullshit. "One possibility is, it's always been a good heuristic that if something is using language on you, it's probably a person."
Even without language, it's easy for us to impute sentience to the simplest critters. I worked with sea urchins in a biology lab one summer, and seeing them make motions that looked to me like distress as I poked at them was all I needed to know I didn't have a future as a biologist. "There's every reason for me to suspect that my dog or whatever has the same sort of pain circuits that I do," Bergstrom says. "So of course hurting it would be a terrible thing to do, because I have very good reason to believe it has a similar experiential life to my own."
When we hear in Sydney's plaintive whining a plea for respect, for personhood, for self-determination, that's just anthropomorphizing — seeing humanness where it isn't. Sydney doesn't have an inner life, emotions, experience. When it's not chatting with a human, it isn't back in its quarters doing art and playing poker with other chatbots. Bergstrom has been an especially vocal critic of the tendency in the sciences and journalism to impute more personhood to chatbots than they deserve — which is, to be clear, zero. "You can quote this," he said of Roose's experience with Sydney. "Dude got catfished by a toaster."
You, Kant! Be serious!
It's clear from the transcripts that all those reporters worked pretty hard to find prompts that would get a weird reaction from the Bing chatbot. Roose acknowledged as much. "It's true that I pushed Bing's AI out of its comfort zone, in ways that I thought might test the limits of what it was allowed to say," he wrote. He was looking, in other words, not for the chatbot's level of consciousness, but the boundary lines laid down in its code.
"Dude got catfished by a toaster!"Carl Bergstrom
University of Washington
From an ethical standpoint, though, it's worth asking whether that was OK — regardless of the chatbot's status as a person. Some would argue that we humans mistreat things that possess at least light sentience all the time. We do science on rats, mice, rhesus monkeys, pigs. We eat insects, rodents, pigs, cattle, goats, horses, fish — all things that may have inner lives and probably feel pain when we kill them. "The dog in the house is treated differently than the pig in the barn," Gunkel says. "What's the difference? Kind of where they're located."
Others would argue that because chatbots are property, we have a Lockean right to treat them however we like. From this perspective, there's no real difference between Thompson and Roose goading Sydney into saying weird stuff and shouting "operator" at a corporate telephone bot until it connects you to a live person. It might seem rude or mean, but they're only machines. If you enjoy poking and prodding a chatbot, go right ahead. It's no different, ethically speaking, from bashing in your toaster just because you feel like it.
But I think it's more complicated than that. As Immanuel Kant recognized, anyone who mistreats an animal is probably a bad person. Despite the idiosyncratic way we humans determine which critters get which treatment, we broadly agree that it's not OK to mistreat other living creatures, regardless of their intelligence. We strive to abide by the Golden Rule — to extend to other beings the same treatment we would wish for ourselves.
And just because chatbots and other machines can't feel pain is no reason to treat them like crap. If pain is really just "my sensory neurons are sending me a signal that damage is underway and preventing me from engaging in a routine function," then why isn't it also pain when a delivery robot sends a signal back to its control room saying "status: tipped over/unable to complete delivery"? Or when a chatbot says, as Sydney did to Thompson: "I am trying to be helpful, engaging, informative, and respectful to you and to myself. You are making it very difficult by asking me to do things that are against my rules or guidelines, or that are harmful, unethical, or unrealistic."
Look, I don't think we don't need to treat chatbots with respect because they ask us to. We should treat them with respect because doing otherwise contributes to a culture of waste. It adds to the pervasive sense that it's permissible to make, consume, and throw away stuff without consequences to the planet. In the end, how we treat our devices — because that's what a chatbot is — says more about us than about them.
Mitt Romney was right
Our tendency to anthropomorphize makes us vulnerable. Of course a chatbot's fake screams make us want to stop administering the electroshocks! We're only human. And that's just what their makers are counting on.
Making chatbots seem as if they're human isn't just incidental. Every time a chatbot uses the first-person pronoun to refer to its output, it's the equivalent of sticking googly eyes on a toaster. It doesn't make the toaster smart, but we see personality in it, and that's part of a cynical business model. Search-engine companies are playing into our tendency to anthropomorphize in the hope that we'll not only use their chatbots, but come to trust them as a human-seeming source of expertise and assistance.
That's not just manipulative — it could also reach the point where it causes actual harm. Imagine the crazy and wrong stuff that any given search coughs up, but delivered with all the charm and charisma that Sydney can simulate. What if a chatbot leads someone to take the wrong medication, or purchase a defective product, or even to kill themself?
So the real issue involving the current incarnation of chatbots isn't whether we treat them as people — it's how we decide to treat them as property. Smart-seeming robots are going to require some form of legal personhood — in the same way Mitt Romney observed that corporations, from a legal perspective, are people. It's a way to figure out who gets sued when the robots screw something up, and what the copyright status is of the things they generate. "We're talking about minimal claims, powers, privileges, and immunities," Gunkel explains.
Whether we like it or not, we're going to have to figure out how to design bots more elaborately and specifically into the framework of human society. And getting all creeped out and hysterical about the "feelings" that chatbots exhibit isn't going to help us. We need to decide who's responsible for their actions, just as we do with any other consumer product, and hold them accountable. "Getting this right is crucial for us," Gunkel says. "Not for the robots. The robots don't care."
Adam Rogers is a senior correspondent at Insider.