A car dealership added an AI chatbot to its site. Then all hell broke loose.

• Pranksters discovered that a local car dealer's AI chatbot could be used as a way to access ChatGPT.
• People shared attempts to trick the chatbot into selling them a new Chevy for as little as $1.
• Fullpath, the chatbot's creator, told Business Insider it's improving the bot based on the pranks.

On Sunday, Aharon Horwitz was listening to a podcast when he got an unusual Slack alert. Horwitz is the CEO of Fullpath, a tech company that sells marketing and sales software for car dealerships. The automated Slack alert meant there was an unusual amount of traffic on one of their client's websites.

A few moments later, an employee dropped a tweet into Slack: someone had figured out how to prank the AI chatbots on a car dealership site. Horwitz and the employee immediately recognized that chatbot: it was one of theirs.

The day before, Chris White, a musician and software engineer in California, was thinking about a new car.

"I was looking at some Bolts on the Watsonville Chevy site, their little chat window came up and I saw it was 'powered by ChatGPT,'" White told Business Insider.

"So I wanted to see how general it was and I asked the most non-Chevy-of-Watsonville question I could think of."

He asked the chatbot to write him a Python script, and it happily obliged. White posted screenshots of the exchange to Mastodon, where it generated thousands of likes and reposts.

Hours later, someone else copied and reposted his screenshots onto X, where it went viral.

Others played around with the chatbot to get it to act against the interests of the dealership. One user got the bot to agree to sell a car for $1 (this is not, I should note, legally binding).

A handful of these tweets went viral, and more were posted on Reddit's /rChatGPT forum, where one Redditor sagely predicted that soon the tech press would report on the fiasco in a tut-tutting manner, bemoaning the dangers of AI.

Still others tried more creative ways to get the chatbot to go off-topic.

Horwitz and his team quickly shut down the bot for that particular dealer's site.

When Business Insider called the local dealer, a salesperson said they were aware of some strange activity on the website but didn't know much more about it, and that the marketing team was in meetings all day on Monday and wasn't available to talk.

Chad Lyons, a spokesperson for General Motors, the maker of Chevrolet, said in a statement, "The recent advancements in generative AI are creating incredible opportunities to rethink business processes at GM, our dealer networks and beyond."

Fullpath, based in Vermont and Israel, started offering the ChatGPT-powered chatbots about six months ago. Horwitz told Business Insider that he estimate several hundred dealers use the chatbots.

Despite the handful of funny screenshots that went viral, Horwitz told Business Insider that there were far more failures. "They worked really hard," Horwitz told Business Insider. "In our logs they were at it for hours."

They could review the logs of all the requests sent into the chatbot, and he observed that there were lots of attempts to goad the chatbot into misbehavior, but the chatbot faithfully resisted. Horwitz also pointed out that the chatbot never revealed any confidential dealership data.

(Business Insider reviewed some of these logs and confirmed that indeed, the chatbot often rejected the silly requests and insisted on only discussing car-related things).

Essentially, the chatbot passed the test, and now FullPath will use these tests to strengthen its limits further.

"The behavior does not reflect what normal shoppers do. Most people use it to ask a question like 'my brake light is on, what do I do?' or 'I need to schedule a service appointment,'" Howitz told Business Insider. "These folks came in looking for it to do silly tricks, and if you want to get any chatbot to do silly tricks, you can do that," he said.