Claire Merchlinsky for BI
Imagine you're a hedge-fund manager who's trying to gain an edge. To maximize returns, you decide to install the latest technology that allows a computer to interpret the changing winds of the market and make thousands of orders in milliseconds. The program helps boost your fund for a while, but the initial excitement turns to dread as you watch the program go rogue and buy hundreds of millions of shares in less than an hour. Your firm scrambles to stop the trades, but they can't, and suddenly you're facing massive losses, all because of a poorly installed algorithm.
Dystopian sounding, right? But this catastrophic scenario isn't some troubling hypothetical about the rising threat of artificial intelligence; it actually happened over a decade ago when a coding mistake led to losses of $440 million for Knight Capital, eventually leading the firm to sell itself at a massive discount.
"Artificial intelligence" isn't "the future" — it's just a marketing term for a slightly updated version of the automation that has been ruling our lives for years. Companies have cycled through a series of names to dress up their tech — automation, algorithms, machine learning, and now AI — but ultimately, these systems all boil down to the same idea: handing over decision-making to computers to execute tasks at speeds far faster than a human could. While there's growing fear that a new breed of AI will infect our daily lives, put millions of people out of work, and generally upend society, most people don't realize just how deeply computerized decision-making has wormed its way into every facet of our existence. These systems are based on datasets and the rules that human beings teach them, but whether it's making money in the markets or feeding us the news, more and more of our lives are in the hands of unaccountable digital systems.
In many cases, these algorithms have proven useful for society. They've helped eliminate mundane tasks or level the playing field. But increasingly, the algorithms that undergird our digital lives are making questionable decisions that enrich the powerful and wreck the lives of average people. There's no reason to be scared of AI making decisions for you in the future — computers have already been doing so for quite some time.
The early internet was a relatively human-curated experience — a disparate collection of web pages that were discoverable only if you knew the address of the site or saw a link to it on another site. That changed in June 1993, when the researcher Matthew Gray created one of the first "web robots," a primitive algorithm designed to "measure the size of the web." Gray's invention helped create search engines and inspired a league of successors — Jump Station, Excite, Yahoo, and so on. In 1998, Stanford students Sergey Brin and Larry Page made the next leap in automating the internet when they published an academic paper about a "large-scale hypertext web search engine" called Google. The paper detailed how their algorithm "PageRank" judged the importance of a web result based on a user's query, serving up the most relevant site based on how many other websites linked to it — which made a lot of sense on a much smaller and more innocent internet.
Somewhere along the way, however, the tech industry tipped over from helpfully automating the jobs that slowed down our lives to distorting society by surrendering crucial decisions to computers.
Nearly three decades after the founding of Google, the internet has only gotten more automated. This has come with plenty of benefits for average people — recommendations on Spotify and Netflix help us find new art, robo-investors can help grow a nest egg at a low cost, and industrial applications like the robotics used to manufacture many modern vehicles have made our economy more efficient. Somewhere along the way, however, the tech industry tipped over from helpfully automating the jobs that slowed down our lives to distorting society by surrendering crucial decisions to computers.
In many ways, the original Google paper feels like a dark warning. It argues that advertising-funded search engines would be "inherently biased" toward those advertisers. It's probably no surprise then that researchers have found that by prioritizing ad dollars over useful results, Google's algorithm is getting worse, degrading a crucial source of information for over 5 billion people. And it's not just search engines. Revenue-focused algorithms behind networks like Facebook, Instagram, TikTok, and Twitter have learned how to feed users a steady stream of upsetting or enraging content to goose user engagement. As human control diminished, the real-world consequences of these algorithms have piled up: Instagram's algorithm has been linked to a mental-health crisis in teenage girls. Twitter admitted that its tech tended to amplify tweets from right-wing politicians, influencers, and news sources, and it's only grown worse since Elon Musk bought the site. Cambridge Analytica used an algorithm to trawl Facebook's data to hyper-target millions of people in the run-up to the UK's vote to leave the European Union and the 2016 US presidential election.
Algorithms that were supposed to make work easier or employees more productive have helped tilt the economy against blue-collar workers and people of color. Companies like Amazon are making hiring and firing decisions based on the whims of a glorified calculator. Customers are getting the short end of the AI stick, too: A 2019 investigation by the Associated Press and The Markup found that algorithms used in making loan decisions were highly biased against people of color, with lenders being 80% more likely to turn down Black applicants than similar white applicants.
And these problems carry over into the public sector, poisoning government services with algorithmic biases. The UK government faced a national scandal in 2020 when administrators replaced nearly 40% of students' A-level exams — a crucial test that can determine a student's ability to go to university — with algorithmically chosen grades. The results significantly underscored students from Britain's tuition-free state schools, instead favoring those who went to private schools in affluent areas and throwing the lives of many young people into disarray. A "self-learning" algorithm used by the Dutch tax authority falsely penalized tens of thousands of people for supposedly defrauding the country's childcare system, pushing people into poverty and leading to thousands of children being placed into foster care. In the US, ProPublica found in a 2016 investigation that an algorithm used in multiple state court systems to judge how likely somebody was to commit a crime in the future was biased against Black Americans, leading to harsher sentences from judges.
Across the public and private sectors, we've handed the keys to a spiderweb of algorithms built with little public insight into how they make their decisions. There is some pushback to this infiltration — the FTC is seeking to regulate how businesses use algorithms, but they've yet to do so meaningfully. And more broadly, it seems that governments are resigned to allowing machines to rule our lives.
Much like the proverbial frog boiling in a pot of water, the slow takeover of algorithms has mostly gone unnoticed by the general public. It's easy to miss a small tweak to Instagram's algorithm or even celebrate the tax software that simplifies your filing. But now, thanks to the new wave of "artificial intelligence," people are starting to notice just how hot the water has gotten. Investor-fueled enthusiasm means that almost every major company is considering or actively integrating generative AI — most of which is mediocre at best — into their services. As this AI hype cycle persists, we're hurtling toward the death of the useful internet.
Across the public and private sectors, we've handed the keys to a spiderweb of algorithms built with little public insight into how they make their decisions.
Large language models like those behind ChatGPT and Google's Gemini are designed to scrape the publicly available internet and various search engines for information. This poses a problem since the web is increasingly full of generic pages designed to game the SEO system rather than provide useful information. Many sites are themselves AI-generated, creating an ouroboros of mediocre, unreliable information. Take Quora, the question-and-answer site that was once beloved for high-quality user-generated answers. Quora now provides answers generated by OpenAI's ChatGPT, which feeds into Google's generative results and eventually tells users that they can melt eggs. And as disconnected executives replace human editors with AI — as Microsoft did with MSN.com, leading to misinformation delivered to over a billion people a month — we'll enter a cycle where the generative models train themselves on the remains of an internet poisoned by generative content. Even the supposedly human domain of social media has become flooded with AI spam, turning X, Facebook, Reddit, and Instagram into a constant battle against misinformation and outright scams.
While generative AI is just the newest extension of the algorithm, it poses a unique threat. Before, humans controlled the inputs and set rules of engagement for the models while the computers produced the output. Now we're allowing these models to set the rules of the game for us, generating both the inputs and outputs. We're already starting to see some of the deleterious effects: In 2023, the National Eating Disorders Association replaced its human helpline staff with an AI chatbot, only to have to pull it shortly thereafter when it started giving harmful weight-loss advice.
The wealthy and powerful will be able to opt out of this algorithmically driven future or shape it in their image by connecting directly to the humans behind companies shrouded by algorithmic processes. When you have a private banker, you don't need to worry about an anonymous, automated financial review — you've got a person with an email address and a phone number. The wealthy won't have to worry about nurses or doctors being replaced by AI processes, because they'll be able to afford concierge medical services. The powerful won't have to worry about their jobs being automated away because they'll be the ones choosing where and when a job is handed over to an automated process.
For the rest of us, it seems as if more and more of our lives will be dictated by the black box of algorithms, and we know how that goes. Automation may be useful to scale and speed up the operations of a business or government, but the tradeoff is almost always human suffering — layoffs, unfair policing, financial losses, a distorted media environment. Even the world's greatest source of information — the internet — is poised to become overwhelmed with content either created to appeal to algorithms or content generated by the algorithms themselves, pushing aside the unique, interesting, and valuable human-generated material that made the internet special.
AI isn't new in this sense, but it is an acceleration of the trend toward cutting people out of the machinations of the world around them. Perhaps an omnipotent superintelligence was never the thing to fear. Perhaps the real threat was the greed that would lead companies to willingly offload critical processes to the point that human control is overwhelmed by a cancerous force — simulacrums informed by simulacrums making real-world calls that erode the ability to decide for ourselves.
Ed Zitron is the CEO of EZPR, a national tech and business public-relations agency. He is also the author of the tech and culture newsletter Where's Your Ed At and the host of the "Better Offline" podcast.