vgajic/Getty Images
- The "coffee-cup test" has resurfaced online as a symbol, for some, of opaqueness in hiring.
- The secret nature of the test can raise similar concerns to the use of AI in hiring.
- Both the coffee test and AI can make the process of getting a job feel arbitrary and nontransparent.
Years ago, I heard a story about a recruiter who would hold up résumés to a light to see whether the watermark stamped onto the paper was facing the right direction.
Some background: Résumés used to be printed on paper. Fancy résumé stock often carried a faint image known as a watermark that would speak to the paper's fine pedigree — or at least how much you'd paid for it.
The legend went that even swanky paper wasn't enough for this loathsome recruiter. If the watermark was upside down or facing backward, it was an indicator of the candidate's slapdash approach to life. Into the trash it went.
For another boss man, today's litmus test isn't paper but ceramic. The hiring manager shows those who show up for interviews where the kitchen is, offers them a coffee, and then rejects those who don't bus their dishes afterward.
The coffee-cup test, like the watermark check before it, can feel like a trivial way to toss candidates who might otherwise be qualified. Little surprise, experts on interviewing often dismiss the gotcha approach.
Yet one reason these stories keep resurfacing online, even years later, is because they speak to our fears that we're hurtling toward a hiring environment with more unfairness and even trickery — trappuccino, anyone? — thanks to technology.
Today, the bogeyman is artificial intelligence. AI can seem just as arbitrary in its decision making as the coffee-cup test.
"One of the reasons people are nervous about AI is that the algorithms used are very blackbox," Josh Millet, an expert on hiring, told Insider. "The applicant has no idea how they're being evaluated."
It's a similar feeling that some might get from the coffee-cup standard. "No one should have secret tests," he said.
AI, of course, could make a hiring process that's still way too subjective that much less so. Yet some worry that ruthless bots will rely on mysterious algorithms to yank job candidates' from consideration because of invisible infractions or minor deficiencies.
Millet, who's founder and CEO of Criteria, a Los Angeles company that works to help companies reduce bias and increase efficiency in hiring, said the promise of AI is to strip unfairness from the recruiting process and make it just about the data. Yet for that to happen, the data sets on which the AI is trained can't be biased. That's a big challenge.
"There's great promise, but there's also risk that it not be used ethically and responsibly," Millet said, referring to AI.
When AI sits in on the interview
There have been notable examples of AI fails when it comes to hiring. Perhaps one of the best known was a trial in which Amazon tested using AI in recruiting. But because the training was done using résumés largely from men, the AI discriminated against female candidates. The company shut down the project.
"There's been some really high-profile faceplants on AI," Millet said. It's little wonder, then, that job seekers would be concerned about the increase in digital gatekeepers. In a survey of some 1,200 employed US adults, about half said that AI tools used in recruiting are more biased than people. The online survey was conducted in June by the Harris Poll for the American Staffing Association.
People are even more squeamish about the notion of our AI overlords getting to make a final call on whether a candidate gets hired. A Pew Research Center survey of 11,000 US adults in mid-December 2022 found that seven in 10 Americans are against letting AI make the ultimate hiring decision.
Sandra Sucher, a professor of management practice at Harvard Business School, told Insider it's understandable why people would have concerns given the record of bias in AI.
"Those aren't imaginary concerns. Those are absolutely appropriate concerns," she said, adding that work remains to address those challenges.
"The promise of equitable hiring is there," Sucher said. "I genuinely would worry about perpetuation of bias depending on the age and the nature of the data set."
Yet the misgivings by some workers and others don't seem to be keeping employers from bringing on AI to do more of the work of hiring. That includes interviewing — presumably without the java.
In one survey of representatives from some US companies, 43% reported that their organizations are using or plan to use AI in the interview process by 2024. The online poll, conducted in June by Resume Builder, involved about 1,000 people who are part of the hiring process at their employers.
Secret tests might not say much about a candidate
The coffee-cup test, then, can feel to critics like another way that people are being judged on things that don't pertain to their jobs. Of course, some have argued the test is a good measure of how considerate a candidate is in the workplace. One Reddit user wrote, "My guess is the people raging against it are the same ones that leave their dirty dishes for other people."
Yet, Millet said, tests that are designed to be a "soul read" into what a person is really like, or how much integrity the candidate has, can often fall short. These measures, for one, don't take into account how nervous a candidate might be in the interview.
Instead of relying on hidden metrics, Millet said those doing the hiring should use standardized questions across candidates and a rubric to score people's answers. For insights into people's character or, preferably, their work ethic, Millet said it's best to use questions that pose scenarios and see how candidates respond. Essentially, anything interviewers do to assess character, personality, or ability should be structured and objective, he said.
Millet said a good rule of thumb for hiring is that the selection criteria should be relatively transparent. "If I want to know something about you, I should ask you a question about it."