Signal design in the age of AI-assisted candidates

Last year my team ran a hiring cycle for nine engineering roles. One of them, a backend opening, drew fifteen thousand applications in the first two days. The full set of nine roles took us six months to close, and along the way we caught many of the candidates cheating somewhere in the interview.

The cheating itself did not surprise me. What surprised me was how easy AI had made it for a weak candidate to look strong on paper and in a first interview. Most resumes looked nearly identical, as if they had all come from the same template. Most interview answers sounded like the candidate was reading them off a screen rather than thinking them through, and when we asked a follow-up question the candidate often stumbled, because the model had not written them an answer for the follow-up. During live coding rounds, we could watch a candidate pause, glance off-camera for a few seconds, and then paste an answer back into the editor.

We tried the obvious things first, and learned what each one was good for:

• Interview-as-a-service platforms gave us consistent rounds, but never flagged a single cheater.
• Proctored online tests caught a few cheaters, but most candidates just kept a phone or another laptop next to them, out of view of the webcam, and read answers off it.
• Manual resume screening worked, but at fifteen thousand applications it was far too slow.
• AI resume shortlisters meant we had one AI reviewing resumes that another AI had written, with the accuracy you would expect.

After a couple of cycles, two things became clear to me.

First, our limit was never how clever a round we could design. It was how many hours one person on my team could spend on screening in a day. We could not run ten interviews per candidate, and at three hundred applications a day we could not even give every resume one real read. If the very first round took an hour of one engineer's time per candidate, the team would run out of hours long before any strong candidate reached a senior interviewer.

Second, our interview process now had to do two different jobs at once, instead of just one. The first job was to filter out the candidates who used AI to look qualified without being qualified. The second was to check how well a candidate works with AI on real engineering tasks, because every engineer on the team now writes code with a model. Most of the writing I read at the time tried to do both jobs in the same interview round.

What follows are six things I now consider when I design a hiring funnel. The first is the structural constraint that governs everything else: how much screening time the team has each day. The next three are filters for candidates who look qualified but cannot do the job, and they run in funnel order: proof of past work in the resume, a quick behavioural read on how the candidate engages with code, and a curve ball the candidate cannot pre-prepare for. The fifth is a deeper round that checks how the candidate works with AI on real engineering tasks. The sixth holds the other five together: candidates and cheating tools adapt, so every round we design will stop working within a few months, and my job is to be ready with the next one.

Start from how much screening time your team has, then design backwards

Before designing any round, work out how much time one person on your team can spend on screening per day. Two hours is realistic for most teams I have worked with. Then work out what a real read of one resume takes. For us it was a few minutes per candidate, enough to look at one or two specific things and form a view. Two hours at three minutes per resume comes out to around forty candidates a day, so if three hundred applications land each day, more than two hundred and fifty of them are not going to get a real read at all. The first cut has to be something cheaper than a read.

Most writing on this problem starts from the question "how do we catch the candidate who used AI?" I think that is the wrong question. The right question is "what is a yes-no signal in the resume that I can check in seconds, that separates the candidates worth a real read from the ones who are not?" Detection tools like proctoring software or eye-tracking do not save time, because someone still has to set them up, review the candidates they flag, and decide who to advance. The glance we settled on is the next change on this list: what I look for in a resume before deciding whether it earns those few minutes of real time.

Look for proof of real work, not a polished resume

The first real filter in the funnel is the resume itself, and this one helps with both jobs at once. It filters out the candidates who only look qualified, because faking a real project with real users is much harder than faking a polished resume. It also surfaces the candidates who have shipped something using AI tools, because shipping a real project is exactly the kind of work that pushes an engineer toward those tools in the first place.

It is no longer enough for a candidate to know a particular framework or language, because a tech stack listed on a resume costs the candidate nothing to write, which means it no longer tells me anything. A model can build them a clean CRUD project in an evening and then write the resume to go on top of it, and the project only tells me the candidate followed a tutorial. What I want to see is a decision the candidate made that the tutorial did not make for them.

For early-career hires, I now look for a side project that reached at least one real user, or a small SaaS the candidate built and ran themselves: something with a user, and a tradeoff the candidate had to think through on their own. For experienced hires, I lean on prior time at engineering teams I respect, referrals from inside our network, and campus visits to programmes whose graduates we have already worked with and trust. One of those campus visits filled all six of our intern openings in a single day.

I do not think this higher bar on proof is a temporary reaction to AI in the candidate pool. The volume of applicants who look strong but cannot do the job will only keep going up, and junior candidates in particular will need proof that stands on its own, because the claims around it can now be generated by anyone. When we hold this line, we stop paying the much larger cost of hiring the wrong person and watching them struggle through onboarding and on-call.

Watch the candidate engage with code, not just produce it

Once the resume pile has been narrowed down, the next thing to find out, quickly, is whether the candidate is familiar with code or just looks like they are on paper. The cheapest signal here is behavioural: how the candidate reacts to code in real time. Behavioural signals are much harder to fake than artefacts, because a model can produce the artefact, but a model cannot sit in the candidate's chair.

The version we ran was a short recorded video where the candidate explained a piece of code on camera: we showed them some code on screen and asked them to record three to five minutes talking through it, camera and microphone on. Other formats produce the same signal (a brief live call with one engineer on the team, a screen-shared walkthrough, a paired session that is mostly the candidate narrating), and the right one depends on the team's bandwidth.

What we graded was how the candidate behaved on the recording while they explained the code. The obvious giveaways, like a long pause before the first sentence or eyes drifting off-screen to read an answer from somewhere, showed up in our very first batch of recordings. The one that stayed with me was a candidate whose friend was sitting just outside the camera frame, whispering the answers.

A candidate cannot fake how their face looks while they read out an answer they did not write. Our version of this round took one to two minutes per candidate to review, which made it the first round in the funnel where we got more out than we put in.

Throw curve balls the candidate cannot pre-prepare for

The last filter for candidates who look qualified but cannot do the job is a curve ball: a question the candidate cannot pre-prepare an answer to, because the right answer needs something a model cannot supply.

The kind of curve ball that worked best for us was a question whose only good answer was a small personal story. The one I used most often was "explain the HTTP 418 status code, the teapot." A model can give you a clean textbook answer. A real engineer remembers a code review where a colleague slipped 418 in as a joke, or a side project where someone added it as a gag. The candidates who gave us only the textbook answer, with no small story attached, were almost always cheating.

Curve balls keep working as a category, and any one specific curve ball stops. Once candidates knew we ask about the teapot, they showed up with a pre-prepared teapot story, and by our second hiring cycle the question had already stopped being useful and we had to write a new one.

Let the candidate use AI, and grade how they use it

Many teams are reluctant to make this next change. Every engineer on my team now writes code with a model, so if we forbid the candidate from using one in the interview, we are testing for a different job than the one we are going to hire them for. What we want to see in the interview is how the candidate works with the model in their hands.

So we added a dedicated round where the candidate uses AI in front of us. We give them a coding problem, often one with deliberately fuzzy requirements, they share their screen and use whichever model they prefer, and we grade seven things while we watch:

• How they handle a fuzzy problem. Before the candidate prompts the model, do they restate the problem in their own words and name the assumptions they are making, or do they paste our question in as-is and let the model decide what we meant?
• How they write the prompt. Does their first prompt give the model the context, the constraints, and at least a small example, or is it a single line followed by hitting submit?
• Whether they write a clean spec when the problem calls for one. A well-written spec helps the model produce better output than any other input the candidate can give it, and the candidates who take the time to write one are the same ones whose AI-assisted code still holds together past the first page.
• How they check the model's output. When the model returns code that looks right, does the candidate run it, test it at the edges, and read it line by line, or accept the first answer that looks plausible and move on?
• Whether they push back on the model. When the model is wrong, does the candidate notice and correct it, or follow it off a cliff?
• How fast they recover from a wrong turn. When the model leads the candidate down a wrong path, how long does it take them to notice and start over?
• Whether they would ship what came out. At the end, we ask the candidate whether they would put the final code into production, and why or why not.

More companies have started running rounds like this for the same reason we did: if we ban the tool in the interview, we end up hiring engineers who cannot use it well, which is the opposite of what we need.

Plan for every round to stop working in three to six months

This last one holds the other five together. The tools candidates use to cheat improve faster than my team can redesign interview rounds, which stops being surprising once you look at who is spending time on what: the people building those tools work on them full time, and we redesign rounds in the gaps between everything else. So my job, as the person designing the funnel, is not to find one perfect round that lasts forever but to keep replacing rounds every few months.

Take the teapot question again. The first time we used HTTP 418, it cleanly separated the candidates into two groups: the ones with a real story stayed in the funnel, and the ones without one dropped out. By the second hiring cycle, candidates were arriving with a pre-prepared teapot story, the right length, with the right amount of self-deprecating detail. The question stopped working, even though the category it belongs to still does.

The same thing will happen to every other round in this list:

• The recorded on-camera round, which felt unbeatable to us in the first cycle, is the next thing the cheating-tool industry will work on. Voice assistants that whisper answers to the candidate, with realistic timing, are already starting to appear.
• We will have to keep adding to our list of what we grade in the AI round, because the underlying models keep getting better at the very things we currently catch candidates on.
• We will have to rewrite our definition of "real shipped work" once candidates learn how to manufacture convincing fake projects.

If I design each round assuming it will work forever, then six months later I am left with a funnel that no longer separates real candidates from fake ones, and a hire bar that has quietly dropped without anyone on the team noticing. If I design each round assuming I will replace it within six months, I always have a few replacement rounds being designed in the background, and the rest of this list stays worth running.

Once the hiring process is reliably telling us which candidates can do the job again, two more questions come up. The first is calibration: how do I keep the same bar we started with, even as the interview format itself keeps changing. The second is growth: once we have hired a real candidate, how do I grow them on a stack the team itself is still learning. After that comes closing the candidate, which gets much easier when the earlier rounds have surfaced who the candidate is, because by the time we make an offer we have met the person we are hiring.