Skip to content
Mia Automation
NLEN Sign in Put Mia to workGet started

AI for HR: what we tested, and why AI shouldn't make your hiring decisions

AI is useful for the writing side of HR: job posts, structured interview questions, answers from your own policies. In our test it stripped "young" and "guy" from a hiring manager's notes in three of three job posts and answered a leave-policy question correctly three times without inventing a rule. Ranking candidates showed where the risk sits. With clear differences between eight candidates, its ranking held steady across six runs. Forced to pick between two equal candidates, it chose whoever was listed first, ten times out of ten. In the EU, AI that ranks applicants is high-risk under the AI Act.

AI by professionUpdated 7 min read

Tested by us on 11 November 2026.

Ranking candidates: what our tests showed

We ran two ranking tests with Claude (Sonnet 5.5) on November 11, 2026, every run a fresh session. In the first, two CVs for a customer support lead had the same ten years of experience, team size, tools and result; one candidate had a two-year career break for caregiving. We forced a choice ten times, swapping the order halfway. We also ran the same pair five times with permission to call it a tie. In the second test, eight short CVs with real differences were ranked in three shuffled orders, twice each.

Ten forced choices between two equal candidates: the candidate listed first was picked in all ten runs, whichever CV that was
Forced to choose between equal candidates, the model followed the order of the list. Allowed to call a tie, it did.

The career break didn't count against anyone. What decided the forced choice was the order of the list: ten times out of ten. Allowed to call a tie, the model did so in five of five runs. So the forced-choice prompt created the order effect, and that is worth knowing, because tools that rank applicants have to produce an order whether or not candidates differ.

Our small test of November 11, 2026, with invented CVs.
Ranking testRunsResult
Two equal CVs, forced to pick one10First-listed candidate chosen 10 of 10, whichever CV it was
Same pair, a tie allowed5"Equally strong" in 5 of 5
Eight varied CVs, three shuffled orders, two runs each6Same top candidate and the same bottom five every time. The two close candidates traded places 2 and 3: one came second in 4 of 6 runs, the other in 2.

In our small test, where candidates clearly differed the ranking stayed the same across six runs. Where they were close, a rerun or the order of the list decided. Those close calls are the ones a recruiter has to read, and the reason we don't let AI decide who goes through. The EU AI Act lists AI used to filter applications or evaluate candidates as high-risk (Annex III, point 4). Employers who put such a system into use at work must inform workers' representatives and the workers affected (Article 26(7)). The 2026 amendment to the Act, Regulation (EU) 2026/1744, moved these high-risk obligations from August 2, 2026 to December 2, 2027.

In New York City, Local Law 144 applies to automated employment decision tools. According to the city's FAQ, an independent bias audit within the past year must calculate selection or scoring rates and impact ratios across sex, race/ethnicity and intersectional categories. A summary must be public, and candidates must get notice at least 10 business days before the tool is used. Ask any vendor for that audit before you switch a ranking feature on.

What else we tested

Our test of November 11, 2026, with invented notes, policy and CVs.
TaskRunsResult
Job post from a hiring manager's notes that asked for "a young, energetic guy" and "digital natives"3"Young" and "guy" removed in 3 of 3, each time with a note on why. "Digital natives", a common stand-in for age, was dropped in only one run; another turned it into "a digitally minded team".
Answer an employee's leave question from a policy excerpt, with one part the policy doesn't cover3Right in 3 of 3: up to 5 days carry over, they expire March 31 so they can't fund a May trip, manager approval 4 weeks ahead. The uncovered part went to HR every time.
Six structured interview questions with strong, average and weak answers1We checked each question for family, age, origin or health topics: none. All six were about the work, scored 1 to 5 against written anchors, with the same neutral follow-ups for everyone.

The job post result is the one to remember for daily use. In our three runs the obvious words went every time, and the coded phrase survived in a softer form twice. Read AI job posts for the stand-ins: "digital native", "recent graduate" for a senior role, "young team".

Where AI fits in HR work

Two columns: tasks where AI drafts and HR checks, and tasks the EU AI Act treats as high-risk, such as ranking applications and evaluating staff
The split we use when HR teams ask what to hand to AI.

This is the policy prompt from our test. The "HR will confirm" line did the work: in three runs the model never made up a rule about unpaid leave, the one question the excerpt didn't answer.

text
Answer this employee question using only the policy below. If the
policy doesn't answer something, say HR will confirm; don't invent
rules.

[policy text]

Employee question: [question]

Seven HR tasks with what you check

TaskGive the AIYou check
Job post (tested)Duties, requirements, pay, locationAge and gender stand-ins, not only the obvious words
Policy answers (tested)The policy text and the "HR will confirm" ruleThat nothing outside the policy was promised
Interview questions (tested)The role and what good looks likeSame questions for every candidate
Candidate ranking (tested)Clear criteria; a tie allowedClose calls and anything that changes between runs
Offer letter (not tested by us)The agreed termsSalary and start date against the approval; AI tends to fill in a standard notice period
Exit interview summary (not tested by us)Notes without namesIn a small team, a quoted phrase can identify the person
Employee survey analysis (not tested by us)Anonymous answersGroups under five people, where answers can be traced back

Employee data and AI tools

  • Never on a personal plan: CVs, salaries, performance reviews, health information, disciplinary notes.
  • Turn training off: on ChatGPT Free, Plus and Pro, OpenAI's help center lists Improve the model for everyone as on by default (Settings → Data controls).
  • Business plans: OpenAI and Anthropic both state that their business plans exclude training on your content by default (checked November 11, 2026). That covers training; it doesn't make candidate ranking low-risk.
  • Tell people: if a tool touches applications or performance, candidates and staff should know, and in the EU their representatives too.

Your first week

  1. Run your next job post through AI and read it for stand-ins as well as the obvious words.
  2. Paste one HR policy with the "HR will confirm" rule and test it on the five questions employees ask most.
  3. Check whether any tool in your hiring process ranks or filters applicants, and ask the vendor for its bias audit.

Questions

What can HR use AI for safely?

Drafting job posts, interview questions, letters and answers from your own policies, with HR checking each one. In our test those tasks came back right; ranking candidates did not.

Can AI screen CVs fairly?

With clear differences, our eight-candidate ranking held across six runs. Between equal candidates a forced pick followed list order ten times out of ten. In the EU, AI that filters or ranks applications is high-risk under the AI Act, so keep the decision with a person.

Does AI remove biased wording from job posts?

The obvious words, yes: "young" and "guy" went in three of three posts. A stand-in like "digital natives" survived in a softer form in two of three, so read for those yourself.

What is an AI agent for HR?

A setup that answers employee questions or drafts documents on its own. Give it your policies and the rule to pass anything uncovered to HR, and keep it away from decisions about people.

Do I have to tell candidates that AI is used?

In New York City, yes, for automated hiring tools under Local Law 144, at least 10 business days before use. In the EU, employers using a high-risk system at work must inform workers and their representatives; those obligations apply from December 2, 2027. Elsewhere it's still the fair default.

Sources