Hire AI evals engineers who can actually tell you when your LLM is regressing.

Two years ago, eval engineering was a research hobby. Now it is the job standing between your latest prompt change and a silent 9% quality regression that leaks into production for six weeks before a customer complains. The 2025–2026 shift everyone in the field calls eval-driven development — writing the eval before the feature, running it on every PR, treating a failed eval like a failed unit test — crossed from research practice into product engineering standard across every scale-up we work with.

Concretely, an evals engineer owns four things. First, the golden-reference datasets: curated input/output pairs with documented disagreement resolution, versioned like code, re-labelled when the task definition drifts. Second, the automated eval harness: programmatic checks for exact match, regex, embedding similarity, JSON-schema validity, latency, cost, and rubric-based LLM-as-judge flows, wired into CI and a dashboard the product team actually looks at. Third, human-in-the-loop rating: rubric design, inter-rater agreement (Cohen's or Krippendorff's alpha), calibration of the LLM-judge against those human scores so you can trust automation beyond a small held-out set. Fourth, adversarial and safety evals: jailbreak suites, prompt-injection, bias probes across protected characteristics, and the red-team harness you need if your product falls under Annex III of the EU AI Act.

Demand reflects the shift. The UK AI Safety Institute's open-source Inspect AI framework, released in 2024 and now at v0.3+, has become the de-facto backbone for safety-grade evals in UK and EU AI teams; the OpenAI Evals repo has 15K+ GitHub stars; Scale AI's Humanity's Last Exam benchmark, released late 2024, re-set what frontier eval design looks like. Every Series B AI-adjacent company we have talked to in the last six months is either hiring an evals engineer or has just made the hire and is already asking for the second. The bottleneck is not supply of framework-literate candidates — it is supply of candidates who have actually operated an eval loop against a real product team under deadline.

One honest distinction up front: an evals engineer is not an evals researcher. The researcher writes papers on benchmark design and publishes at NeurIPS or ICLR. The engineer ships the harness, argues with the PM about the rubric, resolves the six ambiguous labels the raters disagreed on, and owns the green/red light on your Friday release. You almost certainly need the second, not the first, until you are at ~40 AI engineers and actively shipping foundation-model work.

This role confuses generalist recruiters more than any other in our catalogue. Job boards return a blend of traditional QA engineers who wrote a single LLM eval once, and ML researchers who can describe HELM and BigBench but have never versioned a dataset against a product team's Friday deploy. Both profiles miss. Our filter is built to separate the production operator from the framework tourist inside the first screen.

We source across six channels specific to evals engineering: open-source contributor graphs for Inspect AI (AISI's framework), OpenAI Evals, Ragas, TruLens, LangSmith integrations, and Langfuse; authors on the HELM, BigBench, MMLU-Pro and Humanity's Last Exam leaderboards and reproducibility reports; Hugging Face dataset maintainers for eval-focused datasets with sustained downloads; papers-with-code contributors on robustness, faithfulness, and adversarial-eval benchmarks; the Anthropic and OpenAI public red-teaming and model-card communities; and a private network of 640+ CEE AI engineers built by Nikita during his time at Neurons Lab, where the shop delivered 80+ AI projects including several safety-eval and bias-audit engagements for European clients.

Every candidate goes through a 14-minute AI technical interview that probes operator signals, not framework recall: 'walk me through the last time your eval suite caught a regression — what did the delta look like and how did you decide it was signal, not noise?', 'how do you calibrate an LLM-as-judge rubric against human raters when the raters themselves disagree 18% of the time?', 'tell me about a golden dataset you owned — how did you handle ambiguous examples and concept drift?'. The AI asks adaptive follow-ups; a human recruiter reviews the transcript and scores before a shortlist lands in your inbox. Candidates passing our evals filter in 2025-Q4–2026-Q1 had a median of 2.5 years of post-2023 production evals experience and a 93% interview-pass rate at our clients.

The last layer is role-specific: every shortlisted evals engineer must have owned at least one eval loop that blocked a production release — meaning a concrete example where they flagged a regression, the team rolled back or fixed forward before shipping, and the outcome was documented. We verify via a combination of public artifacts (OSS PRs, blog posts, conference talks, model cards) and a reference call with the previous engineering manager. Candidates who have only built evals in a research context do not make the shortlist for production-engineering roles; we route those to a separate pipeline for applied-research teams who want that profile explicitly.