↗MLabsTALENT PARTNERSLet’s talk hiring ↗
← THE HIRING NOTEBOOK / AI HIRING

Inference engineer or ML platform hire? Let the bottleneck decide

Separate production ML, inference and GPU performance needs with a workload brief, serving-incident exercise and role-specific evidence checklist.

By MLabs Talent Partners · · 5 min read

Your LLM feature works, but users wait too long and the serving bill is difficult to explain. The immediate temptation is to hire a GPU optimisation specialist. That may be the right decision, yet the bottleneck could also sit in request handling, deployment configuration, workload management or an unclear product requirement. Choose the level of specialist depth after identifying which evidence is missing.

This guide addresses an established product serving models, rather than choosing a first AI hire. It proposes a hiring decision and a technical assessment built around a fictional serving problem. The aim is to distinguish reliable inference engineering from a persuasive list of optimisation techniques.

Separate three kinds of work

Production ML ownership includes deployment, release processes, observability and recovery. Inference engineering concentrates on how a model is served under a particular workload and set of constraints. GPU performance work may go deeper into profiling, kernels and hardware behaviour. One engineer can span these areas, but a vacancy should identify where independent depth is needed immediately.

Write down what your current team can already do. If nobody can deploy and roll back the service confidently, an expert in low-level optimisation may still need substantial operational support. If the deployment is sound and measurements isolate expensive computation, a more specialised performance hire becomes easier to justify. Avoid turning an unmeasured complaint into a requirement for every tool in the stack.

Bring a workload contract to the hiring meeting

Define the user interaction, representative input and output sizes, expected traffic pattern, quality checks and operating constraints. Specify what “fast enough” means for that interaction. An interactive assistant and an overnight document-processing job should not inherit the same target merely because they use the same model.

Include hardware access, provider constraints, data-handling requirements and who can approve model changes. State which requirements are fixed and which are negotiable. For a team distributed between the UK and Netherlands, explain who owns incidents and when collaboration is expected. A specialist should know whether the job involves operational coverage before accepting it.

Do not require candidates to commit to a percentage saving before inspecting the system. A credible initial outcome could instead be a reproducible baseline, a diagnosis of the dominant constraint and one validated improvement. That is specific without pretending the result is already known.

Give candidates a serving incident to investigate

Prepare a fictional packet showing request arrival times, input lengths, output lengths, queue duration, response timings and failures. State that a new customer has introduced longer requests and that the product team reports a worse experience. Include enough detail to compare request groups; do not hide a magic answer in a missing chart.

Ask the candidate to spend the first part of the interview defining the problem. Which users are affected? Is delay before the first response different from delay during generation? What changed in workload or configuration? Which measurements are taken at the client and which inside the server? Record the questions and their purpose before discussing optimisations.

vLLM’s metrics documentation distinguishes time to first token from inter-token timing and request-level measures. Metric names and semantics should be checked against the deployed version. In the interview, use the distinction to explore user experience rather than requiring memorisation of metric names.

Require a controlled experiment and an exit condition

Ask the candidate to choose one hypothesis and design a bounded experiment. They should state what remains constant, which requests are representative and what result would support or weaken the hypothesis. A proposal to change the model, serving configuration and hardware simultaneously needs an explanation of how the contribution of each change will be understood.

Then introduce a trade-off: the experiment improves aggregate throughput but some interactive requests wait longer. Ask whether to proceed. A strong response refers back to the workload contract, examines affected request groups and considers options without treating a single headline number as the answer. There may be several acceptable decisions, provided their assumptions are explicit.

If the proposed change affects model outputs, require a quality comparison appropriate to the product. If it affects concurrency or scheduling, ask how the experiment will represent load. In every case, request a release plan, monitoring and a reason to stop or revert. The candidate does not need to implement an entire serving stack during the assessment.

Match the evidence to the vacancy

Hiring emphasisEssential evidenceSeparate depth check
Production MLSafe deployment, useful monitoring and incident ownership.Review a release and rollback plan.
Inference engineeringWorkload-aware diagnosis and defensible serving experiments.Interpret a small serving trace or benchmark.
GPU performanceProfiling-led explanation of computation and resource costs.Review a relevant low-level change with a qualified interviewer.

Do not infer GPU implementation ability from an excellent serving discussion. If that depth is essential and nobody on your team can assess it, arrange a suitably qualified technical reviewer. Conversely, do not reject an operationally strong ML engineer because they cannot solve a low-level problem the advertised role will never own.

Make the offer match the system they will inherit

Share an honest account of current observability, technical debt, available hardware and support. Explain who owns evaluation and who can authorise spending. A hire cannot meaningfully optimise a service if they cannot access representative workloads or run controlled experiments. Resolve those practical constraints alongside the search.

Use our production LLM interview guide for broader product judgement, and the benchmark-review assessment for deeper performance evidence. The right hire is the person whose demonstrated depth matches the bottleneck and whose surrounding team can turn that depth into a maintained improvement.

YOUR NEXT CHAPTER

Great teams don’t
happen by chance.

Tell us what you’re building. Let’s find the people to build it.

Discuss your next hire ↗