What should you ask an AI developer before hiring?

These are the questions whose answers reveal whether someone has run AI work in front of real users. Each one has a credible shape of answer and a red flag, and you do not need to be technical to tell them apart: you are listening for a method versus a name.

Ask them in the same words to everyone you are considering, and write the answers down. The comparison is the point.

QuestionA credible answer sounds likeRed flag
How do you know the answers are right?A set of real past questions with known answers, run again after every change, and a number for how often it is wrong"The model is very accurate" or a demo on questions they chose
What happens when the documents do not contain the answer?It says so, and hands off to a person, and that behaviour is tested"It always answers" treated as a feature
How do private documents stay private?Permissions applied when retrieving, as data, so a user only ever searches what they may read"I tell it in the prompt not to share confidential things"
What happens when the model changes or is retired?The model is a replaceable part behind an interface, with the evaluation set re-run on the new oneA build wired directly to one model name with no way to swap it
Can a document tell your assistant what to do?Retrieved text is treated as data, tools are allowlisted, and irreversible actions need approvalNo answer, or "the prompt handles it"
What does it cost to run at my volume?An arithmetic you can follow: what one run sends, how many runs, the published rateA single monthly figure with no volume behind it
Can I see a log of one real run, and replay a bad one?Yes, and they can show you what it decided and whyLogs do not exist, or only the final answer is stored
Whose accounts does it run in?Yours: hosting, model key, repository, with a written handoverEverything lives in their account and leaves when they do

Why do these questions matter more than a portfolio?

Because AI work demos beautifully and fails quietly. A screen recording of an assistant answering five questions correctly tells you almost nothing about whether it will answer the five hundredth correctly, whether it will leak a document it should not have read, or whether it will still work after the provider changes a model.

The failure modes are new enough that plenty of capable developers have not met them yet. That is not a reason to avoid them, it is a reason to ask. Someone who has shipped this work has been bitten by at least one of these and will tell you about it without being asked.

  • Retrieval quality is decided by your documents, not by the model, so ask what they will need from you and what they will do about duplicates and scanned files.
  • Accuracy is a number you measure, not a property a model has. If nobody measured it, nobody knows it.
  • Permissions are a code and data problem. A prompt is not a boundary, and the OWASP guidance on prompt injection is explicit that instructions inside retrieved content are an attack surface.
  • Models change on somebody else's schedule. A build that cannot swap one is a build with an expiry date nobody wrote down.
  • The running bill is a usage bill. Anyone who quotes it without your volume is guessing, and the guess will be wrong in the direction that suits them.

What are the red flags worth walking away from?

None of these means the person is dishonest. Most mean they have not done this at production scale yet, which is fine for a small job and expensive for a system your staff will rely on.

  • They lead with the model or the framework rather than the problem. Tools are the easy part to name and the least informative thing to hear.
  • Every answer is a capability, never a limit. Ask what it cannot do; if nothing comes back, they have not tested the edges.
  • No evaluation set, and no plan to build one. This is the single strongest predictor of an assistant that drifts and gets abandoned.
  • A claim that they train a model on your data for a question-answering job. Fine-tuning is usually the wrong tool for business facts, and it suggests they have not thought about what happens when a price changes.
  • A price with no written scope, or a scope with no failure behaviour in it.
  • Everything runs in their accounts, and the handover is a promise rather than a list.
  • They cannot describe what happens when it is wrong. If there is no answer to that, there is no plan for the day it matters.

How do you check the answers without being technical?

You are not assessing the code. You are assessing whether the person has a repeatable method and can show you evidence. Four things you can ask for that need no technical knowledge to judge:

  • Ask them to answer a question you already know the answer to, from your own documents, and to name the file they read it from. Then ask one your documents do not answer, and see whether it admits it.
  • Ask for the list of questions they test against, and how many are currently failing. A real number, including a bad one, is a good sign.
  • Ask what you have to prepare, and what happens if your files are messy. The honest answer is that this is most of the work and it is partly yours.
  • Ask who else has run something they built, and what broke after launch. Every real project has a story here, and the story is more useful than a logo wall.

When should you hire someone, and when should you not?

The supplier-type question is a different one and it is already answered: agentic developer vs agency vs freelancer compares the kinds of supplier, and how to hire an n8n developer has the ten ownership and failure questions for that tool specifically. This page is about the AI part only.

  • Do it yourself when it is one person, a handful of files, and no private documents. Upload them to a chat tool and see whether the answers are even findable. That costs nothing and tells you whether the project is worth having.
  • Hire someone when other people will rely on the answers, when some documents are private to some staff, or when it has to live inside your own site or app. That is where retrieval quality, permissions and testing decide whether anyone trusts it.
  • Hire someone when a wrong answer costs you a customer or a payment, because the human approval step and the evaluation set are design work, not settings.
  • Do not hire anyone for a general assistant that knows everything about your business. Nobody maintains that knowledge, so it drifts, and staff stop using it inside a month.
  • Do not hire on the strength of a demo alone. Buy a small first piece of work instead, which is the next section.

What should the first piece of work be?

Something small, finishable and written down. A single workflow or a fix starts at $49, and a build around one real process starts at $499, each as one fixed price agreed after a free call. The point of starting small is not the saving, it is the evidence.

A small first job tells you four things an interview cannot:

  • Whether they write down what done means before they start, and whether the list matches what you asked for.
  • Whether they tell you about a problem early or present it as a finished feature later.
  • Whether the thing runs in your accounts and is named so you can read it.
  • Whether they can say what it costs to keep running, with your volume in the arithmetic.

What to remember

  • Ask about evidence, not tools: how the answers are checked, what happens when the model changes, and how private documents stay private.
  • A credible answer names a repeatable method. A weak answer names a model, a framework or a feeling.
  • Ask for a log of one real run and a replay of a bad one, and for the evaluation set including how many cases currently fail.
  • Buy a small first piece of work with a written definition of done, from $49, rather than judging anyone on a demo.