How to Choose a Generative AI Development Company: 10 Questions to Ask

Almost every generative AI development company can show you a demo. A chatbot answering questions over a handful of documents, a summariser, a copilot inside a familiar tool. Demos are cheap to build and they all look good on five hand-picked examples.

What separates partners is what happens after the demo: real data, real users, a monthly bill, and a model that will sometimes be wrong. These ten questions are the ones we would ask if we were on your side of the table. A strong team will answer them specifically and without hesitation.

1. What have you put into production, and who uses it today?

Ask for systems that real users depend on now, not prototypes or hackathon builds. Ask how many people use it, how long it has been running, and what broke in the first month. A team that has run GenAI in production will have stories about latency, cost spikes and edge cases. A team that has only built demos will talk about models.

2. How do you decide between RAG, fine-tuning and prompting?

There is no single right answer, which is exactly why the question is useful. A good partner asks about your data before recommending an approach: how much there is, how often it changes, and whether answers must cite their source. If the answer is the same technique for every problem, keep looking. We wrote about this trade-off in RAG vs fine-tuning.

3. How will you measure quality before launch?

"It looks good" is not a test. Ask how they build an evaluation set from your own data, what success criteria you will agree before building, and how they catch regressions when a prompt or model changes. Prompts should be versioned and tested like code, with results you can see.

4. What happens when the model is wrong?

Every model will produce a wrong answer at some point. The question is what the system does about it. Look for output validation, checks against source documents, confidence thresholds below which the system says it does not know, and a route to a human for high-stakes decisions. Be wary of anyone who promises the model will not make mistakes.

5. What will it cost to run, not just to build?

GenAI has a running cost that grows with usage: model calls, embeddings, vector storage and hosting. Ask for an estimate of the monthly running cost at your expected volume, and how they keep it under control: caching repeated queries, routing simple requests to smaller models, and tracking cost per request from day one.

6. Where will our data go?

Ask which model providers will see your data, in which region it is processed and stored, whether any provider may use it for training, how long it is retained, and who on their team can access it. If you work in a regulated industry, ask how they have handled requirements such as HIPAA or GDPR before, and for an example.

7. Who owns the code, prompts and evaluation sets?

You should own everything built for you: source code, prompts, evaluation data, infrastructure definitions and documentation. Ask what the handover looks like and whether anything depends on a proprietary platform you would have to keep paying for.

8. How do you handle model and provider changes?

Model providers retire models, change prices and have outages. A system wired directly to one model version is fragile. Ask whether they put a thin layer between your application and the provider, so a model can be swapped or a second provider can take over without rewriting the product.

9. Who will actually work on our project?

Ask for the names and roles of the people on your project, the balance of senior and junior engineers, the working hours that overlap with yours, and what happens if someone leaves. The team in the sales meeting is not always the team that builds.

10. How do you structure the engagement?

A sensible structure starts small: a short discovery to confirm the use case and audit the data, then a fixed-scope pilot with clear exit criteria, then the full build. Ask which engagement models they offer, such as a fixed project, a dedicated team or staff augmentation, and when each one fits.

Red flags worth noticing

• No questions about your data, your users or your current systems.
• Promises of a fixed accuracy figure before anyone has seen your data.
• Pricing by headcount only, with no milestones or acceptance criteria.
• No plan for monitoring, cost tracking or support after launch.
• Everything depends on their own platform, with no clean handover.

How we answer these questions

These are the questions we expect from clients, and we would rather answer them in the first call than after a contract is signed. Our Data and AI engineering practice builds GenAI applications, retrieval systems and data platforms, and we also run AI in production ourselves, in products such as QuikSignal. For more on taking a prototype to production, read what we learned shipping GenAI to production, or get in touch to talk through your project.