Evaluating AI Vendors: Questions to Ask Before You Buy

Every week new AI products promise to transform your sales, support or operations. Some deliver real value; many are thin wrappers on the same underlying models, with big claims and small guarantees. A structured evaluation protects your budget, your data and your reputation. Use this checklist before you sign anything.

Start with your problem, not their demo

Define the problem and success criteria first:

What task will this solve, for whom?
What does success look like in numbers (time saved, error rate, response time, revenue)?
What is the current process and cost?
What are the must-haves versus nice-to-haves?
What are the risks if the tool fails?

A polished demo on perfect data says little about your messy reality.

1. Product and performance

Which exact tasks does it perform, and which does it not?
Can we test it on our own data in a pilot, not only on sample data?
What accuracy or quality metrics do you measure, and how? Can we see results on cases similar to ours?
How does it handle our languages, including Nepali, mixed language or local formats and calendars?
What happens when it is uncertain or wrong? Is there a confidence indicator?
Is there human review built in?
How does performance degrade with unusual inputs?

Ask for a trial and prepare 50 to 100 real examples, with answers known to you, to compare vendors fairly.

2. Technology and transparency

Which models does it use: its own, a third-party model or several? Can you change models?
How is it improved over time, and will changes alter behaviour without notice?
Do they publish model cards, evaluation reports or documentation?
What are the known limitations?
Can it be explained or audited?

3. Data privacy and ownership

This is often the most important section.

What data will you collect, where is it stored (country or region) and for how long?
Do you use our data to train your models or anyone else's? Can we opt out in writing?
Who can access our data inside your company? What subprocessors (cloud providers, AI model providers) are involved?
How do you handle deletion at our request and at contract end?
Do you comply with relevant privacy laws and can you sign a data processing agreement?
Who owns the outputs and any fine-tuned models or derived data?
Can we export our data in a standard format at any time?

4. Security

Do you have independent security certifications or audit reports (for example SOC 2 or ISO 27001)?
How is data encrypted in transit and at rest?
How do you manage access: single sign-on, role-based permissions, multi-factor authentication, audit logs?
How do you handle vulnerabilities, penetration testing and incident response? What is your breach-notification timeline?
How do you protect against prompt injection and data leakage between customers?

5. Compliance and risk

Which regulations apply to your product and our use (privacy, sector rules, AI-specific laws)?
What are the product's risk classification and documentation for regulated uses?
Do you provide indemnities for intellectual-property claims related to outputs?
What controls exist for bias and safety testing?

6. Integration and operations

Which systems does it integrate with (email, CRM, ERP, billing, messaging apps)? Is there an API?
How much setup, training and change management is needed?
What is the uptime commitment (service level agreement) and what compensation applies for failures?
What are the limits on usage, rate limits and file sizes?
How do updates and downtime get communicated?

7. Pricing and costs

What is the pricing model: per user, per usage, per token, per document?
Which costs are hidden: setup, integration, premium support, overage fees, extra storage?
How will costs grow if usage doubles?
Are there annual lock-ins, auto-renewals or price-increase clauses?
What is the total cost over three years compared with building or using a general tool?

8. Support and vendor stability

What support hours and languages are available? Time zone differences matter.
Who will be our contact, and how are issues escalated?
How long has the company existed? What is its funding situation? What happens to our data and service if it shuts down or is acquired?
Can we speak with reference customers of similar size and industry?
What does the roadmap look like and how are customer requests handled?

9. Exit strategy

Always ask how you can leave.

Can we export all data, configurations and prompts?
Is there a transition period?
Are there penalties for early termination?
How is data deleted after termination, and can they certify it?

A simple scoring sheet

Rate each vendor from 1 to 5 on the criteria that matter, weighted by importance.

weights = {"quality_on_our_data": 5, "privacy": 5, "security": 4, "integration": 3,
           "total_cost": 3, "support": 2, "exit_options": 2}

vendors = {
    "Vendor A": {"quality_on_our_data": 4, "privacy": 3, "security": 4, "integration": 5,
                 "total_cost": 3, "support": 4, "exit_options": 2},
    "Vendor B": {"quality_on_our_data": 3, "privacy": 5, "security": 4, "integration": 3,
                 "total_cost": 4, "support": 3, "exit_options": 4},
}

for name, scores in vendors.items():
    total = sum(weights[k] * scores[k] for k in weights)
    maximum = sum(weights.values()) * 5
    print(f"{name}: {total}/{maximum} ({total / maximum:.0%})")

A scoring sheet will not replace judgement, but it makes trade-offs visible and helps teams agree.

Red flags

Refusal to let you test with your data
Vague answers about data use and training
Claims like "100% accuracy" or "replaces your whole team"
No security documentation or evasive answers
Pressure to sign quickly
Pricing that is unclear
No export or exit options
No references from comparable customers

Pilot before committing

Run a time-boxed pilot with a clear plan: scope, success metrics, test data, responsible people and a decision date. Include real users in feedback. Measure against the baseline and against alternatives, including doing nothing or using a general-purpose AI assistant.

Frequently Asked Questions

Should I build instead of buy?

Build when the task is core to your business, requires deep customisation or involves sensitive data and you have the skills. Buy for common, well-solved problems.

How long should a pilot last?

Often 2 to 6 weeks, long enough to see real usage patterns but short enough to keep momentum.

Do I need legal review?

For contracts involving personal data, regulated sectors or significant spending, yes.

What if the vendor changes the underlying model?

Ask for notice of changes and the ability to test them, and keep your own evaluation set to detect regressions.

Conclusion

Evaluate AI vendors on real data, with clear success criteria, and scrutinise privacy, security, costs and exit options. A careful pilot and a written checklist protect you from hype and lock-in.