Choosing AI Tools Safely: A Vendor Evaluation Framework

A practical framework for evaluating AI vendors on security, compliance, and data handling. The questions to ask before you sign.

Why safe AI selection matters

AI tools are appearing everywhere, from standalone SaaS products to features embedded in software you already use. The rush to adopt means many organisations are using AI tools without properly evaluating how they handle data.

This creates real risk. Your customer data, employee records, financial information, or proprietary knowledge could be stored in unknown locations, used for model training, or accessible to the vendor's staff.

Evaluation framework

For every AI tool your organisation considers, evaluate these five areas:

  1. Data residency: Where is data processed and stored? Is it in Australia?
  2. Data usage: Is your data used to train or improve the AI model? Can you opt out?
  3. Security controls: Encryption, access controls, audit logging?
  4. Compliance: SOC 2, ISO 27001, GDPR, or Australian-specific certifications?
  5. Data retention: How long is your data kept? Can you request deletion?

Security questions to ask

Before signing up for any AI tool that will handle business data:

  • Where are your servers located? Do you have Australian data centres?
  • Is data encrypted at rest and in transit? What encryption standards?
  • Who at your company can access our data? What access controls exist?
  • Do you have SOC 2 Type II certification? ISO 27001?
  • What happens to our data if we cancel? How long until deletion?
  • Do you have a data processing agreement (DPA) we can review?
  • What's your incident response process? Will you notify us of breaches?

Data handling checklist

Question Ideal answer Red flag
Is data used for model training?No, with opt-out by default"By default, yes" with no opt-out
Where is data stored?Specific region (e.g., AWS Sydney)"Various global data centres"
Can data be deleted?Yes, on request with confirmation"Data may be retained indefinitely"
Third-party access?No sub-processors without disclosureUnnamed third-party processing
Encryption?AES-256 at rest, TLS 1.2+ in transit"Encryption where appropriate"

Red flags to watch for

  • Vague privacy policies: If they cannot clearly explain where data goes and how it is used, assume the worst.
  • No opt-out for model training: Your business data should never be used to train someone else's model without explicit consent.
  • "Free" AI tools: If the product is free, your data is likely the product. Free tiers usually have the weakest privacy protections.
  • No data processing agreement: Any enterprise-grade AI vendor should provide a DPA.
  • Data leaves Australia without disclosure: Under APP 8, this creates legal liability for you.

Our approach: Self-hosted or Bedrock-based AI means your data never leaves your AWS account. No vendor training, no third-party access, full audit control.

Key takeaways

  • Not all AI tools handle your data the same way. Evaluate before you adopt.
  • Key areas: data storage location, training opt-out, encryption, access controls, and compliance certifications.
  • Ask explicit questions about where your data goes, who can access it, and how it's retained.
  • Free tiers often have the weakest data protections. Pay for the plan that matches your risk profile.
Kasun Wijayamanna
Kasun Wijayamanna Founder & Lead Developer

Postgraduate Researcher (AI & RAG), Curtin University - Western Australia

View profile →

Choosing AI tools safely: common questions

What should I check before my business adopts an AI tool?

Check five areas for every AI tool: data residency, data usage, security controls, compliance and data retention. Find out where data is processed and stored and whether that is in Australia, whether your data trains the vendor's model and if you can opt out, what encryption, access controls and audit logging exist, which security certifications the vendor holds, and how long data is kept and whether it can be deleted.

What security questions should I ask an AI vendor?

Ask where the vendor's servers are and whether it has Australian data centres, whether data is encrypted at rest and in transit, and who at the vendor can access your data. Also ask about SOC 2 Type II or ISO 27001 certification, what happens to your data if you cancel and how long deletion takes, whether there is a data processing agreement (DPA) to review, and whether the vendor will notify you of breaches.

Are free AI tools safe to use with business data?

Free AI tools are usually the weakest choice for business data, because free tiers usually have the weakest privacy protections. If the product is free, your data is likely the product. Pay for the plan that matches the business's risk profile, and check for an opt-out from model training, because business data should never train someone else's model without explicit consent.

What are the red flags when evaluating an AI vendor?

Red flags include vague privacy policies, no opt-out from model training, free tools with weak protections, no data processing agreement, and data leaving Australia without disclosure. Vague answers are warning signs too: storage in various global data centres, data that may be retained indefinitely, unnamed third-party processing, or encryption only where appropriate. If a vendor cannot clearly explain where data goes and how it is used, assume the worst.

Why does it matter if an AI tool stores data outside Australia?

If business data leaves Australia without disclosure, Australian Privacy Principle 8 (APP 8) creates legal liability for the business using the tool. That is why data residency is the first area to check. Ask where data is processed and stored, and look for a specific region, such as Amazon Web Services (AWS) Sydney, rather than a vague answer about various global data centres.

Ask the author

Still have a question?

Ask it here and it comes straight to the founder. No sales call, no obligation, and a real answer even if the answer is that you do not need us.

Kasun Wijayamanna, Founder Kasun Wijayamanna
Founder, replies within one business day

Ready to discuss your project?

Tell us what you are working on. We will come back with a practical recommendation and clear next steps.

Australian owned and operated

Built here. Your data stays here.

  • No offshore development. Everything is written by our own team in Australia. Nothing is subcontracted overseas.
  • Your data stays onshore. Hosted in Australia, on infrastructure you own, under Australian law.
  • Every state, not just ours. Perth, Melbourne, Sydney, Brisbane, Adelaide and everywhere between.