Choosing AI Tools Safely: A Vendor Evaluation Framework
A practical framework for evaluating AI vendors on security, compliance, and data handling. The questions to ask before you sign.
A practical framework for evaluating AI vendors on security, compliance, and data handling. The questions to ask before you sign.
AI tools are appearing everywhere, from standalone SaaS products to features embedded in software you already use. The rush to adopt means many organisations are using AI tools without properly evaluating how they handle data.
This creates real risk. Your customer data, employee records, financial information, or proprietary knowledge could be stored in unknown locations, used for model training, or accessible to the vendor's staff.
For every AI tool your organisation considers, evaluate these five areas:
Before signing up for any AI tool that will handle business data:
| Question | Ideal answer | Red flag |
|---|---|---|
| Is data used for model training? | No, with opt-out by default | "By default, yes" with no opt-out |
| Where is data stored? | Specific region (e.g., AWS Sydney) | "Various global data centres" |
| Can data be deleted? | Yes, on request with confirmation | "Data may be retained indefinitely" |
| Third-party access? | No sub-processors without disclosure | Unnamed third-party processing |
| Encryption? | AES-256 at rest, TLS 1.2+ in transit | "Encryption where appropriate" |
Our approach: Self-hosted or Bedrock-based AI means your data never leaves your AWS account. No vendor training, no third-party access, full audit control.
Check five areas for every AI tool: data residency, data usage, security controls, compliance and data retention. Find out where data is processed and stored and whether that is in Australia, whether your data trains the vendor's model and if you can opt out, what encryption, access controls and audit logging exist, which security certifications the vendor holds, and how long data is kept and whether it can be deleted.
Ask where the vendor's servers are and whether it has Australian data centres, whether data is encrypted at rest and in transit, and who at the vendor can access your data. Also ask about SOC 2 Type II or ISO 27001 certification, what happens to your data if you cancel and how long deletion takes, whether there is a data processing agreement (DPA) to review, and whether the vendor will notify you of breaches.
Free AI tools are usually the weakest choice for business data, because free tiers usually have the weakest privacy protections. If the product is free, your data is likely the product. Pay for the plan that matches the business's risk profile, and check for an opt-out from model training, because business data should never train someone else's model without explicit consent.
Red flags include vague privacy policies, no opt-out from model training, free tools with weak protections, no data processing agreement, and data leaving Australia without disclosure. Vague answers are warning signs too: storage in various global data centres, data that may be retained indefinitely, unnamed third-party processing, or encryption only where appropriate. If a vendor cannot clearly explain where data goes and how it is used, assume the worst.
If business data leaves Australia without disclosure, Australian Privacy Principle 8 (APP 8) creates legal liability for the business using the tool. That is why data residency is the first area to check. Ask where data is processed and stored, and look for a specific region, such as Amazon Web Services (AWS) Sydney, rather than a vague answer about various global data centres.
Ask the author
Ask it here and it comes straight to the founder. No sales call, no obligation, and a real answer even if the answer is that you do not need us.
Kasun Wijayamanna
Founder, replies within one business day
Tell us what you are working on. We will come back with a practical recommendation and clear next steps.
Built here. Your data stays here.
Thanks for reaching out. We will get back to you within one business day.
See what else we do