Selling the name of the thing is not always selling the thing.

So be cautious when a salesperson comes knocking on your door with an “AI solution” that fixes everything. Anyone I ever worked with who actually builds things calls it machine learning anyway.

Here are some questions to help establish vendor credibility:

Ask them what broad swath of machine learning is involved in the product .- e.g. supervised, unsupervised, a certain type of neural net architecture, etc

Ask them the names of the algorithms behind the AI - XGBoost, BERT, whatever

What does the model predict for? The actual prediction output- is it 1/0, yes/no, a number?

Ask them what objective function (also called loss or cost function) the model is optimizing for - what is the error the model is trying to minimize?

What metrics do they use to measure model efficacy? Closely related to objective function question. How do they know how well the model is able to generalize on unseen data?

Ask them how often they change up their model - cadence and reasons why

How much are humans involved in this “AI solution”? - many automation tasks like NER and document OCR that get sold as AI fall somewhere on the continuum of machine learning and mechanical turk. What’s the split?

Bonus and especially relevant question - are they selling “GPT as a service” (GaaS- trademark pending, thank you, I’ll see myself out) . This means that their user-facing GUI is a thin wrapper around GPT api and there might not be any added benefit (but added costs) with the service.

from “We’ll call it AI to Sell it, Machine Learning to Build it” by Otakar G. Hubschmann