AI is poised to become a significant and ubiquitous presence in our lives. It has enormous potential value, but we can't contribute meaningfully to a technology we don't understand.
When a user sets out to buy a new piece of technology, they are not particularly interested in what it might do in the future. A potential user needs to understand what a solution will do for them today, how it will interact with their existing technology infrastructure, and how the current version of that solution will provide ongoing value to their business.
But because this is an emerging space that seemingly changes by the day, it can be difficult for these potential users to know what questions they should ask or how to evaluate products at such an early stage in their life cycle.
This article is a high-level guide to evaluating an AI-based solution as a potential customer: a scorecard for business buyers, if you will.
1.- Solve a business problem and the actors really understand said problem
Chatbots, for example, perform a very specific function that helps promote individual productivity. But can the solution scale to the point where it is used effectively by 100 or 1.000 people?
The fundamentals of enterprise software implementation still apply: customer success, change management and the ability to innovate within the tool are fundamental requirements to deliver continuous value to the business. AI shouldn't be thought of as an incremental solution, just a small piece of magic that completely removes a pain point from the experience.
But it's magic if you can literally make something go away by making it autonomous, which comes down to truly understanding the business problem.
2.- What is the security infrastructure like?
The data security implications around AI are at the next level and far exceed the requirements we are used to. You need built-in security measures that meet or exceed your organization's own standards out of the box.
Today, data, compliance and security are critical for any software and are even more important for IT solutions. Artificial Intelligence. The reason for this is twofold:
First, machine learning models are run against enormous amounts of data, and it can be an unforgiving experience if that data is not handled with strategic care.
With any AI-based solution, regardless of what you aim to achieve, the goal is to have a big impact. Therefore, the audience experiencing the solution will also be large. The way in which the data What these large groups of users generate is very important, as is the type of data being used, when it comes to keeping it secure.
Secondly, you must ensure that any solution you have implemented allows you to maintain control of that data to continuously train machine learning models over time. It's not just about creating a better experience, it's also about ensuring your data doesn't leave your environment.
How is data protected and managed, who has access to it and how is it protected? The ethical use of AI is already a hot topic and will continue to be so with imminent regulations on the way. Any AI solution you implement must have been built with an inherent understanding of these dynamics.
The product must improve over time
As machine learning models age, they begin to drift and draw the wrong conclusions. For example, ChatGPT3 only received data through November 2021, meaning it could not understand any events that occurred after that date.
Enterprise AI solutions must be optimized to adapt to changes over time and keep up with new and valuable data. In the world of finance, a model may have been trained to detect a specific regulation that changes along with new legislation.
A security vendor may train their model to detect a specific threat, but then a new attack vector appears. How are those changes reflected to maintain accurate results over time? When purchasing an AI solution, ask the vendor how they keep their models up to date and what they think about model drift in general.
What is the technical team behind the product like?
Good companies will be able to talk in great detail about the machine learning and artificial intelligence models that underpin the technology. If they can't talk in depth about architecture or training models, that should be an immediate red flag.
Knowing the credentials of the people who manage the models and infrastructure is key. Make sure they understand that the infrastructure needs for AI are very different from the next generation of software. It depends much more on data and requires an analysis of completely different security angles.
The team should have enough collective experience to articulate how they are building their solution and why.
The solution provider understands the use case
The vendor should have information about similar use cases and how they have uniquely benefited from specific features. Ask if they can show you what their peers have accomplished using the solution and how that relates to their own organizational challenges.
The right providers tend to be those who have learned and built from first-hand experience. Be wary of providers who seem to simply be chasing a problem versus those who can truly relate to your problem because they have overcome it themselves.
A good AI vendor should reflect your problem back to you, talk to you about how AI can solve it, and ultimately show you how they've created a solution to do just that.
The tool scales, is supported and secure at the enterprise level, or requires third-party support
There are many tools available that allow companies to create their own reference models internally. This “build versus buy” question often comes up when it comes time for CIOs to purchase new solutions.
With AI, it's not really something you should try to build yourself unless you know it's something that can definitely be maintained over time because you have the necessary resources internally. You have to think of it like building a product, with internal customers instead of external ones.
They have feature requests, support requests, maintenance needs, feedback, and complaints. It's difficult to create a product in new territory like AI. Instead, work with companies that spend day and night learning, building and improving in this specific area.
The supplier has relevant and positive references in the sector or area of competence
You must find examples of other organizations like your own that benefit from the solution. If possible, find out if they are willing to talk to you about their experience.
For sellers, this can be a chicken-and-egg situation. Startups need first customers, but those customers want referrals. The good news is that there is enormous excitement and enthusiasm around AI.
Most clients don't know the right path to take, and because this is such a new space, they are typically much more open to working with early-stage vendors.
But hard work isn't about getting that first check. It's about making those first customers hugely successful so they become advocates for the cause. Risk-averse buyers will wait until others try it and eventually join in. More interesting are suppliers who have invested a lot in achieving success, the type for which they would like to be a reference in the future.
