What does it take to make an AI project successful? Considering their high failure rate, it’s a mistake to jump in too quickly without asking this question.
I see success factors in four categories:
- Data,
- Domain knowledge,
- Technology, and
- Team.
Most data scientists are taught disproportionately about technology, with little emphasis on the other three areas. One of the goals of this blog is to correct this asymmetry, ensuring that you have all of the building blocks you’ll need to succeed. In future posts, I’ll be writing in more detail about these topics. Here, I highlight some misconceptions I’ve seen widely held in each one, to get you started.
Data
Myth #1: You need a large quantity of pre-existing data to start a project
Many AI projects begin with the idea that “we have a lot of existing data, maybe we can use it to build an AI model.” Although this is often a good way to start, there are many AI startups that don’t begin this way. Instead, they build a mechanism for accumulating data over time.
For example, Google Voice Local Search was deployed from 2007 to 2010. It allowed users to call GOOG-411 to inquire about the closest restaurant, a fun fact, or local information like what time a dry cleaner closed. The service created training data as a byproduct: Google used the user-provided phone call voice information as raw data to improve its voice recognition performance. Captcha plays a similar game with images.
Myth #2: You can build an ML model just once
In many environments, the situation changes, and so your ML model needs to be updated regularly as well. For example, a model that predicts stock prices, security intrusions, or customer behavior will probably need to be updated regularly to maintain high accuracy. This is not always the case, however, as sometimes patterns are stable over time. A system that recognizes cancer lesions accurately in 1980, for example, will probably not need to be updated for 1990. (And note that a widespread misconception is that “machine learning” means “a model that is updated over time”).
Domain Knowledge
Myth #3: You need domain knowledge to create a functional ML model
A good data set is often enough to create a model that performs well enough to create business value, even without domain knowledge. Creating a successful AI company, on the other hand, is a different matter. To properly select, update, and find a good market fit for an AI product, domain knowledge is usually essential.
Technology
Myth #4: A core technology skill needed in an AI project is the ability to create new algorithms
In many academic AI programs today, students are taught the math inside AI learning systems, with the goal of knowing enough to create new algorithms. An applied environment is different, and much less widely taught: there are dozens of algorithms, and our goal is to use them, usually not to create new ones. In this newer and more challenging domain, critical skills include data manipulation, data analysis, selecting the right algorithm, training and testing a model effectively, and packaging the model into a useful application.
This doesn’t mean that new algorithms will not be needed in applications. There may be innovative algorithms required for very new kinds of applications, or for successfully using the AI system within the context of an application. The point is to recognize that an AI project can often go far without any new algorithm invention.
Myth #5: The only valuable AI use cases are fully autonomous ones
It’s a common misconception that AI technologies will replace humans. But the truth is that many new AI systems will include a combination of humans and autonomous elements, as I introduced in this previous post.
Such an integrated approach is often a better choice, as measured in cost and benefit. For many use cases, companies that determine how to integrate AI systems into their human teams will outpace those that rely on AI or humans alone. Put succinctly: it’s not about AI’s replacing humans, but teams/companies using humans and AI well replacing those that don’t.
Team
Myth #6: AI success depends on having all required key capabilities within your company or team
For many situations it makes sense to outsource expertise in the elements of the key capability categories: Data, Domain Knowledge, and Technology. You don’t need full-time dedicated people for each one, and it’s often financially or practically infeasible.
For example, it is common for a company to start with domain knowledge and to struggle with AI, in which case a partnership with an AI consultant can be much more effective than hiring a full-time AI resource or team. Another approach is to use an AI data preparation specialist. Or an AI specialist company may hire domain experts as it targets a particular use case.
I’ll have much more to say on the key capabilities you’ll need for a successful AI project in future posts. Avoiding the above common myths is a good place to start.
