To understand the problem, perform the following tasks:
- State the goal for the product you are developing or refactoring.
- Determine whether the goal is best solved using predictive ML, generative AI, or a non-ML solution.
- Verify you have the data required to train a model if you're using a predictive ML approach.
State the goal
Begin by stating your goal in non-ML terms. The goal is the answer to the question, "What am I trying to accomplish?"
The following table clearly states goals for hypothetical apps:
| Application | Goal |
|---|---|
| Weather app | Calculate precipitation in six-hour increments for a geographic region. |
| Fashion app | Generate a variety of shirt designs. |
| Video app | Recommend useful videos. |
| Mail app | Detect spam. |
| Financial app | Summarize financial information from multiple news sources. |
| Map app | Calculate travel time. |
| Banking app | Identify fraudulent transactions. |
| Dining app | Identify cuisine by a restaurant's menu. |
| Ecommerce app | Reply to reviews with helpful answers. |
Clear use case for ML
Some view ML as a universal tool that can be applied to all problems. In reality, ML is a specialized tool suitable only for particular problems. You don't want to implement a complex ML solution when a simpler non-ML solution will work.
ML systems can be divided into two broad categories: predictive ML and generative AI. The following table lists their defining characteristics:
| Input | Output | Training technique | |
|---|---|---|---|
| Predictive ML |
Text Image Audio Video Numerical |
Makes a prediction, for example, classifying an email as spam or not spam, guessing tomorrow's rainfall, or predicting the price of a stock. The output can typically be verified against reality. | Typically uses lots of data to train a supervised, unsupervised, or reinforcement learning model to perform a specific task. |
| Generative AI |
Text Image Audio Video Numerical |
Generates output based on the user's intent, for example, summarizing an article, or producing an audio clip or short video. | Typically uses lots of unlabeled data to train a large language model or image generator to fill in missing data. The model can then be used for tasks that can be framed as fill-in-the-blank tasks, or it can be fine-tuned by training it on labeled data for some specific task, like classification. |
To confirm that ML is the right approach, first verify that your current non-ML solution is optimized. If you don't have a non-ML solution implemented, try solving the problem manually using a heuristic.
The non-ML solution is the benchmark you'll use to determine whether ML is a good use case for your problem. Consider the following questions when comparing a non-ML approach to an ML one:
Quality. How much better do you think an ML solution can be? If you think an ML solution might be only a small improvement, that might indicate the current solution is the best one.
Cost and maintenance. How expensive is the ML solution in both the short- and long-term? In some cases, it costs significantly more in terms of compute resources and time to implement ML. Consider the following questions:
- Can the ML solution justify the increase in cost? Note that small improvements in large systems can easily justify the cost and maintenance of implementing an ML solution.
- How much maintenance will the solution require? In many cases, ML implementations need dedicated long-term maintenance.
- Does your product have the resources to support training or hiring people with ML expertise?
Check Your Understanding
Predictive ML and data
Data is the driving force of predictive ML. To make good predictions, you need data that contains features with predictive power. Your data should have the following characteristics:
Abundant. The more relevant and useful examples in your dataset, the better your model will be.
Consistent and reliable. Having data that's consistently and reliably collected will produce a better model. For example, an ML-based weather model will benefit from data gathered over many years from the same reliable instruments.
Trusted. Understand where your data will come from. Will the data be from trusted sources you control, like logs from your product, or will it be from sources you don't have much insight into, like the output from another ML system?
Available. Make sure all inputs are available at prediction time in the correct format. If it will be difficult to obtain certain feature values at prediction time, omit those features from your datasets.
Correct. In large datasets, it's inevitable that some labels will have incorrect values, but if more than a small percentage of labels are incorrect, the model will produce poor predictions.
Representative. The datasets should be as representative of the real world as possible. In other words, the datasets should accurately reflect the events, user behaviors, and/or the phenomena of the real world being modeled. Training on unrepresentative datasets can cause poor performance when the model is asked to make real-world predictions.
If you can't get the data you need in the required format, your model will make poor predictions.
Predictive power
For a model to make good predictions, the features in your dataset should have predictive power. The more correlated a feature is with a label, the more likely it is to predict it.
Some features will have more predictive power than others. For example, in a
weather dataset, features such as cloud_coverage, temperature, and
dew_point would be better predictors of rain than moon_phase or
day_of_week. For the video app example, you could hypothesize that features
such as video_description, length and views might be good predictors for
which videos a user would want to watch.
Be aware that a feature's predictive power can change because the context or
domain changes. For example, in the video app, a feature like upload_date
might—in general—be weakly correlated with the label. However, in
the sub-domain of gaming videos, upload_date might be strongly correlated with
the label.
Determining which features have predictive power can be a time consuming process. You can manually explore a feature's predictive power by removing and adding it while training a model. You can automate finding a feature's predictive power by using algorithms such as Pearson correlation, Adjusted mutual information (AMI), and Shapley value, which provide a numerical assessment for analyzing the predictive power of a feature.
Check Your Understanding
For more guidance on analyzing and preparing your datasets, see Data Preparation and Feature Engineering for Machine Learning.
Predictions vs. actions
There's no value in predicting something if you can't turn the prediction into an action that helps users. That is, your product should take action from the model's output.
For example, a model that predicts whether a user will find a video useful should feed into an app that recommends useful videos. A model that predicts whether it will rain should feed into a weather app.
Check Your Understanding
Based on the following scenario, determine if using ML is the best approach to the problem.
An engineering team at a large organization is responsible for managing incoming phone calls.
The goal: To inform callers how long they'll wait on hold given the current call volume.
They don't have any solution in place, but they think a heuristic would be to divide the current number of customers on hold by the number of employees answering phones, and then multiply by 10 minutes. However, they know that some customers have their issues resolved in two minutes, while others can take up to 45 minutes or longer.
Their heuristic probably won't get them a precise enough number. They
can create a dataset with the following columns:
number_of_callcenter_phones, user_issue,
time_to_resolve, call_time,
time_on_hold.
time_on_hold.