Step 1: Gather Data

Gathering data is the most important step in solving any supervised machine learning problem. Your text classifier can only be as good as the dataset it is built from.

If you don’t have a specific problem you want to solve and are just interested in exploring text classification in general, there are plenty of open source datasets available. You can find links to some of them in our GitHub repo. On the other hand, if you are tackling a specific problem, you will need to collect the necessary data. Many organizations provide public APIs for accessing their data—for example, the X API or the NY Times API. You may be able to leverage these APIs for the problem you are trying to solve.

Here are some important things to remember when collecting data:

• If you are using a public API, understand the limitations of the API before using them. For example, some APIs set a limit on the rate at which you can make queries.
• The more training examples (referred to as samples in the rest of this guide) you have, the better. This will help your model generalize better.
• Make sure the number of samples for every class or topic is not overly imbalanced. That is, you should have comparable number of samples in each class.
• Make sure that your samples adequately cover the space of possible inputs, not only the common cases.

Throughout this guide, we will use the Internet Movie Database (IMDb) movie reviews dataset to illustrate the workflow. This dataset contains movie reviews posted by people on the IMDb website, as well as the corresponding labels (“positive” or “negative”) indicating whether the reviewer liked the movie or not. This is a classic example of a sentiment analysis problem.