Neural networks

Estimated module length: 75 minutes

Learning objectives

• Explain the motivation for building neural networks, and the use cases they address.
• Define and explain the function of the key components of a deep neural network architecture:
  • Nodes
  • Hidden layers
  • Activation functions
• Develop intuition around how neural network predictions are made, by stepping through the inference process.
• Build a high-level intuition of how neural networks are trained, using the backpropagation algorithm.
• Explain how neural networks can be used to perform two types of multi-class classification: one-vs.-all and one-vs.-one.

Prerequisites:

This module assumes you are familiar with the concepts covered in the following modules:

• Introduction to Machine Learning
• Linear regression
• Logistic regression
• Classification
• Working with numerical data
• Working with categorical data
• Datasets, generalization, and overfitting

You may recall from the Feature cross exercises in the Categorical data module, that the following classification problem is nonlinear:

Figure 1. Nonlinear classification problem. A linear function cannot cleanly separate all the blue dots from the orange dots.

"Nonlinear" means that you can't accurately predict a label with a model of the form \(b + w_1x_1 + w_2x_2\). In other words, the "decision surface" is not a line.

However, if we perform a feature cross on our features $x_1$ and $x_2$, we can then represent the nonlinear relationship between the two features using a linear model: $b + w_1x_1 + w_2x_2 + w_3x_3$ where $x_3$ is the feature cross between $x_1$ and $x_2$:

Figure 2. By adding the feature cross x₁x₂, the linear model can learn a hyperbolic shape that separates the blue dots from the orange dots.

Now consider the following dataset:

Figure 3. A more difficult nonlinear classification problem.

You may also recall from the Feature cross exercises that determining the correct feature crosses to fit a linear model to this data took a bit more effort and experimentation.

But what if you didn't have to do all that experimentation yourself? Neural networks are a family of model architectures designed to find nonlinear patterns in data. During training of a neural network, the model automatically learns the optimal feature crosses to perform on the input data to minimize loss.

In the following sections, we'll take a closer look at how neural networks work.

Key terms:

• Activation function
• Hidden layer
• Linear model
• Model
• Neural network
• Nodes
• Nonlinear