# Embeddings

> [!NOTE]
> **Estimated module length:** 45 minutes

> [!NOTE]
> **Learning objectives**
>
> - Visualize vector representations of word embeddings, such as [word2vec](https://wikipedia.org/wiki/Word2vec).
> - Distinguish encoding from embedding.
> - Describe contextual embedding.

> [!CAUTION]
> **Prerequisites:**
>
> This module assumes you are familiar with the concepts covered in the
> following modules:
>
> - [Introduction to Machine Learning](https://developers.google.com/machine-learning/intro-to-ml)
> - [Linear regression](https://developers.google.com/machine-learning/crash-course/linear-regression)
> - [Working with categorical data](https://developers.google.com/machine-learning/crash-course/categorical-data)
> - [Neural networks](https://developers.google.com/machine-learning/crash-course/neural-networks)

Imagine you're developing a food-recommendation application, where
users input their favorite meals, and the app suggests similar meals
that they might like. You want to develop a machine learning (ML) model
that can predict food similarity, so your app can make high quality
recommendations ("Since you like pancakes, we recommend crepes").

To train your model, you curate a dataset of 5,000 popular
meal items, including ,
,
,
,
and .
![Figure 1. A set of illustrations of five food items. Clockwise from
top-left: borscht, hot dog, salad, pizza, shawarma.](https://developers.google.com/static/machine-learning/crash-course/embeddings/images/food_images.png) **Figure 1.** Sampling of meal items included in the food dataset.

You create a `meal` feature that contains a
[**one-hot encoded**](https://developers.google.com/machine-learning/glossary#one-hot-encoding)
representation of each of the meal items in the dataset.
[**Encoding**](https://developers.google.com/machine-learning/glossary#encoder) refers to the process of
choosing an initial numerical representation of data to train the model on.
![Figure 2. Top: a visualization of the one-hot encoding for borscht.
The vector \[1, 0, 0, 0, ..., 0\] is displayed above six boxes,
each aligned from left
to right with one of the vector numbers. The boxes, from left to right
contain the following images: borscht, hot dog, salad, pizza, \[empty\],
shawarma. Middle: a visualization of the one-hot encoding for hot dog.
The vector \[0, 1, 0, 0, ..., 0\] is displayed above six boxes, each
aligned from left to right with one of the vector numbers. The boxes have
the same images from left to right as for the borscht visualization
above. Bottom: a visualization of the one-hot encoding for shawarma. The
vector \[0, 0, 0, 0, ..., 1\] is displayed above six boxes, each aligned
from left to right with one of the vector numbers. The boxes have
the same images from left to right as for the borscht and hot dog
visualizations.](https://developers.google.com/static/machine-learning/crash-course/embeddings/images/food_images_one_hot_encodings.png) **Figure 2.** One-hot encodings of borscht, hot dog, and shawarma. Each one-hot encoding vector has a length of 5,000 (one entry for each menu item in the dataset). The ellipsis in the diagram represents the 4,995 entries not shown.

## Pitfalls of sparse data representations

Reviewing these one-hot encodings, you notice several problems with this
representation of the data.

- **Number of weights.** Large input vectors mean a huge number of [**weights**](https://developers.google.com/machine-learning/glossary#weight) for a [**neural network**](https://developers.google.com/machine-learning/glossary#neural-network). With M entries in your one-hot encoding, and N nodes in the first layer of the network after the input, the model has to train MxN weights for that layer.
- **Number of datapoints.** The more weights in your model, the more data you need to train effectively.
- **Amount of computation.** The more weights, the more computation required to train and use the model. It's easy to exceed the capabilities of your hardware.
- **Amount of memory.** The more weights in your model, the more memory that is needed on the accelerators that train and serve it. Scaling this up efficiently is very difficult.
- **Difficulty of supporting on-device machine learning (ODML).** If you're hoping to run your ML model on local devices (as opposed to serving them), you'll need to be focused on making your model smaller, and will want to decrease the number of weights.

In this module, you'll learn how to create **embeddings**, lower-dimensional
representations of sparse data, that address these issues.

> [!IMPORTANT]
> **Key terms:**
>
> - [One-hot encoding](https://developers.google.com/machine-learning/glossary#one-hot-encoding)
> - [Neural network](https://developers.google.com/machine-learning/glossary#neural-network)
> - [Weight](https://developers.google.com/machine-learning/glossary#weight)

[Help Center](https://support.google.com/machinelearningeducation)