# Step 5: Tune Hyperparameters

We had to choose a number of hyperparameters for defining and training the
model. We relied on intuition, examples and best practice recommendations. Our
first choice of hyperparameter values, however, may not yield the best results.
It only gives us a good starting point for training. Every problem is different
and tuning these hyperparameters will help refine our model to better represent
the particularities of the problem at hand. Let's take a look at some of the
hyperparameters we used and what it means to tune them:

- **Number of layers in the model** : The number of layers in a neural network is
  an indicator of its complexity. We must be careful in choosing this value. Too
  many layers will allow the model to learn too much information about the
  training data, causing overfitting. Too few layers can limit the model's
  learning ability, causing underfitting. For text classification datasets, we
  experimented with one, two, and three-layer MLPs. Models with two layers
  performed well, and in some cases better than three-layer models. Similarly, we
  tried [sepCNN](https://developers.google.com/machine-learning/glossary?utm_source=DevSite&utm_campaign=Text-Class-Guide&utm_medium=referral&utm_content=glossary&utm_term=sepCNN#depthwise-separable-convolutional-neural-network-sepcnn)s
  with four and six layers, and the four-layer models performed well.

- **Number of units per layer** : The units in a layer must hold the information
  for the transformation that a layer performs. For the first layer, this is
  driven by the number of features. In subsequent layers, the number of units
  depends on the choice of expanding or contracting the representation from the
  previous layer. Try to minimize the information loss between layers. We tried
  unit values in the range `[8, 16, 32, 64]`, and 32/64 units worked well.

- **Dropout rate** : Dropout layers are used in the model for
  [regularization](https://developers.google.com/machine-learning/glossary/?utm_source=DevSite&utm_campaign=Text-Class-Guide&utm_medium=referral&utm_content=glossary&utm_term=dropout-regularization#dropout_regularization).
  They define the fraction of input to drop as a precaution for overfitting.
  Recommended range: 0.2--0.5.

- **Learning rate**: This is the rate at which the neural network weights change
  between iterations. A large learning rate may cause large swings in the weights,
  and we may never find their optimal values. A low learning rate is good, but the
  model will take more iterations to converge. It is a good idea to start low, say
  at 1e-4. If the training is very slow, increase this value. If your model is not
  learning, try decreasing learning rate.

There are couple of additional hyperparameters we tuned that are specific to our
sepCNN model:

1. **Kernel size**: The size of the convolution window. Recommended values: 3 or
   5.

2. **Embedding dimensions**: The number of dimensions we want to use to represent
   word embeddings---i.e., the size of each word vector. Recommended values: 50--300.
   In our experiments, we used GloVe embeddings with 200 dimensions with a pre-
   trained embedding layer.

Play around with these hyperparameters and see what works best. Once you have
chosen the best-performing hyperparameters for your use case, your model is
ready to be deployed.