Examining L1 Regularization
This exercise contains a small, slightly noisy, training data set. In this kind of setting, overfitting is a real concern. Regularization might help, but which form of regularization?
This exercise consists of five related tasks. To simplify comparisons across the five tasks, run each task in a separate tab. Notice that the thicknesses of the lines connecting FEATURES and OUTPUT represent the relative weights of each feature.
Task | Regularization Type | Regularization Rate (lambda) |
---|---|---|
1 | L2 | 0.1 |
2 | L2 | 0.3 |
3 | L1 | 0.1 |
4 | L1 | 0.3 |
5 | L1 | experiment |
Questions:
- How does switching from L2 to L1 regularization influence the delta between test loss and training loss?
- How does switching from L2 to L1 regularization influence the learned weights?
- How does increasing the L1 regularization rate (lambda) influence the learned weights?
(Answers appear just below the exercise.)