3. Learning Rate and Epochs¶
Learning rate: how big a step each update takes¶
Every time a network sees training data, it adjusts its weights slightly to do better next time. The learning rate controls how big that adjustment is.
| Learning rate | Effect |
|---|---|
| Too high | Weights swing wildly; the loss graph bounces around instead of smoothly decreasing, and may never settle |
| Too low | Training technically works, but takes far more epochs to get anywhere |
| Well-chosen | Loss decreases steadily and settles in a reasonable number of epochs |
There's no single "correct" learning rate for every problem — it's chosen by experimentation, the same way you experimented with sample variety in Unit 2.
Epochs: how many passes through the data¶
More epochs means more chances to improve — up to a point. Training for far more epochs than needed doesn't keep helping forever; past a certain point, it's how overfitting (Lesson 4) actually happens.
Try it
On the Circle dataset, try an unusually high learning rate and watch the loss graph. Then try an unusually low one. Describe what the loss curve looks like in each case, compared to a well-chosen default.
Tip
If a training run's loss is jumping up and down instead of trending down, check the learning rate first — it's one of the most common causes of unstable training.