Skip to content

3. Learning Rate and Epochs

Learning rate: how big a step each update takes

Every time a network sees training data, it adjusts its weights slightly to do better next time. The learning rate controls how big that adjustment is.

Learning rate Effect
Too high Weights swing wildly; the loss graph bounces around instead of smoothly decreasing, and may never settle
Too low Training technically works, but takes far more epochs to get anywhere
Well-chosen Loss decreases steadily and settles in a reasonable number of epochs

There's no single "correct" learning rate for every problem — it's chosen by experimentation, the same way you experimented with sample variety in Unit 2.

Epochs: how many passes through the data

More epochs means more chances to improve — up to a point. Training for far more epochs than needed doesn't keep helping forever; past a certain point, it's how overfitting (Lesson 4) actually happens.

Try it

On the Circle dataset, try an unusually high learning rate and watch the loss graph. Then try an unusually low one. Describe what the loss curve looks like in each case, compared to a well-chosen default.

Tip

If a training run's loss is jumping up and down instead of trending down, check the learning rate first — it's one of the most common causes of unstable training.

Next: Overfitting →