How does step size affect gradient descent toward a minimum?

A computer can repeatedly adjust a number in the direction that seems to reduce an error—and still make that error grow. The direction alone is not enough. How far each adjustment moves matters. Gradient descent makes this relationship precise by using the slope of a loss function to choose an update.

Take the loss L(x) = (x − 1)². It measures squared error from a target value of one. At x = 4, the error is three and the loss is nine. At x = 1, the loss is zero, the bottom of the parabola. This small example gives us something unusual: both the correct destination and every step of the journey can be calculated exactly.

The derivative L′(x) = 2(x − 1) tells us the local slope. To the right of one it is positive, so moving left can reduce the loss. To the left it is negative, so moving right can help. Gradient descent uses x next = x − αL′(x), where the positive step size α controls how strongly the slope changes x. The algorithm reads the slope at its current position; it does not directly measure the entire journey to the minimum.

Starting at four with α = 0.25, the first slope is six. The update subtracts 0.25 × 6, giving 2.5. Repeating gives 1.75 and then 1.375. Their losses are 2.25, 0.5625 and 0.140625, down from the starting nine. The steps shrink naturally because the slope becomes smaller near the bottom. A small slope does not mean the computer stopped working; it means the same update rule is now making smaller corrections.

To understand every step at once, subtract one from both sides of the update. The new error is (1 − 2α) times the old error. For α = 0.25, that multiplier is one half. For α = 0.75, it is minus one half: starting at four now gives −0.5, 1.75 and 0.625. The values cross the minimum repeatedly, yet the error's magnitude still shrinks. Crossing the target is therefore not, by itself, a sign of failure.

With α = 1, the multiplier is minus one. The values alternate between four and minus two, keeping the loss at nine. With α = 1.25, the multiplier is minus one and a half, so errors grow as their signs alternate. From a nonoptimal start, convergence on this quadratic requires |1 − 2α| less than one, or 0 < α < 1. At α = 0.5, this particular loss reaches its minimum in one update.

Real learning problems can contain millions of adjustable values, differently curved directions and gradients estimated from samples. Their suitable step sizes cannot be copied from this one parabola. Yet the central insight survives: an adjustment combines local information with a chosen response strength. Making that response stronger can speed progress, cause harmless crossing or turn correction into growing instability.