Why this matters
Training is the word you will hear most in machine learning, and gradient descent is what training actually is for most models, including the largest language models. If you understand it, you can tell why a model is not learning, why it learns too slowly, and what the settings in a training script mean.
Step one: measure how wrong the model is
A model makes predictions. We compare them with the true answers using a loss, one number that says how wrong the model is overall. A common loss for predicting numbers is the mean squared error: for each record, take the prediction minus the true value, square it so negatives do not cancel positives, then average. Lower is better. Zero would be perfect.
Step two: the rate of change
A derivative tells you how fast one thing changes when you change another. Picture walking on a hill in thick harmattan haze. You cannot see the valley, but you can feel whether the ground slopes up or down under your feet. The derivative of the loss with respect to a weight is that slope: it says whether increasing the weight makes the loss go up or down, and how steeply.
When a model has many weights, the collection of all these slopes is called the gradient.
Step three: take a small step downhill
Gradient descent repeats a simple loop: