How a model learns

Step 1

Give a mistake a number

A loss function measures how far predictions are from targets. For a numerical prediction, one useful loss is squared error: (prediction - target) ** 2. Squaring makes the error nonnegative and penalizes large misses more strongly.

For several examples, we average those squared errors. This is mean squared error, or MSE. A lower loss is better on the examples being measured; it does not automatically mean the model will work on new examples.

prediction = 4.0
target = 7.0
loss = (prediction - target) ** 2
print("Squared error:", loss)

Step 2

Find which direction is downhill

A derivative is a local slope: how much a result changes when one input changes a little. A gradient collects these slopes for the parameters. You do not need a calculus course to try a slope numerically: make a tiny nudge, measure the change in loss, and divide by the size of the nudge.

A positive slope says increasing the parameter raises the loss locally. To head downhill, move in the opposite direction. The learning rate controls the size of that move. Too large can overshoot; too small can take a long time.

def loss_at(weight):
    prediction = weight * 3.0 + 1.0
    return (prediction - 7.0) ** 2

weight = 0.0
epsilon = 0.0001
slope = (loss_at(weight + epsilon) - loss_at(weight - epsilon)) / (2 * epsilon)
new_weight = weight - 0.05 * slope
print("Slope:", round(slope, 3))
print("Loss before:", loss_at(weight))
print("Loss after:", round(loss_at(new_weight), 3))

Step 3

Repeat the update: a training loop

Here the invented data follows target = 2 * x + 1. We know that rule, but the training calculation only receives input/target pairs.

For one example, let error = weight * x + bias - target. Its squared-error slope with respect to weight is 2 * error * x; with respect to bias it is 2 * error. Average each slope across the examples, then update both parameters. This is the chain rule in action: a change in weight changes the prediction, which changes the loss.

One epoch is one pass over the training examples. This tiny course uses the whole dataset as one batch, so one epoch is also one update. Larger datasets usually take several batch updates per epoch.

xs = [-2.0, -1.0, 0.0, 1.0, 2.0]
targets = [2 * x + 1 for x in xs]
weight, bias = 0.0, 0.0
learning_rate = 0.1

for epoch in range(60):
    errors = [weight * x + bias - y for x, y in zip(xs, targets)]
    loss = sum(error ** 2 for error in errors) / len(xs)
    grad_weight = sum(2 * error * x for error, x in zip(errors, xs)) / len(xs)
    grad_bias = sum(2 * error for error in errors) / len(xs)
    weight -= learning_rate * grad_weight
    bias -= learning_rate * grad_bias
    if epoch % 20 == 0:
        print(f"Epoch {epoch:2d} | loss before update: {loss:.6f}")

print(f"Learned weight: {weight:.3f}, bias: {bias:.3f}")
print(f"Prediction for x=3: {weight * 3 + bias:.3f}")