A machine learns the way you find the bottom of a hill: one small step downhill at a time.

Traditional programming means a human writes explicit rules: "if this, then that." Machine Learning flips that around. Instead of writing the rules ourselves, we show the computer many examples of inputs and correct outputs, and let it discover the rule — a mathematical function — that connects them. Once learned, that function can predict answers for brand-new inputs it has never seen.

A spam filter isn't told "the phrase 'free money' means spam" (spammers adapt in a week). It's shown thousands of labeled emails and learns which patterns correlate with spam. Handwriting readers aren't given a rule for every possible "7" — they see thousands of examples. In every case the common thread is the same: we don't know the rule ourselves, but we have examples of it in action — so we let the data reveal the rule.

This chapter is about supervised learning: we have both inputs and correct answers. (Two siblings exist — unsupervised learning finds structure without answers, and reinforcement learning learns from rewards — but they wait for another chapter.)

Our simplest possible rule assumes price grows in a straight line with size: a bigger house costs proportionally more, on top of some fixed base cost. A straight line has exactly two knobs you can turn: how steep it is, and where it starts. Learning, at its core, is nothing more than tuning these two knobs until the line fits the data well.

Once we have a line — any line, even a terrible guess — we need a way to measure how wrong it is. For each house we compare the model's prediction to the actual price; the difference is that house's error. We then want one single number summarizing how wrong the model is across all houses at once. Smaller is better. That number is called the loss.

Why square each error? Two reasons. First, a raw error can be positive or negative — if we just averaged them, a +10 miss and a −10 miss would cancel to zero, falsely claiming a perfect model. Squaring makes every error positive, so mistakes can never hide each other. Second, squaring punishes big misses much more than small ones — being off by 4 costs 16, being off by 1 costs only 1. Being very wrong should count for more than being slightly wrong.

Plot the loss against every possible pair of knob settings and you get a bowl-shaped surface (you'll see it in Section 9). The best model lives at the bottom of the bowl. So the whole problem becomes: how do you find the bottom of a bowl you can't see all at once?

A derivative answers exactly one question: at the spot where you're standing, how steeply is the ground tilting, and in which direction? Slope pointing up to the right? Then downhill is to the left. Slope of zero? You're standing on flat ground — possibly the very bottom.

We have two knobs, so we ask the slope question twice: if I nudge w a hair, does the loss go up or down, and how fast? And the same for b. The two answers together are called the gradient — a pair of numbers pointing in the direction where loss climbs fastest. Since we want loss to fall, we'll move exactly the opposite way.

We now know which direction makes the loss worse. So we take a small step in the opposite direction, re-measure the slope from our new position, and repeat. Each step nudges w and b a little closer to the values that minimize error. This "step downhill, feel again, step again" ritual is gradient descent — the core learning mechanism behind almost all of modern ML, including deep neural networks.

That's every ingredient. Training a model is simply repeating four steps, over and over: guess, measure the mistake, ask which direction reduces it, adjust. Nothing more mysterious than that.

Let's make everything concrete. Our dataset: three houses, size x = [1, 2, 3], price y = [3, 5, 7]. (Secretly the true relationship is y = 2x + 1 — but the model doesn't know that. It must discover it.) Setup, exactly as in the chapter: start at w = 0, b = 0, learning rate α = 0.1.

Iteration 1 by hand. With both knobs at zero, every prediction is 0, so the errors are [−3, −5, −7] and the loss is (1/3)(9 + 25 + 49) = 27.67. The gradients: ∂L/∂w = (2/3)(−3·1 − 5·2 − 7·3) = −22.67 and ∂L/∂b = (2/3)(−15) = −10. Both slopes are steeply negative — the bowl tilts hard toward larger w and b — so the updates push both knobs up: w becomes 0 − 0.1(−22.67) = 2.267 and b becomes 0 − 0.1(−10) = 1.0. One step, and the loss collapses from 27.67 to about 0.33.

Iteration 2 refines the slight overshoot: the new predictions [3.267, 5.533, 7.8] are all a touch high, the gradients turn small and positive, and the knobs ease back to w ≈ 2.018, b ≈ 0.893. From there the loss keeps shrinking as w → 2 and b → 1 — exactly the true rule y = 2x + 1. Gradient descent, working precisely as intended. Watch it happen:

Now the picture that ties it all together. Plot the loss L for every possible combination of w and b and you get a bowl-shaped surface: high walls where the line fits badly, a single lowest point at w = 2, b = 1 where it fits perfectly. Training never sees this whole bowl — it only ever feels the local slope — yet the path it traces bends steadily inward toward the bottom, loss shrinking with every step. That path below is the actual descent on our house data, starting from w = 0, b = 0.

Here's the payoff. Everything in this chapter — a model with weights and biases, a loss measuring error, gradients from calculus, an update rule stepping downhill — is exactly the same machinery used in logistic regression and in deep neural networks. Only two things ever change in fancier models: the shape of the function f(x), and the specific loss used. The core loop — predict, measure, gradient, update — stays identical. That's why gradient descent is called the workhorse of modern machine learning.