Gradient Descent for Linear Regression: Derive and Run One Update by Hand

See prediction, residual, gradient and parameter update as one continuous calculation. Then verify the lower loss and test what an oversized learning rate does.

KnowledgeGate Team

Exam prep & CS education

Updated 17 Nov 20265 min read
Gradient Descent for Linear Regression: Derive and Run One Update by Hand

You can call an optimiser in one line, yet its returned parameters may feel arbitrary because prediction, error, gradient and update are hidden inside that call. For the points (1,2) and (2,3), starting from w=0.5, b=0.5 with alpha=0.1, the complete calculation shows prediction, error, gradient and update explicitly. One verified update gives (w,b)=(0.7,0.625) and lowers the cost from 0.8125 to 0.3515625.

Gradient descent starts with a model, an error and one exact cost function

For example i, y_hat_i = w x_i + b, where w is the slope, b the intercept, y_hat_i the prediction and y_i the target. Define e_i = y_hat_i - y_i, always prediction minus target. Changing this sign without changing the derivative reverses the update.

For m examples, use the full-batch objective

J(w,b) = (1/(2m)) Σ_i e_i^2 = (1/(2m)) Σ_i (w x_i + b - y_i)^2.

The term e_i^2 belongs to one example; J covers the dataset. Squaring stops opposite residuals from cancelling. The factor 1/2 only cancels the 2 from differentiation and does not change the minimising parameters.

Let theta=(w,b) and grad J=(partial J/partial w, partial J/partial b). Since the gradient points towards steepest local increase, descent moves opposite it. Coding & Skills covers broader programming, AI and machine-learning study.

Derive the linear-regression gradient with the chain rule

For the slope component, retain the inner derivative:

partial J/partial w = (1/(2m)) Σ_i 2e_i (partial e_i/partial w).

Since e_i = w x_i + b - y_i, partial e_i/partial w = x_i. Cancelling the 2 gives

partial J/partial w = (1/m) Σ_i e_i x_i.

For the same residual, an example with a larger x_i contributes proportionally more to the slope gradient. For the intercept, partial e_i/partial b = 1, so

partial J/partial b = (1/m) Σ_i e_i.

Therefore, grad J = ((1/m) Σ_i e_i x_i, (1/m) Σ_i e_i). Update the two parameters simultaneously:

w_new = w_old - alpha(partial J/partial w)

b_new = b_old - alpha(partial J/partial b)

Evaluate both gradients at the same old (w,b). Do not update w before calculating the gradient for b. Optimisation is one mechanism in the wider Approaches to AI map, but not every AI approach trains this model.

Run the forward pass at w=0.5 and b=0.5

Using both points together, the predictions are 0.5(1)+0.5=1.0 and 0.5(2)+0.5=1.5. This forward pass does not change parameters. It records the outputs needed for the cost and gradient, all from the same old values.

i

x_i

y_i

y_hat_i

e_i

e_i^2

e_i x_i

1

1

2

1.0

-1.0

1.0

-1.0

2

2

3

1.5

-1.5

2.25

-3.0

With m=2, the starting cost is

J_0 = (1/(2 × 2))[(-1)^2+(-1.5)^2]

= (1/4)(1+2.25) = 3.25/4 = 0.8125.

The denominator is 2m=4, so 1.625 is not our cost. Both predictions are low, so their residuals are negative. Positive inputs make both e_i x_i terms negative too. Subtracting the resulting negative derivatives will increase both parameters.

Calculate the gradient and complete one simultaneous update

Aggregate the contribution columns from the forward pass:

partial J/partial w = [(-1.0)+(-3.0)]/2 = -4/2 = -2.0

partial J/partial b = [(-1.0)+(-1.5)]/2 = -2.5/2 = -1.25

Thus, grad J(0.5,0.5)=(-2.0,-1.25). Apply alpha=0.1 to both old parameter values:

w_1 = 0.5 - 0.1(-2.0) = 0.7

b_1 = 0.5 - 0.1(-1.25) = 0.625

In vector form, Delta theta = -alpha grad J = (0.2,0.125) and (0.5,0.5)+(0.2,0.125)=(0.7,0.625). Both components came from the gradient at (0.5,0.5), so the update is simultaneous.

The direction makes sense: the starting line underpredicts both targets, so raising its slope and intercept is sensible. Algebra reaches the same conclusion when a model has too many parameters to picture.

Six-stage flow traces two data points through predictions, residuals, gradient contributions, aggregate cost, and the update from (0.5,0.5) to (0.7,0.625).

Verify the update by recomputing the loss

Now run the same two examples through (w_1,b_1)=(0.7,0.625).

This second forward pass is the simplest practical audit. It tests the updated model against the same cost definition instead of assuming that an update helped.

i

Calculation

y_hat_i

e_i

e_i^2

1

0.7(1)+0.625

1.325

-0.675

0.455625

2

0.7(2)+0.625

2.025

-0.975

0.950625

Recompute the objective independently:

J_1 = (1/4)(0.455625+0.950625)

= (1/4)(1.40625) = 0.3515625.

The cost falls from 0.8125 to 0.3515625. The absolute drop is 0.4609375; the relative reduction is (0.4609375/0.8125) × 100, about 56.73%. This describes the toy step, not every dataset or iteration.

As an audit, the new gradient is ([(-0.675)(1)+(-0.975)(2)]/2, [-0.675-0.975]/2) = (-1.3125,-0.825). Its norm, about 1.5503, is below the starting 2.3585. That fits a flatter local slope here, but does not prove convergence generally.

The learning rate decides whether the same gradient helps or overshoots

Reuse the starting gradient (-2.0,-1.25), but deliberately set alpha=1.0. Then w=0.5-1(-2)=2.5 and b=0.5-1(-1.25)=1.75. The predictions become 4.25 and 6.75, so the residuals are 2.25 and 3.75. The resulting cost is

J = (1/4)(2.25^2+3.75^2) = (1/4)(5.0625+14.0625) = 4.78125.

The one-step increase to 4.78125 signals overshoot, and repeated oversized steps can diverge. A very small positive alpha usually gives slow progress. A suitable value can decrease this smooth convex quadratic. An oversized value can cross its valley and increase loss. This does not promise that every step decreases an arbitrary non-convex objective.

For an exact check, design rows (x_i,1) give Hessian (1/m)X^T X = [[2.5,1.5],[1.5,1]]. Its largest eigenvalue is about 3.4271, so fixed-step descent on this quadratic converges for 0<alpha<2/3.4271, about 0.5836. Thus 0.1 is inside and 1.0 outside.

Contour plot from (0.5,0.5) compares the alpha=0.1 descent step to (0.7,0.625) with the alpha=1.0 overshoot to (2.5,1.75).

Gradient descent traps and assessment-style checks

Wrong move

What goes wrong

Correct move

Use y-y_hat but keep the prediction-minus-target derivative

Reversed gradient sign

Keep one residual convention

Forget division by m

Wrong gradient scale

Average both sums over m examples

Mix plain MSE with the defined 1/(2m) cost

Loss and derivative disagree

Keep the stated cost convention

Update w before calculating the gradient for b

The step stops being simultaneous

Calculate both components at old (w,b)

Move with +alpha grad J

Parameters move uphill

Use theta_new=theta_old-alpha grad J

Try three short checks:

  1. If grad J=(3,-4), theta=(2,1) and alpha=0.05, then theta_new=(2,1)-0.05(3,-4)=(1.85,1.20).

  2. Starting from the initial parameters, alpha=0.05 gives (w,b)=(0.6,0.5625).

  3. A zero gradient can mark a minimum, maximum, saddle point or flat region in a general objective. This full-rank linear-regression quadratic has the unique minimum (1,1).

For adjacent learned-model practice, use Artificial Neural Networks MCQs: 10 Solved Questions with Explanations.

Gradient descent short version and the next step

Keep this six-line chain:

  1. Predict with y_hat=wx+b.

  2. Form e=y_hat-y.

  3. Calculate J=(1/(2m))Σe^2.

  4. Derive grad J=((1/m)Σex,(1/m)Σe).

  5. Update both parameters with theta_new=theta_old-alpha grad J.

  6. Recompute the loss before calling the step useful.

Here (0.5,0.5) becomes (0.7,0.625), and J falls to 0.3515625. Cover the table and reproduce both gradient components from the raw points. Then use alpha=0.05 and confirm (0.6,0.5625) before checking the solution. For a broader next learning path, explore Artificial Intelligence (AI) as you move from one hand-computed optimiser step towards wider AI and machine-learning study.