Gradient Boosting Regression: Build Two Residual-Correcting Trees Step by Step

See every stage of gradient boosting regression on five data points, from the initial mean through two residual-fitted stumps and their additive predictions.

KnowledgeGate Team

Exam prep & CS education

Updated 16 Nov 20266 min read
Gradient Boosting Regression: Build Two Residual-Correcting Trees Step by Step

“Each tree fixes the previous tree's mistakes” sounds intuitive, but it hides what becomes the next tree's training label, where the learning rate enters, and how the outputs combine at prediction time. The worked trace uses x = (1,2,3,4,5) and y = (1,2,3,8,12), begins at F_0 = 5.2, and fits two depth-1 trees with eta = 0.5. Training MSE falls from 17.36 to 5.84, then to 2.21.

Gradient boosting is an additive model built in sequence

At stage m, the regression model is

F_m(x) = F_(m-1)(x) + eta T_m(x).

Here, F_(m-1) is the ensemble so far, T_m is the new shallow tree, and eta > 0 scales its correction. Values in (0,1] are commonly used for shrinkage; this trace uses eta = 0.5. A prediction adds the baseline and every shrunken correction, not the latest tree output alone. The Skill Development Courses category provides the broader setting for AI and machine-learning study.

For L_i = (1/2)(y_i - F(x_i))^2, the negative gradient is -partial L_i / partial F(x_i) = y_i - F(x_i) = r_i. This is why squared-error boosting fits residuals. Other differentiable losses use pseudo-residuals, and their optimal leaf updates need not be raw means. Tree m must wait for F_(m-1) to produce its targets, unlike a parallel ensemble. The ML & Data Science hub places this idea in its wider subject area.

Start gradient boosting regression with the best constant prediction

Under squared error, the best constant is the sample mean:

F_0 = (1 + 2 + 3 + 8 + 12) / 5 = 26 / 5 = 5.2.

x_i

y_i

F_0(x_i)

r_i^(0) = y_i - F_0(x_i)

(r_i^(0))^2

1

1

5.2

-4.2

17.64

2

2

5.2

-3.2

10.24

3

3

5.2

-2.2

4.84

4

8

5.2

2.8

7.84

5

12

5.2

6.8

46.24

Negative residuals show that the baseline overpredicts the first three targets. Positive residuals show that it underpredicts the last two. The starting MSE is

(17.64 + 10.24 + 4.84 + 7.84 + 46.24) / 5 = 86.80 / 5 = 17.36.

MSE remains the comparison metric even though the gradient derivation uses half squared error. Multiplying the objective by a positive constant does not change its best splits or fitted values.

Fit the first tree to the baseline residuals

Tree 1 is fitted to (x_i, r_i^(0)), not to the original y_i. Comparing every possible stump split by residual sum of squares gives:

Split after x

1

2

3

4

SSE

64.75

41.1667

10

29

The unique minimum is 10, so the split is x <= 3 versus x > 3. Each leaf returns its mean residual:

left = (-4.2 - 3.2 - 2.2) / 3 = -9.6 / 3 = -3.2

right = (2.8 + 6.8) / 2 = 9.6 / 2 = 4.8

Therefore, T_1(x) = -3.2 on the left and T_1(x) = 4.8 on the right. Applying shrinkage,

F_1 = 5.2 + 0.5T_1.

The first three predictions become 5.2 - 1.6 = 3.6; the last two become 5.2 + 2.4 = 7.6. The leaf values -3.2 and 4.8 are corrections, not final target predictions.

Flow diagram showing residual calculation, a depth-1 split, and updated predictions.

Recompute residuals and fit the second correcting tree

Tree 2 needs fresh targets calculated from F_1:

x

y

F_1

r^(1) = y - F_1

1

1

3.6

-2.6

2

2

3.6

-1.6

3

3

3.6

-0.6

4

8

7.6

0.4

5

12

7.6

4.4

Stage-one MSE is (6.76 + 2.56 + 0.36 + 0.16 + 19.36) / 5 = 29.20 / 5 = 5.84. The split SSE values for r^(1) are 20.75, 14.5, 10, and 5 after x = 1, 2, 3, 4, respectively. Thus tree 2 splits at x <= 4.

Its left output is (-2.6 - 1.6 - 0.6 + 0.4) / 4 = -4.4 / 4 = -1.1; its right output is 4.4. So T_2(x) = -1.1 for x <= 4 and 4.4 otherwise. Adding this correction to F_1, not F_0, gives

F_2 = F_1 + 0.5T_2 = (3.05, 3.05, 3.05, 7.05, 9.80).

The new residuals are (-2.05, -1.05, -0.05, 0.95, 2.20). Their squares sum to 4.2025 + 1.1025 + 0.0025 + 0.9025 + 4.84 = 11.05, so MSE_2 = 11.05 / 5 = 2.21.

Model

Squared-error sum

MSE

F_0

86.80

17.36

F_1

29.20

5.84

F_2

11.05

2.21

For a new x = 4.5, both trees choose their right leaves: 5.2 + 0.5(4.8) + 0.5(4.4) = 9.8.

Ledger showing two boosting stages, residuals, MSE changes, and a prediction for x=4.5.

Learning rate, tree depth and tree count work together

The first tree makes shrinkage visible. At x = 5, F_0 = 5.2 and T_1 = 4.8, so one-tree predictions are 5.68 with eta = 0.1, 7.6 with eta = 0.5, and 10.0 with eta = 1. A smaller learning rate makes each step more cautious and usually needs more trees, but does not guarantee better generalisation.

The controls do different jobs. eta scales each update, max_depth limits the interaction complexity of one tree, and M sets the number of sequential corrections. Depth 1 and M = 2 expose the arithmetic here. In a real model, use held-out validation and early stopping. Continuing to add flexible trees can chase noise or isolated residuals, so retain the iteration with the best validation loss instead of selecting from training loss alone.

Gradient boosting is not bagging, and residual fitting has traps

Bagging and boosting both use trees, but train differently. Approaches to AI: Four Approaches and a Worked Example places supervised learning within the wider AI map.

Feature

Bagging or random forest

Gradient boosting

Tree training

Independent

Sequential

Variation or target

Bootstrap samples and feature randomness

Current pseudo-residuals

Combination

Equal or averaged votes

Baseline plus shrunken corrections

Parallelism

Trees can train in parallel

Each stage depends on earlier stages

Wrong move

What goes wrong

Correct move

Train every tree on y

No residual correction

Train on current negative gradients

Compute tree 2 from F_0

Ignores tree 1

Compute from F_1

Forget eta

Updates and predictions are too large

Scale every tree output

Treat a leaf as the prediction

Drops the ensemble history

Add it as a correction

Use residual means for every loss

Applies a squared-error rule universally

Derive the loss-specific leaf update

With our convention, a positive residual means the ensemble is too low and needs a positive correction. A source using error F - y must change the gradient sign and update consistently.

How assessments test residual-correcting trees

Common question formats ask you to calculate the initial constant, derive residual targets after one stage, reconstruct a prediction from tree outputs and eta, or distinguish sequential boosting from parallel bagging. These retrieval checks test the complete chain:

  1. At x = 4, tree 2's training target is 8 - 7.6 = 0.4.

  2. If F_1(x) = 6.0, T_2(x) = -1.4, and eta = 0.25, then F_2(x) = 6.0 + 0.25(-1.4) = 5.65.

  3. At x = 2 here, F_2 = 5.2 + 0.5(-3.2) + 0.5(-1.1) = 3.05.

Conceptually, if the newest tree is trained independently on the original target, it is not performing this residual-correction step. For adjacent learned-model practice, work through Artificial Neural Networks MCQs: 10 Solved Questions without assuming that they cover this gradient-boosting calculation.

Gradient boosting short version and the next step

Keep this six-step chain ready:

  1. Choose the loss.

  2. Initialise F_0.

  3. Calculate negative gradients or residuals.

  4. Fit a shallow tree to them.

  5. Update F_m = F_(m-1) + eta T_m.

  6. Stop according to validation.

Here, F_0 = 5.2, tree 1 splits at x <= 3, tree 2 at x <= 4, and toy training MSE moves 17.36 -> 5.84 -> 2.21. Cover the ledger and reconstruct x = 5 as 5.2 + 2.4 + 2.2 = 9.8. If you cannot explain the 4.8 and 4.4 corrections, revisit their residual means. Then use AI & ML for Placements | Generative AI Placement Course as a broader next path into AI and machine-learning study.