This guide develops the statistical methods assessed by the Society of Actuaries Statistics for Risk Modeling Examination (SOA Exam SRM). Foundations come first, followed by linear models, model selection, time series, decision trees, and unsupervised learning. Each concept includes a worked example and a specific error to avoid. The explanations assume familiarity with probability, calculus, and mathematical statistics.
Foundations of Statistical Learning
1. Recognizing the learning problem
Supervised learning uses an observed response to learn a predictive relationship. Regression predicts a quantitative response; classification predicts a category or its probability. Unsupervised learning examines structure without a designated response. Classify the task by what must be predicted, rather than by whether the predictors contain numbers.
Worked example: Predicting next year's claim amount is regression. Predicting whether a claim occurs is classification. Grouping policyholders by similar characteristics without claim outcomes is unsupervised learning.
Mistake to avoid: Treating a response coded as zero or one as an ordinary continuous measurement without considering its categorical meaning.
2. Matching accuracy measures to the objective
Mean squared error averages squared numerical prediction errors and penalizes large errors strongly. Classification error measures the proportion of incorrect class assignments. Probability predictions can also be assessed with log loss, which penalizes confident incorrect predictions. The appropriate measure depends on the response and the intended use of predictions.
Worked example: Actual amounts of 10 and 20 with predictions of 12 and 17 give MSE = (4 + 9)/2 = 6.5.
Mistake to avoid: Comparing a regression MSE directly with a classification error rate as though they had the same scale.
3. Understanding the bias–variance tradeoff
For squared-error prediction at a fixed predictor value, expected error separates into squared bias, variance, and irreducible noise. More flexible methods often reduce bias while increasing sensitivity to the training sample. The best predictive choice balances these components; an excellent training fit does not establish low expected test error.
Worked example: Model A has squared bias 9, variance 2, and noise variance 4, giving expected error 15. Model B has components 1, 12, and 4, giving 17. A predicts better.
Mistake to avoid: Choosing the model with the smallest bias while ignoring its sampling variability.
4. Keeping a test set independent
A test set estimates performance on observations that did not influence model fitting or selection. Preprocessing choices, feature selection, and tuning belong inside the training procedure. When observations are dependent, such as repeated records for one policyholder, splitting individual rows may fail to create a genuinely independent assessment.
Worked example: Estimate a predictor's mean and standard deviation from training records, then apply those same values to standardize test records.
Mistake to avoid: Selecting variables using the entire dataset before reporting supposedly independent test performance.
5. Using k-fold cross-validation
Divide observations into k folds. Refit the complete modeling procedure on k − 1 folds and evaluate it on the omitted fold, repeating until every fold has been omitted. Aggregate the held-out losses to compare candidate procedures. Any estimated preprocessing must be repeated within each training fold.
Worked example: Five equally sized folds produce MSE values 8, 10, 7, 9, and 6. Their cross-validation MSE is 40/5 = 8.
Mistake to avoid: Fitting once on all observations and then calling errors within separate subsets cross-validation.
6. Interpreting leave-one-out validation
Leave-one-out cross-validation fits n models, each omitting one observation, and averages the n held-out losses. Each fit uses nearly the full dataset, often reducing validation bias relative to a much smaller training split. However, the fitted models overlap heavily, so their errors are dependent and the resulting estimate need not have low variance.
Worked example: Four held-out squared errors of 1, 4, 9, and 2 give leave-one-out MSE = 16/4 = 4.
Mistake to avoid: Assuming that using more folds always produces a more stable estimate of prediction error.
Ordinary Linear Regression
7. Separating mean assumptions from inference assumptions
Ordinary regression specifies a conditional mean that is linear in its coefficients. Zero conditional error mean supports unbiased coefficient estimation under the usual design conditions. Constant error variance and uncorrelated errors support conventional standard errors; normal errors provide exact classical small-sample inference. These assumptions have different roles and should be checked separately.
Worked example: A straight conditional mean can coexist with residual spread that increases with fitted claim amount. The mean specification may be reasonable while conventional interval calculations need reconsideration.
Mistake to avoid: Concluding that a linear mean automatically implies constant variance or normal errors.
8. Understanding least-squares residual constraints
Ordinary least squares chooses coefficients to minimize the residual sum of squares. At the solution, residuals are orthogonal to each included predictor column. With an intercept, residuals also sum to zero. These are properties of the fitted sample and do not prove that the population errors satisfy the model assumptions.
Worked example: For a design containing an intercept and x = 0, 1, 2, residuals 1, −2, 1 have sum zero and weighted sum 0 − 2 + 2 = 0.
Mistake to avoid: Using the fitted residual mean of zero as evidence that the model has no omitted-variable bias.
9. Calculating a simple regression line
In simple regression, the slope is the centered cross-product sum divided by the centered predictor sum of squares. The intercept is the response mean minus the slope times the predictor mean. The fitted line passes through the two sample means, but its predictions remain conditional on the model and the observed predictor range.
Worked example: For x = 1, 2, 3 and y = 2, 3, 7, the means are 2 and 4. The slope is 5/2 = 2.5, the intercept is −1, and prediction at x = 2.4 is 5.
Mistake to avoid: Calculating the slope from uncentered products without accounting for an intercept.
10. Interpreting partial regression effects
A multiple-regression coefficient describes the change in the fitted conditional mean for a one-unit predictor increase while the other included predictors stay fixed. This differs from the association in a separate simple regression. An observational coefficient alone does not establish a causal effect, especially when relevant variables are omitted.
Worked example: In fitted cost = 200 + 30 × age + 80 × exposure, increasing age from 4 to 6 with exposure unchanged increases predicted cost by 60.
Mistake to avoid: Interpreting the age coefficient while allowing exposure to change at the same time.
11. Reading categorical predictor contrasts
With an intercept, a categorical predictor with L levels commonly uses L − 1 indicator variables. Each coefficient compares its level with the omitted reference level, conditional on the other predictors. Changing the reference level changes coefficient descriptions but preserves fitted values when the same model is represented correctly.
Worked example: Reference group A has fitted mean 100. Indicators for B and C have coefficients 20 and −10, giving fitted means 120 and 90.
Mistake to avoid: Including an intercept and all L level indicators, which creates exact linear dependence.
12. Interpreting interaction terms
An interaction makes one predictor's effect depend on another predictor. In a model containing x, z, and xz, the effect of x is βx + βxz z. Main-effect coefficients therefore describe effects at a particular value of the interacting variable, often zero or the reference category.
Worked example: For fitted y = 10 + 2x + 3z + 4xz with binary z, the x slope is 2 when z = 0 and 6 when z = 1.
Mistake to avoid: Reporting the coefficient 2 as the x effect for both groups.
13. Modeling curvature with transformed predictors
A regression can be linear in coefficients while nonlinear in its predictors. Adding x² allows curvature, and the marginal effect becomes β1 + 2β2x. Polynomial terms can fit smoothly within the observed range but behave poorly when extrapolated. Interpret the combined curve rather than treating its coefficients as separate constant effects.
Worked example: For fitted y = 5 + 3x − 0.5x², the slope at x = 2 is 3 − 2 = 1. At x = 4, it is −1.
Mistake to avoid: Calling the model incompatible with linear regression merely because it contains x².
14. Interpreting a log-transformed response
When log Y is modeled, a predictor coefficient describes a multiplicative change on the original scale. Exponentiating the fitted log mean does not generally recover the arithmetic mean of Y. Under normal log errors with variance σ², the conditional mean is exp(η + σ²/2), whereas exp(η) is the conditional median.
Worked example: With η = ln(100) and σ² = 0.2, the fitted median is 100 and the fitted mean is 100 exp(0.1), approximately 110.52.
Mistake to avoid: Reporting exp(η) as the mean without considering retransformation bias.
15. Connecting explained variation and residual error
For an intercept model, R² = 1 − RSS/TSS describes the proportion of sample response variation explained. Residual variance is estimated by RSS/(n − p − 1), where p excludes the intercept. R² cannot decrease when predictors are added, so it does not by itself establish improved out-of-sample accuracy.
Worked example: With RSS = 24, TSS = 60, n = 10, and p = 3, R² is 0.6 and estimated error variance is 24/6 = 4.
Mistake to avoid: Dividing RSS by n when calculating the conventional unbiased residual variance estimate.
16. Testing an individual coefficient
For a proposed coefficient value β0, the classical t statistic is (estimated coefficient − β0)/standard error. Its reference degrees of freedom are the full model's residual degrees of freedom. A coefficient table reports estimates, uncertainty, and tests; significance does not measure business importance or establish causality.
Worked example: An estimate of 6 with standard error 2 gives t = 3 against zero. With a stated two-sided critical value of 2.1, reject the zero-coefficient hypothesis.
Mistake to avoid: Comparing the raw coefficient with the critical value instead of comparing the t statistic.
17. Testing a group of predictors
A partial F test compares nested ordinary linear models fitted to the same observations. For q restrictions, F = [(RSSreduced − RSSfull)/q]/[RSSfull/dffull]. It asks whether the omitted group improves fit collectively. Its conventional reference distribution requires the classical error assumptions, including normality for exact finite-sample inference.
Worked example: Reduced RSS 100, full RSS 80, q = 2, and full residual degrees of freedom 20 give F = 10/4 = 2.5.
Mistake to avoid: Applying this nested-model comparison to unrelated models or datasets with different observations.
18. Distinguishing confidence and prediction intervals
A confidence interval for the conditional mean reflects uncertainty in the estimated regression surface. A prediction interval for one future observation also includes its unexplained variation and is wider under the same conditions. Both depend on the predictor location; observations far from the training center often have greater estimation uncertainty.
Worked example: A fitted mean of 100, mean standard error 2, and residual standard deviation 5 give intervals 100 ± 4 and 100 ± 2√29, using multiplier 2.
Mistake to avoid: Using the mean confidence interval as the uncertainty range for an individual future claim.
19. Diagnosing regression with residual plots
Residual-versus-fitted plots can reveal curved mean patterns or changing variance. Normal quantile plots examine distributional departures, especially in the tails. Residuals plotted in observation order can reveal dependence. A diagnostic pattern suggests which assumption deserves investigation; no single plot proves that all assumptions hold.
Worked example: A U-shaped residual pattern suggests missing curvature. A funnel-shaped pattern suggests variance changing with the fitted mean. These point to different modeling problems.
Mistake to avoid: Treating every unusual residual pattern as evidence that the response needs the same transformation.
20. Recognizing multicollinearity
Highly related predictors make individual coefficient estimates unstable and inflate their standard errors. For predictor j, VIF = 1/(1 − Rj²), where Rj² comes from regressing that predictor on the others. Multicollinearity need not destroy predictions within the observed predictor combinations, but it complicates separate-effect interpretation.
Worked example: If regressing one predictor on the others gives Rj² = 0.9, its variance inflation factor is 10.
Mistake to avoid: Concluding that every predictor is irrelevant because a jointly useful model has individually imprecise coefficients.
21. Separating outliers, leverage, and influence
A response outlier has an unusual residual. A high-leverage observation has an unusual predictor configuration. Influence concerns how much the fitted model changes when an observation is removed. Strong influence often combines leverage with a substantial residual, but these three descriptions are not interchangeable.
Worked example: A policy with exposure far beyond all other policies may have high leverage. If it lies close to the fitted relationship, removing it may change the coefficients only slightly.
Mistake to avoid: Automatically deleting every high-leverage record without checking data validity and its actual influence.
Generalized Linear Models
22. Choosing a response family and variance relationship
A generalized linear model combines an exponential-family response distribution, a linear predictor, and a link. Distribution choice determines response support and the mean–variance relationship. Normal variance is constant; Poisson variance equals its mean; binomial count variance is np(1 − p); Gamma variance is proportional to the squared mean.
Worked example: A claim count can reasonably motivate a Poisson starting model. A strictly positive, right-skewed claim amount can motivate a Gamma starting model.
Mistake to avoid: Choosing the family only because its link is convenient, while ignoring impossible response values.
23. Applying links and inverse links
The link maps the conditional mean onto the linear predictor: g(μ) = η. The inverse link converts η back to the response scale. Common canonical links include identity for normal means, log for Poisson means, and logit for binomial probabilities. Other suitable links can be used when the model specification permits them.
Worked example: Under a log link, η = ln(3) gives μ = 3. Even η = −2 gives a positive mean, exp(−2), approximately 0.135.
Mistake to avoid: Treating the linear predictor itself as the predicted response mean under every link.
24. Converting logistic coefficients into probabilities
Logistic regression models log odds. Exponentiating a coefficient gives the odds multiplier for a one-unit predictor increase, holding other predictors fixed. Probability is odds/(1 + odds), so a constant odds multiplier does not imply a constant probability difference. Baseline probability matters when explaining the practical effect.
Worked example: Baseline odds of 0.4 imply probability 0.4/1.4, approximately 0.286. A coefficient ln(1.5) raises the odds to 0.6 and probability to 0.375.
Mistake to avoid: Describing an odds ratio of 1.5 as a 50% increase in probability.
25. Accounting for exposure in Poisson regression
For counts observed over differing exposures, a log-link model can include log exposure as an offset with coefficient fixed at one. The remaining predictor models the event rate. This assumes expected counts scale proportionally with exposure, conditional on the included risk characteristics; exposure definitions must therefore be comparable.
Worked example: A fitted rate of 0.03 over exposure 200 gives expected count 6. A predictor multiplying the rate by 1.2 changes the expected count to 7.2.
Mistake to avoid: Comparing raw counts across unequal exposure periods without accounting for their differing opportunities for events.
26. Interpreting Gamma models for positive amounts
A Gamma GLM accommodates strictly positive responses with variance φμ². Thus, for fixed dispersion, standard deviation grows proportionally with the mean. A log link gives multiplicative mean effects, although it is not the Gamma canonical link. The family and link jointly determine interpretation and should not be confused.
Worked example: A baseline mean of 500 and log-link coefficient ln(1.1) give a new mean of 550. If φ = 0.25, the baseline standard deviation is 250.
Mistake to avoid: Using an ordinary Gamma response model for amounts that include zero without addressing the support mismatch.
27. Using deviance and likelihood-ratio comparisons
Deviance compares a fitted model's likelihood with a saturated model. For nested Poisson models on identical observations, the reduction in deviance equals twice the log-likelihood improvement. Under suitable regularity and large-sample conditions, compare that reduction with a chi-square distribution whose degrees of freedom equal the additional parameters.
Worked example: Reduced deviance 123 and full deviance 115 give a statistic of 8. With two additional parameters and a stated critical value 5.99, reject the restrictions.
Mistake to avoid: Using a deviance difference as a nested-model test when the response family or observations differ.
Model Selection and Flexible Regression
28. Applying the SRM forms of AIC and BIC
The September 2026 syllabus restricts AIC and BIC to ordinary linear models, using AIC = (RSS + 2dσ̂²)/n and BIC = (RSS + ln(n)dσ̂²)/n. Here d counts predictors and σ̂² is a common error-variance estimate. Smaller values are preferred; preserve the specified convention when comparing models.
Worked example: For n = 100 and σ̂² = 2, models (RSS, d) = (190, 3) and (180, 5) have AIC 2.02 and 2.00, but BIC approximately 2.176 and 2.261.
Mistake to avoid: Substituting a different likelihood-based formula into an SRM calculation that specifies these forms.
29. Comparing subset and stepwise searches
Best-subset selection examines every predictor subset, while forward selection adds predictors sequentially and backward selection removes them. Stepwise paths are computationally cheaper but can miss the best combination of a given size. Search identifies candidates; a penalty or validation criterion must still decide the preferred complexity.
Worked example: Four predictors create 2⁴ = 16 subsets, including the intercept-only model. Forward selection first compares four one-predictor additions, then three additions to its chosen model.
Mistake to avoid: Assuming the forward-selected pair must equal the best pair found by exhaustive search.
30. Understanding ridge shrinkage
Ridge regression adds λ times the sum of squared slope coefficients to RSS, usually leaving the intercept unpenalized. Increasing λ shrinks slopes toward zero and can stabilize correlated predictors. Predictor scaling matters because the penalty acts on coefficients. Ridge generally retains all predictors rather than setting slopes exactly to zero.
Worked example: Minimizing (3 − b)² + 2b² gives derivative −6 + 6b = 0, so b = 1 instead of the unpenalized value 3.
Mistake to avoid: Expecting ridge to perform automatic variable deletion in the same manner as lasso.
31. Understanding lasso sparsity
Lasso adds a penalty proportional to the sum of absolute slope coefficients. Its penalty can produce exact zero coefficients, combining shrinkage with variable selection. Standardization makes predictor penalties more comparable. With highly correlated predictors, the selected variables can be unstable, so a zero coefficient does not establish that the predictor has no useful association.
Worked example: The objective (3 − b)² + 8|b| is minimized at b = 0: the absolute-value penalty outweighs the benefit of moving away from zero.
Mistake to avoid: Interpreting a lasso-selected subset as a definitive list of causal risk factors.
32. Making predictions with nearest neighbors
K-nearest neighbors predicts from observations closest to the new predictor vector. Regression commonly averages their responses; classification commonly uses their class frequencies or majority vote. Small k creates local flexibility, while larger k smooths predictions. Distance depends on scaling and becomes less informative when many irrelevant predictors are included.
Worked example: The three nearest records have responses 4, 7, and 10, giving a regression prediction of 7. Labels A, A, and B would give majority class A.
Mistake to avoid: Letting a predictor with large numerical units dominate distance merely because it was not scaled.
33. Distinguishing principal component regression from shrinkage
Principal component regression fits the response using selected principal-component scores rather than the original predictors. It reduces dimension, whereas ridge retains all predictor directions with shrinkage. Components are constructed from predictor variation without using the response, so high explained predictor variance does not guarantee high predictive value.
Worked example: If independent x1 has variance 100 but no response relationship, while x2 has variance 1 and y = 10x2, retaining only the first component can discard the predictive direction.
Mistake to avoid: Choosing the number of regression components solely from predictor variance explained instead of predictive validation.
Time Series and Forecasting
34. Separating trend, seasonality, and irregular variation
An additive decomposition represents a series as trend plus seasonal contribution plus irregular variation. Multiplicative structure instead describes seasonal changes proportional to the series level and may motivate a logarithmic transformation for positive data. Identifying these components helps distinguish systematic movement from short-term fluctuations before choosing a forecasting model.
Worked example: An observation of 120 with trend 100 and seasonal contribution 15 has an additive irregular component of 5.
Mistake to avoid: Calling a repeating seasonal pattern random noise simply because successive observations fluctuate.
35. Checking weak stationarity
A weakly stationary process has constant mean, finite constant variance, and covariance depending on the lag rather than calendar time. It need not have independent observations. A deterministic trend violates constant mean, while changing volatility violates constant variance. Stationarity concerns the process, rather than whether a short observed plot looks approximately flat.
Worked example: Xt = 5 + εt with independent mean-zero innovations of variance 4 has mean 5, variance 4, and zero covariance at nonzero lags.
Mistake to avoid: Treating every autocorrelated process as nonstationary.
36. Interpreting autocorrelation
For a stationary process, lag-h autocorrelation is covariance between observations h periods apart divided by the process variance. Positive autocorrelation indicates persistence; negative autocorrelation indicates an alternating tendency at that lag. Sample autocorrelations estimate these relationships and contain sampling noise, so an isolated nonzero estimate does not establish a particular model.
Worked example: A process variance of 3 and lag-one covariance of 1.5 give lag-one autocorrelation 0.5.
Mistake to avoid: Reading autocorrelation as the causal effect of the previous observation on the current one.
37. Recognizing a random walk and differencing
A random walk satisfies Xt = Xt−1 + c + εt. With a fixed starting value and independent innovations, its level variance grows over time, so the levels are not stationary. First differences equal c + εt and can be stationary. Differencing therefore changes both the modeled quantity and the interpretation of forecasts.
Worked example: Starting at 100 with drift 2 and innovation variance 9, the three-step expected level is 106 and forecast-error variance is 27.
Mistake to avoid: Applying stationary level-model reasoning to a random walk without addressing its accumulated uncertainty.
38. Calculating autoregressive forecasts
A stationary AR(1) model can be written Xt − μ = φ(Xt−1 − μ) + εt, with |φ| < 1 and mean-zero innovations. Its h-step conditional mean is μ + φʰ(Xt − μ). Forecasts move toward the long-run mean, potentially alternating around it when φ is negative.
Worked example: With μ = 3, φ = 0.6, and current value 8, the next forecast is 6 and the two-step forecast is 4.8.
Mistake to avoid: Using the equation's intercept as the long-run mean without accounting for the autoregressive coefficient.
39. Calculating autoregressive forecast uncertainty
For an AR(1) with known parameters, h-step forecast-error variance is σε² times the sum of φ²ʲ for j from zero through h − 1. Future innovations accumulate through the autoregressive recursion. With normal innovations, a normal multiplier gives a prediction interval; parameter estimation adds uncertainty beyond this calculation.
Worked example: With innovation variance 4 and φ = 0.5, two-step variance is 4(1 + 0.25) = 5. A forecast of 10 gives an approximate 95% interval 10 ± 1.96√5.
Mistake to avoid: Using the one-step innovation variance unchanged for every forecast horizon.
40. Updating simple exponential smoothing
Simple exponential smoothing updates the estimated level as Lt = αXt + (1 − α)Lt−1, where 0 < α < 1. Larger α responds faster to recent observations; smaller α smooths more strongly. Without trend or seasonal components, forecasts for all future horizons equal the latest estimated level.
Worked example: With α = 0.3, previous level 100, and new observation 110, the updated level is 33 + 70 = 103.
Mistake to avoid: Using a level-only smoothing model to represent persistent trend without adding an appropriate trend component.
41. Distinguishing conditional volatility in ARCH models
An ARCH(1) model specifies conditional error variance as ht = α0 + α1εt−1². A positive intercept and nonnegative slope keep variance positive; α1 < 1 supports a finite unconditional variance under the usual innovation assumptions. Large previous shocks increase predicted volatility even when the conditional mean remains unchanged.
Worked example: With α0 = 0.5, α1 = 0.4, and previous error 3, the next conditional variance is 0.5 + 0.4 × 9 = 4.1.
Mistake to avoid: Interpreting an ARCH volatility forecast as a forecast that the response mean must increase.
Trees and Ensemble Methods
42. Choosing a regression-tree split
A regression tree commonly chooses each split by minimizing the combined within-child RSS. This is a greedy optimization at the current node, rather than a guarantee of the globally optimal tree. Candidate cutpoints partition predictor space into regions whose fitted values are constant until another split is introduced.
Worked example: Responses 2, 4, 8, and 10 have unsplit RSS 40. A split separating 2, 4 from 8, 10 gives child means 3 and 9 and total RSS 4.
Mistake to avoid: Assuming the locally best first split must belong to the best possible complete tree.
43. Predicting from a regression leaf
Under squared-error fitting, a regression leaf predicts the average training response among observations in that region. Trace every split condition to determine the correct leaf before using its value. Predictions are piecewise constant, and ordinary regression trees do not extend a fitted slope beyond the training range.
Worked example: A new record enters a leaf containing responses 50, 70, and 90. Its predicted response is 70, regardless of its exact position within that leaf.
Mistake to avoid: Interpolating between leaf means as though the tree fitted a continuous regression line.
44. Measuring classification-node impurity
Gini impurity is 1 − Σpk², where pk is the node's class proportion. Entropy is −Σpk ln(pk), treating zero-probability terms as zero. Both favor purer nodes. Split quality compares parent impurity with child impurities weighted by their observation counts, rather than simply averaging the child scores.
Worked example: A balanced two-class parent has Gini 0.5. Two equally sized children with proportions 0.75 and 0.25 each have Gini 0.375, giving reduction 0.125.
Mistake to avoid: Ignoring child sizes when comparing impurity reductions.
45. Separating leaf probabilities from class decisions
A classification leaf estimates class probabilities from its training class proportions. A class decision then applies a threshold or decision rule. Majority voting corresponds to a particular equal-cost binary decision setting; different error costs can justify another threshold. Probability estimation and the final action are distinct steps.
Worked example: A leaf contains seven event records and three nonevent records, giving estimated event probability 0.7. Threshold 0.5 predicts an event; threshold 0.8 predicts a nonevent.
Mistake to avoid: Reporting the selected class as though it were the leaf's numerical probability.
46. Selecting a pruned subtree
Cost-complexity pruning trades training fit against tree size. For a regression subtree, an objective such as RSS + α times the number of leaves penalizes complexity. Larger α favors smaller trees. Validation chooses among the resulting subtrees because the lowest unpenalized training RSS typically favors excessive growth.
Worked example: At α = 5, a three-leaf tree with RSS 80 scores 95. A six-leaf tree with RSS 70 scores 100, so the smaller tree is preferred.
Mistake to avoid: Choosing the largest tree solely because it has the lowest training RSS.
47. Reducing variance through bagging
Bagging fits models to bootstrap samples drawn with replacement from the training observations, then aggregates their predictions. Regression usually averages predictions, while classification can aggregate votes. It is especially useful for unstable learners such as deep trees. Its variance reduction depends on how strongly the fitted learners' errors are correlated.
Worked example: Three bagged regression trees predict 100, 110, and 90 for one record. Their averaged prediction is 100.
Mistake to avoid: Treating bootstrap samples as disjoint subsets or expecting averaging to remove systematic bias automatically.
48. Assessing out-of-bag predictions
Each bootstrap sample omits some training observations. An observation's out-of-bag prediction aggregates only trees whose bootstrap samples omitted it. Comparing these predictions with observed responses provides an internal estimate of predictive error. After extensive tuning against that estimate, a separate assessment may still be needed.
Worked example: Three trees that omitted a record predict 80, 90, and 100. Its out-of-bag prediction is 90; if the response is 95, squared error is 25.
Mistake to avoid: Including trees that trained on the record in its out-of-bag prediction.
49. Decorrelation in random forests
Random forests combine bootstrap sampling with random predictor subsets at each split. Restricting candidate predictors prevents a dominant predictor from appearing in nearly every tree's upper splits, which can reduce correlation among trees. The number of candidate predictors balances individual-tree strength against ensemble diversity and should be evaluated empirically.
Worked example: With nine predictors and three candidates per split, a node searches only its randomly selected three predictors, rather than all nine.
Mistake to avoid: Assuming the same three predictors are selected once and used for every split in the forest.
50. Updating predictions through boosting
Boosting builds learners sequentially, using the current model's deficiencies to guide the next fit. For squared-error regression, a new tree can fit residuals and enter the prediction with a learning-rate multiplier. Other losses use corresponding gradient-based targets, so raw residual fitting is not a universal description of every boosting model.
Worked example: Current prediction 100, new tree contribution 8, and learning rate 0.1 produce updated prediction 100.8.
Mistake to avoid: Describing boosting as independent trees fitted in parallel and simply averaged.
51. Balancing boosting hyperparameters
Boosting complexity depends jointly on learning rate, number of iterations, and tree depth. Smaller learning rates often require more iterations; deeper trees permit richer interactions. Additional iterations can eventually worsen validation performance. Tune these settings together rather than assuming that learning rate multiplied by iteration count completely determines the fitted model.
Worked example: With other settings fixed, validation MSE is 30 at 50 trees, 24 at 150, and 29 at 300. The best evaluated iteration count is 150.
Mistake to avoid: Choosing the greatest number of boosting iterations because training loss continues to decrease.
52. Recognizing tree instability
A single tree can change substantially when a small data perturbation changes an early split. Because all later regions depend on that split, instability can propagate through the tree. This explains why a tree can be easy to interpret yet have high prediction variance, and why ensembles may improve its predictive stability.
Worked example: Two fitted trees split at 4.9 and 5.1, with left and right predictions 10 and 20. A record with x = 5 receives different predictions.
Mistake to avoid: Assuming an interpretable split structure must also be stable across training samples.
53. Comparing trees with linear models
Trees naturally represent threshold effects and interactions through successive splits. Ordinary additive linear models represent smooth additive effects unless transformations or interactions are introduced. Linear models can extrapolate their fitted structure; trees usually remain within their leaf predictions. Choose based on the relationship, predictive evidence, and interpretability needs.
Worked example: A synthetic response rises from 100 to 150 only when both age is below 25 and mileage exceeds 20,000. A two-level tree can represent this joint condition.
Mistake to avoid: Concluding that linear models cannot represent interactions when suitable interaction terms can be added.
Principal Components and Clustering
54. Understanding how scaling changes geometry
Distance-based clustering and variance-based principal components depend on predictor units. Centering removes location differences; standardizing divides by each predictor's standard deviation. Standardization is useful when comparable relative variation is intended, but it changes the analysis objective and can amplify low-variance noise. Scaling should reflect what similarity means in the application.
Worked example: Differences of 2 and 100 in variables with standard deviations 2 and 100 become standardized differences 1 and 1, giving Euclidean distance √2.
Mistake to avoid: Assuming raw-unit and standardized analyses must produce the same clusters or components.
55. Defining principal directions and loadings
The first principal component is the unit-length linear combination of centered predictors with maximum variance. Later components maximize remaining variance subject to orthogonality. Loadings describe the direction in predictor space. Multiplying every loading and corresponding score by −1 represents the same component, so its sign has no intrinsic meaning.
Worked example: Loadings 0.8 and 0.6 have squared length 0.64 + 0.36 = 1. The vector −0.8, −0.6 describes the identical axis with reversed orientation.
Mistake to avoid: Interpreting a sign reversal between two PCA outputs as evidence of a different underlying component.
56. Calculating component scores and reconstructions
A component score is the dot product of the processed predictor vector with its loading vector. Retaining selected components reconstructs an approximation by summing score times loading for those components. Convert back using the original centering and scaling when an original-unit reconstruction is required.
Worked example: For processed values (1, 2) and loadings (0.8, 0.6), the score is 2. Retaining only this component reconstructs (1.6, 1.2), rather than the original (1, 2).
Mistake to avoid: Treating a reduced-component reconstruction as exact or forgetting to reverse standardization.
57. Interpreting proportion of variance explained
A component's variance is its associated eigenvalue. Its proportion of variance explained is that eigenvalue divided by the sum of all eigenvalues. Cumulative proportions describe how much predictor variation a retained subspace captures. They do not measure response prediction accuracy, causal importance, or whether a component has a useful business interpretation.
Worked example: Eigenvalues 5, 3, and 2 give first-component proportion 0.5 and first-two-component cumulative proportion 0.8.
Mistake to avoid: Claiming that retaining 80% of predictor variance guarantees 80% predictive accuracy.
58. Applying the K-means objective
K-means minimizes within-cluster squared Euclidean distances to cluster means for a chosen K. Its iterative algorithm alternates nearest-center assignment with center updates. Different initial centers can lead to different local solutions, so multiple starts are useful. Cluster labels are arbitrary identifiers, and the geometry favors roughly compact groups.
Worked example: For observations 0, 2, 8, and 10 with K = 2, groups {0, 2} and {8, 10} have centers 1 and 9 and total squared distance 4.
Mistake to avoid: Assuming one initialization guarantees the global minimum of the clustering objective.
59. Comparing hierarchical linkage rules
Agglomerative hierarchical clustering starts with individual observations and repeatedly merges clusters. Single linkage uses the nearest cross-cluster pair, complete linkage the farthest pair, and average linkage the average cross-cluster distance. These definitions produce different dendrograms: single linkage can chain observations together, while complete linkage favors tighter groups.
Worked example: If cross-cluster distances are 2, 6, and 9, single linkage gives 2, complete linkage 9, and average linkage 17/3, approximately 5.67.
Mistake to avoid: Reading dendrogram merge heights without identifying the distance measure and linkage rule.
60. Choosing the number of clusters
Within-cluster variation decreases as K increases, so its minimum alone cannot select a useful K. An elbow examines diminishing improvement. Silhouette values compare within-cluster cohesion with separation from the nearest alternative cluster. These diagnostics support a decision alongside cluster stability and substantive interpretation; they need not identify one indisputable answer.
Worked example: Within-cluster sums 100, 60, and 55 for K = 1, 2, and 3 suggest an elbow at 2. With a = 2 and b = 5, silhouette is 0.6.
Mistake to avoid: Selecting one cluster per observation because it produces zero within-cluster variation.
Sources
Syllabus reference:
- SOA Statistics for Risk Modeling September 2026 syllabus, format, learning outcomes, weights, and readings
- Statistics for Risk Modeling (SRM) Exam