Module 08 - Sum of Squares Framework

Module 08 - Sum of Squares Framework (TSS, ESS, RSS, R²)

This study reference details the Analysis of Variance (ANOVA) sum of squares decomposition framework used to evaluate ordinary least squares (OLS) regression models. It maps out how total target variability splits into explained and unexplained components, providing the mathematical basis for coefficients of determination.

1. The ANOVA Sum of Squares Decomposition

To evaluate how well a regression line tracks empirical data, the total variation in the dependent target variable is broken down into three component elements:

Total Sum of Squares (TSS)

Measures the baseline total variability present within the dependent target variable. It calculates the sum of the squared differences between each raw observed data point and the general mean of the dependent variable. Graphically, this captures the total variation of the observed values around a horizontal intercept line mapping the target average.

TSS = Σ(y_actual - y_mean)²

Explained Sum of Squares (ESS)

Measures the portion of target variability that is successfully captured and explained by the independent predictor features. It aggregates the sum of the squared differences between the model's predicted values and the baseline mean of the dependent variable. Graphically, this displays the variance of the regression line values around the target average.

ESS = Σ(y_predicted - y_mean)²

Residual Sum of Squares (RSS)

Also known as the Sum of Squared Errors (SSE), this parameter measures the remaining variability in the dependent variable that the model fails to explain. It calculates the sum of the squared differences between the raw observed values and the model's matching predicted coordinates. Graphically, this represents the scatter of observed data points directly around the optimized line of best fit.

RSS = Σ(y_actual - y_predicted)²

These components fit together in a strict additive framework representing total model variance:

2. Correlation vs. Determination Indices

R-squared (R² / Coefficient of Determination): A standardized goodness-of-fit metric that defines the precise proportion of total variance in the dependent variable explained by the regression inputs. Derived directly from the sum of squares decomposition framework:

R² = ESS / TSS = 1 - (RSS / TSS)

A score of 1 represents a perfect fit with zero residual error. However, a high R² score does not automatically guarantee a well-specified model, just as a lower score does not render an experimental dataset invalid; context and assumption testing remain critical.

R (Multiple Correlation Coefficient): In simple linear regression layouts, R equals the exact square root of R-squared, sharing the identical directional sign as the model's slope coefficient. It measures the joint strength and direction linking the estimated values to the raw target data.

r (Pearson's Correlation Coefficient): Measures the linear correlation strength mapping two separate continuous sets of data by dividing their covariance by the product of their respective standard deviations. Unlike regression parameters, r is a symmetrical relationship metric that treats both continuous variables identically without enforcing an independent vs. dependent structure.

3. Model Parsimony & Out-of-Sample Performance

Adjusted R-squared: A modified adaptation of the standard R² coefficient that penalizes the addition of useless independent variables. Standard R² values will mathematically increase or remain constant whenever a new feature is appended—even if that feature contains only random noise. Adjusted R-squared accounts for this by incorporating degrees of freedom parameters:

Adj R² = 1 - [((1 - R²) * (n - 1)) / (n - k - 1)]

Where n equals total observations and k equals total model parameters (excluding the intercept constant). It is a far more reliable metric when evaluating and comparing competing models with varying feature lengths.

Out-of-Sample R-squared (OOS R²): Evaluates model performance using a completely separate validation testing block that was excluded during training optimization. This metric is a vital diagnostic to verify whether an optimized model generalizes well to fresh data rows or is suffering from overfitting.

Degrees of Freedom: The count of independent pieces of statistical data available to estimate a parameter. In variance analysis, it calibrates the penalization matrix to adjust for sample sizes relative to model parameter length.

4. Key Structural Cautions

Interpretation Bounding: Goodness-of-fit metrics must be paired with comprehensive residual diagnostic checks. High fits can still mask critical baseline assumption violations like non-linearity or heteroscedasticity.

Causality Boundaries: Strong linear correlation or high R² determination scores do not imply or prove a causal link between variables. Joint variance alignments can be heavily driven by hidden confounding parameters or random sampling chance.