Least squares is a fundamental mathematical optimization technique used to find the best-fitting curve or line through a set of data points by minimizing the sum of the squares of the residuals (the differences between observed and predicted values). This method serves as the cornerstone of regression analysis and has widespread applications in statistics, engineering, physics, economics, and machine learning.
Mathematical Foundation
The least squares method seeks to minimize the sum of squared residuals between observed data points and values predicted by a mathematical model. For a system of linear equations represented as Ax = b, where A is an m×n matrix of known coefficients, x is an n×1 vector of unknown parameters, and b is an m×1 vector of observed values, the least squares solution finds the vector x that minimizes the Euclidean norm ||Ax - b||².
When the system is overdetermined (more equations than unknowns), the normal equations AᵀAx = Aᵀb provide the solution. The matrix AᵀA is symmetric and positive semi-definite, ensuring the existence of a solution. If A has full column rank, AᵀA is invertible and the unique solution is x = (AᵀA)⁻¹Aᵀb.
Historical Development
The method of least squares was independently developed by Carl Friedrich Gauss and Adrien-Marie Legendre in the early 19th century. Gauss claimed to have used the method as early as 1795 when calculating the orbit of the asteroid Ceres, though he did not publish his work until 1809 in "Theoria Motus Corporum Coelestium." Legendre published the first clear exposition of the method in 1805 in "Nouvelles méthodes pour la détermination des orbites des comètes."
The technique emerged from the need to reconcile inconsistent astronomical observations and measurement errors. Early applications focused on determining planetary orbits and geodetic surveying. The method's success in astronomy quickly established its importance in mathematical physics and statistics.
Linear Least Squares
Linear least squares deals with models that are linear in their parameters, regardless of whether the relationship between variables is linear. The general form is y = β₀ + β₁x₁ + β₂x₂ + ... + βₚxₚ + ε, where β₀, β₁, ..., βₚ are unknown parameters, x₁, x₂, ..., xₚ are independent variables, and ε represents random error.
The solution involves constructing the design matrix X and solving the normal equations (XᵀX)β = Xᵀy. Key assumptions include: errors have zero mean, constant variance (homoscedasticity), and are uncorrelated. Under these conditions, the least squares estimator is the best linear unbiased estimator (BLUE) according to the Gauss-Markov theorem.
Computational methods include QR decomposition, singular value decomposition (SVD), and Cholesky decomposition. SVD is particularly robust for ill-conditioned problems, while QR decomposition offers good numerical stability and efficiency.
Nonlinear Least Squares
Nonlinear least squares addresses models where parameters appear nonlinearly, such as y = β₀e^(β₁x) + ε or y = β₀ + β₁x^(β₂) + ε. Unlike linear least squares, nonlinear problems typically require iterative methods and may have multiple local minima.
The Gauss-Newton algorithm linearizes the model at each iteration using a first-order Taylor expansion. The Levenberg-Marquardt algorithm combines Gauss-Newton with gradient descent, providing better convergence properties. Starting values significantly impact convergence, and careful initialization is often necessary.
The objective function may have multiple local minima, making global optimization techniques sometimes necessary. Convergence criteria typically involve monitoring changes in parameter estimates, objective function values, or gradient norms.
Applications and Extensions
Least squares forms the foundation of linear regression, polynomial fitting, and signal processing. In statistics, it enables parameter estimation, hypothesis testing, and prediction. Engineering applications include system identification, control theory, and structural analysis.
Weighted least squares addresses heteroscedasticity by incorporating known error variances. Generalized least squares handles correlated errors. Ridge regression and LASSO introduce regularization to prevent overfitting and handle multicollinearity.
In modern machine learning, least squares appears in neural networks (mean squared error loss), support vector regression, and principal component analysis. Total least squares accounts for errors in both dependent and independent variables, important in errors-in-variables models.
Robust regression methods like M-estimators reduce sensitivity to outliers that can severely distort least squares solutions. These techniques assign lower weights to observations with large residuals.
Statistical Properties
Under standard assumptions, least squares estimators are unbiased and consistent. The Gauss-Markov theorem establishes their efficiency among linear unbiased estimators. When errors are normally distributed, least squares estimators achieve maximum likelihood and are asymptotically efficient.
The variance-covariance matrix of parameter estimates is σ²(XᵀX)⁻¹, where σ² is the error variance. Confidence intervals and hypothesis tests rely on this covariance structure. The coefficient of determination (R²) measures the proportion of variance explained by the model.
Residual analysis is crucial for validating assumptions. Patterns in residuals may indicate model misspecification, heteroscedasticity, or non-normality. Diagnostic measures include leverage, influence statistics, and goodness-of-fit tests.
Computational advances have made least squares applicable to massive datasets through techniques like stochastic gradient descent and distributed computing, maintaining its relevance in big data applications and real-time systems.