L02 Regressions

We will learn

• Algorithms
  • Linear regression
  • Penalized regression
  • Nonelinear regressions
  • Robust linear regression
• Coding with python

Least squares linear regression

Terminology

• Linear Regression Model

• model parameters:
  • weights or regression coefficients:
  • offset or bias: ()
  • by writing as , we can write as

Least squares estimation

• minimizing the negative log likelihood (NLL):
  • The MLE is the point where:
• the NLL is equal to the residual sum of squares (RSS)

OLS

the normal equation (FOC)

the OLS solution

the solution is unique since tha Hessian is positive definite

Statistical Properties of OLS (finite sample)

• unbiasedness:
• variance:
• Gauss-Markov: the OLS estimator is efficient in the class of linear unbiased estimators. That is, for any unbiased estimator that is linear in , in the matrix sense.

Geometric interpretation of least squares

• orthogonal projection

• projection matrix (hat matrix)

• special case:

Weighted least squares

• heteroskedastic regression

• weighted linear regression

• MLE (weighted least squares estimate):

Measuring goodness of fit

T-total: E-explained: R-residual:

Penalized (linear) regressions

Choosing the strength of the regularizer

• the simple (but expensive) idea
  • try a finite number of distinct values
  • use cross validation to estimate their expected loss
• a practitical method
  • start with a highly constrained model (strong regularizer)
  • gradually relax the constraints (decrease the amount of regularization)
• empirical Bayes approach:
  • get the same result as the CV estimate
  • can be done by fitting a single model
  • use gradient-based optimization instead of discrete search

Lasso regression

• least absolute shrinkage and selection operator(LASSO)

• -regularization: MAP estimation with a Laplace prior

• other norms
  • in general:
  • • even sparser solutions
    • the problem becomes non-convex
  • (norm):

Why does regularization yield sparse solutions?

Hard vs soft thresholding

Consider the partial derivatives of the lasso objective

• the NLL part
  • FOC

• the solution:

• adding the part

• the solution
  • If , so the feature is strongly negatively correlated with the residual, then the subgradient is zero at .
  • If , so the feature is only weakly correlated with the residual, then the subgradient is zero at .
  • If , so the feature is strongly positively correlated with the residual, then the subgradient is zero at .

Regularization path

Plot the values vs (or vs the bound ) for each feature .

Group lasso

• group sparsity
  • many parameters associated with a given variable
  • a vector of weights for variable
  • If we want to exclude variable , we have to force the whole subvector to go to zero

• applications
  • Linear regression with categorical inputs: If the ’th variable is categorical with possible levels, then it will be represented as a one-hot vector of length , so to exclude variable , we have to set the whole vector of incoming weights to .
  • Multinomial logistic regression: The ’th variable will be associated with different weights, one per class, so to exclude variable , we have to set the whole vector of outgoing weights to .
  • Neural networks: the ’th neuron will have multiple inputs, so if we want to “turn the neuron off”, we have to set all the incoming weights to zero. This allows us to use group sparsity to learn neural network structure.
  • Multi-task learning: each input feature is associated with different weights, one per output task. If we want to use a feature for all of the tasks or none of the tasks, we should select weights at the group level.

Coding: Lasso model selection via cross-validation

from sklearn.datasets import load_diabetes

X, y = load_diabetes(return_X_y=True, as_frame=True)
X.head()

Elastic net (ridge and lasso combined)

Nonelinear regression

Polynomial regression

• nonlinear in s
• still linear in the parameters (s)
• similar to multi-linear regression
• polynomial function imposes global structure

Coding: Polynomial regression

import numpy as np
import pandas as pd
import patsy as pt
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
import statsmodels.api as sm
from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model
import warnings
warnings.filterwarnings('ignore')

wage_df = pd.read_csv('./data/Wage.csv')
wage_df = wage_df.drop(wage_df.columns[0], axis=1)
wage_df['education'] = wage_df['education'].map(
  {'1. < HS Grad': 1.0, 
   '2. HS Grad': 2.0, 
   '3. Some College': 3.0,
   '4. College Grad': 4.0,
   '5. Advanced Degree': 5.0
                                                })
wage_df.head()

• preview of the dataset
  
  

Coding: Step function

### Step function steps = 6 # Segment data into 4 segments by age cuts = pd.cut(wage_df['age'], steps) X = np.asarray(pd.get_dummies(cuts)) y = np.asarray(wage_df['wage']) # Fit logistic regression model model = sm.OLS(y, X).fit(disp=0) y_hat = model.predict(X) # PLOT # ---------------------------------- # Setup axes fig, ax = plt.subplots(figsize=(10,10)) # Plot datapoints sns.scatterplot(x='age', y='wage', color='tab:gray', alpha=0.2, ax=ax, data=pd.concat([wage_df['age'], wage_df['wage']], axis=1)); # Plot estimated f(x) sns.lineplot(x=wage_df['age'], y=y_hat, ax=ax, color='blue')

Basis functions

• Polynomial and piecewise-constant regression models are special cases of a basis function approach.
• Basis fucntion: a family of functions or transformations that can be applied to a variable : .
• The model

• some examples of basis fucntions
  • polynomial function
  • piecewise constant function
  • wavelet
  • Fourier series
  • splines

Regression splines

Piecewise Polynomials

• fitting separate low-degree polynomials over different regions of .

• example: piecewise cubic polynomial with a single knot at a point

• degree of freedom

• Using more knots leads to a more flexible piecewise polynomial

Constraints and Splines

piecewise cubic: no constraint continuous piecewise cubic: continuity of cubic spline: continuity of , and degree- spline: a piecewise degree- polynomial, with continuity in derivatives up to degree at each knot

The Spline Basis Representation

• regression spline:

• the spline basis
  • polynomial basis: , , and
  • truncated power basis for each knot

• splines can have high variance at the outer range of the predictors
• natural spline
  • a regression spline with additional boundary constraints
  • the function is required to be linear at the boundary
  • natural splines generally produce more stable estimates at the boundaries

Comparison to Polynomial Regression

natural cubic spline with 15 degrees of freedom vs. degree-15 polynomial natural cubic spline works better on boundaries in general, natural cubic spline produces more stable estimates

Smoothing splines

An Overview of Smoothing Splines

• fitting a curve: minimize RSS to be small.
• should be a smooth function (WHY? & HOW?)
• smoothing spline minimize the following objective

• the smoothing spline is a natural cubic spline with knots at
  • piecewise cubic polynomial with knots at the unique values of
  • continuous first and second derivatives at each knot
  • linear in the region outside of the extreme knots
  • it is a shrunken version of such a natural cubic spline

Coding: Regression spline

# Putting confidence interval calcs into function for convenience. def confidence_interval(X, y, y_hat): """Compute 5% confidence interval for linear regression""" # STATS # ---------------------------------- # Covariance of coefficient estimates mse = np.sum(np.square(y_hat - y)) / y.size cov = mse * np.linalg.inv(X.T @ X) # ...or alternatively this stat is provided by stats models: #cov = model.cov_params() # Calculate variance of f(x) var_f = np.diagonal((X @ cov) @ X.T) # Derive standard error of f(x) from variance se = np.sqrt(var_f) conf_int = 2*se return conf_int # Fit spline with 6 degrees of freedom # Use patsy to generate entire matrix of basis functions X = pt.dmatrix('bs(age, df=7, degree=3, include_intercept=True)', wage_df) y = np.asarray(wage_df['wage']) # Fit logistic regression model model = sm.OLS(y, X).fit(disp=0) y_hat = model.predict(X) conf_int = confidence_interval(X, y, y_hat) # PLOT # ---------------------------------- # Setup axes fig, ax = plt.subplots(figsize=(10,10)) # Plot datapoints sns.scatterplot(x='age', y='wage', color='tab:gray', alpha=0.2, ax=ax, data=pd.concat([wage_df['age'], wage_df['wage']], axis=1)); # Plot estimated f(x) sns.lineplot(x=wage_df['age'], y=y_hat, ax=ax, color='blue'); # Plot confidence intervals sns.lineplot(x=wage_df['age'], y=y_hat+conf_int, color='blue'); sns.lineplot(x=wage_df['age'], y=y_hat-conf_int, color='blue'); # dash confidnece int ax.lines[1].set_linestyle("--") ax.lines[2].set_linestyle("--")

Coding: Natural spline

# Fit a natural spline with seven degrees of freedom # Use patsy to generate entire matrix of basis functions X = pt.dmatrix('cr(age, df=7)', wage_df) # REVISION NOTE: Something funky happens when df=6 y = np.asarray(wage_df['wage']) # Fit logistic regression model model = sm.OLS(y, X).fit(disp=0) y_hat = model.predict(X) conf_int = confidence_interval(X, y, y_hat) # PLOT # ---------------------------------- # Setup axes fig, ax = plt.subplots(figsize=(10,10)) # Plot datapoints sns.scatterplot(x='age', y='wage', color='tab:gray', alpha=0.2, ax=ax, data=pd.concat([wage_df['age'], wage_df['wage']], axis=1)); # Plot estimated f(x) sns.lineplot(x=wage_df['age'], y=y_hat, ax=ax, color='blue'); # Plot confidence intervals sns.lineplot(x=wage_df['age'], y=y_hat+conf_int, color='blue'); sns.lineplot(x=wage_df['age'], y=y_hat-conf_int, color='blue'); # dash confidnece int ax.lines[1].set_linestyle("--") ax.lines[2].set_linestyle("--")

Local regression

computing the fit at a target point using only the nearby training observations

Algorithm

Local linear regression

Generalized additive models

GAMs for Regression Problems

• the multiple linear regression model

• GAM

An Example (using natural spline)

• year and age are quantitative variables
• education is qualitative with five levels: , HS, <Coll, Coll, >Coll
• natural splines

An Example (using smoothing spline)

Pros and Cons of GAMs

• Pros

  • GAMs automatically model non-linear relationships that standard linear regression will miss.

  • The non-linear fits can potentially make more accurate predictions for the response .

  • We can examine the effect of each on individually while holding all of the other variables fixed.

  • The smoothness of the function for the variable can be summarized via degrees of freedom.

• Cons: the model is restricted to be additive.

Coding: GAM

# Use patsy to generate entire matrix of basis functions
X = pt.dmatrix('cr(year, df=4)+cr(age, df=5) + education', wage_df)
y = np.asarray(wage_df['wage'])

# Fit logistic regression model
model = sm.OLS(y, X).fit(disp=0)
y_hat = model.predict(X)
conf_int = confidence_interval(X, y, y_hat)

Robust linear regression

Laplace likelihood

Student- likelihood

• We can fit this model using SGD or EM

Huber loss

• An alternative to minimizing the NLL using a Laplace or Student likelihood is to use the Huber loss:

• It is equivalent to for errors that are smaller than , and is equivalent to for larger errors.

• Huber loss function is everywhere differentiable.

• optimizing the Huber loss is much faster than using the Laplace likelihood

• controls the degree of robustness

  • set by hand
  • by crossvalidation