L03 Classification

We will learn

• Algorithms
  • Logistic regression
  • Generative models for classification (LDA, QDA, naive Bayes)
  • GAM
  • Support Vector Machine
• Coding with python

Introduction

Examples of classification problems

• A person arrives at the emergency room with a set of symptoms that could possibly be attributed to one of three medical conditions. Which of the three conditions does the individual have?
• An online banking service must be able to determine whether or not a transaction being performed on the site is fraudulent, on the basis of the user’s IP address, past transaction history, and so forth.
• On the basis of DNA sequence data for a number of patients with and without a given disease, a biologist would like to fgure out which DNA mutations are deleterious (disease-causing) and which are not.

Regression is not appropriate for classification tasks

• regression methods can not accommodate a qualitative response with more than wo classes

• regression methods can not provide estimation of the conditional probability of responses

The default dataset

default student balance income 1 No No 729.5264952 44361.62507 2 No Yes 817.1804066 12106.1347 3 No No 1073.549164 31767.13895 4 No No 529.2506047 35704.49394 5 No No 785.6558829 38463.49588

Losgistic regression

The logistic model

The probability of default Linear regression logistic function

Estimation and Predictions

Multiple logistic regression

• prediction

A student with a credit card balance of $1,500 and an income of $40, 000 has an estimated probability of default of A non-student with the same balance and income has an estimated probability of default of

Multinomial logistic regression

• classify a response variable that has more than two classes

• the model

  • for : the baseline

• for

• the log odds (for )

Coding: Logistic Regression

import numpy as np
import pandas as pd
from matplotlib.pyplot import subplots
import statsmodels.api as sm
from ISLP import load_data
from ISLP.models import (ModelSpec as MS, summarize)
from ISLP import confusion_table
from ISLP.models import contrast
from sklearn.discriminant_analysis import \
     (LinearDiscriminantAnalysis as LDA,
      QuadraticDiscriminantAnalysis as QDA)
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

Smarket = load_data('Smarket')

Generative models for classification

• The bigidea of generative models for classification
  • model the distribution of the predictors separately in each of the response classes
  • use Bayes' theorem to flip these around into estimates for
• Why do we need the generative models for classification
  • When there is substantial separation between the two classes, the parameter estimates for the logistic regression model are surprisingly unstable
  • If the distribution of the predictors is approximately normal in each of the classes and the sample size is small, then the approaches in this section may be more accurate than logistic regression
  • The methods in this section can be naturally extended to the case of more than two response classes

An Example

and assigns an observation to class 1 if , and to class 2 otherwise The Bayes decision boundary

LDA for

• The multivariate Gaussian distribution

jointed density LHS: and RHS: correlated / have unequal variances

Coding: LDA

lda = LDA(store_covariance=True)
X_train, X_test = [M.drop(columns=['intercept'])
                   for M in [X_train, X_test]]
lda.fit(X_train, L_train)

lda_pred = lda.predict(X_test)
confusion_table(lda_pred, L_test)

Quadratic discriminant analysis (QDA)

each class has its own covariance matrix the Bayes classifier assigns an observation to the class for which maximizes

Coding: QDA

qda = QDA(store_covariance=True)
qda.fit(X_train, L_train)

qda_pred = qda.predict(X_test)
confusion_table(qda_pred, L_test)

Naive Bayes

• Assumption: Within the kth class, the p predictors are independent

• the posterior probability

Estimating the one-dimensional density function using training data

• If is quantitative, then we can assume that
• If is quantitative use a non-parametric estimate for
  • making a histogram for the observations of the th predictor within each class
  • kernel density estimator
• If is qualitative count the proportion of training observations for the th predictor corresponding to each class

Coding: QDA

NB = GaussianNB()
NB.fit(X_train, L_train)

nb_labels = NB.predict(X_test)
confusion_table(nb_labels, L_test)

Generalized additive models

Model the log odds ratio as a generalized additive models：

Support vector machine

• developed in the 1990s
• perform well in a variety of settings
• often considered one of the best "out of the box" classifiers.

Maximal Margin Classifier

Hyperplane

In a -dimensional space: a flat affine subspace of dimension In 2-d: a line In 3-d: a plane In -d (): hard to visualize The mathematical definition (for the -d setting) the 2-d example:

Classification Using a Separating Hyperplane

for a seperating hyperplane

and

Equivalently, a separating hyperplane has the property that

for all .

Separating Hyperplanes

we classify the test observation based on the sign of class 1 class -1 the magnitude of

The Maximal Margin Classifier

margin maximal margin hyperplane (a.k.a. the optimal separating hyperplane) Construction of the Maximal Margin Classifier

The Non-separable Case & Noisy Data

	sometimes data are non-seperateble
	sometimes the maximal margin classifier is very sensitive to noisy data

Support Vector Classifiers

: the i-th obs is on the correct side of the margin : the i-th obs is on the wrong side of the margin : the i-th obs is on the wrong side of the hyperplane

Parameter

is the budget for the amount that the margin can be violated by the observations : no budget for violations to the margin : no more than obs can be on the wrong side of the hyperplane : margin will widen controls the bias-variance trade-off small : low bias, high variance big : high bias, low variance selected via CV An observation: only observations that either lie on the margin or that violate the margin (support vectors) will afect the hyperplane

Coding: SVC

imports import numpy as np from matplotlib.pyplot import subplots, cm import sklearn.model_selection as skm from ISLP import load_data, confusion_table from sklearn.svm import SVC from ISLP.svm import plot as plot_svm from sklearn.metrics import RocCurveDisplay roc_curve = RocCurveDisplay.from_estimator # shorthand create data rng = np.random.default_rng(1) X = rng.standard_normal((50, 2)) y = np.array([-1]*25+[1]*25) X[y==1] += 1 fig, ax = subplots(figsize=(8,8)) ax.scatter(X[:,0], X[:,1], c=y, cmap=cm.coolwarm)

fitting svm_linear = SVC(C=10, kernel='linear') svm_linear.fit(X, y) plotting fig, ax = subplots(figsize=(8,8)) plot_svm(X, y, svm_linear, ax=ax)

You may try with other parameter values (e.g. ).

Support Vector Machines

Support Vector Machines can not handle nonlinearity.

What can we do?

Nonlinear Classifiers Utilizing Polynomial Features

The original features The polynomial features The SVM via Optimization

Kernel Functions

• Definition:

• is a kernel function if and only if the kernel matrix is positive semi-definite for any data .

• Some examples of kernel functions

Suppose and are kernel functions:

• The linear combination of and is a kernel function

• The direct product of and is a kernel function

• For any function , is a kernel function if

SVC vs. SVM

Coding: SVM

create data X = rng.standard_normal((200, 2)) X[:100] += 2 X[100:150] -= 2 y = np.array([1]*150+[2]*50) plotting fig, ax = subplots(figsize=(8,8)) ax.scatter(X[:,0], X[:,1], c=y, cmap=cm.coolwarm)

fitting (X_train, X_test, y_train, y_test) = skm.train_test_split(X, y, test_size=0.5, random_state=0) svm_rbf = SVC(kernel="rbf", gamma=1, C=1) svm_rbf.fit(X_train, y_train) plotting fig, ax = subplots(figsize=(8,8)) plot_svm(X_train, y_train, svm_rbf, ax=ax)

You may try with other parameter values (e.g. ).

SVMs with More than Two Classes

• One-Versus-One (OVO) Classification

  • a.k.a. all-pairs
  • constructs a SVM for each pair of classes
  • classify a test obs using each of the SVMs
  • assign the obs to the class to which it was most frequently assigned in

• One-Versus-All (OVA) Classification

  • a.k.a. one-versus-rest
  • fit SVM for each class (coded as "1" and the rest was coded as "-1")
  • let denote the parameters
  • assign the observation to the class for which is largest

Coding: SVM with Multiple Classes

data and plotting rng = np.random.default_rng(123) X = np.vstack([X, rng.standard_normal( (50, 2))]) y = np.hstack([y, [0]*50]) X[y==0,1] += 2 fig, ax = subplots(figsize=(8,8)) ax.scatter(X[:,0], X[:,1], c=y, cmap=cm.coolwarm) fitting svm_rbf_3 = SVC( kernel="rbf", C=10, gamma=1, decision_function_shape='ovo'); svm_rbf_3.fit(X, y) fig, ax = subplots(figsize=(8,8)) plot_svm(X, y, svm_rbf_3, scatter_cmap=cm.tab10, ax=ax)