Linear Models for Regression and Classification

Suppose you want to predict tomorrow’s temperature, or guess whether a customer will click on an ad. Behind many of these predictions are two simple but powerful ideas: drawing a straight line through data, and bending that line to handle yes-or-no answers. This chapter builds those ideas from the ground up, and shows that they both come from one elegant principle: picking the model that makes the observed data most likely.

The Big Picture#

Linear models are often the first tool you reach for in supervised learning, and they remain benchmarks decades later because they are fast, interpretable, and surprisingly capable. In this chapter we start with linear regression for predicting numbers. Then we meet logistic regression for predicting classes. We will see that both can be understood using maximum likelihood estimation (MLE), which simply asks: “What parameter values would make our data most likely?” By the end, you will know how these models work and why they share the same mathematical foundation.

Linear Regression – The Simple Starting Point#

Imagine you have a scatterplot of house sizes () and their sale prices (). The points form a rough cloud sloping upward. Linear regression draws a single straight line through that cloud to capture the relationship and make predictions.

Linear regression: A supervised learning model that predicts a continuous target as a straight-line function of one or more input features .

For a single feature, the line is written:

Here is the intercept (where the line crosses the -axis) and is the slope (how much changes when increases by 1). With several features , we extend the idea to a hyperplane:

where is a vector of coefficients and is a dot product. The model is linear because the prediction is a linear combination of the inputs, not because itself must enter linearly. You could include or as a new “feature” and the model stays linear in the parameters.

The job of training is to choose and so the line fits the data as well as possible. The classic measure of “fit” is the sum of squared residuals:

Why squares? One reason is geometry: minimizing RSS gives the line that is closest to the points in a vertical direction. Another reason, more probabilistic, is that if we assume the errors (the vertical scatter) follow a normal distribution with constant variance, then minimizing RSS is exactly the same as making the data most likely. We will make that precise in a later section.

For a single feature, the optimal slope and intercept have closed-form solutions:

where and are the sample means. For multiple features, the solution is written compactly with matrices:

This is the normal equation. Here is an matrix whose rows are the feature vectors with a leading 1 for the intercept, and is the vector of targets. When or is huge, we may solve it with gradient descent instead, but the idea is the same: we are finding the parameter vector that minimizes the sum of squared errors.

Interpreting the coefficients is a great thing about linear models. If , then on average, increasing by 1 unit (and holding other features fixed) increases the predicted by 3.2 units. This direct “change in per unit change in ” makes linear regression a natural first model for understanding relationships in data.

📝 Section Recap: Linear regression fits a straight line (or hyperplane) to data by minimizing the sum of squared residuals, and its coefficients are easy to interpret as the expected change in the target per unit change in a feature.

Logistic Regression – From Probabilities to Decisions#

Now suppose you want to predict whether an email is spam () or not spam (). The target is no longer a number that can be any real value; it is a class label. If we simply fitted a straight line to this 0/1 target, the predictions could wander far below 0 or above 1, which makes no sense for a probability. Logistic regression solves this by wrapping the linear score inside a function that squeezes it into the interval .

At the heart of the model is the logistic function (also called the sigmoid):

Its shape is an S-curve: for large negative , approaches 0; for large positive , it approaches 1; at , it equals 0.5. The logistic regression model produces a probability estimate through:

Logistic regression: A linear model for binary classification that estimates the probability of the positive class by applying the logistic (sigmoid) function to a linear combination of the features.

Why the logistic function in particular? One clean answer comes from the log-odds. The odds of an event with probability is . For example, if the probability of rain is 0.75, the odds are 3:1 (3 to 1). Taking the natural logarithm gives the log-odds (or logit):

Logistic regression models the log-odds as a linear function of the inputs:

Solving for yields exactly the sigmoid formula. This log-odds transform maps a probability (0 to 1) to the entire real line , where a straight line is a natural model. It also gives the coefficients a familiar interpretation: for each one-unit increase in , the odds multiply by . If , then , meaning the odds of the positive class increase by about 65% for each unit of (all else equal). This “odds ratio” interpretation is a powerful way to communicate results.

To make a hard classification, we typically set a threshold – often 0.5 – and predict the class with the higher probability. The decision boundary where the two classes are equally likely occurs where the linear score equals zero: . So logistic regression captures the data with a separating hyperplane, just like a linear classifier, but it also provides reliable probability estimates.

Now, how do we find good coefficients? Unlike linear regression with squared error, there is no closed-form formula. Instead we turn to the principle that unites this chapter: maximum likelihood estimation.

📝 Section Recap: Logistic regression models the probability of a class by applying the sigmoid to a linear score; the log-odds transform makes the model linear and gives coefficients an odds-ratio interpretation.

Maximum Likelihood Estimation – A Unified View#

Both linear regression and logistic regression can be learned by asking the same fundamental question: Which parameter values make the observed data most likely? This is maximum likelihood estimation (MLE).

Maximum likelihood estimation (MLE): A method for learning model parameters by maximizing the probability (likelihood) of the observed data under the model’s assumed distribution.

We start by imagining a story for how the data were generated. The story defines a likelihood function – the joint probability of all the training examples, written in terms of the parameters . To make the math easier, we often work with the log-likelihood , which turns products of probabilities into sums and is numerically nicer.

Consider linear regression first. A natural assumption is that the target equals a linear function plus independent, normally distributed noise with variance :

Under this model, each is normally distributed with mean and variance . The likelihood for a single observation is the normal density:

Because the observations are assumed independent, the joint likelihood is the product. Taking the log (and keeping only the terms that depend on ) we get:

Maximizing is clearly the same as minimizing , the residual sum of squares. So the ordinary least squares solution is exactly the maximum likelihood estimate under normal noise. This is why we minimize squares: it comes naturally from the assumption that errors are normally distributed.

Now turn to logistic regression. Here each is a 0 or 1, so the natural assumption is a Bernoulli distribution with parameter . The likelihood for one observation is:

If this equals ; if it equals . Assuming independence, the joint likelihood and its log are:

This log-likelihood is the negative of a quantity known as binary cross-entropy loss. Maximizing it pushes toward 1 when and toward 0 when – exactly what we want. There is no closed-form solution, so we use iterative optimization algorithms like gradient descent or Newton’s method to find the that maximizes .

So both models, though they look different, are learned using the same idea: maximize the likelihood. The only difference is the assumed data distribution: normal for regression, Bernoulli for classification. This insight lets you create new models by simply swapping the distribution; for example, a Poisson distribution for count data gives Poisson regression.

We often add a regularization term to prevent overfitting, but that is a story for another time. For now, remember that learning these linear models is about asking: given what we assume about the noise, what parameters make our training data the most likely outcome?

📝 Section Recap: Maximum likelihood estimation provides a single framework for learning both linear regression (via Gaussian likelihood) and logistic regression (via Bernoulli likelihood), turning loss functions into natural consequences of probabilistic assumptions.

Summary#

Linear regression and logistic regression may seem different – one predicts numbers, the other probabilities – but they share a deep connection. Both build on a linear combination of features, and both are trained by maximum likelihood estimation: pick the parameters that make the data most likely under a reasonable probability model. Understanding this connection gives you a template for designing and interpreting many other supervised learning algorithms: choose a distribution for the target, write down the likelihood, and then optimize. The next time you see a new model, try asking: what probability story is it telling?