Intro to Loss Functions

There are many forms that a loss function can take. Deep learning introduces a much more general approach to constructing loss functions but this intro to loss functions should get you thinking about it the right way.

Models Minimize a Loss Function

A loss function is something that ML algorithms seek to minimize.

Every model comes with its own loss function that it minimizes either implicitly or explicitly Orderinary least squares (OLS) minimizes the sum of squared residuals. Lasso, ridge, and elastic net minimize an additional penalty term to OLS. Regression trees greedily solve $\sum_{j=1}^J \sum_{i \in R_J} (y_i - \hat y_{R_j})^2$ where $j$ indicates predictor spaces (pg. 306 from this book for me info), and neural networks minimize whatever loss function is given them.

Though models minimize loss functions, what are some general types of loss functions and what are each of them good for?

Examples of Regression Loss Functions

Let $d = y - \hat y$.

• Squared Loss: $loss = \sum _{i=1}^n d_i^2 = \pmb d'\pmb d$
• Absolute Deviation: $loss = \sum _{i=1}^n |d_i| = \pmb 1'\text{abs}(\pmb d)$
• Asymetric Loss: $loss = \sum_{i=1}^n d_i\cdot(\tau - 1(d_i<0))$ where $\tau \in (0,1)$ and $1(\cdot)$ is an indicator function that evaluates to $1$ when the condition is satisfied and $0$.

Let's simplify thinking about this function by just considering a single value of $d$ and plotting the loss as a function of a single loss amount $d$.

MSE becomes $loss = d^2$. Which is a parabola.

Absolute deviation becomes $loss = |d|$ which is a V shape with the bottom of the V on the origin.

Asymetric loss becomes

Which is V shaped loss function with the V on the origin, the difference this time is that over estimates (positive values of $d$) have slope $\tau$ while underestimates increase in loss at a rate of $1-\tau$. Note that absolute loss is a special case of asymmetric loss with $\tau = 0.5$

Comparing Regression Loss Functions

What are some differences between squared loss and absolute loss? First note that when $|d| < 1$ absolute loss is greater than squared loss. But note that when $|d| > 1$ MSE is greater and grows exponentially larger than absolute deviation. Knowing this, which model would be more resistant to high noise data when sample size is small? What would be real world example of when MSE would be more appropriate than mean absolute deviation?

Now consider asymmetric loss. When $\tau$ is close to $1$ over predictions receive large loss with almost no loss for under estimates. On the other hand when $\tau$ is close to $0$ the relationship is reversed.

Classification Loss

The loss functions discussed so far only make sense in a regression context. What would the general form for the loss of a binary classifier look like? If $\hat y \in \{0,1\}$ and $y \in \{0, 1\}$ then what values could we have for $d$ with $d = y - \hat y$? $d \in \{-1,0,1\}$. $-1$ would correspond to false positives, $1$ would be a false negative, and $0$ would be if the classification were correct. Therefore our loss function would look like this:

Best solutions for Loss Functions

It can be shown that the best loss prediction given a squared loss function is $E[y|x]$.

The best prediction for absolute loss is $Med(y|x)$.

For asymmetric loss, the best prediction is $Q_Y(1-\tau | x)$. Again, this is the median when $\tau = 0.5$.

For classification loss, the best prediction is