Visualizing Models with Known Functional Form

Intro to data space and parameter space.

Background

Suppose we have a model given by

\begin{align*} \tilde f(x) = \frac{\tilde \theta_{1}}{1 + \exp(\tilde \theta_2 + \tilde \theta_3t)} \end{align*}

In this model $t$ is the independent variable and the $\theta$ 's are parameters of the model. We write $\tilde \theta$ to emphasize that these are the true values of $\theta$ but are unknown. Instead we fit a model of this form and call it

\begin{align*} \hat f^*(x) = \frac{\theta_1^*}{1+ \exp(\theta_2^* + \theta_3^* t)} \end{align*}

In this case we write $\theta^*$ to emphasize that this is the answer we recover from data. If our model is correct, the data generating process is

\begin{align*} y = \tilde f(x) + \epsilon \end{align*}

Where $\epsilon$ is the disturbance term. For sake of this discussion, we won't discuss $\epsilon$ but know that we assume $E[\epsilon| t] = 0$ .

Now, since the goal of research is to find the $\tilde \theta$ 's we design an experiment where we sample at time $t_i$ and recover output $y_i$ . Suppose we sample at $n$ distinct times so that our dataset looks like $\{(t_1, y_1), \dots, (t_n, y_n)\}$ .

Now that we've collected data, we're ready to fit the model with the best parameters.

What does it mean to find the "best" parameters? This depends on how we decide to measure loss. The most conventional loss function is to minimize MSE but depending on the context of your problem, there are many other loss functions that can be chosen.

Visualization

Now that we understand the full context of our modeling problem, we can discuss various visualization of the model

Spaghetti Plot

The spaghetti plot is a visualization of the prediction space for the model. Think of the model as $f(t; \theta_1, \theta_2, \theta_3)$ . Fix $\theta$ 's so that we now have $f(t)$ and trace out the line that is formed by plotting all pairs $(t,f(t))$ as a line.

Now choose different $\theta$ 's and repeat the process tracing out a second line.

Repeat this process many times. The result will be a "spaghetti plot" because the many lines, where each line represents a different realization of the $\theta$ 's.

Finally, we can overlay our data point on the graph by plotting the pairs $\{(t_1,y_1),\dots,(t_n,y_n)\}$ . To keep the metaphor going, we could call these data points "meat balls".

The "best" model in this picture would be the line that passes the "closest" to the data points.

Method:

Choose parameters, $\pmb \theta$ , of the model.
Plot $f(t)$ on $t$ .
Repeat steps (1) and (2) with various $\theta$ .
(Optional) plot observed data. $\{(t_1,y_1), \dots, (t_n,y_n) \}$

Parameter Space

If allow the parameters to be the axis on the graph, and then we calculate the loss over the set $\{(t_1, y_1), \dots, ((t_n, y_n)\}$ . If we only have $2$ parameters, then this plot can be represented as a contour plot.

Can we plot data points in this space? No, but from the previous paragraph we see that we embeded the datapoints as loss in our contour graph.

The best model in this visualization is the one that corresponds to the lowest point on the contour graph.

Method:

Fix all but 2 parameters. so that $\sum_{i=1}^n L(f(\theta_1, \theta_2, t_i), y_i)$ .
Assign the x-axis to $\theta_1$ and the y-axis to $\theta_2$ .
Plot the losses on the space where the value of the loss is represented by the color. This will form contours that are like a mountainous loss landscape.

Data Space

In dataspace, we assign each axis to some $f(t_i)$ . We quite certainly will always have more than $3$ values of $t$ to choose from but to visualize this space we can only pick $3$ values of $t$ . After we've chosen the three $t_i$ 's we still don't know what $f(t_i)$ evaluates to until will pick values for the $\theta$ 's. In this space, we can actually plot all values of $\theta$ . To see this, fix all but on $\theta$ and then let the last $\theta$ move freely to trace out a curve living in this 3D space.

As we continue to allow the $\theta$ 's to vary and tracing out all possible values, we end up with a curved surface (a manifold) living within this space. This manifold is called the model monifold because it shows all the space that model can reach in "data space".

Finally, we can plot the realization of the observed data. In this space, the data can be represented by a single point. Remember the three values of $t$ we picked to evaluate the output of the model at? Call those point $t_i, t_j, t_z$ , then the realized data is simply the point at $(y_i, y_j, y_z)$ . If the speghetti plot didn't have a model realization that passed through all the data points, then the point representing the data in this graph will be off the manifold.

In this space, the best model is the point on the model that has the "smallest" vector reaching to the data point.

Method:

Pick 3 values of $t$ . Call them $\{t_1, t_2, t_3\}$
For all values of $\theta_1, \dots, \theta_p$ plot the point $\big(f(\pmb \theta;t_1), f(\pmb \theta; t_2), f(\pmb \theta; t_3)\big)$ . All these dots will form a curved surface called the model manifold.
(Optional) plot the point in space representing the data. This is $\big((t_1, y_1),(t_2, y_2),(t_3, y_3)\big)$ . Note that this will be off the manifold unless $\epsilon_1 = \epsilon_2 = \epsilon_3 = 0$ .