Preconditioning

Monkeying with the gradient

An essential part of any model update is the gradient

$$\Delta \bold m = \partial \bold r / \partial \bold m\T = \bold F\T \bold r$$ (4)

which vanishes $\bold 0=\bold F\T \bold r$ at the ultimate solution.

With the change of variables $\bold m = \bold S \bold p$ the gradient changes as $\Delta \bold m = \bold S \Delta \bold p$ . In terms of $\bold p$ the regression $\bold 0 \approx \bold r = \bold F \bold m - \bold d$ becomes $\bold 0 \approx \bold r = \bold F \bold S \bold p - \bold d$ with a gradient $\Delta \bold p = \bold S\T \bold F\T \bold r$ . Changing variables back to $\bold m$ gives

$$\Delta \bold m = \bold S \Delta \bold p = \bold S \bold S\T \bold F\T \bold r$$ (5)

The ultimate solution is when the gradient vanishes $\bold 0 = (\bold S\bold S\T) \bold F\T \bold r$ . Most often the ultimate solution is the same with or without the change of variables because most often the transformation matrix $\bold S$ is chosen invertible so $(\bold S\bold S\T)$ has an inverse which will cancel it.

We conclude that when a gradient is multiplied by any positive definite matrix $\bold S\bold S\T$ we are simply solving the original problem in a new coordinate system. Choosing $\bold S\bold S\T$ really matters only when the problem is big so it converges slowly, or when the physics or statistical approach is non-linear (later chapter). The choice of $\bold S$ is a subjective matter. It's an area where prior information or your physical intuition is relevant. Calling both $\bold m$ and $\bold p$ model space, obvious choices for $\bold S$ are weighting functions and filtering functions in model space. You would weight the model space smaller where your intuition tells you your ultimate model will be small or where it might not be learnable from the data.

Preconditioning