Taylor Series and Multivariate Delta Method

I asked this question on https://math.stackexchange.com/ but did not get any answer. Sorry for cross posting.

I’m trying to understand delta method for matrices and vectors to find the variance-covariance matrices for the functions of matrices and vectors. Please see my attempt below. I’m not sure it is right or wrong and especially confused whether it should be
$\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} $ or $\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}^{\prime}\right)}\right\} $. I would highly appreciate experts help in this regard. Thanks

If $\mathbf{X}$ is a matrix then a Taylor series expansion of $f\left(\mathbf{X}\right)$
about $\mathbf{X}_{0}$ is given by:

\begin{align*}
f\left(\mathbf{X}\right) & =f\left(\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right)+\left(\begin{array}{c}
\textrm{2-nd and higher}\\
\textrm{order term}
\end{array}\right)\\
f\left(\mathbf{X}\right) & \approxeq f\left(\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right).
\end{align*}

The approximated variance of $f\left(\mathbf{X}\right)$ about $\mathbf{X}_{0}$
can be computed as

\begin{align*}
\mathbb{V}\left[f\left(\mathbf{X}\right)\right] & \approxeq\left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right]
\end{align*}

Example

If $f\left(\mathbf{X}\right)=\mathbf{X}^{-1}$ then
\begin{align*}
\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}} & =\frac{\partial\mathbf{X}^{-1}}{\partial\mathbf{X}}=-\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}
\end{align*}
Therefore the approximated variance of $f\left(\mathbf{X}\right)=\mathbf{X}^{-1}$
about $\mathbf{X}_{0}$ is
\begin{align*}
\mathbb{V}\left(\mathbf{X}^{-1}\right) & \approxeq\left[-\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[-\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right]\\
& =\left[\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right].
\end{align*}

If $\mathbf{X}$ is a matrix and $\mathbf{y}$ is a column vector
then a Taylor series expansion of $f\left(\mathbf{X},\mathbf{y}\right)$
about $\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)$ is given by:
\begin{align*}
f\left(\mathbf{X},\mathbf{y}\right) & =f\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{y}-\mathbf{y}_{0}\right)+\left(\begin{array}{c}
\textrm{2-nd and higher}\\
\textrm{order term}
\end{array}\right)\\
f\left(\mathbf{X},\mathbf{y}\right) & \approxeq f\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{y}-\mathbf{y}_{0}\right)
\end{align*}

The approximated variance of $f\left(\mathbf{X},\mathbf{y}\right)$
about $\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)$ can be computed
as

\begin{align*}
\mathbb{V}\left[f\left(\mathbf{X},\mathbf{y}\right)\right] & \approxeq\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\mathbf{\mathbb{V}\left(\mathbf{y}\right)}\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]
\end{align*}

Example

If $f\left(\mathbf{X},\mathbf{y}\right)=\mathbf{X}\mathbf{y}$ then
\begin{align*}
\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}} & =\frac{\partial\mathbf{X}\mathbf{y}}{\partial\mathbf{X}}=\frac{\partial\textrm{vec}\left(\mathbf{X}\mathbf{y}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}=\frac{\partial\textrm{vec}\left(\mathbf{I}\mathbf{X}\mathbf{y}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}\\
& =\frac{\partial\left(\mathbf{y}^{\prime}\otimes\mathbf{I}\right)\textrm{vec}\left(\mathbf{X}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}=\left(\mathbf{y}^{\prime}\otimes\mathbf{I}\right)\frac{\partial\textrm{vec}\left(\mathbf{X}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}\\
& =\left(\mathbf{y}^{\prime}\otimes\mathbf{I}\right)\qquad\textrm{and}\\
\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}} & =\frac{\partial\mathbf{X}\mathbf{y}}{\partial\mathbf{y}}=\mathbf{X}.
\end{align*}
Therefore the approximated variance of $f\left(\mathbf{X},\mathbf{y}\right)=\mathbf{X}\mathbf{y}$
about$\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)$ is
\begin{align*}
\mathbb{V}\left(\mathbf{X}\mathbf{y}\right) & \approxeq\left(\mathbf{\mathbf{y}_{0}}^{\prime}\otimes\mathbf{I}\right)^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left(\mathbf{\mathbf{y}_{0}}^{\prime}\otimes\mathbf{I}\right)+\mathbf{\mathbf{X}}_{0}^{\prime}\mathbf{\mathbb{V}\left(\mathbf{y}\right)}\mathbf{\mathbf{X}}_{0}.
\end{align*}

Edited

Let $\mathbf{A}$ be $m\times n$ matrix with $\mathbf{a}_{i}(i=1,\ldots,n)$
as the \emph{$i$}-th column vector. The vectorization operation ,
$\textrm{vec}\left(.\right)$, is an operation from $\mathcal{R}^{m\times n}$
to $\mathcal{R}^{mn}$, with

$
\textrm{vec}(\mathbf{A})=\textrm{vec}\left(\begin{bmatrix}a_{11} & \ldots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{m1} & \ldots & a_{mn}
\end{bmatrix}\right)=\textrm{vec}\left(\left[\begin{array}{ccc}
\mathbf{a}_{1} & \cdots & \mathbf{a}_{n}\end{array}\right]\right)=\left[\begin{array}{c}
\mathbf{a}_{1}\\
\vdots\\
\mathbf{a}_{n}
\end{array}\right].
$

Answer

Attribution
Source : Link , Question Author : MYaseen208 , Answer Author : Community

Leave a Comment