# Recursively updating the MLE as new observations stream in

## General Question

Say we have iid data $$x1x_1$$, $$x2x_2$$, … $$\sim f(x\,|\,\boldsymbol{\theta})\sim f(x\,|\,\boldsymbol{\theta})$$ streaming in. We want to recursively compute the maximum likelihood estimate of $$\boldsymbol{\theta}\boldsymbol{\theta}$$. That is, having computed
$$\hat{\boldsymbol{\theta}}_{n-1}=\underset{\boldsymbol{\theta}\in\mathbb{R}^p}{\arg\max}\prod_{i=1}^{n-1}f(x_i\,|\,\boldsymbol{\theta}),\hat{\boldsymbol{\theta}}_{n-1}=\underset{\boldsymbol{\theta}\in\mathbb{R}^p}{\arg\max}\prod_{i=1}^{n-1}f(x_i\,|\,\boldsymbol{\theta}),$$
we observe a new $$x_nx_n$$, and wish to somehow incrementally update our estimate
$$\hat{\boldsymbol{\theta}}_{n-1},\,x_n \to \hat{\boldsymbol{\theta}}_{n}\hat{\boldsymbol{\theta}}_{n-1},\,x_n \to \hat{\boldsymbol{\theta}}_{n}$$
without having to start from scratch. Are there generic algorithms for this?

## Toy Example

If $$x_1x_1$$, $$x_2x_2$$, … $$\sim N(x\,|\,\mu, 1)\sim N(x\,|\,\mu, 1)$$, then
$$\hat{\mu}_{n-1} = \frac{1}{n-1}\sum\limits_{i=1}^{n-1}x_i\quad\text{and}\quad\hat{\mu}_n = \frac{1}{n}\sum\limits_{i=1}^nx_i,\hat{\mu}_{n-1} = \frac{1}{n-1}\sum\limits_{i=1}^{n-1}x_i\quad\text{and}\quad\hat{\mu}_n = \frac{1}{n}\sum\limits_{i=1}^nx_i,$$
so
$$\hat{\mu}_n=\frac{1}{n}\left[(n-1)\hat{\mu}_{n-1} + x_n\right].\hat{\mu}_n=\frac{1}{n}\left[(n-1)\hat{\mu}_{n-1} + x_n\right].$$