Bias

The bias of an estimate is the difference between the expectation value of the point estimate and value of the parameter.

$$\begin{array}{r}{\text{bias}}_F(\hat{\theta },\theta )=⟨\hat{\theta }⟩-\theta =\int \mathrm{d}x\hat{\theta }f(x)-T(F).\end{array}$$

Note that the expectation value of $\hat{\theta }$ is computed over the (unknown) generative distribution whose PDF is $f(x)$.

Bias of the plug-in estimate for the mean

We often want a small bias because we want to choose estimates that give us back the parameters we expect. Let’s first investigate the bias of the plug-in estimate of the mean. As a reminder, the plug-in estimate is

$$\begin{array}{r}\hat{\mu }=\overline{x},\end{array}$$

where $\overline{x}$ is the arithmetic mean of the observed data. To compute the bias of the plug-in estimate, we need to compute $⟨\hat{\mu }⟩$ and compare it to $\mu$.

$$\begin{array}{r}⟨\hat{\mu }⟩=⟨\overline{x}⟩=\frac{1}{n}⟨\sum _i{x}_i⟩=\frac{1}{n}\sum _i⟨x_i⟩=⟨x⟩=\mu .\end{array}$$

Because $⟨\hat{\mu }⟩=\mu$, the bias in the plug-in estimate for the mean is zero. It is said to be unbiased.

Bias of the plug-in estimate for the variance

To compute the bias of the plug-in estimate for the variance, first recall that the variance, as the second central moment, is computed as

$$\begin{array}{r}{\sigma }^2=⟨x^2⟩-⟨x{⟩}^2.\end{array}$$

So, the expectation value of the plug-in estimate is

$$\begin{array}{r}\begin{array}{rl}⟨{\hat{\sigma }}^2⟩& =⟨\frac{1}{n}\sum _i{x}_i^2-{\overline{x}}^2⟩\\ & =⟨\frac{1}{n}\sum _i{x}_i^2⟩-⟨{\overline{x}}^2⟩\\ & =\frac{1}{n}\sum _i⟨x_i^2⟩-⟨{\overline{x}}^2⟩\\ & =⟨x^2⟩-⟨{\overline{x}}^2⟩\\ & ={\mu }^2+{\sigma }^2-⟨{\overline{x}}^2⟩.\end{array}\end{array}$$

We now need to compute $⟨{\overline{x}}^2⟩$, which is a little trickier. We will use the fact that the measurements are independent, so $⟨x_i{x}_j⟩=⟨x_i⟩⟨x_j⟩$ for $i\ne j$.

$$\begin{array}{r}\begin{array}{rl}⟨{\overline{x}}^2⟩& =⟨{\left(\frac{1}{n}\sum _i{x}_i\right)}^2⟩\\ & =\frac{1}{n^2}⟨{\left(\sum _i{x}_i\right)}^2⟩\\ & =\frac{1}{n^2}⟨\sum _i{x}_i^2+2\sum _i\sum _{j>i}{x}_i{x}_j⟩\\ & =\frac{1}{n^2}(\sum _i⟨x_i^2⟩+2\sum _i\sum _{j>i}⟨x_i{x}_j⟩)\\ & =\frac{1}{n^2}(n({\sigma }^2+{\mu }^2)+2\sum _i\sum _{j>i}⟨x_i⟩⟨x_j⟩)\\ & =\frac{1}{n^2}(n({\sigma }^2+{\mu }^2)+n(n-1)⟨x{⟩}^2)\\ & =\frac{1}{n^2}(n{\sigma }^2+n^2{\mu }^2)\\ & =\frac{{\sigma }^2}{n}+{\mu }^2.\end{array}\end{array}$$

Thus, we have

$$\begin{array}{r}⟨{\hat{\sigma }}^2⟩=(1-\frac{1}{n}){\sigma }^2.\end{array}$$

Therefore, the bias is

$$\begin{array}{r}\text{bias}=-\frac{{\sigma }^2}{n}.\end{array}$$

If ${\hat{\sigma }}^2$ is the plug-in estimate for the variance, an unbiased estimator would instead be

$$\begin{array}{r}\frac{n}{n-1}{\hat{\sigma }}^2=\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x}{)}^2.\end{array}$$

Justification of using plug-in estimates.

Despite the apparent bias in the plug-in estimate for the variance, we will normally just use plug-in estimates going forward. (We will use the hat, e.g. $\hat{\theta }$, to denote an estimate, which can be either a plug-in estimate or not.) Note that the bootstrap procedures we lay out in what follows do not need to use plug-in estimates, but we will use them for convenience. Why do this? The bias is typically small. We just saw that the biased and unbiased estimators of the variance differ by a factor of $n/(n-1)$, which is negligible for large $n$. In fact, plug-in estimates tend to have much smaller error than the confidence intervals for the parameter estimate, which we will discuss next.