The variance of a mixture

Suppose $X$ is a mixture distribution that is the result of mixing a family of conditional distributions indexed by a parameter random variable $\Theta$. The uncertainty in the parameter variable $\Theta$ has the effect of increasing the unconditional variance of the mixture $X$. Thus, $Var(X)$ is not simply the weighted average of the conditional variance $Var(X \lvert \Theta)$. The unconditional variance $Var(X)$ is the sum of two components. They are: $\displaystyle Var(X)=E[Var(X \lvert \Theta)]+Var[E(X \lvert \Theta)]$

The above relationship is called the law of total variance, which is the proper way of computing the unconditional variance $Var(X)$. The first component $E[Var(X \lvert \Theta)]$ is called the expected value of conditional variances, which is the weighted average of the conditional variances. The second component $Var[E(X \lvert \Theta)]$ is called the variance of the conditional means, which represents the additional variance as a result of the uncertainty in the parameter $\Theta$.

We use an example of a two-point mixture to illustrate the law of total variance. The example is followed by a proof of the total law of variance.

Example
Let $U$ be the uniform distribution on the unit interval $(0, 1)$. Suppose that a large population of insureds is composed of “high risk” and “low risk” individuals. The proportion of insured classified as “low risk” is $p$ where $0. The random loss amount $X$ of a “low risk” insured is $U$. The random loss amount $X$ of a “high risk” insured is $U$ shifted by a positive constant $w>0$, i.e. $w+U$. What is the variance of the loss amount of an insured randomly selected from this population?

For convenience, we use $\Theta$ as a parameter to indicate the risk class ( $\Theta=1$ is “low risk” and $\Theta=2$ is “high risk”). The following shows the relevant conditional distributional quantities of $X$. $\displaystyle E(X \lvert \Theta=1)=\frac{1}{2} \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ E(X \lvert \Theta=2)=w+\frac{1}{2}$ $\displaystyle Var(X \lvert \Theta=1)=\frac{1}{12} \ \ \ \ \ \ \ \ \ \ \ \ Var(X \lvert \Theta=2)=\frac{1}{12}$

The unconditional mean loss is the weighted average of the conditional mean loss amounts. However, the same idea does not work for variance. \displaystyle \begin{aligned}E[X]&=p \times E(X \lvert \Theta=1)+(1-p) \times E(X \lvert \Theta=2) \\&=p \times \frac{1}{2}+(1-p) \times (w+\frac{1}{2}) \\&=\frac{1}{2}+(1-p) \ w \end{aligned} \displaystyle \begin{aligned}Var[X]&\ne p \times Var(X \lvert \Theta=1)+(1-p) \times Var(X \lvert \Theta=2)=\frac{1}{12} \end{aligned}

The conditional variance is the same for both risk classes since the “high risk” loss is a shifted distribution of the “low risk” loss. However, the unconditional variance is more than $\frac{1}{12}$ since the mean loss for the two casses are different (heterogeneous risks across the classes). The uncertainty in the risk classes (i.e. uncertainty in the parameter $\Theta$) introduces additional variance in the loss for a randomly selected insured. The unconditional variance $Var(X)$ is the sum of the following two components: \displaystyle \begin{aligned}E[Var(X \lvert \Theta)]&=p \times Var(X \lvert \Theta=1)+(1-p) \times Var(X \lvert \Theta=2) \\&=p \times \frac{1}{12}+(1-p) \times \frac{1}{12} \\&=\frac{1}{12} \end{aligned} \displaystyle \begin{aligned}Var[E(X \lvert \Theta)]&=p \times E(X \lvert \Theta=1)^2+(1-p) \times E(X \lvert \Theta=2)^2 \\&\ \ \ -\biggl(p \times E(X \lvert \Theta=1)+(1-p) \times E(X \lvert \Theta=2)\biggr)^2 \\&=p \times \biggl(\frac{1}{2}\biggr)^2+(1-p) \times \biggl(w+\frac{1}{2}\biggr)^2 \\&\ \ \ -\biggl(p \times \frac{1}{2}+(1-p) \times (w+\frac{1}{2})\biggr)^2 \\&=p \ (1-p) \ w^2 \end{aligned} \displaystyle \begin{aligned}Var(X)&=E[Var(X \lvert \Theta)]+Var[E(X \lvert \Theta)] \\&=\frac{1}{12}+p \ (1-p) \ w^2 \end{aligned}

The additional variance is in the amount of $p(1-p)w^2$. This is the variance of the conditional means of the risk classes. Note that $w$ is the additional mean loss for a “high risk” insured. The higher the additional mean loss $w$, the more heterogeneous in risk between the two classes, hence the larger the dispersion in unconditional loss.

The total law of variance gives the unconditional variance of a random variable $X$ that is indexed by another random variable $\Theta$. The unconditional variance of $X$ is the sum of two components, namely, the expected value of conditional variances and the variance of the conditional means. The formula is: $\displaystyle Var(X)=E[Var(X \lvert \Theta)]+Var[E(X \lvert \Theta)]$

The following is the derivation of the formula: \displaystyle \begin{aligned}Var[X]&=E[X^2]-E[X]^2 \\&=E[E(X^2 \lvert \Theta)]-E[E(X \lvert \Theta)]^2 \\&=E\left\{Var(X \lvert \Theta)+E(X \lvert \Theta)^2 \right\}-E[E(X \lvert \Theta)]^2 \\&=E[Var(X \lvert \Theta)]+E[E(X \lvert \Theta)^2]-E[E(X \lvert \Theta)]^2 \\&=E[Var(X \lvert \Theta)]+Var[E(X \lvert \Theta)] \end{aligned}