sample size n

number of replicates


Marc Lavielle,
July 21, 2015

The Law of Large Numbers

We consider here an infinite sequence \(X_1\), \(X_2\), \(X_3\), …, of independent and identically distributed (i.i.d.) random variables, with mean \(m\) and finite variance \(\omega^2\): \[{\rm E}(X_i) = m \quad ; \quad {\rm Var}(X_i) = \omega^2 < \infty \ , \quad \text{for } i = 1, 2, \ldots\]

Then, the sample average \[\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i\] converges to \(m\) for \(n \to \infty\).

  • The weak law of large numbers (also called Khintchine's law) states that \(\bar{X}_n\) converges in probability towards the expected value \(m\). Indeed, \[ {\rm E}(\bar{X}_n) = m \quad ; \quad {\rm Var}(\bar{X}_n) = \frac{\omega^2}{n} \]

  • The strong law of large numbers states that \(\bar{X}_n\) converges almost surely to the expected value \(m\).

Some probability distributions

The objective of the application is to “visualize'' the convergence (in probability) of \(\bar{X}_n\) for different distributions of \(X_i\):

  • The normal distribution with mean \(\mu\) and variance \(\sigma^2\): \[X_i \sim {\cal N}(\mu , \sigma^2) \quad ; \quad {\rm E}(X_i) = \mu\]

  • The log-normal distribution with parameters \(\mu\) and \(\sigma^2\): \[\log(X_i) \sim {\cal N}(\mu , \sigma^2) \quad ; \quad {\rm E}(X_i) = e^{\mu + \sigma^2/2}\]

  • The \(\chi^2\) distribution with \(\nu\) degree of freedom: \[X_i \sim \chi^2(\nu) \quad ; \quad {\rm E}(X_i) = \nu\]

  • The uniform distribution on [0, b]: \[X_i \sim \text{Unif}([0,b]) \quad ; \quad {\rm E}(X_i) = b/2\]

  • The Poisson distribution with parameter \(\lambda\): \[X_i \sim {\cal P}(\lambda) \quad ; \quad {\rm E}(X_i) = \lambda\]

  • The binomial distribution with parameters \(k\) and \(p\): \[X_i \sim B(k,p) \quad ; \quad {\rm E}(X_i) = kp\]

Some tasks

  • Using the normal distribution, display one sequence \((\bar{X}_n, n\geq 1)\) and see how this sequence converges to \(m=\mu\) (increase the maximum value of \(n\)).
  • Display now \(N\) replicates \((\bar{X}_n^{(1)})\) , \((\bar{X}_n^{(2)})\),…, \((\bar{X}_n^{(N)})\), for \(N=2, 3, 5, 10, 100, 1000\). Comment about these graphs.
  • Display the empirical mean of the \(N\) replicates and \(+/-\) one standard deviation from the mean. How decreases the standard deviation of \(\bar{X}_n\)?
  • Repeat the same exercices with different distributions.