Sum of Independent Random Variables

Suppose two people assemble widgets in a factory, and they work independently. You know the average time it takes each person to assemble a widget. Can you find the average time it would take them to work together to assemble a widget if all you know is their separate average assembly time? In some cases, you can! The key question is whether or not your random variables are independent. So read on to learn about the sum of independent random variables!

Meaning of the sums of independent random variables

You have probably seen how to find the probability generating function of a random variable already. In fact, you have more than likely even looked at what happens when you add two of them together without even realising it!

Let's look at a quick example.

Finding the probability generating function for \(Z=X+Y\) wasn't too bad in the previous example because the individual probability generating functions weren't that complicated. But in a case with more complex probability generating functions this can get complicated very quickly!

In two special cases, there is a much quicker way to find the sum of two probability generating function.

• The first case is where you have two independent discrete random variables \(X\) and \(Y\) and you are asked to find \(Z\) where \(Z=X+Y\).

• The second case is where you are asked to find \(Z\) where \(Z\) is a linear function of discrete random variable \(X\) (i.e. \(Z=aX+b\)).

Let's look at each case.

Finding the probability generating function of \(Z=X+Y\)

There is a very important theorem that covers this case called the Convolution Theorem.

Convolution Theorem: Suppose two independent discrete random variables \(X\) and \(Y\) have probability generating functions \(G_X(t)\) and \(G_Y(t)\). The probability generating function of \(Z=X+Y\) is

\[G_Z(t)=G_X(t)G_Y(t).\]

Let's look at an application.

The main benefit of the Convolution Theorem is that it gives you a way to find the probability generating function without constructing a table, which leads to fewer opportunities for errors.

Finding the probability generating function of \(Z=aX+b\)

Let's take a quick look at how you could build the probability generating function of \(Z=nX\) where \(n\) is a natural number from the probability generating function of \(X\). Starting with \(n=2\),

\[ Z = 2X = X+X\]

so you can use the Convolution Theorem to get that

\[G_Z(t)=G_X(t)G_X(t) = (G_X(t))^2.\]

Then you could use proof by induction to show that for any natural number \(n\), if \(Z=nX\) then

\[G_Z(t)=\underbrace{G_X(t)G_X(t)\cdots G_X(t)}_{n \text{ times}} = (G_X(t))^n.\]

As you know, there are more than just the natural numbers out there, and you still need to account for that '\(+b\)' too. So it helps to look at the alternate definition of the probability generating function:

\[G_Z(t) = \text{E}(t^Z).\]

Then you can use properties of the expected value function to get the following:

\[\begin{align} G_Z(t) &= \text{E}(t^Z) \\ &= \text{E}(t^{aX+b}) \\ &= \text{E}(t^{aX}t^b) \\ &= t^b\text{E}(t^{aX}) \\ &= t^b\text{E}\left((t^a)^X\right) \\ &= t^bG_X(t^a) . \end{align}\]

While this property doesn't have a fancy name, it is worth stating separately.

If \(X\) is a discrete random variable and has a probability generating function of \(G_X(t)\), the probability generating function of \(Z\) where \(Z=aX+b\) is:

\[G_Z(t)=t^bG_X(t^a).\]

Let's take a look at a quick example.

Expectation of the sum of independent random variables

Just like all random variables, sums of independent random variables also have an expectation or mean. You can use the Convolution Theorem and the alternate definition of the probability generating function to find the expectation of the sum of independent random variables, along with the formulas:

• \(G'_X(1) = E(X)\);

• \(\text{E}(aX+b) = a\text{E}(X) + b\); and

• \(\text{E}(X+Y) = \text{E}(X) + \text{E}(Y)\).

Let's look at an example.

Variance of the sum of independent random variables

Just as you have found the mean above, you can also find the variance of sums of independent random variables. To do this you will need the formulas:

• \(\text{Var}(aX+b) = a^2\text{Var}(X)\); and
• \(\text{Var}(Z)= G''_Z(1)+G'_Z(1)-(G'_Z(1))^2\).

Let's look at an example.

Sum of independent random variables examples

You have already seen some examples of funding the sum of independent random variables, along with their mean and variance. However for specific kinds of distributions, like the binomial distribution and the uniform distribution, looking at them, in particular, can be illuminating. So keep going for the specifics!

Sum of independent binomial random variables

Suppose you have two independent random variables that follow binomial distributions. In other words, \(X \sim \text{Bin}(n_X, p_X)\) and \(Y \sim \text{Bin}(n_Y, p_Y)\). You already know that

\[G_X(t) = (1-p_X+p_Xt)^{n_X}\]

and

\[G_Y(t) = (1-p_Y+p_Yt)^{n_Y}.\]

So using the Convolution Theorem, if \(Z = X+Y\) then

\[ \begin{align} G_Z(t) &= G_X(t) G_Y(t) \\ &= (1-p_X+p_Xt)^{n_X} (1-p_Y+p_Yt)^{n_Y}. \end{align}\]

Let's take an example.

Sum of independent uniform random variables

Remember that a discrete uniform random variable takes on equal probabilities for each possible outcome. So if distribution \(X\) has events occurring with probability \(\dfrac{1}{n}\), then

\[ G_X(t) = \frac{t(1-t^n)}{n(1-t)}.\]

So if \(Y\) is a second discrete uniform random distribution that has events occurring with probability \(\dfrac{1}{m}\), and \(Z = X+Y\), then

\[ \begin{align} G_Z(t) &= G_X(t) G_Y(t) \\ &= \left(\frac{t(1-t^n)}{n(1-t)} \right)\left(\frac{t(1-t^m)}{m(1-t)}\right) \\ &= \frac{t^2(1-t^n)(1-t)^m}{nm(1-t)^2} .\end{align}\]

Let's take an example.

It actually isn't necessary to memorise the different formulas for the various kinds of discrete probability distributions as long as you keep in mind the Convolution Theorem!