What Does 94.5% Effective Mean?

November 2020

Today, Moderna announced results of their Covid-19 vaccine trial. This comes a week after Pfizer announced results for their own vaccine.

Headlines celebrate 94.5% effective or 90% effective! But what does that actually mean? And how is it calculated?

Moderna’s release states:

95 cases, of which 90 cases of COVID-19 were observed in the placebo group versus 5 cases observed in the mRNA-1273 group, resulting in a point estimate of vaccine efficacy of 94.5%

Let’s see, 90 of the 95 cases were from the placebo group. And 90 / 95 = 94.7%, which is pretty close. Is that how effectiveness is calculated? Reading the news coverage, it’s a safe bet that many think so.

Notation

Let’s start by defining some classes of patients:

: Given the vaccine
: Exposed to the virus (sufficiently to contract it when unvaccinated)
: Contracted the virus

We’ll use $\neg$ for the inverse. $\#(...)$ means count, E[...] means expected value, and Pr(...) means probability across all patients in the study. So, for example, $Pr(C|\neg V)$ means: the probability that a patient caught the virus, given that they did not receive the vaccine.

Given

Based on the news release, we can state our known facts:

$\#(C,X, V) = 5$ The number of vaccine patients who were exposed to the virus and contracted it was .
$\#(C,X, \neg V) = 90$ The number of placebo patients who were exposed to the virus and contracted it was .
$\#(V) = \#(\neg V)$ Assume that the vaccine and placebo groups are the same size.
$Pr(X | V) = Pr(X | \neg V)$ The chance of being exposed is the same in the vaccine and placebo groups.
$Pr(C | X, \neg V) = 1$ By our definition of "exposure", if you were exposed to the virus without the vaccine, there is a 100% chance you contracted it.

Question

The effectiveness of a vaccine is intuitively defined as:

What are the chances that you will not contract the virus when exposed to it if you were given the vaccine?

Writing the definition in our notation:

$Pr(\neg C | X,V)?$

Derivation

Let’s start by inverting the probability:

$Pr(\neg C | X,V) = 1 - Pr(C | X,V)$

Using raw counts for the conditional probability:

$Pr(\neg C|X,V) = 1 - \#(C,X,V)/\#(X,V)$

Substituting (1) into (7) :

$Pr(\neg C|X,V) = 1 - 5/\#(X,V)$

From (3) and (4) , we can derive that the same number of patients were expected to be exposed in the vaccine and placebo groups:

$\#(X,V) = E[X, V] = E[X, \neg V] = \#(X, \neg V)$

From (2) and (5) , all exposed patients in the placebo group contracted the virus:

$\#(X, \neg V) = \#(C, X, \neg V) = 90$

Finally, from (8,9,10) , we get:

$Pr(\neg C|X,V) = 1 - 5/90$

In summary, "94.5% effective" does not come from —it comes from .

Bonus Observations

Notice that the effectiveness does not depend on group size. As long as we assume that the vaccine and placebo groups are the same size, the terms cancel.
Notice that we had to reason about how many from the vaccine group were exposed to the virus. Even though we have no direct way to measure this value, we are able to arrive at the answer anyway using the other measurements and assumptions.
Notice that if 90 people from the vaccine group had contracted the virus, the vaccine would be 0% effective. That makes intuitive sense.
Notice that if 91+ people from the vaccine group had contracted the virus, the effectiveness would be negative. A negative probability?! Where did we go wrong? I’ll leave this as an exercise to the reader.