The Karma of Kaplan-Meier

Our new book, Modelling Mortality with Actuarial Applications, describes several non-parametric estimators of two quantities:

1. The survival function, \(S_x(t)\), defined as the probability that a person now aged \(x\) will survive at least \(t\) years (\({}_tp_x\) to actuaries), and
2. The integrated hazard function, \(\Lambda_x(t) = \displaystyle\int_0^t\mu_{x+s}ds\).

The estimators of the above quantities are based on two items of data collected at the times of the observed deaths (denoted by \(t_1,t_2,\ldots,t_n\)):

1. The number, \(d_{x+t_i}\), who died at time \(t_i\), and
2. The number, \(l_{x+t_i^-}\), who were alive and under observation immediately before time \(t_i\) (which time we denote by \(t_i^-\)).

Then \(d_{x+t_i}/l_{x+t_i^-}\) is an empirical estimate of the probability of dying at time \(t_i\), conditional on being alive just before time \(t_i\). This contributes to an estimator of the integrated hazard function and two estimators of \(S_x(t)\):

• The Nelson-Aalen estimator of \(\Lambda_x(t)\) is:

\[\hat\Lambda_x(t)=\sum_{t_i\leq t}\frac{d_{x+t_i}}{l_{x+t_i^-}}.\qquad(1)\]

• The Fleming-Harrington estimator of \(S_x(t)\) is:

\[\hat S_x(t)=\exp\left(-\hat\Lambda_x(t)\right).\qquad(2)\]

• The Kaplan-Meier estimator of \(S_x(t)\) is:

\[\hat S_x(t)=\prod_{t_i\leq t}\left(1-\frac{d_{x+t_i}}{l_{x+t_i^-}}\right).\qquad(3)\]

It is clear how the Fleming-Harrington estimator is related to the Nelson-Aalen estimator, but where does Kaplan-Meier fit in? It would be nice to know, not least because the Kaplan-Meier estimator is by far the most commonly used non-parametric estimator in survival modelling (we demonstrate its use with actuarial data in our book).

In my previous blog we found a new representation of the survival function, \(S_x(t)\), as a product integral, which is the middle expression below:

\[S_x(t) = \prod_0^t\left(1-\mu_{x+s}ds\right) = \exp\left(-\int_0^t\mu_{x+s}ds\right).\qquad(4)\]

Essentially, this expresses the multiplicative property of the survival probabilities in the limit as \(ds\to0\).

We often specify a survival model by its hazard rate, \(\mu_{x+s}\), and then expression (4) defines the distribution \(F_x(t)=1-S_x(t)\) of the random future lifetime, \(T_x\), of a person aged \(x\). We are accustomed to using 'nice' hazard rates, such as parametric functions or constant hazard rates over single years of age. This 'niceness' confers a corresponding 'niceness' on \(F_x(t)\), namely that it has no 'lumps' of probability. That is, there is no age \(x+s\) such that \(\Pr[T_x=s]>0\).

However, random variables that take a discrete set of values, such as the binomial or Poisson, are not 'nice' in the sense above — their distribution functions \(F(t)\) add up discrete lumps of probability concentrated on a set of points in the interval \([0,t]\). We ask, what happens to expression (4) when 'niceness' fails in this way?

The answer is that the product integral is fine, but the exponential formula ceases to be valid. If a discrete random variable \(T\) has probability mass \(q_i\) at point \(t_i (t_i=t_1,t_2,\ldots)\) then the product integral becomes the discrete product:

\[\prod_{t_i\leq t}\left(1-\frac{q_i}{1-F(t_i^-)}\right)\qquad(5)\]

and this still is equal to the survival function of \(T\). The fractional term in expression (5) is the conditional probability \(\Pr[T=t_i|T\geq t_i]\), which is the discrete equivalent of the hazard rate — exactly what is estimated empirically by the ratios \(d_{x+t_i}/l_{x+t_i^-}\) in our data.

Does expression (5) look familiar? All can now be explained:

• Expression (4) relates the survival functions and hazard rate in two ways, and is valid for 'nice' hazard rates.
• The observed death probabilities \(d_{x+t_i}/l_{x+t_i^-}\) are empirical estimates of the discretized hazard rates at times \(t_i\).
• The Nelson-Aalen estimator is therefore an empirical estimate of the integrated hazard function.
• The Kaplan-Meier estimator is the product integral of (minus) the Nelson-Aalen estimator (compare expressions (3) and (5)).
• The Fleming-Harrington estimator is the exponential of (minus) the Nelson-Aalen estimator.

Because the increments of the Nelson-Aalen estimator are all jumps, the exponential equality in expression (4) does not hold. Therefore, the Kaplan-Meier estimator is not the same as the Fleming-Harrington estimator (although, as we note in our book, the difference is very small indeed for the large data sets typically used by actuaries). Expression (2) may give the impression that the Fleming-Harrington estimator is the more closely related to the Nelson-Aalen estimator, but it is not. In fact, it is the product integral in expression (4) that is the more fundamental construction; the exponential formula is just nice to have (if it is valid).