\input{head2e}
\newcommand{\Hist}{\mbox{${\cal H}$}}
\newcommand{\Fute}{\mbox{${\cal F}$}}
\newcommand{\tran}{\mbox{\scriptsize \sf T}}
\begin{document}
\begin{centre}
{\Large \bf Notes on Hidden Markov Models}
\end{centre}
\section{Introduction}
\label{sec:intro}
In its simplest form, a hidden Markov model consists of
\begin{alist}
\item a Markov chain of \ub{unobserved} states, and
\item a corresponding sequence of observations whose distribution
depends, amongs other things, upon the associated underlying state.
\end{alist}

Denote the states by $S_1, S_2, \ldots, S_T$.  These states take
values in a ``state space'' which may usually be assumed to be
the finite set $\{1, 2, \ldots, K\}$.

The chain will have a \ub{transition probability} matrix $P =
[p_{ij}]$ where \newline $p_{ij} = P(S_t = j | S_{t-1} = i)$.  You
are reminded that the Markovian assumption is that this conditional
probability is ``all you need to know'', i.e. that
\[
P(S_t = j | S_1 = s_1, \ldots, S_{t-1} = s_{t-1}) =
                               P(S_t = j | S_{t-1} = s_{t-1})
\]

This may be expressed as ``It only matters where you \ub{are}, and
not how you got there.''

A good example of a Markovian structure may be found in genetics; the
genetic makeup of an offspring depends only upon the genes of its
parents.  Preceding generations are irrelevant.

The chain will also have an \ub{initial state probability
distribution} $\pi = (\pi_1, \ldots, \pi_K)$ where $\pi_k = P(S_1 =
k)$.  It rarely makes sense to proceed in any manner other than
assuming that the initial state distribution is equal to the steady
state or equilibrium distribution of the chain.  The steady state
distribution is roughly the long-term unconditional distribution of
the states at any time $t$.  A Markov chain will have a steady state
distribution provided that it is ``irriducible, positive recurrent
and aperiodic''.  These technicalities will not be emphasized in
these notes.

If $\pi$ is the steady state distribution then $\pi$ is the (unique)
stationary distribution and is hence determined by
\[
P^{\tran} \pi = \pi \mbox{ .}
\]

Denote the sequence of observable random variables by $Y_1, Y_2,
\ldots, Y_T$.

We assume that the $Y_t$ are conditionally independent, \ub{given}
the underlying states.

We assume that if $S_t = k$, then $Y_t$ has distribution given by
\[
Y_t \sim f_k(y|\phi)
\]
where each $f_k$ is a probability (density) function.  The functions
$f_k$ will in general depend upon a set of parameters $\phi$ which
will usually have to be estimated.

I refer to $\pi$ and $P$ as the {\em Markovian parameters}, and to
$\phi$ as the {\em model parameters}.

E.g. $K=2$; suppose that the observations are Poisson variates
with means $\lambda_k$, $k = 1, 2$.  Then
\[
f_k(y|\phi) = \frac{e^{-\lambda_k} \lambda_k^y}{y!}
\]
and the vector of parameters $\phi$ is $\phi =
[\lambda_1, \lambda_2]^{\tran}$.

\section{Parameter Estimation}

Consider a sequence of observations $Y_1=y_1, \ldots, Y_T=y_t$.  We
wish to estimate the parameter set $\theta$ consisting of $P$
together with $\phi$.  By the steady state assumption the parameters
$\pi$ are completely determined by $P$.

Estimation is effected by maximum likelihood.

\ub{Given} a state sequence $S$, equal to $S_1 = s_1, \ldots, S_T =
s_T$, the probability of the observations is, because of conditional
independence, equal to
\[
P(Y=y|S=s) = \prod_{t=1}^T f_{s_t}(y_t|\phi) \mbox{ .}
\]
The probability of the state sequence is
\[
P(S=s) = \pi_{s_1} \times \prod_{t=1}^{T-1} p_{s_t s_{t+1}} \mbox{ .}
\]
The joint likelihood is therefore
\begin{eqnarray*}
L(Y,S,\theta) & = & P(Y=y|S=s) \times P(S=s)\\
              & = & P(S=s) \times P(Y=y|S=s)\\
              & = & \pi_{s_1}  \times \prod_{t=1}^{T-1} p_{s_t s_{t+1}}
                \times \prod_{t=1}^T f_{s_t}(y_t|\phi)
\end{eqnarray*}

The joint log likelihood is hence (obviously) equal to
\[
\ln L(Y,S,\theta) = \ln \pi_{s_1} + \sum_{t=1}^{T-1} \ln p_{s_t s_{t+1}}
                  + \sum_{t=1}^T \ln f_{s_t}(y_t|\phi)
\]

To get the marginal likelihood of the actual observations we have to
sum the joint likelihood over all possible (length~$T$) state
sequences:
\begin{eqnarray*}
L(Y,\theta) & = & \sum_s L(Y,s,\theta) \\
            & = & \sum_s \left[ \pi_{s_1} \times
                  \prod_{t=1}^{T-1} p_{s_t s_{t+1}}
                  \times \prod_{t=1}^T f_{s_t}(y_t|\phi) \right] \mbox{ .}
\end{eqnarray*}

There are $K^T$ terms in this sum; e.g. if $K$ is 2 and $T = 100$
then $K^T = 2^{100} \approx 1.27 \times 10^{30}$.

Thus, using a direct approach, it is hard (impossible, actually) even
to evaluate the likelihood, let alone maximize it.  We \ub{can}
evaluate it however, by constructing certain sequences of recursively
defined probabilities and using these sequences, it is even possible
to \ub{maximize} the (log) likelihood, using the EM
(expectation/maximization) algorithm.

\section{A Trivial Example}

\label{sec:triv.ex}
Before going on to describe how the EM algorithm applies to hidden
Markov models, we shall discuss briefly an example with a very
small $T$, equal to 3, whereby the likelihood may be calculated
explicitly.  This example is taken from the paper ``A Hidden Markov
Model for Space-Time Precipitation'' by Walter Zucchini and Peter
Guttorp, {\em Water Resources Research} {\bf 27}, pp. 1917 -- 1923.

In this example the state space of the underlying Markov chain is
$\{1,2,3\}$ and the $Y_t$ are discrete variables taking values in
$\{0,1,2,3\}$ with probability functions given by the following table:

\begin{centre}
\begin{tabular}{c||c|c|c}
 & \multicolumn{3}{|c}{State} \\ \hline
    $Y_t$ & 1 & 2         & 3 \\ \hline \hline
    0 & 1 & $(1-p)^2$ & 0 \\ \hline
    1 & 0 & $p(1-p)$  & 0 \\ \hline
    2 & 0 & $p(1-p)$  & 0 \\ \hline
    3 & 0 & $p^2$     & 1
\end{tabular}
\end{centre}

The transition probability matrix is
\[
P = \frac{1}{3} \left[ \begin{array}{ccc}
			1 & 1 & 1 \\
			1 & 1 & 1 \\
			1 & 1 & 1 \end{array} \right]
\]
and so ``clearly'' the steady state distribution ($\equiv$ the initial
distribution) is $\pi = [1/3, 1/3, 1/3]^{\tran}$.

We observe $Y_1 = 1, Y_2 = 0, Y_3 = 3$.

\ub{Exercise 1:} Show that the likelihood is equal to
\[
\frac{1}{27} p(1-p)(1+p^2)(1 + (1-p)^2) \mbox{ .}
\]

(Hint:  There are $K^T = 3^3 = 27$ summands in the likelihood; however
all but four of these are 0.)

\section{The EM Algorithm}

The EM Algorithm is also called the Baum-Welch Algorithm in the
hidden Markov chain context.  It involves recursively maximizing
\[
Q(\theta, \theta') = \E_S [ \ln L(Y,S,\theta) \; | \; Y, \theta']
\]
with respect to $\theta$.  (That is we find $\theta^{(m+1)}$ by
maximizing $Q(\theta,\theta^{(m)})$ with respect to $\theta$).  Note
that in the formation of $Q(\theta, \theta')$, $L(Y,S,\theta)$ is
calculated (for each $S$) using the parameter set $\theta$, then the
expectation (with respect to $S$, given $Y$) is calculated using
probabilities formed from the parameter values $\theta'$.  The
expectation can be written down explicitly, in terms of the
recursively defined sequences of probabilities.

In fact $Q(\theta, \theta') = \E_S [ \ln L(Y,S,\theta) \; | \; Y, \theta']$
is given by
\begin{eqnarray*}
Q(\theta, \theta') = \sum_{i=1}^K \gamma_1(i) \ln(\pi_i) & + &
\sum_{t=1}^{T-1} \left[ \sum_{i=1}^K \sum_{j=1}^K \xi_t(i,j)
   \ln(p_{ij}) \right] \\
   & + & \sum_{t=1}^T \sum_{i=1}^K \gamma_t(i)
         \ln(f_i(y_t\;|\; \phi))
\end{eqnarray*}
where
\begin{list}{}{}
\item $\gamma_t(i) =$ the probability (given the observations) that
the system is in state $i$ at time $t$, and
\item $\xi_t(i,j) =$ the probability (given the observations) that
the system is in state $i$ at time $t$ and in state $j$ at time $t+1$
\end{list}
In the foregoing expression for $Q(\theta, \theta')$, $\theta'$
is buried in the $\gamma_t(i)$'s and the $\xi_t(i,j)$'s.

Note that the different parts of $\theta$ are involved in separate
components of the foregoing expression and therefore these parts can
be handled separately when we are maximizing with respect to
$\theta$.

\section{The Recursive Probabilities}
The crucial step is to calculate the $\gamma_t$ and $\xi_t$.  Define
\begin{alist}
\item $\Hist_t$ = the ``history'' of the process up to time $t$,
i.e. $\Hist_t = \{Y_1, Y_2, \ldots Y_t\}$.
\item $\Fute_t$ = the ``future'' of the process from time $t$ onward,
i.e. $\Fute_t = \{Y_t, Y_{t+1}, \ldots Y_T\}$.
\item The ``forward'' probabilities:  $\alpha_t(j) = P(\Hist_t, S_t = j)$.
\item The ``backward'' probabilities:  $\beta_t(i) = P(\Fute_{t+1} \;
| \; S_t=i)$\ .  Arbitrarily set $\beta_T(i) = 1$.
\end{alist}

\ub{Exercise 2:} Show that
\begin{enumerate}
\item \mbox{ } \vspace*{-1.0\baselineskip}
\begin{eqnarray*}
\alpha_1(j) & = & \pi_j \times f_j(Y_1|\phi) \mbox{\hspace*{1cm} and}\\
\alpha_t(j) & = & f_j(Y_t|\phi) \sum_{i=1}^K \alpha_{t-1}(i) p_{ij}
             \mbox{\ \ \ for \ \ } t = 2, \ldots, T \mbox{ .}
\end{eqnarray*}

\item \mbox{ } \vspace*{-1.0\baselineskip}
\[
\beta_t(i) = \sum_{j=1}^K f_j(Y_{t+1}|\phi) \beta_{t+1}(j) p_{ij}
             \mbox{\ \ \ for \ \ } t = 1, 2, \ldots, T-1 \mbox{ .}
\]

\item \mbox{ } \vspace*{-1.0\baselineskip}
\[ \gamma_t(i) =
\frac{\alpha_t(i) \beta_t(i)}{\sum_{j=1}^K \alpha_t(j)
					    \beta_t(j)}
	      \mbox{\ \ \ for \ \ } t = 1, 2, \ldots, T \mbox{ .}
\]

(Note that $\gamma_t(i) = P(S_t=i \; | \; \Hist_T)$.)\\

\item \mbox{ } \vspace*{-1.0\baselineskip}
\[
\xi_t(i,j) = \frac{\alpha_t(i) p_{ij} f_j(Y_{t+1}|\phi) \beta_{t+1}(j)}
                      {\sum_{k=1}^K \sum_{\ell=1}^K \alpha_t(k) p_{k\ell}
                       f_{\ell}(Y_{t+1}) \beta_{t+1}(\ell)}
                       \mbox{\ \ \ for \ \ } t = 1, 2, \ldots, T-1 \mbox{ .}
\]

(Note that $\xi_t(i,j) = P(S_t = i \;\; \& \;\; S_{t+1} = j \;
| \;  \Hist_T)$.)\\
\end{enumerate}

Remark:  Note that the likelihood of the observations is simply
equal to
\[
\sum_{i=1}^K \alpha_T(i)
\]
Hence once we have calculated the ``forward probabilities'' we have
succeeded in calculating the likelihood.  Thus, having an effective
way to calculate the likelihood, we could seek to maximize it by
means of some general-purpose numerical procedure maximization
procedure (one which does not depend on derivatives being
available).  Nevertheless the EM algorithm remains the ``method of
choice'' for maximizing the likelihood in the context of hidden
Markov models, and in most contexts involving missing data.

\section{The M-step for the Markovian Parameters}

Maximizing $Q(\theta,\theta')$ with respect to the Markovian parameter
component of $\theta$ is relatively straightforward.  We must maximize
\[
\sum_{i=1}^K \gamma_1(i) \ln(\pi_i) +
\sum_{t=1}^{T-1} \left[ \sum_{i=1}^K \sum_{j=1}^K \xi_t(i,j)
   \ln(p_{ij}) \right]
\]
with respect to the $p_{ij}$.  Note/recall that the rows of the
transition probability matrix $P$ sum to 1, so we must maximize
subject to the constraints $p_{i\cdot} = 1$, $i = 1, \ldots, K$,
where the ``dot'' indicates \ub{summation} with respect
to the index which it replaces.

We must also (in theory) maximize subject to the constraints
that $P^{\tran} \pi = \pi$ (and $\pi_{\cdot} = 1$).  However
this leads to a surprizingly tricky maximization problem.  Moreover
the term
\[
\sum_{i=1}^K \gamma_1(i) \ln(\pi_i)
\]
is ``asymptotically negligible'' as $T \rightarrow \infty$.  Therefore
the practice is to ignore this term and obviate the need for the
constraints relating $P$ to $\pi$.

\ub{Exercise 3:} Show that the maximum of
\[
\sum_{t=1}^{T-1} \left[ \sum_{i=1}^K \sum_{j=1}^K \xi_t(i,j)
   \ln(p_{ij}) \right]
\]
subject to $p_{i\cdot} = 1$, $i = 1, \ldots, K$, is given by
\[
\hat{p}_{ij} = \frac{\omega_{ij}} {\sum_{\ell=1}^K \omega_{i\ell}}
= \frac{\omega_{ij}}{\omega_{i\cdot}}
\]
where $\omega_{ij} = \sum_{t=1}^{T-1} \xi_t(i,j)$.

\section{The M-step for the Model Parameters}

Here the nature of the computations depends upon the structure
of the functions $f_k(y|\phi)$ and may require further iterative techniques.

In the case of the simple example given in section \ref{sec:intro}
the maximizing values (i.e. the updates of the estimates of the $\lambda_k$)
can be written down explicitly.

\ub{Exercise 4:} Suppose that $K=2$; and that the observations are
Poisson variates with means $\lambda_k$, $k = 1, 2$, i.e. that
\[
f_k(y|\phi) = \frac{e^{-\lambda_k} \lambda_k^y}{y!} \mbox{ .}
\]

Show that the maximum of
\[
\sum_{t=1}^T \sum_{i=1}^K \gamma_t(i) \ln(f_i(y_t\;|\; \phi))
\]
with respect to $\lambda_k$, $k=1,2$, is given by
\[
\hat{\lambda}_k = \sum_{t=1}^T y_t \gamma_t(k) \; / \;
                  \sum_{t=1}^T \gamma_t(k) \mbox{ .}
\]

\ub{Exercise 5:} Consider a generalization of the structure described
in section \ref{sec:triv.ex} in which there is some arbitrary number
$K$ of states (rather than 3), $T$ observations $Y_t$, the $Y_t$ take
values in $\{0,1,2,\ldots,L-1\}$, and the probability functions of
the $Y_t$ are given by the $L \times K$ matrix $R = [\rho_{ij}]$;
i.e. $\rho_{ij} = P(Y_t = i | S_t = j)$, $i=0,1, \ldots, L-1$,
$j=1,2, \ldots, K$.  Note that the columns of $R$ sum to 1.

Show that $Q(\theta,\theta')$ is maximized with respect to $R$ by
\[
\hat{\rho}_{ij} = \frac{\sum_{Y_t=i} \gamma_t(j)}{\sum_{t=1}^T \gamma_t(j)}
\mbox{ .}
\]

\section{Replicate Data}

In certain situations we may observe several \ub{replicates} of
data from a hidden Markov chain.  That is we may observe sequences
of observations $Y_{\ell t}$, $\ell = 1, 2, \ldots, L$, $t = 1, 2, \ldots
T_{\ell}$.  Each sequence $Y_{\ell 1}, Y_{\ell 2}, \ldots, Y_{\ell, T_{\ell}}$
is a sequence of observations from the \ub{same} hidden Markov chain,
i.e. the transition probability matrix, the probability density functions
$f_k$, and the associated parameter values $\phi$ are all the same.

If we assume that distinct sequences are \ub{independent} replicates,
then the likelihood of all of the data is simply the product of the
likelihoods of the individual sequences (and the log-likelihood is
the sum of the log-likelihoods).

It is thus quite simple to extend the EM algorithm procedure for
maximizing the likelihood to this slightly more general setting.
One forms the ``over-all $Q$-function'' by summing the $Q$-functions
corresponding to the individual replicates.  The maximization
procedure is then straightforward.

Explicitly
\begin{eqnarray*}
Q(\theta, \theta') = \sum_{\ell=1}^L \sum_{i=1}^K \gamma_{\ell 1}(i)
                     \ln(\pi_i) & + &
\sum_{\ell=1}^L \sum_{t=1}^{T_{\ell}-1} \left[ \sum_{i=1}^K
                \sum_{j=1}^K \xi_{\ell t}(i,j) \ln(p_{ij}) \right] \\
   & + & \sum_{\ell=1}^L \sum_{t=1}^{T_{\ell}} \sum_{i=1}^K \gamma_{\ell t}(i)
         \ln(f_i(y_{\ell t} \;|\; \phi))
\end{eqnarray*}

To maximize $Q(\theta,\theta')$ with respect to $P$ (neglecting the
term in the $\pi_i$'s as before) we let
\[
\omega_{ij} = \sum_{\ell=1}^L \sum_{t=1}^{T_{\ell}-1} \xi_{\ell t}(i,j)
\]
and set
\[
\hat{p}_{ij} = \frac{\omega_{ij}} {\sum_{\ell=1}^K \omega_{i\ell}}
= \frac{\omega_{ij}}{\omega_{i\cdot}}
\]
as before.

Maximizing $Q(\theta,\theta')$ with respect to the model parameters
$\phi$ will depend upon the nature of the functions $f_k$.  If the
$f_k$ are given by a probability matrix as in Exercise 5 then the
entries of that matrix which maximize $Q$ are given by
\[
\hat{\rho}_{ij} = \frac{\sum_{\ell=1}^L \sum_{Y_{\ell t}=i} \gamma_{\ell t}(j)}
                  {\sum_{\ell=1}^L \sum_{t=1}^T \gamma_{\ell t}(j)}
\]
or (perhaps just slightly more perspicuously)
\[
\hat{\rho}_{ij} = \frac{\sum_{\ell=1}^L \sum_{t=1}^{T_{\ell}}
                  \gamma_{\ell t}(j) \times I(Y_{\ell t}=i)}
                  {\sum_{\ell=1}^L \sum_{t=1}^T \gamma_{\ell t}(j)}
\]
where $I(\mbox{condition})$ equals 1 if ``condition'' is true, and
0 if ``condition'' is false.

\section{Why the EM Algorithm Works --- Sort of}

Here we shall show that iteratively maximizing $Q(\theta,\theta')$
with respect to $\theta$ leads to an increasing sequence of
likelihood values.  That is, let $\theta^{(m+1)}$ be the value
of $\theta$ which results from maximizing $Q(\theta,\theta^{(m)})$
with respect to $\theta$.  We shall show that the corresponding
likelihood values then form an increasing sequence.

To simplify notation we write
the likelihood as
\[
L(\theta) = \sum_i p_i(\theta)
\]
and
\[
Q(\theta,\theta') = \sum_i [\ln p_i(\theta)] \times \frac{p_i(\theta')}
			{\sum_j p_j(\theta')} \mbox{ .}
\]

We know that $Q(\theta^{(m+1)},\theta^{(m)}) \geq
Q(\theta^{(m)},\theta^{(m)})$; we want to show that
\[
L(\theta^{(m+1)}) = \sum_i p_i(\theta^{(m+1)})
               \geq \sum_i p_i(\theta^{(m)})
                  = L(\theta^{(m)})\mbox{ .}
\]

Simplifying the notation still further we write

\begin{eqnarray*}
\mbox{Known: } & & \sum_i (\ln p_i) \times \frac{q_i}{\sum_j q_j} \geq
\sum_i (\ln q_i) \times \frac{q_i}{\sum_j q_j} \\
\mbox{To show: } & & \sum_i p_i \geq \sum_i q_i
\end{eqnarray*}

Noting that the denominators $\sum_j q_j$ in the ``Known'' inequality
cancel, and re-arranging a bit, the problem becomes

Show that:
\[
\sum_i \ln(\frac{p_i}{q_i}) q_i \geq 0 \Longrightarrow
\sum_i (p_i - q_i) \geq 0
\]

It suffices to show that $p_i - q_i \geq \ln(\frac{p_i}{q_i}) q_i$ for all i.

\ub{Exercise 6:} Show (by means of elementary calculus!) that
\[
p - q \geq \ln(\frac{p}{q})q
\]
for $p, q > 0$.

\end{document}
