Browse Prior Art Database

Iterative Scheme for the Maximum Mutual Information Training Of Hidden Markov Models

IP.com Disclosure Number: IPCOM000102652D
Original Publication Date: 1990-Dec-01
Included in the Prior Art Database: 2005-Mar-17
Document File: 3 page(s) / 93K

Publishing Venue

IBM

Related People

Gopalakrishnan, PS: AUTHOR [+3]

Abstract

This article describes an effective method for the maximum mutual information estimation (MMIE) of hidden Markov models (HMM) with discrete parameters that for each iteration produces a direction and step size which are guaranteed to improve or at least not worsen the mutual information objective function.

This text was extracted from an ASCII text file.
This is the abbreviated version, containing approximately 52% of the total text.

Iterative Scheme for the Maximum Mutual Information Training Of Hidden Markov Models

       This article describes an effective method for the
maximum mutual information estimation (MMIE) of hidden Markov models
(HMM) with discrete parameters that for each iteration produces a
direction and step size which are guaranteed to improve or at least
not worsen the mutual information objective function.

      It is known (1) that MMIE of HMM for speech with discrete
parameters can provide recognition rates higher than the maximum
likelihood estimation (MLE).  But the disadvantage of current
algorithms for maximization of the mutual information objective
function is that  they are expensive and slow.  For MLE there exists
the forward-backward algorithm that operates in linear time
(proportional to the length of input signal) (2).  We show that as
for MLE one can construct effective algorithms for MMIE.  More
precisely, we show first that there exists a large class of
hill-climbing type of algorithms with the property that was
formulated in the above summary. Secondly, we show that in this class
of algorithms there exists algorithms that can be performed in
forward-backward manner and executed in a linear time.  In addition,
transforming in appropriate way formulas that were used in some of
the above algorithms we derive new very simple heuristic algorithms
for the maximization of the mutual information objective function in
forward-backward manner.

      Precise formulation of the problem for the effective training
of the mutual information objective function one can find in (1,
Chapter 4).  We generalize the problem as

                            (Image Omitted)

 follows:  (A)  Let P({Xij
})> = P1 ({Xij })/P2 ({Xij }) be a ratio of two polynomials P1 ({Xij
}) and P2 ({Xij }) with variables Xij, i = 1,..., p and j =
1, ..., qi .  Let D = {xij / O;    xij = 1} be a
domain over which P({Xij }) is defined.  Let x = {xij } be any point
in D. Find a transformation f:d T D such that if xij = f (x)ij
* xij then P({xij }) > P(x).

      Step 1.  Reduction of the problem (A) to the case...