Dismiss
InnovationQ will be updated on Sunday, Oct. 22, from 10am ET - noon. You may experience brief service interruptions during that time.
Browse Prior Art Database

Maximum Mutual Information Training of Hidden Markov Models With Continuous Parameters

IP.com Disclosure Number: IPCOM000102617D
Original Publication Date: 1990-Dec-01
Included in the Prior Art Database: 2005-Mar-17
Document File: 4 page(s) / 172K

Publishing Venue

IBM

Related People

Gopalakrishnan, PS: AUTHOR [+3]

Abstract

This article describes a method for the maximum mutual information training (MMIE) of hidden Markov models (HMM) with continuous parameters that increases score of the mutual information objective function much faster than the currently available gradient technique.

This text was extracted from an ASCII text file.
This is the abbreviated version, containing approximately 37% of the total text.

Maximum Mutual Information Training of Hidden Markov Models With Continuous Parameters

       This article describes a method for the maximum mutual
information training (MMIE) of hidden Markov models (HMM) with
continuous parameters that increases score of the mutual information
objective function much faster than the currently available gradient
technique.

      The motivation for using MMIE of HMM for speech with continuous
parameters given in [1] is based on the facts that MMIE of HMM can
provide higher correct rate than Maximum Likelihood estimation
because it does not require the presence of the true model in the
parametric family of considered models. Also, using continuous
parameters in principle can help to avoid the loss of information
that is caused by quantization and labeling.  In [2] a general method
was suggested for the effective maximization of the mutual
information objective function with discrete parameters.  The simple
approximate version of that method was successfully implemented for
MMIE of HMM with discrete parameters [2]. In this disclosure we adapt
this method to the maximization of the mutual information objective
function with continuous parameters. In the case of discrete
parameters the main steps of the maximization algorithm are: 1)
reduction of the problem for a rational objective function to one for
a polynomial with possibly negative coefficients; 2) reduction of
this polynomial objective function to a polynomial with non-negative
coefficients for which the standard maximization technique [3,4] is
applicable.  The first step above generalizes immediately to the case
of continuous parameters, but the second step - finding a help
polynomial that has constant value in the domain of parameters which,
when added to the polynomial objective function turns it into a
polynomial with only non-negative coefficients - is difficult to
extend to the case of continuous parameters immediately.  This
happens because the domain of parameters in the continuous case is
more complex than in the discrete one.  In this article we describe a
general method for dealing with the second step for continuous
parameters. This method, roughly speaking, consists of the following
substeps.  First, we introduce help variables into the objective
function.  Treating them as discrete parameters, we find a help
polynomial in terms of these variables.  Then we replace back help
variables by functions of continuous variables in the new polynomial
objective function that has only positive coefficients and maximize
it by standard methods.

      The precise formulation of the problem for the effective
training of the mutual information objective function can be found in
(1, Chapter 4).  For the case of continuous parameters we generalize
it as follows.

                            (Image Omitted)

 Let F = {fij(y:vij)} where fij is a density of
probability distribution (for an i T j arc in HMM) of a...