Browse Prior Art Database

Markov Decision Process for Dynamical Systems

IP.com Disclosure Number: IPCOM000022674D
Publication Date: 2004-Mar-25
Document File: 2 page(s) / 67K

Publishing Venue

The IP.com Prior Art Database

Abstract

Disclosed is a method for speeding the decision process of dynamical systems when the dependent parameters (such as states and spaces) are of high dimension. Benefits include reducing computational time and making stochastic processes deterministic.

This text was extracted from a Microsoft Word document.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 53% of the total text.

Markov Decision Process for Dynamical Systems

Disclosed is a method for speeding the decision process of dynamical systems when the dependent parameters (such as states and spaces) are of high dimension. Benefits include reducing computational time and making stochastic processes deterministic.

Background

Modeling and learning in high-dimensional dynamical systems is very time consuming, because the parameters that represent the dimensions (i.e. different states and spaces) continuously change. Because of this change the convergence criteria alters rapidly, making decisions very difficult; it either involves huge computational power or long computational time, or both. For complex problems, the probability of finding a solution that reaches a steady state is exceedingly small in high-dimensional dynamical systems.

Real-world processes can be represented as dynamical systems. Dynamical systems are described in terms of the evolution of one or more state variables in response to one or more external variables. These state variables and their transitions are stochastic. Present solutions to dynamical systems involve generating various possible solutions or models, then averaging them.

General Description

The disclosed method estimates only the mean of the state or transition at each future time; by associating a gain or loss with the state or transition, the stochastic process becomes a deterministic one. The following equations give an overview of the technique for a one-dimensional state space, which can be easily extended to an n-dimensional state space.

Equation 1

Let be the state at a given time t, and and be the mean and expected states, respectively. For a given distribution of states the estimate (gain or loss) at a future time is expressed in Equation 1, assuming the behavior to be quadratic, where a, b, & c depend on . To calculate the estimate E, full knowledge of state distri...