language: en

[In Advances in Neural Information Processing Systems 10 (Proceedings of NIPS?97), pp.1050?1056. MIT Press.]

Multi-time Models for Temporally Abstract Planning

Doina Precup, Richard S. Sutton
University of Massachusetts
Amherst, MA 01003
fdprecupjrichg@cs.umass.edu

Abstract Planning and learning at multiple levels of temporal abstraction is a key problem for artificial intelligence. In this paper we summarize an approach to this problem based on the mathematical framework of Markov decision processes and reinforcement learning. Current model-based reinforcement learning is based on one-step models that cannot represent common-sense higher-level actions, such as going to lunch, grasping an object, or flying to Denver. This paper generalizes prior work on temporally abstract models [Sutton, 1995] and extends it from the prediction setting to include actions, control, and planning. We introduce a more general form of temporally abstract model, the multi-time model, and establish its suitability for planning and learning by virtue of its relationship to the Bellman equations. This paper summarizes the theoretical framework of multi-time models and illustrates their potential advantages in a gridworld planning task.

The need for hierarchical