language: en

Operant Conditioning in Skinnerbots

David S. Touretzky
Computer Science Department & Center for the Neural Basis of Cognition Carnegie Mellon University
Pittsburgh, PA 15213-3891
dst@cs.cmu.edu
Lisa M. Saksida
Robotics Institute & Center for the Neural Basis of Cognition Carnegie Mellon University
Pittsburgh, PA 15213-3891
saksida@ri.cmu.edu

[To appear in Adaptive Behavior 5(3/4), 1997. Copyright c 1997 The MIT Press.]

Abstract Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning (Watkins, 1989) in that it allows an agent to adapt its actions to gain maximally from the environment while only being rewarded for correct performance. But animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process that attempts to capture some of the aspects that are missing from simple reinforcement learning: conditioned reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space refinement. We apply our model to a task commonly used to study working memory in rats and monkeys: the DMTS (Delayed Match to Sample) task. Animals learn this task in stages. In simulation, our model also acquires the task in stages, in a similar manner. We have used the model to train an RWI B21 robot.

Keywords: operant conditioning, instrumental learning, shaping, chaining, learning robots

[Running Head: Operant Conditioning in Skinnerbots]