Background

Over the last decades, the idea that anticipations guide behavior has been increasingly appreciated. Anticipations, referring to the prediction of behavioral consequences, play a major role in the coordination and realization of adaptive behavior.

Various disciplines have explicitly recognized anticipations. In cognitive psychology, anticipations have been experimentally shown to influence behavior ranging from simple reaction time tasks to elaborate reasoning tasks. It was shown that anticipations influence actual behavior as well as memory mechanisms and attention. Neuropsychology also gained further insights about the role of anticipatory properties of the brain in attentional mechanisms and, conversely, highlighted the role of attentional mechanisms in e.g. the anticipation of objects. In animal behavior studies, learning anticipations is basically manifested in latent learning in which animals show to have learned an environmental representation during an exploration period which is not rewarded and exhibit that they have learned once a differential reward is introduced. However, the animals did not only learn an environmental representation but also managed to exploit that representation to adapt their behavior faster.

More technically, one aspect of the benefit of anticipations becomes obvious in the distinction of supervised, reinforcement, and unsupervised learning. Supervised learning, in which a supervisor, or teacher, explicitly provides the correct answer after each problem instance, is usually the most efficient one. Since the anticipation of the next situation can be confronted with the actual situation at each time step (comparable to an actual supervision), learning to anticipate can be seen as an implicit or self- supervised learning process. Thus, anticipations should be learnable faster than reinforcement behavior and consequently anticipatory driven behavior should adapt itself more efficiently.

Distinct model learning mechanism exist in the area of artificial adaptive behavior. In neural nets, for example, recurrent neural nets but also simpler feed-forward neural nets can be utilized to anticipate behavioral consequences. Also layered or hierarchical structured nets as well as self- organizing nets have been used to build an environmental model. Another approach has been pursued in rule learning systems and especially anticipatory learning classifier systems (ALCSs). In ALCSs, anticipations are usually represented explicitly in each rule in a separate anticipatory or effect part additional to the usual condition and action parts. In reinforcement learning (RL), the dynamic architectures of the Dyna family comprise an anticipatory representation of the world for a further adaptation of behavior. Dyna-like systems build a model of the encountered environment online in terms of results of actions. They utilize the model, for example, to update reinforcement internally.

All the above architectures and all the various disciplines have in common that some internal environmental model containing explicit results of actions must be present in the memory or mind of the system. Anticipation allows to update the model efficiently by direct comparison of the predicted situation and the one actually encountered. But, in turn, this model allows the usage of anticipations to achieve a more competent adaptive behavior. Anticipations essentially control actual behavior as well as guide thoughts and consequent future behavior. Hereby, anticipations influence a wide range of processes ranging from intentions, motivations, and emotions over attention until actual movements.