Background

The workshop on anticipatory behavior in adaptive learning systems is the second of its kind. The first workshop ABiALS-2002 was organized in collaboration with the 7th International Conference on the Simulation of Adaptive Behavior (From Animals to Animats 7, SAB 2002). Revised and extended versions of the workshop contributions as well as further refereed submissions were published in the edited volume on Anticipatory Behavior in Adaptive Learning Systems: Foundations, Theories, and Systems (LNAI 2684). The second workshop intends to expand the gained insights in the realm of Anticipations and Anticipatory Behavior aiming for an interdisciplinary knowledge exchange on anticipatory behavior and learning mechanisms as well as the extension of a conceptual and computational approach to anticipatory mechanisms.

In general, anticipation refers to the prediction of behavioral consequences. The idea is being increasingly appreciated that for animats anticipations are necessary to produce competent adaptive behavior.

Anticipatory behavior refers to behavior that is influenced by expectations about the future, such as future states of the environment, future actions by the animat or by other animats, or merely anticipations about the way things work in a given situation.

Anticipatory behavior in turn influences both planning performance and learning performance. In animal behavior studies it has been shown that once animals have acquired an environmental model through an initial period of exploration in which they are not rewarded, they can alter their behavior dynamically in response to a differential reward. Such behavior cannot be explained by standard reinforcement learning alone.

Anticipations can also be used to improve learning performance. For instance, an animat might learn to identify a manifold in hypothesis space from which tasks are generated, and then use this anticipation to learn new tasks very quickly. Another approach might be to try to identify the class of transformations (over probability distributions) that relate similar tasks to each other. These anticipations require determining and exploiting similarity between learning tasks; a problem which has also been addressed by researchers in other areas such as lifelong learning. Note that using anticipations to improve learning on a new task is different from learning to anticipate.

Various disciplines have explicitly recognized anticipations. In cognitive psychology, anticipations have been experimentally shown to influence behavior ranging from simple reaction time tasks to elaborate reasoning tasks. It was shown that anticipations influence actual behavior as well as memory mechanisms and attention. Neuropsychology also gained further insights about the role of anticipatory properties of the brain in attentional mechanisms and, conversely, highlighted the role of attentional mechanisms in, e.g. the anticipation of objects. In animal behavior studies, learning anticipations is basically manifested in latent learning in which animals show to have learned an environmental representation during an exploration period which is not rewarded and exhibit that they have learned once a differential reward is introduced. However, the animals did not only learn an environmental representation but also managed to exploit that representation to adapt their behavior faster. It has also been suggested that anticipations could present a route to solving long-standing problems like new symbol acquisition, and the development of a shared language.

One aspect of the benefit of anticipations becomes obvious in the distinction of supervised, reinforcement, and unsupervised learning. Supervised learning, in which a supervisor, or teacher, explicitly provides the correct answer after each problem instance, is usually the most efficient one. Since the anticipation of the next situation can be confronted with the actual situation at each time step (comparable to an actual supervision), learning to anticipate can be seen as an implicit or self-supervised learning process.

Various model learning mechanisms have been proposed to learn anticipations, such as feed-forward and recurrent neural networks, layered hierarchical structured networks, self-organizing maps etc. Another approach has been pursued in rule learning systems and especially anticipatory learning classifier systems (ALCSs). In ALCSs, anticipations are usually represented explicitly in each rule in a separate anticipatory or effect part additional to the usual condition and action parts. In reinforcement learning (RL), the dynamic architectures of the Dyna family comprise an anticipatory representation of the world for a further adaptation of behavior. Dyna-like systems build a model of the encountered environment online in terms of results of actions. They utilize the model, for example, to update reinforcement internally. Recently, a new framework to learn a model based on a predictive representation (called "Predictive State Representation") has received significant interest. It seems that representing a problem in terms of anticipations can result in very efficient models; particularly in the context of non-Markov (or perceptually aliased) problems.

There has been relatively little work, however, in using anticipations to improve learning performance, and in exploiting learned anticipations for planning. Current mechanisms exploit available anticipations for novelty detection based on a notion of unexpected consequences or preventive state anticipation mechanisms that predict the usual future changes and execute preventive actions only if necessary. Further conceptual and computational approaches for the exploitation of anticipations to improve learning and behavior such as anticipatory guided attention mechanisms are necessary.

All of the above architectures and all the various disciplines have in common that some internal predictive environmental model, which contains explicit knowledge of action consequences and/or future behavior of the outside environment, is present in the memory, or mind, of the system. Anticipation allows updating the model efficiently by direct comparison of the predicted situation and the one actually encountered. In turn, the model allows the usage of anticipations to achieve more competent adaptive behavior. Essentially, anticipations control actual behavior as well as guide thoughts and consequent future behavior. Thus, anticipations can influence a wide range of mechanisms ranging from intentions, motivations, and emotions mediated by attention to actual behavior execution including planning and learning.