Active Inference, homeostatic regulation and adaptive behavioural control

Another set of predictions stem from the idea that action selection and initiation depends on (multimodal) prediction errors, not only stimuli. This view is compatible with a large body of evidence in the ideomotor theory, according to which actions, including those that are as simple as button presses (as used in most human studies), are planned and selected based on their anticipated effects (Hoffmann, 2003, Hommel et al., 2001, Kunde, 2003, Kunde, 2001, Prinz, 1997; Schuetz-Bosbach and Prinz, 2007). By expanding this framework to consider a role for interoceptive (not only exteroceptive or proprioceptive) prediction errors, it may be possible to address the link between interoceptive states and action selection and initiation, which is currently under-explored in the animal learning literature. Two caveats for experimental studies investigating this link would be the largely anticipatory nature of interoceptive events and the fact the relations between actions and interoceptive consequences can be distal and mediated by other (exteroceptive or proprioceptive) events, see Section 3.1.

To summarize, in this section we have presented a range of empirical predictions that stem from our framework. Some of these predictions are unique to this framework and could serve to disambiguate it from alternatives. Other predictions, have been addressed under other normative perspectives; most notably, reinforcement learning. Perhaps the most important (novel) contribution of our proposal is that it accommodates multifarious phenomena, which have been addressed using different – and often incompatible – computational approaches. Furthermore, the framework can call upon specific process theories that are biologically plausible; for example, predictive coding and cybernetic schemes (Friston, 2010). In other words, there is the potential for an explicit connection between the normative (approximate Bayesian inference) level and the processes underlying neurobiological implementation.

In this article, we have cast “homoeostasis” and “behavioural control” in terms of Active Inference. Under this perspective, drive and goal achievement mandate the suppression of prediction errors of different kinds (interoceptive, proprioceptive, and exteroceptive) within a hierarchical architecture, and their resolution through action. The underlying computations are driven by a unique imperative: free energy minimization. However, when the same free energy minimization is expressed in different conditions (e.g., before, during or after learning of generative models) they produce different forms of behavioural control that correspond to reflexes, conditional responses and goal-directed behaviour.

This paper establishes a link between Active Inference and associative learning theories of animal behaviour, with potential benefits for both. On the one hand, the link extends the scope of Active Inference to the self-regulation of bodily (as opposed to sensorimotor) states and interoceptive inference (Pezzulo, 2013, Seth et al., 2012). Furthermore, the link with animal learning theories clarifies the nature of hierarchical inference. For example, empirical priors in generative models correspond to “drives” and “goals” in the animal learning literature – despite the fact they are called “beliefs” in the Bayesian brain literature. We have suggested that an initial set of priors might correspond to Pavlovian values shaped by evolution, while others are acquired by learning generative models. On the other hand, the connection with Active Inference suggests that the behavioural controllers proposed in the animal learning literature might be seen as the successive contextualization of simpler sensorimotor mechanisms in hierarchical generative models. This view emphasises an inclusive and unified view of functional brain architectures, rather than a collection of distinct and separable modules.

The homeostatic view proposed here clarifies why Active Inference agents are not plagued by the (infamous) “dark room” problem (Friston et al., 2012b). Homeostatic regulation implies a continuous update of empirical priors for action, whose dynamics are dictated by the uninterrupted flux of interoceptive message passing between brain and body. This flux incessantly supplies new set points or goals – and a “moving target” for behaviour. Thus, any organism equipped with a body must face reality rather than live in a darkened room. It is exactly in this sense that (Pavlov, 2010) argues that the brain provides the interface between the internal milieu of the body and the external milieu of the environment. In other words, the interoceptive state is what organisms try to maintain within a viable range – because departures can be dangerous. Exteroceptive states are conserved, provided this sufficient to keep interoceptive states in a physiological range – but here more variability and “exploration” is allowed. Maintaining this delicate balance is an important constraining factor for any robust control system, biological or artificial.

Our analysis also grounds the abstract computations of free energy minimization as described in theoretical neuroscience (Friston, 2010). It suggests that free energy can be minimized at multiple timescales: evolutionary for Pavlovian values, lifelong learning for instrumental control (in particular habitual control), and on-line search-based computations for goal-directed control. In this perspective, the same free energy computations that optimize on-line action might also be used to describe how brain structure is optimized during evolution to embody an organism's needs and how it is adapted during development to retain useful (habitual) strategies and to increase the precision of lower hierarchical layers. These elements are all implicated in free energy principle but here we provide a new perspective on the mechanisms supporting that minimization and the timescales at which they operate.

Our review of different approaches to behavioural control highlights that some terms are used in different ways; for example, the concept of “value”. In the homeostatic view pursued here, value is the complement of surprise or prediction error (because maintaining homeostasis is “valuable”). This implies that any state or action can have an associated value if it links, directly or indirectly, to the minimization of prediction error. Essentially, all adaptive systems should pursue value through action in some way.

It is important to note that although value computations are implemented differently in different systems, they essentially use the same logic, which defines the normative aspect of these behaviours. In most homeostatic theories, the value is pursued directly by engaging (Pavlovian) innate approach or avoidance actions. In reinforcement learning, for instance a temporal-difference reward prediction error is used to train (habitual) controllers that can be considered a proxy for achieving a certain “reward”, which is in turn a proxy for restoring good homeostatic values. In episodic controllers, action sequences that lead to reinforcement are stored and re-enacted in similar circumstances. In Active Inference, free energy minimization can either consider sensory prediction errors directly, or do so indirectly by considering proxies (e.g., prediction errors at higher levels of hierarchical models that link sensory prediction errors over modalities and time). In short, all these approaches rest on similar or even identical value computations, but differ in specific aspects of their mechanistic interpretation and implementation.

Understanding the computational and neuronal basis of motivated behaviour is a key objective of many disciplines including psychology, neuroscience, and (neuro)economics. The framework offered here provides principled and mechanistic hypotheses about how the basic brain design of simpler animals might have been reused and extended through generative models and, while retaining essential embodied aspects, permits higher animals to achieve open-ended goals.

GP is funded by the EU's FP7 Initiative (grant FP7-270108) and the HFSP (grant RGY0088/2014). KJF is funded by a Wellcome Trust Principal Research Fellowship 088130/Z/09/Z). We thank our reviewers for insightful comments.