Active Inference, homeostatic regulation and adaptive behavioural control

Thus, unlike most associative or reinforcement learning theories (Daw et al., 2005), the proposed framework does not describe habitual and goal-directed controllers as necessarily competing. Instead, they are arranged in a hierarchical scheme in which the latter contextualizes the former. There have been recent proposals that address cooperation between the two controllers (Daw et al., 2011, Keramati et al., 2011, Lee et al., 2014, Pezzulo et al., 2013) and explicit proposals that they might form a hierarchy (O’Doherty et al., 2015). Here, we are not only suggesting that the two controllers might be hierarchically arranged, but that their computations are essentially the same (acknowledging that they operate at different hierarchical levels and on different representations and contingencies). This hypothesis contrasts with the view that the human brain includes two separate systems for MB and MF control, plus an arbitration system (O’Doherty et al., 2015). In the perspective pursued here, there is only one controller that realizes both goal directed and habitual behaviour, and in which higher levels (that can be considered more goal-directed in the spectrum) contextualize lower levels (that can be considered more habitual). In this view, contextualization depends on the relative precision processing at different hierarchical levels – where, effectively, precision dynamics subsume the role of arbitration. In this setting, hierarchical arbitration emerges in a principled way from free energy minimization and conforms to Active Inference principles, in which top-down and bottom-up message passing updates predictions and prediction errors, respectively. This view stands in contrast with the proposal that hierarchical message passing involves commands, policies or reward prediction errors (Botvinick, 2008, O’Doherty et al., 2015).

By the same token, the hierarchical scheme of Active Inference differs from hierarchical reinforcement learning (Botvinick, 2008, Botvinick and Weinstein, 2014). In hierarchical reinforcement learning, an agent selects either a primitive action or a higher-level action (e.g., an option); and this hierarchical scheme can extent to many levels. In Active Inference, higher hierarchical levels influence lower levels in a top-down manner, but do not replace them. Furthermore, in Active Inference action selection and the balance between hierarchical levels depend on the precision of the representations at various levels, not on option- or action-specific values. These two features also distinguish the proposed framework from the hierarchical scheme of (Dezfouli et al., 2014, Dezfouli and Balleine, 2012), where the conversion from goal-directed to habitual systems depends on a “chunking” mechanism, and the reverse transformation involves “decomposing action sequences into single actions”.

Furthermore, our scheme for behavioural control is based on Bayesian inference and does not call on reward prediction errors for learning or inference. One advantage of this is the concept of reward is replaced by the realization of prior preferences. This means that epistemic value and pragmatic value (e.g. utility or reward functions) have the same currency and can be accommodated within the same (information hungry) Bayesian scheme – where this Bayesian scheme prescribes how agents can infer or learn reward contingencies (Friston et al., 2015). Also, this normative approach enables an implementation in terms of predictive coding and cybernetic processes, which have some neurobiological plausibility (see Section 5). Although there have been other proposals involving Bayesian inference and behavioural control (Friston et al., 2010, Pezzulo et al., 2013, Solway and Botvinick, 2012), none has simultaneously addressed all the controllers in the ontology of animal learning theories, as offered below.

Another distinguishing aspect is that, in this framework, actions are triggered by goals through prediction errors, whereas in most reinforcement learning schemes, responses are triggered by stimuli via associative links – and prediction errors have only a role in learning (e.g., using Temporal Difference learning) (Sutton and Barto, 1998). In this respect, Active Inference is closer to the cybernetic view of purposive regulation (Miller et al., 1960, Seth, 2014) than to standard associative theories.

The current formulation is also deeply connected to theories that emphasize the importance of interoceptive signals for affective processing and decision-making (Craig, 2010, Damasio and Carvalho, 2013) and especially the work of Seth and collaborators on interoceptive inference (i.e., Active Inference about interoceptive states) as a basis for emotion and conscious presence (Seth, 2013, Seth et al., 2012). Here, we take a complementary perspective and discuss the combined role of interoceptive, exteroceptive and proprioceptive inference for homeostatic control and adaptive action. By starting from different angles, all these theories highlight multilateral roles of interoception, whose implications for enactive approaches to cognition (e.g., the affective consequences of goal achievement, and the associated subjective states) remain an important challenge.

The Active Inference scheme for adaptive control connects well with hierarchical and layered robotic control architectures (Arbib, 1992, Brooks, 1991, Tani and Nolfi, 1999, Verschure et al., 2003), where higher levels encode goal-directed controllers or schemas that contextualize lower-level reflex or habit-like control loops. Some of these architectures also include a (simplified) internal physiology that grounds higher-level goals (Verschure et al., 2014). Active Inference solves the problems of action selection and arbitration between levels that are implicit in these robot architectures by using a unitary, biologically-motivated scheme: a homogeneous process of free energy minimization based on a cascade of (precision engineered) predictions and prediction errors. Within this scheme, the balance between goal-directed and habitual components of behaviour can be found in a principled way based on considerations of model model “accuracy” vs. “complexity”; see above and (FitzGerald et al., 2014, Pezzulo et al., 2013). Furthermore, Active Inference offers a principled scheme to coordinate the two ‘loops’ that, in these architectures, control physiologic needs and drives (via interoceptive signals) and goal-directed action execution (via proprioceptive and exteroceptive signals), respectively. It remains to be seen if and how these features of Active Inference architectures can be translated into more effective design principles for autonomous goal-directed robots.

This new perspective has implications for the role assigned to neuromodulators, and dopamine in particular. In reinforcement learning, phasic and tonic facets of dopaminergic discharges are assumed to encode reward prediction error (Schultz et al., 1997) and average reward expectation, respectively (Niv et al., 2007). In Active Inference, dopamine is associated with expected precision (Friston et al., 2012a), which sets the relative gain of ascending projections signalling prediction error (Friston, 2008),. The ensuing precision control can be understood from several perspectives. First, in a hierarchical scheme, precision regulates the predominance of each level of the hierarchy, with respect to other levels. The function of dopamine that emerges is that of balancing the relative precision of the different hierarchical levels and thus the complexity of the generative model engaged for inference. As a general rule, simulations suggest that phasic dopamine firing elevates the precision of beliefs (probability distributions) over competing and hierarchically composed policies; thereby enabling more precise action selection. This can explain the empirical evidence that higher postsynaptic dopamine availability is associated with enhanced motor vigour, for instance speeding up reaction times (Berridge, 2004, Salamone and Correa, 2012), and with a very fast ‘habitisation’ of behaviour (Gremel and Costa, 2013). Also, this can explain the observation from several cognitive tasks of an inverted-U function, relating dopamine levels to performance (Cools and Esposito, 2011). This would be the consequence of an optimal dopaminergic activity, which would correspond to the engagement of the level of the generative model with the best speed/accuracy trade-off.

The final point speaks to an important issue; namely that the precision or confidence about future behaviour can itself be optimized with respect to free energy or model evidence. In other words, there is an optional precision of beliefs about any choice behaviour under uncertainty.

In Active Inference models of choice behaviour (Friston et al., 2014, Friston et al., 2013), precision reports opportunities to achieve a goal (or more precisely, confidence in the opportunities) that invigorate and select action; see Fig. 7. The expected precision or confidence about preferred outcomes reports on the progress towards the goal – like spatial proximity or sub-goals (Lepora and Pezzulo, 2015, Maisto et al., 2015) – and in accordance with early cybernetic models of purposive action such as the Test-Operate-Test-Exit (TOTE) model (Miller et al., 1960) precision can be updated (in a Bayes optimal fashion) during goal-directed action – rather than just mediating learning by reporting reward prediction errors. This view is compatible with theories that describe dopamine function within a “wanting” system (Berridge, 2004) that is not hedonic (as in the “liking” system) but instrumental to goal pursuit (Salamone and Correa, 2012).

Finally, precision regulates the gain of (postsynaptic) prediction errors that drive associative (synaptic) plasticity (Friston, 2008). The post-synaptic effects of dopaminergic projections from the ventral tegmental area and substantia nigra varies across target regions (e.g., different portions of basal ganglia and cortex), due, for instance to the complex role of receptors such as D1 and D2 (Clark and White, 1987). This prompts the hypothesis that the effects of increased/decreased dopamine activity vary within and across individuals – an idea that has inspired interpretations of several symptoms and signs in neuropsychiatric disorders. For example, there is much current interest in understanding a failure to attenuate sensory precision as an explanation for several symptoms and signs in neuropsychiatric disorders (Edwards et al., 2012, Friston et al., 2012a).

Fig. 7 illustrates how a hierarchical scheme can explain one of the cardinal features of dopaminergic discharges. In this example, we used a T-maze to present conditioned stimuli (CS) in the form of cues that indicated which arm was baited with a reward. Using a simple Bayesian (variational) belief update scheme (see figure legend) it is fairly simple to reproduce dopaminergic discharge profiles that are similar to those observed empirically. In brief, the confidence about subsequent behaviour is encoded by dopamine (whose phasic discharges increase the precision of probabilistic beliefs about competing policies). The figure compares (simulated) dopamine responses to the CS and unconditioned stimulus (US or reward) before the agent knows where the reward is and after it has inferred its location. This example shows that hierarchical inference offers a sufficient explanation for dopamine responses in terms of the confidence or precision about subsequent behaviour. Here, the CS has epistemic value because it resolves uncertainty about subsequent (goal directed) behaviour. Under these circumstances the CS elicits a burst of dopamine firing that is effectively transferred from the US to the CS. See Friston et al. (2015) for details of this particular example.

A final important point concerns the relationship between drives and goals. In most psychological theories, drives are usually linked to basic motivations and homoeostasis (e.g., interoceptive hunger or thirst), while goals are considered to have more elaborate (exteroceptive) sensory and cognitive aspects (e.g., dining in a restaurant), but their relations are often unclear from a mechanistic viewpoint (Pezzulo et al., 2014b, Verschure et al., 2014). In Active Inference, goals are ultimately grounded in expectations about physiological allostasis, and engage a cascade of interoceptive, proprioceptive and exteroceptive loops. The objective is to suppress interoceptive prediction errors (Barrett and Simmons, 2015, Pezzulo, 2013, Seth et al., 2012) through action, which in turn requires bodily signals to be represented centrally and controlled through a deep hierarchical modelling that – crucially – engages proprioceptive and exteroceptive loops. This implies that at every level of the hierarchy, goals of different complexity are represented that can be distally (and evolutionarily) related to homoeostasis but at the same time are (conditionally) independent from them. Here, conditionally independent means that goals have autonomy in steering and controlling behaviour; for example, the goal of ‘going to a fancy restaurant’ is certainly linked to the drive to eat (or socialize) but it can induce restaurant-searching behaviour in the absence of hunger. Furthermore, going to a restaurant does not in itself cause a reversal of hypoglycaemia. The ‘autonomisation’ of goals from primary drives is a key feature of higher animals like humans (Pezzulo and Castelfranchi, 2009) and becomes more evident in pathological conditions, where certain goals such as gambling can become pathological and maladaptive (Montague et al., 2012).

Autonomy is a general characteristic of hierarchically organized architectures that, necessarily, embody conditional independencies and enable organized behaviour over extended timeframes.

This account thus goes beyond purely Hullian, drive-minimization theories of motivation and emphasizes that an important part of motivated behaviour is directly guided by goals and internal representations of desired future states. Basic drives associated with evolutionary imperatives (e.g., good levels of glucose) naturally constrain the acquisition of new empirical priors that define goals at some (high) level of an agent's generative model. In turn, these high-level priors contextualize behaviour that is internally consistent with lower-level drives provided they are afforded sufficient precision. Although the ultimate reason for the emergence of goals is that they satisfy some internal drive, the activation of a specific goal might or might not be caused by concurrent interoceptive inference. Thus, the goal system supports and – in a certain sense – supersedes the drive system: (Montague, 2006) speculates that the goal system capitalizes on existing brain architectures for (reward) prediction errors and reuses it in an open-ended way. In a similar vein, Passingham and Wise (2012) discuss the adaptive advantages of the more advanced (prefrontal) goal systems. They argue that the (new) prefrontal areas of anthropoid primates extended the (older) reinforcement-learning system, increasing its flexibility and adaptability; for example, by generating (foraging) goals from single learning episodes. The resulting architecture for motivated behaviour frees higher animals from the immediate demands of homeostatic regulation. While this may predispose to maladaptive behaviour, it also permits an open-ended proliferation of goals and desires that characterize our human lives – because, as Baruch de Espinoza puts it, “desire is the essence of a man”.

A simplified functional anatomy for the hierarchical Active Inference scheme that we have discussed is shown in Fig. 8. This should be considered as a schematic that organizes ideas about the functional anatomy underlying active or embodied inference. This sort of scheme should not be considered as a comprehensive characterization of functional brain architectures: many components are depicted very simplistically – and some brain areas are not discussed at all. However, it may be useful as a general framework about the functional neural organization and as a working hypothesis for the interpretation and testing of empirical studies. Here anatomical loops of increasing complexity are proposed to embody the ‘gradient’ of control discussed earlier, with the more elaborated goal-directed control loops that contextualize the more elemental sensorimotor constructs within generative models of increasing hierarchical depth. The different levels of the Active Inference hierarchy are represented by a colour gradient from violet (the highest level: prefrontal cortex, PFC), red (supplemental motor area, SMA; Inferotemporal cortex, IC; anterior insular cortex, AIC), orange (premotor/motor cortex, PMC/MC; striatum; ipothalamus, ipoT) to yellow (autonomous nervous system, ANS; motoneurons). Connections among areas represent recurrent projections between areas that follow the usual logic of predictive coding, where backward connections convey descending predictions and forward connections pass prediction errors.

Hierarchically higher areas play a more prominent role in the goal-directed system, whereas orange areas predominate in the Pavlovian and habitual systems. The central position assigned to prefrontal areas at the apex of the goal-directed system is in accordance with a large literature highlighting its importance for flexible cognitive control and executive function (Fuster, 1997, Miller and Cohen, 2001). Evidence is consistent with the hypothesis that contextual representations of increasing abstraction and timescale are processed along a posterior-anterior axis in PFC, suggesting that this region itself is hierarchically organized (Koechlin and Summerfield, 2007). Goal-directed control requires the integration of information from different sources. This corresponds in our model to the fact that the exteroceptive network (including SMA, IC, striatum, MC and motoneurons – bold circles) is anatomically segregated from the interoceptive network (including AIC, ipoT and the ANS – dotted circles), but the two are integrated at the multimodal highest level in the PFC, see Fig. 2 and Gu et al. (2013). Goal-directed control also requires knowledge of the contingencies between actions and outcomes (Balleine and Dickinson, 1998). This form of knowledge is encoded in various ways – and at increasingly longer timescales – in motor, premotor, and prefrontal brain areas, which overall form hierarchical control loops.

Another brain structure with a primary role in goal-directed control is IC and the closely related hippocampus. The ability of the hippocampus to support goal-directed learning and behaviour has been studied principally in spatial navigation (Buzsáki and Moser, 2013), but evidence suggests that this area also supports other advanced cognitive abilities that require prospection, such as imagination of fictive scenarios (Hassabis and Maguire, 2009). From the associative learning perspective, the hippocampus has been linked to a form of instrumental process called ‘episodic’ control. Specifically, the hippocampus is thought to store state-action sequences that successfully attain a goal and then replay those sequences when a constituent state is encountered (Lengyel and Dayan, 2008). This ability to generate fictive sequences of states (Johnson and Redish, 2007, Pfeiffer and Foster, 2013) has been interpreted in terms of reactivation in the context of generative models (Penny et al., 2013, Pezzulo et al., 2014a, Pezzulo et al., 2013). Furthermore, the hippocampus has been consistently linked to memory consolidation (Buzsáki, 1996), a role that can be characterized within Active Inference as the reporting of the expected information, precision and novelty of sequential events (Strange et al., 2005). In short, the hippocampus is an important nexus for both the expression of goal-directed behaviour (possibly via the covert replay of previous experiences encoded in the generative models) and its acquisition (possibly by signalling the opportunity to optimize or revise generative models).

In the hierarchical architecture depicted here, brain structures controlling more complex strategies – that are mandated by goal-directed behaviour and cognitive control – do not replace more ancient brain structures underlying the simpler strategies based on fixed stimulus-response pairs, but contextualize them. In this way, more elaborate forms of action control orchestrate more elementary sensorimotor architectures that support situated action or affordance competition (Cisek and Kalaska, 2010). In the present scheme, this may involve cortical (motor and premotor) and subcortical regions, such as the striatum and hypothalamus.

A widespread view is that different striatal territories (coupled with limbic, associative and sensorimotor cortex) participate in distinct cortico-subcortical loops – limbic, associative, and sensorimotor, respectively – and support different aspects of behavioural control (Yin and Knowlton, 2006). For example, the dorsolateral striatum is thought to be more involved in stimulus-response learning and habitual control, the dorsomedial striatum has been linked to action-outcome learning and goal-directed action, and ventral striatum to Pavlovian values (Glascher et al., 2010, Liljeholm and O’Doherty, 2012, Mannella et al., 2013, O’Doherty et al., 2004). In a related perspective, these striatal territories might compute outcome predictions in parallel using different kinds of information as input (Pennartz et al., 2011). In this perspective, the model-based vs. model-free dichotomy is replaced by a more nuanced view, in which striatal territories embody different predictors (or models): from the more complex outcome predictions in ventral and dorsomedial striatum (analogous to model-based mechanisms), to the simpler predictions about actions based on somatosensory and motor information in the dorsolateral striatum (analogous to model-free mechanisms). The idea that different striatal territories support different aspects of behavioural control is coherent with the Active Inference scheme where generative models are not purely cortical structures but involve cortico-subcortical loops. In the perspective pursued here, however, the striatal subdivision is conceived as in hierarchical terms, rather than in terms of competition among parallel controllers or predictors. This hierarchical view of striatal computation is indeed supported by recent neurophysiological evidence (Ito and Doya, 2015).

Brain loops might extend beyond the striatum and include, for example, the cerebellum, which – given its remarkably homogenous microarchitecture – has been often assumed to be able to encode a large repertoire of internal (generative) models required to predict and control action and its timing (Caligiore et al., 2013, Imamizu et al., 2003, Miall and Wolpert, 1996, Wolpert et al., 1998). Embodied views of cognition suggest that the re-enactment of the same perceptual-motor loops – and internally generated brain dynamics – might realize increasingly complex cognitive functions such as planning, imagery, and conscious thought (Buzsáki et al., 2014, Grush, 2004, Hesslow, 2002, Jeannerod, 2001, Pezzulo, 2011, Pezzulo and Castelfranchi, 2009).

Motoneurons and autonomic effectors are found at the lowest level of the hierarchy. These infrastructures constitute the basic building blocks to enact arc reflexes and are exploited and contextualized by the higher hierarchical levels. It is worth noting that in the Active Inference scheme, motor areas represent proprioceptive predictions and not motor commands (as traditionally assumed), in the same way as sensory (e.g., visual) areas represent exteroceptive predictions (Adams et al., 2013). Motoneurons are responsible for activating arc reflexes that fulfil these predictions, thus guiding overt action. They are thus the locus where predictions are unpacked in a kinematic frame of reference and transformed into overt action.

In addition to the higher-to-lower control hierarchy, the Active Inference scheme requires an attentional network that modulates the relative precision of different levels in the hierarchy. In Fig. 8 this role has been assigned to the blue areas (anterior cingulate cortex, ACC; posterior parietal cortex, PPC; and the dopaminergic ventral tegmental area and substantia nigra, VTA/SN). We hypothesize that ACC and PPC (linked preferentially with higher order areas through recurrent connections) are involved in modulating the relative precision of PFC relative to lower level areas; whereas VTA/SN (projecting widely to all other areas) is more important in regulating the precision of exteroceptive and interoceptive sensations (blue connections). The attentional network has an important role in the balance between more cognitively complex goals and more elementary ones, maintained at different hierarchical levels, which is an important hallmark of cognitive control (Fuster, 1997, Miller and Cohen, 2001). Here, cognitive control entails a high degree of precision of “higher” areas such as PFC, so that more cognitively complex and long-term goals dominate the inference (Pezzulo, 2012, Stoianov et al., 2015). However, in some cases, such as in drug addiction, some aspects of prefrontal function decline and the agent becomes unable to contextualize or suppress (hierarchically lower) cortico – dorsal striatum loops that implement habits (Belin et al., 2009).

In sum, we propose a hierarchical neural architecture that extends from cortical areas to peripheral reflexes. This hierarchy is organized according to the level of abstraction of representations embodied by the different areas. The neural architecture is based on predictive coding, which constitutes a Bayesian inference machine that guides both perception and action.

Active Inference rests upon, and extends, a predictive coding scheme, which provides an explanation for many aspects of functional brain architectures. In brief, it explains the hierarchical nature of cortical connections; the prevalence of backward connections and explains many of the functional and structural asymmetries in the extrinsic (between region) connections that link hierarchical levels (Zeki and Shipp, 1988). These asymmetries include the laminar specificity of forward and backward connections, the prevalence of nonlinear or modulatory backward connections (that embody interactions and nonlinearities inherent in the generation of sensory signals) and their spectral characteristics (Adams et al., 2013, Bastos et al., 2012, Friston, 2008). The specific evidence for predictive coding in the motor system is reviewed in (Shipp et al., 2013).

The studies mentioned above all provide some indirect support to the framework for adaptive behavioural control proposed here, which implements the plausible predictive coding and Active Inference schemes. Furthermore, this new framework allows to explain previous evidence and to make novel predictions that rest on our central premise; namely, that habitual and goal-directed behaviours are context-sensitive expressions of the same hierarchical (active) inference. In other words, by definition, habitual behaviours become habitised by virtue of selecting low-level hierarchical contingencies to make proprioceptive and interoceptive predictions that induce behaviour. The selection rests upon modulatory (gain) control that we presume is reflected in dopaminergic activity. Therefore, neuromodulatory (e.g., dopaminergic) systems should be responsible for hierarchical selection – or mixture – of goal directed or habitual behaviour.

Empirical studies suggest that dopaminergic projections are divergent and modulate, in a complementary fashion, higher and lower hierarchical levels of executive (corticostriatal-thalamic) processing. Our proposal that dopamine encodes the relative precision of the different hierarchical levels fits with such evidence. This divergence and modulatory aspect has been discussed under RL models (O’Doherty et al., 2015). However, we stress that, in the current view, dopamine does not encode reward prediction errors but the precision of prediction errors at various levels of a hierarchy. Because precision is a ubiquitous attribute of all (hierarchically deployed) prediction errors, its computation and broadcasting mandates a convergent-divergent neuroanatomy and a modulatory neurophysiology. Because, in Active Inference, optimization rests upon reciprocal message passing, dopaminergic reference to multiple hierarchical levels must be reciprocated by (monosynaptic or polysynaptic) afferents from these regions.

Although other schemes for hierarchical control have been proposed that use belief precisions or related constructs to modulate the balance of model based and model free controllers (Daw et al., 2005, Gershman and Daw, 2012, Lee et al., 2014, Pezzulo et al., 2013), there are some unique features of the Active Inference scheme that make novel and empirically testable hypotheses. For example, in Active Inference, but not in the aforementioned models, the optimal value of the precision parameter (in terms of free energy minimization) is computed on-line as part of inference. Furthermore, this precision controls the degree of “exploration” of the system, because, formally, it corresponds to the temperature parameter of softmax choice rules. In other words, the degree of exploration is derived from first principles rather than being tuned to behaviour as an ad hoc parameter. This feature of the model entails the novel prediction that a higher precision or confidence in their behavioural plans makes animals less exploratory and thus ultimately produces more stereotyped responses, which again speaks to habitisation. By the same token, exploration is favoured by the presence in the environment of cues that have epistemic value and can improve the precision of the behavioural policies (Friston et al., 2015).

Furthermore, an idea specific to our framework is that habitual and goal-directed behaviour form a continuum rather than a strict dichotomy, and can cooperate rather than just compete. Consistent with this view, recent behavioural and neural evidence supports the idea that at any given moment action selection results from a mixture of goal-directed and habitual control, rather than being under the exclusive control of either of the two systems (Daw et al., 2011, Lee et al., 2014, Otto et al., 2013). Although these data can still be interpreted as resulting from two systems working in parallel and competing at a later stage, a much simpler explanation is the idea of a unitary hierarchy scheme in which the different levels correspond to specific degrees of complexity. In this perspective, here the novelty of the proposal lies in the architectural scheme itself, which does not foresee multiple or even modularized controllers that perform heterogeneous computations plus an arbitration mechanism, but a unique architecture that performs homogeneous computations at all hierarchical levels.

Strictly speaking, in the proposed scheme, habitual behaviour is not completely model free in that it continues to depend on the (simplest) type of predictive model, of the kind “because there is a stimulus, I expect a response”. In Active Inference, it is the expected response that enslaves action, thus even these simple responses are prediction-based. Neurophysiologically, this is compatible with the view that the dorsolateral striatum, which has been long associated with habitual control, encodes simple models that generate predictions based on the animal's somatosensory and motor information (Pennartz et al., 2011). This scheme lends itself to a straightforward hierarchical extension, in which simpler models can be successively contextualized and can incorporate elements of goal-directedness, but only when more complex models at the higher hierarchical levels have sufficient precision to exert an influence. In other words, in this scheme there is no need to see habitual and goal-directed behaviours as regulated by distinct or modular processes.

The key ethological insight afforded by hierarchical Active Inference is that all behaviours are hierarchically contextualized. Not only does this predict that goal-directed and habitual responses can be mixed, but also the specific conditions where this should occur. For example, it predicts that responses should fail to be contextualized – and thus become more habitual – when the system is ‘taxed’ by concurrent cognitively demanding tasks, a phenomenon that has been often interpreted in terms of reinforcement learning theories (Gershman et al., 2014, Otto et al., 2013). Furthermore, it predicts that under certain conditions habitual behaviour can cease to be purely ‘model free’, thereby disclosing evidence of its context sensitivity. For example, phenomena like extinction and the context-dependent reactivation of extinguished associations can be explained in terms of the (hierarchical) selection of learned associations at different periods of exposure to the environment. For example, the extinction of a conditioned response (say, a conditioned response of hunger and salivation following the sound of a bell) would not derive from an attenuation of the sound- food association, but from the fact that the response can be contextualized by a (higher) mechanism that learns how the probability of food availability depends on the sound, in different contexts. In the same way, the response can be reactivated when the right contextual conditions are detected, thus providing the top-down mechanisms enough precision to influence or overcome any behavioural habit developed in the interim; see also (Gallistel, 2012).