Dopamine, affordance and active inference

Clearly, to integrate Equation 5 we need not only the exogenous causes (that specify sequence of visual cues) but also action that depends upon the perceptual inversion of a generative model. The generative model here was chosen to include several features of sensorimotor integration and hierarchical dynamics in the simplest way possible. It considers the brain to model the sensory world as a succession of unstable fixed points in some abstract state space. In other words, the agent expects the world to change continuously, with an itinerant (wandering) trajectory, visiting different states in succession. These reflect prior beliefs about forthcoming sensory events, and are encoded by differential equations that embody metastable dynamics; e.g., winnerless competition; [87], [88]. The resulting attractors can be thought of as central pattern generators that can be nested hierarchically at different time scales [89] to produce exteroceptive and proprioceptive (i.e., sensorimotor) predictions. These predictions are entrained by perception and prescribe motor responses through active inference. An important aspect of these generative models is that high level dynamics determine the context or set that engages lower-level sensorimotor sequences. In what follows, we will exploit this hierarchical aspect to illustrate some generic features of set switching and action selection and how they depend on the delicate balance of precision over different hierarchical levels.

The generative model used by our simulated agent had two levels and the following form (a particular case of Equation 2):(6)At the sensory level, this model has the same form as the actual process generating sensory information (in Equation 5), with the exception that salience is log transformed so that it can be positive or negative (hence the exponential function in the first equality). However, above the sensory mapping (the first equation) the form of a model is very different from the process generating data (Equation 5) and embodies (formal) priors that induce a dynamic interplay between action and perception. It is this interplay that underlies the agent's behaviour. Crucially, the hidden states not only predict visual salience but also predict changes in proprioception. In this sense, they become affordance states. More specifically, the second differential equation means that the agent expects its arm to be drawn to the location specified by a function of hidden affordance states, where encodes the location of the four targets or cues. A softmax function of hidden affordance states just ensures that the hidden state with the largest affordance predominates over the others. It is important to note that if we did not add anything else to this generative model, the agent would simply point to each target when it appeared; cf., [90]. This is because perception would infer that the affordance of a particular target was high. The associated conditional expectations would then induce proprioceptive predictions that would be fulfilled by action; such that the agent would point towards the target. In an experimental setting, these prior beliefs may be instantiated through task instructions.

However, we will consider a more sophisticated and realistic model, in which the agent has prior beliefs about the sequence in which targets will appear. This sort of prior belief could be instantiated by repeated exposure to the same sequence. These prior beliefs are encoded by the function that prescribes a stable heteroclinic channel or winnerless competition among hidden affordance states. In other words, the agent believes that the affordance of the four target locations will change continuously, where each of the four states rises in turn, exciting the next state and suppressing itself (see Figure 2). The speed of these sequential dynamics is governed by the hidden cause . If we stopped here, we would have a simple agent with a limited repertoire of expectations that comprised a fixed sequence of sensorimotor events. These expectations may be consistent with the actual order of cues encountered or they may not be.

The final part of the model endows the agent with the concept that cues may or may not be ordered. We model this in terms of the hidden cause that controls the speed of the sequence. Crucially, this hidden cause is itself a softmax function of a hidden state that is part of a slower itinerant cycle (by factor of eight), governed by the same winnerless competition among the hidden states; . This means that, depending upon the second level hidden states; the sequential dynamics of the hidden affordances at the first level may or may not be engaged. The resulting model may sound complicated; however, its complexity lies in labelling various states of the model. The actual form of the model is both mathematically quite simple and biologically plausible: we have just placed a slow pattern generator on top of a fast pattern generator and have then mapped to sensory consequences. Both pattern generators have the same universal form and show autonomous, metastable dynamics of the sort seen in the real brain [87].

In summary, the agent believes that it will point towards salient cues when they appear. Furthermore, it believes that these cues would appear one at a time; either in a fixed (clockwise) sequence or with no sequential contingencies. Although this is a very simple model of the world, it allows us to demonstrate sensorimotor integration in the context of cued motor actions, biasing of action selection in terms of sequential anticipations and set switching that depends upon recognising the context (sequential or random) in which cues appear. Our particular interest here is in how manipulating the precision (dopamine) at various levels in this hierarchical model will impact on cued responses. The interesting behaviour depends entirely upon the prior beliefs entailed by the form of the generative model and its equations of motion. These are shown schematically in Figure 3, which highlights the difference between the structured and dynamical expectations implicit in the generative model (left panels) and the relatively simple dynamics underlying the generation of sensory input (right panels). This emphasises the fact that real behaviour emerges through the expectations and active sampling of the environment that an agent brings to the world: expectations that are embodied in its generative model. It should be emphasised, that despite the complexity of these models, perception and action can be accounted for by one straightforward principle; namely the minimisation of free energy, as in Equation 1.

If we substitute the generative model in Equation 6 into the message passing (generalised predictive coding) scheme in Equation 3, we arrived at the network architecture shown in Figure 4. To lend this architecture a neurobiological plausibility, we have assigned the prediction and error units to neuronal populations in various cortical and subcortical structures. At the sensory level, we have placed sensory prediction error in extrinsic (visual) coordinates in the parietal cortex and the salience (e.g., illumination) of the four target locations in the superior colliculus; cf., [91]. Predictions about the first level hidden states have been divided into proprioceptive (angular position) and affordance states in the motor and premotor cortex respectively; cf., [92]. The motor cortex sends top-down projections to the parietal cortex and spinal-cord to suppress visual prediction errors and elicit motor reflexes respectively. In contrast, the premotor cortex sends top-down predictions about visual salience to the superior colliculus. Predictions about second level causes and states have been assigned to the basal ganglia and prefrontal cortex respectively. These encode the set (sequential or random context) currently inferred. The basal ganglia and prefrontal cortex exchange predictions and prediction errors through cortico-subcortical loops, while the basal ganglia exchanges signals with the premotor cortex to optimise predictions about affordance. The blue arrows arising from the substantia nigra and ventral tegmental area (SN/VTA) are meant to indicate the main (dopaminergic) projections from this area that we assume modulate the postsynaptic gain of the principal cells (red circles) elaborating prediction errors.

The activities of these (nigrotectal, nigrostriatal and mesocortical) dopaminergic projections encode the precision of prediction errors at different levels of the sensorimotor hierarchy. Although the recent literature on the (mesorhombencephalic) nigrotectal pathway, from SN to the superior colliculus, focuses on GABAergic projections, a substantial proportion of nigrotectal projection neurons use dopamine [93], [94], [95].

The model above is sufficient to engender cued reaching movements, which are anticipatory if the agent correctly infers that the cues are presented in a fixed (clockwise) sequence. However, if we reverse the order of the stimuli, there should be accuracy and reaction time costs, due to the fact that the sequence cannot be predicted under clockwise beliefs about the sequence. Furthermore, there should be a set switching cost as the hidden states at the second (context) level are inferred and the itinerant dynamics at the first (affordance) level are suppressed. When we integrated Equation 1, this is precisely what was found:

Figure 5 shows the results of a simulation using log precisions of four (a relatively high precision) throughout the hierarchy. In this example, the target locations appeared every 12 time bins (of 64 ms) using Gaussian bump functions of time. The first five targets were in the (expected) clockwise order, while last five were presented in an anticlockwise order. The resulting conditional predictions and prediction errors are shown in the top four panels of Figure 5, while the trajectory in extrinsic coordinates and the underlying action are shown in the bottom panels (left and right respectively). The upper left panel shows the conditional predictions of sensory signals and sensory prediction errors (in red). These are errors on the salience, proprioceptive and visual input, which, as can be seen, are small in relation to predictions. The predictions were based upon the hidden states shown on the upper right. One can see the itinerant cycling over conditional expectations of hidden affordances (large amplitude lines) that are inferred with a high degree of conditional confidence (the grey areas correspond to 90% Bayesian confidence intervals). The interesting aspect of these results lie in the middle two panels that show the conditional expectations of the hidden causes and states at the second level, encoding the context or set. These results show that it takes about two movements or trials before there is a confident inference that the context has changed. This inferential set switching is driven by the large (downward) deflection in prediction error shown in red (left middle panel). Note that with these precisions, behaviour is accurate and fast and that the violation of sequential expectations is barely discernible.

In other words, the precision of sensory information is sufficient to override top-down prior expectations of a sequential sort, when they are clearly violated. However, as we will see later, there is a performance cost in terms of reaction time and accuracy.

The same simulated neuronal responses are shown in Figure 6, where they are shown alongside their associated brain structure. This figure tries to illustrate how neuronal dynamics unfold at different timescales in different parts of the brain to produce motor behaviour. Crucially, all the hierarchically deployed dynamics are both entrained by and entrain dynamics in lower levels, through the recurrent message passing implicit in generalised predictive coding. By design, we have placed the slower dynamics in higher (more anterior) brain areas [96], [97], [98]–[99], [100]. The neuroanatomical interpretation of this simulation should not be taken too seriously but illustrates the fact that the scheme is (in principle) biologically plausible, both in terms of its dynamical formulation and the functional anatomy of sensorimotor hierarchies in the brain.

In summary, we have created a generative model that illustrates the itinerant and dynamic sensorimotor constructs that might be used by the brain to predict cued sequential behaviours and set switching in response to changing contingencies. It is worth noting, that this relatively simple model has implicitly modelled (and integrated) a number of apparently disparate processes in cognitive neuroscience: for example, Bayes-optimal sensorimotor integration, evidence accumulation, anticipation, short term (working) memory, action selection, set switching and a simple form of reversal learning (in terms of switching to a new contingency). We mean this in the straightforward sense that to perform accurately, the simulated agent has to remember the sequence of cues in terms of delay period activity in the premotor and prefrontal cortex [101], encoded here in terms of conditional beliefs about the dynamics of hidden states. Furthermore, to respond optimally the agent has to recognize a reversal in the sequence of cues and adjust its internal representation of context accordingly. Interestingly, [102] presents a model of working memory using exactly the same winnerless (generalized Lotka-Volterra) dynamics used in this paper. Using this model, they show that working memory capacity has an upper bound of seven items, under plausible assumptions about lateral neuronal interactions. Crucially, the cognitive processes like working memory do not need to be modelled explicitly but emerge from the Bayesian inversion of a generative model. In future work, we will use the same model to study learning and working memory; however, our current focus is on how Bayes-optimal behaviour degrades when we reduce the precision of prediction errors:

In this section, we repeat the above simulations under different levels of precision in the putative targets of dopaminergic projections. This is meant to simulate depleted levels of dopamine; acting at postsynaptic D1 receptors to reduce postsynaptic gain (see Figure 1). First, we reduced the log precision (by 50% in 6 steps) in the principal cells of the superior colliculus that report the prediction errors on the salience of target locations: .

The effects on conditional expectations of hidden states and causes are shown in Figure 7 for high, intermediate and low levels of precision (dopamine). The upper row shows the conditional predictions of sensory input and sensory prediction errors as in Figure 5, while the middle row shows the conditional expectations of the hidden causes encoding context. The most remarkable thing about these results is the failure to infer a change in the context (or set) when dopamine is depleted. This results in an accumulation of prediction error at the sensory level while, in contrast, the prediction error at the second level (red lines in the middle panels) decreases. This is an intuitive consequence of decreasing the relative precision at lower levels of the hierarchical model, which causes the inference to be over reliant upon top-down priors and less confident about switching to the new context, when sensory prediction errors are less precise.

This means that it takes longer before the second level expectations accumulate sufficient evidence to make them switch, following the reversal of stimulus order. At the lowest level of simulated dopamine, this switch fails completely and the agent always expects the next target to appear in the wrong (clockwise) location. The behavioural consequences of this are shown graphically in the lower panels of Figure 7, in terms of the trajectory of movements over the ten cues (trials). We see here that the trajectory is perturbed progressively as dopamine levels fall; with initial directions being pulled in the direction of falsely anticipated target locations. This is shown more clearly in Figure 8, which shows the trajectory for the lowest level of dopamine. During the first five trials the initial excursion from the lower right target is in the correct direction for the next target in the (clockwise) sequence. However, after the reversal, the initial trajectory from the lower left target is drawn towards the incorrectly anticipated next target, requiring a corrective adjustment to the movement trajectory, when the actual target discloses itself.

Figure 9 shows the behavioural consequences of this precision or dopamine-dependent failure to correctly infer the sequential context: the top panel shows the reaction times (assuming 64 ms time bins) measured as the time from the onset of the cue to the time at which the target was reached (to within a radius of ). The corresponding spatial accuracy is shown in the lower panel as a weighted average of the (inverse) distance to target during each trial. There are two important things to take from these results:

First, irrespective of the level of dopamine, reaction times are faster when the next cue can be anticipated. Furthermore, there is a price to be paid for this anticipatory speeding, when sequential anticipations are violated. This is reflected in the increased reaction times at the point of sequence reversal for a couple of trials. It is these transient decreases in performance that index the switching costs hypothesised earlier. Crucially, the effect of dopamine depletion is to exacerbate both the switching costs and the behavioural slowing when sequential predictions no longer hold. In the limit of very low dopamine, and a complete failure to switch sets (infer a context change), there is a marked impairment in performance that persists following reversal. Perhaps the most important result in Figure 9 is that the set switching costs persist for longer with low levels of dopamine. In other words, there is a perseveration of (suboptimal) anticipatory motor trajectories that is exacerbated by dopamine depletion. This latent bradykinesia and perseveration is reminiscent of the symptoms of Parkinson's disease [103], [104], which was the motivation for these simulations.

On the basis of these results, one might predict that the greatest difference between Parkinson's patients on and off dopaminergic medication would be expressed most acutely in trials that violated expectations established during sequential cueing. Conversely, there should be relatively small differences in reaction times when stimuli are presented in the correct sequence: cf., [105]. Furthermore, differences in reaction times with unpredictable cues should not be marked, once patients have realised that there is no underlying sequence. We will consider these predictions in relation to empirical results in a forthcoming paper. It is also interesting to relate these simulations to the results in [106], who found deficits in probabilistic reversal learning in Parkinson's disease, where “patients also exhibited compromised adaptability to the reversal”. Brown and Marsden [107] investigated set switching in Parkinson's disease using the Stroop task. Subjects had to report either the semantic or physical colour of a word; however, the rule changed every ten trials. The response dimension was cued before each trial or subjects were just reminded to change the rule every ten trials. Patients showed general psychomotor slowing but were further impaired on the uncued condition, especially in the first trial following a rule change.

Finally, we repeated the simulated lesion experiments above by reducing the precision in other cortical and subcortical structures in receipt of SN/VTA projections. The most interesting results are shown in Figure 10 in terms of simulated reaction times over different levels of dopamine: the left panels reproduce the reaction time data of the previous figure, while the middle panel shows the equivalent results obtained when depleting dopamine in the motor cortex (encoding conditional expectations about proprioceptive inputs). The effect of dopamine depletion here is to increase reaction times in a non-specific way. This non-specific slowing was expected, as proprioceptive prediction errors are subverted thereby reducing motor vigour; cf., [108]. Note that inference about affordance and set are not affected, because these are driven by exteroceptive prediction errors. This means there is no change in set switching or perseveration. Conversely, when we lesion mesocortical projections to the premotor cortex (modulating prediction errors about changes in affordance) there are effects on both motor vigour and set switching (right hand panels). Importantly, these effects differ from those produced by the same lesion to the superior colliculus. Here, depleting dopamine actually decreases reaction times. This is perfectly sensible because increasing the precision at higher levels of a hierarchical model has the opposite effect to increasing the precision of lower level prediction errors. In other words, as dopamine levels in the premotor cortex increase, the agent becomes overly confident about its top-down (empirical) prior expectations, even in the face of precise sensory information. This overconfidence is manifest in terms of a slight impairment in reactions elicited by sensory cues.

However, the overconfidence about affordance enables the second level of the generative model to recognise the change in context more efficiently (quickly). This induces a trade-off between the efficiency with which cues elicit movements and the efficiency of set switching.

In this section, we have seen how simulated dopamine depletion is manifest in terms of neuronal responses encoding Bayes-optimal inferences about sensorimotor contingencies and in terms of behavior. The key point made by the simulations is that although dopamine may have a singular mechanism of action and computational function (e.g., to modulate postsynaptic gain and encode precision) the physiological and behavioral correlates of dopamine changes depend on where in the brain they are expressed. This reiterates the point made in the introduction that understanding the role of dopamine may call for a multilateral perspective that accommodates the delicate balance among distributed responses that underlie the functional anatomy of behavior. The simulations in this section can be regarded as a proof of principle that a single mechanism can lead to the diverse functional consequences seen empirically [109], [7], [110]. Here, dopamine had opposite effects on the speed of movements (bradykinesia) depending upon whether it was depleted at higher or lower levels of the sensorimotor hierarchy. Conversely, simulated depletion of dopamine in the superior colliculus (low-level) and premotor cortex (high-level) had similar effects on perseveration. We emphasize this point because it has implications for the computational modeling of dopamine, especially for theoretical accounts of dopamine that consider one optimization process in isolation (for example, associating dopamine with reward prediction error). A nice example of the plurality of deficits following insults to be dopamine system is provided by [111], who conclude that Parkinson's “patients on and off medication both showed attentional shifting deficits, but for different reasons.

Deficits in non-medicated patients were consistent with an inability to update the new attentional set, whereas those in medicated patients were evident when having to ignore distractors that had previously been task relevant.”

In summary, contrary to what is often assumed, dopamine may not report the prediction error on value but the value (precision) of prediction errors. If this is the case, one would anticipate different behavioral deficits following dopamine depletion, depending on which prediction errors were affected. Strategically, it may be better to ask not what the function of dopamine tells you about a model but what a model tells you about the function of dopamine. In this paper, the function of dopamine is to modulate postsynaptic gain, while the model mediates between this (neuronal) function and its behavioral and neurophysiologic consequences. See also [11].

In this paper, we have presented a simple model of cued reaching movements and set switching that is consistent with the notions of salience and affordance. Furthermore, we have simulated some latent symptoms of Parkinsonism by reducing the precision of cues that have affordance. Reducing this precision (dopamine) delays and can even preclude set switching, with associated costs in behavioral accuracy. When the precision of sensory cues is removed completely, we obtain autonomous behavior that is prescribed by the itinerant expectations of the agent (results not shown). Crucially, these simulations are not based on an ad hoc model of dopaminergic function but use exactly the same principles, equations and numerics used previously to address a wide variety of processes in cognitive neuroscience: Table 1 lists the growing number of paradigms and processes that can be explained in terms of free energy minimization and hierarchical Bayesian inference. The simulations in this and related papers can be reproduced from a graphical user interface available in Matlab code (http://www.fil.ion.ucl.ac.uk/spm/).

In short, we have focused on the role of dopamine in balancing the influence of bottom-up sensory information and top-down (empirical) prior expectations during perceptual inference and consequent behavior. This role is consistent with many other theoretical treatments of dopamine in modulating top-down effects and signal-to-noise [49], [44], [11], [112], [2], [113], [45], [39], [40]; and contextualizes them within the formal setting of generalized predictive coding.

The simulations presented in this paper do not attempt to cover all the physiological and computational processes that dopamine may mediate. Our results pertain to classical neuromodulatory postsynaptic consequences of changes in tonic dopamine release (e.g., during depletion due to drugs or disease) on principal cells. There are many aspects of dopaminergic function that are not addressed by these simulations: for example, modulation of long-term synaptic plasticity, or the differential roles of various pre- and postsynaptic dopamine receptors. Furthermore, we have only considered the effect of tonic or baseline dopamine levels (over long time scales) and have ignored dopamine fluctuations on a shorter time scale: for example, the phasic release of dopamine in response to specific cues or its rapid clearance in the striatum by the dopamine active transporter. However, the aim of the present work was to highlight the computational, neurophysiologic and anatomic considerations that suggest a unitary role for dopamine in the encoding of precision.

There is a large literature on modeling dopamine in the context of choice and motor behavior [114], [115], [116], [117], [118], [119]. Some of this literature is based on reinforcement learning and optimal decision theory; and formulated at a rather abstract level in terms of discrete state spaces and time. It is difficult to connect the (predictive coding) process model provided in this paper with the more descriptive (but useful) heuristics provided by reinforcement learning and economics. One link may be via variational Bayesian formulations of reinforcement-learning update equations (see below), in which precision-weighted prediction errors play a key role [120]. On the other hand, there are computational simulations whose biological detail goes beyond the simulations in this paper [112], [121], [122], [123]. In this sense, the simulations reported here should be regarded as an illustration of how far one can get in modeling neuromodulation in the larger setting of Bayes optimal action and perception. In this context, Bayes optimality (implicit in free energy minimization) replaces optimal control (implicit in reinforcement or value learning).

The ideas and simulations in this paper just show that dopamine fits comfortably into a principled if somewhat abstract formulation of action and perception. Specifically, it has the physiological characteristics that would be required to fulfill a central role in Bayes-optimal inference; namely, to encode precision or uncertainty. This is why we have focused on the neuromodulation of postsynaptic gain in cortical cells, mediated by D1 receptors. The theoretical constraints we have considered do not really allow us to say very much about the functional anatomy of the cortico-basal ganglia-thalamic loops. Having said this, the dynamics necessary for active inference are not inconsistent with many aspects of known functional anatomy. For example, the basal ganglia are believed to act on cortex through a process of focused disinhibition [124]. In particular, the basal ganglia output nuclei fire tonically to inhibit cortical activity, but strong striatal activity can reduce this tonic activity to release target thalamic areas and their corresponding cortical areas from inhibition. This basic functional architecture is reflected in our simulations by the selection of an appropriate sensorimotor set by representations of hidden causes in the striatum that selectively enable particular attractors or central pattern generators in premotor cortex. Note that focused disinhibition is a necessary consequence of the dynamics on hidden states and causes that mediate winnerless competition. In one sense, it is difficult to imagine any alternative sort of dynamics that could prescribe unique sequential behavior and motor trajectories.

Although the computations that minimize free energy do not predict the details of functional anatomy (in the same way that maximizing adaptive fitness does not predict the details of a phenotype), it does provide some interesting insights into large-scale functional anatomy. These insights rest upon the notion that the brain is a generative model of its environment. This means that causal regularities in the environment should be recapitulated in its neuroanatomy. The trick here is to think of the brain as generating predictions of sensory input using its backward (top-down) connections; in other words, to imagine the brain without the forward connections that are used for perceptual inversion of its generative model. The picture that emerges is a fine lacework of cortical projections from high-order associative cortex that radiate to sensory cortical areas and then send top-down predictions of sensory input to thalamic nuclei. A nice example here is the distinction between what and where pathways in visual processing [125]. Their very existence suggests that the (hidden) causes of visual inputs to the lateral geniculate nucleus are objects that can belong to different categories and, crucially, can be in different places. One might imagine that in a static universe, where objects were bound to a particular place, the brain would not have separate representations of what and where or their neuroanatomical correlates. So what does this sort of analysis imply for the basal ganglia? We will focus on three specific predictions:

First, the fact that the outputs of the basal ganglia are to thalamic nuclei suggests that the basal ganglia are at a hierarchically lower level than prefrontal cortex. For example, the ventral lateral nucleus receives inputs from the basal ganglia (via the thalamic fasciculus) and sends outputs to the primary motor cortex. The ventral lateral nucleus is juxtaposed to the ventral posterolateral nucleus in receipt of proprioceptive and somatosensory afferents via the posterior column-medial lemniscus pathway. The implicit hierarchical level of the basal ganglia suggests that the prefrontal cortical projections to the striatum should be of the backwards type (originating from deep pyramidal cells). Indeed, the cortical pyramidal neurons projecting to the striatum are located in layers II–VI, but the densest projections come from layer V [126]. Furthermore, the medium spiny neurons that receive these corticostriatal projections express dopaminergic (D1 - direct and D2 - indirect pathway) receptors [127]; as would be expected, if they played the role of prediction error units that are targeted by backward connections.