Dopamine, affordance and active inference

Second, in active inference, the balance between top-down prior beliefs and bottom-up sensory information is controlled by the relative precision of prediction errors at higher and lower levels of cortico-basal ganglia-thalamic hierarchy. This suggests that there will be an antagonistic control of dopaminergic projections to the prefrontal cortex (higher level) and equivalent projections to the striatal (lower-level). In anatomical terms, this may be reflected in the segregation of the cells of origin of the nigrostriatal dopamine projections (from the substantia nigra) and the cortical projections (from the ventral tegmental area). Clearly, there are several important functional associations in dopaminergic systems that may or may not be understandable at this level of analysis. Key examples here include the opposing effects of the direct and indirect pathways [128], [129] and the opposing influences of dopamine on synaptic plasticity [56].

Finally, it is interesting to note that the number of parameters of statistical models is generally much greater than the degrees of freedom of their variance or precision estimates. For example, a classical statistical model with both between and within subject factors can have hundreds of parameters yet can be inverted using just two precisions (modeling within and between subject variability). This might be a simple explanation for the fact that the number of dopaminergic cells (encoding precision or context) is much smaller than the myriad of cells (encoding hidden causes or content) needed to parameterize the world.

The current work presents a covert challenge to standard theories of decision-making and motor control based upon the concept of reward or value. As hinted at in the introduction, there is an inherent circularity in reward-based accounts of behavior that may obscure deeper questions about behavior. This is because (psychological) reward is defined as a process that reinforces or modifies behavior. As such, it is circular to use reward to explain behavior. Although the concept of reward may provide beautifully self-consistent descriptions of behavior in terms of optimizing expected reward; e.g., [20], reward-based accounts (and their neurophysiologic correlates) cannot explain behavior per se. The alternative and simpler approach, provided by active inference, replaces rewards with prior beliefs about how the world should unfold and motivates these priors in evolutionary and ethological terms. Put simply, survival does not depend on seeking out rewards; it depends upon avoiding surprising encounters and physiological states that are uncharacteristic of a given phenotype. It is therefore sufficient to minimize surprise (free energy), which is a problem of (active) inference, not reinforcement learning.

At no point in the formulation above do we refer to reward or value; the only optimization was to minimize surprise about expected outcomes. In the context of hierarchical inference, the potentially potent role of dopamine is obvious. This is because the delicate mixing of bottom-up sensory information, with top-down priors, to produce conditional beliefs (and action) is exquisitely sensitive to the precision or certainty ascribed to representations at different levels of a hierarchy. This is a generic and ubiquitous aspect of hierarchical inference; for example, inference about treatment effects in group studies rests on veridical estimates of within-subject, relative to between-subject variability (precision). The idea here is that dopamine reports the precision (variability) in hierarchical models used by the brain to infer the causes of sensory data. In this view, the task instructions and cues provided to subjects in decision-making experiments do not, in themselves, constitute rewards but are used to instill prior beliefs about how they should behave.

Although we have questioned the epistemological status of reward on the grounds of circularity, one cannot deny that certain cues are inherently rewarding; for example, those induced by appetitive stimuli. In this sense, one might associate reward with the affordance of (hidden) causes in the environment that elicit obligatory volitional and autonomic responses. In this context, the tautology of reward is resolved by noting that reward is a perceptual attribute of a cue not a cause of behavior: in active inference, behavior is caused by prior beliefs and ensuing exchange with environment. These beliefs ensure certain states are occupied frequently, where these states are, by definition, valuable. In this view, value and reward are consequences not causes of behavior. Behavior is caused by the environment and prior beliefs that are established epigenetically or through experience dependent learning. This means that reward is a perceptual (hedonic) consequence of behavior, not a cause.

The argument in this paper is that the information (prediction error) conveyed by rewarding cues cannot be encoded by dopamine, because dopamine cannot excite postsynaptic the responses that would be needed to mediate the influence of that information: Dopamine can only modulate the postsynaptic responses to glutamatergic or other neurotransmitter release. This is not to say that rewarding cues will not excite dopamine cells. Indeed, this is the hypothesis put forward here – namely that rewards or cues with particular affordance can select exteroceptive, interoceptive and proprioceptive processing channels that mediate behavioral responses. In short, dopamine may report the precision or confidence about reward, not reward per se. This account may explain why dopaminergic responses do not behave as reward prediction errors generally. For example, a significant proportion of dopamine neurons increase their firing to aversive stimuli, and cues which predict them [130], [131]. This is the opposite behavior to that predicted by the reward prediction error hypothesis, but follows naturally from the precision encoding framework, since both aversive and appetitive cues signal predictable sensorimotor sequelae. The precision hypothesis also accounts for the finding that dopaminergic neurons fire in response to stimuli that predict the subsequent availability of predictive information about upcoming reward, even though such stimuli convey no information about the reward itself [132]. On the account presented in this paper, such stimuli elicit dopaminergic activity because they signal the onset of a predictable sequence of events. Accounting for all observed patterns of dopaminergic activity falls outside the scope of this paper, but it is clear that the precision hypothesis predicts the involvement of dopamine in a broader range of processes than some existing accounts.

The simulations in this paper focused on the depletion of dopamine. However, increasing levels of, or sensitivity to, dopamine in the basal ganglia has been implicated in several movement disorders, including tardive dyskinesia [133], Huntington's disease [134], Tourette syndrome [135], hemiballismus [136] and levodopa-induced dyskinesia [137]. These disorders produce involuntary, non-purposeful movements that are nonetheless more structured than simple myoclonic jerks or tremor. If dopaminergic transmission was increased pathologically in lower levels of the hierarchy, fluctuations in ascending (afferent) prediction errors could become sufficiently precise (potent) to trigger movement; in other words, subliminal cues would be awarded aberrant affordance and elicit inappropriate action.

Although we have not dealt with the phasic release of dopamine and the implicit optimization of precision, the current formulation does make some strong predictions about the modulation of stimulus bound or event related responses. If tonic dopaminergic levels modulate postsynaptic gain, one would expect to see amplified responses in the post synaptic targets of dopamine projections that report (sensory) prediction errors. This means that one might expect greater evoked responses to the same stimuli when presented in a predictable, as opposed to an unpredictable context. Indeed, this is the basis of an explanation for the mismatch negativity: an increase in the event related potential elicited by a novel or oddball stimulus, presented in a predictable train of stimuli. The proposed mechanism rests on prediction errors that are afforded too much precision [138].

The central role of precision-weighted prediction errors also emerges from other computational perspectives. For example, one can formulate dynamic inference about the causes of sensory cues without reference to neurobiology but as a generic hierarchical Bayesian model [120]. Variational Bayesian methods then yield the same sort of precision-weighted prediction errors as in the present work. Crucially, the variational update equations are formally related to reinforcement learning rules; where precisions play the role of learning rates. This means that one could test whether dopaminergic drug manipulations affect estimates of precision (learning rates) based on empirical behavioral responses during classical reinforcement learning paradigms.

We have not touched upon learning per se, because we have assumed that the prior beliefs used to generate behavior were already known in the simulations. In future work, we will use the simulations above to look at the acquisition of prior beliefs in terms of sequence learning and the effect that dopamine has on this learning. The key issue here is that any effects of dopamine on learning and activity-dependent plasticity (using free energy minimization) are mediated vicariously through its effects on neuronal activity. This means that the only thing specific about dopamine, in the context of learning, is due to the restriction of its projections to the (extended) motor system. This suggests that sensory learning will be largely unaffected by dopaminergic manipulations (simulated or real) to the extent that it is independent of motor learning. We will explore this in future work, in which we will treat (phasic) dopamine release as encoding state-dependent precision, given contextual cues like conditioned stimuli: see [132]. Formally, this is closely related to the use of state-dependent precision to understand attentional gain in perception [30].

One interesting aspect of the simulations in this paper is that they lend themselves nicely to higher-order (operant) learning paradigms: The behavior generated by high-level sensorimotor constructs does not distinguish between sensations caused by motor acts or vice versa, because distinct exteroceptive and motor predictions only arise at lower (sensory) levels of the hierarchy. In other words, we could regard the simulations above as simulations of actions that disclose salient cues (as opposed to salient cues causing action). This perspective may be useful when trying to understand the central role of dopamine in action selection based upon conditioned stimuli on the one hand, and optimizing behavior to access conditioned reinforcers on the other.