Free energy, value, and attractors

Evolutionary theory [61, 62] suggests that species (random attractors) do not arise de novo but evolve through natural selection (e.g., by punctuated equilibria or phyletic gradualism; [63, 64]). We take this to imply that policies are heritable and can be encoded (epigenetically) in terms of value or cost-functions. We will assume the agents are equipped with a cost-function that labels states as attractive or not

(22)c(x ∣ m)≤0: x∈𝒜=⋂ω˜∈Ω𝒜(ω˜),c(x ∣ m)>0: x∉𝒜. Technically, cost indicates whether each state is in a kernel or the set of fixed points of a random attractor [65]. In the deterministic limit Γ → 0 this kernel reduces to an attractor in the usual sense. From now on, we will use 𝒜 ⊂ 𝒪 to mean the kernel of a random attractor or an attractor in the deterministic sense. The introduction of cost allows us to connect attractors in dynamical systems with attractive states in reinforcement-learning and optimal control. Informally, cost labels states as either attractive (e.g. sated) or costly (e.g., thirsty). The cost-function could also be regarded as a characteristic function that indicates whether the current state is characteristic of the class the agent belongs to. This labelling is sufficient to prescribe policies that assure equilibrium solutions, as evidenced by the existence of evolved agents. This question is how? We will begin by considering control theory.

In this section, we look at policies and value from the point of view of optimal control theory and reinforcement-learning. In the previous section, value was considered to arise from a decomposition of flow into curl, and divergence-free parts. In that setting, value simply reports the surprise that a state is occupied. In other words, value is an attribute of the policy. Optimal control theory turns this around and assumes that the policy is an attribute of value. This enables policies to be specified by value, via cost-functions. In this section, we will consider optimal control theory as optimising policies (flows), whose attracting fixed-points are specified by cost-functions. Crucially, because optimal control policies do not specify divergence-free flow, they can only specify policies with attracting fixed points (the maxima of the value function). In the next section, we turn to generalised policies that exploit divergence-free flow to support itinerant policies. We will persist with continuous time formulations in this section and provide discrete time versions of the main results in the appendices.

In optimal control theory and its ethological variants (i.e., reinforcement-learning), adaptive behaviour is formulated in terms of how agents navigate state-space to access sparse rewards and avoid costly regimes. The aim is to find a (proximal) policy that attains long-term (distal) rewards. In terms of the previous section, a policy f = Γ · ∇V + ∇×W was specified via the scalar potential or value V(x˜) also known as (negative) cost-to-go. In optimal control theory, value is defined as the expected path-integral of cost. More formally, the cost-to-go of x˜0∈X is the cost expected over future times t ∈ [t 0, ∞], starting with a point density p(x˜,t0∣m)=δ(x˜0), which evolves according to (3) (see Appendix E),

(23)V(x˜0)=−∫t0∞∫c(x˜)p(x˜,t ∣ m)dx dt⇒c(x˜)=f·∇V(x˜)+∇·Γ·∇V(x˜). Or in the deterministic limit Γ → 0

(24)V(x˜0)=−∫t0∞c(x˜(t))dt⇒c(x˜)=f·∇V(x˜)=V˙(x˜(t)). This definition of value as an expected path-integral of cost (first line) allows cost to be expressed as a function of value (second line). It says that cost is the expected increase in value. This may sound counterintuitive but makes sense if one considers a reward now means less in the future (i.e., a decrease in the value of the next state). Crucially, (24) shows that the maxima of the ergodic density can only exist where cost is zero or less (cf. (22)): at a maximum of p(x˜∣m)=exp⁡(V) we have the following:

(25)∇V(x˜)=0∇·Γ·∇V(x˜)≤0}⇒c(x˜)≤0, with c(x˜)=0 in the deterministic limit. Put simply, costly regions induce value gradients that guide flow towards points where there is no cost (i.e., no gradients). In this sense, value is sometimes called a navigation function. This means that, in principle, we have a way to prescribe ergodic densities with maxima (attracting fixed-points) that are specified with a cost-function. Equation (23) shows that the cost-function can be derived easily, given the policy and implicit value-function. However, to specify a policy with cost, we have to derive the flow from the cost-function. This entails associating a unique flow with the value-function and solving (24) for value: this association is necessary because optimal control does not specify the divergence-free part of the policy. Solving (24) is the difficult problem optimal control and value-learning deal with.

Let optimal control be denoted by π(x˜), where optimal control maximises the ascent of the value-function

(26)π(x˜)=arg max⁡u f(x˜,u)·∇V(x˜). This extremal condition associates a unique (optimal) flow with every value-function such that value can be expressed in terms of cost using (24)

(27)max⁡u f(x˜,u)·∇V(x˜)=f(x˜,π)·∇V(x˜)=c(x˜). This is the celebrated Hamilton-Jacobi-Bellman (HJB) equation. More general forms are provided in Appendix F for the interested reader. The basic problem, posed by the solution of the HJB equation for value, is that the value-function depends on optimal control, so that future cost can be evaluated. However, optimal control depends on the value-function. This circular dependency can only be resolved by solving the self-consistent equations above, also known as the dynamic programming recurrence. This is the raison d'être for value-learning.

In engineering, planning, and control problems, the HJB equation is usually solved by backwards induction (staring at the desired fixed-point and working backwards). However, this is not an ethological option for agents that have to learn the value-function online. An alternative solution exploits the fact that the expected increase in the value of the current state is cost. This leads to a straightforward value-learning scheme

(28)V(x(t))⟵V(x(t))+δ:δ=V˙(x(t))−c(x(t)). Such that, at convergence, the value-function satisfies the deterministic limit of (24), at least for the states visited,

(29)δ(t)→0⇒c(x)=V˙(x(t))=f·∇V(x). Heuristically, this scheme erodes the value landscape, creating gradients that ensure flow through costly regions of state-space. The only points that are exempt from this erosion are maxima with zero flow and cost (or negative cost in the presence of fluctuations). These are the fixed-points of the attracting set prescribed by the cost-function.

In (28), δ(t) reports the difference between the cost predicted V˙(x(t)) and observed c(x(t)). This is the negative cost or reward prediction error. There is a vast literature on reinforcement-learning schemes that solve the discrete time version of the HJB equation; either by backwards induction (model-based schemes) or by using reward prediction error (model-free schemes). Model-free schemes use a discrete time version of (28) using the Robbins-Monro algorithm; [66, 67]. Appendix G provides a brief survey of these schemes. Intuitively, they all involve increasing the value of the last state in proportion to the reward prediction error. This is the basis of temporal difference schemes [2] and Rescorla-Wagner [68] models of conditioning in psychology. See [5] for a comprehensive review. If the agent has no model of flow or state-transitions, similar schemes can be invoked to optimise the policy, (e.g., actor-critic schemes). A generalisation of value-learning, called Q-learning, considers a value or quality Q : X × U → ℝ on the joint space of states and control [69]. Q-learning does not need a model in the form of probabilistic transitions to optimise control, because the quality of an action is encoded explicitly. Perhaps the most important thing to come out of these modelling initiatives is that phasic dopamine discharges in the brain are a prime candidate for reporting reward prediction error [3, 70, 71]. In some cases theoretical predictions preempted empirical findings; for example, “in the absence of an expected reward, value system responses should show a decrease in activity at the time when the reward would normally be delivered” [72], which was confirmed subsequently [73].

In summary, one can decompose any policy or expected flow into a part that is divergence-free and a part that increases value, where value is negative surprise or the log probability of a state being occupied. This means, given expected flow and the amplitude of random fluctuations about that flow, one can compute the ergodic density and associated value-function. Furthermore, if one defines surprise (negative value) of any state as the expected cost accumulated from that state, then it is straightforward to evaluate the cost-functions implied by any given policy. An example is given in Figure 4 using the Lorentz attractor in previous figures.

Using this definition of cost, reinforcement-learning and optimal control theory try to derive value from cost-functions by assuming controlled flow minimises accumulated cost. This involves solving self-consistent equations that entail these assumptions. The ensuing value-function guides flow to ensure cost is minimized under constraints on motion. In dynamical terms, this approach optimises a policy in terms of its scalar potential, whose maxima coincide with points in state-space where c(x) ≤ 0. However, despite its prominence in the neurosciences, optimal control theory is not very useful for understanding self-organisation in biological systems. For example, one could not specify the policy followed by the Lorentz attractor in Figure 4, because its dynamics do not have detailed balance. In other words, although one can derive a cost-function from any flow, one cannot specify any flow with a cost-function: to specify any given policy one would need the vector potentials (or anti-symmetric matrices) above.

Furthermore, optimal control schemes and related heuristics have several shortcomings: (i) they are notoriously slow, requiring hundreds if not thousands of trials to learn even the simplest value-function, (ii) value-learning based on stochastic iteration depends on the same random fluctuations that need to be suppressed to pursue the policy, (iii) optimal control theory says nothing about exploration of state-space, (iv) an exigent limitation of these schemes is that they only account for policies with stationary fixed-points (i.e., agents who would optimally do nothing). This means they cannot account for the itinerant and context-sensitive nature of real behaviour. To resolve these problems we now turn to generalised policies that include divergence-free flow and are constrained directly, as opposed to placing constraints on value functions.

In the previous section, cost was used to specify policies or expected flow in terms of value-functions. However, policies with detailed balance (of the form f = Γ · ∇V) place severe constraints on the attractors they engender. In the deterministic limit, they can only prescribe attractors with a single fixed-point (per basin of attraction), which is a local maximum of value V : X → ℝ. This is not a useful description of real agents that exhibit itinerant dynamics with quasiperiodic and chaotic behaviour. In what follows, we consider policies in generalised coordinates of motion that do not have detailed balance (and exploit divergence-free flow); we will refer to these as generalised policies. These policies provide a rich repertoire of dynamics with attracting sets that can be specified directly by policies. In other words, we dispense with the assumptions of optimal control theory and consider the more general problem of how to specify attracting sets. In what follows, cost-functions are used to specify (unstable) fixed points for fairly generic flows. We will see how this leads to a dynamical formulation of policies and the emergence of optimal itinerancy. This section focuses on the basics and provides some simple examples, noting that there are many potential extensions; some of which have already been addressed in the application of dynamical systems theory to biological systems (see below).

Generalised policies rest on the ensemble dynamics perspective: we start by considering how cost can restrict the probability mass of an ergodic density to a subset of state-space 𝒜 ⊂ X. The Fokker-Planck equation (3) provides a fundamental constraint on flow that must be satisfied when Λp = 0 and

(30)∇·Γ∇p=∇·(fp)=f·∇p+p∇·f⇒p(x˜ ∣ m)=∇·Γ∇p−f·∇p∇·f. This straightforward result shows that as divergence ∇·f decreases, the sojourn time (i.e., the proportion of time a state is occupied) rises. This means divergence decreases value and increases surprise. This is intuitive, in that divergence represents the extent to which flow behaves like a source or a sink at a given point. Attractive or valuable points in state-space are sinks with low divergence. At the peaks of the ergodic density its gradient is zero and its curvature is negative:

(31)p>0∇p=0∇·Γ·∇p<0}⇒∇·f<0. This means divergence must be negative. This generalises an almost trivial result in deterministic systems: divergence is the sum of the flow's real Lyapunov exponents ∇·f=tr⁡(∂x˜f)=∑iRe(λi), where λ i are the eigenvalues of the Jacobian ∂x˜f. If divergence is negative, then all the Lyapunov exponents are negative, implying a stable fixed-point attractor.

This provides a straightforward way of ensuring the peaks of the ergodic density lie in, and only in 𝒜 ⊂ X. This is assured if ∇·f < 0 when x˜∈A and ∇·f ≥ 0 otherwise. We can exploit this using the equations of motion in (21)

(32)f(x,x′)=[x′c(x)·x′−∂xφ]⇒∇·f=c. This flow describes the classical (Newtonian) motion of a unit mass in a potential energy well φ(x), where cost plays the role of negative dissipation or friction. Crucially, under this policy or flow, divergence is cost. This means the associated ensemble density can only have maxima in regions of negative (divergence-based) cost. This provides a means to specify attractive regions 𝒜 ⊂ X by assigning them negative cost, which brings us back to (22),

(33)c(x)≤0: x∈𝒜,c(x)>0: x∉𝒜. Put simply, this scheme ensures that agents are expelled from high-cost regions of state-space and get “stuck” in attractive regions. Equivalently, cost can be regarded as destroying unattractive fixed points at the minima of the potential landscape φ(x). It should be noted that negative divergence does not ensure an attractive fixed point; however, if divergence is sufficiently negative, the point will act as a sink and become, almost surely, part of the attracting set. We will see an example of this below. In summary, in attractive regions with low cost, flow slows down sufficiently to increase sojourn time and implicitly value. This is not dissimilar to Win-Stay, Lose-Shift strategies used to model optimal decisions in game theory [74]. We will now illustrate how divergence-based cost works using the mountain car problem.

Here, we use active inference and a generative model based on (31) to solve a fairly difficult problem in optimal control based purely on the theoretical treatment above. Crucially, the agent that solves this problem has no prior information about constraints on its action and yet it can respond adaptively, when supplied with a cost-function, to find its goal almost immediately. Note that there is no value-learning because we use divergence-based cost, which does not require a value-function. Furthermore, this behaviour is resilient to perturbations, because the policy provides predictions, which are fulfilled by action.

In the mountain car problem, one has to park a mountain car halfway up a hill. However, the car is not sufficiently powerful to ascend the hill directly. This means the only solution to problem is to back away from the parking location and then accelerate towards it, in the hope of acquiring sufficient momentum to access the target. The upper left panel of Figure 5 shows the landscape or potential energy function (with a minimum at position, x = −0.5) that exerts forces on the car. The car is shown at the target position at the top of the hill at x = 1 (red dot). The equations of motion of the car are shown below. Crucially, at x = 0 the force is unity and cannot be overcome by the agent, because a squashing function −1 ≤ σ(a) ≤ 1 is applied to action. The right panels show the agent's generative model in terms of the equations of motion in (13). These correspond to (31), where the cost-function is shown on the upper right. Here, the cost-function c(x, z) has an auxiliary parameter z ∈ ℝ that enables cost to be switched on or off. This can be thought of as satiety, such that when z = 0 cost is positive everywhere, except the target location (see figure legend for details). In this example, the sensory mapping and its assumed form were just g(x˜)=g(x˜)=x˜. Notice that the true equations of motion depend on action while the policy does not. This means the actual behaviour of the agent is selected from a family of flows by action. Figure 6 shows two exemplar flows under different values for action. Under active inference, action tries to realise conditional beliefs that are specified by the policy or empirical priors on motion.

Figure 7 shows how paradoxical but adaptive behaviour (moving away from a target to ensure it is secured later) emerges from this sort of generalised policy on the motion of hidden states, using c(x, 0). These simulations are the results of integrating (1) and (2) (see Appendix H for details). The inferred hidden states (upper right) show that the car explores its landscape until it encounters the target and negative cost or friction increases dramatically to prevent it escaping (i.e., falling down the hill). This ensuing trajectory is shown on the upper left panel. The paler lines provide exemplar trajectories from other trials, with different starting positions. In the real world, friction is constant (one eighth). However, the car expects friction to change with position, enforcing exploration or exploitation. These expectations are fulfilled by action (lower right).

It is important to appreciate what has and what has not been achieved in this simulation: using a single scalar cost-function of position we have been able to induce adaptive goal-directed behaviour without any value-learning or enforced exploration of the environment. This behaviour is elicited immediately without the need for repeated trials or exposures. Furthermore, the requisite exploration and exploitation is manifest as a direct consequence of the agent's priors, without the need for carefully calibrated stochastic terms during training. This sort of adaptive behaviour is reminiscent of foraging seen in insects and other biological agents; where the environment is explored under autonomous dynamics until some reward is encountered. In an experimental context, the above simulation could be regarded in terms of eliciting foot-shock escape behaviour [75]; in that the uniformly high cost can only be avoided by escaping to a low-cost location. It may seem implausible to specify behaviour in terms of cost that is generally high everywhere; however, one can see easily how drive states might correspond to high cost and subsequent foraging or seeking behaviour [76, 77]. Mathematically, this reflects the fact that cost plays a permissive role, in that it ensures maxima of the ergodic density lie in low-cost regions of sate-space by precluding maxima elsewhere. In this sense, the emphasis is on autovitiation of costly fixed points, as opposed to the autopoiesis of attractors.

What we have not addressed here is learning: we have assumed that the agent has already learned the potential energy function φ(x) that corresponds to environmental constraints on its motion: see [6] for an example of this learning. However, there is a subtle but important point about learning: learning the parameters of the potential function corresponds to learning divergence-free flow, which does not affect the ergodic density or fixed points of the attractor. This contrasts with value-learning, in which divergence-flow is unspecified and the parameters of the value-function are learned. We now look at generalising divergence-based schemes and their role in prescribing sequential and itinerant dynamics.

There are clearly many different ways in which we could formulate generalised policies and constrain them with cost-functions: We will concentrate on the use of cost to specify itinerant policies: itinerancy is important because it provides a principled explanation for exploration and foraging in ethology [78]. Furthermore, it provides a key connection to dynamical systems theory approaches to the brain [79] that emphasise the importance of itinerant chaos [80], metastability [81], self-organised critically [82], winnerless competition [83], and attractors [84]. Similar constructs based on metastability have been invoked to understand the dynamics of molecular biology and the emergence of disease states like cancer [85]. The common theme here is the induction of itinerancy though the destruction of fixed-points or the gradients causing them [86]. The ensuing attractor ruins or relics [87] provide a framework for heteroclinic orbits that are ubiquitous in neurobiology, in electrophysiology [88], cognition [89], and large-scale neuronal dynamics [90].