Free energy, value, and attractors

In this section, we introduce the free energy principle as a means of minimising the conditional entropy of an agent's states through action. As noted above, these states and their entropy are hidden from the agent and can only be accessed through sensory states. This means that action cannot minimise the entropy of hidden states directly. However, it can do so indirectly by minimising the entropy of sensory states,

(8)a∗=arg min⁡ aℋ(X ∣ m)=arg min⁡a ℋ(S ∣ m). This equivalence follows from two assumptions: there is a diffeomorphic mapping between hidden and sensory states and that Jacobian of this mapping (i.e., the sensitivity of sensory signals to their causes) is constant over the range of hidden states encountered (see Appendix B). Crucially, because sensory entropy is the long-term average of sensory surprise, the extremal condition above requires action to minimise the path integral of sensory surprise. This means (by the fundamental lemma of variational calculus) for t ∈ [0, T]

(9)δaℋ(S ∣ m)=0⇔∂a(t)ℒ(s˜(t))=0⇔a(t)∗=arg min⁡ a(t)ℒ(s˜(t))ℋ(S ∣ m)=1T∫0Tdtℒ(s˜(t))ℒ(s˜(t))=−ln⁡p(s˜(t) ∣ m). Equation (9) says that it is sufficient for action to minimise sensory surprise to minimise the entropy of sensations (or at least find a local minimum). This is sensible because action should counter surprising deviations from the expected flow of states. However, there is a problem; agents cannot evaluate sensory surprise ℒ(s˜(t)) explicitly, because this would involve integrating p(s˜,x˜,θ∣m) over hidden states and parameters or causes: ϑ=(x˜,θ). This is where the free energy comes in.

Free energy is a functional of sensory and internal states that upper bounds sensory surprise and can be minimised through action (cf. (2)). Effectively, free energy allows agents to finesse a generally intractable integration problem (evaluating surprise) by reformulating it as an optimisation problem. This well-known device was introduced by Feynman [43] and has been exploited extensively in machine learning and statistics [44–46]. The requisite free energy bound is created by adding a nonnegative Kullback-Leibler divergence or cross-entropy term [47] to surprise:

(10)ℱ(t)=ℒ(s˜(t))+DKL(q(ϑ)|| p(ϑ ∣ s˜,m))=〈ln⁡q(ϑ)〉q−〈ln⁡p(s˜,ϑ ∣ m)〉q. This divergence is induced by a recognition density q(ϑ):=q(ϑ∣μ˜) on the hidden causes of sensory states. This density is associated with the agent's internal states μ˜(t) that play the role of sufficient statistics; for example, the mean or expectation of hidden causes. Free energy ℱ(s˜,μ˜)∈ℝ can be evaluated because it is a functional of internal states and a generative model p(s˜,ϑ∣m) entailed by the agent. This can be seen from second equality, which expresses free energy in terms of the negentropy of q(ϑ) and the expected value of ln⁡p(s˜,ϑ∣m).

To ensure action minimises surprise, the free energy must be minimised with respect the internal variables that encode the recognition density (to ensure the free energy is a tight bound on surprise). This is effectively perception because the cross-entropy term in (10) is non-negative, with equality when q(ϑ∣μ˜)=p(ϑ∣s˜,m) is the true conditional density. In short, optimising the recognition density makes it an approximate conditional density on the causes of sensory states. This is the basis of perceptual inference and learning as articulated by the Bayesian brain hypothesis [10, 13, 48–52]. We can now formulate action (9) in terms of a dual minimisation of free energy (see (2) and Figure 1).

(11)a∗=arg min⁡ aℱ(s˜,μ˜),μ˜∗=arg min⁡μ˜ ℱ(s˜,μ˜). Action minimises free energy through changing the generalised motion of hidden states. In essence, it ensures that the trajectory of sensory states conform to the agents conditional beliefs encoded by internal states. Note that action is fundamentally different from a policy in optimal control and reinforcement-learning. Action is not a deterministic function of hidden states and is sensitive to random fluctuation causing sensory states. This means, unlike an optimal policy, it can suppress surprises by countering unexpected fluctuations in sensory states: although optimal control schemes can recover from perturbations, they cannot cancel them actively. However, as we will see below, optimal policies play a key role providing in prior constraints on the flow of hidden states that action tries to disclose.

In what follows, we will assume that the minimisation of free energy with respect to action and internal states (11) conforms to a generalised gradient descent,

(12)a˙=−∂aℱ(s˜,μ˜),μ˜˙=𝒟μ˜−∂μ˜ℱ(s˜,μ˜). These coupled differential equations describe action and perception respectively. The first just says that action suppresses free energy. The second is known as generalised filtering [53] and has the same form as Bayesian (e.g., Kalman-Bucy) filtering, used in time series analysis. The first term is a prediction based upon the differential operator 𝒟 that returns the generalised motion of internal states encoding conditional predictions. The second term is usually expressed as a mixture of prediction errors that ensures the internal states (sufficient statistics) are updated in a Bayes-optimal fashion (see below). The differential equations above are coupled because sensory states depend upon action, which depends upon perception through the conditional predictions. This circular dependency leads to a sampling of sensory input that is both predicted and predictable, thereby minimising free energy and surprise. This is known as active inference.

In generalised filtering, one treats hidden parameters as hidden states that change very slowly: the ensuing generalised descent can then be written as a second-order differential equation: μ¨θ=-∂θℱ-κμθ′, where κ is the (high) prior precision on changes in hidden parameters. See [53] for details. In neurobiological formulations of free energy minimisation, internal states generally correspond to conditional expectations about hidden states and parameters, which are associated with neuronal activity and connections strengths, respectively. In this setting, optimising the conditional expectations about hidden states (neuronal activity) corresponds to perceptual inference while optimising conditional expectations about hidden parameters (neuronal plasticity) corresponds to perceptual learning.

Equation (12) describes the dynamics of action and internal states, whose particular form depends upon the generative model of the world. We will assume this model has the following form:

(13)x˜˙=f(x˜,θ)+ω˜x,s˜(t)=g(x˜,θ)+ω˜s. As in the previous section, (f, g) are nonlinear functions of hidden states that generate sensory states; however, these are distinct from the real equations of motion and sensory mappings (f, g) that depend on action. The generative model does not include action, because action is not a hidden state. Random fluctuations (ω s, ω x) play the role of sensory noise and induce uncertainty about the motion of hidden states. Hidden states are abstract quantities (like the motion of an object in the field of view) that the agent uses to explain or predict sensations. Gaussian assumptions about the random fluctuations in (13) furnish a probabilistic generative model of sensory states p(s˜,ϑ∣m) that is necessary to evaluate free energy. See [53] for a full description of generalised filtering in the context of hierarchical dynamic models. For simplicity, we have assumed that state-space associated with the generative model is the same as the hidden state-space in the world. However, this is not necessary, because exchanges with the environment are mediated through sensory states and action.

Given the form of the generative model (13) and an assumed (Gaussian) form for the recognition density, we can now write down the differential equations (12) describing the dynamics of internal states in terms of (precision-weighted) prediction errors (ε˜s,ε˜x) on sensory states and the predicted motion of hidden states, where (ignoring some second-order terms and using g˜:=g(x˜,θ))

(14)μ˜˙=𝒟μ˜+∂μ˜g˜·ε˜s+∂μ˜f˜·ε˜x−𝒟Tε˜x,ε˜s=Π˜s(s˜−g˜),ε˜x=Π˜x(𝒟μ˜−f˜). The (inverse) amplitude of generalised random fluctuations are encoded by their precision (Π˜s,Π˜x), which we assume to be fixed in this paper. This particular free energy minimisation scheme is known as generalised predictive coding and has become a useful metaphor for neuronal message passing in the brain: see also [12]. The simplicity of this scheme stems from the assumed Gaussian form of the recognition density. This means the internal states or sufficient statistics can be reduced to conditional expectations (see Appendix C). In neural network terms, (14) says that error-units receive predictions while prediction-units are driven by prediction errors. In neurobiological implementations of this scheme, the sources of prediction errors are usually thought to be superficial pyramidal cells while predictions are conveyed from deep pyramidal cells to superficial pyramidal cells encoding prediction error [54].

Because action can only affect the free energy by changing sensory states, it can only affect sensory prediction errors. From (13), we have

(15)a˙=−∂as˜·ε˜s. In biologically plausible instances of active inference, the partial derivatives in (15) would have to be computed on the basis of a mapping from action to sensory consequences, which is usually quite simple; for example, activating an intrafusal muscle fibre elicits stretch receptor activity in the corresponding spindle: see [6] for discussion.

In summary, we can account for the unnatural persistence of self-organising biological systems in terms of action that counters the dispersion of their states by random fluctuations. This action minimises the entropy of their ergodic density by minimising a free energy bound on sensory surprise or self-information as each point in time. To ensure the free energy is a good proxy for surprise, internal states must also minimise free energy and implicitly represent hidden states. This minimisation rests upon a generative model, which furnishes conditional predictions that action can fulfil. These predictions rest of on equations of motion that constitute (empirical) priors [55] on the flow of hidden states in the world. In short, agents are equipped with a model of dynamics in their local environment and navigate that environment to minimise their surprise.

We can now associate the expected flow of the previous section with the empirical priors learned under the generative model: f(x, u) = f(x, μ θ). This rests upon the assumption that action eliminates (on average) the difference between the actual and predicted flow. This means the predicted flow corresponds to the policy. The policy f(x, u) = f(x, μ θ) is an empirical prior because it depends on conditional beliefs about hidden parameters encoded by μ θ. This is an important point because it means that the environment causes prior beliefs about motion (through parameter learning), while these beliefs cause the sampled environment. This circular causality is the basis of autopoietic flow and highlights the fact self-organisation rests on a reciprocal exchange between implicit beliefs about how an agent or system will behave and behavioural constraints that are learned by behaving. Minimising free energy ensures that the beliefs and constraints are consistent and enables the agent to create its own environment. In this view, perceptual inference becomes truly embodied or situated and is an integral part of sustainable interactions with the environment. The previous section suggested that action was the key to understanding self-organised behaviour. This section suggests that action depends on a policy or empirical priors over flow. In what follows, we consider the nature of this flow and its specifications.

The previous section established differential equations that correspond to action and perception under a model of how hidden states evolve. These equations are based on the assumption that agents suppress (a bound on) surprise and, implicitly, the entropy of their ergodic density. We now consider optimising the model per se, in terms of formal priors on flow. These correspond to the form of the equation of motions in (13). In particular, we will consider constraints encoded by a (cost) function c(x˜)⊂m over hidden states. The existence of autopoietic flow is not mysterious, in the sense that agents who do not have a random attractor cannot exist. In other words, every agent (phenotype) can be regarded as a solution to the Fokker-Planck equation, whose policy is compatible with the biophysics of its environmental niche. One might conjecture that each solution (random attractor) corresponds to a different species, and that there may be a limited number of solutions as evidenced by convergent evolution [17]. This section considers the policies that underwrite these solutions and introduces the notion of value in terms of the Helmholtz decomposition. In brief, we will see that flow determines value, where value is negative surprise.

We start with the well-known decomposition of flow into curl- and divergence-free components (strictly speaking, the first term is only curl-free when Γ(x˜)=γ(x˜)·I; that is, the diffusion tensor is isotropic. However, this does not affect the following arguments, which rest on the divergence-free component),

(16)f=Γ·∇V+∇×W. This is the Helmholtz decomposition (also known as the fundamental theorem of vector calculus) and expresses any policy in terms of scalar V(x˜) and vector W(x˜) potentials that prescribe irrotational (curl-free) Γ · ∇V and solenoidal (divergence-free) ∇×W flow. An important decomposition described in [37, 56], formulates the divergence-free part in terms of an antisymmetric matrix, Q(x˜)=-Q(x˜)T and the scalar potential, which we will call value, such that

(17)f=(Γ+Q)∇V⇒∇×W=Q∇V. Using this (standard form) decomposition [57], it is fairly easy to show that p(x˜∣m)=exp⁡(V(x˜)) is the equilibrium solution to the Fokker-Planck equation (4):

(18)p=exp⁡(V)⇒∇p=p∇V⇒Λp=∇·Γ∇p−∇·(fp)=−p(∇·(Q∇V)+(Q∇V)·∇V)=0. Equation (18) uses the fact that the divergence-free component is orthogonal to ∇V (see Appendix D). This straightforward but fundamental result means that the flow of any ergodic random dynamical system can be expressed in terms of orthogonal curl- and divergence-free components, where the (dissipative) curl-free part increases value while the (conservative) divergence-free part follows isoprobability contours and does not change value. Crucially, under this decomposition value is simply negative surprise: ln⁡p(x˜∣m)=V(x˜)=-ℒ(x˜∣m). It is easy to show that surprise (or value) is a Lyapunov function for the policy

(19)V˙(x(t))=∇V·f=∇V·Γ·∇V+∇V·∇×W=∇V·Γ·∇V≥0. Lyapunov functions always decrease (or increase) with time and are used to establish the stability of fixed points in deterministic dynamical systems. This means every policy (expected flow) reduces surprise as a function of time. In other words, it must direct flow towards states that are more probable (and have a greater sojourn time). This is just a formal statement of the fact that ergodic systems must, on average, continuously suppress surprise, to offset the dispersive effect of random fluctuations. Ao reviews the importance and generality of the decomposition in (17) and how it provides a unifying perspective on evolutionary and statistical dynamics [38]: this decomposition shows that fluctuations in Darwinian dynamics imply the existence of canonical distributions of the Boltzmann-Gibbs type. Furthermore, it demonstrates the second law of thermodynamics, without detailed balance. In particular, the dynamical (divergence-free) component responsible for breaking detailed balance does not contribute to changes in entropy. In short, (17) represents “a simple starting point for statistical mechanics and thermodynamics and is consistent with conservative dynamics that dominates the physical sciences” [58]. The generality of this formulation can be appreciated by considering two extreme cases of flow that emphasise the curl and divergence-free components, respectively.

When the random fluctuations are negligible (i.e., Γ → 0), irrotational (curl-free) flow Γ · ∇V = 0 disappears and we are left with divergence-free flow that describes conservative dynamics (e.g., classical mechanics). These flows would be appropriate for massive bodies with virtually no random fluctuations in their motion. A simple example would be the Newtonian mechanics that result from a Lagrangian (surprise) and antisymmetric matrix,

(20)ℒ(x˜)=φ(x)+12x′2Q=[0−110]⇒f=−Q∇ℒ=[x˙x˙′]=[x′−∇φ]. This describes the motion of a unit mass in a potential field φ(x), where the Lagrangian comprises potential and kinetic terms. Things get more interesting when we consider random fluctuations in the velocity,

(21)Γ=[000γ]⇒f=−(Γ+Q)∇ℒ=[x′−∇φ−γx′]. This introduces a motion-dependent reduction in the motion of velocity (acceleration) that corresponds to friction. Note that friction is an emergent property of random fluctuations in velocity (and nothing more). A more thorough treatment of the relationship between the diffusion due to random fluctuations and friction can be found in [57], using the generalised Einstein relation. Consider now systems in which random fluctuations dominate and the conservative (divergence-free) flow can be ignored.

Here, irrotational (curl-free) flow dominates and the dynamics have detailed balance, which means that flow can be expressed as an ascent on a scalar (value) potential: f = Γ · ∇V = −Γ · ∇ℒ. Crucially, because there is effectively no conservative flow, the ergodic density concentrates around the maximum of value (or minimum of surprise), which (in the deterministic limit) induces a fixed point attractor. Curl-free polices are introduced here, because of their central role in optimal control and decision (game) theory: in the next section, we will consider curl-free policies that are specified in terms of value-functions, V(x˜). These range from reinforcement-learning heuristics in psychology to more formal optimal control theory treatments. However, one should note that these approaches are incomplete in the sense they do not specify generic policies: a complete specification of flow would require the vector potential W(x˜) or, equivalently, the anti-symmetric matrix, Q(x˜). This means that it is not sufficient to know (or learn) the value of a state to specify a policy explicitly, unless the environment permits curl-free policies with detailed balance (i.e., with no classical or conservative dynamics).

Ergodic densities under detailed balance are closely connected to quantal response equilibria (QRE) in economics and game theory. QRE are game-theoretical formulations that provide an alternative to Nash equilibria [18]. QRE do not require perfect rationality; players are assumed to make normally distributed errors in their predicted payoff. In the limit of no errors, QRE predict unique Nash equilibria. From the point of view of game theory, the interesting questions pertain to different equilibria prescribed by the policy or state-transitions. These equilibria are analogous to the solutions of the Fokker-Planck equation above, where V(x˜) is called attraction and Γ ∈ ℝ+ is temperature or inverse sensitivity [9, 59]. In this context, the ergodic density p(x˜∣m) =exp⁡(-ℒ) prescribes optimal states or choices probabilistically, in terms of value, where V = −ℒ. This prescription is closely related to softmax or logit discrete choice models [60], which are the most common specification of QRE. In economics, optimal state-transitions lead to equilibria that maximise value or expected utility. These are low-entropy densities with probability mass on states with high utility. We purse this theme in below, in the context of optimal control theory and reinforcement-learning.

In this section, we have seen that a policy or empirical priors on flow (specified by conditional beliefs about the parameters of equations of motion) can be decomposed into curl and divergence-free components, specified in terms of a value-function and antisymmetric matrix that determines conservative flows of the sort seen in classical mechanics. Crucially, this value-function is just negative surprise and defines the ergodic (invariant) probability density over hidden states. However, we have no idea about where the policy comes from. All we know is that it furnishes a solution to the Fokker-Planck equation; an idiocentric description of an agent's exchange with the environment. The remainder of this paper will be concerned with how policies are specified and how they are instantiated in terms of value-functions.