Information and efficiency in the nervous system--a synthesis
This study further established that the temporal profile of the currents underlying APs of some mammalian neurons are nearly perfectly matched to the optimised properties of ionic conductances, so as to minimise the ATP cost. All of these studies show that experimentally measured APs are in fact more efficient than suggested by the previous estimates of Attwell and Laughlin [59]. This was because until 2001 experimental measurements of membrane currents were scant, impeding the study of the overlap between Na+ and K+ currents. The effects of energy-efficient APs on cortical processing were gauged by recalculating Attwell and Laughlin's (2001) estimates by first using the overlap factor of 1.2—found in mouse cortical pyramidal cells—and then assuming the probability that a synaptic bouton releases a vesicle in response to an incoming spike remains at 0.25 [80]. Neurons that are 80% efficient have two notable effects (Figure 3). First of all, the specific metabolic rate of the cortical grey matter increases by 60%, and second, the balance of expenditure shifts from action potentials to synapses (Figure 3, cf. Figure 2) [80].
Dimensionality reduction: Sensory input is high dimensional—a visual scene comprises differences in brightness, colours, numbers of edges, etc. If the retina did not preprocess this visual information, we would have to handle around 36 Gb/s of broadband information, instead of 20 Mb/s of useful data [73]. Preprocessing increases the metabolic efficiency of the brain by about 1,500 times. The requisite dimensionality reduction is closely related to minimising complexity—it is self-evident that internal representations or models of the sensorium that use a small number of dimensions or hidden states will have a lower complexity and incur smaller metabolic costs.
Energy-efficient signalling: Action potentials (APs) are expensive commodities, whether they are used for local computation or long-distance communication [59]. Energy-efficient APs are characterised by Na+ channel inactivation, voltage-dependent channel kinetics, and corporative K+ channels—as described by multiple gating currents, inward-rectifying K+ channels, and high channel densities [7]. These biophysical innovations enable a neuron to produce efficient APs that use the minimal currents necessary to generate a given depolarisation.
Component size and numbers: Action potentials travel considerable distances along densely packed axons, collaterals, and dendrites. The capacitance that must be charged by APs increases with membrane area [101], constraining the number and length of neuronal processes. It is fairly straightforward to show that—to maintain information transfer—the optimal solution is to decrease the number of components. Assuming all neurons have the same thresholds and energy consumption, the energy-efficient solution is to minimise the number of components, under computational constraints dictated by the ecological niche of the animal [101].
Modular design: Very-large-scale integration circuits suggest an isometric scaling relation between the number of processing elements and the number of connections (Rent's rule [102]). Neuronal networks have been shown to obey Rent's rule, exhibiting hierarchical modularity that optimises a trade-off between physical cost and topological complexity—wherein these networks are cost-efficiently wired [103]. A modular design balances the savings in metabolic costs, while preserving computational capacities. Hierarchical modularity also emerges under predictive coding [33]. In this context, the brain becomes a model of its environment, which through the separation of temporal scales necessarily requires a hierarchical connectivity.
Parallel architecture: The brain processes information in parallel—be it frequency analysis in the inner ear or analysing different attributes of a visual scene using functional segregation. This parallel architecture mirrors those used in modern-day microprocessors. For example, a fast single-core microprocessor may consume 5 Watts and execute a program in 10 seconds. If we bring together two single cores, power will double and execution time will halve, still consuming 50 Joules. Alternatively, a slow double-core microprocessor that expends 2.5 Watts of power to execute the program in 15 seconds could consume only 7.5 Joules. This energy saving works because power is proportional to frequency cubed; therefore, halving the frequency reduces the speed by two but conserves eight times the power, making the microprocessor four times as efficient. In short, if parallel architectures are combined with slow computing speeds, the resulting system is energetically more efficient.
Analogue versus digital: If analogue computing is so efficient [104], why don't neurons operate on an all analogue basis? The obvious answer is signal processing in the digital (such as AP) domain enables noise suppression. Noise accumulation in analogue systems [73] speaks to hybrid processing—the use of analogue preprocessing before optimal digitisation. APs are useful in this context because they have an inbuilt threshold mechanism that attenuates noise. If a presynaptic signal is encoded as an AP and transmitted, there is hardly any conduction loss, thereby enabling a reliable transfer of information.
The principle of energy efficiency is not just linked to single neurons. Energy budgets have been calculated for the cortex [82], olfactory glomerulus [83], rod photoreceptors [84], cerebellum [85], and CNS white matter [86], among others. These studies highlight the fact that the movement of ions across the cell membrane is a dominant cost, defined by the numbers and cellular makeup of the neurons and the proportion of synaptic machinery embedded in the cell membrane (Figure 4). Niven and Laughlin [3] have argued that when signalling costs are high and resting costs are low, representations will be sparse; such that neurons in a population preferentially represent single nonoverlapping events (also see [87]). Similarly, when resting costs are high and signalling costs are low, the nervous system will favour the formation of denser codes, where greater numbers of neurons within the population are necessary to represent events [3].
Experimental studies of mammalian cortex suggest that the cortex organises itself to minimise total wiring length, while maximising various connectivity metrics [88]. Minimising wiring lengths decreases the surface area of neuronal processes, reducing the energy required for charging the capacitive cell membrane—to sustain and propagate action potentials. In fact, theoretical analyses in pyramidal and Purkinje cells have shown that the dimensions and branching structure of dendritic arbours in these neurons can be explained by minimising the dendritic cost for a potential synaptic connectivity [89], [90]. This can result from increasing the repertoire of possible connectivity patterns among different dendrites, while keeping the metabolic cost low [89], [90].
In summary, we have reviewed several lines of evidence that evolution tries to minimise metabolic costs, where—in the brain—these costs are primarily incurred by the restoration of transmembrane potentials, whose fluctuations encode or represent hidden states of the world. This raises a question: is energy the only constraint in the evolution of animals? Of course not—functional constraints like reliability, speed, precision, etc. [67] and structural constraints like optimal wiring [91] are equally important. For example, a single action potential in the squid giant axon consumes orders of magnitude more energy than a hippocampal or a pyramidal neuron, yet evolution has invested that extra Joule to buy speed [80], [92]. In short, structure and function interact to determine the fitness of an animal. Having surveyed the key metabolic constraints under which neuronal processing must proceed, we now try to integrate the information theoretic and metabolic perspectives.
In this section, we gather together the imperatives for biological self-organisation reviewed above. We hope to show that minimising variational free energy necessarily entails a metabolically efficient encoding that is consistent with the principles of minimum redundancy and maximum information transfer. In brief, we will show that maximising mutual information and minimising metabolic costs are two sides of the same coin: by decomposing variational free energy into accuracy and complexity, one can derive the principle of maximum mutual information as a special case of maximising accuracy, while minimising complexity translates into minimising metabolic costs.
To connect the thermodynamic work or metabolic energy required to represent hidden states to the variational free energy of those representations, we need to consider the relationship between representational internal states and the underlying thermodynamic microstates. Recall that internal states μ(t) are deterministic quantities that encode a conditional density over hidden states of the world. These macroscopic states can be regarded as unconstrained internal variables of a biophysical system; for example, the molar fractions of different molecules in a cellular compartment. The underlying biophysical system can then be associated with a (thermodynamic) canonical ensemble with internal energy:(10)Here, pi corresponds to the probability of a particular microscopic state and Ei(μ)to its corresponding energy. Given that the total energy is conserved, this probability is given by the Gibbs measure or Boltzmann distribution:(11)The partition function Z(T, μ) ensures the probabilities sum to one, while the last equality follows simply from the definition of entropy H[pi] = Ei[−ln pi]. The Boltzmann distribution describes a system that can exchange energy with a heat bath (or a large number of similar systems) so that its temperature remains constant. The Helmholtz free energy A(T, μ) measures the work obtainable from a closed thermodynamic system at a constant temperature and volume—where a closed system can exchange energy with other systems (but not mass).
The key result we will use from statistical thermodynamics is that the Helmholtz free energy is minimised at equilibrium with respect to any unconstrained internal variables for a closed system at constant temperature T 0,(12)where A 0(T 0, μ) is the free energy of the system at equilibrium or steady state (i.e., constant entropy). This motivates the following Lemma:
Lemma: (complexity minimisation) Minimising the complexity of a conditional distribution—whose sufficient statistics are (strictly increasing functions of) some unconstrained internal variables of a thermodynamic system—minimises the Helmholtz free energy of that system.
Proof: Using standard results from Bayesian statistics [16], we can express free energy as complexity minus accuracy (13)The first complexity term is the divergence between the conditional distribution and the prior distribution under the generative model. This effectively counts the degrees of freedom used to encode or predict sensory input. The accuracy is simply the expected log likelihood of the sensory input under the conditional density encoded by internal states. The prior distribution represents beliefs in the absence of sensory input. This corresponds to the distribution encoded by internal states μ = μ 0 when deprived of input for a suitably long time—at which point, we can assume thermodynamic equilibrium, such that Helmholtz free energy is minimised (see Equation 12):(14)However, in the absence of input, variational free energy reduces to complexity F 0(μ)≥0, which—by Gibbs inequality—has a minimum of zero. This means that complexity is also minimised.(15)In sum, the internal states encoding prior beliefs about hidden states of the world are those that minimise Helmholtz free energy and the complexity defined by variational free energy.
Remarks: All we are saying here is that if a (neuronal) system is deprived of sensory inputs it will obtain thermodynamic equilibrium (or at least a nonequilibrium steady state) and will therefore minimise Helmholtz free energy. This assumes, not implausibly, a constant temperature and volume. Crucially, this is precisely the brain state encoding prior beliefs about sensory input, which means that it is the state of minimum computational complexity. Heuristically, this means that one can associate the complexity cost of variational free energy with metabolic cost—in the sense that they share the same minimum. Crucially, minimising fluctuations in Helmholtz free energy reduces metabolic work by Equation 6. Interestingly, complexity cost also plays a central role in free-energy formulations of optimal control and economic theory [93], [94]. Still et al. arrive at the same conclusions by treating the thermodynamic system as having an implicit model of its inputs—allowing them to establish the fundamental equivalence between model inefficiency or complexity and thermodynamic inefficiency [95]. However, both of these compelling treatments consider homologues of Helmholtz free energy—not variational free energy, which is a functional of a probabilistic model (the conditional distribution).
The complexity minimisation lemma suggests that commonly occurring representational states—that are a priori most probable—are the least costly; for example, resting levels of transmembrane voltage or baseline firing rates. Rare excursions from these states are associated with a high metabolic cost. But how does minimising complexity relate to principles of minimum redundancy? Because representations do not change sensory inputs, they are only required to minimise the free energy of the conditional density. Assuming conditional uncertainty is small, the conditional density can be approximated with a point mass at , such that and the free energy becomes (from Equation 13)(16)The first equality expresses free energy is terms of accuracy and complexity, where the second complexity term just reports the surprise about the conditional representation under prior beliefs. The second equality is the corresponding path integral of free energy (known as free action). Under ergodic assumptions [12], [96] this can be expressed as the conditional entropy of sensory input, given the representations and the entropy of the internal states. Equation 11 has two important implications. First, it shows that minimising free energy, at each point in time, is equivalent to minimising free action —by the fundamental lemma of variational calculus. In other words, Equation 11 is just a restatement of the principle of least action. Second, it shows that minimising free energy maximises the accuracy of representations or minimises their conditional uncertainty (entropy) over time. This is simply a restatement of the principle of minimum redundancy or maximum mutual information [97].
This follows because minimising uncertainty about sensory inputs, given internal states, implicitly maximises the mutual information between sensory and internal states (for any given sensations):(17)This suggests that the infomax principle [97] is a special case of the free-energy principle that is obtained when we discount uncertainty and represent sensory input with point estimates of their causes. In this context, high mutual information is assured by maximising accuracy (e.g., minimising prediction error) and prior beliefs are enforced by minimising complexity. Crucially, minimising complexity minimises metabolic cost.
In short, the infomax principle can be understood in terms of the decomposition of free energy into complexity and accuracy: mutual information or statistical efficiency is optimised when conditional expectations maximise accuracy (or minimise prediction error), while thermodynamic efficiency is assured by minimising complexity. This minimisation ensures that the generative model is not over-parameterized and leads to a parsimonious representation of sensory data that conforms to prior beliefs about their causes. Interestingly, advanced model optimisation techniques use free-energy optimisation to eliminate redundant model parameters [98], suggesting that free-energy optimisation might provide a nice explanation for synaptic pruning and homeostasis in the brain during neurodevelopment [99] and sleep [100]. In developing the link between metabolic and statistical efficiency, we have assumed that internal neuronal states encode hidden states in terms of their most likely value or expectation. Is there any principled reason to assume this form of neuronal code?
Notice from Equation 7 that minimising variational free energy entails maximising the entropy of the conditional density. Intuitively, this is like keeping one's options open when trying to find hypotheses or explanations for sensory input. If we admit an encoding of the conditional density up to second order moments, then the maximum entropy principle [42], implicit in the definition of free energy, requires to be Gaussian. This is because a Gaussian density has the maximum entropy of all forms that can be specified with two moments. Assuming a Gaussian form is known as the Laplace assumption and enables us to express the entropy of the conditional density in terms of its first moment or expectation. This follows because we can minimise free energy with respect to the conditional covariance as follows:(18)Here, the conditional precision Π(μ) is the inverse of the conditional covariance Σ(μ). Equation 18 means the free energy becomes a function of conditional expectations and sensory states. This is important because it suggests the brain may represent hidden states of the world in terms of their expected values. This leads to the Laplace code (defined as neuronal encoding under the Laplace assumption), which is arguably the simplest and most flexible of all neuronal codes [13], [14]. Furthermore, under the Laplace code, one can minimise free energy efficiently using predictive coding [29], [31]. Predictive coding has become one of the most popular ways of understanding message passing in the brain—particularly in the setting of hierarchical perceptual inference. In short, the free-energy principle entails the principle of maximum entropy and leads, in a principled way, to a neuronal encoding of representations in terms of conditional expectations.
The specific nature of the neural code may be exclusive to a species or underlying neural function. Whatever its makeup—expected latency, firing rate, spike timing, phase, etc.—it will exist to harmonize the dialogue between perception and action. In practice, we usually have in mind the instantaneous rate of firing of neuronal populations, which means the internal states encoding posterior beliefs are ensemble averages of ensemble averages—for example, the expectation of (a function of) depolarisation over the neuronal ensemble, where the depolarisation of a single neuron is (a function of) the internal variables of a canonical ensemble.
We have reviewed the thermodynamic and computational (statistical) imperatives for biological self-organisation, with a special focus on neuronal circuits. We have considered the role of classical thermodynamics and the notion of metabolic efficiency—that appears to be an important constraint, under which neurophysiology and neuroanatomy have evolved. From a computational perspective, we have looked at variational free-energy minimisation as the basis for active Bayesian inference and modelling of the environment. The ability to represent and predict hidden environmental states efficiently can be quantified in terms of mutual information. Our synthesis suggests that minimising variational free energy is a sufficient account of the tendency to maximise both metabolic and statistical efficiency. The motivation for minimising variational free energy is to minimise its long-term average to maintain a constant external milieu—as measured by the entropy of an organism's sensory samples over time. By decomposing variational free energy into accuracy and complexity one can understand metabolic efficiency in terms of minimising complexity (which minimises Helmholtz free energy), under the computational constraint that sensory inputs are represented accurately. Conversely, statistical efficiency can be understood in terms of maximising the accuracy (which maximises mutual information), under the constraint that representations have minimal complexity. The link between complexity and metabolic cost rests on the simple observation that, in the absence of sensory input, prior beliefs are encoded by physical variables that minimise Helmholtz free energy.
The nice thing about this formulation is that, under active inference, organisms will selectively sample sensory inputs that conform to their prior beliefs and minimise the complexity of their representations. This means that biological systems will appear to act in a way that minimises fluctuations in Helmholtz free energy—and will aspire to the nonequilibrium steady state that has been assigned to them by evolution.