Information and efficiency in the nervous system--a synthesis

In systems biology, questions concerning the molecular and cellular makeup of an organism are of utmost importance, especially when trying to understand how unreliable components—like genetic circuits, biochemical cascades, and ion channels, among others—enable reliable and adaptive behaviour. The repertoire and speed of biological computations are limited by thermodynamic or metabolic constraints: an example can be found in neurons, where fluctuations in biophysical states limit the information they can encode—with almost 20–60% of the total energy allocated for the brain used for signalling purposes, either via action potentials or by synaptic transmission. Here, we consider the imperatives for neurons to optimise computational and metabolic efficiency, wherein benefits and costs trade-off against each other in the context of self-organised and adaptive behaviour. In particular, we try to link information theoretic (variational) and thermodynamic (Helmholtz) free-energy formulations of neuronal processing and show how they are related in a fundamental way through a complexity minimisation lemma.

The design of engineered and biological systems is influenced by a balance between the energetic costs incurred by their operation and the benefits realised by energy expenditure. This balance is set via trade-offs among various factors, many of which act as constraints. In contrast to engineering systems, it has only been possible recently to experimentally manipulate biological systems—at a cellular level —to study the benefits and costs that interact to determine adaptive fitness [1], [2]. One such example is the nervous system, where metabolic energy consumption constrains the design of brains [3]. In this review paper, we start by defining computation and information in thermodynamic terms and then look at neuronal computations via the free-energy principle. We then consider the efficiency of information processing in the nervous system and how the complexity of information processing and metabolic energy consumption act as constraints. The final section tries to integrate these perspectives: In brief, we will argue that the principle of maximum efficiency applies to both information processing and thermodynamics; such that—for a given level of accuracy—statistically and metabolically efficient brains will penalise the use of complex representations and associated commodities like energy.

A widely used term in neuroscience is “neuronal computation”; but what does computation mean? Simply put, any transformation of information can be regarded as computation, while the transfer of information from a source to a receiver is communication [4]. To understand the physical basis of computation, let us reconsider Feynman's example of a physical system whose information can be read out. The example is intentionally artificial, to keep the physics simple, but has a direct parallel to neuroscience, as we will show at the end. Consider a box that it is filled with an ideal gas containing N atoms. This occupies a volume V 1, in which we can ignore forces of attraction or repulsion between the particles. Now suppose that the answer to a question is “yes” if all N atoms are on the right-hand side of the box, and “no” if they are on the left. We could use a piston to achieve this. By compressing the gas into a smaller volume V 2, a piston performs the work(1)Classical thermodynamics tells us that the pressure and volume of an ideal gas are linked such that(2)where k is Boltzmann's constant and the temperature T is assumed constant. The work done on the gas is then:(3)As we compress the gas, the atoms speed up and attain kinetic energy, hence heating the box. According to the conservation of energy, the work done on the gas is converted to heat. This heat is dissipated to the external environment to keep the temperature constant. This means that the internal energy U of all the particles remains unchanged, such that the work done by the system or change in Helmholtz free energy A = U–TS reduces to the change in thermodynamic entropy S = kH, where H is Shannon entropy:(4)For a single gas particle, with we find that Shannon entropy decreases by ln 2.

This means that by compressing the gas there are fewer places that the particles can occupy and we are less uncertain about their whereabouts. In short, we have gained information. What have we learned from this exercise? To obtain information—in other words, to reduce entropy or average uncertainty —one has to perform work. More generally, Landauer's seminal work showed that energy is required when information is erased or deleted via irreversible operations [5], [6]. In the context of noise or communication, the deletion of incorrect bits therefore requires the dissipation of energy. This dissipation is decreased at lower temperatures because of reduced thermal noise—lower temperatures facilitate a reduction of energy expenditure.

In the brain, volume changes are not the primary mode of conveying information. Instead, the compartments present in the brain, ranging from synaptic clefts to organelles, maintain a relatively constant volume over several seconds at least. What changes on a short time scale are the numbers of molecules, such as transmitters or ions, in these compartments. If we translate volumes to concentrations ci = N/Vi, the change in entropy due to information transfer becomes(5)The work is then . If the molecules are charged, the chemical potential sets up an electrical potential (called the Nernst potential), which is the basis for much of the signalling within the brain. For some molecules, such as Na+ and K+ ions, the concentration changes during electrical signalling are miniscule relative to the total concentrations of these molecules. By linearising δW in the concentration changes, we can easily compute the energetic cost of neuronal signals [7].

In the examples above, the system remains in thermodynamic equilibrium. Recent progress has been made in describing the relationship between Helmholtz free energy and work when the system is driven far from equilibrium—for example, if the gas was compressed quickly. In this more general setting, the Jarzynski equality states [8]:(6)where the expectation E[·]is over an ensemble of paths from the initial to final states. Crucially, the change in Helmholtz free energy (and expected work) does not depend upon the path or the rate at which external parameters (like volume) change. Notice that Equation 4 is a special case of Equation 6, when there is only one (infinitely slow) path.

In summary, changing the state of a system necessarily entails a change in Helmholtz free energy that is equivalent to the work done on the system. Under isothermal conditions, this changes the thermodynamic entropy, which can be regarded as the average uncertainty or information we have about the (microscopic) state of the system. So is this sufficient to establish the link between thermodynamic free energy and information processing? Not really: because the information here is about the (microscopic) state of the system in question. This does not speak to representational information of the sort associated with biological computations or communication: information of this sort reflects how one system represents another. In the next section, we consider a purely information theoretic perspective on computation that invokes free energy and entropy of a fundamentally different sort.

Equation 4 shows how the basic laws of classical thermodynamics connect the Helmholtz free energy of a system to its entropy, where entropy corresponds to the disorder or average uncertainty about its state. In biological systems, there is a natural tendency to resist disorder—at multiple levels of organisation. The maintenance of sensory and physiological states within characteristic bounds is typical of biological systems and usually relies on some sort of regulatory process, i.e., homeostasis [9], [10]. Mathematically, this can be expressed by saying that the (sensory) states of biological systems have characteristically low Shannon entropy, where—under ergodic assumptions—Shannon entropy is (almost surely) the long-term average of self information or surprise (see below). An ergodic system has an invariant phase volume [11], which is a necessary condition for an organism to exist—in the sense that it would otherwise transgress phase boundaries and cease to exist [12].

Here, the Shannon entropy plays the same role as thermodynamic entropy but measures the dispersion not over microstates of a thermodynamic (canonical) ensemble, but over some phase functions or macroscopic variables that change with time. These variables can take values that are relatively frequent (low surprise) or infrequent (high surprise). Shannon entropy reflects the average surprise of these variables as they fluctuate over time. By minimising the surprise associated with environmental fluctuations (sensory input), an organism can maintain its physiological states within bounds [13], [14].

To evaluate surprise, biological systems need to infer the probability of each sensory fluctuation they encounter. In systems like the brain, these inferences need to be made in the blink of an eye. However, calculating the requisite probabilities can be an intricate and lengthy process, making such computations practically intractable. In 1972, the physicist Richard Feynman came up with a clever trick for calculating probabilities (approximately but very efficiently) using variational free energy [15]. The trick is to convert a difficult probability density integration problem into an easy optimisation problem by minimising a free energy bound on the quantity of interest—in our case, the surprise of sensory input. In brief, this entails adjusting probability distributions over the causes of sensory input until they minimise the free energy of sensory input. Notice that we have introduced the notion of causes or hidden states of the world that are responsible for generating sensory samples. Heuristically, this means the system or agent has a model of the world that it uses to evaluate the likelihood or surprise of a sensation. Mathematically, hidden states are fictive variables that are necessary to construct a variational free energy bound on surprise, as we will see next.

Let us assume that self-organising systems like the brain represent their environment probabilistically, in terms of hidden states that cause sensory input. For example, an agent might believe its visual sensations were caused by a bird flying across its field of view. These beliefs can be regarded as real-valued, time-dependent internal or representational states . These internal states encode a conditional probability density over hidden states in the world —such as the motion, colour, and size of the bird. The objective is to minimise the surprise of sensations . Here, denotes a model entailed by a system or an agent, and is the probability of observing a particular state under that model. The model is effectively used to generate hypotheses that explain sensory input in terms of hidden states or representations—such as a bird in flight.

As noted above, minimising surprise directly is an intractable problem, so surprise is replaced with its variational free energy bound [15]. This free energy is a function of sensory and internal states and can now be minimised with respect to the internal states:(7)Here, U(t) = −ln p(s(t), ψ(t)|m) corresponds to an internal energy under a generative model of the world, described in terms of the density over sensory and hidden states p(s,ψ|m). In Equation 7 and throughout H[p] = Ep[−ln p] denotes the entropy of a probability distribution. Comparison with Equation 4 explains why F(t) is called free energy—by analogy with its thermodynamic homologue that is defined as internal energy minus entropy. However, it is important to note that variational free energy is not the Helmholtz free energy in Equation 4—it is a functional of a probability distribution over hidden (fictive) states encoded by internal states q(ψ|μ), not the probability distribution over the (physical) internal states. This is why variational free energy pertains to information about hidden states that are represented, not the internal states that represent them. In other words, the variational free energy measures the information represented by internal states, not internal states per se. Later, we will try to establish the link between variational and Helmholtz free energies. First, we consider the computational implications of minimising variational free energy.

In short, free energy finesses the evaluation of surprise—where an agent can evaluate free energy fairly easily, given the internal energy or a generative model of its environment. The second equality in Equation 7 says that free energy is always greater than surprise, because the second term (Kullback-Leibler divergence) is nonnegative. This means that when free energy is minimised with respect to the internal states, free energy approximates surprise and the conditional density approximates the posterior density over hidden states:(8)This is known as approximate Bayesian inference, which becomes exact when the conditional and posterior densities have the same form [16]. Intuitively, minimising free energy renders the conditional density the true posterior density over hidden states, where both are informed by—or conditioned on—sensory information. In Bayesian parlance, a posterior density describes a belief after sampling some data—in contrast to a prior belief that existed before the data were available. Minimising variational free energy can therefore be regarded using sensory evidence to update prior beliefs to approximate posterior beliefs.

How can we place a concept like variational free energy in the context of neuronal computation? This has a long history—originating in Geoffrey Hinton [17], [18] and Douglas Hofstadter's [19] work using Ising models for inference in artificial neural networks. Hinton and colleagues realised that variational free energy was mathematically equivalent to the cost function for inference in a neural network, such as a Hopfield model [20]—the difference between the prediction made by the neural network and what it actually produced as an output, i.e., the prediction error. These ideas were subsequently absorbed into the free-energy principle [21], [22], whose key insight was that to reduce the entropy of sensations, the system had to act on the environment. The solution is to assume that both the internal states of the system and its action minimise variational free energy (and implicitly surprise). This dual minimisation maps nicely onto perception and action, where variational free energy can be reduced by optimising internal (representational) states or sensory states through active sensory sampling. This is known as active inference and essentially compels organisms to selectively sample what they expect to sample.

Under certain statistical assumptions, free energy is essentially the difference between the agent's predictions and the actual sensations sampled [22]. Therefore, minimising the free energy is equivalent to reducing prediction error and hence surprise [14]. To minimise free energy or prediction error, the brain can either change its prediction to match sensory input or it can change what it samples to match its predictions [21]. This suggests that the brain is continually making predictions and reevaluating them by comparing inputs with internal predictions to make sense of the world. Is there any empirical evidence that this scheme operates in the nervous system?

Volunteers in a magnetic resonance imaging (MRI) scanner watched two sets of moving dots—one random and the other moving coherently. They showed patterns of distributed brain activation that could only be explained in terms of top-down predictions from deep in the brain to visual centres in the occipital cortex. In other words, top-down predictions from the extrastriate cortex appeared to suppress prediction errors in the striate cortex [23]. Assuming the visual system is a hierarchy of cortical areas, such predictive coding enables predictions about hidden states of the world—like coherent motion—to influence processing at lower levels [23]. Similarly, in the auditory cortex, electroencephalographic signals from higher processing centres change brain activity in lower areas [24]. Using dynamic causal modelling, Garrido et al. [24] found that models with top-down connections explained empirical electrophysiological data far better than the models with only bottom-up connections. Garrido et al. [24] argued that these neuronal responses were consistent with the brain's attempt to conciliate predictions at one level with those in other levels—in other words, to reduce hierarchical prediction error.

What sort of neuronal architectures mediate this prediction error minimisation—or predictive coding? In mammalian brains, cortical areas are organised hierarchically [25], [26], wherein populations of neurons can encode expected states of the world and provide top-down predictions to lower or sensory levels [27], [28]. For example, top-down connections from pyramidal neurons in the deeper layers of the cortex are thought to provide predictions to superficial pyramidal populations of a lower area. This enables forward connections from superficial pyramidal neurons to convey prediction errors, creating recurrent dynamics that suppress prediction errors at each level of the cortical hierarchy [29]–[31]. The precision of these errors can be modulated by neuromodulation [32]. Such rescaling of prediction errors in proportion to their precision is simply a form of gain control [33], [34] and may mediate attention. In short, the wetware necessary to minimise free energy appears to be available and is remarkably consistent with its known functional anatomy.

In summary, biological organisms are open self-organising systems that operate far from thermodynamic equilibrium [35]. The free-energy principle suggests that organisms avoid phase transitions by minimising (a variational free energy bound on) the Shannon entropy of their sensory states. But how does one reconcile the need of an animal to survive (by avoiding phase transitions) with its innate tendency to forage or explore? This apparent paradox is resolved by noting that active inference is driven by prior beliefs—and these beliefs can entail exploration. In other words, agents expect to explore and would be surprised if they did not. We will return to the central role of priors in the last section.

Perception minimises prediction error by optimising synaptic activity (perceptual inference), synaptic efficacy (learning and memory), and synaptic gain (attention and salience) [14]. In doing so, we form an optimal representation of the sensorium. Such strategies of optimisation are mathematically equivalent to predictive coding [36], [37] or, as we will see later, maximising the mutual information between sensations and the responses they evoke [38], [39]. In the embodied context of action on the environment, free-energy minimisation can also explain active inference in the exteroceptive domain [40] and homoeostasis through minimising interoceptive prediction errors. In short, the idea of free-energy minimisation, stemming from Feynman's beautiful piece of mathematics, allows us to consider perception and action under a general framework—and produce testable hypotheses.

In the previous section, we described how variational free energy is intricately linked to surprise—the free-energy principle tells us that an organism should strive to reduce its prediction error thereby reducing free energy. The connection between free energy and information—although obvious—is seldom commented upon (see Table 1 in [41]). To minimise free energy, the expected prediction error has to be minimised while, at the same time, the entropy of the conditional density is maximised. This is slightly paradoxical because the purpose of free-energy minimisation is to reduce sensory entropy. However, Equation 7 shows that if the entropy of sensory states H[p(s|m)] is minimised vicariously by minimising free energy over time, then the entropy of the conditional density H[q(ψ|μ)]must be maximised at each point in time. This follows from a need to balance accuracy and complexity of the sort seen in Occam's razor. We will return to this in a later section in the context of the principle of maximum entropy [42]. In this section, we focus on information theory as a way of describing the quality of representations and the constraints under which these representations are formed.

We know that all animals process and transmit information to survive and reproduce in an uncertain environment. A principled way to understand such signal processing was absent until Claude Shannon's seminal work on information theory [43]. To understand how messages can be transferred efficiently via telegraphic wires, Shannon derived powerful formalisms that provided fundamental limits on communication [43]. On one hand, information theory allowed optimisation of complicated devices like satellite communication systems. On the other hand, it fitted comfortably with the bounds established by thermodynamics [44]. Some years after its inception, biologists used information theory to study the efficiency of processing in the nervous system. It was realised that efficient representations were permitted by statistical regularities in the sensorium, i.e., hidden states and their sensory consequences that have low entropy (see [45]). However, the influence of random fluctuations and other constraints prohibit a completely efficient encoding of hidden states in the world.

In the nervous system, limited bandwidth and dynamic range create an information bottleneck due to the limited response ranges of the neurons in sensory epithelia [46]–[48]. Atick [45] suggests that these bottlenecks can also result from computational limitations at higher levels of sensory processing—citing as an example the “attention bottleneck,” where there is constriction of information processing—in bits per unit time—somewhere between area V4 and the inferotemporal cortex. In brief, sensory receptors are required to compress an enormous range of statistically redundant sensory data into their limited range. One way to achieve this is by compression —imagine an architect's plan of your office. This does not include the dimensions of every brick, just the information necessary to build the office. It has been proposed that sensory systems also apply the principle of compression. They sieve redundant information, such that only information that is necessary to encode hidden states is retained [46]—in engineering this is called a factorial code. Of course there are many ways to describe such sensory encoding. Others include but are not restricted to feature detection, filtering, etc. Among these, schemes like linear predictive coding and minimum description length formulations have a particularly close and formal relationship with variational formulations.

Sensory receptors (mechanoreceptors, photoreceptors, and the like) are thought to build a factorial representation of the world—such that only independent bits of information are sampled (Figure 1). Interestingly, this has been observed in the large monopolar cells (LMC) in the blowfly compound eye [49]. Laughlin [49] measured the distribution of the fly's natural environment from horizontal scans of dry woodland and lake-side vegetation and quantified the responses of light-adapted LMCs. Laughlin [49] found that the LMC—known to respond to contrast signals—is most sensitive around the most probable input contrast—with sensitivity dropping to zero as the input became more improbable.

The application of information theory to the nervous system is formally pleasing and has provided some compelling insights. However, it does have limits [50]: although it allows one to quantify the transmission of information, it has no notion of semantics. It only cares about how much information is present but not about what that information represents. A widely used information theoretic metric in neuroscience is the mutual information, which measures how much a random variable tells us about another random variable [51]. If is a stimulus and is the representational response, the mutual information is defined as:(9)Note that the joint density p(s,μ|m) is not the generative model p(s,ψ|m) of the previous section—it describes the joint distribution of sensory and internal states, not the joint distribution of sensory and hidden states. Equation 9 simply describes the divergence or relative entropy between the joint density and the product of its marginals. The mutual information is zero when the neuronal representation is statistically independent of the stimulus and is equal to the entropy of the stimulus when the representation faithfully encodes the stimulus. Since the mutual information must lie between zero and channel capacity, it is only the channel capacity that limits the information transfer between stimulus and neuronal response.

Estimating channel capacity by maximising empirical estimates of mutual information can be a difficult task, especially when the experimenter has only an informed guess about the stimuli that evoke responses. One way to finesse this problem is to use adaptive sampling of inputs, which hones in on stimuli that are maximally informative about observed responses [52]. Assuming one knows the stimuli to use, the next problem is the curse of dimensionality. In other words, one requires an enormous amount of data to estimate the probability densities required to quantify mutual information. Although, sophisticated machine learning tools try to estimate mutual information from limited data [53]–[55], the numerics of mutual information are fraught with difficulties.

Irrespective of the thermodynamic or computational imperatives for a biological system, the simple observation that there should be some statistical dependency between sensory samples and the internal states that encode them means that sensory and internal states should have a high mutual information. This leads to the principles of maximum information transfer (a.k.a. infomax) and related principles of minimum redundancy and maximum efficiency [46]–[48]. Later, we will see how minimising variational free energy maximises mutual information and what this implies for metabolic costs in terms of Helmholtz free energy. First, we will briefly review the biophysical and metabolic constraints on the information processing that underlies active inference.

Hitherto, we have considered the strategies that neurons might use for abstracting information from the sensorium. A reliable representation is necessary for an animal to make decisions and act. Such information processing comes at a price, irrespective of whether the animal is at rest or not [56]. Cellular respiration enables an organism to liberate the energy stored in the chemical bonds of glucose (via pyruvate)—the energy in glucose is used to produce ATP. Approximately 90% of mammalian oxygen consumption is mitochondrial, of which approximately 20% is uncoupled by the mitochondrial proton leak and 80% is coupled to ATP synthesis [57]. Cells use ATP for cellular maintenance and signalling purposes, via ion channels that use ATP hydrolysis to transport protons against the electromotive force. Given that the biophysical “cash-register” of a cell (the ATPases) can only handle ATP—and not glucose—we will discuss brain metabolism in terms of ATP.

In man, the brain constitutes just 2% of the body mass, while consuming approximately 20% of the body's energy expenditure for housekeeping functions like protein synthesis, maintenance of membrane potentials, etc. [58]. What consumes such remarkable amounts of energy? Assuming a mean action potential (AP) rate of 4 Hz, a comprehensive breakdown of signalling costs suggests that action potentials use around 47% of the energy consumed—mainly to drive the Na+/K+ pump (Figure 2) [59]. This pump actively pumps Na+ ions out of the neuron and K+ ions inside [60]. In doing so, the pump consumes a single ATP molecule for transporting three Na+ ions out and two K+ ions in [61]–[63]. Measurements of ATP consumption from intracellular recordings in fly photoreceptors show similar energy consumption to costs obtained from whole retina oxygen consumption [64], [65]. Indeed, in the absence of signalling, the dominant cost of maintaining the resting potential is attributable to the Na+/K+ pump. Attwell and Laughlin [59] further estimated that out of 3.29×109 ATP/s consumed by a neuron with a mean firing rate of 4 Hz, 47% was distributed for producing APs, while postsynaptic receptors accounted for around 40% of the energy consumption (Figure 2). These figures suggest that action potentials and synapses are the main consumers of energy and that they determine the energy cost in the nervous system.

Experimental studies have shown that neuronal performance is related to energy consumption, both during rest and while signalling [65]. What these studies show is obvious—there is no free lunch. Neurons have to invest metabolic energy to process information. The finite availability of ATP and the heavy demand of neuronal activity suggest neuronal processing has enjoyed great selective pressure. Metabolic energy costs limit not only the possible behavioural repertoire but also the structure and function of many organs, including the brain [3], [66], [67]. The nervous system can use many tricks to promote energy efficiency. Neurons that use sparse (or factorial) codes for communication [48], [68] save on the number of action potentials required to encode information, or have topographical connectivity schemes to reduce the surface area of axons connecting different brain areas [69]–[71]. Neurons may also alter their receptor characteristics to match the probability of inputs to form a matched filter [49]. Alternatively, specialised signal processing could be employed to convert signals from analogue representation to pulsatile—prohibiting accumulation of noise during information transfer [72], [73].

In short, nature can use various means to achieve the objective of energy efficiency—see Box 1 for a summary of some strategies. Energy consumption in single neurons depends on the types and the numbers of ion-channels expressed on the lipid bilayer, their kinetics, the cell's size, and the external milieu that changes the equilibrium conditions of the cell. Experimental measures from the blowfly retina show that metabolic efficiency in graded potentials (lacking voltage-gated Na+ channels) is at least as expensive as in those neurons displaying action potentials—with the former capable of higher transmission rates [74]. Similarly, in Drosophila melanogaster photoreceptors, absence of Shaker K+ conductance increases energetic costs by almost two-fold [75], [76]. It has also been suggested that the precise mix of synaptic receptors (AMPA, NMDA, mGlu, Kainate, etc.)—that determine synaptic time constants—influences the energetic cost of the single neuron [77]. Recent evidence indicates that the biophysical properties generating an action potential can be matched to make them energy efficient [78]–[81]. Fast Na+ current decay and delayed K+ current onset during APs in nonmyelinated mossy fibres in the rat hippocampus minimise the overlap between the inward and outward currents, resulting in a reduction of metabolic costs [81]. Similarly, incomplete Na+ channel inactivation in fast-spiking GABAergic neurons during the falling phase of the AP reduces metabolic efficiency of these neurons [78]. Applying numerical optimisation to published data from a disparate range of APs, Sengupta et al. [80] showed that there is no direct relationship between size and shape of APs and their energy consumption.