On Markov blankets and hierarchical self-organisation

These equations are the homologues of (2) for the internal and active states, whose flow performs a gradient ascent on the ergodic density over the internal states and their Markov blanket (note that this density does not involve the external states, in virtue of the dependencies in Eq. (1)). In short, the internal and blanket states that constitute a subsystem are autopoietic, because their (nonequilibrium steady-state or ergodic) probability density is maintained by the flow of the subsystem's internal and active states. In the context of spatially dependent interactions, the flows partition expressed in Eq. (4), afforded by the Markov blanket formalism, relies on the spatial separation of states by a spatial boundary.

The flow of the states therefore describes a gradient ascent on the ergodic density. Analogously, in the setting of the stochastic thermodynamics, the system will minimise its thermodynamic free energy (Seifert, 2012). The link between thermodynamic and variational free energy rests upon associating the amplitude of random fluctuations on the motion of states with temperature – and equipping them with particular units through the use of Boltzmann's constant (Friston, 2019; Seifert, 2012; Sekimoto, 1998). This means that the changes in variational free energy inherent in belief updating can be linked directly to changes in thermodynamic free energy in a way that is consistent with the Jarzynski equality (Jarzynski, 1997) and Landauer's principle (Bennett, 2003; Landauer, 1961). Please see Friston (2019) for a fuller discussion and England (2013) and Parrondo et al. (2015) for a related perspective. Although the ergodic density exists, it is not evaluated explicitly by the system, because this would require access to external states that are hidden behind the Markov blanket. However, it is possible to use an alternative formulation that furnishes a description of the flow in terms of a gradient descent on a variational free energy associated with a generative model of the system in question (Friston et al., 2015):

Here, the flow of internal and active states has been expressed as a gradient descent on variational free energy, which is a function of states that are available to the system. This follows because free energy depends on a variational density q(ψ|μ) over external states that is parameterised by internal states, and a generative model p(ψ, s, a, μ|m), which is the system itself, where m denotes the particular system (Friston et al., 2015).

Under this formulation of density dynamics, internal states will appear to infer external states: the third equality expresses free energy as the self-information (i.e., negative log evidence for the model) expected under the variational density minus the entropy of the variational density. This means that internal and active states maximise the joint probability density – expected under the variational density – over states conditioned on the system or model in question. Moreover, internal states will reduce free energy by parameterising a variational density over external states with maximum entropy; in accordance with Jaynes’ principle of maximum entropy (MacKay, 2003). Although not our focus here, when variational free energy is minimised, the variational density becomes the posterior density over hidden or external states, given blanket states. In this sense, the internal states encode posterior `beliefs’ about external states; despite never seeing them directly.

Crucially, the free energy formulation allows us to prescribe the ergodic density in terms of a generative model. In other words, we can write down a generative model and derive the dynamics according to Eq. (5) as a gradient descent on the free energy equivalent of surprise. In what follows, we will simulate self-organisation by specifying a model about the causes of sensory states – and by specifying the environmental dynamics generating those sensations. This means we need to write down the generative model p(ψ, s, a, μ|m) of the system in terms of the dynamics fψ(ψ, s, a) and fs(ψ, s, a) of the environment and how sensory states are generated. Interestingly, the generative process and model do not have to be isomorphic: the generative model has only to approximate the generative process to minimise free energy (Baltieri and Buckley, 2018). The generative model is usually expressed in terms of random differential equations and nonlinear functions with a hierarchical form. In this paper, we will omit these dynamics for simplicity, and specify the relationship between external and sensory states through the following (static) nonlinear functions:

Under Gaussian assumptions about random fluctuations ω, Eq. (5) prescribes the likelihood and priors defining the generative model or Lagrangian:

Here, Π(i) corresponds to the precision or inverse variance of the random fluctuations. This allows us to completely specify the generative model in terms of beliefs about how sensations are generated and priors about hidden states. The key question we address in the next section is: what are the right priors that enable the emergence of Markov blankets at a higher macroscopic level – that would enable us to interpret the ensuing macroscopic dynamics in terms of the self-evidencing above.

In the simulations of subsequent sections, we integrate Eq. (5) using the Matlab routine spm_ADEM.m in the SPM open source academic software. This generalised Bayesian filtering scheme uses the Laplace assumption; i.e., the assumption that the variational density has a Gaussian form. The use of a Bayesian filtering scheme follows because the variational density q(ψ|μ) over external states approximates the posterior density p(ψ|s, a, μ): please see Friston (2014; 2010) for details. In summary, one can use standard Bayesian filtering to simulate self-organisation. This allows one to specify the form of the ergodic or nonequilibrium steady-state density in terms of the priors of a generative model. The question now is: what sort of priors leads to hierarchical self-organisation?

In what follows, we present two sets of simulations. The first considers the self-organisation of an ensemble of synthetic cells, where each cell possesses its own Markov blanket. The second simulation considers ensembles of ensembles to illustrate hierarchical self-organisation; namely, the self-assembly of Markov blankets of Markov blankets. Crucially, these simulations use simple generative models, embodying the prior ‘belief’ that each member can play the role of an internal, active or sensory state within the ensemble. In other words, Markov blankets at one level of organisation possess prior beliefs there is a Markov blanket partition at the level above. However, each cell has no prior belief about its particular role in the higher level Markov blanket – or the form and composition of this blanket. These elementary priors are easy to specify because each role just depends upon the influences each member of the ensemble can or cannot exert on the others. This means, the only hidden state each member needs to infer is which role it plays at the higher level. We will see that this minimal set of prior beliefs (and subsequent self-evidencing) results in the formation of Markov blankets within the ensemble. The ensuing self-similar organisation can, in principle, be extended to any number of hierarchical levels. We illustrate this kind of hierarchical self organisation using 16 cells, each with their own Markov blanket, that organise into a cellular group or assembly, with its own Markov blanket. We then consider an ensemble of ensembles that organises itself into a little organ encompassed in another Markov blanket.

The first simulation illustrates the self-organisation of an ensemble. Each cell interacts with other cells; in a process that eventually leads to a stable configuration with a boundary separating internal cells from their external milieu. This simulation draws on previous work that interest morphogenesis (Friston et al., 2015). In this setting, self-organisation was simulated by minimising the variational free energy of each cell until they attained a prescribed morphology. This morphology was achieved through spatially dependent (e.g. chemical) signalling – so that every cell sensed every other cell in a way that was consistent with their generative models. The morphology was inscribed in beliefs common to all cells, about cell identity, sensation and secretion. Each cell was interpreted as a Markov blanket surrounding internal states: the action (active states) of a cell was the cause (i.e., external states) of the sensations (i.e., sensory states) of the remaining cells. At the beginning of pattern-formation, cells were undifferentiated, because they were uncertain about their identity in the target morphology. As self-organisation unfolded, each cell inferred a unique identity, location and what they should sense at that location. When every cell was in the right place, these inferences were fulfilled; thereby minimising the free energy (i.e., self information or surprise) of every cell.

In more detail, this inference – in analogy to intracellular cascade signalling and epigenetic mechanisms – was driven by the minimisation of free energy. By generating identity-dependent predictions (e.g. genetic and epigenetic expression) about sensations, every cell moved around and produced extracellular signals until its predictions were confirmed. Predictions about sensations caused by other cells (e.g. extracellular signalling) and its own action (e.g. secretion and position) were constrained by prior beliefs about the role of each cell in the target morphology. These prior beliefs were the same for every cell (c.f., pluripotential or stem cells). In other words, based on its identity, each cell had particular expectations about its sensory states. Because sensations were caused by other cells, surprise could only be minimised when every member of the ensemble had inferred a unique role within the ensemble. In short, priors established a point attractor for the ensemble dynamics, in terms of a free energy minimum, leading to differentiation and self organisation to a target morphology.

In the present work, we use the same strategy: we simulate self-organisation of an ensemble of cells, coupled through spatially decaying (e.g. chemical) signals. However, here, there is no target morphology – only the prior that every cell will play the role of an internal, sensory, or active cell, depending upon what it senses. In other words, the priors embody the conditional independencies implied by the existence of a Markov blanket; in the form of intracellular and extracellular signalling between three cell types. As the external states of each cell are the active states of other cells, the system organises in a pattern that enables each cell to predict signals from its companions as precisely as possible.

From the perspective of the ensemble there are no external states. This is an important point, as self-organisation is by definition autodidactic: it does not require coupling with an external environment. The ensuing process leads to a spatial pattern, wherein components of the system are organised in a predictable fashion with respect to each other. Such a pattern is inscribed in the (e.g., genetically encoded) prior expectations about the sorts of signalling a cell should expect to participate in. More precisely, priors are over parameters that specify the form of the generative model, which shapes the free energy landscape, thus defining the attracting states towards which the dynamics of the ensemble converge (Friston et al., 2015).

We now describe our simulation setup. The system comprised sixteen pluripotential cells, which can become one of three types of cells at the next hierarchical level; namely, internal, active and sensory cells. Each cell type secretes a unique extracellular signal and communicates according to the conditional independencies required by a Markov blanket (see Table 2). The external states of each cell comprised its location ψx∈R2 and the chemical signals ψy∈R3 released. This can be expressed as:

Active states ax and ay (e.g., endoskeleton and secretory apparatus respectively) have an immediate effect on external states; hence the identity mapping. This simplifying assumption means we are ignoring time lags (and attenuation), and implies that there is no environment, other than elements of the ensemble contributing to the external states. Its sensory states are the sensed intracellular (produced by itself) and extracellular (produced by other cells) signals. The latter is a function of distance, assuming signal concentration decreases exponentially over space. This can be expressed as:

Here, the sensory noise ω had a high precision (inverse variance) of exp(16). The sensed extracellular signals are returned by the function α(ψx, ψy), which models the spatial decay of signals, where the extracellular sensations of the ith cell are given by

Here, j indexes all cells other than the ith cell. Each cell generates predictions based on the same generative model, which specifies the mapping from hidden states – namely, the type of the cell ψi – to sensations. The type is then the only hidden state that the cells must infer. This inference is parameterised as an expected probability by their internal states μi. Based on beliefs about its type, each cell then generates predictions about intracellular and extracellular sensations:

Here, pα and py are prior beliefs about secretion and sensation given the type of cell (see Table 2), generating sensory predictions, according to the generative model g(μi) (see Eq. (6)), while σ(μi) is a soft-max function that returns expectations about the cells type. The resulting dynamics of internal and active states of each cell can be expressed as follows:

Here ε=s−g(μ) is called a prediction error, and Π(2) is the precision of a Gaussian prior over internal states that parameterise posterior beliefs about external states. The ~ notation denotes generalised coordinates of motion: see Friston, et al. (2010). The appearance of the precision-weighted prediction errors in this equation arises from the Laplace assumption alluded to earlier. Because variational free energy is expressed in terms of log probabilities, the Laplace assumption licenses a locally quadratic approximation to the variational free energy. As such, a second order Taylor series expansion around the posterior mode is sufficient to characterise this functional. Because the gradient of variational free energy evaluated at the posterior mode is zero (by definition), the linear term of this expansion vanishes. The result is that free energy may be expressed as a function that is quadratic in the distance between each variable and its posterior mode (i.e., quadratic in prediction errors). The gradient of the free energy is then simply expressed as a precision-weighted prediction error. Eq. (12) shows that internal and active states minimise (variational) free energy. Thus, internal and active states perform a descent to minimise prediction errors (Friston, 2010). Under these equations of motion, cells infer their identity based on sensations, while secreting according to their role as the ensemble evolves. At the same time, cells move to a position, where extracellular inputs can be best predicted.

Eq. (12) also evidences how surprise of observations given a particular model is reflected in the free energy. The term `surprise’ (a.k.a. surprisal) is used here in the information theoretic sense that quantifies how improbable a given event is (Tribus, 1961). Surprisal is a negative log probability (where the probability in question is the marginal likelihood of sensory states under the generative model). It is this quantity that is upper bounded by the free energy. Under the quadratic approximation to surprise (and variational free energy) employed in this paper, surprise scales with the squared difference between the expected and observed sensory state. This lets us associate surprise with a squared prediction error. As shown in Eq. (12), free energy is also a function of weighted squared prediction errors. As such, a surprising sensory state, when there is a mismatch between expected and observed data, leads to a large, precise, prediction error (ε) and an increase in free energy.

The results of an exemplar simulation are shown in Fig. 3. Self-organisation leads the ensemble to assume a cell-like morphology, with internal cells in the middle, encircled by active cells, surrounded in turn by sensory cells. Because there are no prior beliefs either about the location or about the number of cells per type, these results constitute an emergent property, resulting from the spatial dependency of interactions among agents. In other words, the number of cells of each type is not pre-specified or part of the generative model – it is an emergent property. Furthermore, the arrangement of differentiated cells is not prescribed by each cell's prior beliefs – the arrangement is an emergent property that is consistent with intercellular signalling. This is interesting in the sense that the only arrangements that are consistent with a cell's beliefs about participating in a Markov blanket are exactly the arrangements that are consistent with a Markov blanket of cells: see also Cademartiri et al. (2012).

Although the results reported in Fig. 3 are sensitive to the priors that constitute each cell's generative model (please see discussion), they do not depend sensitively upon the initial states of each cell. In particular, random fluctuations in the initial positions and states of the cell do not affect the self organisation illustrated in Fig. 3. Exactly the same arrangement can be reproduced quantitatively with different initial randomisations. On the other hand, small changes to the priors, such as the spatial decay of extracellular signals – or the motility of cells – do affect the final configuration. For example, the number of internal cells can be greater than one and the final positions of the cells vary with different priors. Interested readers can repeat these simulations using different initial randomisations and prior settings, using open source code (please see Additional Material).

The simulation presented above illustrates the role of Markov blankets in a simple but plausible world where only local interactions are permitted, in which prior beliefs (e.g., a genetic code) have learned that, in order to exist, a living system has to self-generate boundaries that separate it from – and mediate the coupling with – its environment. As in real biological systems, the constituents of an ensemble interact with each other, leading to signal cascades. This signalling rests on inference (e.g., intracellular dynamics) about the role each cell should play, where action (e.g., chemotactic signalling) realises that role. Cells then differentiate, based upon their prior beliefs (e.g., genetic code). In essence, the ensemble reaches a steady state characterised by an internal milieu, which exists – in virtue of assembling its own Markov blanket – as integral part of the system. One could imagine that genes specify Markovian affordances to produce hierarchical structures; such as organs, tissues, organisms and so on. On this view, self-organisation is then a recursive process that engenders, at every level, the emergence of Markov blankets. We now pursue this using an extended simulation set up.

The second simulation considers ensembles of organelles, each comprising an ensembles of 16 cells, to illustrate hierarchical self-organisation; namely, the self-assembly of Markov blankets of Markov blankets and the requisite coupling between levels. To investigate the autonomous organisation of (256) cells at two levels, every cell is equipped with the same (genetic beliefs or) priors about their local and global identity, that is, they share beliefs about possible roles at both the ensemble (local) and ensemble of ensemble (global) level. Practically, each cell now had two sets of hidden states – and prior beliefs – pertaining to their role at the local (i.e., microscopic) and global (i.e., macroscopic) level. Crucially, these priors are the same as used in the previous simulation; namely, they prescribe conditional independencies that are mandated by a Markov blanket at each level:

Here, the superscripts denote the local (ensemble) and global (ensemble of ensemble) level. The only additional piece of information required in this simulation is how the two levels couple to each other. For computational expediency, we model the microscopic dynamics (cells within an ensemble) of only one ensemble of sixteen cells, whereas for the remaining (fifteen) ensembles, we assume that the average behaviour conforms to the local dynamics of the simulated ensemble. This is a mean field approximation in the sense that we discount local fluctuations within each ensemble and assume only their average behaviour is ‘seen’ by any single ensemble. This allows us to simulate the coupling of sixteen cells of the fully simulated ensemble with other fifteen ensemble means (without simulating the other 15 ensembles explicitly). Notice that the allocation of cells to ensembles does not imply an allocation to a particular type of ensemble. The ensembles self-allocate as the Markov blanket emerges at the higher level. In summary, this simulation illustrates how sixteen cells self-organise in an ensemble that in turn self-organises with other fifteen identical ensembles, while describing the coupling between the local and global level.

In particular, for the fully simulated kth ensemble, the global to local extracellular coupling means that it only senses the average of all other global signals, while the local to global coupling means that the average over its n active states informs the dynamics of the remaining ensembles:

where ζ=1:n. The first and second equalities in (13) refer to the extracellular sensing of cells and the intracellular sensation of the ensemble, respectively. In terms of local to global coupling, as they are part of the same ensemble, these predictions will be congruent with each other and cells will therefore act in concert at the global level:

Here, μζg is the expectation about global coupling for each cell in the ensemble.

In summary, sixteen cells locally self-organise in an ensemble, guided by the (local) priors, while interacting with the remaining fifteen ensembles. Exemplar simulation results are shown in Fig. 4, which illustrates hierarchical self-organisation and pattern formation of Markov blankets within Markov blankets. The lower panels of Fig. 4 show the evolution of each cell's expectations (i.e., differentiation) at the local (left), and global (middle) level. The lower right panel shows the expectations of the cells of the sixth ensemble about their role at the global level. The sixth ensemble is an active ensemble at the global level (colour-coded by green), and this global identity appears to constrain the expectations of every cell at the local level.

A first interesting aspect of these simulation results is the rapidity with which the cells infer their type and implicitly differentiate into internal or blanket cells. This was a generic feature of the simulations, suggesting that the priors that lead to hierarchical self-organisation require a fairly rapid specialisation to enable self-assembly a global attractor. This reflects the circular causality or positive feedback loop between microscopic and macroscopic (Markov blanket) dynamics that underlies this kind of self-organisation. In other words, cells move and secrete chemicals accordingly to their inferred role in the Markov blanket partition; simultaneously, inference becomes more accurate as cells approach their final configuration. In the setting of hierarchical self-organisation, inference at the within ensemble (local) level is characteristically faster than at the between ensemble (global) level. This reflects a ubiquitous separation of temporal scales that characterises hierarchical self-organisation (Jung et al., 2015; Kiebel et al., 2008; Perdikis et al., 2011). A second noteworthy point is the spatial structure emerging in these exemplar simulations: when interactions are spatially dependent, active states – which can influence but are not influenced by external states – are enclosed by sensory states, in a similar way as the cytoskeleton resides within cellular boundaries, or muscles within the epithelium. Analogously, sensory states, that can influence but are not influenced by internal states, are segregated from the latter. Notice finally that as cells must possess a Markov blanket to distinguish each other at one level of description, at the level above there will be other ensembles (not shown) from which the simulated one will have to differentiate itself – by means of a Markov blanket.