On Markov blankets and hierarchical self-organisation

In the example shown in Fig. 4, local expectations (lower left panel) converge quickly, with a discernible differentiation after the first time step. Conversely, local expectations about the global type take much longer to converge, with the degree of uncertainty in some cells that only resolves at the final time step (see cell 12 in the lower right panel). This example may reflect the separation of timescales formalised in synergetics, and in particular by the slaving principle, which deals with self-organisation and pattern formation in open systems far from thermodynamic equilibrium (Carr, 1981; Ginzburg, 1955; Haken, 1978; Tschacher and Haken, 2007). In this setting, slow macroscopic patterns of activity are said to enslave fast microscopic patterns, while the macroscopic patterns (known as order parameters) are constituted by the microscopic patterns; hence circular causality.

In this paper, we have presented a variational treatment of hierarchical self-organisation. Given local interactions, carefully crafted prior (genetic) beliefs about conditional dependencies and independencies endow a system with a point attractor comprising internal states and their Markov blanket. Moreover, applying the same priors at any hierarchical level leads to the emergence of Markov blankets within superordinate Markov blankets. A key feature of the simulations – used in this paper – is the absence of any explicit target morphology within the prior (e.g., genetic) beliefs of the system's denizens. This is an emergent property, which appears to have a top-down effect on the blankets below. The subsequent emergence of a cell-like structure is interesting because it speaks to characteristic spatial boundaries found in most biological systems; namely, cellular membranes. The isomorphism between a statistical and spatial boundary rests on spatially dependent interactions among internal and external states. In other words, the states of a system can include its generalised motion in physical space, such that the blanket states acquire the attribute of a spatial location (and velocity). In turn, this means that spatial boundaries can be identified with statistical boundaries, under conditional dependencies of the flow internal and active states on external states. As noted above, these normally involve short range statistical couplings or, in physical terms, forces or chemical gradients (Ao, 2009; Seifert, 2012). Crucially, this sort of hierarchical self-organisation is a recursive process that can repeat itself at higher levels of description.

Interestingly, this perspective enables to identify ensembles that are or are not self-organising: in this optic, the distinction between a culture of cells and a multicellular organism resides in the emergence of a Markov blanket at the ensemble level. Another consequence of this recursive aspect is the absence of a privileged point of view, when describing hierarchical self-organisation: the dynamics at every level play the role of macroscopic states at the level below, and the role of microscopic states at the level above. An apparent example is morphogenesis, during which chemotaxis and development of a system's components occurs within a global configuration established by morphogen gradients controlling gene expression (Balaskas et al., 2012; Chang et al., 2002). Notably, in our simulations, morphogen gradients could be interpreted as the expression of beliefs about the possible role at the ensemble level that each cell or element at the level below must have. Actuation of these beliefs (morphogenesis) then occurs through an inference process (differentiation), and realisation of the corresponding positional predictions (chemotaxis); for an example that uses exactly the notion of a Markov blanket and chemotactic signalling, please see (Kuchling et al., 2019). At a subcellular scale, the same logic applies, so that self-organisation of microscopic organelles like the cellular membrane and vesicular structures is constrained by beliefs about their role at the cellular level. On the same note, the dialectic between hierarchical layers formalised above accounts for the dynamics of multi-agent, complex systems, ranging from cultural ensembles (Ramstead et al., 2018) to complex urban environments (Hadfi and Ito, 2016).

Here we associate biochemical structures, gradient flows and kinetics with Bayesian priors and belief propagation or updating – as opposed to propositional or representational beliefs. The only thing that licenses the use of the word ‘belief’ is that the macromolecular and cellular kinetics at hand were cast as a gradient flow on variational free energy. This means that one can interpret the resulting dynamics in terms of Bayesian belief updating (Winn and Bishop, 2005). However, this is an `as if’ interpretation; in the same sense that the folding of a macromolecule to minimise its thermodynamic free energy in computational chemistry – e.g., Lammert et al. (2012) – looks `as if’ it is trying to minimise (thermodynamic) free energy.

The advantage of being able to formulate this kind of self-organisation in terms of gradient flows on variational (as opposed to thermodynamic) free energy is that variational free energy is a functional of a generative model. This means that one can prescribe the desired endpoint of self-organisation in terms of priors (Friston et al., 2014). One might ask; where do priors come from? Here, we assume that they are a product of natural selection (i.e., Bayesian model selection) and are therefore entailed in genetics and epigenetics (Campbell, 2016; Frank, 2012; Ramstead et al., 2018). We do not try to provide a detailed account of the ensuing gradient flows in terms of molecular biology; e.g., Tabata and Takei (2004). However, one can imagine hypotheses based on variational free energy gradient flows can be tied to intra-and inter-cellular signalling; e.g., Cervera et al. (2019) and Friston et al. (2015) and Kuchling et al. (2019). Interestingly, exactly the same challenge arises in the neurosciences, where the equivalent gradient flows become neuronal dynamics – and the accompanying challenges become understanding neurophysiology and neuronal microcircuitry in terms of variational message passing; e.g., Friston et al. (2017).

It is interesting to ask how the priors that underwrite this kind of self-organisation are updated in terms of cell biology. The usual response to this is to consider hierarchal processes of free energy minimisation in terms of Bayesian model selection (Allen, 2018; Campbell, 2016; Frank, 2004; Ramstead et al., 2018). This leads naturally to a link between natural selection and Bayesian model selection based upon the evidence bounds afforded by variational free energy. On this reading, natural selection becomes a form of structure learning (a.k.a. Bayesian model selection) based upon the model evidence associated with a particular phenotype. In short, the (marginal) likelihood of a particular phenotypic structure – in an evolutionary setting – is optimised in terms of its prevalence in a population. Because this structure is a model of the external milieu, it entails particular priors. If these priors are fit for purpose in terms of minimising variational free energy then they will be selected. For example, under some mild assumptions, the replicator equation can be cast as a Bayesian filter; exactly along the lines of the above argument: see Frank (2012), Geisler and Diehl (2003) and Ramírez and Marshall (2017) for further discussion.

In our simulations, and more generally, we have made some mild assumptions about the external or environmental states that contextualise self-organisation at the highest scale considered. This speaks to an important conceptual point; namely, that a partitioning of systemic states into Markov blankets at any scale is always contextualised by the scale above. In other words, there must be a permissive context in which self-organisation unfolds at, generally, faster timescales in the level below (Schwabl, 2002; Jeffery et al., 2019). Strictly speaking, this implies an infinite regress; in the sense that we can only talk about Markov blankets at one scale of self-organisation by assuming some attracting set at a higher scale. Indeed, this recursion can be formalised in terms of the renormalisation group that emerges from grouping and course graining (i.e., reduction) operators on the Markov blankets (Friston, 2019). The renormalisation group formulation implies that the time constants of self-organisation at larger scales necessarily increases, when moving from one scale to the next.

Practically, this means that one can assume that the external states that encompass the formation of Markov blankets at the highest scale under consideration are changing slowly in relation to lower scales. The picture that emerges here is that the same basic (Bayesian or variational) mechanics emerge in a scale-free fashion at different levels; from the quantum through to the level of molecular biology; from the scale of cells through to organs, from phenotypes through to species; all the way up to a cosmological scale. Although this might sound fanciful, this perspective has some currency in relation to the differences between quantum, statistical and classical mechanics. These differences rests largely upon the suppression of random fluctuations as one progresses from the small to the large. We have chosen to illustrate coupling between just two levels; namely, the mesoscopic level of cells and cell assembly in biology.

One might ask why we have focused on the free energy principle, as opposed to other formal descriptions of self-organisation (Haken, 1978; Kauffman, 1993; Kelso, 1995; Nicolis and Prigogine, 1977); for example, phase transitions in spin models (Vatansever and Fytas, 2018), attractor landscapes in random Boolean networks (Gershenson, 2012) or Turing style pattern formation via reaction diffusion systems (Halatek et al., 2018). Our motivation for casting self-organisation as a variational principle was threefold: first, the free energy principle provides an integrative formalism that should apply to all the above. In other words, it regards pattern formation and self organisation – as manifest in reaction diffusion systems and other nonequilibrium steady-state dynamics – as realisations of the same principle. This is self-organisation to a random attractor (Crauel and Flandoli, 1994; Friston and Ao, 2012). When this attracting set possesses a Markov blanket the free energy principle must apply (Friston 2013). This means that one can interpret any form of self-organisation – to an attracting set – in terms of a gradient flow on variational free energy and, implicitly, self-evidencing (Hohwy 2016, Ramstead, Badcock et al. 2017). This is important because most existing approaches to the dynamics of self-organisation try to reverse engineer an energy functional (or Lyapunov function), given some dynamics or equations of motion. The free energy principle allows one to invert the problem and write down the dynamics as a gradient flow on a free energy functional that is specified in terms of a generative model (Friston et al., 2014). Crucially, the prior beliefs of this generative model determine the attracting set.

By explicitly writing down a Markov blanket in these priors, we obtain a system whose organisation is the most general and essential possible, implicitly in any self-organising system behaving accordingly to the free energy principle. Finally, using the notion of random dynamical systems (Arnold and Crauel, 1991; Crauel and Flandoli, 1994), the free energy principle allows one to articulate questions about (and simulate) self-organisation at multiple scales. Here, we have focused on the link between just two scales; however, by induction, the conclusions from this paper could be generalised to multiple levels – at least in principle. This hierarchical aspect would be challenging to simulate using conventional approaches to pattern formation.

As noted in the introduction, our aim was to provide a numerical analysis of the minimal conditions under which hierarchical self-organisation emerges – and show that the minimisation of variational free energy provides a sufficient account, under the right sort of generative model. Crucially, this does not mean that any free energy minimising ensemble will show this kind of hierarchical self organisation. Our agenda was not to suggest all systems self-organise hierarchically; rather, we wanted to explain the existence of the hierarchical structures seen in biology, in terms of variational principles.

Practically, the behaviour illustrated in the above simulations depends sensitively on priors in the generative model and initial conditions. In more details, setting prior beliefs that specify an attractor in state space (endowed with a Markov blanket) is not trivial, and growth in the size of the system further complicates the task. This is reminiscent of the emergence – at later evolutionary stages – of bigger or more complex organisms. Furthermore, these simulations show the final stage of self-organisation, where the system finds itself in the vicinity of the attractor; initialising the system too far from this attracting point (e.g., by adding extra perturbation to location) can prevent the system from elaborating a bounded structure (i.e. existing). This, in turn, speaks to the difference between these (minimal and general) simulations and the complexity of (specific) biological systems, endowed with a plethora of control and feedback mechanisms, which underwrite robustness to perturbations. This sensitivity to prior parameters and initial states leads to some interesting questions. For example, questions about the rate at which the structures stabilises – and how this depends upon the parameters (c.f., rate spatial decay constants) that constitute each cell's priors. How do the initial values (e.g., position) affect self-organisation? These questions raise an interesting issue: is there anything special about a hierarchical structure that would explain its prevalence in biotic systems. One speculation here might be that a hierarchical (self-similar) architecture of Markov blankets might be a free energy minimising solution on a longer time scale, such as evolution. This should be possible to address via simulated (pharmacological) lesion experiments that block the formation of higher-order Markov blankets.

One can then measure the free energy with and without hierarchical self-organisation and consider the implications for natural selection. In variational formulations, natural selection is treated as a form of Bayesian model selection, based upon model evidence or variational free energy (Campbell, 2016; Frank, 2012). We hope to pursue this in subsequent work.

This work suggests that Markov blankets are a fundamental characteristic of biological systems. Their presence is necessary for life – as they underwrite an existential separation of the system from its environment, while preserving its interactions. The hierarchical organisation of complex systems – like living organisms – implies that the self-similar organisation of Markov blankets may be evident at any level of biological structure. From the point of view of dynamical systems, Markov blankets are attractors, attracting fast microscopic dynamics, while underwriting the emergence of macroscopic (order) parameters. This circular causality nicely captures the self-organisation of biological systems, which evolve autonomously with a morphology (Markov blanket) that is necessarily predisposed to a selective coupling with external states. The natural place – where these attractors might be specified – is the genetic code. Clearly, this is rather speculative; however, it is possible that the astonishing diversity of flora and fauna we witness might reflect the fact that, in a world where signals are spatially dependent, Markov blankets are synonymous with existence.

Simulations: The simulations reported in this paper can be reproduced using the open access academic software SPM (http://www.fil.ion.ucl.ac.uk/spm/software/). The key routines are DEM_cells.m and DEM_cells_cells.m that illustrate self-organisation of a single ensemble and ensemble of ensembles respectively.

DEM_cells.m: This demo illustrates self-organisation in an ensemble of (sixteen) cells using the same principles described in DEM_morphogenesis, but using a simpler generative model. Overall, the dynamics of these simulations show how one can prescribe a point attractor for each constituent of an ensemble that endows the ensemble with a point attractor to which it converges. In this example, we consider the special case where the point attractor is itself a Markov blanket. In other words, cells come to acquire dependencies, in terms of intracellular signalling, that conform to a simple Markov blanket with intrinsic or internal cells, surrounded by active cells that are, in turn, surrounded by sensory cells. This organisation rests upon intracellular signals and active inference using generalised (second-order) variational filtering. In brief, the hidden causes driving action (migration and signalling) are expectations about cell type. These expectations are optimised using sensory signals; namely, the signals generated by other cells. By equipping each cell with prior beliefs about what it would sense if it was a particular cell type (i.e., internal, active or sensory), they act (i.e., move and signal) to behave and infer their role in an ensemble of cells that itself has a Markov blanket. In a DEM_cells_cells.m, we use this first-order scheme to simulate the hierarchical emergence of Markov blankets; i.e., ensembles of cells that can be one of three types at the local level; independently of their time at the global level.

DEM_cells_cells.m: This demo is a hierarchical extension of DEM_cells.m, where we have 16 ensembles comprising 16 cells. Each cell has a generative model (i.e., prior beliefs) about its possible local and global cell types (i.e., internal, active or sensory). Given posterior beliefs about what sort of self it is at the local and global level, it can then predict the local and global intracellular signals it would expect to receive. The ensemble of ensembles then converges to a point attractor; where the ensemble has a Markov blanket and each element of the ensemble comprises a cell – that is itself a Markov blanket. The focus of this simulation is how the local level couples to the global level and vice versa. For simplicity (and computational expediency) we only model one ensemble at the local level and assume that the remaining ensembles conform to the same (local) dynamics. This is effectively a mean field approximation, where expectations of a cell in the first ensemble about its global type are coupled to the corresponding expectations and the ensemble level, and vice versa. The results of this simulation are provided in the form of a movie and graphs.

Ensor Rafael Palacios and Karl Friston conceived the ideas, wrote the manuscript and performed the simulations. Adeel Razi conceived the ideas, reviewed the manuscript text and contributed to the simulations. Thomas Parr and Michael Kirchhoff reviewed the manuscript.

Ensor Rafael Palacios: Conceptualization, Methodology, Writing - original draft, Visualization. Adeel Razi: Conceptualization, Writing - original draft. Thomas Parr: Writing - original draft. Michael Kirchhoff: Writing - original draft. Karl Friston: Conceptualization, Methodology, Writing - original draft, Visualization.

This work was funded by the Wellcome trust. KJF is funded by a Wellcome Trust Principal Research Fellowship (Ref: 088130/Z/09/Z). We also thank our reviewers for helpful guidance in describing this work.