On Markov blankets and hierarchical self-organisation
Biological self-organisation requires emergence of boundaries, namely Markov blankets.
Hierarchical self-organisation entails emergence of Markov blankets at multiple scale.
Keywords: Self-organisation, Markov blanket, Dynamical systems, Free energy, Active inference
There is growing interest in the role of Markov blankets and associated partitions in understanding self-organisation– and the accompanying self-evidencing that arises from Bayesian mechanics (Friston, 2019). A key aspect of this self-organisation is the hierarchical decomposition of Markov blankets of Markov blankets. This notion has emerged in the literature at several levels; ranging from conceptual analyses in the context of ethology and evolution (Allen, 2018; Clark, 2017; Friston et al., 2015; Kirchhoff et al., 2018; Pellet and Elisseeff, 2008; Ramstead et al., 2018), through to the emergence of multicellular organisms (Kuchling et al., 2019) to the implicit renormalisation group that furnishes a particular perspective on (quantum, statistical and classical) mechanics (Friston, 2019; Friston et al., 2014).
However, despite the potential importance of these conceptual and mathematical analyses, no one has yet provided a proof of principle that Markov blankets of Markov blankets can emerge using numerical analyses. In this paper, we report such a proof of principle by illustrating the emergence of blankets of blankets under the unitary principle of (variational) free energy minimisation. We frame this in terms of self- organisation or pattern formation in cells, to emphasise the simplicity and biological plausibility of the underlying dynamics – although this framing is more by analogy than any detailed consideration of inter-and intracellular communication. Our primary aim was to show that hierarchal compositions of Markov blankets of Markov blankets can emerge from gradient flows on variational free energy, under an appropriate generative model.
In brief, the notion of a Markov blanket allows one to define any system or structure in a way that distinguishes it from the environment or milieu in which it resides. The Markov blanket plays the role of a statistical boundary that allows one to talk about a system per se (Alcocer-Cuarón et al., 2014; Schrödinger, 1944). Such structures can be described at multiple scales, from macromolecules such as ribonucleic acids, through organelles to organs and organisms and even beyond. A Markov blanket is a set of states that separates the internal or intrinsic states of a structure from extrinsic or external states. Importantly, when interactions between states are spatially dependent, as is the case for states pertaining to the physical description of biological organisms, this separation can be spatial in nature. Consequently, in this setting, a Markov blanket describes a spatial boundary. Moreover, this boundary comprises sensory and active states, as is the case for biological systems, like membrane receptors and the cytoskeleton underneath them. In other words, a (biological) physical boundary is a Markov blanket (with sensory and active states), where dependencies between states are determined by location in space. And obvious example here would be the membranes that surround organelles and cells. The crucial aspect of a Markov blanket is that it provides a formal definition of what it means for the internal states of a structure to exist in a way that is conditionally independent of its external states. This definition precludes a (spatially dependent) direct coupling between internal and external states, such that they only influence each other vicariously through the Markov blanket (Friston 2013).
Markov blankets play a central role in several disciplines. For example, in Bayesian statistics and machine learning, they organise the architecture of message passing in neuronal networks and, indeed, the way we implement many statistical tests (Pellet and Elisseeff, 2008). In control theory, they underlie the circular interactions between system and environment (Baltieri and Buckley, 2018). In theoretical biology, they are the cornerstone of variational approaches to self-organisation under the free energy principle. These variational treatments have been applied at many levels; ranging from variational ethology and evolution (Ramstead, Badcock et al. 2017, Constant, Ramstead et al., 2018), through to self organisation, and adaptive behaviour in neuroscience (Friston 2010, Limanowski and Blankenburg 2013), down to morphogenesis and pattern formation at the cellular level (Kiebel and Friston 2011, Friston et al., 2015), and up to mental manipulation and imagination (Hohwy, 2016; Yufik and Friston, 2016). In this sense, the Markov blanket is a scale free concept that underwrites the dynamics of all self-organising systems, at some level.
Markov blankets are not necessarily spatially extensive membranes; they are just a set of states that separates internal and external states. For example, the brain's Markov blanket might include all its sensory receptors and neuromuscular junctions. At the cellular level, Markov blankets can be associated with membranes that surround cells and intracellular organelles, or to the surface of membrane-less organelles where a liquid-liquid phase separation takes place (Mitrea and Kriwacki, 2016). Notably, any spatially dependent interactions between system and environment can exist only in virtue of the permissive role of a statistical boundary; that is, a Markov blanket. However, this does not constrain such interactions within the physical boundary, such in the case of channels allowing ions to cross the cellular membranes or the production of heat by warm-blooded animals (Virgo, 2011; Virgo et al., 2011).
Previous work has already addressed the inextricable link between Markov blankets and living organisms. In particular, any biological self-organising system can be viewed as generating and maintaining Markov blankets at multiple scales (Friston, 2013). Consequently, morphogenesis at any particular level of description becomes the process of constructing a Markov blanket with a particular structure, as exemplified by the organisation of an ensemble of undifferentiated cells into a differentiated target morphology (Friston et al., 2015; Kuchling et al., 2019). In a companion paper, we have articulated the implications that the emergence of nested Markov blankets have for our understanding and interpretation of an organism's dynamics, with the important consideration that Markov blankets do not have to be co-extensive with the biophysical boundaries of an organism (Kirchhoff et al., 2018). These arguments are in turn tightly connected with considerations about a system's cognitive domain that exuberates spatial boundaries, analogously to the cognitive domain of a `glinder’ (a set of On states surrounded by Off states) in the Game of Life (Beer, 2014). In the present paper, we ask how an ensemble of constitutive parts, endowed with Markov blankets, could self-organise to create a Markov blanket at a higher scale; namely, a Markov blanket of Markov blankets. In particular, we focus on the minimal set of prior beliefs, a hierarchically organised system must express, and how these beliefs at different scales are linked.
Our basic conclusion is that a single principle is sufficient to explain the emergence of hierarchical structure; namely the variational free energy principle. This does not imply that hierarchical organisation is an emergent feature of any coupled random dynamical systems; rather, with the right sort of generative model, an ensemble of Markov blankets (e.g., cells) can self-assemble a Markov blanket around the ensemble (e.g., an organ). A generative model here refers to a probabilistic model of how external states influence the Markov blanket that is implicit in the dynamics of internal states. In the current setting, having the right sort of generative model can be regarded as having the right sort of prior (probabilistic) beliefs that are endowed by evolution. In what follows, we will use simulations to provide a numerical proof of principle that minimising variational free energy (under a suitable generative model) leads to hierarchical self-organisation. Throughout the paper ``free energy” will refer to variational free energy. While this is closely related to the thermodynamic concept of free energy (see Friston, 2019 for details), variational free energy is an informational quantity that provides an upper bound on surprise (a.k.a., surprisal or the negative log probability of sensory data).
This paper is organised as follows: first, we review the concept of a Markov blanket in biological systems and draw the link between statistical independence and physical boundaries. By doing so, we provide an intuition on the chief role that Markov blankets have in self-organisation within the free energy principle. We then consider the implications of the existence of a Markov blanket for the behaviour of random dynamical systems obeying the variational free energy minimisation principle. A technical treatment of Markov blankets in the emergence of physical structures and associated (quantum, stochastic, and classical) mechanics can be found in Friston, 2019). This section emphasises the autopoietic nature of systems that (Maturana, 1974), through the dynamics of their internal and active states, resist a natural tendency to disorder. In the final sections, we describe simulations of self-organisation at two levels: these furnish a proof of concept for self-organisation into Markov blankets and the hierarchical formation of blankets of blankets, respectively. We conclude with a discussion of how this treatment relates to other characterisations of biological self organisation.
Biological systems generally segregate themselves from their environment to form boundaries, which define the distinction between what is internal to the system and what is external (Alcocer-Cuarón et al., 2014; Kauffman, 1993; Kelso, 1995; Nicolis and Prigogine, 1977; Schrödinger, 1944). In this paper, these boundaries are formalised in terms of Markov blankets; namely, statistical boundaries that separate internal and external states (e.g., a cellular membrane separating intracellular and extracellular dynamics). In particular, spatial boundaries are an instantiation of the statistical independencies, when physical state interactions are spatially dependent, as is often the case for biological systems. This separation is a fundamental property of self-organising systems, because their very existence implies the presence of a boundary that distinguishes inside (i.e., self) from the outside (i.e., environment).
Living systems maintain the integrity of their boundaries (i.e. Markov blankets), in the face of an ever-changing environment. This means that life has evolved mechanisms for the generation, maintenance, and repair of Markov blankets. A system endowed with such mechanisms connotes an autopoietic organisation that autonomously assembles its own components; in particular its boundaries, (Maturana, 1974; Varela et al., 1974). This autonomy does not imply isolation from the environment, which – on a thermodynamic account – is needed to provide energy (Whitesides, 2002). Therefore, living organisms are operationally closed, while presenting as thermodynamically open. The interaction between system and environment is then mediated by the boundary. Notably, this coupling is non-trivial, in that the organism must actively realise an ‘informational control’ of the environment (i.e., possess a teleology), by filtering, canalising and categorising signals that carry information about their external causes (Auletta, 2010). This implies that the system does not merely respond to sensory states, but reacts to them to infer some (useful) information about the world. At the same time, the boundaries must contain machinery that allows the system to act on external states. In short, definitive borders are essential for living systems, as any dynamics that happens within and between systems can only take place in virtue of their existence (Friston, 2013).
Living organisms are complex systems, denoted by non-linear interactions between multiple hierarchically arranged and nested components (Hilgetag et al., 2000; Kauffman, 1995, 1993; Kirchhoff et al., 2018). As such, characterising how they self-organise requires not only an understanding of how single components couple to each other, but also how microscopic and macroscopic levels interact. This invokes the notion of top-down influences on the low level dynamics (Ellis et al., 2012) and vice versa.
Self-organisation has been addressed extensively in theoretical biology using tools from statistical thermodynamics and information theory to explain how biological systems resist a natural tendency to disorder. This holdout is an apparent violation of the second law of thermodynamics, or at least standard descriptions of it (Evans and Searles, 2002; Seifert, 2012). A more recent line of work within this framework (Friston 2013) sees living organisms as placing an upper (free energy) bound on their self-information (i.e., negative log likelihood of sensed states). This imperative is motivated by the fact that biological systems have to maintain sensory states within physiological bounds. This means the Shannon entropy (i.e., dispersion) of sensory states is necessarily bounded. Shannon entropy is the path or time average of self-information; also known as surprisal or surprise. In short, self-organisation can be regarded as synonymous with systems that place an upper bound on their self-information or surprise. In this variational formulation of self-organisation – that emphasises its inferential aspect – living organisms are understood as placing a (free energy) bound on surprise.
These arguments rest upon ergodicity assumptions (implicit in the fact that the sorts of systems we are interested in have characteristic measures that persist over time). Ergodicity implies that, over a sufficiently long period, the time spent in a particular location of state-space is equal to the probability that the system will be found at that location when sampled at random (Friston, 2013). If this probability measure is finite, it means that any system will revisit all its states (or their neighbourhoods) time and time again. It is this peculiar behaviour that underwrites self-organisation; namely, the existence of an attracting set of states that endow living systems with characteristic states that they visit time and time again.
The existence of an attracting set means that one can interpret the long-term average of surprise as the entropy of the systems sensory states. Crucially, because surprise is (negative) Bayesian model evidence, minimising free energy – defined as an upper bound on surprise – is the same as maximising a lower bound on the evidence for an implicit model of how sensory states are generated. In other words, the system can be regarded as a (generative) model of its environment (Conant and Ashby, 1970), and will look as if it is gathering evidence for its own existence. This has been called self-evidencing (Hohwy, 2016). It follows that – by minimising free energy – biological systems place an upper bound to the entropy of their sensations by inferring their causes; this is also known as active inference (Friston et al., 2010), and is closely related to formulations of the perception-action cycle in the life sciences, like embodied cognition (Clark, 2009), artificial intelligence (Ay et al., 2008), and cognitive neuroscience (Fuster, 2004). In short, self-organisation entails the bounding of self-information that can be cast as self-evidencing.
In what follows, we use mathematical and numerical analyses that build upon a free energy formulation of pattern formation (Friston et al., 2015). We start with subsystems whose dynamics possess a Markov blanket as an attracting set. We then integrate the system until it self-organises into a stable configuration. Subsequently, we extend the simulation to consider hierarchical systems; namely, configurations of configurations (i.e., blankets of blankets) that could, in principle, be extended indefinitely.1 These simulations were used to test the following hypothesis: if the maintenance of Markov blankets can be cast as self-evidencing, then self-organisation should be an emergent property (Kirchhoff et al., 2018) of subsystems that `believe’2 they participate in – or are enclosed by – a Markov blanket. Because Markov blankets are defined by conditional independencies, the requisite beliefs can be specified simply, in terms of communication or signalling between subsystems. In other words, it should be possible to reproduce hierarchical self-organisation by equipping subsystems with beliefs about how they influence – and are influenced by – other subsystems. The next section considers the formal basis of our simulations, based upon random dynamical systems and their probability density dynamics.
This section associates random dynamical systems with living organisms, where the states of a system stand for its internal states (e.g., intracellular states), its blanket states (e.g., receptors on a cell membrane and the actin filaments of the cytoskeleton) and external states (e.g., extracellular milieu). The systems under consideration are complex (i.e., non-linear and hierarchical) and organise independently of any applied or external gradient: we will see that such systems exhibit a process of pattern generation that lead to definitive boundaries (i.e. Markov blankets), defining internal states and their relationship with external states.
The notion of a Markov blanket was originally proposed in the context of Bayesian networks or graphs (Pearl, 1988), where it refers to the parents of the set of states (that influence it), its children (that are influenced by it), and the children's parents. The Markov blanket defines the conditional independencies between a set of states Λ (the system) and a second set of states Ψ (the environment). This concept can be translated into a biological setting: for example, the intracellular milieu of a cell represents the internal states and the plasmalemma corresponds to the Markov blanket, through which communication between intracellular and extracellular states is mediated (Auletta, 2013; Friston, 2013). Crucially, the Markov blanket can be decomposed into sensory S and active A states, which are and are not children of the external states, respectively. Thus, the existence of a Markov blanket S × A induces a partition of states in x∈X=Ψ×S×A×Λ; external states act on sensory states, which influence, but are not influenced by internal states. Internal states couple back through active states, which influence but are not influenced by external states (Table 1). This partition ensures a statistical separation between internal and external states in the sense they are independent, when conditioned on the Markov blanket.
How this statistical concept can be translated into a biological setting, and why is its presence so important? We start by considering a system in which long-range (e.g., electromagnetic) interactions are possible, and states’ identity rests upon their coupling. Here, every state interacts with all others, irrespective of its spatial position; every state is therefore indistinguishable from the remainder, because the fully interconnected nature of the system precludes any statistical separation of one state from another (Fig. 1a). To engender statistical structure (i.e., an identity), coupling has to be limited via interactions restricted in space. One possible scenario displays two sets of states located far enough for them to be statistically independent: the state of one set does not influence the other and vice versa (Fig. 1b). However, in this case interactions between sets are also precluded. When considering biological systems and their environment, this scenario becomes unrealistic, given the definition of biological organisms as open systems (Whitesides, 2002). Therefore, two sets of states can be associated in a meaningful way to a biological organism's intrinsic (i.e. internal) and extrinsic (i.e. external) states only when a Markov blanket exists, which defines conditional or spatial independencies (i.e. identities) and interactions between the two states. Notably, it is the restriction in space of interactions that justifies the association between statistical and spatial independence, thus between Markov blankets and spatial boundaries. This is the minimal thus most general description of a biological self-organising organism possible (Fig. 1c).
Notably, interactions occurs in virtue of the partition of the blanket states into sensory and active states, mediating the vicarious influence of external on internal states and the influence of internal on external states, respectively.
In reality, segregation emerges in the presence of coupling. In other words, a subsystem differentiates itself from the environment, but remains (statistically or energetically) coupled to it. This is possible when two sets of states are conditionally independent not just because of their spatial separation, but in virtue of a third set; namely, blanket states (Fig. 1c). These blanket states comprise sensory and active states, mediating the vicarious influence of external on internal states and the influence of internal on external states, respectively. This concludes our description of a minimal partition that enables a meaningful separation of internal and external states.
How can the concept of Markov blanket expand when considering the hierarchical structure of biological systems? Let us group internal and blanket states into a single (macroscopic vector) state. If this macroscopic state participates in some meaningful structure, a macroscopic Markov blanket has to emerge, whose sensory and active states – and the internal states insulated within – will each be composed of microscopic Markov blankets. Hence, the formation of Markov blankets at any level of hierarchical organisation is intimately linked to the maintenance of Markov blankets `all the way down’ (Fig. 2). On this view, self-organisation is a recursive process of boundary formation that spans all levels of hierarchical organisation. Later, we provide a proof of concept for this argument by simulating the hierarchical self-organisation of Markov blankets.
In summary, self-organisation has to feature the emergence of boundaries that define an internal state space, separating it from external states, while allowing for vicarious coupling. It follows that hierarchical self-organisation requires the emergence of Markov blankets of Markov blankets. In the next section, we turn to the nature of the dynamics that underwrite this emergence.
We will be dealing with random dynamical systems expressed as Langevin equations of the following form:
This describes the dynamics of a system with a Markov blanket in terms of the flow f(x) of its states and random fluctuations ω. The flow of external ψ ∈ Ψ, sensory s ∈ S, active a ∈ A and internal states μ ∈ M in the second equality, conforms to the dependencies implied by a Markov blanket (see Table 1). External states can only be influenced by internal states through their Markov blanket, and are therefore called hidden states, because they are hidden behind the Markov blanket. In the specific setting of biological systems, the partition in Eq. (1) relies on the spatial location of states, and identifies sensory and active states as components of spatial boundaries.
An alternative formulation of Eq. (1) is in terms of a Lagrangian, which allows us to describe the system's dynamics in terms of a gradient flow using the Helmholtz decomposition. This rests upon ergodic assumptions implied by the existence of an attracting set, called a pullback or random global attractor. Following Crauel et al. (1999), Crauel and Flandoli (1994) and Friston (2013), one can express the flow of states in terms of a divergence-free component and a curl-free descent on a Lagrangian L(x) that corresponds to the self-information or surprise associated with any state. This rests upon ergodic assumptions implying the existence of an attracting set (conditioned of the model) in the state-space, called a pullback or random attractor (Crauel et al., 1999; Crauel and Flandoli, 1994), and an associated probability density, the ergodic density.
Here, diffusion tensor Γ is half the covariance of the random fluctuations, and Q is an antisymmetric matrix that satisfies Q(x)=−Q(x)T. The equality p(x|m)=exp(−L(x)) is the solution of the Fokker-Planck equation describing the density dynamics (Friston and Ao, 2012; Frank, 2004), where m denotes a particular system or model (see Variational Free energy section). This ergodic or nonequilibrium steady-state density is the probability density at which its rate of change is zero. Eq. (2) means the states of a system m at nonequilibrium steady-state are performing a gradient ascent on the ergodic density. This is revealing, because it shows that the system's flow counters the dispersive effects of random fluctuations – by flowing towards the attracting states.
This gradient flow formulation also applies to the flow of internal and active states