Evolutionary Connectionism: Algorithmic Principles Underlying the Evolution of Biological Organisation in Evo-Devo, Evo-Eco and Evolutionary Transitions

For example, the ‘free-living’ ancestors of eukaryote organelles initially evolved close symbiotic relationships with the host cell (they were still separate evolutionary units at this stage, but reproducing together), and latterly became reproductively centralised and synchronised (Margulis 1981, 1993).

The consequence of this positive feedback is captured by a principle of correlation becomes causation. That is, variables (phenotypic characters, species populations, evolutionary units) whose behaviours originally co-varied because of a correlated external stimulus (or by accident, or because of selection acting at a lower level) come to have behaviours that co-vary because of their internal interaction structures (i.e., developmental interactions, ecological partnerships or reproductive dependencies). In the developmental domain, Riedl describes this observation as the evolution of developmental architectures that “mimic” the functional constraints on phenotypes (Riedl 1977; Wagner and Laubichler 2004). See also the conversion of alternate “ecosystem states” (configurations that are forced by changes to environmental conditions) into alternate “community states” (configurations that are intrinsic attractors of the ecological population dynamics) (Beisner et al. 2003; Power et al. 2015), and the concept of symbiogenesis in the evo-ego domain (Margulis 1981, 1993; Maynard Smith and Szathmáry 1995).

Although this positive feedback between topology and behaviour (organisation and evolution) is simple when considering individual connections between a pair of entities, the consequences of this principle for the dynamics of larger systems is much more interesting but not immediately obvious. However, connectionist models of learning and memory have characterised the capabilities and limitations of learning models based on this type of feedback in great detail. Such connectionist models can thus be used to recognise common principles across multiple domains (Farmer 1990). We show how this can be used to advance our understanding of this feedback principle in evolutionary systems—hence, evolutionary connectionism. Moreover, we argue that the different consequences of this feedback in evolutionary systems (affecting developmental, ecological and reproductive networks) have specific analogues in different applications of correlation learning (namely, reinforcement correlation learning, unsupervised correlation learning and deep correlation learning, respectively) that help us to understand how natural selection changes the processes of variation, selection and inheritance, respectively, in evolutionary systems.

Connectionism is an approach to cognitive modelling that attempts to explain the cleverness of cognitive processes not by ascribing sophistication to the individual component parts (such as individual neurons) but to the organisation of the connections between them (Garson 2015). Artificial neural network models provide a mechanistic basis to this idea (Hinton et al. 1986; Rumelhart et al. 1986). The field of artificial neural networks has been extraordinarily successful in providing a substantially different way of conceptualising the possible machineries of memory, learning, perception and problem solving as decentralised and distributed processes (Clark 1995; Hinton et al. 1986). They have also been, and continue to be, extremely successful in providing practical machine learning methods for classification, pattern recognition, clustering, data compression and optimisation in innumerable application domains (Rumelhart et al. 1986; Hinton and Sejnowski 1999; O’Reilly and Munakata 2000; Hinton 2007), thus demonstrating that such distributed and decentralised mechanisms can exhibit computationally powerful collective behaviours.

The kind of neural networks that are relevant here are very simple and very well-studied. Such a network is characterised by a number of nodes, in a network of connections, where the activation of each node is a non-linear weighted sum of the input activations it receives from other nodes (note that in gene networks, the expression potential of a gene is generally modelled as a non-linear weighted sum of expression potentials of other genes, and in ecological networks the rate of growth of a species is often modelled as a non-linear weighted sum of other species densities; Watson et al. 2014; Power et al. 2015). The Hopfield network, in particular, has been used as a model for dynamical systems and emergent collective behaviours in many different domains (Hopfield 1982). This is a network where every node is potentially connected to every other node bi-directionally. Neural network models are useful to us in this context because they show that -Many interesting and non-trivial collective behaviours can arise from a network of individually simple components if the connections between them are appropriately organised.Organisations sufficient to produce such behaviours can arise from very simple learning mechanisms that modify connections incrementally.

These learning mechanisms gradually modify the organisation of the system by incrementally adjusting the strength of connections in the network. Modifying connection strengths in this manner has the effect of altering the correlation between the activation of one node and the activation of another. A positive connection produces positive correlation in the activation of the nodes it connects; conversely a negative connection means that when one is activated the activation of the other is suppressed. Adjusting connections in this manner is therefore a type of correlation learning (Hinton and Sejnowski 1999). This type of learning is just a way of implementing the very general idea of associative learning which has influenced cognitive modelling for centuries (Kallich 1945), i.e. learning which objects or ideas go together, or learning which stimuli go together with which outcomes, or behaviours with rewards.

In practice, neural network learning methods often calculate the appropriate change for each connection based on the observed error, i.e. the difference between desired and actual outputs—see, e.g. the Delta Rule and Back-Propogation algorithms (Rumelhart et al. 1986). Such supervised learning assumes that information is provided about what the correct output is during training. But this is not necessary. Correlations can also be learned via reinforcement learning without a priori knowledge about what the correct output is. This can be as simple as modifying connections at random and retaining modifications that improve the output. Whether it is by trial and error (reinforcement learning) or by such supervised learning calculations, the direction of change that provides improvements is the same (at least in the limit where changes affect one connection at a time). Accordingly, an equivalence with the action of natural selection can be verified (Valiant 2013; Watson et al. 2010a). In addition to improving the output of the system immediately, the consequence of such learned connections on the future behaviour of the system is to make those combinations of outputs that were rewarded in the past more likely to occur again in future (i.e. positive feedback on correlations). The subsequent behaviour of the system is thus partly determined by the current inputs at that point time, but partly determined by the past experience of the system recorded, in a distributed manner, in the network of relatively slow-changing interactions.

In contrast to supervised and reinforcement learning, some learning mechanisms are unsupervised which means they do not use a task-based performance metric at all (Hinton and Sejnowski 1999). One very well-known example of this employs a type of Hebbian learning (Hebb 1949), often paraphrased as ‘neurons that fire together wire together’—meaning that the synaptic connection between two neurons is strengthened when the neurons are both stimulated at the same time (e.g. by the same input or stimulus). Under Hebbian learning, the direction of change in the connection is determined by the current output of the system (e.g. whether the two neurons are currently firing) not by a task-specific performance metric. This type of unsupervised learning mechanism is equivalent to reinforcement learning that favours amplification (increase in magnitude) of the current outputs regardless of what they are (i.e. their signs). Because this amplification is enacted through changes to connections rather than independent variables, it has the consequence of reinforcing combinations of values in the current output. This causes those combinations of outputs to become more stable and resilient to perturbation. That is, if one or a small number of the system variables are changed, the weighted connections from other variables that have not changed will force it to change back (or will reduce the external input necessary to change it back). For a given distribution of initial conditions, this means that that particular combination of values is more likely to re-occur (in dynamical systems terms, the initial conditions that lead to a particular attractor state is the ‘basin of attraction’ for that pattern, and the effect of this type of learning is to increase the size of this attractor basin, i.e. to increase the number of initial conditions that lead to that attractor).

Thus, whereas reinforcement learning strengthens correlations that are good (making changes that improve rewards and make good combinations of outputs more likely to occur in future), unsupervised learning merely strengthens correlations that are frequent (making changes that amplify or stabilise the current output and make those combinations of outputs more likely to occur again in future).

Hebbian learning is a fully distributed learning mechanism (i.e. the change in a connection is a function only of the activation in the two nodes it connects) and the change is not a function of system-level performance or any other system-level quantity. This type of learning nonetheless has predictable consequences for the behaviour of the network as a whole. In particular, it effectively ‘internalises’ correlations that are frequent in the input, i.e. two neurons that originally fired together because they frequently responded to the same external inputs, subsequently fire together because of the synaptic connection that has been strengthened between them. This causes the internal structure of the learning system to mimic or ‘mirror’ the structure of patterns observed in the input/external environment (i.e. learning which features of the input co-vary and which are independent), and the activation dynamics of the system to recreate patterns of activation that are ‘familiar’ given its past experience. This is called an associative memory (Hopfield 1982) which has many interesting properties: the ability to store and recall multiple activity patterns, to recall patterns of activation from partial stimuli, to cluster data points into intrinsically similar groups, to classify novel patterns into such categories, to repair corrupted patterns (toward the nearest training pattern), to generate generalised patterns based on structural similarity, perform dimensional reduction/data compression, and to produce idealised exemplars of class from noisy or corrupted training samples (Hopfield 1982; Hinton and Sejnowski 1999). All of these functions are consequences of the simple incremental changes to connections that reinforce frequent correlations.

Correlation learning is thus based on the same positive feedback between topology and behaviour that we observed in the other biological networks. That is, neurons whose behaviours originally co-varied because they were rewarded at the same time, or co-varied because of a common external stimulus, come to have behaviours that co-vary because of their internal interaction structures (i.e., synaptic connections). In the context of connectionist models of learning, it is clear that there are many interesting and well-understood consequences for the subsequent behaviour of the system that follow directly from this basic principle. Crucially, these consequences follow inevitably from this basic principle and are not special to neural networks; any network that exhibits this same principle at the level of individual connections, will also exhibit the same system-level behaviours. In other words, the learning algorithm that is implemented in artificial neural networks is substrate independent and will thus be instantiated in any network that has the same kind of positive feedback.

We introduce the term “evolutionary connectionism” to recognise that, by processes that are functionally equivalent to connectionist models of memory and learning, natural selection acting on the relationships within and between evolutionary entities can result in organisations that produce complex system-level behaviours in evolutionary systems and improve the adaptive capabilities of natural selection over time. The basis of evolutionary connectionism is that the simple principle of positive feedback on the organisation of a system, well-understood in the context of neural network models, is also common to the evolution of developmental, ecological and reproductive organisations. This has the potential to unlock a well-established field of research, with specific conceptual and mechanistic models and results, which can be utilised to understand the evolution of biological organisations under natural selection (Farmer 1990). In particular, this framework helps us to make sense of the bi-directional feedback between evolutionary processes and structural organisations, and also to understand how the consequent changes to the organisation of the parts modifies the collective behaviour and optimisation of the whole. Hence it provides a theoretical framework where we can begin to understand how it is possible for the Darwinian machine to change its own operational mechanisms over time, and how this can improve its ability to produce adaptive change by internalising and exploiting past experience.

It is not too difficult to see that, given heritable variation in interactions and selection, evolving systems will exhibit changes to connections analogous to correlation learning by reinforcement (Watson et al. 2010a; Valiant 2013; Watson and Szathmáry 2015). The evolutionary conditions that correspond to reinforcement learning are simply those that reward changes to interactions that improve fitness (e.g. changes to a gene network that increase fitness by changing the gene expression pattern/phenotype that the network produces, Watson et al. 2014).

If the current output of the network is already the desired output of the network, or even the locally optimal output, then the distinction between reinforcement learning and unsupervised learning is moot (i.e. reinforcing the current behaviour and reinforcing the good behaviour is the same). Unsupervised correlation learning is a suitable model for evolutionary change in such cases. For example: (1) When selection favours increased robustness. Although there may be other quite different phenotypes that are higher in fitness to the current phenotype, local selective gradients may nonetheless favour changes to network structure that amplify the current phenotype or reduce phenotypic variability. If this is enacted by changing correlations (i.e. reinforcing the co-occurrence of features that already co-occur) and not merely by removing variability from individual features, this is equivalent to unsupervised correlation learning. Selection for robustness is another way of saying that the current output is the desired output, but in a locally optimal sense (i.e. all small variations are inferior). (2) The effect of individual-level selection acting on interactions between different evolutionary units within collectives (e.g. individuals within an ecological community). Because individuals adopt behaviours that (locally) maximise individual fitness, individual selection favours variations that enable them to retain their current behaviour (or do more of the same behaviour). This is not, in general, good for the fitness of the collective (the sum of individual fitnesses); whenever individual behaviours are subject to a social dilemma, the behaviours adopted under individual selection do not maximise collective fitness. Nonetheless, in reinforcing the current behaviour of each individual the configuration of the collective is made more stable or robust.

If these changes are enacted by altering the coordination of social behaviours with other individuals, and not merely by altering individual behaviours, this is equivalent to unsupervised correlation learning at the system level; i.e. reinforcement learning at the individual level can produce unsupervised learning at the system level (Power et al. 2015; Watson et al. 2010a). This is yet another way of saying that the current behaviour is the desired behaviour—but here the point is that the current behaviour of the system is the locally desirable behaviour for individuals even if it is not ‘the desirable behaviour’ for collective fitness. In sum, both reinforcement correlation learning and unsupervised correlation learning mechanisms can occur in evolutionary systems; unsupervised correlation learning results from selection for robustness at the system level or from selection acting at a lower level (maximising the utility of the components rather than the collective).

It is not a coincidence that learning systems and evolving systems exhibit the same organisation principles. The reason that connections are changed by reinforcement (or unsupervised) learning in the direction that they are is because that is the direction that improves the output of the system (or amplifies the current output). And the reason that natural selection evolves changes to connections in the same direction is because that is likewise the direction that improves the output of the system (or retains its current fitness levels). It is simply the result of selection for good or non-worse patterns of correlation. By adopting structural configurations that mimic the selective environments they have experienced, or by canalising their current response to it, they can increase fitness or prevent it from being decreased. The consequence of these selected changes is that evolving systems internalise information about the environment (e.g. what combinations of phenotypic features are fit, or what combinations of species can coexist given the prevailing abiotic conditions).

In fact, this kind of change is so basic and natural that it occurs spontaneously in any dynamical system built from a network of malleable interactions—in this sense, neural and evolutionary organisations are both examples of the same underlying principles. For example, even a purely physical system of particles and springs, where the springs, as well as being elastic, are also slightly plastic (e.g. weaken with stress), will exhibit this type of positive feedback (we assume that the spring constants change slowly compared to changes in the particle states that they control). This does not require natural selection; rather, spring-changes merely minimise energy (i.e. adopt a configuration that causes them to do less work, a.k.a. wearing out or deforming under forcing) given the structure of the perturbations exerted on them by the environment. We argue that organisations evolving under natural selection necessarily follow the same basic principle. Each adopts an organisation that reflects the structure of the environment that they are exposed to because organisations that do not will be changed by the action of the environment (Ashby 1956, 1960; Godfrey-Smith 1994; Gell-Mann 1994).

Whereas the consequences of this type of feedback in developmental, ecological and reproductive organisations is very poorly understood, the consequence of this type of feedback in neural networks has been thoroughly characterised by decades of research. This enables us to understand how the action of this feedback on individual connections modifies the organisation and dynamical behaviour of the system as a whole—in particular, its problem solving or optimisation capabilities.

A learning system improves its performance at some behaviour (e.g. classification, clustering, foraging) by incrementally optimising the parameters of that behaviour. Although the behaviour being improved is not usually an optimisation process, it can be. Our recent work shows how a learning system can learn how to optimise more efficiently and effectively over time by beneficially biasing the parameters of the optimisation process (Watson et al. 2010b, 2011a, b). For an optimisation process based on trial and error (formally, ‘generate’ and ‘test’, Watson et al. 2011b), there are logically two different ways in which the trajectory of the search process through solution space can be biased: (a) by changing the test function or selection function, i.e. the way new solutions are evaluated or their apparent value, or (b) by changing the way new solutions are generated. Intuitively, changing the apparent value of a solution (e.g. by artificially increasing the value of certain combinations of variables that are ‘familiar’ given past experience), may enable an optimisation process to escape a local peak in a reward function by raising the value of some points and/or lowering others. Whereas, changing the way solutions are generated (e.g. by re-using a particular combination of variables, i.e. a module, and substituting it for another) may enable an optimisation process to escape a local peak in a reward function by creating directed ‘jumps’ in solution space (Watson et al. 2011b).

This approach can be demonstrated by combining two previously unrelated behaviours, each independently well-understood in the Hopfield network but not previously brought together (Watson et al. 2009a, 2010b):(i)The connections of a (non-learning) network can be defined to represent the constraints of a constraint optimisation problem, and running the network with this interaction structure causes it to find activation patterns that are locally optimal solutions to that problem (Hopfield and Tank 1985, 1986).(ii)A learning network shows an ability to form a generalised memory of past experience by internalising correlation structures observed/experienced in that environment (Hopfield 1982, Fontanari 1990).

Combining these two behaviours in the same network (but on different timescales) defines an interesting new kind of dynamical system. This combines fast state dynamics (with occasional perturbations) and relatively slow changes to connections. Initially the behaviour of the activation dynamics merely finds locally optimal solutions to the problem, as in (i). But at the same time the network is learning. It is not learning a predefined set of patterns as in (ii), however. Rather it is mimicking the patterns of activation that are found at locally optimal configurations. We call this a self-modelling system (Watson et al. 2010b) because the changes to its connections effectively form an associative memory of its own behaviour. This bidirectional feedback between the behaviour of the system and the organisation of the system means that as the organisation of the network begins to change, it changes how the system behaves, and hence changes the solutions that it finds. Specifically, we show that this causes the system to find better solutions more reliably over time, and can enable it to find high-quality solutions that would otherwise be, not just found less frequently, but highly unlikely to be found at all (Watson et al. 2010b). In some cases, unsupervised correlation learning (or equivalently, reinforcement learning applied at the component level), is sufficient to attain these behaviours.

The explanation of why a self-modelling system finds better solutions over time has three parts (Watson et al. 2009b, 2010b, 2011a). First, in systems involving many low-order (e.g. pairwise) constraints, the size of dynamical attractors (if there are multiple attractors) is positively correlated with their depth (i.e. low energy attractor states can be reached from more initial conditions than higher energy attractor states). For this not to be the case the slope of the energy landscape would have to be arbitrarily steep, and this cannot be the case when the energy function is built from the sum of many low-order, e.g. pairwise, interaction terms each of limited magnitude. Second, the positive feedback between the activation dynamics on the network and changes to the topology of the network means that configurations that are most frequently observed are most frequently reinforced. Given the first point, this increases the likelihood that the system finds good configurations that it has found in the past, and decreases the likelihood that it will find inferior solutions over time. Third, and more interestingly, correlation learning forms a generalised associative memory of a set of patterns (Fontanari 1990) and not just a ‘rote’ memory. This means that if there is an underlying structural regularity common to easy-to-find locally optimal solutions that can be represented in a correlation model, then the structure that the network learns will reflect the correlation structure of those patterns and not the patterns per se. The network is therefore able to favour novel patterns with this underlying regularity and is not merely limited to favouring the specific patterns it has already discovered.

Thus, by mimicking the correlation patterns observed in the problem the system can generalise over a distribution of easy-to-find locally optimal solutions to enlarge the basin of attraction for exceptionally high-quality solutions even before such solutions have been visited for the first time (Watson et al. 2010b). In terms of adaptive systems more generally, these conditions create a link between simple habituation behaviours (that canalise the current system state or make it more robust) and adaptation that generates and exploits novel configurations that may be superior to any previously experienced past state. In evolutionary terms, this is the link between robustness and evolvability.

Computationally, this is a simple form of model-building optimisation (Pelikan et al. 1999; Hauschild and Pelikan 2011). These are techniques that learn a model of the problem structure and then use this structure to find better solutions to the problem. But in this case, this occurs without using specialised machine learning mechanisms. It uses only very simple positive feedback principles, based on incremental changes to connections that are generic across a broad range of adaptive networks (Watson et al. 2010b, 2011a; Mills 2010).

Although the optimisation capabilities of self-modelling dynamical systems are demonstrably superior to those of non-learning networks, the capabilities of non-hierarchical systems are limited. Specifically, if a problem has structural regularities that cannot be represented in pairwise correlations, incremental modification of pairwise connections will not be able to capture or exploit this regularity. However, deep or multi-layered networks (Hinton 2007; Rumelhart et al. 1986) can represent higher-order structural regularities that cannot be represented in single-level networks.

The problem-solving principles underlying deep learning are related to problem decomposition and ‘chunking’ (Mills 2010; Mills et al. 2014; Watson 2006), i.e. breaking a complex or high-dimensional problem down into more manageable sub-problems and then assembling together the solutions to these sub-problems in different ways to solve larger problems, and so on. The tricky thing is how to do this ‘bottom-up’ i.e. without a teacher, providing knowledge of how to decompose the problem, to guide the learner through the necessary steps. In neural networks, this type of hierarchical learning is recently referred to as ‘deep learning’ (Hinton 2007; Hinton et al. 2006; Hinton and Salakhutdinov 2006), but the ambition to build deep learning models has been around a long time, e.g. by building networks with many layers; each layer taking inputs from the previous layer, transforming it into a new representation, and passing that on the next layer (Rumelhart et al. 1986). In these cases the overall learning task is often supervised (i.e. a performance metric on the output layer is used). However, the trouble with conventional approaches is that assessing the error on the output of the network massively underdetermines the changes that are required on intermediate layers (a.k.a. hidden layers) of the network (Rumelhart et al. 1986; Hinton 2007). That is, there are many different possible intermediate representations that can give the same outputs for a given input (but only some of these will generalise well). Moreover, it is not clear how to devise an appropriate supervised learning function that operates directly on the intermediate layers (that is, an intermediate representation that is ‘correct’ for one output layer may be useless for a different output layer). An exciting new development in neural network research, Deep belief networks (Hinton et al.

2006; Hinton 2007), has revived interest in deep learning by providing a new approach to this problem. This technique uses unsupervised learning to build intermediate levels of representation one at a time, and ‘freezes’ what has been learned in each layer of connections before the next level of organisation is added on top. This exploits the ability of unsupervised learning to find representations that mimic the intrinsic structure of the problem (without feedback on performance) to reduce the dimensionality of the problem before higher-level layers are constructed. This low-dimensional representation can make task-learning at the next level easier, and thus provides good generalisation for supervised learning at a higher-level of the network even though that level of the network didn’t exist at the time the lower layer was learned. In this manner, a combination of unsupervised and supervised learning is far more computationally efficient than either alone.

Using these deep learning ideas to improve an optimisation process leads to what we might term deep optimisation (closely related to multi-scale search, Mills et al. 2014). The idea here is a process that first uses a simple hill-climbing optimisation process to find local optima, then learns and exploits low-order regularities over a distribution of these optima to find better optima, then learns higher-level regularities from a distribution of these optima to improve optimisation further, and so on. We have demonstrated that this type of learning can efficiently solve optimisation problems that are provably difficult for single-scale search (Mills 2010; Mills et al. 2014). This technique exploits the multi-scale aspect of an individual-based evo-ego model (Watson et al. 2009b, Watson et al., submitted) and implements it in a machine learning method (without the need for an individual-based simulation model). Related but more sophisticated principles are involved in grammar-based optimisation methods. Here a schema grammar, representing the hierarchical correlations observed in a distribution of above-average solutions, is learned and exploited to re-scale the search process at a higher level of organisation (Cox 2012; Cox and Watson 2014).

A deep optimisation approach can also be implemented in a fully distributed neural network model using only local learning mechanisms. This is an extension of the self-modelling dynamical system above and learns connections in exactly the same way, i.e. by reinforcing connections that are frequently observed at local optima. The difference is in what the connections mean—their causal role in the dynamics of the system behaviour. In the simpler self-modelling system, changes to connections have the effect of altering the fitness function of the optimiser, or equivalently, the energy function of the dynamical system—making familiar configurations lower energy (i.e. more likely to be retained). In the ‘deep’ version of the self-modelling dynamical system (a.k.a. “rHN-g”, Watson et al. 2011b), changes to connections do not alter the energy function but they alter the way movements in configuration space are sampled. Specifically, the learned links are used to create clusters of state variables that vary in a coordinated fashion. These higher-level units, or modules, change subsets of variables simultaneously, and these multiple state changes will either all be kept or all rejected, as a unit, depending on the change in energy/fitness that they confer. This enables directed jumps in configuration space—changes to many variables that may be beneficial in combination even if each individual variable-change involved is individually deleterious. Higher-level, clusters of clusters can be created recursively or hierarchically in the same way. This has been shown to solve modular constraint problems that cannot be solved by self-modelling networks that do not have the ability to do this recursive encapsulation (Mills 2010; Mills et al. 2014; Watson et al. 2011b).

These optimisation techniques provide a solution to the problem of inventing a representation that makes a problem easy to solve by recoding the original high-dimensional problem into a lower-dimensional representation of the problem space (Watson et al. 2011b). The positive feedback principle is still apparent in the way connections are learned and used; i.e. variables whose values are frequently correlated become ‘wired’ together in such a way that, after they have been connected, they cannot vary independently—they are transformed into a new emergent or higher-level state variable.