A hierarchy of time-scales and the brain

First, we simulated a surprising song, in which the last chirps were omitted. We stopped the bird's singing after 1.4 seconds, which effectively removes the last two chirps (Figure 3A and 3B). The recognition system, at the first level, correctly predicts zero amplitude auditory input, after the interruption. However, this does not happen immediately but after a short period of about 100 ms. At the second level, the uncertainty about the cause increases massively and maintains its trajectory, following the expected sequence of chirps. At both levels, for a brief period after the interruption, there is a large prediction error (Figure 3C). In summary, the system's response shows that both levels work together to explain the unexpected cessation of sensory input. While first-level dynamics suppress prediction error by fitting sensory data, the second-level representations increase their uncertainty.

This example was chosen to show how hierarchical models might disclose themselves empirically. Consider the simulated LFP responses based on prediction error in Figure 3C. The marked responses at the premature termination of the song (red arrow) can only be explained by a violation of predictions (surprise) over time. This is because we have simulated an evoked response to the omission of a stimulus. In the absence of predictions, a stimulus that is not there cannot elicit any response. The hierarchical nature of these predictions derives from two aspects of the model. The dynamical hierarchy, encoded by the generalised motion within each level, and the structural hierarchy entailed by the two-levels. In the next simulation, we examine their relative contribution to omission-related prediction error responses by removing the second level. We hoped to show that the omission response was attenuated because the prediction from the slower temporal scale was no longer available.

Here we simulated a synthetic bird whose second level had been removed. In Figure 4A, we show the inversion of the ensuing single-level model using the same data as above. The prediction error at the first level in Figure 4C is greater than for the two-level system (Figure 2B). This is expected because the single-level model is not informed about the slowly changing parameter from the second-level attractor. In other words, the two-level system attains a lower prediction error because it can model slow environmental dynamics, which results in a better description of sensory input.

Figure 4B shows what happens when the song stops prematurely. As predicted, the omission response of the single-level system is smaller than for the two-level system and reaches zero more quickly (Figure 4D). This means that the two-level system is less forgiving of violations in long-term temporal structure, when predicting sensory input. This is an important result because it means that, given unexpected input, the two-level model produces a larger prediction error than the simpler single-level model. Usually, models that produce smaller prediction errors are better than models that produce larger prediction errors. In other words, if our task were to model interrupted birdsongs, the two-level model is worse than the single-level model. The critical point is that although this behaviour can be framed as a disadvantage from a modelling perspective, it entails several advantages for the agent:

First, the larger and more enduring prediction error of the two-level system signals that something unexpected and potentially important has happened (a cat might have put an abrupt end to the rendition). The second-level prediction error could then be explained away by supraordinate causes (i.e., a nearby predator) whose representation may be essential for survival. In short, hierarchical systems can register and explain away surprising violations of temporal succession, on extended time-scales. Second, the two-level system can infer slowly changing causes to which the single-level system is blind. These second-level dynamics may carry useful information; for example, that the singing bird is strong and well-fed. Missing this information may pose a serious disadvantage when it comes to choosing a mate. Finally, the second level adds stability to the inversion process and renders recognition more robust to random fluctuations in the environment. The coupling of the fast to the slow level improves inference on degraded sensory input by providing empirical priors. This is shown in Figure 5A, where we increased the noise level of the sensory input by an order of magnitude. The two-level model can cope with this level of noise (although the third syllable is missed; Figure 5A). In contrast, the single-level system fails to predict the sensory data completely (Figure 5B). This difference in recognition is due to veridical prior knowledge from the second level, which confers a more enduring prediction of sensory sequences.

A key aspect of the recognition model above rests on the nonlinearity of the internal model. It is this nonlinearity that allows high-level states to act as control parameters to reconfigure the motion of faster low-level states. If the equations of motion at each level were linear in the states, each level would simply convolve its supraordinate inputs with an impulse response function. This precludes the induction of faster dynamics because linear convolutions can only suppress or amplify the input frequencies; they cannot create new frequencies. However, the environment is nonlinear, where long-term causes may disclose themselves through their influence on the autonomous nonlinear dynamics of other systems. To predict the ensuing environmental trajectories accurately, top-down effects in the agent's internal model must be nonlinear too.

The simulations have shown how environmental trajectories at two different time-scales can be extracted from fast sensory input. This simple example of how a synthetic bird recognises songs provides a metaphor for how the human brain might exploit temporal structure in the environment. Obviously, the brain affords many more levels than two and operates on much higher-dimensional input. However, the principle of hierarchical inference, with separation of time-scales, could be an inherent part of neuronal computations. If the generative model employed by the brain embodies autonomous dynamics that are coupled nonlinearly by control parameters, each level in the hierarchy may represent a specific time-scale. In the following, we will discuss two bodies of neuroscientific evidence for such a mapping: (i) modulatory backward connections which operate at slower time-scales than forward connections and (ii) a cortical gradient of environmental time-scales. We then relate the principle of hierarchical inference to other theoretical accounts in neuroscience.

There is extensive literature on the hierarchical organisation of the brain, in particular of the cortex [41]–[44]. This organisation has been studied most thoroughly in the visual system, where cortical areas are regarded as forming a hierarchy; with lower areas being closer to sensory input. The notion of a hierarchy rests upon the distinction between forward and backward connections [41], [45]–[48]. This distinction is based on the specificity of the cortical layers that are the predominant sources and origins of extrinsic connections in the brain. Forward connections arise largely in superficial pyramidal cells, in supra-granular layers and terminate in spiny stellate cells of layer four or the granular layer of a higher cortical area [41],[49]. Conversely, backward connections arise largely from deep pyramidal cells in infra-granular layers and target cells in the infra- and supra-granular layers of lower cortical areas. Intrinsic connections are both intra- and inter-laminar and mediate lateral interactions between neurons that are a few millimetres away. Due to convergence and divergence of extrinsic forward and backward connections, low visual levels like the primary visual cortex (V1) have small spatial receptive fields, whereas higher visual areas have larger receptive fields; e.g., lateral-occipital cortex [50].

There is a key functional distinction between forward and backward connections that renders backward connections more nonlinear or modulatory in their effects on neuronal responses, e.g., [48]. This is consistent with the deployment of voltage sensitive and non-linear NMDA receptors in the supra-granular layers that are targeted by backward connections. Typically, the synaptic dynamics of backward connections have slower time constants [51]. This has led to the notion that forward connections are driving and elicit an obligatory response in higher levels, whereas backward connections have modulatory effects and operate over greater spatial and temporal scales. This is crucial, because modulatory influence, from a higher level, with slow time constants, suggests that information from the higher level provides contextual information to the lower level. These experimental findings are matched by our theoretical account. In our simulations, evidence for a slow modulatory influence on a lower level can be observed in Figure 3A. Here, contextual top-down influence during online inversion prevents the first-level from reacting quickly to a surprising (unlikely) change in the sensory input. It takes a relatively long period (100 ms) before the dynamic inversion recognizes the unexpected end of the song. This slow transition to a new input regime is due solely to the slow contextual influence of the second level; the transition is much faster (∼10 ms) when one removes the contextual influence (Figure 4B). Note that we did not tune the inversion algorithm to ensure higher levels provide slow contextual guidance for lower levels. Rather, the generative model of a temporal hierarchy enforces that hierarchical separation of temporal scales.

Assuming that the brain employs a temporal hierarchy and that ‘wiring costs’ [52] among levels are minimised, one might expect (i) that low levels of the cortical hierarchy are anatomically close to primary sensory areas and (ii) that the juxtaposition of time-scales (fast to slow) is conserved, when mapped to hierarchically disposed cortical areas. Indeed, systems neuroscience provides experimental evidence that there is a rostro-caudal gradient in cortex, along which the time-scales of representations generally increase, from fast (caudal) to slow (rostral). In Table 1, we list brain areas/systems for which we review the evidence that these form levels in an anatomic-temporal hierarchy in supporting material (Text S1). The time-scales of environmental dynamics in Table 1 are rough estimates based on this review. In this picture, cortico-cortical long-range connections allow for coupling among time-scales. Note that although the view presented in this paper is entirely cortico-centric, we speculate that a cortical anatomic-temporal hierarchy is also expressed in subcortical structures.

The concept of modelling sensory dynamics and their relation to neuronal representations can be related to several approaches in theoretical physics [53]–[59]. The most important is ‘synergetics’ described in Haken [56], where Jirsa and Haken [58] further elaborated the theory to relate it to electromagnetic observations of brain activity. Synergetics embodies the principle that fast dynamics are ‘enslaved’ by slow dynamics, governed by a few ‘order parameters’ naturally incorporating time-scale separation. Synergetics has been demonstrated in behavioural dynamics like bimanual coordination, where the dynamics of finger movements are modelled in terms of fast and slow dynamics. As shown in [59], this framework can be used to analyze brain dynamics as measured with magnetoencephalography. In [57], the synergetics approach was employed to model the recognition of behavioural patterns like arm movements. The principle of a temporal hierarchy might also be linked to accounts of environmental or neuronal multi-scale dynamics, e.g., [53],[54]. In another related approach from theoretical physics, it has been shown that, under certain constraints, coupled nonlinear systems can transfer information from fast to slow time-scales [55].

There is extensive literature on the hierarchical structure of human behaviour, see [60] for a recent example and [61],[62]. In [63], Botvinick proposed a hierarchical model of behavioural sequences, using recurrent neural networks, where high levels in a hierarchy encode slow time-scales, while low levels encode fast input/output. The temporal hierarchy emerged, after learning, without imposing specific constraints. This is an important result, that is shared with several accounts in the robotics literature, where a hierarchy of time-scales in recurrent networks emerges naturally from optimizing a robot to perform navigation tasks [21],[22],[64],[65].

There are several theories that relate to the hypothesis that the operations of specific brain systems pertain to temporal structure of the environment. An exemplary approach is Fuster's sensorimotor hierarchy [12],[66],[67]. Fuster postulates that prefrontal cortex integrates behaviour (motor) over time, while interacting with posterior (sensory) cortical areas. This theory rests on two interacting hierarchies (see Figures 1and 2 in [66]). In spirit, this model is close to what we have formulated. However, one conceptual difference is that we regard the whole of cortex as a single hierarchy. In our model, the unifying feature of the hierarchy is a rostro-caudal gradient of time-scales. Fuster derives the need for two sub-hierarchies from the division of motor and perceptual resources. We believe that this division might prove unnecessary because, according to the free-energy principle, the brain has the singular task of predicting sensory input. This means that the generators of motor output simply predict sensory consequences of anticipated [intended] movements, e.g., [40],[68].

Other models, in particular from motor control theory, try to explain perception and action via forward modelling and reinforcement learning, e.g., [17],[69]. There are several important differences, between these accounts and the approach used above. Our approach uses an explicit separation of time-scales. Another key difference lies in the generality of our inversion algorithm, with nonlinear evolution and output functions at each level (recurrent networks often use linear mixing of the input and a sigmoid output nonlinearity). In addition, our algorithm enables inference on the state precisions such that dynamic uncertainty is quantified. This is probably important for an adaptive agent because behaviour should not only depend on some estimated state of the environment but also on the agent's uncertainty about these estimates. Other differences exist at a more technical level: we use a variational Bayesian framework in generalised coordinates, which enhances the stability and simplicity of the online inversion scheme [36].

There is a large experimental and theoretical literature on coupled neuronal dynamics, e.g., [25],[26],[54],[70], which is distinct from the current treatment. The neuronal dynamics considered in this work are determined by the free-energy principle (Equation 5). This means that any separation of temporal scales emerges explicitly from the generative model which is transcribed from the environment. This separation is not an inherent property of coupled neuronal systems per se. One important implication is that neuronal dynamics themselves may not relate directly to dynamics of sensory input but rather to the inversion scheme used to optimise the model of that input. However, it is interesting to note that there are reports of a simple relationship between the temporal aspects of sensory input and neuronal dynamics, particularly in the auditory domain [70],[71].

We have proposed that the brain employs a hierarchical model, where nonlinear coupling among hierarchical levels endows each with a distinct temporal scale. At low levels of this hierarchy; e.g., close to primary sensory areas, neuronal states represent the trajectories of short-lived environmental causes. Conversely, high levels represent the context in which lower levels unfold. Critically, at each level, representations depend on, and interact with, representations at other levels. We presented simulations that provide a proof of concept that a temporal hierarchy is a natural model to recover information about dynamic environmental causes. In addition, we have discussed empirical findings, which support the conclusion that cortical structure recapitulates a hierarchy of temporal scales.

The principle of a temporal hierarchy provides a theoretical framework for experiments in systems neuroscience. The predictions based on this account could be addressed by making time-scale an experimental factor. For visual areas, Hasson et al. [72] provide a compelling example of such paradigms.

We thank Katharina von Kriegstein for valuable discussions and her comments on an earlier version of the manuscript. We thank Christian Ruff, Chris Frith, Jérémie Mattout, Debbie Talmi, Sven Bestmann, and Felix Blankenburg for their comments on earlier versions of the manuscript.