Explore-exploit trade-off and foraging heuristics

Reid and colleagues (2016) describe the problem as one that faces most organisms. Options must be explored before a commitment is made, and changing conditions force constant re-evaluation.² They treat the multi-armed bandit as the classic experiment for it. The name comes from casino slot machines, where choosing which machine to play so as to maximise payoff is nearly impossible for the average gambler to solve consistently.² Reid (2023) adds that the paradigm has been applied to humans, birds, fish and social insects, and also to Physarum.⁵ Both works frame the question the same way: exploit what is familiar, or risk exploring for more.²

In the two-armed bandit test of Reid and colleagues (2016), plasmodia sat between two environments that differed in food availability and profitability. Reid (2023) reports that the study showed Physarum comparing options, integrating over sequential samples, and combining reward frequency with reward magnitude.⁶ The authors proposed ten candidate rules of varying complexity. The simplest was autocorrelation, which means moving in the same direction as the previous timestep.⁷ It selects the arm with the highest index, which accounts for expected future reward from both exploring and exploiting an arm.³ The passages here do not say which rule the organism best matched.

Exploring and exploiting also appear in definitions of cognition. Ciaunica, Shmeleva and Levin (2023) quote Lyon's definition: the mechanisms an organism has for becoming familiar with, valuing and interacting productively with its environment, by exploring, exploiting and evading.⁸ Rorot (2022) quotes a similar description of basal cognition as a toolkit of capacities that act on environmental features in service of existential goals.⁴ Reid (2023) draws the practical conclusion that neural and non-neural decision-making share at least some underlying principles. He argues that these cannot be understood until decision-making outside a brain is better studied.⁵

Friston and Ao (2012) offer a different account in simulation. A single scalar cost function of position produced adaptive goal-directed behaviour without value-learning or enforced exploration of the environment. Exploration and exploitation appeared as a direct consequence of the agent's priors, with no carefully calibrated stochastic terms during training.¹ They compared this to insect foraging, in which the environment is explored under autonomous dynamics until a reward is met.¹ Their account differs from the bandit heuristics above, which are rules for sampling and switching among options.