Goals and control · Article

Expected free energy, policy selection and utility versus value

In active inference, an agent selects a policy, a sequence of actions, by minimising expected free energy. Friston and colleagues argue that this removes the separate value function that classical control theory requires.¹² They also distinguish the utility of states from the value of policies.⁴ Friston and colleagues (2013) present this distinction as a link to classic formulations and as showing utility is relative.⁴

Earliest held: 1859, Darwin
Most discussed in: Active Inference, homeostatic regulation and…, 2015
In the library: 411 passages in 50 works
Rewritten: 2026-10-20

The classical value function

Friston, Daunizeau and Kiebel (2009) set active inference against dynamic programming and value-learning. In those schemes a policy is optimised on the basis of a value-function of hidden states, which corresponds to expected reward or negative cost.¹ They state two requirements of this route. The agent must have access to the hidden states, and the value-function must be solved for. They call that solution non-trivial.² Their alternative prescribes the policy in terms of free energy, which they say encodes optimal control.²

Value as negative surprise

Friston and Ao (2012) treated value from the side of dynamics. They showed that a policy, written as empirical priors on flow, can be split into curl and divergence-free components. In that decomposition the value-function is just negative surprise, and it defines the ergodic probability density over hidden states.³ The authors add that this does not say where the policy comes from. They state that the rest of the paper concerns how policies are specified and how value-functions instantiate them.³

Utility versus value

Friston and colleagues (2013) drew the contrast between utility and value. Utility is a fixed attribute of states that agents are attracted to, while the value of a policy is context sensitive and depends on the current state.⁴ They state that active inference has no value function and that free energy is the only quantity optimised.⁵ On their account, value is a consequence of behaviour rather than its cause, since some states are simply occupied more often.⁵ Pezzulo, Rigoli and Friston (2015) noted that precision corresponds to the temperature of softmax choice rules, so that exploration follows from the scheme and is not tuned as an ad hoc parameter.⁶

Expected free energy and affordance

Later papers apply expected free energy to organisms and their niches. Kirchhoff and colleagues (2018) state that the only self-consistent prior in active inference is that actions minimise expected free energy.⁷ Constant and colleagues (2018) describe salience as information gain or epistemic value, which they place within expected free energy.⁸ Ramstead and colleagues (2019) cast affordances as expected free energy differences. For discrete-state models they equate abilities with policies, and they quantify each affordance by the expected free energy under that policy.⁹

SourcesEach quotation was checked word for word against the passage it opens.

Dynamic programming and value-learning try to optimise a policy based on a value-function of hidden states, which corresponds to expected reward or negative cost.Friston KJ, Daunizeau J, Kiebel SJ, 2009 · Reinforcement learning or active inference? · open at passage 45
The free-energy formulation circumvents these problems by prescribing the policy in terms of free-energy, which encodes optimal control (Equation 6)Friston KJ, Daunizeau J, Kiebel SJ, 2009 · Reinforcement learning or active inference? · open at passage 47
Crucially, this value-function is just negative surprise and defines the ergodic (invariant) probability density over hidden states.Friston K, Ao P, 2012 · Free energy, value, and attractors · open at passage 59
Utility is a fixed attribute of states that agents are attracted to. In contrast, the value of a policy is context sensitive and depends upon the current state.Friston K, Schwartenbeck P, Fitzgerald T, Moutoussis M…, 2013 · The anatomy of choice: active inference and agency · open at passage 33
In active inference, there is no value function: free energy is the only quantity that is optimized.Friston K, Schwartenbeck P, Fitzgerald T, Moutoussis M…, 2013 · The anatomy of choice: active inference and agency · open at passage 7
this precision controls the degree of “exploration” of the system, because, formally, it corresponds to the temperature parameter of softmax choice rules.Pezzulo G, Rigoli F, Friston K, 2015 · Active Inference, homeostatic regulation and adaptive behavioural control · open at passage 86
Generally speaking, active inference assumes that the only self-consistent prior is that the actions undertaken by organisms minimize expected free energy.Kirchhoff M, Parr T, Palacios E, Friston K, Kiverstein J, 2018 · The Markov blankets of life: autonomy, active inference and the free energy… · open at passage 45
Salience is the information gain, reduction of uncertainty, Bayesian surprise, or epistemic value that constitutes an important part of expected free energy.Constant A, Ramstead MJD, Veissière SPL, Campbell JO…, 2018 · A variational approach to niche construction · open at passage 53
these abilities correspond to the policies and their affordance is quantified in terms of the expected free energy under each policy.Ramstead MJD, Constant A, Badcock PB, Friston KJ, 2019 · Variational ecology and the physics of sentient systems · open at passage 37

Linked ideas

Free energy principle and active inferencepart of / contains

Policy selection by expected free energy is the action side of the wider free energy principle.

Affordance and action selectionrelated to

Ramstead and colleagues quantify affordance as expected free energy under each policy.

Explore-exploit trade-off and foraging heuristicsrelated to

Pezzulo and colleagues tie precision to softmax temperature, which sets the degree of exploration.

Precision, synaptic gain and neuromodulationrelated to

Precision of beliefs about future states plays the role of sensitivity in policy choice.

Goal-directednessparallels

Goal-directed behaviour is cast here as policy selection under prior beliefs rather than reward maximisation.

Autopoiesis and the Markov blanketrelated to

Kirchhoff and colleagues frame active inference as maintaining a Markov blanket by minimising expected free energy.

Homeostatic regulation and hierarchical reflex control as active inferencerelated to

Covers how action policies are chosen once instrumental responses are placed in the model.

Predictive processing and interoceptive inferencepart of / contains

Allen & Friston place active inference under the predictive-processing umbrella; B's policy selection by free energy is its action-side extension, where Friston 2009 reduces optimal-policy learning to free-energy minimisation.

Niche constructionrelated to

Passages frame niche construction variationally, using active inference and free energy; VANC extends individual action-perception loops to organism–niche coupling. Neither passage mentions expected free energy or policy selection.

Quantum reference frames and the membrane as boundaryrelated to

Passages treat the Markov blanket as the boundary separating internal from external states; active inference has its active and internal states minimise variational free energy. Quantum reference frames and expected free energy go unmentioned.

Temporal depth of generative models and conscious versus unconscious inferencerelated to

Choosing among multiple future trajectories is the policy selection problem that marks deep models in the Friston account.

Where it is discussedPassages matching active inference, policy, expected free energy, utility of states

2015	Active Inference, homeostatic regulation and adaptive behavioural control · Pezzulo G, Rigoli F, Friston K	56
2012	Free energy, value, and attractors · Friston K, Ao P	49
2013	The anatomy of choice: active inference and agency · Friston K, Schwartenbeck P, Fitzgerald…	32
2019	Variational ecology and the physics of sentient systems · Ramstead MJD, Constant A, Badcock PB…	24
2022	Active inference, morphogenesis, and computational psychiatry · Pio-Lopez L, Kuchling F, Tung A…	24
2018	The Markov blankets of life: autonomy, active inference and the free energy… · Kirchhoff M, Parr T, Palacios E…	24
2009	Reinforcement learning or active inference? · Friston KJ, Daunizeau J, Kiebel SJ	24
2016	Probabilistic adaptation in changing microbial environments · Katz Y, Springer M	14
2022	Competency in Navigating Arbitrary Spaces as an Invariant for Analyzing… · Fields C, Levin M	14
2018	A variational approach to niche construction · Constant A, Ramstead MJD, Veissière…	13