rate of such a code, MacKay and McCulloch realised they could do better. Thinking instead of the time between spikes as the code allowed a neuron to transmit much more information.

The technique went under the name ‘dynamic programming’,

If the vestibular system says the body is now moving leftwards, for example, the bump gets pushed to the left. In this way, movement along the ring doesn’t create errors in memory, but rather updates the memory based on new information. ‘Working’ memory earns its name.

Whereas a descriptive equation is like a cartoon animation of an event – capturing its appearance but without any depth – a model is a re-enactment.

what temporal difference learning helps you learn is a value function.

The work of Bush, Mosteller, Resorla, Wagner and Sutton all turned stacks of data from conditioning experiments into strings of symbols that could describe the algorithm needed to do the learning part of reinforcement learning.

There is no central planner that can look down and say: ‘It looks like signals would flow better if that neuron over there was connected to this one here.’6 That is why the brain needs to overproduce and prune.

To calculate the information in a symbol you must therefore ask: ‘To what power must I raise two in order to get the inverse of the symbol’s probability?’

The presence of the buzzer blocks the ability of the light to be associated with the food.