Rationality

Evolutionary selection pressures are ontologically distinct from the biological artifacts they create.

Evolutions are slow. How slow? Suppose there’s a beneficial mutation that conveys a fitness advantage of 3%: on average, bearers of this gene have 1.03 times as many children as non-bearers. Assuming that the mutation spreads at all, how long will it take to spread through the whole population? That depends on the population size. A gene conveying

This is how I see the story of life and intelligence—as a story of improbably good designs being produced by optimization processes. The “improbability” here is improbability relative to a random selection from the design space, not improbability in an absolute sense—if you have an optimization process around, then “improbably” good designs become

AI is much harder than people instinctively imagined, exactly because you can’t just tell the ghost what to do. You have to build the ghost from scratch, and everything that seems obvious to you, the ghost will not see unless you know how to make the ghost see it. You can’t just tell the ghost to see it. You have to create that-which-sees from scra

Because natural selection started out so inefficient (as a completely accidental process), this tiny handful of meta-level improvements feeding back in from the replicators—nowhere near as complicated as the structure of a cat—structure the evolutionary epochs of life on Earth.

Multicellular organisms can only exist because they’ve evolved powerful internal mechanisms to outlaw evolution. If the cells start evolving, they rapidly evolve to extinction: the organism dies.