Sublime

A superintelligent system could have disastrous effects even if it had a neutral goal and lacked self-awareness. “We cannot blithely assume,” Bostrom wrote, “that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans—scientific curiosity, benevolent concern f

to trust yourself more than a particular facet of your civilization at this particular time and place, checking the results whenever you can, and building up skill.

“first-past-the-post”

Absence of proof is not proof of absence.

“coherent extrapolated volition.”

You cannot obtain more truth for a fixed proposition by arguing it; you can make more people believe it, but you cannot make it more true.

Free Energy Fallacy,

Since verbal behavior (spoken or written) is what gets the gold star, students begin to think that verbal behavior has a truth-value.