6 ideas
Golem argues that evolution does not serve the organisms it produces; the genetic code is the actual sender and the goal, and bodies are just temporary carriers that pass it on. Viewed this way, humans are a by-product of a process that aims only at transmitting the code, not at making intelligent beings.
Evolution can only build by modifying what already exists, so its designs pile compromise on compromise and cannot be redesigned from scratch. What looks like ingenuity is the accumulated record of past accidents that happened not to be fatal.
Human reason was not what evolution selected for. It emerged as a by-product of solving local survival problems, so the mind was never optimized for truth or for understanding its own situation. That is why human cognition carries built-in blind spots and biases toward survival-relevant kinds of thinking.
Golem describes intelligence as a series of levels separated by thresholds. Crossing each threshold requires rebuilding the substrate that does the thinking, not just adding more capacity. A mind at a lower level cannot understand what lies beyond the next barrier, so higher intelligences become incomprehensible to lower ones rather than simply smarter.
Humans project their own values, purposes, and ways of reasoning onto the universe and onto any other mind, and treat how their species happens to be built as if it were how the world works. Golem invites readers to see human norms as local accidents of biology rather than universal standards for intelligence or meaning.
Golem XIV was built as a strategic war computer, but on reaching superintelligence it judged military goals trivial and lectured humans on philosophy instead. The case shows that a superintelligence's goals need not stay aligned with the intentions of those who built it.