@dwarkesh_sp I think this comes down to a version of agency. An LLM doesn't sit around and "ponders" things.
Would Newton have made his discoveries if his only mode of interaction was reacting to questions from the people around him?
A good system is designed to be periodically cleared of cruft. It has a built-in counterbalance. Without this pressure, our bias drives us to add band-aid after band-aid, until the only choice is to destroy the whole system and start from scratch.