This piece by Bright Simons makes the point that an LLM is a model of human social reasoning at a given point in time. AI doesn’t really think, “it remembers how we thought together.” The limit is the system of thought it’s trying to model:
Suppose you could travel to Egypt in 3000 BC and copy, in flawless hieroglyphics, the contents of every temple library, every architectural plan, every priestly manual, every commercial ledger. Then suppose you travelled to Mesopotamia and did the same in cuneiform. Consolidate everything you could find in the languages of that era, and then proceed to train a large language model on it. Full transformer architecture, self-attention, the whole enchilada.
The result would be a system capable of a certain kind of intelligence. It could predict floods from astronomical cycles. It could draft administrative correspondence. It could generate plausible religious commentary. But it would have no capacity for what the Greeks would later call the syllogism. It would carry no trace of Roman legal abstraction, and have no conception of the empirical method that wouldn’t emerge for another four millennia.
The claim is that the intelligence of a large language model isn’t in its architecture — it’s in the pre-existing “social complexity of the civilisation whose language it digested.” An LLM model can record and approximate the complexity, but it is stuck there.