This piece by Bright Simons describes an LLM as a model of human social reasoning at a given point in time. AI doesn’t really think, “it remembers how we thought together.” The limit is the system of thought (the society) it’s modelled upon:
Suppose you could travel to Egypt in 3000 BC and copy, in flawless hieroglyphics, the contents of every temple library, every architectural plan, every priestly manual, every commercial ledger. Then suppose you travelled to Mesopotamia and did the same in cuneiform. Consolidate everything you could find in the languages of that era, and then proceed to train a large language model on it. Full transformer architecture, self-attention, the whole enchilada.
The result would be a system capable of a certain kind of intelligence. It could predict floods from astronomical cycles. It could draft administrative correspondence. It could generate plausible religious commentary. But it would have no capacity for what the Greeks would later call the syllogism. It would carry no trace of Roman legal abstraction, and have no conception of the empirical method that wouldn’t emerge for another four millennia.
The claim is that the intelligence of a large language model isn’t in its architecture — it’s in the pre-existing “social complexity of the civilisation whose language it digested.” An LLM can record and approximate the complexity, but it is stuck there.
Jaron Lanier argued similarly a couple of years ago:
The new programs mash up work done by human minds. What’s innovative is that the mashup process has become guided and constrained, so that the results are usable and often striking. This is a significant achievement and worth celebrating—but it can be thought of as illuminating previously hidden concordances between human creations, rather than as the invention of a new mind.