Form follows function? The inductive prejudice and the stories that help us see it.
On how a phrase coined by an architect in 1896 describes, backwards, what happens today in computing systems trained on data.
A note on vocabulary. In this text I do not use the phrase "artificial intelligence" or its abbreviation (AI), and I avoid expressions that attribute states of mind or intentions to a machine: "thinks", "understands", "knows", "wants", "sees", "chooses", "advises", "keeps company". I speak of computing systems, statistical models and conversational interfaces, and I describe what they produce, calculate or reproduce, not what they "feel" or "understand". I keep only the established technical names ("machine learning", "neural network", "training") and the titles of the works cited, so as not to falsify sources. The characters in fairy tales think and feel; analogies with them concern only the structure of a mechanism, not a mind of the machine.
A workshop, a phrase and a reversal
My father repaired engines. His workshop smelled of oil, had a lamp hanging above the workbench, a radio always on, and tools laid out in an order that, to me as a child, seemed incomprehensible. It was not an order by labels or alphabet; it was his. And yet, when he needed a spanner, his hand knew where it was, without looking.
If I asked him why he kept them that way, I got no answer, but a phrase he said often, in front of anything broken: "It can still be mended." The workshop's form had not been designed to a plan. It had settled, over years, after his hands had done the same things thousands of times.
I often think of that workshop when I read about computing systems trained on data, because it poses, on a small scale, a much larger question: who follows whom — does form follow function, or function form?
The formula that put the question into circulation belongs to the American architect Louis Sullivan. In 1896, in an essay about the tall office building, he wrote that form ever follows function.
The formula became the slogan of modernism: first you know what a thing is for, then you give it the appearance that suits it. A chair has four legs because it must stand, not because four legs look nice.
In systems trained on data, this relationship turns upside down. And that is precisely why a whole series is worth writing about it.
Let me explain why. When you build a system by training, you cannot know in advance, in the strong sense of the word, what function it will end up calculating. You know what function you want to obtain: to classify pictures of cats, to produce a translation, to estimate the weather. But the function it actually calculates — the one it reproduces on data encountered only after training — depends on a series of choices of form: the network's architecture, the training procedure, the data used, the constraints imposed. Engineers have a precise name for this set of choices. They call it inductive bias, that is, inductive prejudice.
The word "function" has, in addition, two senses that meet here. One is mathematical: a machine-learning model is, in essence, a function mapping inputs to outputs, and the training procedure selects, from among countless possible functions, a suitable one. The other is social: what do people, in fact, do with this technology? The two senses do not overlap, but they influence one another, and much of this text tries to hold them in the same frame.
So that the discussion does not remain abstract, I did something I often do in my texts: I called upon the Romanian fairy tale for help. Not as decoration, but as a mnemonic. A reader of seventeen or twenty has almost certainly met, at school, Harap-Alb (the hero of Ion Creangă's classic Moldavian fairy tale), the two daughters in Creangă's tale, or Făt-Frumos — the Romanian "Prince Charming" — who sets out after "youth without old age". These images are already in their head. They can be used as hooks on which to hang ideas that would otherwise slip away.
One warning, though, so as to be fair to the fairy tale. The analogies in this text are interpretations, not technical equivalences. Harap-Alb is not a neural network and was never thought of as one. Where the analogy holds, I use it; where it breaks, I say it breaks. One further point of precision: Harap-Alb and the tale of the daughters are literary fairy tales, written by Ion Creangă on popular motifs, while Master Manole is a ballad and legend, not a fairy tale proper. I gather them under the same roof anyway, because they work as shared images of a culture.
The text follows eight steps. It begins with what the inductive prejudice concretely is and where the idea comes from. It continues with the way modern architectures build in hypotheses about the world. Then it passes through the field's most important dispute, Rich Sutton's "bitter lesson" and the replies to it. It stops at what we know, publicly and at this moment, about what functions these systems in fact fulfil in current use. And it closes with the limits and the costs of form.