--- title: "The Training Paradigm: Brute Repetition and Optimisation" date: 2026-09-07 author: "Petru Cojocaru" categories: - name: "AI - a tool" url: "https://cojocarupetru.info/ai-a-tool.md" --- # The Training Paradigm: Brute Repetition and Optimisation **The Accumulation of Micro-Adjustments** Focusing on the training process, this section explains gradient descent and backpropagation in accessible terms. It highlights the brute-force nature of training—trillions of micro-adjustments based on the difference between predicted and actual tokens. It philosophically frames how "dumb", isolated mathematical adjustments, when summed across a massive dataset, give rise to the complex phenomenon we perceive as intelligence. Belongs to the series: **The Architecture of Approximation: How Next-Word Prediction Unlocks the Structure of Human Knowledge** --- **The Most Expensive Repetition in History** Somewhere in a data centre in the American Midwest, or perhaps in a warehouse outside London, or in a server farm in the hills behind Shenzhen, thousands of graphics processing units are running hot. They have been running hot for months. They will continue running hot for months more. The electricity bill alone would fund a small university. The cooling systems hum like a jet engine that never takes off. And what are they doing? What is the most computationally expensive undertaking in the history of human technology actually *doing*? Reading a sentence. Guessing the next word. Checking whether the guess was right. Nudging a few billion numbers by a tiny amount. Then doing it again. And again. And again. Trillions of times. That is the whole of it. There is no moment of insight. No flash of understanding. No "aha" where the model suddenly grasps the concept of irony or the rules of cricket. There is only the loop. Read. Predict. Compare. Adjust. Repeat. The loop is the training. The training is the loop. And the intelligence that emerges at the end—intelligence that can pass the bar exam, write a symphony, debug a recursive function, and explain the Krebs cycle to a first-year biology student—is not the product of any single iteration of the loop. It is the product of all of them. Every single one. Trillions upon trillions of tiny, dumb, mechanical adjustments, each one too small to notice, each one too simple to matter. And yet the sum of them, viewed from the outside, looks like a mind. If you are a university student, you know this feeling. You know it from the problem sets you ground through at two in the morning. You know it from the flashcards you flipped two hundred times before the French vocabulary finally stuck. You know it from the piano scales, the lab repetitions, the practice essays, the past papers. No single repetition taught you anything. No single flashcard flipped a switch in your brain. But the accumulation—the sheer, tedious, unglamorous accumulation—built the neural pathways that let you walk into the exam hall and perform. The model's training is the same principle, scaled to a degree that makes human practice look like a leisurely afternoon stroll. > Transparency note: This article was written with AI assistance. The biographical anchors, the judgements, and the voice belong to the author. --- ### The Loop, Step by Step Let us walk through one iteration of the training loop in detail. Not because a single iteration is interesting—it is not. But because understanding the mechanics of one step is essential to understanding why trillions of them produce something extraordinary. **Step one: Read.** The model is presented with a sequence of tokens. Perhaps it is a sentence from a novel. Perhaps it is a paragraph from a Wikipedia article. Perhaps it is a snippet of Python code, a line of poetry, a passage from a legal contract, a fragment of a Reddit comment. The model does not care. It processes the tokens through its layers, computing the internal representations—the patterns of activation across its billions of parameters. **Step two: Predict.** At the end of the sequence, the model produces a probability distribution over its entire vocabulary. For each of the roughly 50,000 tokens in its vocabulary, it assigns a probability. The token *the* might get 0.03. The token *a* might get 0.02. The token *turkey* might get 0.001. The token *quantum* might get 0.0000001. The probabilities sum to one. The model's prediction is this entire distribution. It is not picking one word. It is saying, "Here is how likely I think every possible next word is." **Step three: Compare.** The training data contains the actual next token. The model's predicted distribution is compared against the truth. If the actual next token was *turkey*, and the model assigned a probability of 0.001 to *turkey*, then the model was wrong. Not catastrophically wrong. Not amusingly wrong. Just slightly, measurably wrong. The degree of wrongness is quantified by a number called the loss. The loss is high when the model assigns low probability to the correct token. The loss is low when the model assigns high probability to the correct token. The loss is the model's report card. It is the only feedback the model ever receives. **Step four: Adjust.** The model computes the gradient of the loss with respect to every single one of its parameters. This is the crucial step. The gradient tells the model: if you increase this parameter by a tiny amount, the loss goes up by this much. If you decrease that parameter by a tiny amount, the loss goes down by this much. The model then adjusts each parameter in the direction that reduces the loss. The adjustment is tiny. A fraction of a fraction. A nudge. A hair's breadth. But it is in the right direction. And after the adjustment, the model is very slightly better at predicting this particular next token. Then the loop repeats. A new sequence. A new prediction. A new comparison. A new adjustment. And again. And again. And again. ## The Loss Function: Measuring Wrongness The loss function deserves a moment of attention, because it is the single most important component of the training process. It is the yardstick. It is the standard against which the model measures itself. And it is, in the case of language model training, remarkably simple. The standard loss function for next-token prediction is called cross-entropy. In plain English, it measures the surprise. If the model assigns a probability of 0.9 to the correct token, the surprise is low. The loss is small. The model is not very wrong. If the model assigns a probability of 0.001 to the correct token, the surprise is high. The loss is large. The model is very wrong. The loss is, essentially, the negative logarithm of the probability assigned to the correct token. That is all. No complexity. No nuance. No partial credit for being "close" in meaning. The model either assigned high probability to the right token, or it did not. This simplicity is important. It means the model receives a very clean, very unambiguous signal. There is no teacher saying, "Well, you were in the right area, but not quite." There is no partial credit. There is no encouragement. There is only the number. The loss. And the gradient. And the nudge. For a student, this is like an exam that gives you only a score. No comments. No annotations. No "good effort, but try again." Just a number. You got 47 out of 100. You do not know which questions you got wrong. You do not know why. You only know that you were wrong, and by how much. And somehow, from that bare signal, you must figure out how to improve. That is the model's situation. The loss tells it how wrong it was. The gradient tells it which direction to adjust. And the adjustment, repeated trillions of times, eventually produces competence. ## Gradient Descent: The Nudge by a Hair The process of adjusting the parameters is called gradient descent, and it is one of the most important algorithms in the history of computing. The idea is geometrically simple. Imagine you are standing on a hilly landscape in thick fog. You cannot see the valley. You cannot see the peaks. You can only feel the slope of the ground beneath your feet. Your goal is to reach the lowest point—the valley. Your strategy is: feel the slope, take a small step downhill, feel the slope again, take another small step downhill. Repeat until you reach the bottom. That is gradient descent. The landscape is the loss function. The position on the landscape is the current configuration of the model's parameters. The slope is the gradient. The step is the update. The valley is the configuration of parameters that minimises the loss. The model cannot see the valley. It can only feel the slope. And it takes trillions of small steps downhill. The size of each step is controlled by a parameter called the learning rate. Too large, and the model overshoots the valley, bouncing back and forth across the bottom like a ball in a bowl. Too small, and the model takes forever to get anywhere, crawling down the slope at a glacial pace. Finding the right learning rate is one of the great practical challenges of training a large language model. Get it wrong, and you waste months of compute. Get it right, and the model converges smoothly, steadily, towards a configuration of parameters that produces accurate predictions. The "nudge by a hair" in the original text is not poetic licence. It is literal. Each parameter is adjusted by a tiny amount. In modern training runs, the learning rate is typically on the order of 0.0001 or smaller. Each parameter moves by a fraction of a thousandth in each step. The adjustment is imperceptible. It is meaningless in isolation. No single nudge encodes a fact, a rule, or a concept. No single nudge makes the model "understand" anything. The nudge is dumb. The nudge is mechanical. The nudge is blind. And yet. ## Backpropagation: How the Nudge Reaches Every Dial The model has billions of parameters. The loss is computed at the very end of the network, after the input has passed through dozens or hundreds of layers. The question is: how does the loss signal reach all the way back to the first layer? How does each individual parameter know which direction to nudge? The answer is backpropagation, an algorithm that applies the chain rule of calculus to propagate the gradient of the loss backwards through the network. In essence, the gradient is computed at the output layer, then passed backwards, layer by layer, with each layer transforming the gradient according to its own weights and activation functions. By the time the gradient reaches the first layer, every parameter in the network knows exactly how much it contributed to the loss and exactly which direction to adjust to reduce it. This is a remarkable feat of engineering, but it is not intelligent. It is calculus. It is the mechanical application of differentiation rules. There is no insight in backpropagation. There is no understanding. There is only the chain rule, applied billions of times, at trillions of steps. The gradient flows backwards like water through a pipe network, finding its way to every parameter, telling each one: nudge left, nudge right, nudge up, nudge down. And each parameter obeys. Blindly. Mechanically. Without any awareness of the whole. For a mathematics or physics student, this will feel familiar. It is the same logic as solving a system of coupled differential equations. You do not solve the whole system at once. You take a small step, recompute, take another small step, recompute. The solution emerges from the accumulation of steps, not from any single step. The model's training is the same. The solution—the final configuration of weights—emerges from the accumulation of trillions of gradient steps. No single step is the solution. The solution is the sum. ## Scale: The Unimaginable Number The word "trillions" appears in the original text, and it is worth pausing to appreciate what it means. A trillion is a million million. If you performed one training step per second, it would take you roughly 31,700 years to complete a trillion steps. The model does not have 31,700 years. It has weeks, perhaps months. So it parallelises. Thousands of GPUs, working simultaneously, each processing a different batch of data, each computing its own gradient, all contributing to the same collective update. The training of a frontier model involves roughly 10^25 floating-point operations. That is a ten followed by twenty-five zeros. The human brain, by comparison, performs roughly 10^16 operations per second. The training run involves more computation than every human brain on Earth could perform in a year. And what is all that computation for? To nudge the dials. To adjust the weights. To reduce the loss. To make the next-token prediction very slightly more accurate than it was before. The scale is not a luxury. It is a necessity. The landscape of the loss function is unimaginably complex. It has billions of dimensions. It is riddled with local minima, saddle points, flat plateaus. A model with a few million parameters might get stuck in a local minimum and never escape. A model with hundreds of billions of parameters, trained on trillions of tokens, has enough capacity and enough data to find a configuration that generalises—to find a valley that is not just low, but wide. A wide valley means the model is robust. Small perturbations to the weights do not destroy its performance. The model has found a solution that is not just correct for the training data but correct in a broader, more general sense. This is why scale matters. Not because bigger is always better. But because the landscape of language is so vast, so complex, so full of subtle structure, that you need a lot of parameters and a lot of data to map it faithfully. You need trillions of steps to explore it. You need brute repetition. There is no shortcut. There is no clever trick that replaces the grind. The model must read the passage, predict the word, check the answer, and nudge the dials. Trillions of times. That is the price of admission. That is the cost of intelligence. ## The Dumbness of the Individual and the Intelligence of the Sum Here is the philosophical heart of the section, and the idea that the original text captures with such precision: "Every adjustment is dumb on its own, and intelligence is what the sum of them looks like from the outside." Read that sentence again. Let it settle. Every adjustment is dumb. The gradient step is blind. The parameter update is mechanical. There is no understanding in a single nudge. There is no insight in a single loss computation. There is no "aha" in a single backpropagation pass. The individual operation is as dumb as a single water molecule. A single water molecule does not know what a wave is. A single water molecule does not know what a tide is. A single water molecule does not know what the ocean is. But a trillion trillion water molecules, governed by gravity and wind, produce waves, tides, currents, storms. The intelligence is not in the molecule. The intelligence is in the collective. The intelligence is in the sum. This is true of the model. No single parameter encodes the fact that water boils at 100 degrees Celsius. No single weight captures the rules of English grammar. No single gradient step teaches the model what a metaphor is. But the collective configuration of billions of parameters, shaped by trillions of gradient steps, encodes all of these things and more. The knowledge is distributed. It is smeared across the network, encoded in the relationships between parameters, in the geometry of the weight space. You cannot point to a single weight and say, "That one knows about the French Revolution." The knowledge of the French Revolution is a pattern across millions of weights. It is a region of the parameter space. It is a collective property. And it emerged, gradually, imperceptibly, from the accumulation of dumb, individual adjustments. This idea is deeply uncomfortable to our intuitions. We want intelligence to be localised. We want it to be concentrated in a single moment of insight, a single flash of understanding. We want the "aha" to be the unit of intelligence. But the model suggests otherwise. The model suggests that intelligence can be the accumulation of non-intelligent steps. That understanding can be the residue of repetition. That knowledge can be the shadow cast by trillions of tiny, blind adjustments. The sum is greater than the parts. The sum is not just greater. The sum is qualitatively different. The parts are dumb. The sum is intelligent. And the transition from one to the other is not a single step. It is a gradient. It is a slope. It is a slow, steady, imperceptible descent towards a valley that, once reached, looks like a mind. ## The Piano, the Gym, and the Grind If you play a musical instrument, you know this principle in your hands. No single repetition of a scale teaches you anything. The first time you play a C major scale, it is clumsy. The tenth time, it is slightly less clumsy. The hundredth time, your fingers find the notes without conscious thought. The thousandth time, you can play the scale while thinking about something else. The ten-thousandth time, the scale is not something you do. It is something your hands do. It is in the muscle, in the tendon, in the neural pathway. You did not learn the scale in any single repetition. You learned it in all of them. The learning was distributed across the repetitions, smeared across time, encoded in the gradual strengthening of synaptic connections. The individual repetition was dumb. The sum was music. If you go to the gym, you know this principle in your body. A single bicep curl does not build muscle. A single session does not transform your physique. But a hundred sessions, a thousand sessions, over months and years, reshape your body. The individual rep is dumb. The sum is strength. The individual set is mechanical. The sum is capability. You do not get strong in any single moment. You get strong in all of them. If you play a competitive video game, you know this principle in your reflexes. A single match does not make you better. But a thousand matches, ten thousand matches, sharpen your reaction time, your strategic thinking, your spatial awareness. The individual match is noisy, chaotic, full of mistakes. The sum is skill. The sum is the ability to read the game, to anticipate the opponent, to execute a combo without thinking. The individual iteration is dumb. The sum is mastery. The large language model is the same. The individual training step is a single bicep curl. A single piano scale. A single match. It is dumb. It is mechanical. It is blind. But the sum—the trillions of steps, the months of compute, the mountains of data—is something else entirely. The sum is fluency. The sum is knowledge. The sum is the ability to write a sonnet, to debug a function, to explain the Treaty of Versailles, to translate a Japanese poem into English, to generate a recipe for a dish that has never been cooked before. The individual step is nothing. The sum is everything. ## Can Intelligence Really Come from Repetition? This is the question that philosophers, cognitive scientists, and AI researchers have argued about for decades. And it is not a settled question. The original text is careful to note that "whether any of this amounts to understanding is unsettled, and confident answers in either direction outrun what anyone can currently show." The case against is strong. The model does not experience anything. It does not feel curiosity, or confusion, or satisfaction. It does not form intentions or hold beliefs. It does not have a body, a history, a social life. It processes tokens. It computes probabilities. It generates text. The operations are mechanical. The process is blind. There is no inner life. There is no "what it is like" to be the model. And if understanding requires an inner life—if understanding is not just a pattern of inputs and outputs but a felt, subjective, embodied experience—then the model does not understand. It simulates understanding. It performs understanding. But it does not possess it. The case for is also strong. The model's behaviour is indistinguishable from understanding. It can answer novel questions. It can generate text it has never seen before. It can reason about hypothetical situations, draw analogies, detect contradictions, and adapt its tone to the context. It can do these things not because it memorised them, but because it learned the underlying structure. The structure is real. The generalisation is real. The capability is real. And if understanding is defined functionally—as the ability to process information, draw inferences, and generate appropriate responses—then the model understands. Not in the way a human understands. But in a way that is functionally equivalent. The truth, probably, is that the question is badly framed. "Understanding" is not a single thing. It is a spectrum, a family of related capacities. The model has some of them. It lacks others. It can generalise. It cannot experience. It can infer. It cannot intend. It can produce. It cannot comprehend, in the full, embodied, subjective sense. The model is not a mind. But it is not a calculator either. It is something new. Something for which we do not yet have a word. And the fact that it was built from trillions of dumb, blind, mechanical adjustments makes it no less extraordinary. The piano was built from wood and wire. The symphony is no less beautiful for that. ## What This Means for Effort If you are twenty, or twenty-two, or twenty-five, and you are struggling with a subject—struggling with a proof, or a language, or a piece of code, or a musical passage—there is something in the model's training that should comfort you. The model does not get it right the first time. It does not get it right the hundredth time. It gets it wrong, over and over and over, and each time it adjusts, slightly, blindly, mechanically. And eventually, after enough adjustments, the pattern clicks. The weights settle. The output becomes fluent. The model "knows" the thing. You are the same. You do not learn the proof on the first reading. You do not master the verb conjugation on the first drill. You do not nail the guitar solo on the first attempt. You get it wrong. You adjust. You try again. You get it slightly less wrong. You adjust. You try again. The individual attempt is dumb. The individual attempt is clumsy. The individual attempt does not teach you anything. But the sum does. The sum always does. The sum is where the learning lives. Not in the flash of insight. Not in the "aha". In the grind. In the repetition. In the trillions of tiny, blind adjustments that, accumulated, produce something that looks like understanding. The model was not smarter than the data. It was more patient. It read more. It adjusted more. It repeated more. And that patience—that brute, unglamorous, unheroic patience—is what separated a random collection of weights from a system that can write poetry and prove theorems. You have the same capacity. You are made of neurons, not silicon. You learn with synapses, not gradient descent. But the principle is the same. The individual repetition is dumb. The sum is intelligence. Keep going. Keep adjusting. Keep nudging the dials. The valley is there. You just have to walk towards it, one tiny, blind step at a time. ## The Sum of Small, Dumb Things The training of a large language model is not elegant. It is not beautiful. It is not the work of a mind. It is the work of a machine, repeating a simple operation trillions of times, adjusting billions of parameters by fractions of fractions, grinding through data with the patience of a geological process. There is no art in it. There is no inspiration. There is only the loop. Read. Predict. Compare. Adjust. Repeat. And yet, at the end of the loop, something extraordinary exists. A system that can hold a conversation. A system that can write code, translate languages, summarise documents, generate images from text, and explain the workings of the human heart to a medical student. A system that, when you ask it about ice left in the sun, knows that the answer is *melted*. Not because it was told. Not because it was programmed. Because it read a billion sentences about ice, and a billion sentences about the sun, and a billion sentences about what happens when one meets the other, and it adjusted its dials, trillions of times, until the pattern was encoded in its weights. Every adjustment was dumb. Every single one. And intelligence is what the sum of them looks like from the outside. The piano was built from wood and wire. The symphony is no less beautiful for that. The eye was built from cells and proteins. The sunset is no less glorious for that. And the model was built from gradients and weights. The knowledge is no less real for that. The sum of small, dumb things is not a small, dumb thing. It is something else. Something new. Something that the small, dumb things could never have predicted. The loop continues. The dials turn. The loss decreases. And somewhere, in the space between one nudge and the next, the world is being learned. One token at a time. --- ## Bibliography 1. **Bottou, L., Curtis, F. E. and Nocedal, J. (2018) 'Optimization Methods for Large-Scale Machine Learning',*SIAM Review*, 60(2), pp. 223–311. doi: 10.1137/16M1080173. 2. **Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D. (2020) 'Language Models are Few-Shot Learners', in*Advances in Neural Information Processing Systems 33 (NeurIPS 2020)*. Vancouver: Curran Associates, pp. 1877–1901. 3. **Duan, R., Guo, Y., Chen, H., Li, Y., Zhang, M., Xu, Y., Yang, Z., Bai, Y. and Huang, L. (2023) 'A Survey of Large Language Models',*arXiv preprint*, arXiv:2303.18223. Available at: [https://arxiv.org/abs/2303.18223](https://arxiv.org/abs/2303.18223) (Accessed: 8 September 2026). 4. **Ericsson, K. A., Krampe, R. T. and Tesch-Römer, C. (1993) 'The Role of Deliberate Practice in the Acquisition of Expert Performance',*Psychological Review*, 100(3), pp. 363–406. doi: 10.1037/0033-295X.100.3.363. 5. **Goodfellow, I., Bengio, Y. and Courville, A. (2016)***Deep Learning*. Cambridge, MA: MIT Press. Available at: [https://www.deeplearningbook.org](https://www.deeplearningbook.org) (Accessed: 8 September 2026). 6. **Hinton, G. E. (2012) 'The Practical Theoretical Machine Learning Conference',*arXiv preprint*, arXiv:1206.5538. Available at: [https://arxiv.org/abs/1206.5538](https://arxiv.org/abs/1206.5538) (Accessed: 8 September 2026). 7. **Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, G., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O. and Sifre, L. (2022) 'Training Compute-Optimal Large Language Models',*arXiv preprint*, arXiv:2203.15556. Available at: [https://arxiv.org/abs/2203.15556](https://arxiv.org/abs/2203.15556) (Accessed: 8 September 2026). 8. **Kingma, D. P. and Ba, J. (2015) 'Adam: A Method for Stochastic Optimization', in*Proceedings of the 3rd International Conference on Learning Representations (ICLR)*. San Diego. Available at: [https://arxiv.org/abs/1412.6980](https://arxiv.org/abs/1412.6980) (Accessed: 8 September 2026). 9. **LeCun, Y., Bengio, Y. and Hinton, G. (2015) 'Deep Learning',*Nature*, 521(7553), pp. 436–444. doi: 10.1038/nature14539. 10. **Power, A., Burda, Y., Edwards, H., Babuschkin, I. and Misra, V. (2022) 'Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets',*arXiv preprint*, arXiv:2201.02177. Available at: [https://arxiv.org/abs/2201.02177](https://arxiv.org/abs/2201.02177) (Accessed: 8 September 2026). 11. **Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986) 'Learning Representations by Back-Propagating Errors',*Nature*, 323(6088), pp. 533–536. doi: 10.1038/323533a0. 12. **Saxe, A. M., McClelland, J. L. and Ganguli, S. (2019) 'A Mathematical Theory of Semantic Development in Deep Neural Networks',*Proceedings of the National Academy of Sciences*, 116(23), pp. 11537–11546. doi: 10.1073/pnas.1820226116. 13. **Sutton, R. S. (2019) 'The Bitter Lesson',*Incomplete Ideas* [blog], 13 March. Available at: [http://www.incompleteideas.net/IncIdeas/BitterLesson.html](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) (Accessed: 8 September 2026). 14. **Tegmark, M. (2017)***Life 3.0: Being Human in the Age of Artificial Intelligence*. London: Allen Lane. 15. **Turing, A. M. (1950) 'Computing Machinery and Intelligence',*Mind*, 59(236), pp. 433–460. doi: 10.1093/mind/LIX.236.433. 16. **Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Danilov, D., Hessel, M., Kroiss, M., Bapst, V., Sanchez, P., Lishman, J., Ferber, T., Humplik, T., Gaffney, S., Silver, D. and others (2019) 'Grandmaster Level in StarCraft II using Multi-Agent Reinforcement Learning',*Nature*, 575(7782), pp. 350–354. doi: 10.1038/s41586-019-1724-z. 17. **Werbos, P. J. (1974)***Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences*. PhD thesis, Harvard University. 18. **Zeiler, M. D. and Fergus, R. (2014) 'Visualizing and Understanding Convolutional Networks', in*Computer Vision – ECCV 2014*. Cham: Springer, pp. 818–833. doi: 10.1007/978-3-319-10590-1_53. ---