---
title: "The Mechanics of Context and Weights"
date: 2026-09-07
description: "It details how context shifts the activation of these weights, allowing the model to seamlessly transition between different semantic regions (e.g., moving from"
author: "Petru Cojocaru"
categories:
  - name: "AI - a tool"
    url: "https://cojocarupetru.info/ai-a-tool.md"
---

# The Mechanics of Context and Weights

**Navigating the Latent Space of 'Dimmer Switches'**

 This section demystifies the neural network architecture, specifically the Transformer model and its attention mechanisms. It uses the metaphor of "billions of dimmer switches" to explain neural weights and parameters. It details how context shifts the activation of these weights, allowing the model to seamlessly transition between different semantic regions (e.g., moving from a culinary context to a geographical one) without losing coherence.

 Belongs to the series:**The Architecture of Approximation: How Next-Word Prediction Unlocks the Structure of Human Knowledge**

 
---

 **A Room Full of Dimmer Switches**

 Picture a room. Not a small room. A room the size of a football stadium, every wall covered, floor to ceiling, with dimmer switches. Billions of them. Each one is a small, round dial, and each one governs the strength of a connection between two things. Not two physical things. Two *concepts*. Two *words*. Two *ideas*. The dial between *cook* and *turkey* is turned up high. The dial between *rain* and *clouds* is turned up high. The dial between *king* and *crown* rides near maximum. The dial between *king* and *bicycle* sits near zero, barely whispering.

 This is not a metaphor for how the model works. This is, in a meaningful sense, what the model *is*. Strip away the marketing language, the corporate presentations, the breathless commentary, and what remains is a vast collection of numerical parameters—weights—each one encoding the strength of an association between elements in a high-dimensional space. There are roughly 175 billion of them in GPT-3, 540 billion in PaLM, and over a trillion in the largest models currently in operation. Each one is a floating-point number. A decimal. A dial setting. And the entire intelligence of the system, everything it can do, everything it appears to know, is encoded in the collective configuration of those dials.

 For a moment, just sit with that. The entirety of what looks like knowledge, like reasoning, like fluency, lives in the positions of billions of tiny numerical dials. No single dial is intelligent. No single dial encodes a fact. But the pattern of all of them together—the specific, precise, hard-won configuration that emerged from trillions of training steps—contains a compressed representation of the structure of human language and, through language, of the world that language describes.

 
## What a Weight Actually Is

 Let us get slightly more precise, because precision matters here. In a neural network, a weight is a coefficient in a matrix multiplication. When the model processes a token, that token is represented as a vector—a list of numbers, typically several thousand entries long. This vector passes through a layer of the network, where it is multiplied by a matrix of weights. The result is a new vector, slightly transformed. It passes through another layer, another matrix, another transformation. And another. And another. In a large model, this happens dozens or hundreds of times before the final output is produced.

 Each entry in each matrix is a weight. Each weight determines how strongly one component of the input influences one component of the output. If the weight is large, the influence is strong. If it is small, the influence is weak. If it is zero, there is no influence at all. The dimmer switch analogy maps directly onto this: a high weight means the dial is turned up, a low weight means it is turned down.

 But here is the crucial point. The weights are not set by hand. They are not programmed. They are *learned*. During training, the model starts with random weights—dials set to arbitrary positions. It makes predictions. The predictions are wrong. The error signal propagates backwards through the network, and each weight is nudged by a tiny amount in the direction that would have made the prediction slightly less wrong. Then the model sees another example. And another. And another. Trillions of times. Each nudge is infinitesimal. But the cumulative effect, over trillions of examples and billions of parameters, is a configuration of weights that encodes an extraordinarily rich and nuanced model of linguistic relationships.

 Think of it like tuning a radio. You turn the dial slowly, listening for the signal. At first, all you hear is static. Then, gradually, a voice emerges. You fine-tune. The voice becomes clearer. You are not creating the broadcast. You are adjusting your receiver to pick it up. Training is the same. The structure of language—the relationships between words, the grammar, the semantics, the logic—is already present in the data. The model is adjusting its billions of dials to tune into that structure. The knowledge is in the signal. The weights are the receiver.

 
## Cook and Turkey: The Anatomy of an Association

 The original text gives a beautifully simple example. *Cook* and *turkey* sit beside each other constantly in the training data. Recipes. Thanksgiving articles. Cooking shows. Food blogs. "Cook the turkey for three hours." "The turkey is ready to cook." "How to cook a perfect turkey." The two words appear together so frequently, in so many contexts, that the weight connecting them—the dial between them—is turned up very high. When the model encounters *cook* in a culinary context, the activation flows strongly towards *turkey*. And vice versa.

 But notice something important. The association is not a simple, fixed link. It is conditional on context. If the text is about American football, *turkey* activates a completely different set of associations: *Thanksgiving*, *game*, *Dallas Cowboys*, *November*. The word *turkey* has at least two distinct regions in the model's representational space, and which one is activated depends entirely on the surrounding words. The dial between *turkey* and *cook* is high, but only when the context is culinary. The dial between *turkey* and *football* is also high, but only when the context is sporting. The model holds both settings simultaneously. It does not choose one meaning in advance. It keeps all possible meanings in superposition, weighted by context, until the context resolves the ambiguity.

 This is remarkably similar to how the human brain handles polysemy. When you read the word *bank*, your brain does not immediately commit to *financial institution* or *river edge*. Both meanings are activated, briefly, in parallel. Then the surrounding context—the rest of the sentence, the paragraph, the topic of conversation—suppresses the irrelevant meaning and amplifies the relevant one. You are not consciously aware of this process. It happens in milliseconds, below the threshold of awareness. But it is happening. The model does the same thing, except that its "milliseconds" are matrix multiplications and its "awareness" is a probability distribution over 50,000 tokens.

 
## Kill and Spider: Context Is Everything

 The original text offers a more provocative example. *Kill* and *spider*. In isolation, these two words have a weak association. Most people do not go around killing spiders. But in certain contexts, the association strengthens dramatically. "I walked into the bathroom and there was a massive spider on the wall. I had to kill it." Here, *kill* and *spider* are tightly bound. The context is domestic, visceral, slightly panicked. The dial between them rides high.

 But now change the context. "During the meditation retreat, the facilitator asked us to observe the spider in the corner without judgement. We were not to kill it, not to move it, simply to let it be." Here, *kill* and *spider* still co-occur, but the relationship is inverted. The text is about *not* killing. The dial between them is still active, but the polarity has shifted. The model must capture not just the co-occurrence but the *nature* of the relationship. In the bathroom, *kill* is what you do to the spider. In the retreat, *kill* is what you refrain from doing.

 This is where the dimmer switch metaphor starts to strain, and we need to reach for something richer. The weights in a transformer model are not simple, fixed associations. They are contextual. The same pair of words can have different effective weights depending on the surrounding context. This is achieved through the attention mechanism, which we will examine shortly. The key point is that the model does not have one dial for *kill–spider*. It has a mechanism for computing, on the fly, how strongly *kill* should attend to *spider* given the current context. In the bathroom, the attention is strong and direct. In the retreat, the attention is present but qualified, modulated by words like *not*, *without*, *simply*, *let it be*.

 For a student of linguistics, this will feel familiar. It is the difference between semantic meaning and pragmatic meaning. The words are the same. The syntax is similar. But the communicative intent is different, and the model must capture that difference. It does so not through explicit rules but through the statistical patterns of the training data. In the corpus, *kill* and *spider* appear together in contexts of fear, disgust, and domestic pest control. They also appear together in contexts of mindfulness, nature observation, and ethical restraint. The model encodes both patterns. The context selects between them.

 
## The Latent Space: Where Concepts Live

 Now we arrive at the concept that is, arguably, the most important and the most strange in modern AI: the latent space. The latent space is the high-dimensional mathematical arena in which the model represents meaning. Every token, every word, every phrase that the model processes is mapped to a point in this space. The space has thousands of dimensions—far more than the three we can visualise—but the principle is the same as a two-dimensional map. Words that are similar in meaning are close together in the space. Words that are different are far apart.

 The classic example, from the word2vec era of 2013, is the vector arithmetic: *king* minus *man* plus *woman* equals *queen*. The model's latent space encodes the concept of royalty as a direction, and the concept of gender as another direction. Moving along the royalty direction from *man* takes you to *king*. Moving along the gender direction from *king* takes you to *queen*. The relationships are geometric. Meaning is position.

 In modern large language models, the latent space is vastly more complex. It does not just encode simple analogies. It encodes entire scenes, narratives, arguments, and emotional registers. When you ask the model about cooking a turkey, the model's internal state moves into a region of the latent space where food-related concepts cluster: *oven*, *roast*, *stuffing*, *gravy*, *thermometer*, *golden brown*, *rest the meat*. The next token is sampled from this region. It is not chosen from the entire vocabulary of 50,000 tokens. It is chosen from the neighbourhood of the current position in the latent space. The context has navigated the model to the right part of the map.

 Now say the same words—*turkey*, *cook*, *bird*—inside a conversation about a country. About geopolitics. About NATO membership. About the lira. The model's internal state moves to a completely different region of the latent space. A region where *Ankara*, *Erdogan*, *Bosphorus*, *inflation*, *membership*, *alliance* cluster together. The same surface words activate entirely different neighbourhoods. The machinery is the same. The weights are the same. But the context has steered the model to a different part of the map, and the next token arrives from there.

 This is, in a very real sense, how meaning works in the model. Meaning is not a fixed property of a word. It is a position in a space, and that position shifts depending on context. The word *turkey* does not have one meaning. It has a cloud of potential meanings, and the context collapses the cloud into a point. The model does not look up the definition of *turkey*. It computes the position of *turkey* in the current context, and that position determines what comes next.

 
## The Attention Mechanism: How the Dials Talk to Each Other

 The architecture that makes all of this possible is the Transformer, introduced in a 2017 paper by Vaswani and colleagues at Google. The key innovation of the Transformer is the attention mechanism. Before Transformers, language models processed text sequentially, one word at a time, passing information forward like a bucket brigade. The Transformer dispensed with the bucket brigade. Instead, every token in the input attends to every other token simultaneously. Every word looks at every other word and asks: how relevant are you to me, right now, in this context?

 The answer to that question is a set of attention weights. These are not the same as the model's trained parameters, though they are computed using them. The attention weights are dynamic. They change with every input. They are the model's way of saying, "In this particular sentence, *kill* should pay a lot of attention to *spider* and *bathroom*, but very little attention to *meditation*." Or, "In this particular sentence, *turkey* should pay attention to *oven* and *gravy*, not to *NATO* and *Bosphorus*."

 The attention mechanism is what gives the model its contextual sensitivity. Without it, the model would treat every word the same way regardless of context. With it, the model can modulate the influence of every word on every other word, dynamically, in real time, based on the specific input. It is, in effect, a system for computing which dimmer switches should be turned up and which should be turned down for this particular sentence, this particular paragraph, this particular conversation.

 For a 22-year-old computer science student, the attention mechanism can be understood as a form of soft information retrieval. Each token generates three vectors: a query, a key, and a value. The query is the token's way of asking, "What am I looking for?" The key is the token's way of advertising, "This is what I contain." The value is the actual information the token carries. The attention score between two tokens is computed by taking the dot product of one token's query with the other's key. A high dot product means high relevance. The final representation of each token is a weighted sum of all the values in the sequence, weighted by the attention scores. Tokens that are highly relevant contribute a lot. Tokens that are irrelevant contribute almost nothing.

 This mechanism, repeated across dozens of layers and multiple "heads" (each head attending to different aspects of the input), gives the model an extraordinarily rich and flexible capacity for contextual processing. It can track entities across long passages. It can resolve pronouns. It can maintain the state of a narrative, the logic of an argument, the structure of a piece of code. All of this emerges from the simple operation of computing dot products between queries and keys. The machinery is simple. The emergent behaviour is not.

 
## The Same Machinery, Different Music

 One of the most striking features of the Transformer architecture is its generality. The same model, with the same weights, can write a poem, debug a Python script, translate a French legal document into English, summarise a medical paper, and generate a recipe for pad thai. The weights do not change between tasks. The architecture does not change. What changes is the input—the prompt, the context, the sequence of tokens that the model is asked to continue.

 This is analogous to a musical instrument. A piano has 88 keys. The keys do not change between pieces. The strings do not change. The hammers do not change. What changes is the sequence in which the keys are pressed. The same instrument can play Chopin and jazz, Beethoven and boogie-woogie. The machinery is fixed. The music is determined by the input.

 The large language model is the same. The weights are the instrument. The context is the score. The output is the performance. Ask the model about cooking a turkey, and it plays in the key of culinary English. Ask it about the geopolitical situation in Turkey, and it plays in the key of political analysis. The instrument is the same. The music is entirely different. And the model does not need to be retuned or reprogrammed between performances. The context does all the work. The context selects the region of the latent space. The context sets the attention patterns. The context determines which dimmer switches are turned up and which are turned down. The model simply responds.

 This is what makes the model so versatile, and also what makes it so difficult to control. The model does not have a "mode" for cooking and a "mode" for politics. It has one mode: continue the text in the most probable way given the context. If the context is ambiguous, the model's output will be ambiguous. If the context is misleading, the model's output will be misleading. The model has no independent sense of what it "should" be doing. It has only the context and the weights. The weights are fixed. The context is the lever. And whoever controls the context controls the output.

 
## The Human Parallel: Weights in the Brain

 It is tempting, and not entirely misleading, to draw a parallel between the model's weights and the synapses in the human brain. A synapse is a junction between two neurons. The strength of a synapse—the degree to which the firing of one neuron influences the firing of the next—is modifiable. This modifiability is the basis of learning and memory. When you learn a new word, a new fact, a new skill, what changes is the strength of specific synaptic connections. The pattern of synaptic strengths across the brain encodes your knowledge, your habits, your personality.

 The parallel is real but limited. The human brain has roughly 100 trillion synapses. The largest language model has roughly a trillion parameters. The brain's synapses are not simple numerical weights. They are complex, dynamic, biochemical structures that change over multiple timescales—from milliseconds to years. The brain is embodied, situated, awash in neurotransmitters, shaped by hormones, modulated by emotion. The model is none of these things. It is a static function, a fixed set of weights, applied to an input to produce an output. It does not learn in real time. It does not forget. It does not get tired or hungry or anxious.

 And yet, at the level of abstraction that matters for information processing, the parallel holds. Both systems encode knowledge in the strengths of connections. Both systems use context to modulate which connections are active. Both systems produce behaviour that is sensitive to the statistical structure of the environment. The brain learned the statistical structure of the physical and social world. The model learned the statistical structure of text. The mechanisms are different. The principle is the same. Narrow pressure builds machinery that encodes the structure of the environment. And the machinery, once built, can be deployed in contexts the pressure never anticipated.

 
## The Limits of the Map

 The latent space is powerful, but it is not infinite. It is bounded by the training data. The model can only represent concepts, relationships, and patterns that are present in the corpus it was trained on. If a concept is absent from the corpus, it is absent from the latent space. If a relationship is underrepresented, it will be weakly encoded. If a pattern is rare, the model will struggle to reproduce it.

 This has practical consequences. The training data for most large language models is heavily skewed towards English, towards Western perspectives, towards text that is available on the internet. Languages with fewer digital resources are underrepresented. Dialects, slang, and non-standard forms are underrepresented. Knowledge that is primarily oral, or primarily held in non-digitised archives, is underrepresented. The latent space is a map, and like all maps, it reflects the biases of the cartographer. The territories that were not surveyed do not appear on the map. The rivers that were not recorded do not flow.

 For a generation that is increasingly reliant on these models for information, for drafting, for coding, for learning, this is not an abstract concern. It is a practical one. The model's knowledge is not universal. It is not neutral. It is a statistical reflection of a specific, bounded, historically contingent corpus of text. The weights encode the patterns of that corpus, including its gaps, its silences, its distortions. The dimmer switches are set by the data. And the data was written by people, with all the limitations, biases, and blind spots that entails.

 
## The Room, Revisited

 Let us return to the room of dimmer switches. You are standing in the middle of it. Billions of dials surround you, each one humming faintly, each one set to a specific position. The room is vast, incomprehensibly vast. You cannot see the far walls. You cannot comprehend the full configuration. But you can reach out and touch the nearest dial. You can turn it, slightly, and observe what happens. You can see how the change propagates, how it affects the output, how it shifts the model's behaviour.

 This is, in essence, what researchers do when they probe the internals of a large language model. They reach into the room. They adjust a dial. They observe the effect. They try to map the territory. And what they find, over and over, is that the territory is stranger and more complex than they expected. The dials are not independent. They are entangled. Turning one affects thousands of others. The configuration is not designed. It is evolved. It is the product of trillions of small, undirected adjustments, each one nudging the system towards slightly better prediction. The result is a landscape of meaning that no human designed, no human fully understands, and no human can fully control.

 The room is the model. The dials are the weights. The configuration is the knowledge. And the context—the words you type, the question you ask, the conversation you are having—is the hand that reaches into the room and selects which dials matter, which associations to activate, which region of the latent space to navigate. The model does not choose. The context chooses. The model simply responds, faithfully, mechanically, to the configuration of its dials and the pattern of its input.

 You are reading this sentence, and somewhere in a data centre, a room full of dimmer switches is humming. The switches are set. The context is flowing. The next token is being computed. And the machinery, blind and patient and vast, does what it has always done. It guesses. And in the guessing, it maps the world.

 
---

 
## Bibliography

 
1. **Bahdanau, D., Cho, K. and Bengio, Y. (2015) 'Neural Machine Translation by Jointly Learning to Align and Translate', in*Proceedings of the 3rd International Conference on Learning Representations (ICLR)*. San Diego. Available at: [https://arxiv.org/abs/1409.0473](https://arxiv.org/abs/1409.0473) (Accessed: 8 September 2026).
2. **Bengio, Y., Ducharme, R., Vincent, P. and Jauvin, C. (2003) 'A Neural Probabilistic Language Model',*Journal of Machine Learning Research*, 3, pp. 1137–1155.
3. **Elmo (Peters, M. E.), Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K. and Zettlemoyer, L. (2018) 'Deep Contextualized Word Representations', in*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*. New Orleans: Association for Computational Linguistics, pp. 2227–2237.
4. **Hinton, G. E. and Salakhutdinov, R. R. (2006) 'Reducing the Dimensionality of Data with Neural Networks',*Science*, 313(5786), pp. 504–507. doi: 10.1126/science.1127648.
5. **Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D. (2020) 'Scaling Laws for Neural Language Models',*arXiv preprint*, arXiv:2001.08361. Available at: [https://arxiv.org/abs/2001.08361](https://arxiv.org/abs/2001.08361) (Accessed: 8 September 2026).
6. **Lakoff, G. (1987)** *Women, Fire, and Dangerous Things: What Categories Reveal About the Mind*. Chicago: University of Chicago Press.
7. **LeCun, Y., Bengio, Y. and Hinton, G. (2015) 'Deep Learning',*Nature*, 521(7553), pp. 436–444. doi: 10.1038/nature14539.
8. **Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. and Dean, J. (2013) 'Distributed Representations of Words and Phrases and their Compositionality', in*Advances in Neural Information Processing Systems 26 (NIPS 2013)*. Lake Tahoe: Curran Associates, pp. 3111–3119.
9. **Nagel, T. (1974) 'What Is It Like to Be a Bat?',*The Philosophical Review*, 83(4), pp. 435–450. doi: 10.2307/2183914.
10. **Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S. (2020) 'Zoom In: An Introduction to Circuits',*Distill*. doi: 10.23915/distill.00024.
11. **Park, K., Goldstein, A., Polimeni, J. R., Pieper, C., Xu, W., Halevy, A. and Finkelstein, A. (2023) 'The Semantic Geometry of Language Models',*arXiv preprint*, arXiv:2301.04757. Available at: [https://arxiv.org/abs/2301.04757](https://arxiv.org/abs/2301.04757) (Accessed: 8 September 2026).
12. **Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986) 'Learning Representations by Back-Propagating Errors',*Nature*, 323(6088), pp. 533–536. doi: 10.1038/323533a0.
13. **Searle, J. R. (1980) 'Minds, Brains, and Programs',*Behavioral and Brain Sciences*, 3(3), pp. 417–424. doi: 10.1017/S0140525X00005756.
14. **Toneva, M., Fedus, W., Roberts, C., Dhariwal, P. and Sutskever, I. (2022) 'Predicting and Interpreting Embedding Spaces of Large Language Models',*arXiv preprint*, arXiv:2206.01122. Available at: [https://arxiv.org/abs/2206.01122](https://arxiv.org/abs/2206.01122) (Accessed: 8 September 2026).
15. **Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017) 'Attention Is All You Need', in*Advances in Neural Information Processing Systems 30 (NIPS 2017)*. Long Beach, CA: Curran Associates, pp. 5998–6008.
16. **Wittgenstein, L. (1953)** *Philosophical Investigations*. Translated by G. E. M. Anscombe. Oxford: Blackwell.
17. **Yun, C. and Li, Y. (2020) 'Are Transformers Universal Approximators of Sequence-to-Sequence Functions?', in*Proceedings of the 8th International Conference on Learning Representations (ICLR)*. Addis Ababa. Available at: [https://openreview.net/forum?id=BygQRTVtPS](https://openreview.net/forum?id=BygQRTVtPS) (Accessed: 8 September 2026).

---