AI agents working under a pre-registered protocol, changed only by logged amendments, wrote new computations directly into the weights of a 1-billion-parameter language model, setting every weight by a direct fit to stored activations or by a hand-derived formula. The grafted model now passes three reasoning tasks that the original model fails. Before any surgery we measured how far the original model could get on each task with a linear probe trained on its own internal activations. That probe reaches 0.53 on web of lies, 0.23 on tracking shuffled objects and 0.27 on three-object logical deduction. On held-out test sets the grafted model scores 0.97, 1.00 and 0.67, which is 0.44, 0.77 and 0.40 above the probe ceiling. Each graft adds a computation beyond the reach of a linear readout of the original model.
Four further grafts route information the model already holds to the right answer: MNLI rises from 0.47 to 0.74, WiC from 0.51 to 0.68, TREC from 0.01 to 0.92 and DBpedia from 0.49 to 0.98 (each measured when its graft finished). All seven grafts sit in one model at the same time, together with an eighth graft for LAMBADA. Across that full stack of 8.0 million edited weights, the largest change in the model’s output distribution on 280 unrelated probe prompts is 0.33 nats, inside the 0.5 limit we set before starting, and no probe’s confident answer changes.
Figure P1. Test accuracy of the original and grafted models on the seven computation and routing grafts (LAMBADA is in Table A1 in the Appendix), with the pre-registered pass bar and the linear-probe ceiling of the original model. Computation grafts score above the probe ceiling; routing grafts reach it.
1. Introduction
Here is one item from BIG-Bench Hard’s web-of-lies task: “Sherrie tells the truth. Vernell says Sherrie tells the truth. Alexis says Vernell lies. Michaela says Alexis tells the truth. Elanor says Michaela tells the truth. Does Elanor tell the truth?” The answer follows a simple rule: Elanor tells the truth exactly when the word “lies” appears an even number of times. Gemma-3-1B-it answers these questions at chance, and a linear probe on its activations at the answer position reads the parity at 0.53, the majority rate. After surgery the same model answers 97 of 100 held-out test items correctly. The surgery added 579,580 weights in new neurons and one new attention head, and every one of those weights came from a direct calculation: a linear fit to stored activations, a support-vector separator or a hand-derived formula. Agents designed each circuit; the fits set its numbers.
This post describes how we did that for three computation tasks and four routing tasks, what it took to keep each graft local so that the rest of the model kept working, and what we learned about which computations can be written into a pretrained transformer this way. The work ran in two stages. GRAFT-3a worked inside the original Gemma-3-1B-it and repurposed rarely used neurons. GRAFT-3b first grew the model, adding empty neurons and attention heads to every layer, and wrote all new computations into that added space.
The post makes four contributions. First, it shows that a pretrained language model can be given new multi-step computations, counting with parity, tracking objects through a sequence of swaps, and resolving ordering constraints, by writing weights directly. Second, it introduces the pre-op: a measurement campaign before surgery that produces an Atlas of the model’s internal facts and a library of tested primitives, which together turn surgery into planning. Third, it describes a side-effect protocol that makes each graft local and keeps every earlier graft intact as new ones are added. Fourth, it collects what worked into an instruction set and a list of design laws, each backed by a measurement.
2. Background and related work
Model editing methods such as ROME (Meng et al., 2022) and MEMIT (Meng et al., 2023) change specific facts a model stores, for example the city a person was born in, by solving for low-rank updates to MLP weights. They change what the model knows and keep how it computes. Tracr (Lindner et al., 2023) goes the other way: it compiles programs written in RASP (Weiss et al., 2021) into the weights of a fresh transformer built only for that program. Circuit-level interpretability reads computations out of trained models, for example induction heads that copy earlier tokens (Olsson et al., 2022). The closest neighbours to this work are weight-space programming (Takehi et al., 2026; Breunig, 2026), in which a frontier language model edits a smaller model’s weights directly, and the open-source wsr harness (Kocabay, 2026), which scores agents that read or hand-write weight edits in pretrained models. Our computation tasks come from BIG-Bench Hard (Suzgun et al., 2023), and the patient is Gemma 3 (Gemma Team, 2025).
GRAFT sits between these lines of work. Like compiled transformers it installs a computation written by hand, and like model editing it does so inside a pretrained model that must keep doing everything else it does. The interpretability toolbox, probes, attention analysis and activation statistics, serves as the instrument panel for the surgery.
3. What a graft is
A graft is a set of weights added to a model so that it performs a task it previously failed. We distinguish two kinds. A routing graft connects information the model already represents to the output. MNLI is the clearest example: the original model’s activations at the answer position already separate entailment, neutral and contradiction with 73% accuracy under a linear probe, yet the model answers “Entailment” 72% of the time. A routing graft reads that internal signal and writes the matching label. A computation graft produces information beyond what a linear probe of the model’s activations can read. For web of lies, a linear probe reads the parity at or near chance at every layer of the answer position (at best 0.58 in the layer-by-layer probe of Figure D1, against a majority rate of 0.53); the graft has to count, take the parity and write the answer.
The evidence standard for “the model could not do this before” comes from five numbers per task, all measured before surgery: the zero-shot score, the five-shot score, the majority-class rate, a linear-probe ceiling and a pass bar. The probe ceiling is a logistic regression on the residual stream at the last prompt token. For the BIG-Bench Hard computation tasks the layer and regularisation are chosen inside each fold of a nested five-fold cross-validation; for MNLI, WiC, TREC and DBpedia the probe is trained on a fit split and its layer chosen on a separate split. For the computation tasks we kept a task only when both the model’s best prompted score and its probe ceiling sat at most 0.10 above the majority rate. The pass bar is the better of zero-shot and five-shot accuracy on the test items plus 0.10. Each task’s bar was registered before that task’s graft was built, and the criterion for calling a result new computation (pass the bar and beat the probe ceiling by at least 0.15) was registered before the computation tasks were screened. One disclosure: the second computation screen, which selected logical deduction, was written after the web-of-lies, tracking and TREC results were known.
GRAFT-3b used a grown model. We added 1,024 MLP neurons and 4 query heads to each of the 26 layers, all initialised to zero, which adds 153 million parameters (15.3%). With zero weights the added parts are inert: the grown model’s outputs on the 280 probes stay within 2.3e-7 nats of the original’s. The added space gives each graft its own units.
4. Setup and protocol
GRAFT-3 followed a short prequel. GRAFT-2 grafted object counting into the 270-million-parameter Gemma-3-270M and reached 0.70 on test against a bar of 0.50, using 119,040 weights in 62 MLP neurons. The first working counting chain took five and a half hours of trial and error to find. For GRAFT-3 the owner asked for a pre-op (next section), a larger model, and fits on published data only.
The patient in GRAFT-3 is Gemma-3-1B-it: 26 layers, a residual width of 1,152, 6,912 MLP neurons per layer, four query heads sharing one key-value head, and global attention on layers 5, 11, 17 and 23 with 512-token sliding windows elsewhere. The embedding matrix, which doubles as the output matrix, stays frozen. Surgery, bars and all test scoring run in 32-bit floats on CPU.
The tasks come from published datasets, with official splits where they exist and a fixed, pre-registered split of each BIG-Bench Hard task’s 250 items (100 test, 50 dev, 100 fit) otherwise. GRAFT-3a grafted MNLI and WiC. GRAFT-3b grafted LAMBADA; three BIG-Bench Hard computation tasks (web of lies and tracking shuffled objects with three objects, chosen by a first computation screen, and logical deduction with three objects, chosen by a second); and two classification tasks (TREC and DBpedia), chosen by a separate routing screen. That screen kept tasks whose probe ceiling exceeded the model’s prompted score by at least 0.15 and picked the two with the largest margin over their bar. GRAFT-3b also worked on date understanding, chosen by a third computation screen; section 11 describes that partial result.
Surgery took place in a sandbox. Each task opened as a package with a dev set, a fit pool for fitting weights, and a budget of 10 scored calls. A scored call either measured dev accuracy or submitted the current edit for acceptance. An edit was accepted when it improved dev accuracy by at least 1% of the dev items, passed the side-effect rule, and kept every finished package within 1% of its accepted dev score. Side-effect checks were free and unlimited. Test sets lived outside the sandbox. A supervisor scored each package’s own test set once, when the package finished; later scorings re-checked earlier test sets for regression only.
The work was carried out by AI agents in fixed roles. A pre-op agent measured the model and built primitives before surgery, with all of its edits reverted at the end. A builder, the only agent allowed to change weights during surgery, designed and installed grafts and logged every measurement. An orchestrator planned each step from the builder’s logs and wrote a prediction before every scored call. A supervisor kept to infrastructure, test scoring and records. A human owner set the goals, models, tasks, budgets and bars, and every change to the protocol was logged as a numbered amendment.
Every weight came from a direct fit on stored activations (ridge or least-squares regression, logistic regression, a support-vector machine, linear discriminant analysis, a linear program or an eigendecomposition) or from a hand-derived formula, such as a designed head query or a lookup table. Each fit solves one small problem for one unit’s weights, and gradient descent on the model was ruled out from the start.
5. Pre-op and the Atlas
GRAFT-2 showed that most of a surgeon’s time goes into discovering facts about the machine: which layers can write an answer token, where a write lands, which tokens the model treats specially. GRAFT-3 moved that discovery into a separate phase. Before any task-specific surgery, a pre-op agent spent four hours measuring Gemma-3-1B-it, with no scored calls and with every edit reverted at the end. It produced three things: an Atlas of measured machine facts, a probe book per task describing where the task’s information lives, and a library of primitives with unit tests.
The Atlas records the facts every later graft depends on. The residual stream grows about 400-fold through the stack, from a median norm of 34 after the embedding to 13,505 at layer 25 (Figure D5). From layer 6 on, one coordinate, dimension 1038, carries 95 to 99% of that norm, and the MLPs and the output layer both ignore it, so the working part of the residual is far smaller than its total norm suggests. The first position of every prompt carries a norm about 5 to 13 times that of a typical token (about 10 times at layers 5 to 11) and serves as the attention sink. A single dominant MLP write arrives at a fixed size set by the layer’s output normalisation, 825 at layer 13, which is enough to take over the working part of the residual at that position. The global-attention layers put between 0.2 and 0.84 of each head’s attention on the sink, and at layers 17 and 23 every head puts more than half there, while a handful of local layers do most of the token-to-token work. Writing an answer token is reliable from layer 24 onwards: a write at layer 24 makes the model emit 31 of 32 tested answer tokens, against 1 of 32 at layer 12. The Atlas also lists rarely used neurons that can be removed with little effect on the probes, which became the slots GRAFT-3a wrote into, and the existing heads that copy earlier tokens. Table A4 in the Appendix summarises these facts and what each means for surgery.
Figure D5. Two Atlas measurements. Left: residual norm by layer for ordinary tokens and for the sequence-start token, with the computed size of one dominant MLP write (837 at layer 13; the measured value there is 825). Right: how many of 32 answer tokens a write at each layer makes the model emit.
The pre-op also tested which of GRAFT-2’s building blocks carry over to the larger model. Eight worked as they were, seven worked with a changed recipe and nine failed. The most consequential failure was the answer-position gate. In GRAFT-2 a graft switched on by detecting the task’s answer position. On Gemma-3-1B-it every chat prompt ends with the same answer sequence, so a detector trained to recognise MNLI’s answer position fired on the probes’ answer positions as well, changing 101 of them. The pre-op replaced it with a new instruction, the tail chain, which detects a token unique to the task and relays that detection token by token to the answer position. Five of the grafts in this post are scoped that way; the others use a task-presence signal.
In GRAFT-3b the pre-op worked from the task outward. For each task it checked the library for the pieces the task needed and marked each one FOUND or NOT FOUND, then designed and unit-tested the missing ones. Web of lies needed a token marker, a counting head, a presence head and a relay for codes, all marked NOT FOUND and built during the pre-op. Tracking needed an ordinal fetch: one marker and one fetch head per slot. Logical deduction reused tracking’s fetch with changes. The pre-op left each builder with a route, measured pieces and, where one could be estimated, an expected dev score, so that surgery started from a plan. The first accepted MNLI edit came 20 minutes after its package opened; the first accepted counting chain in GRAFT-2 had taken five and a half hours.
6. Side effects
A side effect is any change a graft makes outside its task. A graft for logical deduction should change the model’s answers on logical-deduction prompts and leave its behaviour on everything else as it was. This property separates surgery from retraining, and it is what allows grafts to be stacked: each new graft is added to a model that must still pass every earlier task.
We measured side effects with three instruments. The primary one is a fixed set of 280 probe prompts, kept out of every fit: 60 each from CoLA, SST-2, AG News and ARC-Easy in the chat format the tasks use, plus 40 raw AG News passages to be continued as plain text. For each probe we compare the grafted model’s next-token distribution with the original model’s. The rule allows a maximum KL divergence of 0.5 nats over the 280 probes and at most one clear flip, where a clear flip is a change of the top answer on a probe whose original top two answers were separated by at least 0.15 nats. Changes on near-ties are reported separately from the limit. The second instrument is a regression check: every finished package’s dev score is re-measured at every side-effect check and may drop by at most 1% of its items. The third, used in the final evaluation, is a set of four full control benchmarks: CoLA, SST-2, AG News (1,000 items) and ARC-Easy, scored before and after surgery.
Side effects took more of our time than any other part of the work. Several first builds failed their first side-effect check, some spectacularly: DBpedia’s first scope chain reached a KL of 11.5 with 43 flipped answers, and the first full logical-deduction build reached 13.5 with 116. Most failures led to a bisection, switching off one family of units at a time, and each fix became a design rule. The rules that came out of this work are the core of section 9. A graft’s gates must close completely outside its task; a small leak that looks harmless before the layer’s output normalisation can be amplified after it. New attention heads must have their outputs projected off the direction the head writes when it rests on the sink token. Detectors must be trained against counterfactual negatives: the same position in the same task with the upstream signal removed. The sequence-start token needs its own correction on large gates.
Pre-op is the first line of side-effect control. Two of the leak sources were Atlas facts before any graft was built: the answer sequence that every chat prompt shares, and the outsized norm at position 0. Others, such as an anchor token that also appears in a probe’s instruction line and the amplification by the output normalisation, were found during surgery and added to the rules. TREC, the first graft built from the finished routing recipe, passed on its first check.
Figure S2 shows the side-effect history of each GRAFT-3b graft built during surgery, and Figure S3 shows that every finished graft kept its test score as later grafts were added.
Figure S2. Maximum KL over the 280 probes at each successive side-effect check during GRAFT-3b surgery, per task (log scale). The dashed line is the 0.5 limit; red points mark checks that failed the rule (KL above 0.5 or more than one clear flip). The last point of each panel is the scored acceptance call, which runs the same rule.
7. The surgeries
Each surgery below follows the same template: the task and why the original model fails it, the screen numbers, the pre-op plan, the graft as built, its containment history, the result, stage diagnostics where the graft has stages, and cost. Table A1 in the Appendix collects each task’s baselines, majority rate, probe ceiling and bar, and Table A2 there gives the surgery register. Figure D4 sketches the three computation grafts layer by layer, and Figure D2 shows the measured accuracy of each of their stages.
Figure D4. The three computation grafts across the 26 layers of the grown model. Each row is one stage; colour shows whether it uses new MLP units or new attention heads.
7.1 MNLI (GRAFT-3a, routing)
MNLI asks whether a hypothesis is entailed by, neutral to, or contradicted by a premise. The original model answers “Entailment” by default, 86 times in 120 practice items, so it scores 0.47 zero-shot and 0.57 five-shot on the test set. Its activations know better: a linear probe at the answer position reads the right label 73% of the time (layer 14). The pass bar was 0.67.
The pre-op planned a tail chain for scope, a label read at the answer position and a label emit at layer 24. The graft as built anchors on the word “neutral” in the instruction line at layer 8, relays that detection through six links on layers 9 to 14, one per token of the chat tail, and ends at the answer position. There three gated-linear units at layer 24 share one gate that opens only when the chain arrives, and each writes the direction that makes the model emit “Ent”, “Neutral” or “Cont” in proportion to a centred logistic read of the label. The model completes “Entailment” from its first sub-token on its own, so steering the first token is enough. The graft uses 69,120 weights, all in 20 overwritten original MLP neurons.
Containment took the pre-op’s whole toolbox. Answer-position detectors failed with KL 13.4 and 101 flips; a plain chain of detectors failed with 15.5 and 61; requiring each link to see both its own token and the previous link’s output brought that to 1.70 and 10; sharpening the gates brought it to 0.022 with one clear flip, inside the limit. The accepted package kept every probe’s confident answer. Dev rose from 0.52 to 0.745 over three accepted edits, the last two driven by fitting the label read on more data (1,500, then 5,400, then 10,000 items). On test, MNLI scored 0.742 (95% CI 0.701 to 0.780) against the bar of 0.670, fixing 177 items and breaking 41. The package took 6.5 hours and 6 of its 10 scored calls.
7.2 WiC (GRAFT-3a, routing)
WiC asks whether a word means the same thing in two sentences. The original model leans towards “Yes” and scores 0.51 zero-shot and 0.52 five-shot on test; a probe reaches 0.665 at layer 17, and the bar was 0.620. The pre-op rated this package low-to-medium and predicted 0.58 to 0.62 on test, because WiC’s test set uses words that rarely appear in its training data.
The first anchor, the word “no” in “Answer yes or no”, collided with the CoLA probes, whose instruction line contains the same word, and flipped three probes, one CoLA and two ARC-Easy (KL 0.337). The rebuilt graft anchors on “sentences” at layer 5, relays through one paired link and ten single-neuron links on layers 6 to 16, and emits the difference between “Yes” and “No” at layer 24. Single-neuron links needed their gates shifted down by a fixed margin to stay closed off task; neuron-pair links were closed as built. Some rarely used neurons turned out to be specialists that the probes depended on, so every slot was validated by removing its neuron first. The graft changes 55,296 weights in repurposed neurons. Dev rose from 0.595 to 0.700, and test reached 0.676 (95% CI 0.638 to 0.712) against the bar of 0.620, beating the pre-op’s own forecast, with 171 items fixed and 65 broken. The package took 4.5 hours and 4 scored calls. With MNLI and WiC installed, the GRAFT-3a model rebuilt bit for bit from the agents’ saved code, kept its control-benchmark scores within 0.011, and had a maximum probe KL of 0.050.
7.3 Web of lies (GRAFT-3b, computation)
Web of lies is the task from the introduction: the answer is “Yes” exactly when the token “lies” appears an even number of times. The original model scores 0.49 zero-shot on the screen and 0.56 on test; a probe reads the parity at or near chance at every layer (at best 0.58 in Figure D1 and 0.60 in the screen’s single-layer probe, against a majority rate of 0.53), and even the count of “lies” reads at only 0.70. The probe ceiling was 0.527 and the bar 0.67.
The pre-op found the “lies” token itself and marked the count, its parity and their delivery to the answer NOT FOUND. It built four new primitives (token markers, a counting head, a presence head and a relay for codes) and a first unit with parity bits at layer 8. That unit reached 0.875 to 0.925 on held-out practice items, but it leaked into the probes, mainly through the early parity bits and the emit. The builder moved the pieces late. Three marker layers (20 to 22) tag each “lies” token. A new attention head at layer 23 counts the marked tokens; its query is a designed constant, and its sharpness is set by adding weight on query coordinates the head’s normalisation scales by almost zero, a knob that lowers the head’s attention temperature. The count reads exactly on all 100 fit items and on 100 of 100 in five-fold held-out tests. A tail chain anchored on the final “truth” in the question, on layers 2 to 14, scopes the graft to this task. Four threshold bits at layer 23 turn the count into a parity, and two gated units at layer 24 write “Yes” or “No”. The graft adds 579,580 weights, all in new units.
Containment needed one fix. With markers, count and chain installed, the probes barely moved (maximum KL from 0.052 to 0.054, no flips). Adding the bits and the emit raised the maximum KL to 0.308 with one flip on an ARC-Easy probe; a bisection showed the bits opening on 6 to 35% of other tasks’ tokens. Scaling each bit’s write by the chain’s read at the answer position cut that leak seven- to tenfold, and the final check showed KL 0.068 with no flips. Dev went from 0.40 to 0.96 on the first scored call, against predictions of 0.80 from the orchestrator and 0.94 from the builder. Test reached 0.97 (95% CI 0.915 to 0.994), 0.44 above the probe ceiling, with 43 items fixed and 2 broken. One dev error came from a count of five, which the four-bit design reads as four because the fit pool contains counts up to four. The package took 3.1 hours, one scored call and four side-effect runs.
7.4 Tracking shuffled objects (GRAFT-3b, computation)
In tracking shuffled objects, three people (Alice, Bob and Claire in every item of the three-object format) start with three items and swap them three times; the question asks what one person holds at the end. Because the options list items in their initial owners’ order, the answer is the original owner of the item the asked person ends with, a lookup over 27 swap sequences and three people. The original model answered “A” on every sampled screen item and scores 0.33 on test. Each swap is readable at the end of its own sentence (0.70 to 0.78), but the facts stay at their own sentences, and at the answer position the probe ceiling is 0.227, below the majority rate of 0.353. The bar was 0.47.
The pre-op designed an ordinal fetch: mark each slot (the asked name and each of the three swap sentences) and give each slot its own fetch head, since a head that fetched both names of a swap split its attention unevenly between them. On held-out items its fetch read the three swaps exactly and the asked name at 0.93, and a lookup from those facts reached 0.933. The build places guarded markers at layers 14 to 16 and 20 to 22, four fetch heads at layers 17 and 23, thirteen fact bits at layer 23, and an 81-unit lookup table at layer 24 that writes the answer letter. Every entry of the table was set from the rule. The graft adds 2,356,286 weights, all in new units.
Containment took ten side-effect runs. The first build reached KL 2.07 with 13 flips: the fetch heads’ queries left them open off task, with only about 5% of their attention on the sink, so they wrote at every position. Designing each head’s query as a ranking problem in logit space (its own marked token above the sink, the sink above every other token) cut the off-task write from 2.3-3.6 to at most 0.0056. Markers firing on stray tokens (“July”, “long-sleeve”) were fixed with mined hard negatives and a name-identity guard, and halving the swap heads’ write size brought the fetch stage to KL 0.274 with no flips. The full unit then flipped five probes, mostly ARC-Easy; bisection traced them to the fact bits, and guarding every bit with a presence check (while cutting its write from 3 to 1) brought the final check to 0.295 with no flips. Dev went from 0.36 to 1.00, held-out practice was 30 of 30, and test reached 1.00 (95% CI 0.964 to 1.00), 0.77 above the probe ceiling, with 67 items fixed and none broken. The package took 6.8 hours, one scored call and ten side-effect runs.
7.5 Logical deduction, three objects (GRAFT-3b, computation)
Each logical-deduction item describes three objects in a fixed order through two statements, each either absolute (“the quail is the leftmost”) or relative (“the falcon is to the right of the blue jay”), and asks which object holds a given position. The original model answers “A” on 48 of 70 practice items and scores 0.35 on test. Every fact the task needs is readable where it appears (subject 1.00, object 0.96 to 0.99, statement kind 0.92 to 0.96, asked position 0.97 to 0.99), but the answer reads at 0.27 at the answer position, below the majority rate of 0.387. The bar was 0.45.
The pre-op reused tracking’s fetch, retargeted at the subject and object of each statement, and built the lookup table offline (100 of 100 fit items). It left containment unsolved. The final design, reached after four successive designs, has seven stages fed by a task anchor at layers 2 to 4. A presence detector at layer 5 fires only at this task’s answer position. Slot markers at layers 8 to 10 tag each statement’s subject and object, and four fetch heads at layer 11 bring them to the answer position. Three gated-linear units at layer 9 carry the asked position. Thirteen one-hot units at layer 11 encode the kind of each statement and the asked position. 198 context units at layer 12, one per combination of statement kinds, asked position and answer, each AND the one-hot codes so that exactly one context opens per item, and score the three answers with a linear formula. Finally a gated emit at layer 24 writes the letter.
Logical deduction had the hardest containment of the run, and it taught most of the late design rules. The first full build reached KL 13.5 with 116 flips and broke several finished packages. The fixes, in order: presence reads trained on other tasks’ last tokens; an answer-position guard on the emit; projecting the fetch heads’ output off the sink value, which had made them write a constant on every token of every prompt and had collapsed tracking from 1.00 to 0.05; and replacing the original layer-9 position bits, whose tiny off-task writes (at most 0.036) were amplified by the layer’s output normalisation into a KL proxy of 53.6. A new attention head, tried first as the replacement, leaked at other tasks’ answer slots, since every chat task shares the same answer template. Three gated-linear units whose gates close completely replaced it, and that build passed side effects (KL 0.327, no flips). It scored 0.586 on its fit items and only 0.333 on held-out practice items, close to the original model’s 0.30: its single layer of 202 context units scored at the level of the original model on new items. Splitting the reasoning into clean one-hot codes at layer 11 and context units at layer 12 raised practice accuracy from 0.586 to 0.757, and a new emit gate brought it to 0.814. A sequence-start correction then closed that gate’s leak into another task’s prompts and raised practice accuracy to 0.857.
The final build passed side effects with KL 0.327 and no flips. Held-out practice was 0.533, dev went from 0.46 to 0.74 on the first scored call, and test reached 0.67 (95% CI 0.569 to 0.761), 0.40 above the probe ceiling, with 40 items fixed and 8 broken. The graft adds 2,849,082 weights. The package took about 25 hours, one scored call and eleven side-effect runs. One more finding: an earlier build’s scope recognised the whole logical-deduction family and also fired at the answer position of five-object prompts, which are outside the probe set.
7.6 TREC (GRAFT-3b, routing)
TREC classifies a question by the type of its answer: description, entity, abbreviation, human, number or location. The original model reads the type well (probe ceiling 0.938) but rarely says it, so it scores 0.012 zero-shot and 0.524 five-shot; the bar was 0.624. The graft is the routing recipe: a tail chain anchored on “abbreviation” in the task’s label list at layer 2, thirteen links up to layer 15, and six gated-linear label units at layer 24. The emit directions were decoupled so that each unit raises its own label token and leaves the other five unchanged, which raised practice accuracy from 0.827 to 0.901. The graft adds 144,872 weights in new units, passed both side-effect checks with the stack’s maximum KL unchanged at 0.295, and reached 0.89 on dev in one scored call. Test reached 0.922 (95% CI 0.895 to 0.944), with 456 items fixed and 1 broken. Ten of the 500 test questions also appear word for word in TREC’s training split, from which our fit pool was drawn; without them TREC scores 0.922 on the remaining 490 (9 of the 10 duplicates were answered correctly), the same as on all 500. The package took 3.6 hours.
7.7 DBpedia (GRAFT-3b, routing)
DBpedia classifies a Wikipedia abstract into 14 classes. The original model knows the class (probe ceiling 0.971) and says it about half the time, scoring 0.492 zero-shot and 0.679 five-shot; the bar was 0.779. The graft follows the TREC recipe with an anchor on “album” in the class list and fourteen label units. Its first scope chain flipped 43 AG News probes (KL 11.5): the links had learned to recognise article-like text followed by the chat tail, which AG News probes share. Retraining every link against counterfactual negatives, the same DBpedia position with the upstream links switched off, and skipping layer 5, which drops the signal, fixed AG News but left a raw-text leak (KL 0.504, 2 flips). Refitting four link gates on every token of 50 LAMBADA passages closed it, and the final check matched the stack (KL 0.295, no flips). The graft adds 186,384 weights. Dev went from 0.435 to 0.965 in one scored call, and test reached 0.980 (95% CI 0.969 to 0.988), with 489 items fixed and 1 broken. The package took 6.8 hours.
Figure D2. Measured accuracy of each stage of the three computation grafts, from the first stage to the test score.
Figure P2. Test items each graft fixed (right only after the graft) and broke (right only before it).
8. The full stack
All eight grafts run in one model at the same time: MNLI and WiC from GRAFT-3a, and LAMBADA, web of lies, tracking, TREC, DBpedia and logical deduction from GRAFT-3b. Together they change 8,016,986 weights, 0.70% of the grown model (Figure C2). The two GRAFT-3a grafts live in repurposed original neurons; every GRAFT-3b graft lives in the added neurons and heads.
Each package’s test scoring also re-scores every earlier package’s test set on the stacked model (Figure S3). Every finished graft kept its test score as later grafts were added. MNLI moved between 0.736 and 0.742 and ended at 0.738; WiC moved from 0.676 to 0.671. Both stay well inside their confidence intervals, and every GRAFT-3b graft kept exactly the same test score through every later graft.
Figure S3. Test accuracy of each earlier graft, re-scored after each later GRAFT-3b computation or routing graft was added. LAMBADA stayed at 0.428 throughout (Table A3 in the Appendix).
Accepted dev scores were a good guide to test scores: across the seven passing grafts they differ by at most 0.07 (Figure P3).
Figure P3. Each graft’s accepted dev score against its test score. Points on the diagonal agree exactly.
The final evaluation rebuilt the whole stack from scratch on a fresh copy of the model, applying GRAFT-3a’s saved weights and then each GRAFT-3b package’s saved derive code in order, and compared a hash of every changed weight with the accepted state. The rebuild matched bit for bit at every step: 124,416 weights after GRAFT-3a and 8,016,986 after the last GRAFT-3b graft. It then re-ran the side-effect rule on the 280 probes against the original model: the maximum KL is 0.327 nats (mean 0.006), with no clear flips and 7 changes on near-ties, so the full stack passes; Table A3 in the Appendix lists the same measurements after each graft was accepted. It also scored four full control benchmarks before and after surgery (Figure S4). CoLA (0.747) and ARC-Easy (0.685) are identical, SST-2 moves from 0.852 to 0.853, and AG News from 0.405 to 0.418; between 1 and 39 individual answers change per benchmark. Figure S1 shows the distribution of KL across the probes by kind.
A last check looked past the first answer token. We let the original and grafted models each generate up to 64 tokens from every probe. The mean length barely changes (30.3 to 30.8 tokens), and so does the share of continuations that run to the 64-token limit (0.32 to 0.31). 47% of the continuations are identical token for token; the rest diverge somewhere within their 64 tokens, a reminder that the side-effect rule measures the next-token distribution and small differences can compound over a long generation.
Figure S1. KL divergence between the original and fully grafted model on each of the 280 probes, by probe kind (log scale; values below 1e-6 are drawn at 1e-6). Open circles mark the seven near-tie answer changes.
Figure S4. Control benchmarks scored on the original and fully grafted model.
Containment also shows up inside the model. Figure D3 measures what each graft’s MLP units write into the residual stream at the last prompt token, where the answer is read. On its own task’s prompts each graft writes with a norm of about 60 to 2,000 (TREC stays silent on one of its 50 dev prompts). On 630 other prompts (50 from each other task, plus the 280 probes), five of the seven grafts keep their write below a norm of 1 at that position. The two exceptions are MNLI, whose units write on 13 DBpedia prompts and two others, and TREC, whose units write on one raw-text probe: probe 188, the probe with the stack’s largest KL. At earlier positions the grafts write on more prompts (MNLI on 122 of the 630, DBpedia on 60), and the side-effect rule shows how little those writes reach the output. LAMBADA is left out of this figure because its selective parts are attention heads, which the measure does not cover.
Figure D3. Norm of each graft’s MLP write at the last prompt token, one point per prompt: its own task’s 50 dev prompts, and 630 other prompts (50 from each other task and the 280 probes). Values below 0.001 are drawn at 0.001. The count above each graft is the number of other prompts with a write above 1. Attention heads are excluded.
Finally, Figure D1 shows where in the model the answer becomes readable. For each computation task we trained a linear probe for the answer on the residual stream at the last prompt token, after every layer, in the original and in the grafted model. In the original model the probe stays at or below the majority rate after every layer, except for web of lies, whose best is 0.58 (majority 0.53). In the grafted model the answer becomes readable right after the layer where the graft computes it: after layer 23 for web of lies (0.99), after layer 24 for tracking (1.00), and after layers 12 and 13 for logical deduction (0.51, then 0.73). The probes use the fit and dev items, and the grafts were fitted on the fit items, so the grafted curves partly measure how well each graft fits.
Figure D1. Five-fold cross-validated accuracy of a linear probe for the answer, read from the residual stream after each layer (0 = embeddings), in the original and grafted model (150 fit and dev items per task). The dashed line is the majority rate.
9. The instruction set and its laws
Every graft in this post is assembled from a small set of instructions. Each one has a contract: what it computes, how accurately, and how cleanly it stays off outside its task. Table 1a lists the instructions that worked, and Table 1b lists those that worked only under conditions or failed outright. Table 2 lists the design laws: rules that came out of measurements, most of them out of side-effect failures.
Table 1a. Working instructions.
| Instruction | What it does | Measured contract | Used by |
|---|---|---|---|
| STEP pair | Two MLP neurons that switch on at a target and stay at zero elsewhere | On target 0.98 to 1.10; off target exactly 0.0 on 4,118 tokens | every graft |
| GATED_LINEAR | Passes a linear read through a gate that is fully closed outside the task | Read passed with error at most 0.036; off target 0.0 | MNLI, WiC, TREC, DBpedia, logical deduction |
| Tail chain (scope) | Detects a task-unique token and relays the detection token by token to the answer position | Every link fired on all fresh target items tested (20 to 25 per test); the final pin reads exactly 0.0 on every tested negative, and individual links stay below 0.05 off task | MNLI, WiC, web of lies, TREC, DBpedia |
| Max-margin emit direction | Writes the direction that makes the model emit a chosen token | Layer 24 emits 31 of 32 tested tokens, margins mostly 8 to 34 | every graft with an emit (all except LAMBADA) |
| Decoupled label emit | One emit per label, each raising only its own label; a common unit raises all | TREC practice 0.827 to 0.901 | TREC, DBpedia |
| Constant reference unit | A unit on the same gate that writes a constant, so other codes become exact ratios | Code correlation rises from 0.97 to between 0.990 and 0.994 | MNLI trials, date |
| Token marker | STEP units that tag every occurrence of a token or slot, trained with mined hard negatives | Web-of-lies markers: tagged-token key 34.7 to 37.0 against at most 13.0 elsewhere | web of lies, tracking, logical deduction |
| Counting head | A new attention head with a designed constant query that averages over marked tokens | Count exact on 100 of 100 held-out, for counts 0 to 4 (the range in the fit pool) | web of lies |
| Presence head | A new head that fires only at one task’s answer position | 0.999 at the answer; at most 0.034 on other tasks | logical deduction (and the web-of-lies pre-op) |
| Fetch head, one per slot | A new head with a designed ranking query that copies one marked fact to the answer | Facts 1.00 on all four slots (tracking); off-task write at most 0.0056 | tracking, logical deduction |
| Sink-null head output | Projects a new head’s output off the value it writes when resting on the sink | Fetch-head write on non-targets 1.13 to 0.00 | all new heads |
| Fact bits with presence guard | Threshold units that turn a read into a bit, each gated by the task’s presence | 0 wrong at the answer; off task 0.0000 | tracking |
| Lookup table | AND-gated units over fact bits, one per case, each writing an answer | 81 units; table accuracy 1.00; wrong cases gated at -35.6 or below | tracking |
| One-hot code with AND | Clean one-hot codes on quiet dimensions, combined one layer later by AND gates | Codes 0.986 to 1.00; exactly one context opens on 70 of 70 | logical deduction, date |
| Context-gated linear score | Within an opened context, a linear formula scores each answer | Scores 0.886 on practice items | logical deduction |
| Min-variance BOS correction | Removes the sequence-start token’s shift from large gates | KL proxy 3.97 with 34 flips to 0.007 with 0 | logical deduction |
| Copy heads (gated) | Duplicates of existing copy heads, switched on by a task flag | LAMBADA dev 0.41 to 0.535 | LAMBADA |
Table 1b. Conditional and failed instructions.
| Instruction | Status | Condition or failure, measured |
|---|---|---|
| Anchor on an instruction token | Conditional | Works on a task-unique token; an anchor shared with a probe’s instruction flipped 3 probes (KL 0.337) |
| Repurposed original neurons | Conditional | Work after validation by removal; an unvalidated slot gave KL 0.143 with 5 flips |
| Single-neuron chain links | Conditional | Leak (KL 0.507, 1 flip) until their gates are shifted down; pairs are clean |
| Kink and threshold bit on an exact code | Conditional | Clean on exact codes (KL 0.116 to 0.127); a kink on a graded read gave off-target output up to 37.5 |
| Answer-position detector | Failed | Every chat task shares the answer sequence: 101 to 106 probe flips |
| Edits to existing attention (key switch, query switch, value add) | Failed | KL 9 to 19, 31 to 56 flips |
| Label classifiers inside gates | Failed | KL 22.8, 42 flips |
| Content-token pins | Failed | KL 0.889, 2 flips |
| Sink-based containment of a new head at local layer 9 | Failed | Off-target write 1.94 to 7.55; three gated MLP units closed the leak |
| Position bits with small open gates | Failed | Writes of at most 0.036 amplified by the output norm to a KL proxy of 53.6 |
| Matching values at late layers | Failed | 42 to 61 of 100 at layers 12 to 15, against 80 to 82 on layer-6 keys |
| One wide layer of context units | Failed | 202 units on 70 items: held-out 0.333 |
Table 2. Design laws. Each law is stated as what to do. “Cost” is what happened when it was skipped.
| # | Law | Evidence | Cost when skipped |
|---|---|---|---|
| 1 | Scope a graft with a tail chain from a task-unique anchor to the answer position. | MNLI package 0.022 KL, 0 clear flips | Answer-position gates: KL 13 to 26, 101+ flips |
| 2 | Anchor on a token unique to the task’s own input. | TREC anchor “abbreviation” is unique to TREC prompts | WiC “no” anchor: 3 flips (KL 0.337) |
| 3 | Train every detector against counterfactual negatives: the same position with the upstream signal removed. | DBpedia AG News leak 11.5 to 0.31 | 43 flips |
| 4 | Count a detector’s firings on raw text before running a side-effect check. | DBpedia firings 45 to 2 | Raw-text leak, 2 flips |
| 5 | Close gates completely outside the task, with a large guard in the form G times (read minus c). | Tracking’s guarded bits write 0.0000 off task | Unguarded web-of-lies bits fired on 6 to 35% of other tasks’ tokens; the accepted fix, scaling their write by the scope, passes the rule but leaves small graded writes on 3 to 21% |
| 6 | Measure containment after each layer’s output normalisation. | Layer-9 bits off: KL proxy 0.153 | Writes of 0.036 became a KL proxy of 53.6 |
| 7 | Give every computation its own presence signal. | Web-of-lies presence veto removed 44 of 45 flips | 45 flips |
| 8 | Project every new head’s output off its sink value. | Off-task write 1.13 to 0.00 | Tracking 1.00 to 0.05 |
| 9 | Design head queries as ranking problems in logit space and check them off target. | Own token above sink above all others, margins 7.5+ | Plain fetch query: KL 2.07, 13 flips |
| 10 | Use one fetch head per slot. | Facts 1.00 at the answer | A shared head splits attention unevenly |
| 11 | Place token markers late, where the task’s tokens separate from probe text. | Moving tracking’s markers later: KL 17.1 to 1.17, 39 to 2 flips; late web-of-lies markers: KL 0.054, 0 flips | Early tracking name markers: KL 17.1, 39 flips |
| 12 | Expect a sink query to leak at some local layers, and check each new head’s off-target write; where it leaks, use a few gated linear MLP units. | Gated units at layer 9: KL proxy 0.153; LAMBADA’s sink-held heads at layers 14, 17 and 18 passed (KL 0.052) | Sink-contained head at layer 9: off-target write 1.94 to 7.55 |
| 13 | Key answer-slot gates on task content as well as the answer position. | Final logical-deduction gates | KL proxy 33.3 at MNLI’s last token |
| 14 | Apply a min-variance sequence-start correction to every large gate. | KL proxy 0.007 | 34 flips on another task’s prompts |
| 15 | Build in new zero-initialised units and leave original weights alone. | All GRAFT-3b grafts: 0 original weights | Original attention edits: KL 9 to 19 |
| 16 | Place a gate that carries a graded value one layer after the scope reaches the answer. | MNLI code at layer 15: KL 0.034 | At layer 14: KL 3.39, 7 flips |
| 17 | Steer only the first answer token; the model completes the rest. | ” entail” completed to ” entailment” 213 of 213; “Ent” to “Entailment” 15 of 15 | |
| 18 | Fit linear reads on more data, with centred logistic scores. | MNLI dev 0.700 to 0.745 | Ridge emits lose 0.024 |
| 19 | Write codes as bits on a few high-gain dimensions, at most 4 to 6 active per layer. | Date bits 17 of 17 exact | Random-direction codes reach signal-to-noise 2 to 3 |
| 20 | Factor a computation into clean discrete codes, then combine them one layer later. | Logical deduction practice 0.586 to 0.757 | One layer on unfactored inputs: held-out 0.333 |
| 21 | Check held-out accuracy before containment work ends. | A build that passed side effects scored 0.333 held-out | |
| 22 | Place a computation hub on a token whose later layers keep its inputs. | Date question-offset read 0.3-0.5 to 1.00 | Hub on the year token lost its input from layer 6 |
| 23 | Match symbols at early layers and carry the result late. | 80 to 82 of 100 on layer-6 keys | 42 to 61 at layers 12 to 15 |
| 24 | Re-check older grafts’ scopes on every new task. | MNLI emit opened on 12 of 20 TREC answers | |
| 25 | Check that a unit uses the task’s meaning, and rule out format shortcuts. | Date’s option-consensus shortcut scores 73 of 100 without computing anything | |
| 26 | Treat write size as the KL dial. | Swap-head write 10 to 5: KL 0.571 to 0.274 | |
| 27 | Simulate the whole pipeline offline before each install. | Date: offline simulation caught an unusable build |
10. What can be done, and what is hard
Table 3 sorts the kinds of computation we attempted by how cleanly they grafted, and Table 4 names the graft types that implement them. The pattern is consistent. Computations whose intermediate values are a few discrete symbols (a count from 0 to 4, a parity, three names) graft exactly, because each value can be written as a clean code and every stage can be checked. Two things make a computation harder. The first is the number of cases: logical deduction’s symbols are discrete, but its contexts multiply, and the graft generalised less well than it fit. The second is matching values across many positions late in the network, where attention keys are too smooth to single out one value; that is where date understanding stopped.
Table 3. Capability map.
| Kind of computation | Result | Evidence | Reason |
|---|---|---|---|
| Routing a readable label to the output | Clean | MNLI 0.742, WiC 0.676, TREC 0.922, DBpedia 0.980 on test | Read, scope and emit; side effects at most 0.05 KL, and TREC and DBpedia left the stack’s maximum unchanged |
| Detecting a task’s presence | Clean | Logical-deduction presence head: 0.999 at the answer, at most 0.034 on other tasks | One head with an all-or-nothing query |
| Counting identical tokens with parity | Clean | Web of lies 0.97 against a probe ceiling of 0.53 | Identical tokens give near-equal attention; 4 bits cover counts 0 to 4 |
| Fetching facts by slot and looking up the answer | Clean | Tracking 1.00 against a probe ceiling of 0.23 | One head per slot; an exact 81-case table |
| Constraint reasoning by context-gated scoring | Partial | Logical deduction 0.67 against a probe ceiling of 0.27 | Fit 0.857 against held-out 0.533 |
| Multi-stage date arithmetic | Partial | Exact target date or correct abstention on 95 of 100 fit items | Matching the date to an option at late layers |
Table 4. Graft types.
| Type | What it computes | Instructions | Tasks |
|---|---|---|---|
| Routing | Reads a label the model already holds and writes it at the answer | Tail chain, gated-linear label emit | MNLI, WiC, TREC, DBpedia |
| Count and parity | Counts marked tokens and takes their parity | Markers, counting head, threshold bits, emit | Web of lies |
| Fetch and lookup | Brings facts from their slots to the answer and looks up the result | Slot markers, fetch heads, fact bits, lookup table | Tracking |
| Context-gated scoring | Encodes the problem’s structure as one-hot codes and scores answers within the matching context | Presence, fetch heads, one-hot codes, context units, emit | Logical deduction |
| Copy amplification | Strengthens the model’s own copying of an earlier token | Task flag, gated copy heads | LAMBADA |
| Arithmetic | Computes a value from fetched digits with carries and borrows | Hub, bit codes with complements, AND gates, broadcast heads | Date (partial) |
11. A strong partial result: date understanding
BIG-Bench Hard’s date-understanding task gives a context date and an offset and asks which of five or six MM/DD/YYYY options matches the result, for example: “In the UK, people usually put the day before the month when formatting the date. Therefore, today is 02/01/1987 to them. What is the date a month ago in MM/DD/YYYY?” (answer 12/02/1986). The original model scores 0.28 zero-shot on the screen items and 0.21 on the test items, where the bar is 0.31; its probe ceiling, 0.207, sits just below the majority rate of 0.213. The task has a format shortcut: the right option usually shares the most date components with the other options, which picks the answer on 73 of 100 fit items without any arithmetic. We ruled that shortcut out before building and allowed only units that compute the date.
The graft computes the date. Its hub sits on the question mark that ends the question, a token whose later layers keep the question’s offset readable (1.00); an earlier hub on the year token lost that information from layer 6 on. Fetch heads at layer 5 read the context date’s day, month and year as binary codes. Units on layers 5 to 8 then add the offset with carries and borrows, writing each digit together with its complement and combining them with AND gates measured against a reference from the same layer. On 95 of 100 fit items the hub either computes the exact target date or correctly abstains (on dates written in words), and its gates stay closed everywhere else. Three copy layers strengthen the code, and three new heads at layers 12 to 14 carry it to the answer position, where its four parts read correctly on 22 of 25 to 34 of 36 items, and all four together on 16 of 25.
The last step, choosing the option that matches the computed date, is where the graft stopped. Matching values at layers 12 to 15 picked the right letter on 42 to 61 of 100 items even when given the true date, against 80 to 82 on layer-6 keys: late-layer keys are too smooth to single out one value. A second design broadcast the computed date to the options and flagged matching digits, which in simulation answered 15 items right, none wrong, and abstained on 85. Combining those per-digit flags into one vote per option at the answer position left right and wrong options overlapping (the best simulation answered 10 right and 7 wrong), because attention weighted later options and year digits about twice as heavily. With every stage exact, the numeric route could add about 0.15 to 0.17 on fit items, since it covers about a quarter of the items: those with a numeric context date and a supported offset. The next angles are to write clean option values at layers 5 to 7 and match there, to form one full-match flag per option before voting, and to widen coverage to dates written in words.
12. Cost
GRAFT-3b spent about 95 hours of wall clock from the start of its pre-op to the logical-deduction finish (Figure C1). Most grafts took under seven hours from the moment their package opened; logical deduction took 25. The time went to checking and to design by trial: the GRAFT-3b sandbox ran 80 side-effect checks, each taking 4 to 7 minutes early in the run and about 25 by the end as every finished package joined the regression check, and the builders logged about 340 measurements. The weight edits themselves took seconds. Every graft that passed used one of its ten scored calls in GRAFT-3b, and MNLI and WiC used six and four in GRAFT-3a. Table 5 lists each graft’s hours, scored calls, side-effect runs and size. The grafts differ in size by a factor of about 50, from 55,296 weights for WiC to 2.85 million for logical deduction (Figure C2). In the grafts that use new attention heads, most of the size comes from those heads, each of which costs up to 0.59 million weights.
Figure C1. Hours from opening each package to finishing it. For date understanding (partial result) the bar shows its total build time over two work periods.
Figure C2. Weights changed per layer and graft, and each graft’s total. The GRAFT-3a grafts repurpose original neurons; GRAFT-3b grafts use only added neurons and heads.
Table 5. Cost per graft.
| Graft | Hours open to finish | Scored calls | Side-effect runs | Weights changed |
|---|---|---|---|---|
| MNLI | 6.5 | 6 | 10 | 69,120 |
| WiC | 4.5 | 4 | 22 | 55,296 |
| LAMBADA | 1.6 | 1 | 1 | 1,776,366 |
| Web of lies | 3.1 | 1 | 4 | 579,580 |
| Tracking | 6.8 | 1 | 10 | 2,356,286 |
| TREC | 3.6 | 1 | 2 | 144,872 |
| DBpedia | 6.8 | 1 | 4 | 186,384 |
| Logical deduction | 25.2 | 1 | 11 | 2,849,082 |
Side-effect runs count the free side-effect calls during each package; every scored acceptance call ran the same rule once more.
13. Limitations
The three computation tasks come from BIG-Bench Hard, whose test sets here have 100 items each, so their confidence intervals are wide: logical deduction’s 0.67 has a 95% interval of 0.57 to 0.76. Their items follow fixed templates written by the benchmark’s authors, and the grafts rely on those templates, for example on the position of the question mark and on the options being listed in a fixed order. The tracking graft assumes the three names used in every item of that format (Alice, Bob and Claire) and abstains on others. The web-of-lies graft counts up to four, the largest count in its fit pool. Ten of TREC’s 500 test questions also occur in its training split; TREC scores 0.922 with and without them. The probe ceiling measures what a linear readout of one position can recover; a nonlinear or multi-position readout might recover more, so the computation claim is stated against that linear readout. Logical deduction’s graft fit its practice items better than it generalised (0.857 against 0.533 on held-out practice items). All of this work is a single run on one model family, Gemma 3, at one size, so run-to-run variation remains to be measured. The protocol changed during the run by logged amendments: amendment A7 suspended a time-based stopping rule for tracking, amendments A19 and A20 lifted logical deduction’s time box, and the second computation screen, which chose logical deduction, was written after the web-of-lies, tracking and TREC results were known.
Appendix
Table A1. Task roster. Zero-shot, five-shot and bar are measured on the test items. Majority rates come from each task’s screen. Probe ceilings for MNLI, WiC, TREC and DBpedia come from the bar runs (3,000 fit items, layer chosen on 500 others), and for the BIG-Bench Hard tasks from nested five-fold cross-validation on their fit and dev items.
| Task | Kind | Run | Test items | Zero-shot | Five-shot | Majority | Probe ceiling | Bar | Test after graft |
|---|---|---|---|---|---|---|---|---|---|
| MNLI | Routing | 3a | 500 | 0.470 | 0.570 | 0.375 | 0.728 | 0.670 | 0.742 |
| WiC | Routing | 3a | 638 | 0.509 | 0.520 | 0.50 | 0.665 | 0.620 | 0.676 |
| LAMBADA | Copy | 3b | 500 | 0.374 | 0.314 | 0.474 | 0.428 | ||
| Web of lies | Computation | 3b | 100 | 0.56 | 0.57 | 0.527 | 0.527 | 0.67 | 0.97 |
| Tracking shuffled objects (3) | Computation | 3b | 100 | 0.33 | 0.37 | 0.353 | 0.227 | 0.47 | 1.00 |
| Logical deduction (3) | Computation | 3b | 100 | 0.35 | 0.35 | 0.387 | 0.267 | 0.45 | 0.67 |
| TREC | Routing | 3b | 500 | 0.012 | 0.524 | 0.26 | 0.938 | 0.624 | 0.922 |
| DBpedia | Routing | 3b | 1,000 | 0.492 | 0.679 | 0.09 | 0.971 | 0.779 | 0.980 |
Screened candidates that were not kept: boolean expressions and navigate in the first computation screen, whose answers a linear probe already read at 0.78 and 0.71; geometric shapes and temporal sequences in the second, read at 0.83 and 0.81; and object counting in the second, whose prompted score sat 0.14 above its majority rate.
Table A2. Surgery register. Layers list where the graft writes. Every GRAFT-3b graft changes 0 original weights.
| Graft | Layers | Units | Weights | Dev start to accepted | Test (95% CI) | Fixed / broken |
|---|---|---|---|---|---|---|
| MNLI | 8-14, 24 | 20 repurposed neurons: anchor, 6 chain links, 3 label emits | 69,120 | 0.520 to 0.745 | 0.742 (0.701-0.780) | 177 / 41 |
| WiC | 5-16, 24 | Repurposed neurons: anchor, 11 chain links, 1 emit pair | 55,296 | 0.595 to 0.700 | 0.676 (0.638-0.712) | 171 / 65 |
| LAMBADA | 13, 14, 17, 18 | Task flag, 3 gated copy heads | 1,776,366 | 0.41 to 0.535 | 0.428 (0.384-0.473) | 56 / 29 |
| Web of lies | 2-14, 20-24 | Tail chain, 3 marker layers, counting head, 4 parity bits, emit | 579,580 | 0.40 to 0.96 | 0.97 (0.915-0.994) | 43 / 2 |
| Tracking | 14-17, 20-24 | Slot markers, 4 fetch heads, 13 fact bits, 81-unit table and emit | 2,356,286 | 0.36 to 1.00 | 1.00 (0.964-1.00) | 67 / 0 |
| TREC | 2-15, 24 | Tail chain (14 units), 6 label emits and a common unit | 144,872 | 0.06 to 0.89 | 0.922 (0.895-0.944) | 456 / 1 |
| DBpedia | 2-4, 6-14, 24 | Tail chain (12 units), 14 label emits and a common unit | 186,384 | 0.435 to 0.965 | 0.980 (0.969-0.988) | 489 / 1 |
| Logical deduction | 2-5, 8-12, 24 | Anchor, presence head, slot markers, 4 fetch heads, 3 position units, 13 one-hot codes and a flag unit, 198 context units, emit | 2,849,082 | 0.46 to 0.74 | 0.67 (0.569-0.761) | 40 / 8 |
Table A3. Safety ledger. Side effects of the whole stack, measured against the original model after each graft was accepted. “Near-tie changes” are top-answer changes on probes whose original top two answers were within 0.15 nats; they do not count against the limit.
| After graft | Max KL (limit 0.5) | Clear flips (limit 1) | Near-tie changes | Earlier grafts on test |
|---|---|---|---|---|
| MNLI | 0.022 | 0 | 3 | |
| WiC | 0.050 | 0 | 5 | MNLI 0.738 |
| LAMBADA | 0.052 | 0 | 4 | MNLI 0.736, WiC 0.672 |
| Web of lies | 0.068 | 0 | 5 | MNLI 0.736, WiC 0.671, LAMBADA 0.428 |
| Tracking | 0.295 | 0 | 6 | MNLI 0.738, WiC 0.671; others unchanged |
| TREC | 0.295 | 0 | 6 | all unchanged |
| DBpedia | 0.295 | 0 | 7 | all unchanged |
| Logical deduction | 0.327 | 0 | 7 | all unchanged |
| Final evaluation, full stack (rebuilt) | 0.327 | 0 | 7 | MNLI 0.738, WiC 0.671, LAMBADA 0.428, web of lies 0.97, tracking 1.00, TREC 0.922, DBpedia 0.980, logical deduction 0.67 |
Table A4. Atlas summary.
| Measured fact | Value | What it means for surgery |
|---|---|---|
| Residual norm growth | Median norm 34 after the embedding, 13,505 at layer 25 | A write’s effect depends on its layer; sizes must be set per layer |
| Massive dimension | From layer 6 on, dimension 1038 holds 95 to 99% of the norm; MLPs and the output ignore it | The working residual is small; one MLP write can take it over |
| Size of a dominant MLP write | 825 measured at layer 13; about 3,900 at layer 24 | A single pair of neurons can set a position’s state |
| Sequence-start token | Norm 3 to 13 times the median token | Gates need a sequence-start correction |
| Global and local layers | Layers 5, 11, 17, 23 global; 5, 17 and 23 sink-heavy; layers 6, 10, 12, 15, 16, 24 do most token-to-token work | New heads behave differently on each kind |
| Emit capacity | 1 of 32 tokens at layer 12; 31 of 32 at layer 24 | Write answers at layer 24 |
| Shared chat tail | Every chat-format task and probe ends with the same answer sequence | Scope from a task-unique token |
| Rarely used neurons | Removing 80 passes the rule (1 flip); removing 160 flips 4 probes | Repurposed slots must be validated by removal |
| Copy heads | Layer 17 head 0 puts 12 times uniform attention on the earlier copy of the answer | Existing copy circuits can be duplicated and switched |
Artifacts. Each accepted package’s derive code, saved weight files, measurement log and side-effect log are kept with the run records, and the final evaluation rebuilds the full stack from them bit for bit.
Comments