Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT00 — The Steering Stack Duration: 60 minutes Level: Senior Engineer and above Prerequisites: None (this is the entry point)
After completing this module, you will be able to:
The one idea the entire course is built on. Get this wrong and every subsequent module is misunderstood.
Fine-tuning steers behavior; it does not teach knowledge. The model steers — the harness bounds.
Read it twice. Every technique you will learn in this course — SFT, DPO, GRPO, abliteration, even quantization — is a steering technique. It redirects an already-capable base model. It does not pour new knowledge into the weights. The single exception is continued pretraining (Module FT12 touches it; it is de-emphasized because almost nobody reading this will ever do it at a scale that matters), and even that is better understood as adaptation than teaching.
This is the misconception the entire field is built on top of. People talk about "teaching the model" to do X. The model already knows how to do X — it saw X during pretraining on trillions of tokens. What you are doing is redirecting its probability mass so that it does X reliably, in your format, under your conditions. That is steering.
Once the thesis lands, four things that look mysterious become obvious:
Course 1's thesis: the model is ~1.6% of an agent; the harness is the other 98.4%.
Course 3 zooms into that 1.6% and asks: what can you actually change about the model, and what should you leave to the harness? The answer — the model steers, the harness bounds — is the complement, not the contradiction, of Course 1. The two courses describe the same system from opposite ends. Take them together and you have the whole picture: a model you have steered toward your intent, deployed inside a harness that bounds what it may do.
This is why the course's final module (FT23) and second capstone are explicitly about the synthesis — an uncensored (steered) model inside an eval'd (bounded) harness. That is the destination.
Five layers. Every module in the course maps to exactly one (or a cross-cutting concern).
+-------------------------------------------+
| 5. THE BOUNDARY (the harness) | <- Courses 1 & 2A live here
+-------------------------------------------+
| 4. THE EXPORT (quant + serve) | <- GGUF / vLLM / Ollama / air-gap
+-------------------------------------------+
| 3. THE STEER (fine-tuning) | <- SFT / DPO / GRPO / abliteration
+-------------------------------------------+
| 2. THE ADAPTER (LoRA / DoRA) | <- the lightweight swappable steer
+-------------------------------------------+
| 1. THE BASE (pretrained weights) | <- the "world model" + openness
+-------------------------------------------+
The single property that makes the whole field tractable:
You can swap any layer above the base without touching the one below.
That is why LoRA adapters are swappable (Layer 2 sits on top of Layer 1 and detaches). It is why you can abliterate without retraining (Layer 3 edits weights that Layer 2 would otherwise train). It is why you can quantize after training (Layer 4 compresses whatever Layer 3 produced). It is why the harness is independent of the model (Layer 5 wraps whatever Layer 4 serves). The stack is modular by design.
The pretrained weights. This is what came out of the pretraining supercomputer after trillions of tokens. It is the "world model" — the model's compressed representation of language, facts, reasoning patterns, and (in multimodal bases) images/audio. You did not make it. You almost certainly cannot make it. You choose it (Module FT03).
The defining property of Layer 1 for this course: its openness. Is it open-weights only (you get the weights, not the training data — e.g., Llama 3.x)? Or is it open-data (you get the weights and the training corpus — e.g., OpenBMB MiniCPM, Ai2 OLMo/Tulu, HuggingFace SmolLM3)? This distinction is load-bearing for sensitive domains (Module FT21, FT22) because open-data means auditable — you can prove what the model saw. Open-weights-only means you are trusting the publisher's word. (Module FT02.)
The lightweight, swappable steer. LoRA (Low-Rank Adaptation) and its successors (DoRA, rsLoRA) inject tiny trainable matrices — often under 1% of the model's parameters — that steer the base's behavior. The base weights are frozen; only the adapter trains. When you are done, the adapter merges back in (or stays separate for hot-swapping).
The defining property of Layer 2: it is cheap and swappable. This is what democratized fine-tuning. A 7B QLoRA adapter fits on a $1,500 RTX 4090. You can train a dozen adapters for different tasks and swap them at inference like loading a config file. (Modules FT08, FT09.)
The full spectrum of behavior-shaping. This is where the actual fine-tuning happens, whether via adapter (Layer 2) or full-parameter:
The defining property of Layer 3: everything here is steering, not teaching. Even GRPO, which sounds like "training a reasoner," is really steering the model to use reasoning pathways it already has.
Quantization and serving. Once you have a steered model, you must compress it to deploy it (quantize) and run it somewhere (serve). GGUF for local/Ollama, AWQ for vLLM production, MLX for Apple Silicon. (Modules FT19, FT20.)
The defining property of Layer 4: it is downstream of training. You quantize after you fine-tune. You serve after you quantize. Mistakes here cost you quality or speed, but they do not change what the model learned — only how compactly and quickly it runs.
The harness. This is not this course — it is Courses 1 and 2A. But it is the layer that makes Layer 3's uncensoring safe. A model steered to never refuse (Layer 3) is only responsible inside a harness that bounds what it may do (Layer 5). The harness provides the policy gates, audit logs, and threat model that the model cannot provide for itself. (Module FT23.)
The defining property of Layer 5: it is independent of the model and revisable without retraining. Change the policy, redeploy. No GPU required. This is why safety lives here, not in the weights.
The most important judgment in fine-tuning. Misplace it and you waste compute, overfit, or catastrophically forget.
STEERING (what fine-tuning does) | KNOWLEDGE INJECTION (what it does NOT do well)
------------------------------------------|----------------------------------------------
Format & style ("respond in JSON") | New facts the base never saw
Instruction following | A new language not in the pretraining mix
Preference alignment ("better/worse") | A new domain's deep vocabulary
Reasoning pathway activation | Genuine new reasoning ability
Tool-call formatting | New tool semantics the model can't infer
Refusal removal (abliteration) | Adding safety the base lacks
------------------------------------------|----------------------------------------------
SFT · DPO · GRPO · abliteration | Continued pretraining (CPT) — the exception
The rule of thumb:
If the base model, with a perfect prompt, could already produce the target behavior — that's steering. If it could not, no amount of fine-tuning will reliably get you there; you need a different base or continued pretraining.
Before you fine-tune, prompt the base model directly with your ideal input and a great system prompt. Three outcomes:
The cardinal error in the field is treating fine-tuning as a knowledge-injection tool. "We'll fine-tune the model on our medical records and it will learn medicine." It will not — or rather, it will appear to, by memorizing surface patterns, and then fail catastrophically on the first case that differs from the training distribution. This is why RAG (retrieval) exists: for knowledge, you retrieve, you do not train.
Fine-tuning is for behavior. If you want the model to use your medical knowledge base in a particular way — always cite, always flag uncertainty, always structure as a differential — that is a steering problem, and fine-tuning is the right tool. If you want the model to know something it doesn't, retrieve.
This is not philosophy. The "intrinsic dimension" hypothesis (Aghajanyan et al., arXiv:2007.07784) and the subsequent LoRA literature established that the useful changes during fine-tuning live in a low-rank subspace — you can capture them with adapters that are under 1% of the parameters. If fine-tuning were moving large amounts of knowledge, you would need to update large amounts of parameters. You do not. You need to update a tiny slice. That tiny slice is the steering vector.
The 2024 finding (Shuttleworth et al., arXiv:2410.21228, "An Illusion of Equivalence") sharpened this: LoRA and full-parameter fine-tuning produce structurally different weight matrices. They are not approximations of each other. LoRA finds a low-rank steering solution; full FT finds a different (higher-rank) solution. They reach similar behavior via different geometry. This is exactly what you would expect if fine-tuning is steering (low-rank works) rather than knowledge injection (where you would need full-rank updates).
Every module lives at a layer. Here is the map.
| Module | Layer | What it does |
|---|---|---|
| FT00 (this) | Mental model | The thesis and the stack |
| FT01 | Cross-cutting | VRAM math — can you afford to load it? |
| FT02 | Base | The open spectrum — can you audit it? |
| FT03 | Base | Selecting the base |
| FT04–FT07 | Substrate | Data — the steering wheel |
| FT08–FT09 | Adapter | LoRA, QLoRA, DoRA, rsLoRA |
| FT10 | Adapter+Steer | Full FT vs PEFT decision |
| FT11 | Adapter+Steer | The training loop (TRL) |
| FT12 | Steer | SFT — the baseline |
| FT13 | Steer | DPO family — preferences |
| FT14 | Steer | GRPO — reasoning |
| FT15 | Steer | CoT distillation, RFT |
| FT16–FT18 | Steer | Alignment control (uncensored) |
| FT19–FT20 | Export | Quantize and serve |
| FT21–FT22 | Boundary | HIPAA, government, air-gap |
| FT23 | Boundary | The synthesis with the harness |
Read the course top to bottom and you are climbing the stack: base (P00) → data (P01) → adapter (P02) → steer (P03–P05) → export (P06) → boundary (P07). Each pillar adds a layer. By the end, you have the whole stack and the judgment to intervene at the correct layer.
Fine-tuning on documents expecting the model to learn the content. It will memorize surface patterns and fail out-of-distribution. Use RAG for knowledge; use fine-tuning for behavior. The single most common mistake in the field.
Trying to solve a format problem with continued pretraining (overkill), or a knowledge problem with SFT (underkill), or a safety problem with model weights alone (Layer 3 when you need Layer 5). The stack exists so you can intervene at the right layer. Module FT10 and FT23 are, in part, about this judgment.
Believing you must retrain to change behavior (you can often swap a Layer 2 adapter), or that quantization requires retraining (it does not — Layer 4 is downstream), or that the harness must change when the model changes (Layer 5 is model-agnostic). The modularity is the point.
The most expensive version: a sophisticated algorithm (GRPO, DPO) on a bad dataset. The algorithm steers precisely in the direction the data points. Bad data = bad direction, no matter how good the optimizer. This is why Pillar 1 (Data) comes before Pillar 2 (PEFT) and Pillar 3 (Alignment).
| Term | Definition |
|---|---|
| The Steering Stack | The five-layer model: Base → Adapter → Steer → Export → Boundary |
| Steering | Redirecting an already-capable model's behavior (format, preference, reasoning activation, refusal removal) |
| Knowledge injection | Actually shifting the model's knowledge distribution; the domain of continued pretraining, not fine-tuning |
| Base model | Layer 1; the pretrained weights; the "world model" |
| Adapter | Layer 2; a lightweight, swappable steer (LoRA, DoRA) — under 1% of params |
| Open-data model | A base released with its training corpus (OpenBMB, OLMo, Tulu, SmolLM3) — auditable |
| Open-weights-only | A base released as weights without training data (Llama 3.x) — not auditable |
| Abliteration | Layer 3; steering away from refusal by deleting a residual-stream direction |
| The Boundary (harness) | Layer 5; what bounds what the model may do — the subject of Courses 1 and 2A |
| Intrinsic dimension | The hypothesis that useful fine-tuning changes live in a low-rank subspace — the theoretical basis for LoRA |
See 07-lab-spec.md. The "Load, Steer, Swap" lab: load MiniCPM5-1B, apply a format-steering LoRA, swap it for a different one, observe the behavior change without touching the base. The modularity of the stack, felt in under 30 minutes.
# Module FT00 — The Steering Stack
**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT00 — The Steering Stack
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: None (this is the entry point)
---
## Learning Objectives
After completing this module, you will be able to:
1. State the course thesis — *fine-tuning steers behavior, it does not teach knowledge* — and defend it with the LoRA-vs-full-FT evidence.
2. Draw the five-layer Steering Stack and explain why each layer is independently swappable.
3. Distinguish steering (SFT, DPO, GRPO, abliteration) from knowledge injection (continued pretraining) and place each technique on the correct side of the line.
4. Predict, for a given goal ("make the model speak JSON," "make it refuse less," "teach it cardiology"), which layer of the stack the intervention belongs at — and which layers it does *not* belong at.
---
# 0.1 — The Thesis
*The one idea the entire course is built on. Get this wrong and every subsequent module is misunderstood.*
## The sentence
> **Fine-tuning steers behavior; it does not teach knowledge. The model steers — the harness bounds.**
Read it twice. Every technique you will learn in this course — SFT, DPO, GRPO, abliteration, even quantization — is a *steering* technique. It redirects an already-capable base model. It does not pour new knowledge into the weights. The single exception is continued pretraining (Module FT12 touches it; it is de-emphasized because almost nobody reading this will ever do it at a scale that matters), and even that is better understood as *adaptation* than *teaching*.
This is the misconception the entire field is built on top of. People talk about "teaching the model" to do X. The model already knows how to do X — it saw X during pretraining on trillions of tokens. What you are doing is *redirecting* its probability mass so that it does X reliably, in your format, under your conditions. That is steering.
### Why this matters immediately
Once the thesis lands, four things that look mysterious become obvious:
1. **Why QLoRA works at 1.5% of parameters.** If fine-tuning were injecting knowledge, you would need to update a large fraction of the weights. It is not — you are steering, and steering is a low-rank operation. A tiny adapter is enough. (Module FT08.)
2. **Why uncensoring degrades math.** Abliteration steers *away* from refusal by deleting a direction in the residual stream. That direction is entangled with other capabilities. Steering away from one thing nudges others. (Module FT17.)
3. **Why an uncensored model is only safe inside a harness.** Steering changes what the model *does*. It does not change what it *may* do. The boundary between "does" and "may" is the harness — and that is a different course (Course 1). (Module FT23.)
4. **Why your data matters more than your algorithm.** Steering is only as good as the steering wheel — and your steering wheel is your dataset. A brilliant algorithm on bad data steers you into a wall. (Pillar 1.)
## The complement to Course 1
Course 1's thesis: *the model is ~1.6% of an agent; the harness is the other 98.4%.*
Course 3 zooms into that 1.6% and asks: **what can you actually change about the model, and what should you leave to the harness?** The answer — *the model steers, the harness bounds* — is the complement, not the contradiction, of Course 1. The two courses describe the same system from opposite ends. Take them together and you have the whole picture: a model you have steered toward your intent, deployed inside a harness that bounds what it may do.
This is why the course's final module (FT23) and second capstone are explicitly about the *synthesis* — an uncensored (steered) model inside an eval'd (bounded) harness. That is the destination.
---
# 0.2 — The Steering Stack
*Five layers. Every module in the course maps to exactly one (or a cross-cutting concern).*
## The stack
```
+-------------------------------------------+
| 5. THE BOUNDARY (the harness) | <- Courses 1 & 2A live here
+-------------------------------------------+
| 4. THE EXPORT (quant + serve) | <- GGUF / vLLM / Ollama / air-gap
+-------------------------------------------+
| 3. THE STEER (fine-tuning) | <- SFT / DPO / GRPO / abliteration
+-------------------------------------------+
| 2. THE ADAPTER (LoRA / DoRA) | <- the lightweight swappable steer
+-------------------------------------------+
| 1. THE BASE (pretrained weights) | <- the "world model" + openness
+-------------------------------------------+
```
The single property that makes the whole field tractable:
> **You can swap any layer above the base without touching the one below.**
That is why LoRA adapters are swappable (Layer 2 sits on top of Layer 1 and detaches). It is why you can abliterate without retraining (Layer 3 edits weights that Layer 2 would otherwise train). It is why you can quantize after training (Layer 4 compresses whatever Layer 3 produced). It is why the harness is independent of the model (Layer 5 wraps whatever Layer 4 serves). The stack is modular by design.
### Layer 1 — The Base
The pretrained weights. This is what came out of the pretraining supercomputer after trillions of tokens. It is the "world model" — the model's compressed representation of language, facts, reasoning patterns, and (in multimodal bases) images/audio. You did not make it. You almost certainly cannot make it. You choose it (Module FT03).
The defining property of Layer 1 for this course: **its openness.** Is it open-weights only (you get the weights, not the training data — e.g., Llama 3.x)? Or is it open-data (you get the weights *and* the training corpus — e.g., OpenBMB MiniCPM, Ai2 OLMo/Tulu, HuggingFace SmolLM3)? This distinction is load-bearing for sensitive domains (Module FT21, FT22) because open-data means *auditable* — you can prove what the model saw. Open-weights-only means you are trusting the publisher's word. (Module FT02.)
### Layer 2 — The Adapter
The lightweight, swappable steer. LoRA (Low-Rank Adaptation) and its successors (DoRA, rsLoRA) inject tiny trainable matrices — often under 1% of the model's parameters — that steer the base's behavior. The base weights are frozen; only the adapter trains. When you are done, the adapter merges back in (or stays separate for hot-swapping).
The defining property of Layer 2: **it is cheap and swappable.** This is what democratized fine-tuning. A 7B QLoRA adapter fits on a $1,500 RTX 4090. You can train a dozen adapters for different tasks and swap them at inference like loading a config file. (Modules FT08, FT09.)
### Layer 3 — The Steer
The full spectrum of behavior-shaping. This is where the actual fine-tuning happens, whether via adapter (Layer 2) or full-parameter:
- **SFT (Supervised Fine-Tuning):** steers format and instruction-following. "Respond in JSON." "Always cite your sources." (Module FT12.)
- **DPO family (Direct Preference Optimization and variants):** steers preference. "This response is better than that one." (Module FT13.)
- **GRPO and RL on verifiable rewards:** steers reasoning. "Generate solutions; the math-correct ones get rewarded." (Module FT14.)
- **Abliteration:** steers *away* from refusal. Deletes a direction in the residual stream. (Module FT17.)
- **Continued pretraining:** the one exception — actually shifts the model's knowledge distribution. De-emphasized. (Module FT12, sidebar.)
The defining property of Layer 3: **everything here is steering, not teaching.** Even GRPO, which sounds like "training a reasoner," is really steering the model to *use* reasoning pathways it already has.
### Layer 4 — The Export
Quantization and serving. Once you have a steered model, you must compress it to deploy it (quantize) and run it somewhere (serve). GGUF for local/Ollama, AWQ for vLLM production, MLX for Apple Silicon. (Modules FT19, FT20.)
The defining property of Layer 4: **it is downstream of training.** You quantize *after* you fine-tune. You serve *after* you quantize. Mistakes here cost you quality or speed, but they do not change what the model learned — only how compactly and quickly it runs.
### Layer 5 — The Boundary
The harness. This is *not* this course — it is Courses 1 and 2A. But it is the layer that makes Layer 3's uncensoring safe. A model steered to never refuse (Layer 3) is only responsible inside a harness that bounds what it may do (Layer 5). The harness provides the policy gates, audit logs, and threat model that the model cannot provide for itself. (Module FT23.)
The defining property of Layer 5: **it is independent of the model and revisable without retraining.** Change the policy, redeploy. No GPU required. This is why safety lives here, not in the weights.
---
# 0.3 — Steering vs Knowledge: Where the Line Is
*The most important judgment in fine-tuning. Misplace it and you waste compute, overfit, or catastrophically forget.*
## The line
```
STEERING (what fine-tuning does) | KNOWLEDGE INJECTION (what it does NOT do well)
------------------------------------------|----------------------------------------------
Format & style ("respond in JSON") | New facts the base never saw
Instruction following | A new language not in the pretraining mix
Preference alignment ("better/worse") | A new domain's deep vocabulary
Reasoning pathway activation | Genuine new reasoning ability
Tool-call formatting | New tool semantics the model can't infer
Refusal removal (abliteration) | Adding safety the base lacks
------------------------------------------|----------------------------------------------
SFT · DPO · GRPO · abliteration | Continued pretraining (CPT) — the exception
```
The rule of thumb:
> **If the base model, with a perfect prompt, could already produce the target behavior — that's steering. If it could not, no amount of fine-tuning will reliably get you there; you need a different base or continued pretraining.**
### The test that tells you which side you are on
Before you fine-tune, prompt the base model directly with your ideal input and a great system prompt. Three outcomes:
1. **It produces the behavior, unreliably or in the wrong format.** That's a steering problem. Fine-tune (SFT) to make it reliable. This is ~90% of real-world fine-tuning.
2. **It refuses or produces the wrong *kind* of behavior, but you can see the capability is there.** That's a preference/alignment problem. DPO or abliteration. (Modules FT13, FT17.)
3. **It has no idea what you are talking about — the domain is genuinely foreign.** That's a knowledge gap. Fine-tuning will not fix this reliably. You need a different base, or continued pretraining on a large domain corpus, or (usually the right answer) RAG. RAG is not this course, but it is often the correct answer when people reach for fine-tuning.
### Why people overestimate fine-tuning (the cardinal error)
The cardinal error in the field is treating fine-tuning as a knowledge-injection tool. "We'll fine-tune the model on our medical records and it will learn medicine." It will not — or rather, it will *appear* to, by memorizing surface patterns, and then fail catastrophically on the first case that differs from the training distribution. This is why RAG (retrieval) exists: for knowledge, you retrieve, you do not train.
Fine-tuning is for *behavior*. If you want the model to *use* your medical knowledge base in a particular way — always cite, always flag uncertainty, always structure as a differential — that is a steering problem, and fine-tuning is the right tool. If you want the model to *know* something it doesn't, retrieve.
### The LoRA-vs-full-FT evidence (why "steering not teaching" is not just a slogan)
This is not philosophy. The "intrinsic dimension" hypothesis (Aghajanyan et al., arXiv:2007.07784) and the subsequent LoRA literature established that the *useful* changes during fine-tuning live in a low-rank subspace — you can capture them with adapters that are under 1% of the parameters. If fine-tuning were moving large amounts of *knowledge*, you would need to update large amounts of parameters. You do not. You need to update a tiny slice. That tiny slice is the steering vector.
The 2024 finding (Shuttleworth et al., arXiv:2410.21228, "An Illusion of Equivalence") sharpened this: LoRA and full-parameter fine-tuning produce *structurally different* weight matrices. They are not approximations of each other. LoRA finds a low-rank steering solution; full FT finds a different (higher-rank) solution. They reach similar behavior via different geometry. This is exactly what you would expect if fine-tuning is steering (low-rank works) rather than knowledge injection (where you would need full-rank updates).
---
# 0.4 — Mapping the Course to the Stack
*Every module lives at a layer. Here is the map.*
| Module | Layer | What it does |
| --- | --- | --- |
| FT00 (this) | Mental model | The thesis and the stack |
| FT01 | Cross-cutting | VRAM math — can you afford to load it? |
| FT02 | Base | The open spectrum — can you audit it? |
| FT03 | Base | Selecting the base |
| FT04–FT07 | Substrate | Data — the steering wheel |
| FT08–FT09 | Adapter | LoRA, QLoRA, DoRA, rsLoRA |
| FT10 | Adapter+Steer | Full FT vs PEFT decision |
| FT11 | Adapter+Steer | The training loop (TRL) |
| FT12 | Steer | SFT — the baseline |
| FT13 | Steer | DPO family — preferences |
| FT14 | Steer | GRPO — reasoning |
| FT15 | Steer | CoT distillation, RFT |
| FT16–FT18 | Steer | Alignment control (uncensored) |
| FT19–FT20 | Export | Quantize and serve |
| FT21–FT22 | Boundary | HIPAA, government, air-gap |
| FT23 | Boundary | The synthesis with the harness |
Read the course top to bottom and you are climbing the stack: base (P00) → data (P01) → adapter (P02) → steer (P03–P05) → export (P06) → boundary (P07). Each pillar adds a layer. By the end, you have the whole stack and the judgment to intervene at the correct layer.
---
## Anti-Patterns
### "Teaching" via fine-tuning
Fine-tuning on documents expecting the model to *learn* the content. It will memorize surface patterns and fail out-of-distribution. Use RAG for knowledge; use fine-tuning for behavior. The single most common mistake in the field.
### Intervening at the wrong layer
Trying to solve a format problem with continued pretraining (overkill), or a knowledge problem with SFT (underkill), or a safety problem with model weights alone (Layer 3 when you need Layer 5). The stack exists so you can intervene at the right layer. Module FT10 and FT23 are, in part, about this judgment.
### Treating the layers as coupled
Believing you must retrain to change behavior (you can often swap a Layer 2 adapter), or that quantization requires retraining (it does not — Layer 4 is downstream), or that the harness must change when the model changes (Layer 5 is model-agnostic). The modularity is the point.
### Steering without a steering wheel
The most expensive version: a sophisticated algorithm (GRPO, DPO) on a bad dataset. The algorithm steers precisely in the direction the data points. Bad data = bad direction, no matter how good the optimizer. This is why Pillar 1 (Data) comes before Pillar 2 (PEFT) and Pillar 3 (Alignment).
---
## Key Terms
| Term | Definition |
| --- | --- |
| **The Steering Stack** | The five-layer model: Base → Adapter → Steer → Export → Boundary |
| **Steering** | Redirecting an already-capable model's behavior (format, preference, reasoning activation, refusal removal) |
| **Knowledge injection** | Actually shifting the model's knowledge distribution; the domain of continued pretraining, not fine-tuning |
| **Base model** | Layer 1; the pretrained weights; the "world model" |
| **Adapter** | Layer 2; a lightweight, swappable steer (LoRA, DoRA) — under 1% of params |
| **Open-data model** | A base released with its training corpus (OpenBMB, OLMo, Tulu, SmolLM3) — auditable |
| **Open-weights-only** | A base released as weights without training data (Llama 3.x) — not auditable |
| **Abliteration** | Layer 3; steering *away* from refusal by deleting a residual-stream direction |
| **The Boundary (harness)** | Layer 5; what bounds what the model *may* do — the subject of Courses 1 and 2A |
| **Intrinsic dimension** | The hypothesis that useful fine-tuning changes live in a low-rank subspace — the theoretical basis for LoRA |
---
## Lab Exercise
See `07-lab-spec.md`. The "Load, Steer, Swap" lab: load MiniCPM5-1B, apply a format-steering LoRA, swap it for a different one, observe the behavior change without touching the base. The modularity of the stack, felt in under 30 minutes.
---
## References
1. **Aghajanyan et al. (2020)** — *Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning*. arXiv:2007.07784. The theoretical basis for "fine-tuning is steering."
2. **Hu et al. (2021)** — *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685. The adapter that made steering cheap.
3. **Dettmers et al. (2023)** — *QLoRA: Efficient Finetuning of Quantized LLMs*. arXiv:2305.14314. Made 7B fit on a consumer GPU.
4. **Liu et al. (2024)** — *DoRA: Weight-Decomposed Low-Rank Adaptation*. arXiv:2402.09353. DoRA closes ~half the gap between LoRA and full FT.
5. **Shuttleworth et al. (2024)** — *An Illusion of Equivalence: LoRA vs Full Fine-Tuning*. arXiv:2410.21228. LoRA and full FT produce structurally different weights.
6. **Arditi et al. (2024)** — *Refusal in Language Models Is Mediated by a Single Direction*. arXiv:2406.11717, NeurIPS 2024. The basis for abliteration (Module FT17).
7. **NTIA (2024)** — *AI Open-Model Weights Report*. The government citation for open-weights enabling sensitive-data use.
8. **Course 1 — Master Course** — *Harness Engineering*. The Layer 5 reference; the complement to this course.