When a general-purpose LLM doesn’t quite do what you need, whether that’s the raw model or an agent built on top of it, the reflex, especially for people who’ve built classic ML before, is to fine-tune it: train it on your data, make it yours. That instinct comes from a world where the model was the system and retraining was the main lever you had. With today’s LLMs it usually isn’t where you should start. There are three ways to close the gap between a general model and your specific problem: give it better grounding context, optimize the harness around it, or fine-tune the weights. Most teams reach for the heaviest of the three first.
What changed my mind isn’t a benchmark. It’s watching the same reflex play out across three very different needs, and noticing that each one actually wanted a different lever. A recipe-design assistant in a pharma MES that had to be domain-aware and grounded. A general-purpose data-and-anomaly agent running in that same operation. And that same kind of workload pushed out to the edge. Put them side by side and “which lever” stops being an argument about taste and becomes a question you can answer from the need.
When the need is domain knowledge, the lever is grounding. I saw this most clearly on a pharmaceutical project: a recipe-design assistant inside a pharma MES. A recipe engineer takes an existing manufacturing recipe and authors a new one (every asset integration, which step executes where, all the checks and balances wired in), and the whole thing has to be expressed in the right pharma terms and obey strict company- and process-specific guidelines at every line. It’s a specialized, regulated domain (think 21 CFR Part 11, where an electronic record has to carry an audit trail and traceability, on top of being correct) where an answer you can’t trace is worse than no answer. The prevailing view in the room was strong and specific. A general model can’t be trusted here, so it has to be trained for the segment; fine-tune it on the domain’s data. Nobody argued the goal. What I kept snagging on was that “aware and grounded” is, above all, a grounding problem. Put the recipe library, the term definitions, and the governing guidelines in front of the model at authoring time, with the citations and guardrails the domain demands, and a lot of the gap closes without touching a single weight. Fine-tuning is worth it only for what grounding genuinely can’t reach: a reasoning pattern or output discipline the base model can’t hold even when the right documents are in front of it. And the reason not to lead with it is the cost shape. The model pass itself is the cheap part. A parameter-efficient fine-tuning run is often a few tens of dollars of compute*, so it’s easy to wave through. The real bill is the labeled-data pipeline behind it and the standing commitment to retrain as the regulated ground shifts. Counting data prep and iteration, a serious enterprise effort runs on the order of $10K to $100K+*, re-run every time guidance moves. In a regulated segment, the ground moves. That didn’t feel like an end state to me. It felt like a treadmill you agree to walk forever.
Where they actually landed was context management with RAG: grounding rather than fine-tuning, and over the treadmill that was the right instinct. Thinking about it today, I’d lean harder on the harness than we did. A good part of what remained wasn’t missing knowledge, it was behavior: getting the model to apply the guidelines and assemble a valid recipe step by step. That’s harness work, not weights. Grounding was the right lever to start with; the harness was the one we under-used.
When the need is behavior, the lever is the harness. Stay in that same MES, but change the job: a general-purpose agent working alongside a human rather than replacing the judgment. Its objective runs along a spectrum. At one end it’s a generalized data-insight assistant: read the table, open the file, pull the structured records and the messy notes, extract the insight, hand it back in a form someone can act on. At the other it’s a narrow anomaly agent: watching a time-series stream or joining logs against relational data, flagging what looks off, attaching a confidence to the call, and passing it to a person to triage or decide. The old way to cover many anomaly types was a stable of individually trained ML detectors, one per anomaly, each with its own labels and retraining. A single general agent, tuned through its harness and grounded in the asset’s context, can span a range of them without a bespoke trained model for each. The base model can already do all of that. What decides whether it works isn’t domain knowledge and it isn’t the weights. It’s whether the model uses its tools and its context well: reads the whole table instead of the first screen, calibrates a confidence it won’t overstate, and knows when to hand the call back to the human. That’s the harness (system prompts, tool descriptions, a bit of middleware), and it’s the lever most teams skip straight past.
A result landed recently that pushed me to finally write this down; it crystallized something I’d been circling for a while. I read it less as a product story than as evidence: an open model, NVIDIA’s Nemotron 3 Ultra, matched top closed frontier models on LangChain’s own agent benchmark with no retraining at all, at roughly a tenth of the cost per run, the entire gain coming from engineering the harness rather than the weights. One fix is a perfect tell for what “harness” even means. The model kept answering end-of-file questions from the first page of a long file because nothing told it the file continued. The fix wasn’t a smarter model; it was a middleware note saying “there’s more, keep reading.” The capability was already there. The harness just stopped it from misusing it. (The playbook; NVIDIA’s step-by-step.) For an MES agent whose whole job is reading tables and files without getting them wrong, that’s exactly the lever.
When the need is performance, the deployment target picks the lever, and it can flip the order. Take that same MES-style workload and change where it runs. In the cloud, compute is elastic: you can serve a big model behind a fat retrieval prompt and scale out to hold latency, so grounding-first is perfectly reasonable and you reach for fine-tuning mainly when the economics of volume push you there. Move it to the edge, an inline cell on the factory floor with a fixed GPU budget and a hard latency ceiling (which is where a lot of my head is anyway), and the order inverts. “Faster or cheaper” stops being a preference and becomes the spec. The giant model behind a fat prompt won’t run in the power and time you have, so the heaviest lever to build becomes the first one you reach for: a small model that fits the hardware, fine-tuned or distilled down from a bigger one. And the harness pulls its weight in an unglamorous way: not by adding cleverness but by removing waste. Fewer steps, shorter tool loops, a tighter prompt, middleware that lets the smaller model hold the task instead of forcing you back up to bigger hardware. Same problem, two homes, opposite first move.
Line the three up and the pattern is the whole point. They sort by how much you commit, grounding lightest and fine-tuning heaviest, but you don’t climb that ladder blindly, cheapest-first, because each rung fixes a different kind of need. Missing domain knowledge points at grounding, a capable-but-clumsy agent points at the harness, a hard performance envelope points at shrinking the model. And the axes cross: the lightest lever to build is often a tax at runtime, while the heaviest to build can be the runtime win, so the order you’d pick to ship fast is close to the reverse of the order you’d pick to run cheap at scale. They also stack: a real system often grounds, tunes its harness, and runs a shrunk model underneath, all at once. The table below is that whole picture at a glance.
| Lever | The need it fixes | What you commit | Build cost | At runtime |
|---|---|---|---|---|
| Grounding context | Missing domain knowledge (the pharma case) | Nothing in the weights: retrieval + prompt | Low | A tax: extra retrieval and a fatter prompt on every call |
| Harness optimization | Capable but clumsy behavior (the MES agent) | How the model behaves, not what it is | Low–medium (needs a real eval) | Usually neutral to better: less wasted work |
| Fine-tuning / distillation | A hard performance envelope, or capability grounding can’t reach (the edge) | The weights, a data pipeline, a retrain loop | High (~$10K–$100K+ loaded)* | Often the win: a small model beats big-model-plus-prompt |
So the point isn’t that fine-tuning is bad, or that the harness beats training. It’s that fine-tuning became the default reach when it should be the considered one. Diagnose the need first: is it knowledge, behavior, or performance. Name the constraint: quality or cost. Then reach for the least-committing lever that actually closes that need, knowing the three stack and that the cheapest thing to build is rarely the cheapest thing to run. The gap tells you what’s broken; the deployment tells you which lever you’re allowed to fix it with.
I want to stop hand-waving this and put a number on it. I’m planning to take an open model through exactly this loop soon, to see how far grounding and harness tuning get before fine-tuning genuinely earns its keep, and write up what I find.
* Figures are generalized industry reference ranges. A parameter-efficient (LoRA/QLoRA) run is typically ~$10–30 in raw compute, while a full enterprise fine-tuning effort loaded with data prep and iteration commonly lands around $10K–$100K+. They show the cost shape, not this specific engagement’s numbers.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA. I may have gotten something wrong or missed a detail, and I’m always happy to be corrected.