Prompts vs Fine-Tuning: When to Use Each for Real Work

Most teams reach for fine-tuning too early. The model returns the wrong shape twice, someone says "let's just fine-tune it," and a week of data labeling starts before anyone has written a prompt with a real output contract. That's the expensive way to fix a cheap problem.

The prompts vs fine-tuning question isn't ideological. It's a cost-and-control decision, and for most jobs the answer is "prompt first, and probably prompt only." Fine-tuning earns its keep in a narrow band: high volume, stable requirements, and a behavior a prompt genuinely can't reach. This post draws the line, with the model-by-model behavior that actually decides it.

What each one changes

Prompting shapes the input. You write instructions, examples, and an output format, and the base model does the rest. Nothing about the model changes. Swap the prompt, swap the behavior. That's the whole appeal: it's reversible in seconds.

Fine-tuning shapes the model. You train the base model further on your own labeled examples so it leans toward a behavior by default. The weights change. You get consistency and shorter prompts, but you've spent training compute and pinned yourself to a snapshot you now have to maintain.

Here's the part the generic explainers skip: fine-tuning doesn't retire the prompt. A fine-tuned model still reads whatever you send it, and a weak prompt drags it down the same way it drags down any model. Fine-tuning moves the floor. The prompt still carries the task.

The decision, as a table

Ranking pages on this query almost always run a feature-by-feature grid (cost, speed, data needs). Useful, but it doesn't tell a developer what to do. This one maps each situation to an action.

The honest default sits in the top rows. You fine-tune when prompting has been pushed hard and hit a wall you can name, not when the first two attempts came back wrong.

Why "the model returned bad JSON" is almost never a fine-tuning problem

The most common trigger for a premature fine-tuning project is format instability, and it's almost always a prompt-structure issue you can fix in an afternoon.

Model behavior here is specific, not generic. Claude honors a ## Output format heading placed at the end of the prompt more reliably than an inline "respond in JSON" instruction buried mid-paragraph. GPT-4o tends to drift on long pasted inputs unless the schema is restated on the final line, because it weights the most recent tokens heavily. Gemini follows an explicit example of the exact output shape better than a prose description of it. None of that is fixed by training. It's fixed by moving the contract to the end and showing three examples in the exact target shape.

If a prompt with that structure, three consistent few-shot examples, and a pinned model version still can't hit the target, now fine-tuning is a rational next step. You've proven the base model won't do it under instruction.

What fine-tuning is genuinely better at

Fine-tuning isn't a loser here. It wins clearly in three places, and it's worth being precise about them so the choice stays honest.

Consistency at scale is the big one. If a task runs ten thousand times a day and the tone or format must never wobble, baking that behavior into the weights beats hoping every call reads a long prompt correctly. It also shortens the prompt: a fine-tuned model that already knows your house format doesn't need the format re-explained on every call, and that token saving compounds across millions of requests.

The second is reaching behavior outside the base distribution. If the task involves a domain vocabulary, a labeling scheme, or a reasoning style the base model simply wasn't trained on, no amount of prompting conjures it. Examples in the prompt help, but at some point the model needs the pattern in its weights.

The third is latency. A shorter prompt is a faster prompt. For real-time systems where every hundred milliseconds matters, trimming a 2,000-token instruction block down to a 200-token task prompt (because the rest lives in the weights) is a real win.

What fine-tuning is not good at: fresh facts. Training freezes knowledge at a snapshot. For anything that changes, retrieval on top of a prompt beats fine-tuning every time. And no, you don't fine-tune to teach the model this week's pricing.

The variables that make a prompt reusable

The reason "just prompt it" gets a bad name is that most prompts aren't built to be reused. A one-off prompt someone typed into a chat window isn't a fair comparison against a fine-tuned model. A prompt built as a reusable asset is.

That means variables and a locked contract. Instead of hardcoding a request, you leave slots: {{source_text}}, {{target_format}}, {{constraints}}. The instructions stay fixed, the inputs change, and the output contract keeps every run the same shape. That's the difference between a prompt you retype and a prompt pack you run.

A prompt with those slots and a contract at the end handles most of what people assume needs fine-tuning. That's the opinion worth defending: for the median task, a well-structured reusable prompt beats a fine-tune on total cost of ownership, because you keep the ability to change it in seconds and you never maintain a training pipeline.

Getting started without over-engineering

You don't need a decision committee. You need to try the cheap option properly before you pay for the expensive one.

1. Write the task as a reusable prompt. Role first, then context, then the output contract at the very end.
2. Add three few-shot examples in the exact target shape. Not four in three different shapes — three identical ones.
3. Pin the model version so behavior doesn't drift under you mid-evaluation.
4. Run it on twenty real inputs, not two happy-path ones. Watch where the format breaks.
5. If it holds, you're done. Ship the prompt. No training run.
6. If it doesn't, name the failure precisely — is it format, domain knowledge, or latency? That names your fine-tuning case.
7. Only then scope a fine-tune, and keep the prompt on top for per-run specifics.

For the structure and context-ordering side of that, the Context Engineering Harness pack gives you the prompt scaffold that survives long inputs — the exact ordering fix that resolves most "I need to fine-tune" moments. Pair it with how to budget a context window for AI agents so the prompt stays inside the model's reliable range.

Fine-tuning is a real tool with a narrow, valuable job. But it's the second thing to try, not the first. Prove the prompt can't do it, name the reason, and you'll know exactly when to spend the training budget. Most of the time you won't need to. For the buy-versus-build side of that same call, see where to buy ready-made AI prompts, or start from the full catalog if you'd rather not write the scaffold yourself.