How to Build an LLM Model Selection Rubric for Any Workload

Most teams pick a model the same way they pick a lunch spot: whatever's top of mind. A launch made headlines, a colleague swears by it, the leaderboard moved. Six months later the bill is triple what it needs to be and nobody can say why that model, for this job.

An llm model selection rubric fixes that. It's a scored framework that rates candidate models against criteria weighted for one specific workload, so the pick is defensible and repeatable. Not "Claude feels smarter." A number, a weight, and a reason you can show a skeptical reviewer.

This isn't about finding the "best" model. There isn't one. There's the right model for a support-ticket classifier, a different one for long-document summarization, and a third for a latency-sensitive autocomplete. The rubric is how you tell them apart.

Why habit and hype pick the wrong model

Two failure modes dominate. The first is habit: the model that worked on the last project gets reused on a job with completely different constraints. The second is hype: a launch dominates a news cycle and suddenly it's the default, benchmark-topping numbers and all.

Both skip the only question that matters. What does this workload actually need? A summarizer for legal briefs cares about faithfulness and long-context handling. A real-time chat assistant cares about time-to-first-token. Those are different jobs, and the leaderboard doesn't know which one you're running.

Public benchmarks make this worse. They're measured on inputs that look nothing like yours. Galileo's writeup on instance-specific rubrics makes the related point for output grading: generic criteria "produce a dangerous kind of false confidence, with scores trending upward while visible quality stagnates." The same trap applies to model choice. A generic ranking flatters every model and tells you nothing about your task.

Here's the opinionated part: the highest benchmark score is a weak signal, and treating it as the answer is how teams end up overpaying for capability they never use. A frontier model priced for reasoning-heavy work is money set on fire when the job is single-label classification a mid-tier model nails at a tenth the cost.

What a good model selection rubric actually contains

A usable rubric has four moving parts. Skip any one and it collapses into a vibe.

1. A workload profile. What goes in, what comes out, how much, how fast, and what a wrong answer costs. Vague profiles produce vague scores.
2. Weighted criteria. Accuracy, latency, cost per unit, safety behavior, and integration fit — each weighted by how much this job depends on it. Weighting is the whole game.
3. Honest scoring. Score each candidate per criterion, and mark unknowns as unknowns. A fabricated 8/10 is worse than an honest "not tested yet."
4. A decisive bake-off. Run the top two on a small set of real inputs before you commit. The rubric shortlists; the bake-off decides.

The order matters. Weight the criteria before you look at any scores. Do it after and you'll unconsciously inflate the weight of whatever your favorite model happens to be good at. That's not analysis. That's rationalization with a spreadsheet.

The prompt that runs the rubric for you

Doing this by hand means a spreadsheet, a lot of tab-switching, and a scoring pass that drifts every time you get tired. A prompt pack turns it into a reusable job. The LLM Model Selection Rubric is built as four chained prompts, and the anatomy is worth seeing:

Variables → Prompt → Output

{{workload_description}}   the job in plain language
{{constraints}}           latency budget, cost ceiling, compliance limits
{{candidate_models}}      the models you're actually considering
{{traffic_profile}}       volume, peak load, request shape
        ↓
Prompt 1: Workload Profile Extractor    → {{workload_profile}}
Prompt 2: Candidate Scoring Matrix       → weighted scores + unknowns
Prompt 3: Bake-Off Protocol Designer     → a runnable test on real samples
Prompt 4: Migration and Revisit Plan     → {{decision_summary}} + revisit trigger
        ↓
Output: a scored matrix, a bake-off you can run, and a dated decision

The output contract is what makes it stick. Every score carries a one-line justification and an explicit confidence level. Unknowns stay flagged, not guessed. So when someone asks "why not GPT-class reasoning here?" the answer's already written down: this workload weighted cost at 40%, latency at 30%, and the reasoning premium bought nothing the profile needed.

Model behavior worth building into the profile

Scoring isn't just accuracy and price. Models behave differently in ways that break workloads if you don't test for them. Claude honors a ## Output format heading more reliably than an inline "respond in JSON" instruction, so a strict-schema job scores it higher on format compliance. GPT-4o often needs the schema restated on the final line of the prompt when the input is long. Gemini's long-context window is real, but recall across a 200k-token paste isn't uniform — the middle sags.

None of that shows up on a leaderboard. All of it shows up in production. Bake it into the {{scoring_notes}} so the rubric captures the behavior you'll actually depend on.

Step-by-step: run your first selection

1. Write the workload profile

Describe the job in {{workload_description}}: the input, the desired output shape, and what a wrong answer costs. "Classify support tickets into 6 categories, 40k/day, an error routes a customer wrong" is a profile. "We need a good chatbot" isn't.

2. Set constraints and weights

Put your hard limits in {{constraints}} — the latency budget, the monthly cost ceiling, any data-residency rules. Then weight the criteria. If cost dominates, say so numerically before scoring.

3. Score the candidates

List real options in {{candidate_models}}. Score each against every weighted criterion. Mark anything untested as unknown; don't paper over it.

4. Run the bake-off

Take the top two and run the generated protocol on 20–50 real inputs. This is where a leaderboard darling loses to a cheaper model that's simply good enough at your exact task.

5. Record the decision and its expiry

Write the pick, the weights, and a revisit trigger into {{decision_summary}}. Models ship new versions constantly. A decision with no expiry date is a decision that quietly rots.

Variables you'll set

A quick caveat: the rubric is a decision aid, not an oracle. It structures the trade-offs and forces honesty about unknowns. The bake-off on real data is still what settles a close call, and you should trust that over any score on paper.

Getting started

1. Grab a workload you're currently running on a model nobody chose deliberately.
2. Write its profile into {{workload_description}}.
3. Set {{constraints}} and weight the criteria — before looking at candidates.
4. Score two or three models honestly, unknowns flagged.
5. Run the bake-off on real samples.
6. Record the decision and a revisit date.
7. Do the same for your next unexamined workload.

If model choice is a recurring decision — new features, new workloads, quarterly cost reviews — a repeatable rubric beats re-arguing it from scratch every time.

Once you've picked a model, the next question is what it costs at your volume. The token cost profiling prompt quantifies that, and if you're choosing between specific frontier models for planning-heavy work, the o3 vs Claude planning comparison walks through one such head-to-head. Both pair well with a rubric that's already told you what your workload weights.