How to Choose an LLM for Your Use Case: A Criteria Checklist

“Which LLM is best?” is the wrong question. The useful question is: which model is good enough for this specific task, at a cost and speed you can live with, under constraints you can’t break? The answer changes from one feature to the next, and it changes over time as providers release, reprice and retire models.

This guide gives you a repeatable checklist. Work through it once per use case and you’ll end up with a short list of candidates and a clear reason for the final pick — something you can explain to a teammate or revisit in six months.

Start with the job, not the model

Before you look at any model list, write down what the model has to do in one or two sentences. Be concrete:

• “Classify incoming support tickets into one of 12 categories and extract the order number.”
• “Answer questions about a 200-page policy manual, citing the section.”
• “Suggest code review comments on pull requests in a TypeScript monorepo.”

Then note the shape of a typical request: roughly how much text goes in, how much comes out, how many requests per day, and how fast a response needs to arrive. These four numbers drive most of the decisions below.

The checklist

Score each candidate against the criteria that matter for your job. Not every row applies to every use case — a nightly batch job doesn’t care about latency; a chat widget does.

Criterion	Question to ask	How to check
Task quality	Does it do my task well?	Your own eval set (see below)
Instruction following	Does it respect format and constraints?	Eval with strict output checks
Context window	Do my inputs fit with headroom?	Token count of a worst-case request
Output length limit	Can it produce the longest answer I need?	Provider docs: max output tokens
Latency	Is time-to-first-token and total time acceptable?	Measure from your region
Cost	What is cost per request and per month?	Token volumes × current prices
Modalities	Does it need images, audio, PDFs, tool use?	Provider docs
Structured output	Does it support JSON / schema-constrained output?	Provider docs + test
Deployment	API only, cloud marketplace, or self-hosted weights?	Licensing and hosting options
Data handling	Retention, training use, region, compliance needs	Provider terms and agreements
Stability	How long will this model version be available?	Deprecation policy
Rate limits	Can it handle peak traffic?	Account tier limits

Task quality comes first — and only your data can measure it

Public benchmarks are a reasonable first filter: they tell you whether a model is broadly in the right tier for reasoning, coding or multilingual work. But they measure someone else’s tasks, scored in someone else’s way, and models may have been tuned with those benchmarks in mind. A model that leads a leaderboard can still be mediocre at extracting fields from your invoices.

The fix is a small, private evaluation set: 30–100 real examples with the answers you expect. We cover how to build one in how to evaluate models on your own tasks.

Context window: fit plus headroom

Add up everything that goes into one request — system prompt, instructions, examples, retrieved documents, conversation history — and add the expected output. That total must fit in the context window, and you want headroom, because real-world inputs vary. Also remember that a large window doesn’t guarantee the model uses every part of it equally well. Our context window explainer goes deeper.

Cost: calculate, don’t guess

Most API models charge separately for input and output tokens, and output is usually the more expensive of the two. That means two models with similar “headline” prices can differ a lot for your workload depending on whether it’s input-heavy (summarising long documents) or output-heavy (generating long drafts).

Work it out with your own volumes. The formula, and a worked example with hypothetical numbers, is in what tokens are and how to estimate API cost.

Latency: measure it yourself

Latency depends on model size, output length, provider load and your distance to the endpoint. For interactive features, track two numbers: time to first token (how quickly the user sees something) and total generation time. Streaming hides some of the wait; it doesn’t make long outputs faster.

Data, compliance and deployment

These criteria are often pass/fail and should be checked early so you don’t waste time evaluating models you can’t use. Questions to answer:

• Will request data be retained, and for how long?
• Is it used for training by default, and can that be turned off?
• Can processing happen in a specific region?
• Do you need open weights so you can self-host?

Turn the checklist into a decision

A simple, defensible process looks like this:

1. Filter on hard constraints. Drop any model that fails a pass/fail criterion: data handling, deployment, modality, context fit.
2. Shortlist three to five candidates across different size tiers — include at least one smaller, cheaper model. Teams often skip this and overpay.
3. Run your eval set on each candidate with the same prompt. Record a quality score.
4. Measure latency on a sample of real requests.
5. Calculate monthly cost using your actual token volumes and the provider’s current prices.
6. Pick the cheapest model that clears your quality bar — not the highest scorer, unless the quality gap justifies the cost gap.

It helps to put the results in a single table. Here’s what one might look like. All numbers below are hypothetical, invented to illustrate the layout:

Candidate (hypothetical)	Eval score	Median latency	Est. monthly cost	Passes constraints
Large model X	94%	3.1 s	$1,800	Yes
Mid model Y	91%	1.4 s	$420	Yes
Small model Z	78%	0.6 s	$60	Yes

If your quality bar is 90%, model Y wins: it clears the bar at a fraction of X’s cost. If the bar is 95%, none qualify and you need to improve the prompt, add retrieval, or widen the shortlist.

Common mistakes

• Choosing by reputation. “Everyone uses model X” is not evidence that X is right for your task.
• Ignoring output tokens. A cheap input price can hide an expensive output price.
• Testing on three examples. Small samples make lucky or unlucky runs look meaningful.
• Not re-evaluating. New models and price changes can make last quarter’s choice suboptimal. Keep the eval set and rerun it.
• One model for everything. Different features can use different models. Routing easy requests to a smaller model is one of the most effective ways to cut API costs.

Document the decision

Write a short record: the task, the candidates, eval scores, latency, cost estimate, the date, and why you chose what you chose. When a provider changes prices or ships a new model, you can rerun the same process in an afternoon instead of starting from scratch.

That repeatable comparison — candidates side by side with sources and dates — is exactly what we’re building MaximumAI to make easier.