Most "best AI models" lists are a leaderboard screenshot with a paragraph under each name. By the time you finish reading, a new model has landed and the order has changed. I've watched that happen every month this year, and it's why I stopped picking models by rank.
My name is Artem, and I run the Writingmate blog. I spend a big part of every week putting the same prompt through several models and comparing what comes back, because the model that tops a chart is often not the one that finishes my job. This post is the process I use: pick by task, read benchmarks with suspicion, then confirm with a 15-minute test you can repeat whenever something new launches.
You'll get a decision table by task and budget, a short guide to what benchmarks measure (and where they mislead), a scoring rubric, and the math for cost per usable answer. No brand loyalty required.
Why "best AI model" is the wrong question
"Best" depends on three things: the job, the budget, and how much a wrong answer costs you. A model that writes beautiful prose may stumble on a refactor across six files. A model that is brilliant at code can be oddly flat when you ask for a warm customer email. And the cheapest model in the catalog is often the best one for the boring 5,000-row job you'd never pay a premium model to do.
So instead of asking "which is the best LLM?", ask "which model finishes this job with the fewest retries at a price I'm okay with?" That framing also survives launches. New models come out weekly, but your jobs stay the same.
Here's the thing most lists skip: the answer is usually two or three models, not one. A strong general model for hard work, a cheap fast model for volume, and maybe a long-context specialist. That's the short list I'll help you build below.
Best AI models by task: a decision table
I'm deliberately not ranking brands here, because the specific names move too fast. Instead, this table says what to look for in each job and what to test first. Browse the Writingmate models directory to filter by context window, price, and modality, then pick two or three candidates per row.
Job | What matters most | Look for | Budget tier |
|---|---|---|---|
Long-form writing | Voice, structure, following style notes | A flagship general model with strong instruction following | Mid to high |
Coding and debugging | Correct code, multi-file reasoning, tool use | A model tuned for code or agentic work, large output limit | Mid to high |
Research with long documents | Context window, recall, citing the right passage | Long-context models (check the real window, not the headline) | Mid |
High-volume cheap tasks | Cost per call, speed, consistent formatting | Small or "flash"-class models | Low |
Multimodal (images, PDFs, screenshots) | Reading charts and messy scans accurately | Models that list image or file input in their specs | Mid |
Two quick notes on that table. First, price differences between tiers are huge. In our recent news recap, one new long-context model was listed at $2 per million input tokens and $10 per million output, while one small model from a major lab started at $0.10 per million input tokens. That's a gap of 20x on input alone, so "cheap bulk work" is a real category, not an afterthought. Check current prices before you commit, since they change.
Second, "multimodal" means different things. Some models read images well but handle PDFs poorly. Test with your actual files, not a stock photo.
What AI benchmarks actually measure (and where they mislead)
Benchmarks aren't useless. They're just narrower than the headline suggests. A score tells you how a model did on one fixed set of questions, under one setup, on one day. Here are the failure modes I watch for.
- Contamination. If benchmark questions or close paraphrases leak into training data, a model can recall answers rather than reason. Public, older benchmarks carry the highest risk. Benchmarks built from problems published after a model's training cutoff (LiveCodeBench-style) are harder to game.
- Saturation. When several top models score above roughly 90%, the test stops separating them. A one-point lead on a saturated benchmark is noise, not a reason to switch.
- Setup sensitivity. Change the prompt, the harness, the compute budget, or even answer order, and rankings can shift. Researchers have shown that small perturbations can reorder leaderboards, and that rephrasing questions can cause measurable score drops, which hints at memorization.
- Cherry-picked sets. A vendor chart usually shows the benchmarks where its model wins. If a model "wins everything," ask which tests were left out.
- Style-voted leaderboards. Arena-style rankings reward answers people like at a glance. That's useful for chat feel, but it isn't the same as correct code or a faithful summary.
- Retired scoreboards. Even the field's favorite coding test has been questioned. One 2026 write-up notes that OpenAI's evals team announced it would stop reporting SWE-bench Verified because of contamination concerns.
The trend is also toward harder tests. A recent post on X from Deedy captured the mood:
"The creators of SWE-Bench just dropped a really simple new benchmark every LLM gets 0% on. ProgramBench asks: can models recreate real executable programs (ffmpeg, SQLite, ripgrep) from scratch with no internet? We are far from saturated on model quality." — @deedydas on X
Notice what that tells you: old coding scores look great, while a differently designed test gives 0%. Both can be true. Neither one is your workload.
The community has been saying this for years. A well-known r/LocalLLaMA thread titled "Karpathy on LLM evals" drew comments tying the problem to Goodhart's Law: once a metric becomes a target, it stops being a good measure. You can read it on Reddit.
Thread title: "Karpathy on LLM evals" — r/LocalLLaMA on Reddit (commenters connect the evals problem to Goodhart's Law)
How I compared: the ground rules
Here's what I checked when building this process, so you can judge how far to trust it.
- I treated benchmark scores as a way to build a shortlist, never to make the final call.
- I used the same prompt, same settings, and same input files for every model in a round.
- I ran each prompt at least twice, because one lucky answer proves nothing.
- I scored outputs with a fixed rubric before looking at prices, so cost didn't bias my judgment.
- For the video above, I used it as a reference on how leaderboards get gamed. I'm not attributing any specific numbers to it, and I'd suggest watching it with the same skeptical eye you'd bring to any chart.
I haven't published numeric results in this post on purpose. Scores from my prompts would be useless for yours. The point is the method.
The 15-minute same-prompt test
This is the part that replaces the leaderboard. It takes about 15 minutes once you've done it a couple of times.
- Pick one real task. Not a toy question. Paste in the actual brief, draft, bug, or document you'd work on today.
- Pick 3 to 4 candidates. One flagship, one mid-priced model, one cheap model, and optionally one long-context or multimodal specialist.
- Send the identical prompt to all of them. In Writingmate you can open the models page, choose several models, and compare responses side by side in one workspace, so you're not copying and pasting across five tabs.
- Run it twice. Regenerate each answer once. If the second run is much worse, that model is inconsistent for this job.
- Score each answer with the rubric below. Do it blind if you can: cover the model names first.
- Write down cost. Note the token usage or per-call price for each run.
A simple four-point rubric
Score each answer 0 to 3 on each line. Max is 12. I call anything at 9 or above "usable," meaning I'd ship it with light edits.
Criterion | 0 = fail | 3 = pass |
|---|---|---|
Correctness | Wrong facts, broken code, invented sources | Everything checks out when verified |
Instruction following | Ignores format, length, or constraints | Hits every constraint in the prompt |
Usefulness | Generic, needs a full rewrite | Specific to my situation, minor edits only |
Consistency | Second run contradicts or degrades | Second run lands at the same quality |
A tip from experience: write the correctness check before you read the answer. For code, run it. For research, click two of the claimed sources. Models are very good at sounding right.
Cost per usable answer: the number that beats price per token
Price per million tokens is a trap on its own. A cheap model that needs four tries costs more, in money and in your time, than a pricier one that nails it once. The formula is simple:
Cost per usable answer = cost per run ÷ share of runs scoring 9 or above.
Here's a made-up example to show the math (these are illustrative numbers, not measured results):
Model type | Cost per run | Usable runs (of 10) | Cost per usable answer |
|---|---|---|---|
Flagship | $0.040 | 9 | $0.044 |
Mid-priced | $0.012 | 8 | $0.015 |
Small / flash | $0.004 | 5 | $0.008 |
On raw cost the small model looks 10x cheaper than the flagship. On cost per usable answer it's about 5x cheaper, and you still spend your own time reviewing the five failures. For a bulk job like tagging support tickets, where a person isn't reading every output, that 5x saving may still win. For a client-facing draft, the flagship's extra four cents is a bargain. This is why one model never covers everything.
Add your own time to the equation too. If a failed run costs you three minutes of editing, price that in. Most people find it dwarfs the API cost.
Build a short list of two or three defaults
After the test, you probably have a clear winner for the hard job, a clear winner for volume, and maybe a runner-up. Write them down:
- Default 1: the heavy lifter. Your go-to for writing, code, and anything where mistakes are expensive.
- Default 2: the volume model. Cheap, fast, and good enough for classification, extraction, and first drafts.
- Default 3 (optional): the specialist. Long context for big documents, or multimodal for screenshots and scans.
Then re-run your test only when something changes: a new launch in your price tier, a price cut, or a model that suddenly gets worse on your task. I do a quick check roughly once a month and ignore the rest of the launch noise. If you want help deciding how many subscriptions you really need, I wrote about that in our subscription cost breakdown, and the pricing page shows what multi-model access costs on one plan. If you're only starting out, the free AI apps guide is a gentler first step.
Common mistakes when choosing a model
- Switching for a two-point benchmark gain. Unless your task matches the benchmark, you won't feel it.
- Testing with one prompt. One prompt measures that prompt. Use at least two from the same job.
- Trusting the headline context window. A model can accept a million tokens and still miss details in the middle. Hide a specific fact deep in your test document and see if it's found.
- Ignoring output limits. Some models cut long answers short. That wrecks long-form writing and big code files.
- Judging by tone. Confident writing isn't correct writing. Verify first, admire later.
- Locking into one vendor. If your whole workflow lives in one app, switching costs you a week. Keeping a multi-model workspace avoids that.
So which model should you pick?
Honestly? I can't name one for you, and any article that does is out of date within weeks. What I can say is this: use benchmarks to shortlist, use your own prompt to decide, and use cost per usable answer to set the budget. Keep two or three defaults, check them monthly, and ignore the leaderboard drama in between.
If you want to run the test without juggling accounts, open the Writingmate models page, pick a few candidates for your next real task, and compare them side by side. It takes about 15 minutes, and you'll know more than any ranking can tell you.
See you in the next one!
Artem
Frequently Asked Questions
Sources
- LLM Benchmarks Are Broken—The Leaderboard Illusion (YouTube)
- @deedydas on X
- r/LocalLLaMA on Reddit
- Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models (arXiv)
- Benchmark Contamination: How to Read an AI Coding Leaderboard After OpenAI Quit SWE-bench
- How to Read AI Leaderboards Without Getting Fooled
- Writingmate models page
- our subscription cost breakdown
- pricing page
- free AI apps guide
Written by
Artem Vysotsky
Ex-Staff Engineer at Meta. Building the technical foundation to make AI accessible to everyone.
Reviewed by
Sergey Vysotsky
Ex-Chief Editor / PM at Mosaic. Passionate about making AI accessible and affordable for everyone.