Every time a new AI model launches, we get a new set of benchmarks telling us how much better it is at reasoning, coding, writing, math, or whatever else it has been optimized for.
Those benchmarks are helpful, but they usually do not answer the question I actually care about: what should I use this new model for?
AI use can get expensive, especially when you are working with premium models or making lots of API calls. Knowing when to use a more powerful model, and when a faster or cheaper model will work just as well, can save money and improve the quality of your work.
Aaron Rasmussen has a great example he posted on LinkedIn when Fable was made released. Every time there is a new model, he asks the model to produce the image of a man from a bird’s-eye view, holding a lantern in a forest, with accurate shadows around him.
By running the same task, he can quickly see what the model does better, where it still struggles, and whether those improvements are meaningful for the work he cares about.
A lot of other people I know have similar small tests that help them understand where new models really matter for them. With a couple of days left for people on Claude’s basic plan to try Fable - I thought this would be a good week to create those tests for ourselves.
Before we start: if you are getting this email forwarded to you and are not yet part of the lab - you can join the tiny AI lab here.
The Challenge:
Build your own AI benchmark.
Pick one to three tasks from your real work that you can really assess the output for.
The goal is to create a simple way to compare how different models handle the work you do with AI day to day. When a new model launches, your benchmark should help you understand what is different about its performance and what you should use it for.
The prompt to get you started
Drop this into your AI of choice to get started:
Heavy AI users often have a few repeatable tasks they use to understand new models. One image expert asks every new image model to generate a man holding a lantern in a forest, viewed from above, with accurate shadows. Because he understands lighting well, the same task helps him quickly see what has improved, where the model still struggles, and whether those changes matter for his work.
I want to create my own benchmark for the work I do.
Interview me one question at a time. Learn what I do, how I currently use AI, which tasks matter most in my work, and where differences in model quality are most noticeable or useful.
Then design one to three simple benchmark tasks based on my real work.
For each task, give me:
the exact prompt to use
what I should pay attention to in the output
Keep the full benchmark simple enough to run in under ten minutes.
Start with your first question.
Too Easy:
Run your benchmark across three models you already have access to.
Compare the outputs and create a simple model cheat sheet. Note which model you would use for each task, where each one struggles, and what you should test again the next time a major model update is released.
This gives you the beginning of your own model-routing system.
If you are using APIs, you can take this one step further by routing different types of work to different models. You might use a faster, lower-cost model for simple or high-volume tasks and reserve premium models for the work where the additional quality matters.
This helps you optimize for both cost and performance instead of automatically sending every task to the most expensive model.
Why this matters?
Global benchmarks are useful because they help us understand model progress at a large scale. But they often lack the specificity to tell you how that progress should change the way you work.
Most people use whichever model is selected by default. They open an AI tool, type a prompt, and accept the model choice that has already been made for them.
Different models are better at different things. One may be stronger at research, another at writing or coding. As models get better and more specialized, knowing which one to use for which task is becoming an important part of using AI well.
That’s it for this week.
Hit reply and tell me what you thought of the challenge this week.
I read every response, and your experience helps me shape these challenges. If you get stuck or have questions, reach me at [email protected]