happy to see ant taking a real shot at this, I had the intuition it was an important problem space some months ago already with this open-source take https://github.com/edonadei/caliper
Hey, glad that you appreciate it! Model version, yes you can either overwrite with the --model and --judge-model you can overwrite at runtime or in the yaml spec.
For the seed: no I haven't added it, and I don't think the harnesses like claude code or Codex supports it nowadays. (You're right OpenAI API exposes a best-effort seed parameter though)
But TBH I was not trying too much to control the LLM itself. I remember reading nice work by thinking machines on it. Here I decided to focus way more on pass@k because I'm focusing on a "classic" usage that is not trying too hard to play with seed or temperature, just the model with sane defaults.
If you feel that could be useful, please file me an issue and I'll consider adding it :)
Hey everyone
I found myself constantly searching for updated information about different LLMs and their capabilities, so I built thesota.fyi - a simple dashboard that compares AI language models.
What it does right now:
Shows key metrics for popular models (GPT-4, Claude, Gemini, etc.)
Compares performance in different categories (coding, math, instruction following)
Updates regularly to maintain current information
Simple, clean interface focused on readability
Why I built it:
Needed a quick way to compare models for different use cases (Inspired by levels.fyi)
Wanted to track progress as models improve
Found existing solutions either outdated or too complex
Current features:
Overall performance scores
Specialized task scores (coding, math, instruction)
Basic model information (organization, license type, knowledge cutoff)
Regular updates to keep information current
Looking for feedback on:
Is this useful for you? Would you use it?
What features would make this more valuable?
What information about models do you find most important?
Any suggestions for improving the scoring system?
You can check it out at: thesota.fyi
I'm planning to keep this tool free and hopefully make it more comprehensive based on community feedback. Any thoughts or suggestions would be really appreciated!
reply