b. Bryan Zane Smith / llmbench

An open-source CLI that benchmarks language models across providers. Browse four published leaderboards in one table, refreshed daily, and run your own speed and image-generation benchmarks against any provider you have a key for.

$ uvx llmbench

Also pipx run llmbench or pip install llmbench. Python 3.11+, MIT.

Source on GitHub ↗

What llmbench does

01

Measure

One prompt set, many models, two ways to score.

throughput
Time to first token, tokens/sec, inter-chunk and total latency, token usage.
image_gen
Generation latency plus the saved PNGs, for side-by-side review.

Every run writes a self-contained HTML gallery you can scroll through.

02

Configure

One file, ~/.llmbench/config.yaml, holds every API key and every model. Twenty providers are known by name.

  • openai
  • anthropic
  • gemini
  • moonshot
  • deepseek
  • xai
  • groq
  • mistral
  • together
  • fireworks
  • openrouter
  • perplexity
  • cerebras
  • qwen
  • nvidia
  • nebius
  • deepinfra
  • sambanova
  • flux

Local, no key required

  • ollama
  • vllm
  • lmstudio
  • llamacpp

Four wire protocols cover all of them. Environment variables still work as a fallback, so CI needs no config file.

03

Compare

Published scores from four sources, refreshed daily. No API key needed to read them, in the terminal or in the table below.

huggingface
Open LLM Leaderboard v2: IFEval, BBH, MATH, GPQA, MUSR, MMLU-PRO.
lmarena
LMArena ELO, from head-to-head human preference voting.
aider
Aider Polyglot: multi-language code-editing pass rate.
bundled
Snapshot shipped inside the package, so it works offline.

llmbench leaderboard --source lmarena --top 10

Published leaderboards

Scores from HuggingFace Open LLM v2, LMArena, and Aider Polyglot, plus an offline snapshot, in one sortable table. Refreshed daily. These are published numbers, not measured by llmbench.

Loading…
# Model Organization Source Score Metric