LLM Benchmarks
| Model | Created By | License | Context Window | Intelligence Index | MMLU-Pro (Reasoning & Knowledge) | GPQA Diamond (Scientific Reasoning) | Humanity's Last Exam (Reasoning & Knowledge) | LiveCodeBench (Coding) | SciCode (Coding) | HumanEval (Coding) | MATH-500 (Quantitative Reasoning) | AIME 2024 (Competition Math) | Multilingual Index (Artificial Analysis) | Input Price (USD/1M Tokens) | Output Price (USD/1M Tokens) | Median Tokens/s | P5 Tokens/s | P25 Tokens/s | P75 Tokens/s | P95 Tokens/s | Median First Chunk (s) | P5 First Chunk (s) | P25 First Chunk (s) | P75 First Chunk (s) | P95 First Chunk (s) | Total Response (s) | Reasoning Time (s) | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| o4-mini (high) | OpenAI | Proprietary | 200k | 70 | 83% | 78% | 18% | 80% | 47% | 99% | 99% | 94% | $1.93 | $1.10 | $4.40 | 128.6 | 74.7 | 101.3 | 145.1 | 172.2 | 39.96 | 39.96 | 17.23 | 26.41 | 53.84 | 72.69 | 43.85 | 0 | |
| Gemini 2.5 Pro | Proprietary | 1m | 69 | 86% | 84% | 17% | 78% | 40% | 99% | 98% | 87% | $3.44 | $1.25 | $10.00 | 148.4 | 139.1 | 143.4 | 154.2 | 160.4 | 40.54 | 40.54 | 29.87 | 34.42 | 44.65 | 68.49 | 43.91 | 0 | ||
| o3 | OpenAI | Proprietary | 128k | 67 | 85% | 83% | 20% | 53% | 41% | 99% | 99% | 90% | $17.50 | $10.00 | $40.00 | 234 | 145.9 | 186.8 | 287.7 | 386.6 | 14.19 | 14.19 | 7.59 | 10.61 | 16.55 | 22.13 | 16.33 | 0 | |
| Grok 3 mini Reasoning (high) | xAI | Proprietary | 1m | 67 | 83% | 79% | 11% | 70% | 41% | 98% | 99% | 93% | $0.35 | $0.30 | $0.50 | 75.8 | 38.9 | 49 | 99.1 | 166.4 | 0.28 | 26.65 | 0.19 | 0.26 | 0.51 | 19.98 | 33.24 | 26.37 | |
| o3-mini (high) | OpenAI | Proprietary | 200k | 66 | 80% | 77% | 12% | 73% | 40% | 99% | 86% | $1.93 | $1.10 | $4.40 | 156.6 | 110 | 135.8 | 167.8 | 197.9 | 56.65 | 56.65 | 28.34 | 48.35 | 63.86 | 84.11 | 59.85 | 0 | ||
| o3-mini | OpenAI | Proprietary | 200k | 63 | 79% | 75% | 9% | 72% | 40% | 97% | 97% | 77% | $1.93 | $1.10 | $4.40 | 149.9 | 88.9 | 125.4 | 177 | 216.8 | 15.01 | 15.01 | 9.76 | 11.36 | 18.32 | 21.66 | 18.35 | 0 | |
| Qwen3 235B A22B (Reasoning) | Alibaba | Open | 128k | 62 | 83% | 70% | 12% | 62% | 40% | 93% | 84% | $0.30 | $0.20 | $0.60 | 41.3 | 18 | 30.3 | 51.3 | 87.8 | 0.55 | 48.99 | 0.26 | 0.45 | 0.63 | 2.56 | 61.1 | 48.43 | ||
| o1 | OpenAI | Proprietary | 200k | 62 | 84% | 75% | 8% | 68% | 36% | 97% | 97% | 72% | $26.25 | $15.00 | $60.00 | 129.4 | 99 | 109.9 | 158.3 | 187.9 | 22.09 | 22.09 | 15.47 | 20.18 | 27.18 | 36.92 | 25.95 | 0 | |
| Llama 3.1 Nemotron Ultra 253B Reasoning | NVIDIA | Open | 128k | 61 | 83% | 73% | 8% | 64% | 35% | 95% | 75% | $0.90 | $0.60 | $1.80 | 42.5 | 37.8 | 41.7 | 42.6 | 42.7 | 0.64 | 47.69 | 0.61 | 0.64 | 0.66 | 0.7 | 59.45 | 47.04 | ||
| Gemini 2.5 Flash (Reasoning) | Proprietary | 1m | 60 | 80% | 70% | 12% | 51% | 36% | 98% | 84% | $0.99 | $0.15 | $3.50 | 351.6 | 314.3 | 332 | 357.4 | 372.4 | 9.02 | 9.02 | 7.3 | 7.93 | 10.25 | 15.03 | 10.45 | 0 | |||
| DeepSeek R1 | DeepSeek | Open | 128k | 60 | 84% | 71% | 9% | 62% | 36% | 98% | 97% | 68% | $0.96 | $0.55 | $2.19 | ||||||||||||||
| Qwen3 32B (Reasoning) | Alibaba | Open | 128k | 59 | 80% | 67% | 8% | 55% | 35% | 96% | 81% | $0.17 | $0.10 | $0.38 | 43.2 | 20.9 | 34.5 | 330.5 | 335.6 | 0.55 | 46.8 | 0.23 | 0.41 | 0.71 | 3.04 | 58.37 | 46.25 | ||
| QwQ-32B | Alibaba | Open | 131k | 58 | 76% | 59% | 8% | 63% | 36% | 98% | 96% | 78% | $0.47 | $0.40 | $0.55 | 106.2 | 33.6 | 83.7 | 360 | 447.9 | 0.44 | 23.89 | 0.2 | 0.34 | 0.72 | 2.92 | 28.59 | 23.45 | |
| Claude 3.7 Sonnet Thinking | Anthropic | Proprietary | 200k | 57 | 84% | 77% | 10% | 47% | 40% | 98% | 95% | 49% | $6.00 | $3.00 | $15.00 | ||||||||||||||
| Qwen3 14B (Reasoning) | Alibaba | Open | 128k | 56 | 77% | 60% | 4% | 52% | 32% | 96% | 96% | 76% | $0.12 | $0.08 | $0.24 | 76.7 | 42 | 61 | 88 | 88.8 | 0.55 | 26.62 | 0.49 | 0.51 | 0.65 | 0.91 | 33.14 | 26.07 | |
| Qwen3 30B A3B (Reasoning) | Alibaba | Open | 128k | 56 | 78% | 62% | 7% | 51% | 28% | 96% | 75% | $0.19 | $0.10 | $0.45 | 125.8 | 38.6 | 77.2 | 137.1 | 186.1 | 0.55 | 16.45 | 0.18 | 0.5 | 0.61 | 0.82 | 20.42 | 15.9 | ||
| o1-mini | OpenAI | Proprietary | 128k | 54 | 74% | 60% | 5% | 58% | 32% | 97% | 94% | 60% | $1.93 | $1.10 | $4.40 | 219.4 | 160.9 | 195.7 | 236.2 | 259.1 | 9.9 | 9.9 | 4 | 8.7 | 12.99 | 15 | 12.18 | 0 | |
| DeepSeek V3 (Mar' 25) | DeepSeek | Open | 128k | 53 | 82% | 66% | 5% | 41% | 36% | 92% | 94% | 52% | $0.48 | $0.27 | $1.10 | ||||||||||||||
| GPT-4.1 mini | OpenAI | Proprietary | 1m | 53 | 78% | 66% | 5% | 48% | 40% | 95% | 93% | 43% | $0.70 | $0.40 | $1.60 | 73.1 | 39.6 | 56.2 | 81.7 | 92 | 0.58 | 0.58 | 0.48 | 0.53 | 0.65 | 0.8 | 7.42 | 0 | |
| GPT-4.1 | OpenAI | Proprietary | 1m | 53 | 81% | 67% | 5% | 46% | 38% | 96% | 91% | 44% | $3.50 | $2.00 | $8.00 | 123.7 | 70.6 | 98 | 158 | 188.8 | 0.46 | 0.46 | 0.34 | 0.39 | 0.57 | 0.81 | 4.5 | 0 | |
| Gemini 2.0 Flash Thinking exp. (Jan '25) | Proprietary | 1m | 52 | 80% | 70% | 7% | 32% | 33% | 94% | 50% | $0.00 | $0.00 | $0.00 | ||||||||||||||||
| DeepSeek R1 Distill Qwen 32B | DeepSeek | Open | 128k | 52 | 74% | 62% | 6% | 27% | 38% | 95% | 94% | 69% | $0.22 | $0.21 | $0.24 | 45 | 12.4 | 20 | 48.6 | 50.9 | 0.27 | 44.76 | 0.18 | 0.24 | 1.22 | 6.09 | 55.88 | 44.49 | |
| Qwen3 8B (Reasoning) | Alibaba | Open | 128k | 51 | 74% | 59% | 4% | 41% | 23% | 90% | 75% | $0.06 | $0.04 | $0.14 | 90.7 | 46.8 | 85.6 | 100 | 101.2 | 0.69 | 22.74 | 0.63 | 0.68 | 0.72 | 0.96 | 28.26 | 22.05 | ||
| Llama 3.3 Nemotron Super 49B Reasoning | NVIDIA | Open | 128k | 51 | 79% | 64% | 7% | 28% | 28% | 96% | 96% | 58% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Grok 3 | xAI | Proprietary | 1m | 51 | 80% | 69% | 5% | 43% | 37% | 91% | 87% | 33% | $6.00 | $3.00 | $15.00 | 72 | 21.9 | 56.8 | 85.3 | 94 | 0.42 | 0.42 | 0.31 | 0.34 | 0.57 | 1.25 | 7.37 | 0 | |
| Llama 4 Maverick | Meta | Open | 1m | 51 | 81% | 67% | 5% | 40% | 33% | 88% | 89% | 39% | $0.35 | $0.20 | $0.85 | 137.4 | 60.3 | 107.9 | 244.7 | 788 | 0.36 | 0.36 | 0.2 | 0.29 | 0.48 | 2.05 | 3.99 | 0 | |
| GPT-4o (March 2025) | OpenAI | Proprietary | 128k | 50 | 80% | 66% | 5% | 43% | 37% | 96% | 89% | 33% | $7.50 | $5.00 | $15.00 | 145.9 | 105.1 | 135.4 | 159 | 195.4 | 0.3 | 0.3 | 0.26 | 0.27 | 0.34 | 0.36 | 3.73 | 0 | |
| Gemini 2.0 Pro Experimental | Proprietary | 2m | 49 | 81% | 62% | 7% | 35% | 31% | 95% | 92% | 36% | $0.00 | $0.00 | $0.00 | 21.4 | 1.8 | 4.9 | 33.6 | 37.5 | 17.21 | 17.21 | 16.23 | 16.76 | 17.92 | 19.49 | 40.55 | 0 | ||
| DeepSeek R1 Distill Qwen 14B | DeepSeek | Open | 128k | 49 | 74% | 48% | 4% | 38% | 24% | 93% | 95% | 67% | $0.88 | $0.88 | $0.88 | 169.7 | 44.2 | 45.6 | 170.7 | 172.6 | 0.41 | 12.19 | 0.26 | 0.28 | 0.74 | 0.92 | 15.14 | 11.78 | |
| Mistral Medium 3 | Mistral | Proprietary | 128k | 49 | 76% | 58% | 4% | 40% | 33% | 90% | 91% | 44% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Gemini 2.5 Flash | Proprietary | 1m | 49 | 78% | 59% | 5% | 41% | 23% | 93% | 43% | $0.26 | $0.15 | $0.60 | 274.2 | 234.3 | 257.9 | 291.5 | 308.7 | 0.4 | 0.4 | 0.23 | 0.3 | 0.42 | 0.45 | 2.22 | 0 | |||
| DeepSeek R1 Distill Llama 70B | DeepSeek | Open | 128k | 48 | 80% | 40% | 6% | 27% | 31% | 97% | 94% | 67% | $0.60 | $0.54 | $0.87 | 120 | 30.5 | 59.6 | 353.4 | 2 | 559 | 0.54 | 17.2 | 0.19 | 0.3 | 0.94 | 2.72 | 21.36 | 16.66 |
| Claude 3.7 Sonnet | Anthropic | Proprietary | 200k | 48 | 80% | 66% | 5% | 39% | 38% | 95% | 85% | 22% | $6.00 | $3.00 | $15.00 | 77.1 | 66.8 | 73.2 | 78.7 | 82.2 | 1.3 | 1.3 | 0.57 | 0.93 | 1.72 | 3.74 | 7.79 | 0 | |
| Gemini 2.0 Flash | Proprietary | 1m | 48 | 78% | 62% | 5% | 33% | 31% | 90% | 93% | 33% | $0.17 | $0.10 | $0.40 | 235.4 | 176.9 | 221.4 | 243.8 | 260.2 | 0.38 | 0.38 | 0.26 | 0.33 | 0.41 | 0.46 | 2.5 | 0 | ||
| Qwen3 4B (Reasoning) | Alibaba | Open | 32k | 47 | 70% | 52% | 5% | 47% | 4% | 91% | 93% | 66% | $0.12 | $0.08 | $0.24 | 157.1 | 69 | 146.3 | 158.1 | 158.6 | 0.53 | 13.27 | 0.47 | 0.49 | 0.62 | 0.69 | 16.45 | 12.73 | |
| Reka Flash 3 | Reka AI | Open | 128k | 47 | 67% | 53% | 5% | 44% | 27% | 95% | 89% | 51% | $0.35 | $0.20 | $0.80 | 56.4 | 55 | 55.9 | 57.5 | 58 | 0.96 | 36.42 | 0.86 | 0.9 | 0.98 | 1.06 | 45.29 | 35.46 | |
| Qwen3 235B A22B | Alibaba | Open | 128k | 47 | 76% | 61% | 5% | 34% | 30% | 90% | 33% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| Gemini 2.0 Flash (exp) | Proprietary | 1m | 46 | 78% | 64% | 5% | 21% | 34% | 91% | 91% | 30% | $0.00 | $0.00 | $0.00 | 237.6 | 207.8 | 226.7 | 245.4 | 259.9 | 0.26 | 0.26 | 0.21 | 0.24 | 0.28 | 0.34 | 2.37 | 0 | ||
| DeepSeek V3 (Dec '24) | DeepSeek | Open | 128k | 46 | 75% | 56% | 4% | 36% | 35% | 91% | 89% | 25% | $0.48 | $0.27 | $1.10 | ||||||||||||||
| Qwen2.5 Max | Alibaba | Proprietary | 32k | 45 | 76% | 59% | 5% | 36% | 34% | 93% | 84% | 23% | $2.80 | $1.60 | $6.40 | 49.1 | 39.8 | 45.3 | 50.6 | 53 | 1.26 | 1.26 | 1.16 | 1.2 | 1.43 | 1.58 | 11.44 | 0 | |
| Gemini 1.5 Pro (Sep) | Proprietary | 2m | 45 | 75% | 59% | 5% | 32% | 30% | 90% | 88% | 23% | $2.19 | $1.25 | $5.00 | 91.8 | 86.4 | 87.6 | 94.9 | 102.3 | 0.43 | 0.43 | 0.39 | 0.41 | 0.45 | 1.09 | 5.87 | 0 | ||
| Claude 3.5 Sonnet (Oct) | Anthropic | Proprietary | 200k | 44 | 77% | 60% | 4% | 38% | 37% | 93% | 77% | 16% | $6.00 | $3.00 | $15.00 | 74.7 | 66.5 | 70.8 | 78.3 | 82.3 | 1.61 | 1.61 | 0.59 | 1.08 | 2.71 | 5.63 | 8.31 | 0 | |
| Qwen3 32B | Alibaba | Open | 128k | 44 | 73% | 54% | 4% | 29% | 28% | 90% | 87% | 30% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Sonar | Perplexity | Proprietary | 127k | 43 | 69% | 47% | 7% | 30% | 23% | 82% | 82% | 49% | $1.00 | $1.00 | $1.00 | 94.6 | 36.3 | 79.9 | 138.4 | 161.6 | 1.75 | 1.75 | 1.42 | 1.66 | 1.87 | 2.25 | 7.04 | 0 | |
| Llama 4 Scout | Meta | Open | 10m | 43 | 75% | 59% | 4% | 30% | 17% | 83% | 84% | 28% | $0.27 | $0.16 | $0.54 | 126.9 | 34.7 | 87.2 | 166.8 | 1 | 974.7 | 0.4 | 0.4 | 0.2 | 0.3 | 0.66 | 2.23 | 4.34 | 0 |
| Sonar Pro | Perplexity | Proprietary | 200k | 43 | 76% | 58% | 8% | 28% | 23% | 85% | 75% | 29% | $6.00 | $3.00 | $15.00 | 81.1 | 58.1 | 67.9 | 131.8 | 165.7 | 2.41 | 2.41 | 1.7 | 1.85 | 3.02 | 4.49 | 8.58 | 0 | |
| QwQ 32B-Preview | Alibaba | Open | 33k | 43 | 65% | 56% | 5% | 34% | 4% | 87% | 91% | 45% | $0.23 | $0.17 | $0.43 | 63.3 | 33.1 | 45.1 | 93.2 | 98.8 | 0.4 | 32 | 0.19 | 0.28 | 0.52 | 0.62 | 39.9 | 31.6 | |
| Nova Premier | Amazon | Proprietary | 1m | 43 | 73% | 57% | 5% | 32% | 28% | 91% | 84% | 17% | $5.00 | $2.50 | $12.50 | 63.9 | 60.4 | 62.8 | 66.2 | 72.8 | 0.79 | 0.79 | 0.72 | 0.76 | 0.85 | 0.87 | 8.62 | 0 | |
| Qwen3 30B A3B | Alibaba | Open | 128k | 43 | 71% | 52% | 5% | 32% | 26% | 86% | 26% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| GPT-4o (Nov '24) | OpenAI | Proprietary | 128k | 41 | 75% | 54% | 3% | 31% | 33% | 93% | 76% | 15% | $4.38 | $2.50 | $10.00 | 136.7 | 104.8 | 120.3 | 147.8 | 159.3 | 0.54 | 0.54 | 0.46 | 0.52 | 0.56 | 0.62 | 4.2 | 0 | |
| Gemini 2.0 Flash-Lite (Feb '25) | Proprietary | 1m | 41 | 72% | 54% | 4% | 19% | 25% | 88% | 87% | 28% | $0.13 | $0.07 | $0.30 | 201.6 | 182.1 | 194.7 | 209 | 216.6 | 0.28 | 0.28 | 0.22 | 0.26 | 0.3 | 0.31 | 2.76 | 0 | ||
| Llama 3.3 70B | Meta | Open | 128k | 41 | 71% | 50% | 4% | 29% | 26% | 86% | 77% | 30% | $0.60 | $0.59 | $0.70 | 112.2 | 25.5 | 41.8 | 165.8 | 1 | 903 | 0.45 | 0.45 | 0.23 | 0.34 | 0.62 | 2.73 | 4.91 | 0 |
| GPT-4.1 nano | OpenAI | Proprietary | 1m | 41 | 66% | 51% | 4% | 33% | 26% | 88% | 85% | 24% | $0.17 | $0.10 | $0.40 | 223.2 | 181.5 | 192.7 | 238.6 | 255.6 | 0.54 | 0.54 | 0.31 | 0.47 | 0.82 | 0.85 | 2.78 | 0 | |
| Qwen3 14B | Alibaba | Open | 128k | 41 | 68% | 47% | 4% | 28% | 27% | 87% | 28% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| GPT-4o (May '24) | OpenAI | Proprietary | 128k | 41 | 74% | 53% | 3% | 33% | 31% | 94% | 79% | 11% | $7.50 | $5.00 | $15.00 | 108.5 | 52.4 | 79.4 | 174.2 | 207 | 0.37 | 0.37 | 0.29 | 0.35 | 0.43 | 0.66 | 4.98 | 0 | |
| Llama 3.1 405B | Meta | Open | 128k | 40 | 73% | 52% | 4% | 31% | 30% | 85% | 70% | 21% | $3.25 | $3.25 | $3.25 | 37 | 26.3 | 31.5 | 89.2 | 165.7 | 0.67 | 0.67 | 0.38 | 0.43 | 1.16 | 2.38 | 14.19 | 0 | |
| Qwen2.5 72B | Alibaba | Open | 131k | 40 | 72% | 49% | 4% | 28% | 27% | 88% | 86% | 16% | $0.00 | $0.00 | $0.00 | 49.3 | 42.7 | 47.5 | 53.2 | 54.6 | 1.05 | 1.05 | 0.97 | 0.99 | 1.14 | 1.29 | 11.19 | 0 | |
| MiniMax-Text-01 | MiniMax | Open | 4m | 40 | 76% | 58% | 4% | 25% | 25% | 86% | 75% | 13% | $0.42 | $0.20 | $1.10 | 33.8 | 26.9 | 30.6 | 38.9 | 46.5 | 0.79 | 0.79 | 0.63 | 0.68 | 1 | 1.63 | 15.58 | 0 | |
| Phi-4 | Microsoft Azure | Open | 16k | 40 | 71% | 57% | 4% | 23% | 26% | 87% | 81% | 14% | $0.22 | $0.13 | $0.50 | 33.5 | 12.7 | 24 | 44.6 | 45.5 | 0.46 | 0.46 | 0.36 | 0.43 | 0.52 | 1.35 | 15.37 | 0 | |
| Command A | Cohere | Open | 256k | 40 | 71% | 53% | 5% | 29% | 28% | 82% | 82% | 10% | $4.38 | $2.50 | $10.00 | 104.9 | 100.4 | 103.9 | 108.8 | 111.9 | 0.2 | 0.2 | 0.19 | 0.19 | 0.21 | 0.23 | 4.97 | 0 | |
| Tulu3 405B | Allen Institute for AI | Open | 128k | 40 | 72% | 52% | 4% | 29% | 30% | 89% | 78% | 13% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Llama 3.3 Nemotron Super 49B v1 | NVIDIA | Open | 128k | 39 | 70% | 52% | 4% | 28% | 23% | 83% | 78% | 19% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Grok 2 | xAI | Proprietary | 131k | 39 | 71% | 51% | 4% | 27% | 28% | 86% | 78% | 13% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Gemini 1.5 Flash (Sep) | Proprietary | 1m | 39 | 68% | 46% | 4% | 27% | 27% | 84% | 83% | 18% | $0.13 | $0.07 | $0.30 | 191.1 | 179.9 | 187.3 | 195.2 | 201.1 | 0.19 | 0.19 | 0.16 | 0.18 | 0.2 | 0.23 | 2.81 | 0 | ||
| Mistral Large 2 (Nov '24) | Mistral | Open | 128k | 38 | 70% | 49% | 4% | 29% | 29% | 90% | 74% | 11% | $3.00 | $2.00 | $6.00 | 67.3 | 22.9 | 46.9 | 79.8 | 87.9 | 0.45 | 0.45 | 0.36 | 0.39 | 1.46 | 15.81 | 7.88 | 0 | |
| Qwen3 1.7B (Reasoning) | Alibaba | Open | 32k | 38 | 57% | 36% | 5% | 31% | 4% | 85% | 89% | 51% | $0.00 | $0.00 | $0.00 | 48.9 | 44.5 | 45.6 | 137.9 | 167.4 | 0.63 | 41.5 | 0.56 | 0.61 | 0.69 | 0.73 | 51.72 | 40.88 | |
| Gemma 3 27B | Open | 128k | 38 | 67% | 43% | 5% | 14% | 21% | 89% | 88% | 25% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| Grok Beta | xAI | Proprietary | 128k | 38 | 70% | 47% | 5% | 24% | 30% | 87% | 74% | 10% | $7.50 | $5.00 | $15.00 | 66.5 | 60.1 | 64.5 | 67.5 | 68.6 | 0.3 | 0.3 | 0.24 | 0.28 | 0.35 | 0.39 | 7.82 | 0 | |
| Pixtral Large | Mistral | Open | 128k | 37 | 70% | 51% | 4% | 26% | 29% | 85% | 71% | 7% | $3.00 | $2.00 | $6.00 | 41.2 | 28.1 | 35.9 | 42 | 42.9 | 0.38 | 0.38 | 0.34 | 0.36 | 0.63 | 91.77 | 12.53 | 0 | |
| Qwen2.5 Instruct 32B | Alibaba | Open | 128k | 37 | 70% | 47% | 4% | 25% | 23% | 90% | 81% | 11% | $0.15 | $0.10 | $0.30 | ||||||||||||||
| Llama 3.1 Nemotron 70B | NVIDIA | Open | 128k | 37 | 69% | 47% | 5% | 17% | 23% | 82% | 73% | 25% | $0.24 | $0.18 | $0.40 | 48.1 | 21.4 | 39.1 | 63.4 | 72.3 | 0.58 | 0.58 | 0.28 | 0.35 | 0.67 | 2.57 | 10.98 | 0 | |
| Nova Pro | Amazon | Proprietary | 300k | 37 | 69% | 50% | 3% | 23% | 21% | 83% | 79% | 11% | $1.40 | $0.80 | $3.20 | ||||||||||||||
| Qwen3 8B | Alibaba | Open | 128k | 37 | 64% | 45% | 3% | 20% | 17% | 83% | 24% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| Mistral Large 2 (Jul '24) | Mistral | Open | 128k | 37 | 68% | 47% | 3% | 27% | 27% | 89% | 71% | 9% | $3.00 | $2.00 | $6.00 | 39.2 | 16 | 36.7 | 40.9 | 41.9 | 0.45 | 0.45 | 0.4 | 0.41 | 0.56 | 0.87 | 13.2 | 0 | |
| Qwen2.5 Coder 32B | Alibaba | Open | 131k | 36 | 64% | 42% | 4% | 30% | 27% | 90% | 77% | 12% | $0.15 | $0.14 | $0.19 | 52.7 | 41.9 | 46.9 | 79.7 | 88.8 | 0.54 | 0.54 | 0.23 | 0.32 | 0.97 | 2.99 | 10.03 | 0 | |
| GPT-4o mini | OpenAI | Proprietary | 128k | 36 | 65% | 43% | 4% | 23% | 23% | 88% | 79% | 12% | $0.26 | $0.15 | $0.60 | 71.3 | 34.5 | 49.7 | 75.6 | 93.8 | 0.63 | 0.63 | 0.32 | 0.48 | 0.65 | 0.83 | 7.64 | 0 | |
| Llama 3.1 70B | Meta | Open | 128k | 35 | 68% | 41% | 5% | 23% | 27% | 81% | 65% | 17% | $0.56 | $0.56 | $0.73 | 64.9 | 30.2 | 41.3 | 132.4 | 171.6 | 0.51 | 0.51 | 0.24 | 0.32 | 0.67 | 1.92 | 8.22 | 0 | |
| Mistral Small 3.1 | Mistral | Open | 128k | 35 | 66% | 45% | 5% | 21% | 27% | 86% | 71% | 9% | $0.15 | $0.10 | $0.30 | 151.7 | 72.7 | 135.6 | 166.6 | 172.7 | 0.27 | 0.27 | 0.25 | 0.25 | 0.29 | 0.39 | 3.57 | 0 | |
| Mistral Small 3 | Mistral | Open | 32k | 35 | 65% | 46% | 4% | 25% | 24% | 85% | 72% | 8% | $0.15 | $0.10 | $0.30 | 146.8 | 112.7 | 136.7 | 157.2 | 166.6 | 0.27 | 0.27 | 0.26 | 0.26 | 0.3 | 0.35 | 3.67 | 0 | |
| Qwen3 4B | Alibaba | Open | 32k | 35 | 59% | 40% | 4% | 23% | 17% | 84% | 21% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| Claude 3 Opus | Anthropic | Proprietary | 200k | 35 | 70% | 49% | 3% | 28% | 23% | 85% | 64% | 3% | $30.00 | $15.00 | $75.00 | 28 | 24.4 | 26 | 29.6 | 30.9 | 1.13 | 1.13 | 0.99 | 1.05 | 1.38 | 1.48 | 18.96 | 0 | |
| Claude 3.5 Haiku | Anthropic | Proprietary | 200k | 35 | 63% | 41% | 4% | 31% | 27% | 86% | 72% | 3% | $1.60 | $0.80 | $4.00 | 66.1 | 57 | 62 | 67.7 | 68.6 | 0.9 | 0.9 | 0.57 | 0.64 | 1.21 | 2.24 | 8.47 | 0 | |
| DeepSeek R1 Distill Llama 8B | DeepSeek | Open | 128k | 34 | 54% | 30% | 4% | 23% | 12% | 84% | 85% | 33% | $0.04 | $0.04 | $0.04 | 43.5 | 34.5 | 37 | 53.6 | 55.9 | 0.67 | 46.65 | 0.66 | 0.66 | 74.76 | 378.86 | 58.14 | 45.98 | |
| Gemma 3 12B | Open | 128k | 34 | 60% | 35% | 5% | 14% | 17% | 83% | 85% | 22% | $0.06 | $0.05 | $0.10 | 23.8 | 13.2 | 18.6 | 35.6 | 50.7 | 0.61 | 0.61 | 0.33 | 0.56 | 0.79 | 1.03 | 21.6 | 0 | ||
| Gemini 1.5 Pro (May) | Proprietary | 2m | 34 | 66% | 37% | 4% | 24% | 27% | 83% | 67% | 8% | $2.19 | $1.25 | $5.00 | 66.8 | 63.8 | 65.4 | 67.7 | 69.3 | 0.35 | 0.35 | 0.34 | 0.35 | 0.39 | 11.29 | 7.84 | 0 | ||
| Qwen Turbo | Alibaba | Proprietary | 1m | 34 | 63% | 41% | 4% | 16% | 15% | 85% | 81% | 12% | $0.09 | $0.05 | $0.20 | 106.4 | 94.4 | 103 | 109.8 | 112.3 | 1.01 | 1.01 | 0.96 | 0.97 | 1.09 | 1.3 | 5.71 | 0 | |
| Llama 3.2 90B (Vision) | Meta | Open | 128k | 33 | 67% | 43% | 5% | 21% | 24% | 82% | 63% | 5% | $0.72 | $0.72 | $0.72 | 36.5 | 23.4 | 32.1 | 51.1 | 60.9 | 0.27 | 0.27 | 0.18 | 0.21 | 0.52 | 1.06 | 13.96 | 0 | |
| Qwen2 72B | Alibaba | Open | 131k | 33 | 62% | 37% | 4% | 16% | 23% | 83% | 70% | 15% | $0.00 | $0.00 | $0.00 | 31 | 30.8 | 30.9 | 31.1 | 31.2 | 1.3 | 1.3 | 1.24 | 1.26 | 1.46 | 1.59 | 17.43 | 0 | |
| Nova Lite | Amazon | Proprietary | 300k | 33 | 59% | 43% | 5% | 17% | 14% | 84% | 77% | 11% | $0.10 | $0.06 | $0.24 | 287 | 265 | 274 | 302.5 | 319.2 | 0.32 | 0.32 | 0.3 | 0.3 | 0.33 | 0.35 | 2.06 | 0 | |
| Gemini 1.5 Flash-8B | Proprietary | 1m | 31 | 57% | 36% | 5% | 22% | 23% | 12% | 69% | 3% | $0.07 | $0.04 | $0.15 | 287.2 | 277.8 | 282.6 | 293.3 | 299.9 | 0.19 | 0.19 | 0.16 | 0.17 | 0.2 | 0.22 | 1.93 | 0 | ||
| DeepHermes 3 - Mistral 24B | Nous Research | Open | 32k | 30 | 58% | 38% | 4% | 20% | 23% | 75% | 60% | 5% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Jamba 1.5 Large | AI21 Labs | Open | 256k | 29 | 57% | 43% | 4% | 14% | 16% | 24% | 61% | 5% | $3.50 | $2.00 | $8.00 | ||||||||||||||
| Hermes 3 - Llama-3.1 70B | Nous Research | Open | 128k | 29 | 57% | 40% | 4% | 19% | 23% | 75% | 54% | 2% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Jamba 1.6 Large | AI21 Labs | Open | 256k | 29 | 56% | 39% | 4% | 17% | 18% | 70% | 58% | 5% | $3.50 | $2.00 | $8.00 | 57.9 | 11.2 | 45.7 | 63.4 | 67.5 | 0.56 | 0.56 | 0.35 | 0.45 | 0.79 | 1.23 | 9.2 | 0 | |
| Gemini 1.5 Flash (May) | Proprietary | 1m | 28 | 57% | 32% | 4% | 20% | 18% | 72% | 55% | 9% | $0.13 | $0.07 | $0.30 | 315.1 | 294.9 | 306.8 | 330.8 | 356.6 | 0.26 | 0.26 | 0.24 | 0.25 | 0.27 | 0.28 | 1.85 | 0 | ||
| Nova Micro | Amazon | Proprietary | 130k | 28 | 53% | 36% | 5% | 14% | 9% | 80% | 70% | 8% | $0.06 | $0.04 | $0.14 | 322.6 | 284.5 | 307 | 334.5 | 356.1 | 0.3 | 0.3 | 0.28 | 0.29 | 0.32 | 0.33 | 1.85 | 0 | |
| Yi-Large | 01.AI | Proprietary | 32k | 28 | 59% | 36% | 3% | 11% | 19% | 74% | 56% | 7% | $3.00 | $3.00 | $3.00 | 66.2 | 62.7 | 64.9 | 69.7 | 80.9 | 0.4 | 0.4 | 0.33 | 0.35 | 0.42 | 2.31 | 7.95 | 0 | |
| Claude 3 Sonnet | Anthropic | Proprietary | 200k | 28 | 58% | 40% | 4% | 18% | 23% | 71% | 41% | 5% | $6.00 | $3.00 | $15.00 | 60.5 | 52.4 | 58.6 | 62.4 | 63.5 | 0.81 | 0.81 | 0.54 | 0.6 | 1.16 | 1.64 | 9.08 | 0 | |
| Codestral (Jan '25) | Mistral | Proprietary | 256k | 28 | 45% | 31% | 5% | 24% | 25% | 85% | 61% | 4% | $0.45 | $0.30 | $0.90 | 108.6 | 67.3 | 96.9 | 120 | 137.4 | 0.28 | 0.28 | 0.26 | 0.27 | 0.32 | 0.48 | 4.89 | 0 | |
| Llama 3 70B | Meta | Open | 8k | 27 | 57% | 38% | 4% | 20% | 19% | 79% | 48% | 0% | $0.84 | $0.73 | $0.84 | 47.4 | 17.9 | 19.5 | 151.8 | 335.9 | 0.45 | 0.45 | 0.22 | 0.36 | 0.88 | 1.75 | 10.99 | 0 | |
| Mistral Small (Sep '24) | Mistral | Open | 33k | 27 | 53% | 38% | 4% | 14% | 16% | 81% | 56% | 6% | $0.30 | $0.20 | $0.60 | 85.5 | 70.7 | 78.4 | 91.4 | 100.3 | 0.28 | 0.28 | 0.27 | 0.27 | 0.29 | 0.49 | 6.13 | 0 | |
| Phi-4 Multimodal | Microsoft Azure | Open | 128k | 27 | 49% | 32% | 4% | 13% | 11% | 73% | 69% | 9% | $0.00 | $0.00 | $0.00 | 21 | 14.5 | 19.3 | 21.7 | 23.7 | 0.35 | 0.35 | 0.3 | 0.32 | 0.36 | 0.38 | 24.15 | 0 | |
| Qwen2.5 Coder 7B | Alibaba | Open | 131k | 27 | 47% | 34% | 5% | 13% | 15% | 90% | 66% | 5% | $0.03 | $0.02 | $0.06 | 199.9 | 133.6 | 191.5 | 210.3 | 212.5 | 0.48 | 0.48 | 0.46 | 0.47 | 0.53 | 0.65 | 2.98 | 0 | |
| Mistral Large (Feb '24) | Mistral | Proprietary | 33k | 26 | 52% | 35% | 3% | 18% | 21% | 71% | 53% | 0% | $6.00 | $4.00 | $12.00 | 30.7 | 29.4 | 30 | 31.1 | 31.8 | 0.45 | 0.45 | 0.43 | 0.44 | 0.46 | 0.56 | 16.73 | 0 | |
| Mixtral 8x22B | Mistral | Open | 65k | 26 | 54% | 33% | 4% | 15% | 19% | 72% | 55% | 0% | $3.00 | $2.00 | $6.00 | 57.1 | 42.7 | 48.2 | 69.2 | 86.4 | 0.33 | 0.33 | 0.29 | 0.31 | 0.36 | 0.47 | 9.08 | 0 | |
| Phi-4 Mini | Microsoft Azure | Open | 128k | 26 | 47% | 33% | 4% | 13% | 11% | 74% | 70% | 3% | $0.00 | $0.00 | $0.00 | 58.5 | 53 | 56.1 | 59.7 | 61.2 | 0.33 | 0.33 | 0.28 | 0.31 | 0.35 | 0.37 | 8.88 | 0 | |
| Qwen3 1.7B | Alibaba | Open | 32k | 25 | 41% | 28% | 5% | 13% | 7% | 72% | 10% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| Phi-3 Medium 14B | Microsoft Azure | Open | 128k | 25 | 54% | 33% | 5% | 15% | 12% | 0% | 46% | 1% | $0.30 | $0.17 | $0.68 | 53 | 50.4 | 52.1 | 53.6 | 54.2 | 0.41 | 0.41 | 0.38 | 0.4 | 0.44 | 0.46 | 9.84 | 0 | |
| Gemma 3 4B | Open | 128k | 24 | 42% | 29% | 5% | 7% | 6% | 72% | 77% | 5% | $0.03 | $0.02 | $0.04 | 124.1 | 81.2 | 108.7 | 146 | 159.8 | 0.22 | 0.22 | 0.15 | 0.17 | 0.24 | 0.26 | 4.24 | 0 | ||
| Claude 2.1 | Anthropic | Proprietary | 200k | 24 | 50% | 32% | 4% | 20% | 18% | 16% | 37% | 3% | $12.00 | $8.00 | $24.00 | 14 | 13.5 | 13.7 | 14.2 | 14.4 | 0.91 | 0.91 | 0.82 | 0.88 | 1 | 1.35 | 36.6 | 0 | |
| Llama 3.1 8B | Meta | Open | 128k | 24 | 48% | 26% | 5% | 12% | 13% | 67% | 52% | 8% | $0.10 | $0.10 | $0.10 | 222.2 | 46.1 | 93.6 | 457.9 | 1 | 792 | 0.32 | 0.32 | 0.16 | 0.23 | 0.51 | 1.04 | 2.57 | 0 |
| Pixtral 12B | Mistral | Open | 128k | 23 | 47% | 34% | 5% | 12% | 14% | 78% | 46% | 0% | $0.15 | $0.15 | $0.15 | 100.5 | 91.6 | 98.1 | 103.6 | 105.2 | 0.29 | 0.29 | 0.27 | 0.29 | 0.31 | 0.56 | 5.26 | 0 | |
| Qwen3 0.6B (Reasoning) | Alibaba | Open | 32k | 23 | 35% | 24% | 6% | 12% | 3% | 49% | 75% | 10% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Mistral Small (Feb '24) | Mistral | Proprietary | 33k | 23 | 42% | 30% | 4% | 11% | 13% | 79% | 56% | 1% | $1.50 | $1.00 | $3.00 | 138.5 | 75.9 | 107.8 | 163.8 | 176.1 | 0.27 | 0.27 | 0.25 | 0.25 | 0.3 | 0.76 | 3.88 | 0 | |
| Mistral Medium | Mistral | Proprietary | 33k | 23 | 49% | 35% | 3% | 10% | 12% | 41% | 4% | $4.09 | $2.75 | $8.10 | 46.4 | 26.4 | 38.3 | 58.7 | 89.6 | 0.42 | 0.42 | 0.34 | 0.36 | 0.53 | 1.06 | 11.2 | 0 | ||
| Ministral 8B | Mistral | Open | 128k | 22 | 39% | 28% | 5% | 11 | 12 | 77% | 57% | 4% | $0.10 | $0.10 | $0.10 | 134.8 | 119.8 | 126.3 | 137.2 | 139.2 | 0.27 | 0.27 | 0.26 | 0.26 | 0.28 | 0.36 | 3.98 | 0 | |
| Gemma 2 9B | Open | 8k | 22 | 50% | 31% | 4% | 13 | 1 | 65% | 52% | 0% | $0.12 | $0.12 | $0.15 | |||||||||||||||
| Phi-3 Mini | Microsoft Azure | Open | 4k | 22 | 44% | 32% | 4% | 12 | 9 | 25% | 46% | 4% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| LFM 40B | Liquid AI | Proprietary | 32k | 22 | 43% | 33% | 5% | 10 | 7 | 51% | 48% | 2% | $0.15 | $0.15 | $0.15 | 169.1 | 153.8 | 164 | 173.8 | 200.2 | 0.21 | 0.21 | 0.14 | 0.16 | 0.95 | 0.99 | 3.17 | 0 | |
| Command-R+ | Cohere | Open | 128k | 21 | 43% | 34% | 5% | 11 | 12 | 63% | 40% | 0% | $4.38 | $2.50 | $10.00 | 48.2 | 43.1 | 47.2 | 49.4 | 50.8 | 0.26 | 0.26 | 0.24 | 0.25 | 0.27 | 0.3 | 10.64 | 0 | |
| Llama 3 8B | Meta | Open | 8k | 21 | 41% | 30% | 5% | 10 | 12 | 71% | 50% | 0% | $0.09 | $0.06 | $0.14 | 104.7 | 59.6 | 74.1 | 187.4 | 1 | 353.7 | 0.34 | 0.34 | 0.21 | 0.29 | 0.46 | 0.97 | 5.12 | 0 |
| Gemini 1.0 Pro | Proprietary | 33k | 21 | 43% | 28% | 5% | 12 | 12 | 2% | 40% | 1% | $0.75 | $0.50 | $1.50 | |||||||||||||||
| Codestral (May '24) | Mistral | Open | 33k | 20 | 33% | 26% | 5% | 21 | 22 | 80% | 35% | 0% | $0.30 | $0.20 | $0.60 | 105.2 | 93.8 | 102.2 | 108.5 | 110.1 | 0.3 | 0.3 | 0.28 | 0.29 | 0.32 | 0.44 | 5.05 | 0 | |
| Aya Expanse 32B | Cohere | Open | 128k | 20 | 38% | 23% | 5% | 14 | 15 | 68% | 45% | 0% | $0.75 | $0.50 | $1.50 | 118.3 | 114.8 | 116.3 | 123.6 | 127.2 | 0.16 | 0.16 | 0.14 | 0.15 | 0.17 | 0.21 | 4.39 | 0 | |
| Llama 2 Chat 13B | Meta | Open | 4k | 20 | 41% | 32% | 5% | 10 | 12 | 33% | 2% | $0.00 | $0.00 | $0.00 | |||||||||||||||
| Command-R+ (Apr '24) | Cohere | Open | 128k | 20 | 43% | 32% | 5% | 12 | 12 | 64% | 28% | 1% | $6.00 | $3.00 | $15.00 | 72 | 62.7 | 67.3 | 75.3 | 77.5 | 0.21 | 0.21 | 0.19 | 0.21 | 0.23 | 0.27 | 7.15 | 0 | |
| DBRX | Databricks | Open | 33k | 20 | 40% | 33% | 7% | 9 | 12 | 67% | 28% | 3% | $0.00 | $0.00 | $0.00 | ||||||||||||||
| Ministral 3B | Mistral | Proprietary | 128k | 20 | 34% | 26% | 6% | 7 | 9 | 74% | 54% | 0% | $0.04 | $0.04 | $0.04 | 225.6 | 159.8 | 220.6 | 229.7 | 232.1 | 0.26 | 0.26 | 0.24 | 0.25 | 0.28 | 0.47 | 2.48 | 0 | |
| Mistral NeMo | Mistral | Open | 128k | 20 | 40% | 31% | 4% | 6 | 10 | 65% | 40% | 0% | $0.15 | $0.15 | $0.15 | 140.8 | 91.1 | 133 | 147.7 | 149.9 | 0.29 | 0.29 | 0.27 | 0.27 | 0.3 | 0.34 | 3.84 | 0 | |
| Llama 3.2 3B | Meta | Open | 128k | 20 | 35% | 26% | 5% | 8 | 5 | 56% | 49% | 7% | $0.04 | $0.03 | $0.05 | 107.4 | 70 | 84.8 | 222.8 | 1 | 591.4 | 0.46 | 0.46 | 0.19 | 0.23 | 0.61 | 1.16 | 5.12 | 0 |
Models compared: OpenAI: GPT 4o Audio, GPT 4o Realtime, GPT 4o Speech Pipeline, GPT-3.5 Turbo, GPT-3.5 Turbo (0125), GPT-3.5 Turbo (0301), GPT-3.5 Turbo (0613), GPT-3.5 Turbo (1106), GPT-3.5 Turbo Instruct, GPT-4, GPT-4 Turbo, GPT-4 Turbo (0125), GPT-4 Turbo (1106), GPT-4 Vision, GPT-4.1, GPT-4.1 mini, GPT-4.1 nano, GPT-4.5 (Preview), GPT-4o (April 2025), GPT-4o (Aug ’24), GPT-4o (ChatGPT), GPT-4o (March 2025), GPT-4o (May ’24), GPT-4o (Nov ’24), GPT-4o Realtime (Dec ’24), GPT-4o mini, GPT-4o mini Realtime (Dec ’24), o1, o1-mini, o1-preview, o1-pro, o3, o3-mini, o3-mini (high), and o4-mini (high), Meta: Code Llama 70B, Llama 2 Chat 13B, Llama 2 Chat 70B, Llama 2 Chat 7B, Llama 3 70B, Llama 3 8B, Llama 3.1 405B, Llama 3.1 70B, Llama 3.1 8B, Llama 3.2 11B (Vision), Llama 3.2 1B, Llama 3.2 3B, Llama 3.2 90B (Vision), Llama 3.3 70B, Llama 4 Behemoth, Llama 4 Maverick, Llama 4 Scout, and Llama 65B, Google: Gemini 1.0 Pro, Gemini 1.0 Ultra, Gemini 1.5 Flash (May), Gemini 1.5 Flash (Sep), Gemini 1.5 Flash-8B, Gemini 1.5 Pro (May), Gemini 1.5 Pro (Sep), Gemini 2.0 Flash, Gemini 2.0 Flash (exp), Gemini 2.0 Flash Thinking exp. (Dec ’24), Gemini 2.0 Flash Thinking exp. (Jan ’25), Gemini 2.0 Flash-Lite (Feb ’25), Gemini 2.0 Flash-Lite (Preview), Gemini 2.0 Pro Experimental, Gemini 2.5 Flash, Gemini 2.5 Flash (May ’25), Gemini 2.5 Flash (May ’25) (Reasoning), Gemini 2.5 Flash (April ’25) (Reasoning), Gemini 2.5 Pro, Gemini 2.5 Pro Preview (May’ 25), Gemini Experimental (Nov), Gemma 2 27B, Gemma 2 9B, Gemma 3 12B, Gemma 3 1B, Gemma 3 27B, Gemma 3 4B, Gemma 3n E4B, Gemma 7B, and PALM-2, Anthropic: Claude 2.0, Claude 2.1, Claude 3 Haiku, Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Haiku, Claude 3.5 Sonnet (June), Claude 3.5 Sonnet (Oct), Claude 3.7 Sonnet Thinking, Claude 3.7 Sonnet, Claude 4 Opus, Claude 4 Opus Thinking, Claude 4 Sonnet, Claude 4 Sonnet Thinking, and Claude Instant, Mistral: Codestral (Jan ’25), Codestral (May ’24), Codestral-Mamba, Devstral, Ministral 3B, Ministral 8B, Mistral 7B, Mistral Large (Feb ’24), Mistral Large 2 (Jul ’24), Mistral Large 2 (Nov ’24), Mistral Medium, Mistral Medium 3, Mistral NeMo, Mistral Saba, Mistral Small (Feb ’24), Mistral Small (Sep ’24), Mistral Small 3, Mistral Small 3.1, Mixtral 8x22B, Mixtral 8x7B, Pixtral 12B, and Pixtral Large, DeepSeek: DeepSeek Coder V2 Lite, DeepSeek LLM 67B (V1), DeepSeek Prover V2 671B, DeepSeek R1, DeepSeek R1 (FP4), DeepSeek R1 Distill Llama 70B, DeepSeek R1 Distill Llama 8B, DeepSeek R1 Distill Qwen 1.5B, DeepSeek R1 Distill Qwen 14B, DeepSeek R1 Distill Qwen 32B, DeepSeek V3 (Dec ’24), DeepSeek V3 (Mar’ 25), DeepSeek-Coder-V2, DeepSeek-V2, DeepSeek-V2.5, DeepSeek-V2.5 (Dec ’24), DeepSeek-VL2, and Janus Pro 7B, Perplexity: PPLX-70B Online, PPLX-7B-Online, R1 1776, Sonar, Sonar 3.1 Huge, Sonar 3.1 Large, Sonar 3.1 Small , Sonar Large, Sonar Pro, Sonar Reasoning, Sonar Reasoning Pro, and Sonar Small, xAI: Grok 2, Grok 3, Grok 3 Reasoning Beta, Grok 3 mini, Grok 3 mini Reasoning (low), Grok 3 mini Reasoning (high), Grok Beta, and Grok-1, OpenChat: OpenChat 3.5, Amazon: Nova Lite, Nova Micro, Nova Premier, and Nova Pro, Microsoft Azure: Phi-3 Medium 14B, Phi-3 Mini, Phi-4, Phi-4 Mini, Phi-4 Multimodal, Phi-4 mini reasoning, Phi-4 reasoning, and Phi-4 reasoning plus, Liquid AI: LFM 1.3B, LFM 3B, and LFM 40B, Upstage: Solar Mini, Solar Pro, and Solar Pro (Nov ’24), Databricks: DBRX, MiniMax: MiniMax-Text-01, NVIDIA: Cosmos Nemotron 34B, Llama 3.1 Nemotron 70B, Llama 3.1 Nemotron Nano 8B, Llama 3.3 Nemotron Nano 8B v1 (Reasoning), Llama 3.1 Nemotron Ultra 253B Reasoning, Llama 3.3 Nemotron Super 49B v1, and Llama 3.3 Nemotron Super 49B Reasoning, IBM: Granite 3.0 2B, OpenVoice: Granite 3.0 8B, Inceptionlabs: Mercury Coder Mini, Mercury Coder Small, Mercury Instruct, and Mercury Small, Reka AI: Reka Core, Reka Edge, Reka Flash (Feb ’24), Reka Flash, and Reka Flash 3, Xiaomi: MiMo 7B RL, Baichuan: Baichuan 4 and Baichuan M1 (Preview), Other: LLaVA-v1.5-7B, Cohere: Aya Expanse 32B, Aya Expanse 8B, Command, Command A, Command Light, Command R7B, Command-R, Command-R (Mar ’24), Command-R+ (Apr ’24), and Command-R+, Bytedance: Duobao 1.5 Pro, Seed-Thinking-v1.5, Skylark Lite, and Skylark Pro, AI21 Labs: Jamba 1.5 Large, Jamba 1.5 Large (Feb ’25), Jamba 1.5 Mini, Jamba 1.5 Mini (Feb 2025), Jamba 1.6 Large, Jamba 1.6 Mini, and Jamba Instruct, Snowflake: Arctic and Snowflake Llama 3.3 70B, Alibaba: QwQ-32B, QwQ 32B-Preview, Qwen Chat 14B, Qwen Chat 72B, Qwen Chat 7B, Qwen1.5 Chat 110B, Qwen1.5 Chat 14B, Qwen1.5 Chat 32B, Qwen1.5 Chat 72B, Qwen1.5 Chat 7B, Qwen2 72B, Qwen2 Instruct 7B, Qwen2 Instruct A14B 57B, Qwen2-VL 72B, Qwen2.5 Coder 32B, Qwen2.5 Coder 7B , Qwen2.5 Instruct 14B, Qwen2.5 Instruct 32B, Qwen2.5 72B, Qwen2.5 Instruct 7B, Qwen2.5 Max, Qwen2.5 Max 01-29, Qwen2.5 Omni 7B, Qwen2.5 Plus, Qwen2.5 Turbo, Qwen2.5 VL 72B, Qwen2.5 VL 7B, Qwen3 0.6B, Qwen3 0.6B (Reasoning), Qwen3 1.7B, Qwen3 1.7B (Reasoning), Qwen3 14B, Qwen3 14B (Reasoning), Qwen3 235B, Qwen3 235B (Reasoning), Qwen3 30B A3B, Qwen3 30B A3B (Reasoning), Qwen3 32B, Qwen3 32B (Reasoning), Qwen3 4B, Qwen3 4B (Reasoning), Qwen3 8B, and Qwen3 8B (Reasoning), and 01.AI: Yi-Large and Yi-Lightning.
Compare + LLM Banchmarks at RankLLMs.com
