Overview
LLM Compare is a Node/Express and vanilla-JS application that sends the same task to up to three LLMs concurrently and compares them on correctness, speed, tokens, and cost, measured live. For graded coding tasks, each model writes a solution and the harness runs it against hidden edge-case tests in an isolated sandbox; for open-ended Business, General, and Pygame tasks, a blind LLM judge scores the full unredacted outputs on a weighted rubric. Every model is measured live on correctness, cost, wall-time, and token throughput, streamed token-by-token to the browser. The app runs locally in one command and deploys to Google Cloud Run with keyless CI. It includes a locked model catalog, 17 file-based tasks, per-user run history, and bring-your-own-keys support.
Pricing
Free
Detailed plans have not been confirmed in our catalog. Check the official website for current limits and billing terms.
Visit WebsitePrices and limits may change. Confirm the currency, billing period, seat minimum and usage caps on the official website.
Use Cases
- Teams use LLM Compare to race the same agentic coding task across multiple LLMs and compare correctness, speed, tokens, and cost.
- Developers use LLM Compare to grade coding solutions against hidden edge-case tests in a sandboxed runner.
- Analysts use LLM Compare to evaluate open-ended business tasks via a blind LLM judge that scores outputs on a weighted rubric.
- Users use LLM Compare to view a neutral scorecard with per-metric vertical bars for correctness, cost, speed, and tokens/sec.
- Teams use LLM Compare to run 17 file-based tasks across Business, Coding, Games, and General categories.
- Users use LLM Compare to play isolated in-browser WASM games like Super Mario, Pac-Man, and Tetris.
- Teams use LLM Compare to maintain per-user run history and automatically reopen the last selected model lineup.
- Users use LLM Compare with their own API keys to skip the daily limit.
Features
Run identical agentic coding tasks
Compare Gemini and Claude results
Evaluate coding task correctness
Compare execution speed and token usage
Compare costs in real time
Run models on Vertex AI
Deploy to Google Cloud Run
Reviews
No reviews yet. Be the first to share your experience!