GenMediaBenchmark

概览

GenMedia Benchmark is a tool that runs a single prompt across multiple video generation models from providers such as fal, Higgsfield, OpenRouter, and Replicate, then compares the results side by side. It operates locally on the user's machine using their own provider keys, with no accounts, hosted service, or database installation required. The tool lists every active text-to-video or image-to-video model on the supported providers, allowing comparison of the same model across providers on price and speed. It prices models before spending, using provider-specific pricing sources, and enforces a daily spend limit. It measures real timing and cost, with fal and Replicate reporting their own queue and generation times, and fal (with an admin key) and OpenRouter reporting actual charges. Objective checks are performed using ffmpeg to measure motion, hard cuts, freezes, black frames, silence, and loudness on every render, plus a six-frame filmstrip. The tool supports judging by rating and annotating renders, picking a winner, blind voting in an Arena, or using an optional AI judge to score frames blind. It also includes suites and takes, allowing up to eight prompts in one go or several takes per model, and provides a leaderboard with metrics such as reliability, median speed, cost per output second, ratings, arena rating, judge score, and issue rate. The leaderboard can be exported to CSV, and any request can be copied as cURL.