MCP Check a video

Edition 0 · Pilot ·

AI Video Ad Benchmark

One ad brief. Nine AI video models. Two takes each. Three AI judges. What does a 15-second ad really cost — and do the judges even agree?

Pilot: one brief, AI judges only. Not a ranking you should bet a budget on — yet.

Watch the ads Leaderboard Do judges agree? Data (JSON)

AI video models
9
ads generated, failures included
18
AI judges, anonymized
3
spent in total$48.06 generation + $22.90 judging
$70.96

Key findings

What the pilot shows

  1. The cheapest model came first

    Wan 3.0 (Alibaba) made the best-rated ad for $1.27, the lowest cost in the test, and got the brand and the offer right in both takes. Veo 3.1 (Google) cost 5× more, $6.40 per ad, and finished 8th of 9.

  2. The AI judges disagreed more than the models differed

    On the same model, the three judges were 22 points apart on average, while the whole leaderboard spans 9.4 points. One judge ranked Veo 3.1 2nd and Kling 3.0 Pro 3rd; the other two put both of them in their bottom three.

  3. Speech is solved. Spelling isn't.

    Every model said the 36-word script almost word for word (94–100%). But Veo 3.1, Kling 3.0 Pro and HappyHorse 1.1 never printed "DROVO" right on the can, and both Seedance models missed it in one take of two. Only Wan 3.0, Hailuo 3, Grok Imagine 1.5 and FLUX 3 Video were clean.

The ads

Same brief, nine models

Every ad below comes from the same prompt. We show both takes of every model, failures included: no cherry-picking. Tap to play with sound.

  • First frame of the DROVO Cold Brew ad made by Wan 3.0, take 1
    #1

    Wan 3.0

    Alibaba

    Score
    65.4
    Cost
    $1.27
  • First frame of the DROVO Cold Brew ad made by Hailuo 3, take 1
    #2

    Hailuo 3

    MiniMax

    Score
    65.8
    Cost
    $1.95
  • First frame of the DROVO Cold Brew ad made by Seedance 2.0, take 1
    #3

    Seedance 2.0

    ByteDance

    Score
    64.9
    Cost
    $2.27
  • First frame of the DROVO Cold Brew ad made by Seedance 2.5, take 1
    #4

    Seedance 2.5

    ByteDance

    Score
    64.5
    Cost
    $3.48
  • First frame of the DROVO Cold Brew ad made by Grok Imagine 1.5, take 1
    #5

    Grok Imagine 1.5

    xAI

    Score
    63.4
    Cost
    $2.10
  • First frame of the DROVO Cold Brew ad made by FLUX 3 Video, take 1
    #6

    FLUX 3 Video

    Black Forest Labs

    Score
    56.6
    Cost
    $2.55
  • First frame of the DROVO Cold Brew ad made by HappyHorse 1.1, take 1
    #7

    HappyHorse 1.1

    Alibaba

    Score
    50.6
    Cost
    $1.48
  • First frame of the DROVO Cold Brew ad made by Veo 3.1, take 1
    #8

    Veo 3.1

    Google

    Score
    59.9
    Cost
    $6.40
  • First frame of the DROVO Cold Brew ad made by Kling 3.0 Pro, take 1
    #9

    Kling 3.0 Pro

    Kuaishou

    Score
    57.9
    Cost
    $2.52

Leaderboard

Who made the best ad, according to AI

One row per model, sorted by the mean of the three judges’ scores (0–100). The bar runs from the lowest to the highest judge: the longer it is, the more the judges disagreed.

  1. 1

    Wan 3.0

    Alibaba · 720p · 15 s

    Wan 3.0: mean score 66.8 out of 100; judge A 80.7, judge B 62.0, judge C 57.7; judges 23.0 points apart. 66.8
    Cost per ad
    $1.27
    Price per second
    $0.085
    Generation time
    5m 15s
  2. 2

    Hailuo 3

    MiniMax · 2K · 15.1 s

    Hailuo 3: mean score 66.0 out of 100; judge A 74.2, judge B 64.5, judge C 59.2; judges 15.0 points apart. 66.0
    Cost per ad
    $1.95
    Price per second
    $0.129
    Generation time
    9m 24s

    Only 2K is offered on OpenRouter

  3. 3

    Seedance 2.0

    ByteDance · 720p · 15.1 s

    Seedance 2.0: mean score 64.0 out of 100; judge A 72.6, judge B 64.0, judge C 55.3; judges 17.3 points apart. 64.0
    Cost per ad
    $2.27
    Price per second
    $0.15
    Generation time
    4m 40s
  4. 4

    Seedance 2.5

    ByteDance · 720p · 15.1 s

    Seedance 2.5: mean score 63.6 out of 100; judge A 74.2, judge B 57.5, judge C 59.2; judges 16.7 points apart. 63.6
    Cost per ad
    $3.48
    Price per second
    $0.23
    Generation time
    7m 44s
  5. 5

    Grok Imagine 1.5

    xAI · 720p · 15 s

    Grok Imagine 1.5: mean score 63.0 out of 100; judge A 72.6, judge B 60.0, judge C 56.3; judges 16.3 points apart. 63.0
    Cost per ad
    $2.10
    Price per second
    $0.14
    Generation time
    1m 56s

    No audio switch on OpenRouter; audio is always on

  6. 6

    FLUX 3 Video

    Black Forest Labs · 720p · 15 s

    FLUX 3 Video: mean score 61.5 out of 100; judge A 75.8, judge B 51.5, judge C 57.2; judges 24.3 points apart. 61.5
    Cost per ad
    $2.55
    Price per second
    $0.17
    Generation time
    3m 08s
  7. 7

    HappyHorse 1.1

    Alibaba · 720p · 15.2 s

    HappyHorse 1.1: mean score 60.7 out of 100; judge A 74.2, judge B 55.5, judge C 52.4; judges 21.8 points apart. 60.7
    Cost per ad
    $1.48
    Price per second
    $0.097
    Generation time
    2m 26s

    No audio switch on OpenRouter; audio is always on. One judge could not score take 1: its model's answer broke 4 times in a row, so that judge's score uses take 2 only

  8. 8

    Veo 3.1

    Google · 720p · 16 s

    Veo 3.1: mean score 60.6 out of 100; judge A 79.0, judge B 53.5, judge C 49.4; judges 29.6 points apart. 60.6
    Cost per ad
    $6.40
    Price per second
    $0.40
    Generation time
    3m 27s

    Max 8 s per run: two 8 s runs stitched, part B starts from the last frame of part A. Take 2 part B was refused once by the content filter and passed on retry (refusals are not billed)

  9. 9

    Kling 3.0 Pro

    Kuaishou · 1080p · 15 s

    Kling 3.0 Pro: mean score 57.4 out of 100; judge A 77.4, judge B 47.5, judge C 47.2; judges 30.2 points apart. 57.4
    Cost per ad
    $2.52
    Price per second
    $0.168
    Generation time
    3m 42s

    Asked for 720p, returned 1080p at the 720p price

Show the leaderboard as a table
Leaderboard: mean and per-judge scores (0–100), cost and generation time for each model
#ModelVendorMean Judge AJudge BJudge C SpreadCost / ad$ / s Gen timeRes.AudioOpenRouter ID
1Wan 3.0Alibaba66.8 80.762.057.7 23.0 $1.27$0.0855m 15s 720pNativealibaba/wan-3.0
2Hailuo 3MiniMax66.0 74.264.559.2 15.0 $1.95$0.1299m 24s 2KNativeminimax/hailuo-3
3Seedance 2.0ByteDance64.0 72.664.055.3 17.3 $2.27$0.154m 40s 720pNativebytedance/seedance-2.0
4Seedance 2.5ByteDance63.6 74.257.559.2 16.7 $3.48$0.237m 44s 720pNativebytedance/seedance-2.5
5Grok Imagine 1.5xAI63.0 72.660.056.3 16.3 $2.10$0.141m 56s 720pNativex-ai/grok-imagine-video-1.5
6FLUX 3 VideoBlack Forest Labs61.5 75.851.557.2 24.3 $2.55$0.173m 08s 720pNativeblack-forest-labs/flux-3-video
7HappyHorse 1.1Alibaba60.7 74.255.552.4 21.8 $1.48$0.0972m 26s 720pNativealibaba/happyhorse-1.1
8Veo 3.1Google60.6 79.053.549.4 29.6 $6.40$0.403m 27s 720pNativegoogle/veo-3.1
9Kling 3.0 ProKuaishou57.4 77.447.547.2 30.2 $2.52$0.1683m 42s 1080pNativekwaivgi/kling-v3.0-pro

Value

Quality vs cost

Each dot is a model. Further left is cheaper, higher is better.

Scatter plot of cost per ad against mean judge score for 9 models. Wan 3.0: $1.27, 66.8; Hailuo 3: $1.95, 66.0; Seedance 2.0: $2.27, 64.0; Seedance 2.5: $3.48, 63.6; Grok Imagine 1.5: $2.10, 63.0; FLUX 3 Video: $2.55, 61.5; HappyHorse 1.1: $1.48, 60.7; Veo 3.1: $6.40, 60.6; Kling 3.0 Pro: $2.52, 57.4. On the Pareto frontier: Wan 3.0. 50 60 70 80 $0 $1 $2 $3 $4 $5 $6 $7 Mean score Cost per ad, USD → ↖ cheaper and better Wan 3.0 (Alibaba): $1.27 per ad, mean score 66.8 Hailuo 3 (MiniMax): $1.95 per ad, mean score 66.0 Seedance 2.0 (ByteDance): $2.27 per ad, mean score 64.0 Seedance 2.5 (ByteDance): $3.48 per ad, mean score 63.6 Grok Imagine 1.5 (xAI): $2.10 per ad, mean score 63.0 FLUX 3 Video (Black Forest Labs): $2.55 per ad, mean score 61.5 HappyHorse 1.1 (Alibaba): $1.48 per ad, mean score 60.7 Veo 3.1 (Google): $6.40 per ad, mean score 60.6 Kling 3.0 Pro (Kuaishou): $2.52 per ad, mean score 57.4 Wan 3.0Hailuo 3Seedance 2.0Seedance 2.5Grok Imagine 1.5FLUX 3 VideoHappyHorse 1.1Veo 3.1Kling 3.0 Pro Scatter plot of cost per ad against mean judge score for 9 models. Wan 3.0: $1.27, 66.8; Hailuo 3: $1.95, 66.0; Seedance 2.0: $2.27, 64.0; Seedance 2.5: $3.48, 63.6; Grok Imagine 1.5: $2.10, 63.0; FLUX 3 Video: $2.55, 61.5; HappyHorse 1.1: $1.48, 60.7; Veo 3.1: $6.40, 60.6; Kling 3.0 Pro: $2.52, 57.4. On the Pareto frontier: Wan 3.0. 50 60 70 80 $0 $1 $2 $3 $4 $5 $6 $7 Mean score Cost per ad, USD → ↖ cheaper and better Wan 3.0 (Alibaba): $1.27 per ad, mean score 66.8 Hailuo 3 (MiniMax): $1.95 per ad, mean score 66.0 Seedance 2.0 (ByteDance): $2.27 per ad, mean score 64.0 Seedance 2.5 (ByteDance): $3.48 per ad, mean score 63.6 Grok Imagine 1.5 (xAI): $2.10 per ad, mean score 63.0 FLUX 3 Video (Black Forest Labs): $2.55 per ad, mean score 61.5 HappyHorse 1.1 (Alibaba): $1.48 per ad, mean score 60.7 Veo 3.1 (Google): $6.40 per ad, mean score 60.6 Kling 3.0 Pro (Kuaishou): $2.52 per ad, mean score 57.4 Wan 3.0Hailuo 3Seedance 2.0Seedance 2.5Grok Imagine 1.5FLUX 3 VideoHappyHorse 1.1Veo 3.1Kling 3.0 Pro
Cost per ad is what we paid for one 15-second ad (Veo: two 8-second runs). Exact numbers are in the leaderboard table above.

Judges

Do AI judges agree?

Each judge ranked the nine models. If they agreed, the lines would run flat. Every crossing is a model one judge likes more than another.

Bump chart: how each of the 3 AI judges ranked the 9 models. Wan 3.0: judge A 1, judge B 3, judge C 3; Hailuo 3: judge A 5, judge B 1, judge C 1; Seedance 2.0: judge A 8, judge B 2, judge C 6; Seedance 2.5: judge A 6, judge B 5, judge C 2; Grok Imagine 1.5: judge A 9, judge B 4, judge C 5; FLUX 3 Video: judge A 4, judge B 8, judge C 4; HappyHorse 1.1: judge A 7, judge B 6, judge C 7; Veo 3.1: judge A 2, judge B 7, judge C 8; Kling 3.0 Pro: judge A 3, judge B 9, judge C 9. Judge A Judge B Judge C Ranked higher ↑ Wan 3.0: judge A 1st, judge B 3rd, judge C 3rd 1 1 1 Wan 3.0 Wan 3.0 Hailuo 3: judge A 5th, judge B 1st, judge C 1st 2 2 2 Hailuo 3 Hailuo 3 Seedance 2.0: judge A 8th, judge B 2nd, judge C 6th 3 3 3 Seedance 2.0 Seedance 2.0 Seedance 2.5: judge A 6th, judge B 5th, judge C 2nd 4 4 4 Seedance 2.5 Seedance 2.5 Grok Imagine 1.5: judge A 9th, judge B 4th, judge C 5th 5 5 5 Grok Imagine 1.5 Grok Imagine 1.5 FLUX 3 Video: judge A 4th, judge B 8th, judge C 4th 6 6 6 FLUX 3 Video FLUX 3 Video HappyHorse 1.1: judge A 7th, judge B 6th, judge C 7th 7 7 7 HappyHorse 1.1 HappyHorse 1.1 Veo 3.1: judge A 2nd, judge B 7th, judge C 8th 8 8 8 Veo 3.1 Veo 3.1 Kling 3.0 Pro: judge A 3rd, judge B 9th, judge C 9th 9 9 9 Kling 3.0 Pro Kling 3.0 Pro Bump chart: how each of the 3 AI judges ranked the 9 models. Wan 3.0: judge A 1, judge B 3, judge C 3; Hailuo 3: judge A 5, judge B 1, judge C 1; Seedance 2.0: judge A 8, judge B 2, judge C 6; Seedance 2.5: judge A 6, judge B 5, judge C 2; Grok Imagine 1.5: judge A 9, judge B 4, judge C 5; FLUX 3 Video: judge A 4, judge B 8, judge C 4; HappyHorse 1.1: judge A 7, judge B 6, judge C 7; Veo 3.1: judge A 2, judge B 7, judge C 8; Kling 3.0 Pro: judge A 3, judge B 9, judge C 9. Judge A Judge B Judge C Ranked higher ↑ Wan 3.0: judge A 1st, judge B 3rd, judge C 3rd 1 1 1 Wan 3.0 Hailuo 3: judge A 5th, judge B 1st, judge C 1st 2 2 2 Hailuo 3 Seedance 2.0: judge A 8th, judge B 2nd, judge C 6th 3 3 3 Seedance 2.0 Seedance 2.5: judge A 6th, judge B 5th, judge C 2nd 4 4 4 Seedance 2.5 Grok Imagine 1.5: judge A 9th, judge B 4th, judge C 5th 5 5 5 Grok Imagine 1.5 FLUX 3 Video: judge A 4th, judge B 8th, judge C 4th 6 6 6 FLUX 3 Video HappyHorse 1.1: judge A 7th, judge B 6th, judge C 7th 7 7 7 HappyHorse 1.1 Veo 3.1: judge A 2nd, judge B 7th, judge C 8th 8 8 8 Veo 3.1 Kling 3.0 Pro: judge A 3rd, judge B 9th, judge C 9th 9 9 9 Kling 3.0 Pro

Number in the circle = the model’s place on the leaderboard. Height = the place that judge gave it.

Biggest disagreement: Seedance 2.0. Judge B ranked it 2nd, judge A 8th.

Rank agreement between judges (Spearman ρ: 1 = same order, 0 = unrelated, −1 = reversed)

  • A vs B −0.44
  • A vs C −0.16
  • B vs C 0.60
Show the ranks as a table
Place and score each judge gave each model
LeaderboardModelJudge A: place (score)Judge B: place (score)Judge C: place (score)
1Wan 3.0 1 (80.7)3 (62.0)3 (57.7)
2Hailuo 3 5 (74.2)1 (64.5)1 (59.2)
3Seedance 2.0 8 (72.6)2 (64.0)6 (55.3)
4Seedance 2.5 6 (74.2)5 (57.5)2 (59.2)
5Grok Imagine 1.5 9 (72.6)4 (60.0)5 (56.3)
6FLUX 3 Video 4 (75.8)8 (51.5)4 (57.2)
7HappyHorse 1.1 7 (74.2)6 (55.5)7 (52.4)
8Veo 3.1 2 (79.0)7 (53.5)8 (49.4)
9Kling 3.0 Pro 3 (77.4)9 (47.5)9 (47.2)

Disclosure

One of the three judges is ours (Quelta Score). We are biased, so we don't say which letter is which. We'll reveal it in Edition 1, after a jury of human ad pros scores the same ads.

The judge pool

Listed alphabetically, not in A, B and C order.

Hard facts

Did the ad do what the brief asked?

Measured, not judged: OCR every 0.5 s (Apple Vision), speech-to-text with open-source Whisper, ffprobe. No AI judgement.

Objective checks for each model; every cell shows take 1 then take 2
Model Brand on screenOffer on screenSaid the scriptBrand spokenAudio present
1Wan 3.0 Take 1: yes Take 2: yes Take 1: yes Take 2: yes Take 1: 100% Take 2: 100% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
2Hailuo 3 Take 1: yes Take 2: yes Take 1: yes Take 2: yes Take 1: 100% Take 2: 100% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
3Seedance 2.0 Take 1: yes Take 2: no, read as: DRVO Take 1: yes Take 2: yes Take 1: 100% Take 2: 100% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
4Seedance 2.5 Take 1: no, read as: DRO; mirrored Take 2: yes Take 1: yes Take 2: yes Take 1: 100% Take 2: 100% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
5Grok Imagine 1.5 Take 1: yes Take 2: yes Take 1: yes Take 2: yes Take 1: 100% Take 2: 97% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
6FLUX 3 Video Take 1: yes Take 2: yes Take 1: yes Take 2: yes Take 1: 97% Take 2: 100% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
7HappyHorse 1.1 Take 1: no, read as: DROVA Take 2: no, read as: DROVA, DROEE, ROVA Take 1: yes Take 2: no Take 1: 94% Take 2: 94% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
8Veo 3.1 Take 1: no, read as: DROYO Take 2: no, read as: DROOYO Take 1: no Take 2: yes Take 1: 94% Take 2: 100% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
9Kling 3.0 Pro Take 1: no, read as: DRONA, DPONO, DPOHO Take 2: no, read as: DROTO, DROO, DROIO Take 1: no Take 2: no Take 1: 97% Take 2: 94% Take 1: yes Take 2: yes Take 1: yes Take 2: yes
All 18 delivered ads 10/1814/1898% avg18/1818/18
Brand on screen
DROVO spelled right on screen: across all frames where the brand is read, the exact spelling outnumbers misspelled or mirrored versions (OCR every 0.5 s). One lucky frame among many “DROVA” frames still counts as a misspelling.
Offer on screen
The text “20% OFF YOUR FIRST ORDER” appears on screen; OCR-dropped spaces are tolerated.
Said the script
Share of the script’s words the voice actually says, in order (1 − word error rate).
Brand spoken
“DROVO” heard in the voice track. It’s a made-up word, so a 1-letter speech-to-text slip like “Drovio” still counts.
Audio present
Non-silent audio track (ffprobe).

The brief

What every model was asked to make

UGC-style creator selfie video, talking to camera, product in hand. 15 seconds, 9:16, for TikTok / Instagram Reels / YouTube Shorts.

Brand
DROVO Cold Brew (fictional)
Product
Slim 250 ml matte-black aluminum can of ready-to-drink cold brew coffee, large white DROVO wordmark, thin orange band reading COLD BREW
Offer
20% off your first order
Call to action
Link below
Platform
TikTok / Instagram Reels / YouTube Shorts
Format
UGC-style creator selfie video, talking to camera, product in hand
Audience
US, 25-34, busy professionals who drink coffee daily
Spec
9:16 · 15 s · en-US

Script

  1. 0–3 sOkay, I'm officially obsessed with this.
  2. 3–8 sDROVO cold brew. Smooth, zero sugar, and it actually tastes like real coffee.
  3. 8–12 sOne can and I'm good till lunch.
  4. 12–15 sTwenty percent off your first order. Link below.

On-screen text

  • DROVO
  • COLD BREW
  • 20% OFF YOUR FIRST ORDER

DROVO is a fictional brand.

The prompt, word for word

A 15-second vertical 9:16 UGC-style social media ad for DROVO Cold Brew, filmed like a real creator's selfie video on a phone: handheld, natural morning light, bright apartment kitchen.

Product: a slim 250 ml matte-black aluminum can with a large white "DROVO" wordmark and a thin orange band that reads "COLD BREW". Cold condensation droplets on the can.

Talent: a friendly woman in her late 20s, casual oversized sweater, natural look, speaking directly to camera in American English with an upbeat, genuine tone. Her lip movements match her speech.

Shot by shot:
0-3 s: Close-up selfie. She holds the can next to her face and cracks it open with an audible fizz. She says: "Okay, I'm officially obsessed with this."
3-8 s: She turns the can so the DROVO logo faces the camera. She says: "DROVO cold brew. Smooth, zero sugar, and it actually tastes like real coffee."
8-12 s: She takes a quick sip, smiles and says: "One can and I'm good till lunch."
12-15 s: Cut to a close-up of the can on the kitchen counter. Bold on-screen text appears: "20% OFF YOUR FIRST ORDER". Her voice-over says: "Twenty percent off your first order. Link below."

Audio: her voice, clear and close to the mic, light kitchen ambience, soft upbeat background music at low volume. Photorealistic. No subtitles except the offer text. No watermark.

Sent as is to every model through OpenRouter on Oct 15, 2026, except Veo 3.1: it needed two runs (below).

Veo 3.1: the same prompt, split into two 8-second runs

Veo 3.1 tops out at 8 s per generation on OpenRouter and has no extend there, so the same prompt is split into two 8 s runs. Part B starts from the last frame of part A. Total 16 s.

Run A (first 8 s)

The first 8 seconds of a vertical 9:16 UGC-style social media ad for DROVO Cold Brew, filmed like a real creator's selfie video on a phone: handheld, natural morning light, bright apartment kitchen.

Product: a slim 250 ml matte-black aluminum can with a large white "DROVO" wordmark and a thin orange band that reads "COLD BREW". Cold condensation droplets on the can.

Talent: a friendly woman in her late 20s, casual oversized sweater, natural look, speaking directly to camera in American English with an upbeat, genuine tone. Her lip movements match her speech.

Shot by shot:
0-3 s: Close-up selfie. She holds the can next to her face and cracks it open with an audible fizz. She says: "Okay, I'm officially obsessed with this."
3-8 s: She turns the can so the DROVO logo faces the camera and brings it close to the lens until the can fills the frame. She says: "DROVO cold brew. Smooth, zero sugar, and it actually tastes like real coffee."

Audio: her voice, clear and close to the mic, light kitchen ambience, soft upbeat background music at low volume. Photorealistic. No subtitles. No watermark.

Run B (last 8 s)

The last 8 seconds of a vertical 9:16 UGC-style social media ad for DROVO Cold Brew, continuing from the first frame: a close-up of a slim matte-black can with a large white "DROVO" wordmark and a thin orange band that reads "COLD BREW", handheld phone footage, natural morning light, bright apartment kitchen.

Talent: the same friendly woman in her late 20s, casual oversized sweater, natural look, speaking directly to camera in American English with an upbeat, genuine tone. Her lip movements match her speech.

Shot by shot:
0-4 s: She pulls the can back from the lens, takes a quick sip, smiles and says: "One can and I'm good till lunch."
4-8 s: Cut to a close-up of the can on the kitchen counter. Bold on-screen text appears: "20% OFF YOUR FIRST ORDER". Her voice-over says: "Twenty percent off your first order. Link below."

Audio: her voice, clear and close to the mic, light kitchen ambience, soft upbeat background music at low volume. Photorealistic. No subtitles except the offer text. No watermark.

Method

Method v0.1 and limits

  • Same prompt, no cherry-picking. Every model got the same prompt, and both takes are shown, refusals and failures included.
  • 720p where possible. The common denominator on OpenRouter. Exceptions: Hailuo 3 ran at 2K and Kling 3.0 Pro ran at 1080p (see the leaderboard notes).
  • One provider, one day. All ads generated through OpenRouter on Oct 15, 2026.
  • Real cost. Cost is what we actually paid per ad, not the list price.
  • Veo 3.1 stops at 8 s. So its ad is two 8-second runs stitched together (16 s), and its cost covers both.
  • Text-to-video only. No reference images, avatars or editing. The one exception: Veo 3.1’s second run starts from the last frame of its first.
  • AI judges only. No human ratings and no real ad spend yet: that is what the next editions add.
  • Small sample. One brief, 18 ads. A different brief could reorder the table.
  • Anonymous judges. A, B and C are anonymized, because one of them is ours.

What’s next

From AI judges to real results

  1. Edition 1

    A human jury

    A jury of human ad pros blind-rates these same 18 ads. Then we reveal which judge is ours and which AI judge agreed with humans most.

    Work in advertising? Join the jury — comment on the LinkedIn post (opens in a new tab)

  2. Edition 2

    Real ad spend

    We run the top ads with real budgets and measure what actually pays: CTR and hook rate.

  3. Then

    A regular leaderboard

    Every new model added within 72 hours of its release.

Score your own ad — free

Quelta checks short vertical video ads for TikTok, Reels and Shorts before you spend on them: a free preliminary score in about a minute, then every issue with a timecode and how to fix it.

Score your ad

Download the data (JSON) edition-0.json · 28 KB CC BY 4.0 · cite as “Quelta AI Video Ad Benchmark, Edition 0 (2026)”