SMALL SAMPLE · VERSIONED · REPRODUCIBLE

A benchmark with its failure left in.

Ten synthetic cases test one model against declared cells and complete rows. The cropped scan fails exactly where its source is missing.

Download results JSON Read the method

Benchmark ID

2026-07-19-paddlevl16-v1

System version

8584fdd

Provider model

PaddleOCR-VL-1.6

Dataset

10 cases · 11 pages

HOW IT WAS SCORED

Recall, structure, and completion are separate.

Expected cells

Headers and non-empty declared values are normalized for Unicode, whitespace, case, decimal form, and ISO dates, then matched against saved actual workbooks.

Complete rows

A row passes only when every expected value appears contiguously in one output row. The crop case therefore scores zero for full rows.

Run completion

Every submitted case remains in the denominator. A provider or conversion failure would score zero rather than disappearing from the report.

Cell recall: 98.2% 214/218
Full-row exact match: 90.3% 28/31
Completed runs: 100% 10/10

BY FIELD TYPE

The missing crop is visible in headers and numbers.

Headers98.2%54/55 exact matches

Text100%88/88 exact matches

Numbers94.9%56/59 exact matches

Dates100%16/16 exact matches

DOCUMENTED FAILURE

A cropped source cannot yield the missing amount column.

The right-edge stress case removes “Amount USD” and all three values before conversion. It completed technically, but scored 80.0% for expected cells and 0% for complete rows.

Input PDF Actual XLSX

Missing header: Amount USD
Missing values: 425.30 · 1180.00 · 690.75
Review action: Return to the source and rescan the complete page.

ALL TEN CASES

Inputs and actual model outputs are public.

Case	Input quality	Pages	Cell recall	Full rows	Artifacts
Scanned register	Clean image-only	1	100%	100%	PDF XLSX
Multi-page register	Digital text	2	100%	100%	PDF XLSX
Bank statement	Digital text	1	100%	100%	PDF XLSX
Commercial invoice	Digital text	1	100%	100%	PDF XLSX
Customs declaration	Digital text	1	100%	100%	PDF XLSX
Bill of lading	Digital text	1	100%	100%	PDF XLSX
Air waybill	Digital text	1	100%	100%	PDF XLSX
Scanned register	Resampled + 1.2° skew	1	100%	100%	PDF XLSX
Scanned register	Blur + low contrast + 2.4° skew	1	100%	100%	PDF XLSX
Scanned register	Right edge cropped	1	80.0%	0.0%	PDF XLSX

READ BEFORE REUSING THE NUMBER

What this benchmark does not prove.

Ten synthetic cases and eleven pages are too small for a general product accuracy claim.
Four scan-quality cases derive from one layout. Handwriting, camera perspective, language variation, and naturally noisy archives are absent.
Expected-value recall does not penalize unrelated extra cells and is not a semantic field-mapping metric.
Each business document type has one case, so its rate is descriptive, not a stable estimate.

Dataset manifest Run record Scored results

REPRODUCE IT

Every input, output, hash, and formula is inspectable.

python3 scripts/generate_conversion_quality_dataset.py

go run ./cmd/quality-benchmark \
  --benchmark-id 2026-07-19-paddlevl16-v1 \
  --dataset-manifest apps/web/public/benchmarks/conversion-quality/2026-07-19-paddlevl16-v1/dataset-manifest.json \
  --system-version <commit>

node scripts/score_conversion_quality_benchmark.mjs \
  --benchmark-id 2026-07-19-paddlevl16-v1