https://c-allergic.github.io/tags/benchmarking/Recent content in Benchmarking onHugoenMon, 11 May 2026 00:00:00 +0800https://c-allergic.github.io/blog/reproducing-capsbench/Mon, 11 May 2026 00:00:00 +0800https://c-allergic.github.io/blog/reproducing-capsbench/<h2 id="why-caption-evaluation-matters">Why Caption Evaluation Matters</h2> <p>Automated image captioning has come a long way, but evaluation hasn&rsquo;t kept up. Standard metrics like BLEU, METEOR, and CIDEr were designed for reference-based machine translation — they weren&rsquo;t built to capture the nuanced quality dimensions of modern generative captioners. More recent metrics like CLIPScore and RefCLIPScore improve on alignment but still miss fine-grained issues like hallucination, style appropriateness, and factual grounding.</p> <h2 id="capsbench-a-closer-look">CapsBench: A Closer Look</h2> <p><a href="https://arxiv.org/abs/2409.10695">PGV3</a>&rsquo;s <strong>CapsBench</strong> is one of the most comprehensive efforts to benchmark caption evaluation metrics. It tests evaluators across multiple dimensions: object hallucination, attribute accuracy, relationship correctness, and more. I&rsquo;ve been reproducing CapsBench to understand its strengths and — more importantly — where it breaks down.</p>