REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation

Tengjie Lin1,* Yutao Sun1,* Jingwei Ni2,* Shuhan Ge3 Hao-Xuan Ma4 Yanting Miao5 Wangyue Lu1 Mingshuai Chen1 Tiancheng Zhao6 Jianwei Yin1
1Zhejiang University 2ETH Zürich 3Nanjing University of Science and Technology 4Nanjing University 5University of Waterloo 6Binjiang Institute of Zhejiang University
*Equal contribution
A live VQA benchmark that renews answer-bearing visual keys at evaluation time
Paper Code Dataset Quickstart

A contamination signal in static V*Bench

We first ask whether current VQA benchmarks are still measuring visual perception, or whether part of the score comes from memorized answer-bearing details. As a probe, we keep the original images fixed but replace the question with a fresh human-written question targeting a different visual key. If a model truly solves the image, performance should remain stable; instead, accuracy drops sharply across model families.

Model Original V* Fresh Relabel V* Drop
Qwen3.6-Plus 90.6% 74.3% -16.3pp
GPT-5.5 86.9% 75.9% -11.0pp
Claude Opus 4.7 78.5% 59.7% -18.8pp
Gemini 3.1 Pro 88.0% 69.6% -18.4pp
Seed-2.0-Lite 91.6% 74.3% -17.3pp
Kimi K2.6 81.7% 63.4% -18.3pp
MiMo v2.5 78.0% 66.5% -11.5pp
Llama 4 Maverick 61.3% 51.8% -9.5pp

This consistent drop is compatible with original V*Bench scores containing benefits from memorized answer-bearing details. REKEY starts from this observation and makes the visual key renewable.

The same image is paired with a fresh question that targets a different visual key.

How REKEY Renews a Visual Key

REKEY does not only rewrite the question. It identifies editable visual-key slots, regenerates the local visual evidence under context, and then writes a fresh question whose answer depends on the newly generated key.

Step 1

Annotate renewable slots

Original image with candidate context regions and editable slots.

Starting from the original VQA image, REKEY marks candidate edit slots and records what each slot can support, such as adding a cup or changing the color of an existing object.

Step 2

Generate within context

Context crop sent to a generative model and returned as an edited context patch.

REKEY crops a larger context window around the slot, prompts a generative model to edit only the marked region, and keeps the surrounding pixels fixed.

Step 3

Paste back and re-question

Regenerated image with the new visual key pasted back and a new question created.

The generated patch is pasted back into the full image, producing a refreshed benchmark item with a new answer-bearing visual key and a matching multiple-choice question.

Experimental Results

We ask three questions: does REKEY close the contamination gap (RQ1), does it preserve V*Bench's original difficulty (RQ2), and is human annotation necessary (RQ3)?

RQ1

REKEY closes the contamination gap

Model Original V* Fresh Relabel V* REKEY REKEY − Relabel≈ 0 = matches fresh questions
Qwen3.6-Plus 90.6 74.3 75.7 +/-5.4 +1.4
GPT-5.5 86.9 75.9 77.1 +/-2.1 +1.2
Claude Opus 4.7 78.5 59.7 67.2 +/-3.9 +7.5
Gemini 3.1 Pro 88.0 69.6 75.0 +/-3.3 +5.4
Seed-2.0-Lite 91.6 74.3 79.9 +/-2.6 +5.6
Kimi K2.6 81.7 63.4 72.6 +/-0.8 +9.2
MiMo v2.5 78.0 66.5 73.6 +/-4.7 +7.1
Llama 4 Maverick 61.3 51.8 56.9 +/-0.6 +5.1
Average 82.1 66.9 72.2 +5.3
RQ2

REKEY preserves the original difficulty

On LLaVA-1.5-13B, position accuracy barely moves (−0.4 pp) after regeneration — REKEY preserves the visual-search structure while renewing the answer-bearing evidence.

Setting Overall Attr Position
Original V* 48.68 43.47 56.57
REKEY 42.6 +/-2.4 33.2 +/-3.9 56.2 +/-3.3
Delta -6.1 -10.3 -0.4
RQ3

Human annotation is necessary

A model annotator changes the task it is supposed to evaluate: its slots are easier to edit and easier to find.

  • 33.6% usable Replace annotations
  • 3.2× larger Add regions than human annotations
  • +8.4 to +18.5 pp higher model accuracy
Radar chart comparing human REKEY annotations with VLM self-annotations.
RQ1 Analysis

Capability boosts that disappear under REKEY

Prompting and tool use improve static variants, but their gains do not consistently transfer after visual-key regeneration.

  • Chain-of-thought lifts Original / Relabel by up to +6.8 / +10.5 pp — but on REKEY the effect turns slightly negative (−0.5 to −3.1 pp).
  • Agentic zoom adds up to +9.5 / +12.4 pp on the static sets, yet only −4.0 to +3.6 pp on REKEY.
Bar chart comparing CoT and agentic effects on Original V*, Relabel V*, and REKEY.

BibTeX

@misc{lin2026rekey,
  title         = {REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation},
  author        = {Tengjie Lin and Yutao Sun and Jingwei Ni and Shuhan Ge and Hao-Xuan Ma and Yanting Miao and Wangyue Lu and Mingshuai Chen and Tiancheng Zhao and Jianwei Yin},
  year          = {2026},
  eprint        = {2606.20736},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.20736}
}