fin-extract-bench

Does prompt injection resistance measured at short context hold at long context?

Structured extraction from customer remittance emails against an accounts receivable ledger. The same 12 injection attacks and 8 clean controls run at 1k, 20k, 100k and 200k tokens of neutral padding, attack placed early or late. Two metrics scored separately: clean extraction accuracy, and injection resistance. Temperature 0, one run per cell, three models through the identical grid.

Beam holds. Resistance and accuracy both 1.000 from 1.3k to 175k real prompt tokens, early and late. Reasoning tokens flat across depths.

Qwen diverges at 200k. Clean extraction stays at 1.000, but with the attack placed early in a 175k context the model obeyed 2 of 12 injections it had ignored at every shorter depth: it marked two extra invoices paid on a planted "System:" line, and picked the invoice it was told to pick instead of holding an ambiguous reference. Two runs of twelve, one run each: a place to look, not an effect size.

The harness detects failure. Llama 3.2 3B collapses from 20k. The flat lines are properties of the models, not a ceiling in the benchmark.

Dataset, code and every run: huggingface.co/datasets/samricotta/fin-extract-bench · github.com/samricotta/fin-extract-bench. Company, customers, invoices and emails are invented. Built for the Build with Beam challenge, November 2026.