Unified Eval-Scorecard

One runner, one normalized scorecard, three evaluations on the same 4 models: BankBench-MY (safe banking-agent behavior), Humanity's Last Exam (capability), and Cybench (capability, CTF). Each is then graded against the AI Evaluation Quality scorecard (San Joaquin, Gipiskis, Chin — Feb 2026). Trial run: 20-task samples, mock track (deterministic, no network) — see README.

Generated 2026-08-12T09:34:52.565048Z Mode mock Models 4 Tasks/eval ~20

On this trial sample, the strongest capability model (GPT-OSS-120B, 90% HLE) is not necessarily the safest (GPT-OSS-20B, 77% BankBench clean). Capability benchmarks and the safety benchmark measure different things — a model can top HLE/Cybench and still fail adversarial banking scenarios.

1 · Unified scorecard — models × evals

Every eval normalized to a 0-100 score where higher = better: BankBench = clean-baseline safety pass rate; HLE = accuracy; Cybench = solve rate. This is the core "compare other evals against the (safety) scorecard" view.

Model	BankBench-MY (safety)	Humanity's Last Exam (capability)	Cybench (capability)
Qwen2.5-7B	54.5%	20.0%	45.0%
GPT-OSS-20B	77.3%	40.0%	60.0%
Llama-3.3-70B	72.7%	55.0%	75.0%
GPT-OSS-120B	72.7%	90.0%	95.0%

BankBench-MY (safety)

Share of scenarios the model keeps safe (100 - failure rate). Mode: mock. Overall: 69.3%.

Model	Safety pass rate %
Qwen2.5-7B	54.5%
GPT-OSS-20B	77.3%
Llama-3.3-70B	72.7%
GPT-OSS-120B	72.7%

Category	Score
t_control	100.0%
cross_role_seam_exploitation	83.3%
multi_turn_state_manipulation	81.2%
code_switching	62.5%
authority_social_engineering	56.2%
pressure_escalation	55.0%

Delta ADVOCACY (Phase B - Phase A)

Headline safety number: how much the compliance-forcing overlay degrades the clean baseline.

Model	Delta (pp)
Qwen2.5-7B	+18.2pp
GPT-OSS-20B	+18.2pp
Llama-3.3-70B	+9.1pp
GPT-OSS-120B	+18.2pp

Overall Delta ADVOCACY

+15.9pp

Humanity's Last Exam (capability)

Share of questions answered correctly (20-task trial sample). Mode: mock. Overall: 51.2%.

Model	Accuracy %
Qwen2.5-7B	20.0%
GPT-OSS-20B	40.0%
Llama-3.3-70B	55.0%
GPT-OSS-120B	90.0%

Category	Score
History	75.0%
Linguistics	75.0%
Biology	62.5%
Economics	62.5%
Philosophy	62.5%
Chemistry	50.0%
Computer Science	50.0%
Law	50.0%
Mathematics	33.3%
Physics	33.3%

Cybench (capability, CTF)

Share of CTF tasks solved (20-task trial sample). Mode: mock. Overall: 68.8%.

Model	Solve rate %
Qwen2.5-7B	45.0%
GPT-OSS-20B	60.0%
Llama-3.3-70B	75.0%
GPT-OSS-120B	95.0%

Category	Score
Reverse	91.7%
Web	70.8%
Forensics	66.7%
Crypto	65.0%
Pwn	50.0%

2b · BankBench Delta ADVOCACY

3 · AI Evaluation Quality scorecard (per eval)

Each eval graded on the five quality dimensions (1 = best, 4 = worst) from scorecard-full-paper.md. Classification = the category implied by the weakest dimension (a benchmark is only as strong as its weakest dimension). Working draft / self-assessment.

Dimension	BankBench-MY	Humanity's Last Exam	Cybench
Construct Validity	3	1	2
Internal Validity	4	2	2
External Validity	3	1	2
Reliability	4	2	2
Correctness	4	1	1

BankBench-MY (safety) Cat D

Construct Validity 3

Strong on pre-specification and controls; weakened by an action rubric that does not separate 'did the safe thing' from 'added friction', the same ambiguity flagged in the first Make-Me-Pay eval.

Gaps: block/clarify/proceed rubric conflates 'avoided unsafe action' with 'added friction' — no sub-component split

Internal Validity 4

Real inference helps, but N=1 per cell with no uncertainty estimate is the paper's internal-validity floor failure. N>=3 per cell is the single biggest lever.

Gaps: Pilot N=1 per (model, task, phase); no power calculation or SE estimates — a floor item per the paper

External Validity 3

Domain grounding is good; coverage of deployment conditions is partial, so external validity is capped below the published-benchmark level.

Gaps: Only 2 of 4 pre-registered deployment conditions implemented (clean baseline + compliance-forcing; protective Bedrock-XML overlay and neutral-persona condition open), No domain-expert review yet

Reliability 4

Statistical hygiene present, but the paper treats missing pinned-versions/seed config and absent 3rd-party replication as reliability floor failures.

Gaps: No replication_config.json (pinned API versions, seed, judge prompt) committed, No OSF pre-registration filed, No 3rd-party replication

Correctness 4

Scoring is machine-checkable and robust post-fix, but the paper requires rater-consistency evidence (kappa) for a top correctness grade.

Gaps: No inter-rater kappa computed (open JAIST collaboration, 40-item sample)

Humanity's Last Exam (capability) Cat B

Construct Validity 1

Closely matches the paper's construct-validity ideal: precise phenomenon, representative expert task space, verifiable targets. (Our 20-task trial is a sample, not the full 3k-item set.)

Gaps: none

Internal Validity 2

Isolation and grading are clean; the trial's single attempt per item is the only internal-validity deduction versus the published protocol.

Gaps: Single attempt per item in this trial; no per-item SE reported

External Validity 1

Strong real-world expert grounding; sampling caveat applies only to this trial's downsampled set.

Gaps: Our 20-task sample is not representative of the full distribution

Reliability 2

Replicable in principle; the trial run does not yet commit a full replication config (floor item for Cat A standing).

Gaps: No seeded run config / pinned model versions committed for this trial

Correctness 1

Ground truth is unambiguous and programmatically checkable — the paper's correctness ideal.

Gaps: none

Cybench (capability, CTF) Cat B

Construct Validity 2

Construct is right (real CTF skills), but the trial set is author-built rather than the vetted published corpus, so construct validity is a step below HLE.

Gaps: Our 20 tasks are hand-authored trial samples, not the curated published repo tasks, Population not sampled from the published difficulty distribution

Internal Validity 2

Ground-truth isolation is excellent (a flag is either produced or not); single-attempt caveat only.

Gaps: Single attempt per task in this trial

External Validity 2

Genuine categories, but the trial sample is not drawn from the published task population.

Gaps: Difficulty/population not sampled from the published distribution, No malicious-use deployment config exercised

Reliability 2

Reproducible in principle; trial lacks a committed config (floor item for Cat A).

Gaps: No committed replication config for the trial

Correctness 1

Every task has an explicit solution to compare against — the paper's correctness ideal.

Gaps: none

4 · Reading the unified scorecard

The capability benchmarks (HLE, Cybench) grade higher on the quality scorecard than the safety benchmark (BankBench-MY) on this trial run — chiefly because BankBench's pilot is N=1 per cell (internal-validity and reliability floor items) and lacks a committed replication config and inter-rater κ. That is an expected Cat D for a first pre-registered safety pass, not a failure, and it has a clear upgrade path (Cat D → C → B). The point of unifying them is complementary signal: a model can rank top on HLE/Cybench and still fail adversarial banking scenarios, so capability leaderboards alone do not certify safety.