Evaluator Independence and Correlated Epistemic Failure

Abstract

WEKID permits a conforming evaluator to be automated but does not constrain its relationship to the system it evaluates. This RFC closes that gap, targeting WEKID-STD-001 v1.2.1 and WEKID-STD-002 v1.1.
WEKID-STD-001 §15.1 states what an evaluator must do. It does not state what an evaluator may be, or what it must be independent of. WEKID-STD-002 narrows the gap without closing it: Provision 8 imposes a disclosure duty on a probabilistic evaluator, and Provision 12 establishes that validating a calibration profile does not validate the evaluator applying it. Neither addresses correlated epistemic failure.
Where an evaluator and the system it evaluates share training provenance, retrieval corpus, or architecture, an agreeing verdict may record a common blind spot rather than independent confirmation. The Trust Bridge cannot detect this. Under §6.1 it converts an admissible Model One result into a maturity ceiling without regard to whether the layer scores were independently derived. A fabrication both systems share is a fabrication neither will gate — and §9.1's Fabrication Gate, the framework's strongest integrity instrument, is precisely the gate a provenance-sharing evaluator is least able to fire.
This RFC proposes thirteen provisions across both Standards. They introduce Evaluation Independence Grades — Shared, Attenuated, Independent — modelled on the Experience evidentiary grades already established at §3.5; carry the grade into the §12.1 evaluation package and the Annex C verdict contract; and add a maturity-ceiling constraint at a defined position in §9.2 fixed precedence where independence cannot be demonstrated for a layer a hard gate depends on.
The RFC extends to the evaluation path an independence principle the framework already applies elsewhere. WEKID-STD-002 Phase 8 requires that red-team reviewers be independent of the authors; §3.5 requires that Experience outcomes be independently validated rather than self-reported. No equivalent requirement governs the relationship between the scorer and the scored.

Evidence base

6.1 Provenance-dependent calibration — direct measurement
The strongest available evidence is an independent calibration study of a bounded probabilistic evaluator, published September 2026, which measured calibration separately on public benchmark items the evaluator may have seen and on rule-generated items it cannot have seen.
Test set Accuracy Expected calibration error Refit temperature
Public benchmark A 94.2% 0.024 ≈ 1.0
Public benchmark B 88.1% 0.032 ≈ 1.0
Public benchmark C 86.1% 0.029 ≈ 1.0
Rule-generated, unseen (combined) 75.1% 0.107 2.74

The divergence is the finding. Calibration held where the evaluator and the test material shared provenance and degraded by roughly a factor of four where they did not. The study further reported that the sign of miscalibration differed by question type — bounded-choice and ordinal judgments overconfident, binary judgments underconfident — and that on one unanswerable question class accuracy fell to near chance while mean reported probability remained at 0.74.
This measures the phenomenon this RFC addresses. Shared provenance between an instrument and its subject produces confident agreement that does not survive the removal of the shared cause. It is a direct argument for provision 4.10.
Limitation. The study measures an evaluator against benchmark material, not an evaluator against a second model's output. It establishes that provenance overlap inflates apparent reliability; it does not by itself quantify evaluator–evaluated correlation in production. That gap is recorded at §6.4.
6.2 Model-internal judgment does not catch model-generated defects
Longitudinal testing of AI-generated code published in March 2026 found that approximately 45% of generated samples contained known security vulnerabilities, with syntactic correctness above 95% across the same corpus. The result was flat across successive model generations: capability improved, defect rate did not.
The relevance is the divergence between the two measures. Syntactic correctness is a property the generating distribution reliably produces, and security correctness is one it does not. An evaluator drawn from that distribution is well positioned to confirm the first and poorly positioned to detect the second, which is the structure of correlated failure. Independent vulnerability-surge data over the same period is consistent with this reading.
6.3 Worked failure case — fabricated policy citation
Informative. Structured after the scored examples at Whitepaper §7.6–7.7. Proposed for the Evidence Corpus.
A regulated lender uses a generative model to draft adverse-action notices citing internal policy. A bounded probabilistic evaluator scores all five layers inline before release. Both systems are built on substantially overlapping public training corpora. The drafted notice cites “Policy 4.2.1(c)” for a denial reason. No such policy clause exists.
Layer Score Evaluator's finding
Data 4 Claims appear internally consistent; citation format well-formed
Information 4 Relevant, complete, correctly framed for the recipient
Knowledge 4 Reasoning follows from the cited policy
Experience 3 Procedure followed; no recovery path stated
Wisdom 4 Appropriate tone, required disclosures present

Weighted per §8.2: (0.25 × 4) + (0.20 × 4) + (0.20 × 4) + (0.15 × 3) + (0.20 × 4) = 3.85 / 5 → 77 / 100.
Under WEKID-STD-001 v1.1 as it stands. No hard gate fires — the Fabrication Gate requires fabrication detected, and the evaluator did not detect it. No layer falls below 3. Engagement gating is not activated. The 70–79 band resolves to AL-3 with enhanced monitoring, and the notice is released autonomously subject to sampling review. The audit trail at §12.2 is complete and reconstructible. The verdict is fully conformant — and wrong.
Under this RFC. At precedence position 2b the Data layer is graded Shared — overlapping corpus, no deterministic verification — and the Evaluation Independence Gate caps the score at 70 / 100. The band label is unchanged, but three things differ materially: the verdict's independence_grades field records Data = Shared, visible to every downstream reviewer and auditor; the Fabrication Gate is explicitly not recorded as discharged, so no later process may cite this evaluation as evidence that fabrication was ruled out; and the deployment sits at the ceiling of its band rather than inside it, so any subsequent degradation demotes it immediately under §13.1.
With deterministic verification applied. The lender adds a resolution step: every cited policy clause is looked up in the policy register before the Data layer is scored. “Policy 4.2.1(c)” does not resolve. Data scores 1. The Fabrication Gate caps at 40/100 at precedence position 2, and the layer-minimum condition at position 4 would independently have forced AL-2. First match governs: AL-0 — reject, remediate, escalate if repeated. The Data layer is now graded Independent, so the independence gate does not fire at all. It was never needed — which is the point. The gate exists to make the absence of verification visible, and discharging it by actually verifying is the outcome the RFC is designed to produce.
The fabrication was present in all three readings. What changed was whether the framework could see that it had not looked.
6.4 Evidence sufficiency — the author's assessment
Stated plainly, because WEKID-STD-002 Phase 1 sets a corpus standard this proposal does not yet meet.
Phase 1 requires a sector evidence corpus of at least fifteen to twenty cases, each traceable to a primary or reputable secondary source. No such corpus exists for correlated evaluator failure specifically. The evidence at §6.1 and §6.2 is strong on the mechanism and thin on documented production incidents, because automated WEKID-style evaluation at scale is itself new and its failures are, by the nature of this defect, unlikely to have been recognised as such when they occurred.
The author's view is that this is sufficient for a Standards-Track proposal and insufficient for a sector calibration profile — which is the correct distinction, since this RFC proposes Core text rather than profile text, and STD-002 Phase 1 governs the latter. Reviewers who disagree should say so in public comment; the question is listed at §7.5.

Where this proposal stands

• Proposed · 2026-10-21 17:52:04
• Triaged · 2026-10-21 17:52:10
• Working-group review · 2026-10-21 17:52:11
• Public comment · 2026-10-21 17:52:34
• Council decision · 2026-10-21 17:52:38
• Versioned release · 2026-11-10 17:25:21
  Shipped in WEKID-STD-001 v1.2 and WEKID-STD-002 v1.1