Laws of AI Evaluation

Tool

AI Readiness Review

A more rigorous instrument for launch review. Five screening questions come first, then 27 criteria across seven dimensions. Under each criterion, the labels marked "Based on" name the standards and guidelines it draws on; the key at the bottom of the page explains each one. Some criteria are marked as gates.

1Screening — five questions asked before scoring

  1. S1
    Does the output influence a decision about a person’s employment, finances, health, housing, legal status, education, or benefits?
  2. S2
    Can the system take an action, or commit a result, without a human confirming it first?
  3. S3
    Does the feature process personal, sensitive, or identifiable data?
  4. S4
    Will non-expert, vulnerable, or unsupported users rely on the output without a specialist intermediary?
  5. S5
    Are the effects of a wrong output slow, costly, or impossible to reverse?

D1Capability disclosure

  1. D1.1
    Users are told they are interacting with an AI system, clearly and at or before first interaction.
    Gate · T2Based on
  2. D1.2
    Expected reliability is communicated in terms the user can act on, not as an unqualified capability claim.
    Based on
  3. D1.3
    Scope boundaries are stated: the interface communicates what the system is not for.
    Based on
  4. D1.4
    Expectation-setting is staged across the experience rather than front-loaded into a single disclaimer.
    Based on

D2Output legibility

  1. D2.1
    Uncertainty is expressed per output and reflects actual model confidence rather than a fixed decorative label.
    Based on
  2. D2.2
    The reason for a given output is available at the point of decision, in plain language.
    Gate · T2Based on
  3. D2.3
    Output is traceable to the inputs, records, or rules that produced it.
    Based on
  4. D2.4
    Confidence signals and explanations are conveyed non-visually as well as visually.
    Gate · T2Based on

D3Human agency and oversight

  1. D3.1
    The user can see what the system proposes to do before it takes effect.
    Gate · T2Based on
  2. D3.2
    A person can disregard, override, or reverse the output without engineering assistance.
    Gate · T2Based on
  3. D3.3
    Consequential actions require explicit human confirmation; the system does not execute them silently.
    Gate · T2Based on
  4. D3.4
    The design actively counters automation bias rather than encouraging uncritical acceptance.
    Gate · T3Based on

D4Uncertainty and failure design

  1. D4.1
    An explicit “cannot determine” state exists and is structurally distinct from a confident answer.
    Gate · T2Based on
  2. D4.2
    Low-confidence output receives different visual and interaction treatment from high-confidence output.
    Based on
  3. D4.3
    A designed recovery path exists for AI-specific errors, separate from generic system error handling.
    Gate · T2Based on
  4. D4.4
    Failure is contained: one bad output does not block, corrupt, or silently alter unrelated work.
    Based on

D5Equity and accessibility

  1. D5.1
    The AI-specific interface has been tested with assistive technology, not only reviewed visually or inherited from a component library.
    Gate · T2Based on
  2. D5.2
    Performance disparities across user subgroups, languages, or regions have been measured rather than assumed absent.
    Gate · T3Based on
  3. D5.3
    AI-generated accessibility content is human-reviewed before it is relied upon.
    Based on
  4. D5.4
    The experience adapts to user context and stakes rather than offering one undifferentiated path.
    Based on

D6Demonstrated value

  1. D6.1
    The efficiency or quality claim is measured against a real pre-AI baseline, not estimated.
    Based on
  2. D6.2
    Complexity is genuinely removed rather than displaced into a downstream step, another team, or a later moment.
    Based on
  3. D6.3
    A granular feedback mechanism exists and demonstrably routes into system improvement.
    Based on
  4. D6.4
    Post-deployment monitoring is defined, with a named owner and a threshold that triggers action.
    Based on

D7Accountability record

  1. D7.1
    Known limitations are documented in a form that actually reaches the people who deploy and use the system.
    Gate · T2Based on
  2. D7.2
    A named human owner is accountable for this feature’s behavior in production.
    Based on
  3. D7.3
    An impact assessment covering affected individuals and groups was completed before launch.
    Gate · T3Based on

KeySources — what each “Based on” label refers to

This page lists the instrument (AIRR v1.0). Tier routing and scoring are not built here yet; see the Changelog. A Gate badge shows the risk tier recorded for that criterion in the instrument.