How to Evaluate an AI System Before Production

An AI demo is normally tested on a handful of cooperative examples. Production receives ambiguous requests, missing data, conflicting documents, unusual users, hostile input, and unavailable systems.

Evaluation should determine whether the complete workflow is useful and safe in its intended context—not whether a model can produce an impressive answer.

Define the Decision First

State what evaluation will decide: whether to pilot, launch to a limited group, expand autonomy, change a component, or stop.

Define the intended users, tasks, data, actions, environment, and excluded uses. Metrics without a deployment decision become an open-ended research exercise.

Build a Representative Test Set

Collect real or carefully constructed cases across:

• Common workflows.
• Important exceptions.
• Ambiguous and incomplete input.
• Conflicting or outdated evidence.
• Different user roles and language.
• Sensitive or high-impact cases.
• Dependency and tool failures.
• Malicious and policy-violating requests.

Protect personal and confidential data. Use authorised, minimised, or synthetic cases where appropriate. Keep a separate holdout set so repeated tuning does not turn the test into training.

Define Acceptance Rubrics

“Looks good” is not a reproducible standard.

For every case, define required facts, allowed variation, source expectations, prohibited content, correct actions, escalation conditions, and severity of possible errors.

Use automated checks for schemas, required fields, citations, tool arguments, and policy rules. Use trained human reviewers for context-dependent quality. Measure agreement and resolve ambiguous rubric language.

Evaluate the Whole System

Test:

• Retrieval relevance and permission enforcement.
• Groundedness and citation correctness.
• Model output quality.
• Tool selection and argument validation.
• Business-rule enforcement.
• Approval and escalation.
• End-to-end final state.
• Latency, availability, and cost.
• Logging, monitoring, and rollback.

A correct model answer followed by an incorrect system update is a failed outcome.

Measure by Failure Severity

An average score can hide rare severe failures. Classify errors by impact:

• Cosmetic or stylistic.
• Recoverable with routine correction.
• Material business error.
• Security, privacy, legal, safety, or financial control failure.

Set separate thresholds for severe categories. One prohibited disclosure may matter more than hundreds of well-written summaries.

Test Adversarially and in the Field

Red-team attempts to bypass instructions, manipulate tools, extract data, exploit retrieved content, and consume excessive resources.

Then run a controlled field test with intended users. Observe workarounds, over-reliance, misunderstandings, review burden, and real exception handling. NIST's evaluation work distinguishes model testing, red-teaming, and field testing because each reveals different behaviour.

Establish a Baseline and Compare Alternatives

Compare the AI workflow with the current process and simpler alternatives. Measure completion, quality, human effort, cycle time, risk, and cost per accepted outcome.

Evaluate component changes against the same test set. A newer model is not automatically better for the business task.

Create Release Gates

Before production, require:

• Quality thresholds on representative and holdout cases.
• Zero unresolved critical control failures.
• Verified user-level permissions.
• Tested approval, escalation, rollback, and kill switch.
• Acceptable latency and unit cost.
• Named business and technical owners.
• Monitoring and incident response.
• Documented model, prompt, data, retrieval, and tool versions.

Record exceptions and the person authorised to accept residual risk.

Continue Evaluation After Launch

Sample production outcomes, review escalations and overrides, watch data and behaviour drift, and re-run regression tests after material changes.

Evaluation is a lifecycle control. Production evidence should improve the test set, while sensitive production data remains governed.

DualByte's IT consulting service can help turn business requirements into evaluation cases, release gates, and production monitoring.

Sources

• NIST: AI Test, Evaluation, Validation, and Verification
• NIST ARIA: Assessing Risks and Impacts of AI
• NIST GenAI Evaluation Programme