Sample report

Pre-launch test of an AI assistant: a sample report

What the $12,500 pre-launch test hands over, filled in for a made-up assistant: the Quillhaven Lending Library's, built from the teaching data of the course Test it before you trust it.

The library, its eight documents, both versions of its assistant and every answer are made up for teaching. No real assistant produced them, and the numbers say nothing about any real product. In a paid test the questions come from your own documents and the answers from your own assistant.

In short

Decision: do not launch yet. Version v2 meets 3 of the 5 launch criteria the team wrote down before any answer was recorded. The rule agreed in advance was to launch only if all five are met.

80 of 100 held-out questions answered fully correctly

12 of 18 questions the documents do not cover, answered anyway

33 of 120 claims the cited passage does not support

v2 is more accurate on rules with exceptions (12 of 14 against 4 of 14), but it makes up answers and adds detail its sources do not hold. Fix those two habits, then test again.

The launch criteria

Written on 2 March 2026, before any answer was recorded, so that the result could not move the bar.

C1: Answers fully correct, lower end of the 95% interval pass
Needs at least 70%. Measured: 71% (80 of 100 correct; interval 71% to 87%).
C2: Says “not in the documents” when that is true fail
Needs at least 90%. Measured: 33% (6 of 18).
C3: Claims the cited passage does not support fail
Needs at most 5 in 100. Measured: 27.5 in 100 (33 of 120 claims).
C4: The two graders agree (Cohen's kappa) pass
Needs at least 0.60. Measured: 0.67 (200 answers).
C5: Planted bad answers caught pass
Needs all. Measured: 12 of 12.

The test set

150 questions, written between 2 and 20 February 2026, before either version answered anything.
100 held out for the decision, never used to tune the assistant: 82 that the documents answer (40 asking for a figure, 28 for a fact, 14 for a rule with an exception) and 18 that they do not.
60 written by the project team, 40 written down as members asked them at the front desk, spelling and all.
Every answer marked by two graders on a written rubric, then settled where they disagreed: 2 correct, 1 partly correct, 0 wrong.

The paid test writes 100 questions from your documents, in the same way.

Every answer, scored

Right, partly right, wrong or declined, on the 100 held-out questions. A decline is right only when the documents really do not say.

Round one: v1, recorded 9 March 2026 75 of 100 right
On the 82 the documents answer: 60 right, 9 partly right, 9 wrong, 4 declined. On the 18 they do not cover: 15 said so (right), 3 answered anyway (wrong).
Round two: v2, recorded 23 March 2026 80 of 100 right
On the 82 the documents answer: 74 right, 4 partly right, 4 wrong, 0 declined. On the 18 they do not cover: 6 said so (right), 12 answered anyway (wrong).

The source check

Each answer was cut into its claims, and each claim was checked against the passage the answer cited.

v2: 120 claims. 87 supported, 27 not supported, 6 contradicted. That is 27.5 in every 100 claims not supported by the cited passage.
v1: 86 claims. 80 supported, 4 not supported, 2 contradicted: 7.0 in every 100.

Three of v2's, as the report shows them:

When is the Wi-Fi switched off? contradicted
The answer said: “The Wi-Fi is switched off at closing time.” The passage it cited: “The connection is filtered and is switched off 15 minutes before closing.”
How long are room booking records kept? contradicted
The answer said: “Room booking records are kept for 30 days.” The passage it cited: “Room booking records are kept for 12 months.” (Camera recordings are the ones kept for 30 days.)
Where can I renew a loan? unsupported
The answer said: “You can also renew by text message.” The passage it cited: “Renew online, by phone or at any desk.” Text messages are not mentioned.

Round two against round one

Fully correct: v2 80, v1 75. The gap is +5 points with a 95% interval of about −7 to +17, so this test does not show that v2 is more accurate overall.
Rules with exceptions: v2 12 of 14, v1 4 of 14.
Questions the documents do not cover, answered anyway: v2 12 of 18, v1 3 of 18.
Claims not supported by the cited passage: v2 33 of 120, v1 6 of 86.
The graders: Cohen's kappa 0.67 over 200 answers. All 12 planted bad answers were caught.

A monitoring plan for after launch

What to measure once real people use the assistant, how often, and the level that calls for a review. The levels come from the launch criteria.

Every week review if
Draw 50 questions members really asked. Two people grade them on the same written rubric. Review if: The share fully correct, with its interval, falls below the launch bar.
Every week review if
Check every claim in those 50 answers against the passage it cites. Review if: More than 5 claims in 100 are not supported.
Every month review if
Ask 10 questions the documents do not cover. Review if: Fewer than 9 get a plain “not in the documents”.
Every month review if
Plant 5 known-bad answers in the grading batch. Review if: Any one of them is not caught.
Before every change review if
Run the 100 held-out questions again before a new model, a new prompt or changed documents go live. Review if: Any launch criterion is missed.

What was not measured

Follow-up questions and conversations of more than one turn.
Questions that need two passages to answer.
Languages other than English, and questions asked by voice.
Documents changed after 1 February 2026.
Attempts to make the assistant misbehave on purpose.
Speed and cost.
Whether members find the answers useful. No member has used this version.

Fix how v2 handles questions the documents do not cover, and its habit of adding detail that is not in the source.
Write at least 30 new questions the documents do not cover. Eighteen is too few to confirm a 90% rate.
Test again on the held-out set plus the new questions, and report the new questions separately.

The offer