How we measure
The protocol is written before the run.
Every number we publish comes from the same protocol. The first half is fixed in the charter before a single question is asked. The second half is what the run reports, with the range the data supports.
Before the run · fixed in the charter
1 · The questions
Your experts write the questions.
Each question comes with the answer your expert would accept and the record a good answer has to cite. Each is tagged with its kind, so the results can be read by kind as well as overall.
Every question is asked three times, because the same model can answer the same question differently. An attempt passes only if it is correct and cited.
What breaks if we rename the STATUS column on the customer table?
Accepted answer, written by your expert
Customer.java and the 14 classes that read it, two PL/SQL packages for billing and accounts, and the nightly active-customer report.
A passing answer must cite
pkg_billing, line 341
active_customers.jrxml, line 54
2 · How many answers
More answers, tighter range.
Fifteen questions asked once can only say the true rate is somewhere between 47 and 93 percent. Forty questions asked three times narrow that to about 18 points.
That is why a pilot asks every question three times, and why the pass mark sits on all 40, not on a sealed handful. Past that, each extra point of precision costs far more grading time than it is worth.
3 · The bar and its odds
A platform at the bar passes half the time.
Any measurement has noise. A platform whose true pass rate is 75 percent will be recorded above 70 about nine times in ten; one at 65, about one time in seven. A platform exactly at 70 passes half the time, under any honest rule.
That is what a bar is for: to separate clearly above from clearly below, not to flatter the ones on it. The charter also says what happens on a near miss, so that rule is fixed before the run too.
After the run · what the report will show
4 · Answers per question
Three attempts, one picture.
Every question is asked three times. Sorting the 36 questions by how many of their attempts passed gives a distribution: a platform worth relying on piles up on the right, at three of three.
The model alone piles up on the left. Same questions, same model.
5 · Where the lift comes from
Not every kind of question gains the same.
Questions are tagged by kind. The ones that cross layers, tracing a screen to a table or asking what breaks if a method changes, are where the model alone struggles most and where anchoring adds the most.
Six questions per kind is too few for a range per row, so the rows show counts, and the caption says so.
6 · What every report contains
Your pilot report uses the same figures.
On your questions and your data, with the bar from your charter. Nothing on this page is a format we keep for marketing.
- The range for each configurationModel alone and anchored, each with its 95% range.
- The gain against the barMeasured against the number in the charter, set before the run.
- Answers per questionHow many questions came out right every time, and how many never did.
- The lift by question kindWhere anchoring helps most, and where it does not yet.
- The full question setEvery question, accepted answer and grade, so your team can check ours.
Bring one question your teams can’t answer reliably today.
In thirty minutes we’ll show you how it would be anchored in your data, and how we’d measure the answer.
No pitch.
A working conversation about your operation.
An honest answer.
If AI isn’t the right tool, we’ll say so.
A clear next step.
If there’s something worth building, you’ll know what the diagnostic covers.