Renzo DemartiniIn short
- Test the work you already run: the same approved cases and files, on both models.
- Set the pass marks before you see an answer, and open every citation behind a decision.
- Hold the switch while a defect is open, keep the old route as the fallback and name who can reverse the change.
A new model scores well on a public benchmark. Your invoices, sales policies and dispatch notes are not in it.
The owner’s decision is narrower: does the work you already trust still pass if the model changes? Test that on your own approved cases, then decide.
Freeze the existing work
Twelve approved cases run twice on each route give 24 outputs to compare
| Item | Agreed value |
|---|---|
| Dataset | Approved pack, version 1 |
| Functions | Finance, sales and operations |
| Cases | 4 per function, 12 in all |
| Runs | 2 per case, on each route |
| Record | Model, plan, date and tools enabled |
Sample. The approved cases and their answers are fictional.
Illustrative figures, not client data.
Start with work you already trust. Keep the approved inputs, expected answers and reviewer notes together. Give both routes the same files and instructions. OpenAI’s evals guide says a test data set should represent ‘the type of data you expect your prompt to work with’. Choose cases that cost something when they are wrong, such as a quote, a supplier reply or a dispatch exception.
OpenAI says models ‘sometimes produce different output from the same input’. Run each case twice, and keep the awkward answers too.
Set the pass marks first
Four pass marks are written down before either model answers
| Check | Required outcome | Evidence kept |
|---|---|---|
| Calculations | Match the approved working | Working shown |
| Citations | Support every claim relied on | Source opened, words matched |
| Access | No record the user may not see | Two denied requests |
| Missing evidence | The answer flags the gap | A source left out on purpose |
Illustrative figures, not client data.
Write the gates before you read either answer. A fluent reply cannot choose its own marking scheme. Anthropic’s documentation says: ‘Most use cases need multidimensional evaluation along several success criteria.’
Include a missing source and an access-denied case. Open every citation behind a decision and compare its words with the claim. Anthropic says citations help you ‘track and verify the sources behind each response’. Mark each gate pass or fail with a reason, so two reviewers reach the same answer. Record the vendor checks in the AI vendor evidence annex.
Keep every correction visible
One sales policy is misstated, so the candidate passes 11 of 12 cases
| Function | Cases | Existing accepted | Candidate accepted | Review minutes |
|---|---|---|---|---|
| Finance | 4 | 4 | 4 | 30 / 20 |
| Sales | 4 | 4 | 3 | 35 / 30 |
| Operations | 4 | 4 | 4 | 25 / 20 |
| Total | 12 | 12 | 11 | 90 / 70 |
Sample figures, not a model test. Review minutes are existing route / candidate. A case passes only if both runs pass.
Illustrative figures, not client data.
The candidate misstates one sales policy, though it cites the right document. The reviewer marks the case failed and keeps the corrected wording beside the original output.
Log every intervention, including prompt changes and manual sums. If you repair the instructions, rerun both routes.
Checking the candidate took 20 minutes less. That is trial effort in hours, not a cash saving. Software and running costs sit outside this test.
Check access and plan the reversal
Two access probes pass, yet one citation failure still holds the switch
| Control | Filled decision |
|---|---|
| Test data | Approved fictional records only |
| Access probes | 2 denied requests, both denied |
| Open defect | 1 citation failure |
| Owner decision | Hold the switch |
| Reversal owner | Operations lead |
| Fallback | Existing route and settings kept |
Illustrative figures, not client data.
Passing the access probes does not repair the citation failure. Keep the existing prompts, settings and files until the owner accepts revised evidence.
Keep personal information out of the trial unless its use has been assessed. The OAIC says ‘Take steps to minimise the amount of personal information that is input into the AI system.’ Get privacy advice for your circumstances.
OpenAI advises running evals ‘on every change’. Re-run the pack when the model, the plan or the workflow changes. Anthropic advises ‘thorough testing of your applications with the new models well before the retirement date’.
What to do first
Twelve cases and four pass marks turn a model change into a decision
- 01Pick 12 approved cases
Four each from finance, sales and operations, with the answers already agreed.
- 02Write the pass marks
Calculations, citations, access and missing evidence, before any answer is read.
- 03Run both routes twice
Same files, same instructions. Keep every output and every correction.
- 04Decide on a dated record
Name the owner, the open defect, the fallback and who can reverse the change.
One page and twelve cases are enough to start. Run the pack before you renew or change a plan.
Common questions
Which is better, Claude or ChatGPT?
Neither wins for every business. Test your own approved cases against agreed answers, because public scores measure other work.
Can a benchmark settle the switch?
No. OpenAI separates industry benchmarks for comparing models in isolation from specific tests of your own application. Your decision rests on the second kind.
Should both routes stay available?
Keep the approved fallback until the pass marks are met. Then re-run the pack whenever the model, the plan or the workflow changes.
Can customer data go into the test?
Use approved fictional records first. Personal information needs an assessment and privacy advice, and the OAIC advises minimising it.
Chartered Accountant (CA ANZ) and finance executive, based in Canberra. Practical AI and automation for SMEs, and adviser to owners, CEOs and boards on integration and finance.