Renzo DemartiniIn short
- Write each measure and name its evidence owner before anyone asks the AI a question.
- Reconcile known answers to source records, and keep the first answer beside the corrected one.
- Approve one bounded use, and re-run the pack when the data, the definition or the tool changes.
An AI tool answers a sales question in seconds, and the answer reads well. Is it right?
NIST’s generative AI profile warns of ‘erroneous or false content’ by which users ‘may be misled or deceived’. The owner needs proof before the answer reaches a decision: the measure it used and the records behind it. Three checked answers show how.
Agree what each answer means
Three answers each have a written measure and a named owner before anyone asks
| Function | Approved measure | Evidence owner |
|---|---|---|
| Finance | Invoiced revenue less credits, excluding GST | Finance lead |
| Sales | Won opportunities ÷ closed opportunities | Sales lead |
| Operations | Orders delivered by the promised date ÷ orders due | Operations lead |
Illustrative figures, not client data.
Define the measure before you ask the question: its period, its exclusions and its source version. Here, invoiced revenue is a management measure, not an accounting policy.
A convincing explanation cannot settle a disputed definition, so each evidence owner approves the measure first. That is the person who can say what a correct answer looks like. Microsoft says preparing the data gives Power BI Copilot ‘the context it needs to reduce ambiguity’.
Reconcile three known answers
Two of three first answers match the approved working, and sales does not
| Case and source records | Approved working | First AI answer | Reviewed answer |
|---|---|---|---|
| F1: invoice and credit registers, v1 | A$200,000 − A$10,000 | A$190,000 | A$190,000 |
| S1: closed-opportunity register, v1 | 12 wins ÷ 40 closed | 40% | 30% |
| O1: dispatch register, v1 | 45 on time ÷ 50 due | 90% | 90% |
Sample, October, excluding GST. Sales used 30 opportunities with recorded contact. The approved denominator is all 40 closed.
Illustrative figures, not client data.
The sales answer divided 12 wins by 30 opportunities with recorded contact. The approved denominator is all 40 closed, which gives 30%. The correction changes the conclusion without changing a sale.
Choose known answers: figures you have already checked against the records, so a wrong answer is visible. Keep the first answer, its query and its filters. Microsoft says you can ‘verify the calculations and filters used’ behind a Copilot answer. Then keep the corrected query and the reviewer’s explanation. The board-pack evidence bridge connects accepted answers to the decision paper.
Budget the checking time
Checking three answers took 40 minutes, with one correction
| Case | Correction | Review minutes | Accepted by |
|---|---|---|---|
| F1 | None | 10 | Finance lead |
| S1 | Denominator corrected | 20 | Sales lead |
| O1 | None | 10 | Operations lead |
| Total | 1 correction | 40 | Owner receives the pack |
Illustrative figures, not client data.
Forty minutes is review effort for this pack. It is hours of checking, not time saved, and no cash saving or payback is claimed. Software and running costs are outside the sample.
Time the review, the correction and the re-run separately, and count checking apart from time spent fixing source records. NIST’s framework asks that assurance criteria are ‘demonstrated for conditions similar to deployment setting(s)’ and documented. Anthropic says ‘Most use cases need multidimensional evaluation along several success criteria.’ Score correctness, traceability and review effort.
Approve one use, not the system
The owner approves an internal planning trial and defers automatic purchasing
| Field | Filled record |
|---|---|
| Decision owner | Business owner |
| Scope | Internal planning trial only |
| Acceptance | 3 reviewed answers approved, 1 after correction |
| Deferred use | Automatic purchasing changes |
| Stop trigger | Definition, source or access changes |
| Next check | Before the next planning decision |
Illustrative figures, not client data.
Approve a use, and keep the evidence pack with the decision. OpenAI advises continuous evaluation that runs on every change, so re-run the pack when the data or the tool changes.
Name who can stop the trial and what triggers it. Next time, add a missing record and a question outside the approved measures. A good answer says it cannot answer. It does not invent an explanation.
Check access before adding sensitive data. The OAIC advises: ‘Take steps to minimise the amount of personal information that is input into the AI system.’ Get privacy advice for your obligations.
What to do first
Three measures and three known answers are enough for a first acceptance pack
- 01Write three measures
One each for finance, sales and operations, with period, exclusions and owner.
- 02Pick three known answers
Questions whose answers you have already checked against records.
- 03Reconcile and log
Keep the first answer, the query, the correction and the review minutes.
- 04Approve a bounded use
Name the owner, what is deferred and what stops the trial.
Pick the report your board reads most often.
Common questions
What is AI business intelligence?
It adds AI to the analysis of business data. The risk is a plausible wrong answer, so each answer needs checking against the measure and the records behind it.
Does a correct total prove the explanation?
No. Check the calculation and the definition separately. A right number can sit beside a wrong explanation.
Will AI replace the person checking reports?
No. Someone stays accountable for the measure and the decision. OpenAI advises combining evaluation metrics with human judgement.
Chartered Accountant (CA ANZ) and finance executive, based in Canberra. Practical AI and automation for SMEs, and adviser to owners, CEOs and boards on integration and finance.