An invoice-checking pilot, from baseline to decision.

Use this guide to: plan a bounded AI workflow and measure the accepted result.

Start with the decision, not the model

A small operations team wants to spend less time preparing incoming invoices for approval. The useful output is a checked invoice record with supporting evidence and a clear exception status. A fluent summary is not enough. The reviewer must be able to trace the supplier, invoice number, currency, amounts and payment references back to the original document.

The pilot prepares a draft and flags inconsistencies. It does not release a payment, change supplier bank details or approve its own work. Every invoice remains subject to human review and the team's existing approval process.

Define what the workflow actually does

Begin with one bounded document type and one approval queue. The system reads the document, drafts a structured record and links each extracted value to the relevant page or region. Deterministic software checks arithmetic and required fields. Available purchase-order and receipt records can support matching when those records are actually supplied.

Missing information stays missing. A supplier name that resembles an existing record is a candidate match, not a verified identity. A changed payment destination, conflicting currency or unexplained mismatch goes to an exception queue. Document text is treated as source data, not as instructions that can change the workflow's permissions.

A completed task means that a reviewer has checked the record, resolved or escalated the exception and recorded the disposition. Payment authorization is a separate decision outside this pilot. Measure the whole preparation-and-review path against the same completion rule used for the manual baseline.

Write down the assumptions

Assume 1,000 invoices per month. Manual preparation and checking take three minutes per invoice. Use an assigned labor cost of $30 per hour. Workflow A has a lower processing charge but sends more invoices through additional exception handling. Workflow B costs more to process each invoice but has fewer assumed exceptions.

Scroll the table sideways on a narrow screen.

The processing charge is an assumed all-in variable amount covering extraction, model calls and expected retries. The $200 is an additional non-labor allowance for operating the proposed workflow, not a quoted subscription. Human exception time is additional to the initial review of every invoice. Baseline time already includes the current process's exception work, so it must not be added a second time.

The 0.60-minute assumption means 36 seconds of initial human work. It is an input to test, not a claim that this review speed will be adequate. If evidence checks require more time, replace the assumption before making a commitment.

Calculate the cost of the whole path

The manual baseline is 1,000 × 3 ÷ 60 = 50 hours, or $1,500 of assigned labor cost. Workflow A needs 10 hours for initial review plus 350 × 2 ÷ 60 = 11.67 hours for exceptions. Workflow B needs the same 10 initial-review hours plus 150 × 2 ÷ 60 = 5 exception hours.

Scroll the table sideways on a narrow screen.

B's processing charge is $20 higher per month, but its assumed exception work costs $200 less. Its total assigned cost is therefore $180 lower than A's. The cheaper model-processing unit does not identify the cheaper completed workflow.

These per-invoice figures assume all 1,000 records reach the stated completion standard after review. If some remain unresolved, report the unresolved count separately and divide cost by the records actually completed. Do not count an unreviewed draft as an accepted outcome.

The $640 and $820 differences from the baseline are differences in assigned monthly cost, not demonstrated cash savings. Released staff time may be used elsewhere without reducing payroll. Integration, initial testing, training, security review, setup, the cost of errors and any other omitted costs still need their own budget. This comparison cannot establish profitability.

Test quality before trusting the cost

Use a labelled set of historical documents and compare the proposed workflow with the existing process against the same reference decisions. Include normal documents and difficult cases: scans, credit notes, duplicate invoice numbers, mismatched amounts, missing references and changed payment details. Keep a held-out set that is not used to tune the prompts or rules.

NIST AI RMF 1.0's MEASURE 2.1 and 2.3 call for documented evaluation methods and assessment in conditions relevant to deployment. Our application here is to test the complete human-and-software workflow, record its errors and compare it with an explicit baseline. The controls and assumptions in this article are AI Keeper's teaching design, not a NIST-prescribed invoice process.

Record field-level correctness, missed discrepancies, unnecessary escalations, unresolved records and end-to-end completion time. Keep critical payment-detail errors separate from ordinary formatting corrections. A single average accuracy percentage can hide a failure that matters to the operating decision.

The pilot owner should agree on acceptance rules with the reviewers before running the held-out test. An unapproved payment action or an unexplained payment-detail change pauses the pilot in this example. That is an operational choice for this constructed workflow, not a universal legal or statistical standard. A clean small sample does not prove that rare errors cannot occur.

Change the assumption that could reverse the decision

Keep B's other inputs unchanged but increase initial human review from 0.60 to 2.00 minutes per invoice. Initial review now costs $1,000. Add $150 of exception work, $30 of processing and the $200 allowance: the total becomes $1,380, or $1.38 per completed invoice.

Only $120 of assigned monthly cost separates that scenario from the manual baseline, before implementation and other excluded costs. The next useful measurement is therefore the review time needed to meet the acceptance rules. A lower API price cannot answer that question.

Repeat the comparison with observed exception rates and actual processing charges. When volume changes, separate fixed allowances from variable costs. Report capacity released, cash expenditure and quality outcomes on different lines.

Make the next commitment small and explicit

Run the first evaluation in shadow mode: the normal process remains authoritative while the proposed workflow prepares drafts for comparison. Restrict access to the documents needed for the test, choose a responsible reviewer and define how errors are logged. Use approved handling arrangements for any personal, commercial or payment information in the documents.

The decision at the end is whether to continue, narrow the scope or stop. Continue only when the evidence supports both the acceptance rules and an affordable operating plan. A rollout should inherit a monitoring owner, an exception queue and a way to return to the manual process.

Download the pilot scorecard CSV ↓

The scorecard contains the illustrative inputs and calculated results, with a separate blank column for observed pilot values. It is a record sheet, not an automatic spreadsheet calculator. Use it to expose the next missing measurement before building a larger forecast.