Evaluating AI Bookkeeping Software: A Buyer's Checklist

Every AI bookkeeping product demos well. The model reads a clean invoice, the entry appears, the audience nods. Demos are selected for success — so a demo tells you almost nothing, because the questions that matter all begin the same way: what happens when the AI is wrong?

Here is the checklist we would use to evaluate any product in this category, written so you can apply it to anyone — including us. For each item: the question to ask, why it matters, and what a good answer sounds like.

1. Is there a validation layer separate from the model?

Ask: What sits between the AI's output and my books? What are the invariants, and can the model bypass them?

Why it matters: A probabilistic drafter writing directly into a ledger means every model failure becomes a bookkeeping failure. A deterministic validation layer — one with no AI in it — turns model failures into visible, rejectable events. In our case that layer is AgentLedger: every transaction must balance to zero, reference accounts that exist, and carry a sane date, or it is rejected with a reason. Rejected, note — not "auto-corrected". A validator that repairs is a validator that guesses.

Good answer: Named invariants, deterministic checking, rejection with machine-readable reasons. Bad answer: "The model is very accurate." That is a claim about the drafter, offered in place of an answer about the safety layer.

2. Who, by name, is accountable for each entry?

Ask: For any entry in my ledger, can you show me the human who approved it — or the explicit policy under which no human did?

Why it matters: "Human-in-the-loop" is marketing until it is specific. The workable standard: every entry is approved, corrected, or queried by an identifiable person under an agreed review policy, and anything that posts automatically does so under an explicit, versioned policy with confidence and materiality thresholds someone signed off. Vague "our team reviews activity" answers describe a vibe, not a control.

Good answer: A named reviewer per entry, three recorded verbs (approve / correct / query), and written per-account automation policies. Ask to see the policy screen, not the philosophy.

3. How complete is the audit trail?

Ask: Show me, for one entry: the source document, the AI's draft as proposed, its confidence score, every edit and editor, the approval, and — if it auto-posted — the policy that allowed it.

Why it matters: When HMRC or an auditor asks why an entry exists, "the AI did it" is not an answer. The tell is whether the vendor keeps the draft before correction: the difference between proposed and approved is the proof that review actually happens. Systems that store only final entries can claim any amount of oversight; systems that preserve drafts can demonstrate it.

Good answer: The full chain, reconstructable per entry, kept permanently. Bonus points if they will show you a rejected draft and a duplicate held for a human decision — the unglamorous records are the honest ones.

4. How is access controlled — especially for AI integrations?

Ask: How do integrations and AI agents authenticate? Are tokens scoped? Can I see and revoke every grant? Is email access read-only? Is every external call logged?

Why it matters: AI features multiply the connections into your financial data, and each connection is a place things can go wrong. Least privilege is the standard: scoped OAuth tokens rather than shared passwords, read-only wherever writing is not essential, one-click revocation, and a log of every call. For AI agents specifically, one boundary is worth demanding outright: no external agent should be able to post to the ledger — the most it should do is propose drafts that go through the same validation and human approval as everything else.

Good answer: OAuth 2.1, scoped tokens, visible grants, revocation, per-call logging. Bad answer: anything involving storing your email password.

5. Can you leave with your books?

Ask: If I go, what do I get, in what format, and does it mean anything without your software?

Why it matters: Export freedom is where a vendor's confidence shows. Books held as opaque database rows exportable only as summary PDFs are a hostage situation with a dashboard. A plain-text double-entry ledger plus your original source documents is a complete record another accountant or another tool can actually use.

Good answer: Open, documented format; originals included; no exit fee for your own data. A vendor should want to keep you by being good, not by being difficult to leave.

6. Do the claims survive contact with the documentation?

Ask: For every impressive number or adjective, where is the method behind it?

Red flags worth naming: accuracy percentages with no stated methodology; "fully autonomous" bookkeeping (autonomy in accounting is a liability wearing a cape); "guaranteed" refunds or savings (outcomes depend on your facts, and nobody controls HMRC); and "HMRC approved" (HMRC lists software as compatible with Making Tax Digital — see the GOV.UK software list — it does not approve anyone's bookkeeping judgement). A vendor honest about limits — where the AI is weak, what still needs a person, what could go wrong — is telling you they have actually operated the thing.

7. What is the correction loop?

Ask: When I dispute an entry, what happens, and does the system learn where it is weak?

Why it matters: Every model miscategorises sometimes. What separates products is whether corrections are recorded against the original drafts, whether correction patterns inform review thresholds, and whether duplicates are held for a person rather than silently auto-resolved in either direction.

The one-sentence version

Buy the workflow, not the model: the model will be wrong on the day it matters, and on that day the validation layer, the named human, the audit trail, and your export rights are what you actually own. Our version of those answers is one sentence long — AI drafts. AgentLedger validates. People approve. — and we would rather you test it against this checklist than take it on faith.

Check eligibility / Ask for Details