Duplicate Detection by Design: Three Layers and a Human Decision

Duplicates are one of the quiet failure modes of bookkeeping. Post the same invoice twice and your costs are overstated, your profit understated, your tax figures wrong in a direction HMRC tends to notice, and — if the duplicate flows through to payment — a supplier may be paid twice. None of this announces itself. It sits in the books looking plausible until someone reconciles carefully or asks an awkward question.

Multi-channel ingestion makes the problem worse before it makes anything better. Our platform accepts evidence by upload, by read-only email connection, by browser extension from vendor portals, and by WhatsApp or Telegram. That convenience has an obvious consequence: the same invoice can genuinely arrive three times — once as an email attachment, once photographed and messaged over, once pulled from the supplier's portal. A system that ingests from everywhere must be serious about deduplication, or it is a duplicate-generating machine with good marketing.

Here is how ours works, and the one design rule that governs all of it.

Layer 1: file hash — the same bytes twice

The cheapest check comes first. Every incoming file is hashed, and the hash is compared against everything previously ingested for your organisation. If the identical file arrives again — same PDF forwarded from two mailboxes, same photo uploaded twice — the match is exact, mechanical, and free of judgement.

Even here we do not silently drop the second copy. The duplicate file is recorded as received, linked to the original, and surfaced. There is information in the fact that a document arrived twice — it tells you two channels are carrying the same traffic — and quietly binning evidence, even redundant evidence, is not a habit a bookkeeping system should have.

Layer 2: draft-level detection — the same transaction taking shape

File hashing misses the interesting cases, because the same invoice rarely arrives as the same bytes. A scanned PDF and a phone photo of one document hash differently. So the second layer watches the drafts themselves: when the AI proposes an entry, it is compared against existing entries and pending drafts for tell-tale agreement — same counterparty, same amount, same or adjacent dates, same reference where one exists.

This is deliberately a suspicion layer, not a verdict layer. Two transactions can legitimately look identical: a tradesperson buying the same fixings twice in a week, two identical coffees on a client day, a supplier's flat monthly charge. Draft-level matching exists to raise its hand, not to swing an axe.

Layer 3: invoice content hash — the same document in different clothing

The third layer targets the classic multi-channel case directly. From each extracted invoice we derive a content hash built from what identifies the document itself — the supplier, the invoice number, the total — rather than from the bytes of the file that carried it. The emailed PDF and the WhatsApp photo of the same invoice produce different file hashes but the same content hash, and the match survives the document arriving in different formats, resolutions, and lighting conditions.

Content hashing catches what file hashing cannot, and it does so deterministically — no model in the loop, so a match is a reconstructable fact, not an opinion.

The rule that governs all three layers: detection is automated, disposal is human

Whatever layer raises the flag, the outcome is the same: the suspected duplicate is held. It is not auto-posted, because posting a duplicate corrupts the books. It is not auto-discarded, because discarding a false positive destroys a real transaction — and of the two errors, silently deleting genuine business activity is the one you cannot see afterwards.

A person — your accountant, working the review queue — sees both candidates side by side: documents, extracted details, proposed entries, and which layer matched them. Then a human decision is made and recorded. Genuinely the same? One entry posts, both pieces of evidence stay attached to it. Genuinely different? Both post, and the pairing is dismissed with the reason on the record. Unclear? It becomes a query to you: "Did you buy from Howdens twice on the 14th, or is this one document twice?"

Every one of those decisions lands in the audit trail with who decided, what they decided, and when. If anyone later asks why two similar-looking transactions both stand in the ledger, the answer is not a shrug — it is a named person's recorded judgement with the evidence attached.

Why we think this is the right trade

A fully automatic deduplicator would look better in a demo. It would also be quietly making irreversible evidence decisions on the strength of pattern matching, and the failure cases — recurring identical charges are common in real businesses — are exactly the ones automation gets wrong with confidence. Our design accepts a small human cost (a held item takes a moment of review) to avoid an unbounded silent one.

It is the same philosophy that runs through the whole platform: AI drafts. AgentLedger validates. People approve. Machines are excellent at noticing that two things look alike. Deciding what that resemblance means for your books is judgement, and judgement is a job we give to people, on the record.

Check eligibility / Ask for Details