Back to blogs

    AI Freight Audit Agent: Role of Data in Audit Decisions

    By TransportOne Editorial Team•October 21, 2026•7 min read
    A Hapag-Lloyd shipping container in a container yard

    Your freight audit platform reports that 71 percent of invoices went straight through last month. That number tells you how often nobody looked. It can't tell you how often nobody needed to.

    The distance between those two readings is where the money sits, and it's the central problem of AI freight audit with incomplete data. It gets sharper once you hand the work to autonomous AI agents: software that takes a goal, works through multi-step tasks across your systems and acts with limited human review. An invoice that cleared because every charge matched the contract is a win. An invoice that cleared because the system couldn't resolve the load and quietly took the carrier's figure is an overpayment. You'll never go looking for it, since nothing in your reporting records that it happened.

    The touch rate rewards guessing

    Straight-through processing is the number the whole accounts payable profession is graded on. Ardent Partners' Accounts Payable Metrics that Matter in 2026, as summarised by Basware, puts best-in-class teams at 55.2 percent of invoices moving through with no manual touch, against 31.4 percent for everyone else. Measuring it is fine. The trouble starts when it becomes the target.

    A touch rate with no error budget measures coverage and gets read as quality. You can lift it any afternoon. Widen a tolerance, or let a fuzzy match count as a match, or tell the system to take the closer of two plausible loads. Each move reads as progress on the dashboard, and each turns a visible exception into an invisible payment.

    Why language models make it worse

    In September 2025, Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang argued in Why Language Models Hallucinate that models confidently produce plausible falsehoods because training and evaluation reward guessing over admitting uncertainty. Most benchmarks grade in a way that penalises an uncertain response, so a system optimised to score well learns to answer anyway.

    Point something shaped by that pressure at a freight invoice and grade it on touch rate. You've reproduced the same scoring flaw inside your back office.

    A model's own confidence score won't catch it

    The standard reassurance for AI agents in logistics is a threshold. Let the agent attach a number to each extraction, approve above 0.9, and route the rest to a person. That only works if the number means what it appears to mean, and on invoice documents it often doesn't.

    What Spotify found

    Spotify's engineering team published a case study in December 2024 on building confidence scores for exactly this job: parsing vendor invoices that arrive in many languages and layouts, under ITGC and SOX review obligations. They tested averaged token log probabilities and found no clear correlation between that score and accuracy. Majority voting across several models did correlate. Even so, they ran Platt scaling over the result before anyone was allowed to trust it.

    What a calibrated score changes

    A measured version of the same finding landed last month. Yichao Jin and colleagues, in Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents (September 2026), evaluated vision language models on real invoices, synthetic invoices and ad-buy forms. The models' self-reported confidence scored 0.54 to 0.74 AUROC, where 0.5 is a coin toss. Their decomposed score, built from perception, layout and validation signals, reached 0.90 to 0.99.

    The next pair of figures shows what that means in operation. Holding the target error rate below 10 percent, the models' own confidence cleared between 0.1 and 7.0 percent of fields for automatic approval. The calibrated score cleared 49 to 72 percent. Same documents, same extraction. The defensible automation rate moves by an order of magnitude depending on what sat behind the threshold, so a touch rate quoted without its error budget isn't something you can act on.

    AI freight audit with incomplete data needs a third outcome

    Most audit systems offer two exits: approve or dispute. Whatever fits neither gets pushed through whichever door is cheaper to operate, and in practice that's approve. A third exit fixes it, provided it's a named result with reasons attached. A dumping ground won't do.

    In pre-payment freight invoice audit, Penny treats needs data as a decision. She reaches it when the freight itself withholds the answer:

    • A detention charge arrives with no gate-in or gate-out timestamp from either party. Nothing in the file establishes who held the truck, which is the one question a detention charge has to answer.
    • An accessorial is billed under a code that appears in no contract version you hold, because the amendment adding it was signed regionally and never loaded.

    When the data conflicts

    Conflicting data is harder than missing data, because the agent has something to pick. Take a PRO number that resolves to two loads, both moving that week, on that lane, for that shipper. The candidates differ by a few hundred dollars and one scores marginally higher on the match. Selecting it would nudge the touch rate up.

    It would also bind an audit result to the wrong shipment, the failure described in why load resolution is the freight audit bottleneck. Choosing is worse than declining, and only one of the two shows up on any report. In the invoices we work through, the ambiguous match tends to look most like a clean one.

    Restraint has to be built

    Don't expect the model to supply that restraint unaided. AbstentionBench, released by FAIR at Meta in June 2025, tested 20 frontier models across 20 datasets of unanswerable, underspecified and false-premise questions, and concluded that abstention remains unsolved. Scale didn't help: larger models were no better at declining. Reasoning fine-tuning actively hurt, degrading abstention by 24 percent on average. You construct the holding back around a model, from rules and evidence checks, plus a queue where the answer has somewhere to land.

    What Should an AI Agent Include in Its Results?

    A queue of items reading "manual review required" costs more than no queue at all, because an analyst now rebuilds from nothing the work the agent already did. The trace is the deliverable.

    What the item settlesBare exceptionNeeds data with a trace
    The questionManual review requiredThe single fact that would resolve it, in one sentence
    What was consideredNothing shownEach candidate, with the fields that agree and the fields that don't
    EvidenceThe analyst re-pulls every documentRate confirmation, bill of lading and delivery receipt already attached
    StakesUnknown until someone opens itThe dollar spread between the readings, against the invoice total
    OwnerThe queueThe named party holding the missing artefact
    DeadlineWhenever somebody noticesA stated date, after which the item pays or disputes by rule

    That final row carries more weight than it looks. An unresolved item that sits forever is a slower way of approving by default, and payment terms don't pause for your backlog. Every item needs a clock and a written rule for what happens when it expires.

    Four numbers that replace the one

    Retire the standalone touch rate and report these instead, with a standing rule that the first is never quoted on its own.

    • Auto-approval rate at a stated error budget. "71 percent" is unfalsifiable. "71 percent, at a measured error rate under 1 percent on sampled review" is a claim somebody can test.
    • Silent error rate. Pull a random sample of auto-approved invoices each week and audit them by hand. Nothing else tells you what the automation cost. A team that never samples has no measurement at all, whatever its dashboard shows.
    • Resolution time, split by reason. Items waiting on a carrier and items waiting on your own contract repository are different problems with different owners. Reporting them together hides both.
    • Share of items answered by a document you already held. When that fraction runs high, the shortfall is plumbing, and plumbing is cheaper to repair than uncertainty.

    The fourth metric is the cheapest to act on. A missing contract amendment is one example. So is a POD sitting in a carrier portal nobody connected, or a rate table loaded for one business unit and not the other. None of those needs a better model, and each one raises the honest automation rate. That's the practical reason AI agents need accurate data: an agent can only clear what the file lets it prove. If your business units sign carrier amendments separately, check that gap first. It applies directly to freight audit for consumer goods shippers.

    Thirty invoices will tell you more than the dashboard

    Is your 71 percent real? Pull 30 of last month's automatically approved freight invoices at random and audit each one by hand against the contract and the execution evidence.

    Sort what you find into two piles. In one go the charges that were correct. In the other go the charges that passed because nothing in the system could tell either way. That second pile shows your actual automation rate, and it won't match the dashboard.

    Then book a walkthrough with the same 30 invoices. We'll run them through Penny and show each needs-data item the way your analysts would receive it, with candidates, evidence, owner and clock on the item.

    Frequently asked questions

    How do AI agents improve logistics operations?
    In freight audit, an agent like Penny checks each invoice against the contract and the execution evidence before payment, so your analysts only see the items that need them. The gain is real only when the agent reports what it couldn't verify instead of approving on a guess. Measure it by the auto-approval rate at a stated error budget, plus a weekly hand-audited sample of approved invoices, not by touch rate alone.
    Why do AI agents need accurate data?
    An agent can only approve what the file lets it prove. When a gate timestamp is missing, a contract amendment was never loaded or a PRO number resolves to two loads, the agent either guesses or stops. Guessing turns a visible exception into a silent overpayment, so fixing missing documents and contract data often raises your honest automation rate more than a better model would.
    How are AI agents different from traditional logistics automation?
    Traditional automation follows fixed rules and usually offers two exits, approve or dispute, so anything that fits neither tends to get approved. An AI agent works through multi-step tasks across your systems, such as resolving the load, checking the contract and testing the evidence. A well-built one can also return a third outcome, needs data, which names the missing fact, attaches the evidence gathered and assigns an owner and a deadline.
    What should an AI freight audit agent do when data conflicts?
    It should decline to pick and show the candidates. If a PRO number resolves to two loads on the same lane in the same week, choosing the closer match can bind the audit to the wrong shipment. The agent should list each candidate with the fields that agree and the fields that don't.
    Can I trust an AI model's confidence score for freight invoices?
    Not on its own. Spotify found no clear correlation between averaged token log probabilities and accuracy on vendor invoices. Research published in September 2026 found self-reported confidence scored 0.54 to 0.74 AUROC, while a calibrated score built from perception, layout and validation signals reached 0.90 to 0.99.

    See it on your freight

    Stop Overpaying On Freight Invoices

    Penny checks every invoice against your contracts and what the shipment actually did, then disputes what doesn’t hold up. You pay 30% of what she saves you, and nothing if she saves nothing.

    See how Penny works

    Found this useful? Share it with your team.

    Share on LinkedIn