Licences activated and tokens spent are not productivity. They are spend, and most businesses are still measuring the wrong one.
Two point three million conversations in a single month. That was the headline number Klarna, the Swedish payments company, put out in February 2024: an AI assistant doing the work of roughly 700 full-time support agents.
Handle time fell from eleven minutes to under two. Repeat inquiries dropped by a quarter.
Fifteen months later, Klarna started hiring humans back. Its chief executive told Bloomberg why, in May 2025: cost had been "a too predominant evaluation factor," and the result was lower quality.
Read the first paragraph again. Every figure in it describes volume. None of it describes whether the customer's problem actually got solved.
That's the trap sitting in your own dashboard too, whether or not you had heard of Klarna before this piece.
You are counting activity. You should be counting output.
Logins, conversations handled, tokens spent: all of it measures how hard the tap is running. None of it measures what filled the bucket. That's not a productivity metric. That's a very busy tap.
The confusion is not unique to one company. MIT's Project NANDA studied enterprise AI spending broadly and published its findings in July 2025: despite $30 billion to $40 billion in combined enterprise investment, 95 percent of organisations were getting no measurable profit-and-loss impact from their AI systems.
Adoption was never the hard part. Over 80 percent of the organisations MIT surveyed had piloted a generalist tool. The share with a measurable result attached stayed near 5 percent.
That gap is not a technology problem. It is a measurement problem.
A vendor can show you your login count inside their own product on the day you sign up. They cannot show you which report used to take three people two days and now takes one person an afternoon, because that number lives in your operations, not theirs.
So build it yourself, or admit you have none.
Pick five tasks your business does every week. Time them now. Time them again next quarter. A number that moves once and then goes quiet was a demo, not a result.
Klarna's own numbers looked like proof in February. Its customers said otherwise by May.
So before your next renewal lands on your desk: what did the spend actually build, and would you know without checking the licence count first?