The Meter Is Always Running

Another weekend, another blog post. For those who read these, thank you, and for those who don't, no dramas, I've long since accepted this is cheaper than therapy and I get to keep the receipts.

So this week I want to talk about a mistake. Not one of mine for once, which makes a nice change, but a couple of very expensive ones made by companies far bigger and smarter than me. And the reason I want to talk about them is that underneath all the noise they're actually a story about something I've been wrestling with on my own little projects at home.

Let me set the scene.

In May, Amazon quietly switched off an internal leaderboard called Kirorank. It ranked employees by how many AI tokens they burned through. The intent was honourable enough, a bit of gentle nudging to get people using their shiny new tools. What actually happened is that staff worked out they could climb the board by firing pointless busywork through AI agents, running up the compute bill while solving precisely nothing. Someone even coined a word for it: tokenmaxxing. Eventually a senior Amazon VP had to stand up and tell everyone, more or less, to stop using AI just for the sake of using AI.

Now here's the bit that made me smile. Amazon wasn't even first. Meta had already killed off its own version, a leaderboard reportedly called Claudeonomics, a month earlier, for exactly the same reason. Two of the smartest engineering shops on the planet, completely independently, built the same well-meaning scoreboard and completely independently watched it get gamed into a bonfire of cash. If that doesn't make you feel a bit better about your own mistakes, I don't know what will.

And it wasn't just the leaderboards. Uber's CTO went a bit viral for admitting the company had blown through its entire annual Claude Code budget by April. Their COO then followed up, rather drily, to point out that all that spending hadn't produced gains anywhere near the size of the bill.

It would be easy to look at all this and go "see, AI is overhyped and it's a money pit." That's a lazy take, and I don't think it's quite right. The interesting thing here isn't the technology at all. It's the measuring.

An old story newly dressed up

There's an old idea, usually pinned on the economist Charles Goodhart, that goes roughly: when a measure becomes a target, it stops being a good measure. The moment you tell people "your score is how many tokens you use," you haven't measured productivity. You've measured token use. And people, being people, will cheerfully optimise for exactly the thing you rewarded, which is the number, not the outcome.

When a measure becomes a target, it stops being a good measure.

This is not an AI problem. It's the same disease that gives you developers gaming lines-of-code counts, salespeople sandbagging their pipeline, and just about every KPI that ever quietly strangled the thing it was meant to protect. I spent seventeen years in mining software watching perfectly well-intentioned metrics curdle into perverse incentives. The tokens are the new bit. The mistake is as old as the hills.

So when Amazon and Meta and Uber all tripped over this, they weren't tripping over some flaw in the models. They were tripping over cost governance, which is a genuinely different thing to cost efficiency.

Cost efficiency is "am I using the cheapest tool that does the job." Cost governance is "does anyone actually know whether this spend is buying value, and can they steer it before the budget's gone." Uber didn't really have an efficiency problem, they had a governance one. There was no feedback loop connecting the money going out to the value coming in, right up until the money had already gone out. Amazon's leaderboard was a governance failure wearing an encouragement scheme as a disguise. Nobody was asking the only question that matters, which isn't "how much AI did we use" but "did the AI actually help us solve a real issue."

Three guardrails, courtesy of other people's scars

The good news is you can pull three fairly clean rules out of this mess, and even better, they cost nothing and need no new software.

Don't measure what you don't want optimised

This is the Amazon and Meta lesson, and it's the big one. The instant token usage became visible and rankable, it became a target, and the second it became a target it was useless as a signal. If you must track tokens, track them the way you track your power bill: as a cost to keep half an eye on, never a score to win. The metric that's actually worth anything sits a layer below. Did vetted code ship? Did the customer's problem get solved? Amazon's replacement, apparently, is something they call "normalised deployments," which is just a posh way of saying "useful stuff that actually made it to production." That's the number. The rest is vanity.

Build the feedback loop before you open the taps, not after

This is the Uber lesson. Blowing your annual budget by April isn't really a spending problem, it's a blindness problem. If you can't see cost against value in something close to real time, you're not driving, you're just a passenger who finds out where the car went when the receipt turns up. The loop doesn't have to be clever. It just has to exist before the throttle's wide open.

Make reaching for the big model a decision, not a reflex

This is the one I find most interesting, because it's the one that applies whether you're a giant burning millions or a bloke in Perth tinkering on a Sunday. The expensive tier, the frontier model, should be something you reach for on purpose, for a reason, not the thing every single request slams into out of habit. Most of the tokenmaxxing horror stories boil down to reaching for the biggest hammer on every job, including the ones that only needed a teaspoon.

Which, of course, brings me round to my own garage.

What I actually do, and where it wobbles

For a while now I've been building a personal setup I call KieranOS. The shape of it is dead simple. A local model running on my own machine via Ollama does the first pass, and only certain work gets bumped up to a frontier model through the Claude API. The local layer is free in the way that counts here, it doesn't tick the meter, so anything it can handle on its own is money I simply never spend.

The logic for deciding what goes where is, right now, deliberately dumb. It's task-type routing. Planning, summarising, restructuring, the reading-and-thinking work, all stays local. Generation, the actual production of code and finished output, goes up to the frontier. It's a static rule, and I picked it precisely because it's static. It's easy to follow, cheap to reason about, and I can always predict what it's going to do. For a first version, boring and predictable beats clever and mysterious every day of the week.

But I'd be doing exactly what I've just criticised everyone else for if I pretended it was the answer, so here's where it wobbles.

Task-type routing treats "planning" as if it's one thing, and it really isn't. Planning a bog-standard CRUD endpoint and planning a cache invalidation strategy across a distributed system are both, technically, "planning." One of them my local model handles in its sleep. The other it'll happily have a crack at, produce something that looks like a plan, and hand back with a completely straight face. And that's the failure mode that actually worries me, because it's quiet. The local model doesn't fail loudly by throwing its hands up. It fails softly by being plausible. Then the frontier model, doing exactly what I asked of it, builds something lovely on top of a dodgy foundation, and I've saved a few tokens on the planning step in exchange for a more expensive mess to untangle later.

If that pattern sounds familiar, it should. It's a tiny, personal version of the exact thing the whole industry is chewing its nails about, the worry that AI produces confident output pointed in the wrong direction while the humans understand less and less of what they're standing on. My daft little routing rule can cook up that problem all on its own, in my house, on a Tuesday night.

There's a second wobble I should own up to as well. That local first pass isn't actually free, it's just free of token cost. It adds a bit of latency, and every now and then it produces a worse starting point that the frontier model then has to spend tokens fixing, which can end up costing more than if I'd just gone straight to the top in the first place. Sometimes the local tier is genuinely saving me money. Sometimes it's just ceremony. And the slightly uncomfortable truth is that, as things stand, I can't always tell you which.

The bit I'm building next

So the thing I'm actually working on for KieranOS is exactly the granularity that dumb task-type routing is missing. Not "is this a planning task" but "how hard is this planning task, and is it the kind of hard that my local model quietly gets wrong." The decision I want isn't local-versus-frontier as a category. It's a judgement about complexity, about how many moving parts a task is trying to hold in its head at once, because that seems to be the real line where the little model stops being trustworthy.

And here's the honest punchline, the reason this blog post and my weekend project turn out to be the same problem: I haven't cracked it yet. I'm still working it out. Which, if I'm being truthful, is a far more comfortable place to be writing from than "I've solved AI cost management, gather round and behold my architecture."

Because the real lesson from Amazon and Meta and Uber isn't "route to local models and you'll be grand." It's smaller and more durable than that. The tier you reach for should be a decision with a reason behind it, and that reason should answer to whether the work was any good, not to how much of it there was. That's true if you're a hyperscaler with a data centre and a budget with a lot of zeroes on it. And it's true if you're me, on a MacBook in Perth, trying to work out whether the little model on my desk actually earned its keep this time.

The meter's always running. The only question worth asking is whether the fare bought you anything.

More on the projects I'm tinkering with over on the Apps & Games page, or head back to the blog.