on bullshit metrics

If you use LLM to generate metrics just know that you are running on bullshit metrics. There used to be a time where grunts like me would shy away from using words like p99 and p95 without understanding what they mean. Today with the advent of generative coding it’s never been easier to tip toe into the garden of knowledge reserved for clever(er) beings.

LLMs lie and will happily slap a rocket science term into a bullshit formula. LLMs will happily slap a nice looking graph too. And probably a nicely redacted comment on top of it. Once we start relying on this metric to base our decisions, we will start running loops around a bullshit metric. We will burn through tokens building increasingly complex layers on top of layers until a point of no return, where someone will scrap it off and start it over, or it will remain abandoned waiting for a grunt to come and remove it.

Just be wary of x10 productivity gains, x100 product release, x1000 merged PRs, x5000 resolved incidents, because it’s possible it is all just a fantasy LARP game.