AI and the perpetual beta mindset
Summary
AI is rarely perfect or production-ready. Working with AI means embracing a perpetual beta mindset. And perpetual betas always need humans in the loop.
I often introduce myself as an outdoor photographer who earns his living from tech. Flippant as that introduction might be, I spend a lot of time behind the camera. The camera I use, the Canon R5, debuted in 2020 with a revolutionary feature: animal eye tracking and focus. The eyes are the window to the soul, they say, and photographers always seek to nail focus on the eyes of their subject. Sony was the OG of this technology when they introduced human eye auto focus in 2017 or thereabouts. However, Canon’s AI autofocus technology opened a whole new world of possibilities in the 2020s.
Five years is a long time in tech, and you’d assume that deep, supervised learning algorithms that Canon and Sony would have matured considerably by now. After all, deep learning is good enough to drive cars, so you’d expect algorithms to detect a subject’s eyes without fail. Right? Wrong!
Experienced wildlife photographers using modern animal eye AF typically see:
60–90% sharp eye retention with slower or stationary subjects;
and 20–60% during fast action.
Overall keeper rates (photos we don’t delete) hover around 50–70%, depending on conditions and the animals we photograph. And I don’t envy the job these algorithms must do, given their very limited compute capacity. Think of this. The tiny programs that run in a camera body must detect something that resembles an eye in a rather busy scene. Eye detection can be easy when you’re pointed at a lion. But a leopard is a whole other ball game. It has a body full of eyes! Add to this the complexity of the animal kingdom, comprising reptiles, amphibians, fish, mammals, birds, and invertebrates, and you have the perfect recipe for confusion.
Compared to a lion, a leopard’s body confuses even the best eye tracking AI
Of course, in the most challenging focusing situations, manual focus by a human being will often outperform auto-focus by the camera’s inbuilt AI. When you compare AI’s spectacular failure in this scenario with a human’s obvious superiority, the intellectual chasm between machines and humans becomes evident.
Despite our bullishness about AI’s capabilities, there are limits to machine intelligence in 2025. Humanity’s last exam (HLE) makes these limits painfully clear. The best large language models available to us today are only capable of low accuracy as compared to expert-level academics. Worse, they suffer from poor calibration errors - i.e. they’re rather confident even when they’re wrong!
Just this month, Apple dropped a massive truth bomb about the true capabilities of the latest reasoning models. Their paper, “The illusion of thinking”, suggests that LLMs merely mimic reasoning with simple problems and eventually give up when they encounter higher complexity problems. That finding is consistent with the impressions you may draw from HLE’s data on model performance.
So, if AI is still incapable of competing with the best of us, what's the brouhaha about? The answer lies in the mindset with which we utilise AI over the next few years.
The perpetual beta mindset
If LLMs are amongst the most advanced AI technology we have access to, and the best LLMs only achieve 20% accuracy or thereabouts compared to the smartest humans, then we have a long way to go before we can trust AI to be “production grade”. For reference, my camera’s AI feels like a five-year beta!
And herein lies the rub.
Across companies, there’s often a top-down mandate to drive up productivity with AI.
There’s a huge chasm between the best of humans and the worst of AI.
This chasm often causes the best workers to also be the most sceptical about AI.
That scepticism eventually leads to poor AI adoption, even where AI may deliver the most value.
How might we then extract the most value from AI? In my view, we’re best off thinking about AI technology as a perpetual beta. Don’t compare the tails - i.e. the best and worst of AI with the best and worst of humans. Instead, ask yourself if middle-of-the-road work can benefit from AI integration. I think we’ll all notice that on average, for the average, commonly understood tasks, the average AI will perform as well as the average human, if not better. I know that sentence contains a lot of averages. That’s by design. If there were one word you should take away from this article, I’d say it’d be “average”.
AI delivers most value in the very middle of the bell curve
Value at the core, not at the fringes
As AI tools become more like heavy-lifting cranes than the one-trick ponies they used to be, it’s tempting to seek value at the fringes of their current capability. It’s unsurprising that we then think of AI as a replacement for humans; something Erik Brynjolfsson calls the Turing trap:
“A trap where we sacrifice the real benefits of machines—namely, their potential to complement humans and do things we can’t.”
I wonder if embracing the perpetual beta nature of AI can help us derive more value from these technologies. Can a focus on the “average” stuff help us create more value with AI?
How can we employ AI to make the average developer more effective?
How can AI make existing teams more effective by doing tasks that humans find boring?
How can AI help us collaborate with our most adjacent colleagues by helping us practice some adjacent skills in safety?
How can we utilise AI to handle the core, repetitive, well-understood, and routine work, so that humans can focus their 40 hours on bona fide reasoning and problem-solving at the fringes?
Instead of thinking of AI as a replacement for humans, can AI improve the mix of drudgery and novelty in our portfolio of work?
Improving the portfolio of modern knowledge work
The perpetual beta mindset can help assuage concerns that our smartest colleagues have with the over-enthusiastic, techno-utopian spiel they hear about AI. Most people will sign up to reduce drudgery at the well-understood core. Most people will help you unpack their jobs if they don’t fear that AI will replace them at some point. But AI transformation will continue to hit roadblocks, not in the least from the best knowledge workers, if job losses loom large and if executives rate AI as more than a perpetual beta.
As for my wildlife photography adventures, my camera nails focus for the most common scenarios I encounter in the wilderness. It helps me be far more effective when photographing fast action than I’ve ever been in the past. Yes, my keeper rate is 50-70%, but that’s probably 3x what I could achieve before AI became part of my workflow. Of course, it’ll fail from time to time. That’s a novel situation for me to be the human in the loop and exercise my judgment. For the rest of the drudgery of mucking with camera controls, AI has my back!
