TL;DR. Foundation Model AI have effectively reached their
current limit and that isn’t good enough to replace most educated
workers.
I have heard, both
in person and on-line, CEOs of AI-based companies saying that their
new technology is going to make most workers redundant. They claim we
won’t need most skilled workers in the future. Exactly when? Really
soon.
This is just a
marketing lie. Everyone who understands the technology knows it is
not true, but the goal is to feed the dreams of the super-rich and
convince them to invest in AI technology. Anyone who understands the
technology knows the claim isn’t true. My goal, with this post, is
to help the average reader understand why this so, that is why AI is
not going to replace skilled workers.
To get there we will
have to talk about things that most people don’t currently know.
They are a little bit complex, but not nearly as hard to understand
as what the tech bros want you to think they are. But you will face a
bunch of text without pretty pictures, and I’m afraid you’re
going to have to suffer through, if you want to understand the
changes happening around you.
The Basics
First we need to
have some understanding of two terms that sound complex, but at their
core are reasonably simple. They are Latent Space and a
Universal Function Approximator. Most people don’t
know these terms, but it will take you about three minutes to
understand them well enough to know why skilled labor isn’t going
anywhere, so I’m hoping you’ll stick with me.
Latent Space.
A latent space is a compressed representation of something. The idea
is that we can strip off a lot of details and get to the core of
something in a simpler form. For example, imagine you correlated
peoples ages and heights. Older people tend to be taller, at least
during childhood. But once adult height is reached, we stay pretty
much the same over our lives, until we shrink a little when we become
elderly. I could develop a mathematical function, using fancy
regression, that predicts the average person’s height, given their
age. For kids, you could use a linear regression of a line going up
with time. We can add some fancy bits to flatten the line at
adulthood.
That function is a
latent representation of the relationship between age and height. It
will be pretty good, but not great, at predicting a persons height,
if you give it their age.
I could add
additional things to the equation that will make it better. I could
add the person’s nutritional background, and maybe some genetic
markers, and maybe their parent’s height when they were the same
age. All of this would make my function do a better job of predicting
a person’s height, but it would still not be perfect. But it would
be a latent space for predicting a person’s height. Adding
information about Mary Shelly’s book Frankenstein will not
improve this equation—this become important later.
Universal
Function Approximator. Deep learning using something call
Artificial Neural Networks. These are fancy and complex mathematical
tools that function as universal function approximators. Remember
that function we wanted that would take a person’s age, nutritional
background, parent’s heights and genetic markers, and predict a
persons height? To find that function in the real world will take a
massive amount of information and several hours of collaborative
meetings between domain experts and biostatisticians. But we could
use artificial neural networks to approximate that function. These
tools can find something that is almost as good as if you had the
experts do it. But not exactly as good.
Deep learning can
approximate a complex latent space, provided it has enough training
data relative to the problem. It uses some fancy logic to estimate
the function, and the corresponding latent space, and the more data
it has, the better the job it does. But it will never be as good as
doing the analysis correctly from first principles. Though, it will
often be good enough.
The Limits of
Training Data
There are lots of
fancy terms being kicked around to describe different ways to put
together these artificial neural networks and create useful latent
spaces. The biggest ones are now called Foundation Models and these
are what the Tech Bros are trying to sell you.
Aside.
Remember the Tech Bros are not really trying to sell the service to
you. They are trying to convince people who have billions of dollars
that their technology will replace the workers the billionaires have
to pay in order to make more money. They just need you to believe
what they are saying so the billionaires will believe them also.
So Foundation Models
take massive training data sets and try to find a Latent Space that
can predict anything. The “G” in chatGPT stands for Generalized.
A GPT is a pretrained (that’s the “P”) generalized tool. You
can ask it anything, and it will have an answer. It has be trained to
have a latent space to predict anything. It can tell you how many
“r”s are in Strawberry. How Frankenstein ends. And how
tall a person is at age 3. We’ve trained this latent space to have
every answer all at once.
The models aren’t
really that good, but they are improving. The people I know use them
to answer easy questions in areas they aren’t expert in. What is
the code to write a t-test in the programming language R?
But most of us have
noticed that the models are no longer improving very much. A few
years ago you could believe a CEO when they told you that they were
weeks away from having a GPT that did everything. Because every week
they seemed to get better. But then they stopped.
There is one figure that you need to see in order to understand why
they stopped improving.
I added the purple line by hand—it is not a regression line, just
my guess as to where such a line would go. But I think pretty much
everyone can see the pattern.
This image is taken
from https://epoch.ai/benchmarks/eci.
What it shows is the relationship of Foundation Models between their
performance—how well they work according to a certain set of
metrics—and the size of their training data set. Well, actually the
log of the size of their training data set. So the left most point
which has a performance score a little below 70, was trained on a
data set of a certain size. And it was a huge training set. And
chatGPT5 up near the top right, with a score of almost double the
left most point was trained on a data set 10,000 times larger!
Increasing the
training set by 10,000 times only doubled performance. And worse, it
appears to be slowing down. Models trained on even larger sets seem
to have little or no improvement. We are close to the maximum that
the current architecture can produce.
Building larger data
centers and training models with even larger data sets is not going
to make them particularly better. Only innovation in the AI
architecture is going to do that, and the people working on AI are
too busy trying to fleece the billionaires to spend any time on that.
It is not that we couldn’t build better AI. It is simply not going
to be worth it. The people who could build new architectures that
generate better models from the same size training sets aren’t
working on that problem. It is not an easy problem, and it certainly
can’t be solved by a vibe-coding kid.
A Foundation Model’s
performance (for a given architecture) depends on the size of its
training set. Oh, there is some noise around it. This one is a little
better than that one, but in general, they improve with exponentially
more training data.
And they are out of
training data. They have taken everything that is easily available.
Another Aside.
In order to see real innovation in the AI industry, what you need to
watch for is new AI architectures that gives improved performance
with the same amount of training data. This would be represented as a
curve above the one I drew. Such a curve will likely have a new slope
and will represent a real improvement, not just marketing hype.
Conclusions
Remember that fancy
regression we did at the start of this post? It will never be
perfect, but it can be both close and useful. Provided you have
enough experts working on it. Our AI approximation will always be a
little worse (or a lot worse) than the bespoke model.
Foundation Models
are trained from data stolen off the web. But Shakespeare is dead.
He’s not writing any more. Same with Mary Shelly. So all the new
data you can get is from the on-line content currently living people
are creating. (This blog will likely be slurped into some models
without my consent.)
But we don’t write
our deepest expertise on-line. We write things that we want people
who know less than us to understand. This blog is to help intelligent
readers with little AI exposure understand the limitations of AI, not
my thoughts on using visible AI to interpret omic data sets. Only my
coworkers hear those. Eventually we will write new peer reviewed
papers that the Tech Bros will steal to add to their models. But only
AFTER we are using them. And such papers are always incomplete.
The current approach
to AI will always lag.
This post is already
getting long, so I’ll stop here, but there is a whole other story
on AI performance related to the density of training data. Questions
that fall near areas in the latent space that are relatively rich in
training information will yield better results than those out and
away from training data. Just like our regression example is trained
on human age and height and will not work well for either mice or
elephants, questions asked to Foundation Models do best when they are
asked about things close to their training data.
Models asked about
how to write R code for calculating a t-test do relatively well.
There are hundreds of web sites that answer that question. If you ask
the same question about hierarchical linear models (which have very
little representation in the training data) the models do poorly and
often give factually incorrect answers. And this isn’t going to
change any time soon.
As always, thank you
for read this and I always welcome your comments and questions below.
Also, I am trying to make this approachable to a broad audience and
always appreciate feedback on where I was unclear.