I have heard, both in person and on-line, CEOs of AI-based companies saying that their new technology is going to make most workers redundant. They claim we won’t need most skilled workers in the future. Exactly when? Really soon.
This is just a marketing lie. Everyone who understands the technology knows it is not true, but the goal is to feed the dreams of the super-rich and convince them to invest in AI technology. Anyone who understands the technology knows the claim isn’t true. My goal, with this post, is to help the average reader understand why this so, that is why AI is not going to replace skilled workers.
To get there we will have to talk about things that most people don’t currently know. They are a little bit complex, but not nearly as hard to understand as what the tech bros want you to think they are. But you will face a bunch of text without pretty pictures, and I’m afraid you’re going to have to suffer through, if you want to understand the changes happening around you.
The Basics
First we need to have some understanding of two terms that sound complex, but at their core are reasonably simple. They are Latent Space and a Universal Function Approximator. Most people don’t know these terms, but it will take you about three minutes to understand them well enough to know why skilled labor isn’t going anywhere, so I’m hoping you’ll stick with me.
Latent Space. A latent space is a compressed representation of something. The idea is that we can strip off a lot of details and get to the core of something in a simpler form. For example, imagine you correlated peoples ages and heights. Older people tend to be taller, at least during childhood. But once adult height is reached, we stay pretty much the same over our lives, until we shrink a little when we become elderly. I could develop a mathematical function, using fancy regression, that predicts the average person’s height, given their age. For kids, you could use a linear regression of a line going up with time. We can add some fancy bits to flatten the line at adulthood.
That function is a latent representation of the relationship between age and height. It will be pretty good, but not great, at predicting a persons height, if you give it their age.
I could add additional things to the equation that will make it better. I could add the person’s nutritional background, and maybe some genetic markers, and maybe their parent’s height when they were the same age. All of this would make my function do a better job of predicting a person’s height, but it would still not be perfect. But it would be a latent space for predicting a person’s height. Adding information about Mary Shelly’s book Frankenstein will not improve this equation—this become important later.
Universal Function Approximator. Deep learning using something call Artificial Neural Networks. These are fancy and complex mathematical tools that function as universal function approximators. Remember that function we wanted that would take a person’s age, nutritional background, parent’s heights and genetic markers, and predict a persons height? To find that function in the real world will take a massive amount of information and several hours of collaborative meetings between domain experts and biostatisticians. But we could use artificial neural networks to approximate that function. These tools can find something that is almost as good as if you had the experts do it. But not exactly as good.
Deep learning can approximate a complex latent space, provided it has enough training data relative to the problem. It uses some fancy logic to estimate the function, and the corresponding latent space, and the more data it has, the better the job it does. But it will never be as good as doing the analysis correctly from first principles. Though, it will often be good enough.
The Limits of Training Data
There are lots of fancy terms being kicked around to describe different ways to put together these artificial neural networks and create useful latent spaces. The biggest ones are now called Foundation Models and these are what the Tech Bros are trying to sell you.
Aside. Remember the Tech Bros are not really trying to sell the service to you. They are trying to convince people who have billions of dollars that their technology will replace the workers the billionaires have to pay in order to make more money. They just need you to believe what they are saying so the billionaires will believe them also.
So Foundation Models take massive training data sets and try to find a Latent Space that can predict anything. The “G” in chatGPT stands for Generalized. A GPT is a pretrained (that’s the “P”) generalized tool. You can ask it anything, and it will have an answer. It has be trained to have a latent space to predict anything. It can tell you how many “r”s are in Strawberry. How Frankenstein ends. And how tall a person is at age 3. We’ve trained this latent space to have every answer all at once.
The models aren’t really that good, but they are improving. The people I know use them to answer easy questions in areas they aren’t expert in. What is the code to write a t-test in the programming language R?
But most of us have noticed that the models are no longer improving very much. A few years ago you could believe a CEO when they told you that they were weeks away from having a GPT that did everything. Because every week they seemed to get better. But then they stopped.
There is one figure that you need to see in order to understand why they stopped improving.
I added the purple line by hand—it is not a regression line, just my guess as to where such a line would go. But I think pretty much everyone can see the pattern.
This image is taken from https://epoch.ai/benchmarks/eci. What it shows is the relationship of Foundation Models between their performance—how well they work according to a certain set of metrics—and the size of their training data set. Well, actually the log of the size of their training data set. So the left most point which has a performance score a little below 70, was trained on a data set of a certain size. And it was a huge training set. And chatGPT5 up near the top right, with a score of almost double the left most point was trained on a data set 10,000 times larger!
Increasing the training set by 10,000 times only doubled performance. And worse, it appears to be slowing down. Models trained on even larger sets seem to have little or no improvement. We are close to the maximum that the current architecture can produce.
Building larger data centers and training models with even larger data sets is not going to make them particularly better. Only innovation in the AI architecture is going to do that, and the people working on AI are too busy trying to fleece the billionaires to spend any time on that. It is not that we couldn’t build better AI. It is simply not going to be worth it. The people who could build new architectures that generate better models from the same size training sets aren’t working on that problem. It is not an easy problem, and it certainly can’t be solved by a vibe-coding kid.
A Foundation Model’s performance (for a given architecture) depends on the size of its training set. Oh, there is some noise around it. This one is a little better than that one, but in general, they improve with exponentially more training data.
And they are out of training data. They have taken everything that is easily available.
Another Aside. In order to see real innovation in the AI industry, what you need to watch for is new AI architectures that gives improved performance with the same amount of training data. This would be represented as a curve above the one I drew. Such a curve will likely have a new slope and will represent a real improvement, not just marketing hype.
Conclusions
Remember that fancy regression we did at the start of this post? It will never be perfect, but it can be both close and useful. Provided you have enough experts working on it. Our AI approximation will always be a little worse (or a lot worse) than the bespoke model.
Foundation Models are trained from data stolen off the web. But Shakespeare is dead. He’s not writing any more. Same with Mary Shelly. So all the new data you can get is from the on-line content currently living people are creating. (This blog will likely be slurped into some models without my consent.)
But we don’t write our deepest expertise on-line. We write things that we want people who know less than us to understand. This blog is to help intelligent readers with little AI exposure understand the limitations of AI, not my thoughts on using visible AI to interpret omic data sets. Only my coworkers hear those. Eventually we will write new peer reviewed papers that the Tech Bros will steal to add to their models. But only AFTER we are using them. And such papers are always incomplete.
The current approach to AI will always lag.
This post is already getting long, so I’ll stop here, but there is a whole other story on AI performance related to the density of training data. Questions that fall near areas in the latent space that are relatively rich in training information will yield better results than those out and away from training data. Just like our regression example is trained on human age and height and will not work well for either mice or elephants, questions asked to Foundation Models do best when they are asked about things close to their training data.
Models asked about how to write R code for calculating a t-test do relatively well. There are hundreds of web sites that answer that question. If you ask the same question about hierarchical linear models (which have very little representation in the training data) the models do poorly and often give factually incorrect answers. And this isn’t going to change any time soon.
As always, thank you for read this and I always welcome your comments and questions below. Also, I am trying to make this approachable to a broad audience and always appreciate feedback on where I was unclear.
No comments:
Post a Comment