The banner I used for this list is a handy figure we use at my day-job to show the relationship between different AI technologies. The key features are that Generative AI is different from Discriminant AI and Machine Learning, and that Large Language Models are just one type of Generative AI.
TL;DR. This is a list of information about how the use of LLMs and related Generative AI methods underperforms relative to traditional approached.
It is 2025 and you don't need me to tell you that Generative AI just hit the peak of the Hype Cycle. As it starts descending, I figured I'd keep a list of useful references about how it is overhyped.
Generative AI Underperforms
Model Evaluation & Threat Research (METR): Using LLMs to help you code, slows you down. Here is an on-line link. Here is a PDF.
Media Coverage: AI Agents wrong about 70% of the time, here.
Jason Sanford: A journalist writes about GenAI underperforming, here.
Chinese Call Center Problems, here.
Economics
Tom's Hardware reports on an economist who is concerned that we are heading into a Tech Company bubble larger than the old dot-com. It can be found here.
I've been trying to find something like a peer-reviewed metric for measuring LLM performance, but everything that I find appears to be written by marketing groups of major corporations. If I find any, I'll add them here.
Bonus Quantum Computing
In my world the people who over sell Generative AI usually talk about Quantum Computing. Which I find funny, and a bit sad.
I felt I had to keep track of this story from the Register.
If you know of any that I could add to my list (or if you have other suggestions), please feel free to comment below.
No comments:
Post a Comment