the previous couple of weeks, we have now seen the discharge of highly effective LLMs comparable to Qwen 3 MoE, Kimi K2, and Grok 4. We’ll proceed seeing such speedy enhancements within the foreseeable future, and to match the LLMs towards one another, we require benchmarks. On this article, I focus on the newly launched ARC AGI 3 benchmark and why frontier LLMs battle to finish any duties on the benchmark.
Motivation
My motivation for writing this text is to remain on prime of the most recent developments in LLM expertise. Solely within the final couple of weeks have we seen the Kimi K2 mannequin (finest open-source mannequin when launched), Qwen 3 235B-A22B (at the moment finest open-source mannequin), Grok 4, and so forth. There may be a lot taking place within the LLM house, and one option to sustain is to trace the benchmarks.
I believe the ARC AGI benchmark is especially fascinating, primarily as a result of I wish to see if LLMs can match human-level intelligence. ARC AGI puzzles are made in order that people are in a position to full them, however LLMs will battle.
It’s also possible to learn my article on Utilizing Context Engineering to Significantly Enhance LLM Performance and take a look at my website, which contains all my information and articles.
Desk of Contents
Introduction to ARC AGI
ARC AGI is actually a puzzle recreation of sample matching.
- ARC AGI 1: You’re given a collection of input-output pairs, and have to finish the sample
- ARC AGI 2: Just like the primary benchmark, performing sample matching on enter and output examples
- ARC AGI 3: Right here you’re taking part in a recreation, the place you must transfer your block into the purpose space, however some required steps in between
I believe it’s cool to check out these puzzle video games and full them myself. Then, you may see LLMs initially battle with the benchmarks, after which improve their efficiency with higher fashions. OpenAI, for instance, scored:
- 7.8% with o1 mini
- 75% with o3-low
- 88% with o3-high
As you can too see within the picture beneath:
Taking part in the ARC AGI benchmark
It’s also possible to strive the ARC AGI benchmarks your self or construct an AI to carry out the duties. Go to the ARC AGI 3 website and begin taking part in the sport.
The entire level of the video games is that you don’t have any directions, and you must determine the principles your self. I get pleasure from this idea, because it represents determining a wholly new downside, with none assist. This highlights your capacity to be taught new environments, adapt to them, and resolve issues.
You’ll be able to see a recording of me playing ARC AGI 3 here, encountering the issues for the primary time. I used to be sadly unable to embed the hyperlink within the article. Nevertheless, it was tremendous fascinating to check out the benchmark and picture the problem an LLM has to undergo to unravel it. I first observe the setting, and what occurs after I carry out the totally different actions. An motion on this case is urgent one of many related buttons. Some actions do nothing, whereas others have an effect on the setting. I then proceed to uncover the purpose of the puzzle (for instance, get the article to the purpose space) and attempt to obtain this purpose.
Why frontier fashions obtain 0%
This article states that when frontier fashions have been examined on the ARC AGI 3 preview, they achieved 0%. This may sound disappointing to some folks, contemplating you have been most likely in a position to efficiently full loads of the duties your self, comparatively rapidly.
As I beforehand mentioned, a number of OpenAI fashions have had success with the sooner ARC AGI benchmarks, with their finest mannequin attaining 88% on the primary model. Nevertheless, initially, fashions achieved 0%, or within the low single-digit percentages.
I’ve just a few theories for why frontier fashions weren’t in a position to carry out duties on ARC AGI 3:
Context size
When engaged on ARC AGI 3, you don’t get any details about the sport. The mannequin thus has to check out quite a lot of actions, see the output of these actions (for instance, nothing occurs, or a block strikes, and many others). The mannequin then has to judge the actions it took, together with the output, and think about its subsequent strikes.
I consider the motion house on ARC AGI 3 may be very massive, and it’s thus tough for the fashions to each experiment sufficient to seek out the proper motion and keep away from repeating unsuccessful actions. The fashions basically have an issue with their context size and using the complete size of it.
I not too long ago learn an fascinating article from Manus about how they develop their brokers and handle their reminiscence. You need to use methods comparable to summarizing earlier context or utilizing a file system to retailer vital context. I consider this might be key to growing efficiency on the ARC AGI 3 benchmark.
Coaching dataset
One other main cause frontier fashions are unable to finish ARC AGI 3 duties efficiently is that the duties are very totally different from their coaching dataset. LLMs will nearly at all times carry out means higher on a activity if such a activity (or the same one) is included within the coaching dataset. On this occasion, I consider LLMs have little coaching information on working with video games, for instance. Moreover, an vital level right here can also be the agentic coaching information for the LLMs.
With agentic coaching information, I imply information the place the LLM is using instruments and performing actions. I consider we’re seeing a speedy improve in LLMs used as brokers, and thus, the proportional quantity of coaching information for agentic habits is quickly growing. Nevertheless, it could be that present frontier fashions nonetheless should not nearly as good at performing such actions, although it’ll seemingly improve quickly within the coming months.
Some folks will spotlight how this proves LLMs would not have actual intelligence: The entire level of intelligence (and the ARC AGI benchmark) is to have the ability to perceive duties with none clues, solely by inspecting the setting. To some extent, I agree with this level, and I hope to see fashions carry out higher on ARC AGI due to elevated mannequin intelligence, and never due to benchmark chasing, an idea I discover later on this article.
Benchmark efficiency sooner or later
Sooner or later, I consider we’ll see huge enhancements in mannequin efficiency on ARC AGI 3. Largely as a result of I believe you may create AI brokers which can be fine-tuned for agentic efficiency, and that optimally make the most of their reminiscence. I consider comparatively low-cost enhancements can be utilized to vastly enhance efficiency, although I additionally anticipate dearer enhancements (for instance, the discharge of GPT-5) will carry out effectively on this benchmark.
Benchmark chasing
I believe it’s vital to depart a bit about benchmark chasing. Benchmark chasing is the idea of LLM suppliers chasing optimum scores on benchmarks, reasonably than merely creating the very best or most clever LLMs. This can be a downside as a result of the correlation between benchmark efficiency and LLM intelligence just isn’t 100%.
Within the reinforcement studying world, benchmark chasing can be known as reward hacking. A state of affairs the place the agent figures out a option to hack the setting they’re in to attain a reward, with out correctly performing a activity.
The explanation LLM suppliers do that is that at any time when a brand new mannequin is launched, folks normally have a look at two issues:
- Benchmark efficiency
- Vibe
Benchmark efficiency is normally measured on identified benchmarks, comparable to SWE-bench and ARC AGI. Vibe testing can also be a means LLMs are sometimes measured by the general public (I’m not saying it’s a great way of testing the mannequin, I’m merely saying it occurs in follow). The issue with this, nonetheless, is that I consider it’s fairly easy to impress folks with the vibe of a mannequin, as a result of vibe checking tries some very small proportion of the motion house for the LLM. You might solely be asking it sure questions which can be found on the internet, or asking it to program an utility which the mannequin has already seen 1000 cases of in its coaching information.
Thus, what it’s best to do is to have a benchmark by yourself, for instance, an in-house dataset that has not been leaked to the web. Then you may benchmark which LLM works finest on your use case and prioritize utilizing this LLM.
Conclusion
On this article, I’ve mentioned LLM benchmarks and why they’re vital for evaluating LLMs. I’ve launched you to the newly launched ARC AGI 3 benchmark. This benchmark is tremendous fascinating contemplating people are simply in a position to full a number of the duties, whereas frontier fashions rating 0%. This thus represents a activity the place human intelligence nonetheless outperforms LLMs.
As we advance, I consider we’ll see speedy enhancements in LLM efficiency on ARC AGI 3, although I hope this is not going to be the results of benchmark chasing, however reasonably the intelligence enchancment of LLMs.
