Meta formally launched LLaMA 4 (Giant Language Mannequin Meta AI) — the fourth era of its open massive language fashions.
LLaMA 4 represents a major leap in AI structure, combining Combination-of-Specialists (MoE) design with native multimodal integration (textual content + picture), a record-breaking 10 million token context window, and main enhancements in reasoning, coding, and comprehension.
With efficiency rivaling and infrequently exceeding proprietary fashions like OpenAI’s GPT-4, Anthropic’s Claude, and Google’s Gemini, LLaMA 4 is designed for open innovation whereas sustaining state-of-the-art capabilities. This text provides a complete deep dive into LLaMA 4’s structure, coaching methodology, efficiency, configurations, use circumstances, limitations, and tips on how to entry it.
1.1 Combination-of-Specialists (MoE) Structure
For the primary time within the LLaMA sequence, LLaMA 4 introduces a Combination-of-Specialists design. The mannequin accommodates quite a few knowledgeable modules however solely prompts a subset per enter, conserving inference environment friendly whereas massively growing parameter capability. For instance:
- Scout prompts 17B parameters with 109B whole
- Maverick additionally makes use of 17B energetic, with 400B whole parameters
This permits LLaMA 4 to outperform a lot bigger dense fashions, reminiscent of LLaMA 3’s 405B dense variant, with far much less compute.
1.2 Native Multimodality (Early Fusion)
Not like LLaMA 3, LLaMA 4 is natively multimodal. It fuses textual content and picture tokens on the enter layer, processing them by way of a unified transformer spine. An improved MetaCLIP encoder converts pictures into tokens. This tight integration permits seamless vision-text reasoning — e.g., answering questions on charts, studying screenshots, or analyzing paperwork.
