Llama 4 Maverick (17Bx128E)

Overview

Llama 4 Maverick (17Bx128E) is a natively multimodal, instruction-tuned large language model (LLM) introduced by Meta as part of the Llama 4 series. Developed for both research and commercial applications, Llama 4 Maverick integrates text and visual information by leveraging sophisticated model architectures and training methodologies. It is optimized for multimodal reasoning, extensive multilingual support, coding, and advanced agentic workflows. This article presents a scientific overview of its technical foundations, benchmark performance, training approaches, and intended applications, while situating the model within the broader Llama 4 family.

Model Architecture and Technical Innovations

Llama 4 Maverick introduces a Mixture-of-Experts (MoE) architecture featuring 17 billion active parameters from a total pool of 400 billion, distributed across 128 experts and a shared expert in each MoE layer. Each input token is processed jointly by the shared expert and one routed expert, enhancing computational efficiency by activating only a subset of model parameters at any time. The architecture leverages interleaved attention layers without standard positional embeddings, instead relying on the iRoPE (interleaved Rotary Positional Embedding) approach and inference time temperature scaling for improved generalization across long sequences.

Multimodal capabilities are enabled through early fusion, allowing seamless integration of text and vision tokens within a unified model backbone. The vision encoder, derived from MetaCLIP, is pre-trained jointly with a frozen Llama backbone to align visual and textual information effectively. Llama 4 Maverick's training pipeline incorporates FP8 numerical precision to maximize computational throughput. For hyperparameter optimization, the novel MetaP training strategy automatically configures per-layer learning rates and initialization scales for model scalability and stability.

Training Methods and Data

The model was pre-trained on approximately 22 trillion tokens of multimodal data, encompassing a broad range of publicly available, licensed, and Meta-generated text, image, and video content. Notably, over 200 languages are represented, with at least 100 languages each contributing more than a billion tokens, resulting in high-coverage multilingual support. The No Language Left Behind initiative informed the construction of these multilingual datasets.

Llama 4 Maverick employed co-distillation from a larger "teacher" model, Llama 4 Behemoth, which features 288 billion active parameters. A custom distillation loss function was developed, dynamically weighting soft and hard training targets to improve generalization. Post-training incorporates lightweight supervised fine-tuning, continuous online reinforcement learning (RL), and direct preference optimization. Additionally, training data was progressively filtered to exclude over 50% of "easy" prompts, focusing learning on more challenging instances as judged by automated model-based assessment.

Performance and Benchmark Results

Llama 4 Maverick demonstrates strong results across a comprehensive suite of benchmarks, with particularly notable improvements in multimodal understanding, reasoning, and code generation compared to previous Llama models and contemporary competitors. In the "Needle-in-a-haystack" (NiH) retrieval evaluation, which tests the model's ability to locate relevant information within long context windows, Llama 4 Maverick achieves successful retrieval at context lengths up to 1 million tokens for text data.

On widely recognized benchmarks, Llama 4 Maverick attains the following outcomes:

• MMMU (0-shot, image reasoning): 73.4% accuracy
• MathVista (0-shot): 73.7% accuracy
• ChartQA (0-shot, image understanding): 90.0% relaxed accuracy
• DocVQA (0-shot): 94.4 ANLS
• LiveCodeBench (0-shot, code generation): 43.4% pass@1
• MMLU Pro (0-shot, reasoning): 80.5% macro avg/acc
• MGSM (0-shot, multilingual math): 92.3% avg/em

Comparative analyses indicate that the model performs competitively or surpasses models such as GPT-4o and Gemini 2.0 Flash on multimodal tasks, while matching DeepSeek v3 on reasoning and coding with fewer active parameters. Experimental chat deployments reported an ELO rating of 1417 on the LMArena leaderboard.

Applications and Usage

Llama 4 Maverick is designed as a general-purpose assistant well-suited for chat applications, sophisticated agentic systems, and tools requiring detailed image and text understanding. It supports multimodal prompts comprising text and up to five images in the instruction-tuned version (tested up to eight images during post-training). The model facilitates advanced use cases such as visual recognition, multi-image reasoning, code synthesis, and multilingual dialogue. Function-calling is supported in both conventional Python-like and JSON formats, enhancing its applicability to tool-enabled agentic environments and workflow automation.

As part of Llama 4's suite, Maverick's outputs are also leveraged for synthetic data generation and downstream model distillation, supporting broader research and model improvement efforts. The system prompt can be customized, aiming to avoid unnecessary refusals while maintaining conversational naturalness and balance.

Limitations and Safety Considerations

Despite improvements in response balance and reduction of refusals on contentious topics, Llama 4 Maverick retains inherent LLM challenges, including potential biases and the generation of inaccurate or harmful content. Image understanding is primarily limited to English captions and queries. While pre-training data spans 200 languages, only a subset are actively supported for high-quality output. The model's performance is validated for up to five images in multimodal input, with additional images requiring developer-led evaluation.

Deployment contexts should incorporate appropriate safety guardrails, given that Llama 4 Maverick is not intended for use in isolation. Alignment with the Acceptable Use Policy, alongside ongoing robustness and trust evaluation—such as through the Llama Guard hazards taxonomy—is recommended for any application.

Licensing

Llama 4 Maverick is distributed under the Llama 4 Community License Agreement, which grants a non-exclusive, worldwide, royalty-free right to use, modify, and create derivative works. Redistribution requires retention of the license, prominent attribution ("Built with Llama"), and compliance with the Acceptable Use Policy. Additional commercial terms apply for deployments exceeding 700 million monthly active users. The agreement does not convey Meta trademarks except where explicitly permitted for compliance. Meta reserves the right to terminate the license in case of noncompliance.

Model Family and Related Models

Llama 4 Maverick is situated between Llama 4 Scout, a compact model optimized for ultra-long context (up to 10 million tokens and 17 billion active parameters), and Llama 4 Behemoth, an experimental teacher model with 288 billion active parameters and a nearly 2-trillion-parameter pool. Performance and input capabilities vary between family members, with Scout excelling at text/video retrieval at scale and Behemoth leading in STEM benchmarks, further informing distillation and training strategies across the suite.

External Resources

• Llama 4 Model Card
• Llama 4 License
• Llama.com Model Downloads
• Hugging Face Meta Llama Profile
• Llama 4 Use Policy
• Llama Cookbook (Technical Reference)
• No Language Left Behind
• LMArena Leaderboard
• Interleaved Attention Layers research
• Meta Privacy Policy
• Llama Trust and Safety