Qwen2.5 14B 1M

Overview

Qwen2.5-14B-Instruct-1M is a large language model developed by the Qwen Team at Alibaba Group. As part of the Qwen2.5-1M series, the model introduces the capacity for processing input sequences up to one million tokens in length. This increase in context length supports applications that require reasoning across extensive documents, repository-level codebases, or collections of heterogeneous information. Qwen2.5-14B-Instruct-1M is released under an open-source license, with full technical documentation and benchmarks publicly available. The innovations in context handling, optimization, and training strategies address challenges intrinsic to long-context language modeling, supporting performance across both standard and expanded-context tasks (Qwen2.5-1M Technical Report, Qwen2.5-1M blog).

Model Architecture and Training Innovations

Qwen2.5-14B-Instruct-1M is based on a Transformer architecture, incorporating architectural and training components. Notably, it employs Grouped Query Attention (GQA) to optimize cache efficiency in the attention mechanism, and SwiGLU for activation, facilitating improved convergence and stability. Rotary Position Embeddings (RoPE) are used for encoding positional information, complemented by QKV bias and RMSNorm for normalization and training robustness (model architecture, HuggingFace model card).

The model contains 14.7 billion parameters (13.1 billion non-embedding), distributed over 48 layers. Support for extended long-context inputs is realized through a series of training-stage innovations.

A progressive pre-training scheme gradually expands the context window from an initial size of 4,096 tokens up to 262,144 tokens, using both natural and synthetic corpora to optimize performance at scale. Subsequently, a two-stage supervised fine-tuning process is applied: the first stage focuses on shorter tasks (≤32,768 tokens), while the second co-trains on mixed-length data, including extended sequences up to 262,144 tokens. Offline Reinforcement Learning, akin to Direct Preference Optimization (DPO), is used for model alignment, ensuring human-like preference in responses (training stages).

Techniques for Long-Context Modeling

Qwen2.5-14B-Instruct-1M uses methods for length extrapolation, which allow the model to operate effectively on contexts beyond the lengths encountered during training. The Dual Chunk Attention (DCA) mechanism is employed to remap relative positional indices, mitigating accuracy degradation in ultra-long sequences. This method, integrated with attention scaling via the YaRN method, enables extrapolative inference at lengths up to one million tokens without additional training (Dual Chunk Attention, Qwen2.5-1M blog).

To further accelerate long-context inference, the model integrates a sparse attention mechanism called MInference. This approach reduces the computational and memory costs of the prefill phase and includes features such as chunked prefill and sparsity refinement for high-precision processing of million-token sequences (MInference and optimization). These innovations are supported by engine optimizations in BladeLLM, which implements dynamic chunked pipeline parallelism and advanced scheduling strategies for efficient large-scale inference.

Evaluation and Benchmark Performance

Qwen2.5-14B-Instruct-1M shows performance across a range of benchmarks, including long-context scenarios. In the Passkey Retrieval task, the model sustains perfect accuracy in retrieving information from sequences up to 1 million tokens, irrespective of the answer’s position in the document (benchmark results, Qwen2.5-1M blog).

On the RULER benchmark, which measures long-context reasoning, Qwen2.5-14B-Instruct-1M achieved a score of 95.7 (average), outperforming GPT-4 and GPT-4o-mini at 128K contexts (RULER benchmark table). For LV-Eval (evaluating up to 256K context tokens) and LongBench-Chat (up to 100K tokens), the model shows performance among models.

On short-context and general-purpose tasks, Qwen2.5-14B-Instruct-1M maintains parity with its 128K-token predecessor and competitive models like GPT-4o-mini, while still offering the capability to process much longer sequences (performance data). Standard benchmarks such as MMLU-Pro, MATH, HumanEval, Arena-Hard, and MTBench confirm the model’s proficiency on a broad spectrum of reasoning, mathematics, coding, and alignment tasks.

Applications and Related Models

The capabilities of Qwen2.5-14B-Instruct-1M make it suitable for tasks that demand extensive context integration, such as code generation with repository-scale context, literature review over large document corpora, or integrative information retrieval from ultra-long records (use cases). Tasks requiring synthesis from multiple evidence fragments, or alignment with human preferences in extended dialogues, are also supported.

Other models in the Qwen2.5-1M series include the open Qwen2.5-7B-Instruct-1M, which also achieves 1M context support with lower resource requirements, and Qwen2.5-Turbo, a Mixture of Experts model accessible via API. These models are complemented by previous Qwen2.5 architectures with shorter context (128K tokens) and larger models such as Qwen2.5-72B-Instruct, which show performance in select benchmarks (model family overview, Qwen2.5-1M blog).

Model Limitations and Development

Despite its long-context capabilities, Qwen2.5-14B-Instruct-1M has notable limitations. For local deployment at full 1M-token context, substantial VRAM is required; for the 14B model, the minimum recommended is 320GB, restricting accessibility for certain users (deployment details, HuggingFace documentation). The model’s sparse attention implementation can be sensitive to sparsity configuration, and optimal settings may require additional tuning for ultra-long contexts.

While length extrapolation methods avoid the need for retraining when scaling to larger contexts, small accuracy drops can still occur, especially when legacy inference frameworks are used instead of the recommended optimized implementations. The research team continues to investigate approaches for scaling, reduced-memory training, and further optimization (ongoing development).

Qwen2.5-14B-Instruct-1M is provided under the Apache 2.0 license (license details).

Helpful External Resources

• Qwen2.5-1M Technical Report (PDF)
• Qwen Team on HuggingFace
• Qwen Team ModelScope
• Qwen2.5-1M HuggingFace Demo
• Qwen2.5-1M ModelScope Demo
• Qwen Team GitHub
• vLLM GitHub Repository
• Qwen Documentation on vLLM Usage
• Qwen-Agent GitHub