WizardLM 13B

Overview

WizardLM 13B is a large language model (LLM) designed for advanced instruction following and complex reasoning tasks. Developed through fine-tuning from the Llama-2 13B model, WizardLM 13B incorporates innovations in automatic instruction generation and has demonstrated strong results on several competitive natural language processing benchmarks. It was publicly released on July 25, 2023, under the Llama 2 License, furthering research into robust, open large language models.

Model Architecture and Training Methodology

WizardLM 13B is based on the Llama-2 13B architecture and utilizes a post-training fine-tuning pipeline focused on strengthening instruction-following capabilities. The core innovation is the Evol-Instruct method, a technique for automatically generating high-quality instruction data of varying complexity. This approach iteratively rewrites and evolves initial instructions, creating a rich dataset for fine-tuning large language models and enabling them to respond effectively to complex queries. Detailed descriptions of the Evol-Instruct pipeline and its impact on model training are outlined in the research paper.

The fine-tuning process employs the Vicuna conversational prompt format to facilitate multi-turn dialogs and polite, helpful replies. Training optimization uses the Adam optimizer, a learning rate of (2 \times 10^{-5}), a maximum token sequence of 2048, batch sizes of four, and distributed training with DeepSpeed Zero-3. The training regimen consists of three epochs over a period of approximately 140 hours, allowing comprehensive adaptation to the evolved instruction data. The resulting model supports flexible dialog and reasoning across a variety of topics.

Evol-Instruct: Automated Instruction Dataset Generation

At the heart of WizardLM's performance is the Evol-Instruct pipeline, which serves to expand and diversify the model's training data. Starting from the 52,000-instruction dataset used to train Alpaca, Evol-Instruct employs a combination of large language models—including OpenAI's ChatGPT and, later, Llama-2-based models—to evolve instructions across four rounds. Each round can apply different strategies, such as increasing instruction complexity, deepening reasoning requirements, introducing concrete constraints, or broadening topical coverage. The process aims to expand both the depth and breadth of the resulting dataset, contributing to the model's generalization to real-world and nuanced instructions. Evolved instructions with insufficient information gain, problematic wording, or which lead to poor model responses are filtered out by a dedicated eliminator module.

Ultimately, over 250,000 unique instructions were synthesized, with a representative sample of 70,000 used as the final training data to match comparison baselines. Response data for each instruction was generated using ChatGPT, enabling the model to learn from a wide variety of challenging and authentic instruction-response pairs. The methodology and its generalizability to other base models and instruction-following frameworks are described in the original research publication.

Benchmark Performance and Evaluation

WizardLM 13B has been evaluated on several competitive benchmarks, including MT-Bench, AlpacaEval, WizardEval, and HumanEval. As of July 2023, WizardLM-13B V1.2 reported the following results: an MT-Bench score of 7.06, AlpacaEval accuracy of 89.17%, WizardEval score of 101.4%, and a HumanEval pass@1 score of 36.6. These metrics indicate performance in instruction following, code generation, and mathematical reasoning.

The model demonstrates capabilities in handling multi-step and intricate instructions, with human evaluations via the WizardEval testbed indicating performance on criteria such as relevance, response accuracy, reasoning, calculation, and overall accuracy. These evaluations used 218 real-world human-created instructions, providing a measure of generalization and instruction-following ability. Results and leaderboard standings can be found on the MT-Bench leaderboard and the AlpacaEval leaderboard.

Applications and Use Cases

WizardLM 13B is well-suited for advanced tasks requiring complex instruction interpretation, multi-step reasoning, and nuanced language understanding. Key application areas include code generation, as demonstrated by its HumanEval results, mathematical reasoning (GSM8k and related tasks), and scenario-based dialog systems involving complex information synthesis or decision making. Its instruction-following capabilities also make it suitable for developing conversational agents, research assistants, and tools that require reliable compliance with detailed user prompts.

The instruction-tuning and data generation techniques applied in WizardLM have broader applicability, as demonstrated in extensions such as WizardMath for mathematical reasoning and WizardCoder for code-centric applications.

Model Limitations and Licensing

While WizardLM 13B demonstrates capabilities in instruction following, several limitations remain. The underlying training dataset is not publicly released, owing to ongoing legal and audit reviews, despite the open availability of the model weights. The authors also note limits in their evaluation methodologies, particularly regarding scalability and representativeness across diverse user domains. Automatic and human assessments may not account for every possible use or context.

The model is made available under the Llama 2 License, which allows for a range of research and non-commercial activities. Earlier WizardLM releases were distributed under more restrictive non-commercial terms.

Timeline and Related Models

WizardLM 13B V1.2 was released on July 25, 2023, following the initial introduction of the WizardLM research in April 2023. The WizardLM model family encompasses additional sizes and variants, including WizardLM-7B, WizardLM-30B, and specialized offshoots such as WizardCoder for code generation and WizardMath for mathematical tasks. These models illustrate the adaptability and modularity of the Evol-Instruct framework, contributing to the research trajectory of instruction-optimized large language models.

Helpful Resources and External Links

• Original research describing the model and techniques
• Model weights and documentation available in the Hugging Face repository
• Source code and project materials on GitHub
• Example evaluation data for WizardEval
• Comparative benchmarking results on the MT-Bench Leaderboard
• Instruction-following model benchmarking on the AlpacaEval Leaderboard
• Llama 2 License Information
• Detailed description of the code-focused WizardCoder variant
• Details on the mathematical reasoning WizardMath extension