Neural Chat 7B

Overview

Neural Chat 7B is a family of large language models (LLMs) developed by Intel, designed to address a broad range of natural language processing tasks. The latest iteration in this lineage, Neural-Chat-7B-v3-3, features enhanced fine-tuning procedures, mathematical reasoning capabilities, and optimized performance on Intel hardware platforms. Released in December 2023, the model incorporates alignment and training techniques relevant to the field of language models.

Model Architecture and Training

Neural Chat 7B is constructed as a 7-billion parameter generative transformer-based language model. The v3-3 version is a fine-tuned derivative of Neural-Chat-7B-v3-1, which itself was originally adapted from Mistral-7B. The model employs a context window of 8192 tokens, facilitating its ability to process and generate extended sequences of text.

Fine-tuning for Neural-Chat-7B-v3-3 was conducted using the Intel Gaudi 2 processor, leveraging an array of eight Gaudi2 accelerator cards. This facilitates efficient model training and enhanced throughput, particularly in large-scale fine-tuning regimes. Neural-Chat-7B-v3-3 utilizes the Direct Preference Optimization (DPO) approach for alignment, focusing on preference-driven supervision via the Intel/orca_dpo_pairs dataset.

Datasets and Alignment Methods

A distinctive characteristic of Neural-Chat-7B-v3-3 is its strong mathematical reasoning, achieved through specialized fine-tuning on the MetaMathQA dataset. This resource builds upon the established GSM8k and MATH training sets, while meticulously avoiding contamination from the GSM8k test set, preserving the integrity of model evaluation. The alignment pipeline employs DPO to tune the model’s outputs to human-preferred behaviors, thereby improving the quality of generated text.

Further information about this alignment procedure and supervised fine-tuning on Habana Gaudi2 hardware is detailed in the blog post "The Practice of Supervised Fine-tuning and Direct Preference Optimization on Intel Gaudi2".

Inference and Precision Modes

Neural-Chat-7B-v3-3 is engineered for performance and flexibility across a variety of inference environments. It supports multiple precision levels, catering to diverse deployment requirements:

• Floating-point 32 (FP32) inference can be performed using standard implementations in the Transformers library.
• Brain Floating Point 16 (BF16) and INT4 quantized inference are available via the Intel Extension for Transformers (IET) and Intel Extension for PyTorch, leveraging the WeightOnlyQuantConfig for efficient low-precision execution without substantial degradation of model accuracy.

These optimizations are intended to offer improved inference speed and reduced memory consumption, thereby enabling efficient deployment across inference pipelines.

Applications and Performance

The design of Neural-Chat-7B-v3-3 caters primarily to language modeling tasks such as text generation, reasoning, and serving as an expert assistant for mathematics. The model is recognized for generating step-by-step solutions and explanations to mathematical problems, reflecting advancements in mathematical understanding accrued from targeted training on MetaMathQA.

Benchmark evaluation on the Open LLM Leaderboard provides comprehensive performance metrics against peer models. As of its release, Neural-Chat-7B-v3-3 achieved an average score of 69.83, with sub-task scores including 66.89 on ARC (25-shot), 85.26 on HellaSwag (10-shot), 63.07 on MMLU (5-shot), 63.01 on TruthfulQA (0-shot), 79.64 on Winogrande (5-shot), and 61.11 on GSM8K (5-shot), as compiled on the leaderboard's detailed results page.

Limitations and Considerations

Despite its robust training and alignment regimen, Neural-Chat-7B-v3-3 retains key limitations inherent to large-scale language models. It may generate outputs that are factually incorrect, imprecise, or biased, depending on the task context. The model's inherited and fine-tuned datasets, while curated, cannot guarantee immunity from offensive, biased, or lewd outputs. Consequently, developers and users are advised to conduct comprehensive safety evaluations and consider post-processing solutions for mitigating unwanted outputs when deploying the model in applications.

Usage of Neural-Chat-7B-v3-3 is governed by the open-source Apache 2.0 license, supporting broad research, modification, and redistribution.

Model Family and Resources

The Neural Chat 7B lineage comprises several iterations, with v3-3 directly succeeding v3-1, and both tracing their origins to the foundation established by Mistral-7B. Each variant integrates new fine-tuning strategies and alignment data while building on proven architectural baselines. The open-source ecosystem is supported by a suite of tools and sample code for training, quantization, and inference, available via Intel’s repositories.

Helpful Links

• Intel Neural Compressor GitHub Repository
• Intel Extension for Transformers GitHub Repository
• The Practice of Supervised Fine-tuning and Direct Preference Optimization on Intel Gaudi2
• Open LLM Leaderboard
• MetaMathQA Project Page
• MetaMathQA Paper on arXiv
• Hugging Face Leaderboard Results for Neural-Chat-7B-v3-3
• GitHub Sample Code to Reproduce Neural Chat 7B v3-3
• Intel Developers Discord