Starling 7B (Beta)

Overview

Starling 7B (Beta) is an open large language model developed by the Nexusflow Team. It is designed to provide capabilities in helpfulness, harmlessness, and performance across a range of dialogue and programming tasks. The model builds upon the OpenChat-3.5-0106 refinement of the Mistral 7B architecture and incorporates a reinforcement learning pipeline leveraging AI-based feedback systems. Released in November 2023, its development involved a collaborative approach, drawing upon datasets and techniques contributed by several organizations and open-source communities.

Model Architecture and Training

At its foundation, Starling 7B (Beta) contains approximately 7.24 billion parameters using BF16 tensor operations, adapted from the Mistral 7B architecture via the OpenChat-3.5-0106 checkpoint. A key aspect of its development is its reinforcement learning from AI feedback (RLAIF) pipeline. This method employs a specialized reward model, Nexusflow/Starling-RM-34B, that provides automated evaluation signals, replacing human feedback with feedback generated by AI systems. The policy refinement utilizes the Proximal Policy Optimization (PPO) algorithm to fine-tune the model’s responses based on preferences encoded in its training data.

The overall training process incorporates both ranking and reward-modulated supervised data, with preference optimization aimed at aligning responses with user expectations for clarity, safety, and informativeness. The reward modeling technique addresses limitations of scalability and consistency sometimes encountered in human-annotated feedback loops.

Datasets and Data Sources

The training of Starling 7B (Beta) utilizes a diverse range of datasets. A notable dataset is the berkeley-nest/Nectar ranking dataset, which is used for comparative preference modeling. The model’s reward and preference tuning pipeline also draws from additional open community resources, including the lmsys-chat-1M dataset, which contributes to conversational depth and diversity.

The developmental process utilized contributions from open-source projects and institutions such as Anthropic, Llama, Hugging Face H4, OpenChat, OpenBMB, Flan, and ShareGPT, which provided datasets or base model checkpoints used in Starling's development. This amalgamation supports coverage of single-turn queries, multi-turn dialogue, and code-related instructions, contributing to generalized performance.

Technical Capabilities and Performance

Starling 7B (Beta) is optimized for a blend of helpfulness, harmlessness, and breadth of application across conversational and programming domains. Its RLAIF and PPO-based fine-tuning facilitate adaptive dialogue behavior, supporting single-turn user queries, contextual multi-turn exchanges, and specialized coding assistance. According to automated MT-Bench evaluation via GPT-4, Starling 7B (Beta) achieves a score of 8.12, positioning it among models of comparable scale within open-access benchmarks.

The Mistral 7B architecture and data pipeline allow it to generate text suitable for both general-purpose and technical use cases, including explanations, step-by-step reasoning, and programming solutions.

Application Scenarios and Usage Patterns

The design of Starling 7B (Beta) supports its deployment in a range of applications. It facilitates single-turn conversations, multi-turn dialogues, and task-specific coding interactions. The instruction-following behaviors are shaped by prompt engineering, utilizing templates derived from those used in OpenChat-3.5-0106. This chat prompting format involves explicit role designation and meta-tokens such as <|end_of_turn|> for turn demarcation.

For example, a typical single-turn prompt might start with, “GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant:”, while coding mode prompts follow structures like “Code User: Implement quicksort using C++<|end_of_turn|>Code Assistant:”. Text generation is influenced by adherence to these prompt formats. The model can be operated with various open-source libraries, including transformers, provided parameters such as max_length, pad_token_id, and eos_token_id are correctly specified.

Starling 7B (Beta) is suitable for education, research, technical interviews, code completion, and general conversation tasks that require consistent output moderation and informational content.

Limitations and Licensing

While Starling 7B (Beta) is engineered for concise and helpful responses, rare occurrences of unnecessarily verbose output are noted and may require mitigation through prompt adaptation, such as setting the temperature parameter to zero. The performance of the model can degrade if chat prompts deviate from the prescribed template.

Starling 7B (Beta) is distributed under the Apache-2.0 license, with additional stipulations prohibiting its use in direct competition with OpenAI. Users must comply with the terms associated with any incorporated data, particularly those governed by OpenAI’s Terms of Use and ShareGPT's Privacy Practices.

See Also and External Resources

• Nexusflow: Starling 7B (Beta) Model Card
• Starling-RM-34B Reward Model
• Mistral-7B-v0.1
• berkeley-nest/Nectar Dataset
• lmsys-chat-1M Dataset
• OpenAI Terms of Use
• ShareGPT Privacy Practices