Pygmalion 2 7B

Overview

Pygmalion 2 7B is an instruction-tuned generative language model developed by PygmalionAI with a primary focus on fiction writing, character-driven conversation, and roleplaying. Building on the Llama-2 7B architecture from Meta AI, the Pygmalion 2 7B model has been fine-tuned using custom datasets and methodologies to enhance its capacity for interactive storytelling and lifelike dialogue generation.

Model Architecture and Design

The foundation of Pygmalion 2 7B is the transformer-based Llama-2 7B model, which consists of approximately 6.74 billion parameters and employs the BF16 tensor type for efficient computation. Its architecture enables natural language understanding and generation, suitable for a range of conversational AI applications.

Pygmalion 2 7B distinguishes itself through instruction tuning performed via supervised fine-tuning strategies. The fine-tuning process utilized the open-source Axolotl framework, which supports scalable training of large language models. Modifications to the model’s prompting structure—particularly the adoption of the "Metharme" format—allow for nuanced control over conversational context. Special control tokens such as <|system|>, <|user|>, and <|model|> define speaker roles and enable the chaining of conversation turns, facilitating more dynamic and context-aware exchanges.

Training Data and Methodology

To optimize for roleplaying and imaginative narrative output, Pygmalion 2 7B was trained on a diverse array of datasets. The primary training set includes PIPPA, a large-scale, proprietary dataset curated by PygmalionAI, containing instruction-following samples, synthetic conversations, fictional stories, and content extracted from roleplay forums. Supplementary data was incorporated from general instruction datasets, enhancing the model’s flexibility in following varied prompts.

Fine-tuning relied on supervision to guide the model’s outputs toward creative, user-aligned content. Conversations with system-level directives set using the <|system|> token help establish scenario context and modify reply styles or verbosity. This training regimen ensures that Pygmalion 2 7B can handle both structured instructional exchanges and freeform, open-ended narrative tasks.

Prompting Format and Conversational Dynamics

The Metharme prompting format in Pygmalion 2 7B’s interface was developed to ease the construction of multi-turn dialogues and complex interactions. Unlike the classic Pygmalion format, Metharme leverages explicit role markers: <|system|> for contextual or behavioral instructions to the model, <|user|> for end user inputs, and <|model|> for the model’s replies. These tokens may be repeated and nested, thus preserving extended conversational state and enabling intricate roleplay or storytelling sessions.

Within this framework, prompt engineering becomes intuitive; for example, a <|system|> prompt can establish a narrative scenario or instruct the model to provide brief or elaborate answers. This structuring supports use cases involving back-and-forth character dialogues, collaborative fiction writing, or interactive storytelling.

Applications and Intended Use

Pygmalion 2 7B primarily serves as a tool for creative writing and entertainment. Its design is optimized for producing fiction, engaging in roleplaying scenarios, and holding character-driven conversations. The model is frequently employed in environments where maintaining narrative continuity and persona is paramount, such as in collaborative novels, text-based games, or chat-based interactive experiences. Through its instruction-following and context-sensitive mechanisms, the model accommodates both guided narrative creation and spontaneous dialogue.

Limitations and Ethical Considerations

While Pygmalion 2 7B is suited for creative applications, the model was not specifically fine-tuned for safety or factual reliability. Both the base Llama-2 model and the fine-tuned outputs may contain content that is profane, inappropriate, or otherwise socially unacceptable. The lack of dedicated safety alignment means that generated text may deviate from factual accuracy and may not be suitable for use cases outside of entertainment and fictional writing. Users are advised to employ the model within its intended scope and with appropriate oversight.

Licensing and Availability

Pygmalion 2 7B is available under terms compatible with the Llama-2 community license, allowing for both commercial and non-commercial use. The training and dissemination of the model received sponsorship from SpicyChat, facilitating open access to the resource for research and development purposes.

Helpful Links

• PygmalionAI Project Page
• PIPPA Dataset
• Llama-2 7B
• SpicyChat Sponsor
• Axolotl Training Framework