L3.3 70B Euryale v2.3

Overview

L3.3 70B Euryale v2.3 is a large-scale generative AI model developed by Sao10K, positioned as the direct successor to Euryale v2.2. Built on top of Meta’s Llama 3.3 70B Instruct platform, this model incorporates extensive fine-tuning and a blend of custom datasets to address natural language generation tasks, particularly those favoring creative instruction and role-playing. Euryale v2.3 incorporates architectural modifications and training strategies for language modeling.

Model Architecture and Technical Foundations

L3.3 70B Euryale v2.3 is built on the Llama 3.3 70B Instruct base, itself a continuation of the Llama 3.1 70B series. The model leverages the AutoModelForCausalLM type and uses the AutoTokenizer for input handling, which are standard in large language model deployments. Employing the Axolotl deep learning framework (v0.5.2), Euryale v2.3 is trained using a modular and scalable process.

Architectural features include a sequence length of 16,384 tokens and the integration of flash attention, which functions to accelerate and optimize memory usage during attention computation. Adapters are implemented through LoRA, using parameters such as lora_r=128, lora_alpha=16, lora_dropout=0.1, and targeting linear layers. Additional architecture-level features, available via various Liger plugins, include rotary positional encoding (liger_rope), RMS and fused layer normalization, GLU activations, and fused linear cross-entropy.

These configurations result in approximately 70.6 billion parameters.

Training Data and Methodology

The training of L3.3 70B Euryale v2.3 was conducted over a single epoch, drawing on a collection of custom datasets that emphasize creative and diverse text generation. The principal datasets entail a cleaned and separated “unalignment” file (amoral-full-sys-prompt.json), role-playing and creative instruction data (mimi-superfix-RP-filtered-fixed.json), and instruction-tuning contributions from Hespera (hespera-smartshuffle.json). All datasets adopt the customllama3 type, which is compatible with the model’s architecture and objectives.

The optimization process incorporates a short warmup of 15 steps, gradient accumulation to manage memory efficiency, micro-batching (size 1), and unsloth-based gradient checkpointing. Training utilizes the paged_ademamix_8bit optimizer, implemented alongside 8-bit quantization techniques. Parameter tuning follows a cosine learning rate schedule at a learning rate of 0.000004, with regularization by weight decay (0.1), and gradient norm clipping at 25.0. Distributed training and memory efficiency are managed using DeepSpeed via a custom zero3_bf16 configuration.

Capabilities and Use Cases

L3.3 70B Euryale v2.3 is designed primarily for text generation tasks. Its curated datasets and instruction-tuning emphasize adaptability to open-ended conversation, creative writing, and simulation of role-playing dialogues. The model demonstrates alignment towards both structured instruction-following and flexible contextual storytelling, and is used for applications in interactive assistants, fictional or character-based chatbots, and content creation tools.

The model leverages the Llama 3-instruct prompt style. According to developer guidance, prompt structure and system prompts are highly influential in producing outputs, allowing users to tailor outputs for diverse scenarios, from creative writing to technical explanations.

Usage, Configuration, and Considerations

Euryale v2.3 is prompted using the Llama-3-Instruct format. Suggested decoding parameters are a temperature of 1.1 (for output diversity) and a minimum probability threshold (min_p) of 0.1 to balance creativity with coherence. The model's large sequence length accommodates extensive conversations or deeply nested prompts.

Users are advised that evaluation results are not currently displayed on the Hugging Face leaderboard due to metadata errors related to dataset specification. As with all large generative AI models, responses may vary, and prompt engineering can significantly impact output relevance and informativeness.

Model Family and Development Roadmap

L3.3 70B Euryale v2.3 is directly identified as the successor to Euryale v2.2, with claims by the developer indicating it surpasses the earlier Hanami-x1 model in certain regards. It exists within a lineage of models built on the Llama 3.3 70B Instruct foundation, with ongoing community and developer-led improvements.

Future plans for this model's family include refining dataset quality, increasing the availability of secondary chat and creative-leaning content, and active development on adjacent projects. Continued dataset enhancements and tuning are planned to modify the model's applicability and reliability in both creative and instructional domains.

Helpful Resources

• L3.3 70B Euryale v2.3 model homepage on Hugging Face
• Axolotl deep learning framework GitHub repository
• Developer's website and contact information
• Pixiv profile of てぃあ, the artist for official model artwork