Nous Hermes 13b

Overview

Nous Hermes 13B is a large language model developed by Nous Research, designed to perform a wide spectrum of natural language processing tasks. It is a fine-tuned variant of Meta AI's Llama 13B model, with enhancements in instruction-following and general language ability. The model was partially curated and fine-tuned by Teknium and Karan4D, with additional contributions from Redmond AI in providing computational resources. The development of Nous Hermes 13B aligns with ongoing efforts to broaden public access to capable open-source language models for research and other non-commercial purposes.

Technical Characteristics and Architecture

At its core, Nous Hermes 13B utilizes the Llama 13B architecture, which is notable for its balance of capability and efficiency. The model was fine-tuned on a context window of 2,000 tokens, enhancing its ability to sustain longer and more coherent responses compared to its predecessor. Released initially in the FP16 precision format, additional quantized versions—such as GGML and GPTQ 4-bit—are planned to optimize model deployment and accessibility. The architecture and training processes for Nous Hermes 13B emphasize robust instruction compliance, minimal hallucination rates, and the exclusion of restrictive moderation systems found in closed-source models such as those from OpenAI.

Training Data and Methodology

The fine-tuning of Nous Hermes 13B drew upon a diverse and extensive dataset of more than 300,000 instructions. The majority of this data was composed of synthetic outputs generated by GPT-4, contributing to the overall quality of instruction-following tasks. Key sources included the GPTeacher dataset (covering general tasks, roleplay, and code instructions), the CodeAlpaca dataset, Evol_Instruct Uncensored, GPT4-LLM, and Unnatural Instructions. Additionally, the dataset incorporated specialized content from Camel-AI's STEM-focused resources and Airoboros' GPT-4 dataset. The training procedure was conducted over a span of more than 50 hours utilizing advanced infrastructure, specifically an 8x A100 80GB DGX platform. These efforts yielded a model designed to generate accurate, contextually relevant, and creative outputs across a broad array of instructions and domains.

Benchmark Performance and Evaluation

Nous Hermes 13B has achieved competitive results on a range of academic NLP benchmarks. In evaluations using the ARC-challenge (ARC-c), ARC-easy (ARC-e), BoolQ, HellaSwag, OpenBookQA, PIQA, and Winogrande datasets, the model demonstrated strong performance in both accuracy and normalized scoring. Notably, Nous Hermes 13B demonstrated leading performance on benchmarks such as ARC-c, ARC-e, HellaSwag, and OpenBookQA, when compared with entries in the GPT4All benchmarking suite. For instance, it recorded an accuracy of 0.4915 on ARC-c and 0.7769 on ARC-e, aligning it closely with, and in some cases exceeding, several contemporaneous open models in these tasks.

Applications and Use Cases

Nous Hermes 13B is designed to excel across a variety of NLP applications. Its capacities span creative text generation, complex instruction following, coding assistance, and contextual dialogue. The model's design also supports multi-turn dialogue systems, making it well-suited for implementation in both research and entertainment chatbot scenarios. Illustrative applications include its integration in conversational and role-playing bots, as demonstrated by available open-source repositories.

Prompting Format and Usage

The model follows the Alpaca prompt format, facilitating a standardized approach to instruction input and response output. A typical prompt consists of an "Instruction" heading, with the optional inclusion of an "Input" field for supplementary context, followed by the "Response." This structure is compatible with widely-adopted prompting conventions and ensures consistency across development environments. Examples and templates for interaction using this format can be found in community repositories and in technical documentation released by model contributors.

Collaborators and Development

The creation of Nous Hermes 13B was a collaborative project involving Teknium, Karan4D, Nous Research, and computational support from Redmond AI. Further contributions were made by Huemin Art and key dataset authors, including nlpxucan, jondurbin, and organizations like Camel-AI and Microsoft. The development team also benefited from community support in debugging and performance optimization, reflecting the open and participatory ethos behind much of contemporary open-source AI research.

Limitations and Licensing

Details regarding specific model limitations and license terms for Nous Hermes 13B are not explicitly delineated in publicly available releases. Users are advised to consult the official Hugging Face model page and accompanying documentation for the most current information on permissible uses and deployment restrictions.