Dippy SN11: Creating The World's Best Open-Source Roleplay LLM

Please check our Launch Tweet for our vision of creating the world's best open-source roleplay LLM.*

---

• Introduction
• Roadmap
• Overview of Miner and Validator Functionality
  • Miner
  • Validator
• Running Miners and Validators
  • Running a Miner
  • Running a Validator
• Contributing
• License

---

Introduction

Note: The following documentation assumes you are familiar with basic Bittensor concepts: Miners, Validators, and incentives. If you need a primer, please check out https://docs.bittensor.com/learn/bittensor-building-blocks.

Dippy is one of the world's leading AI companion apps with 1M+ users. The app has ranked #3 on the App Store in countries like Germany, been covered by publications like Wired magazine and the average Dippy user spends 1+ hour on the app.

The Dippy Roleplay subnet on Bittensor aims to create the world's best open-source roleplay LLM by leveraging the collective efforts of the open-source community. This subnet addresses the critical issue of loneliness, which affects a significant portion of the population and is linked to various mental and physical health problems.

Current SOTA LLMs (Claude, OpenAI etc.) are designed for the assistant use case and lack the empathetic qualities necessary for companionship. While some companies (like Character AI and Inflection) have developed closed-source roleplay LLMs, the open-source alternatives lag significantly behind in performance.

Roadmap

Given the complexity of creating a state of the art roleplay LLM, we plan to divide the process into 3 distinct phases.

Phase 1:

• Subnet launch with robust pipeline for roleplay LLM evaluation on public datasets and response length
• Public model leaderboard based on evaluation criteria
• Introduce Coherence and Creativity as a criteria for live model evaluation

Phase 2:

• Publicly release front-end powered by top miner submitted model of the week
• Integrate top miner submitted model in Official Dippy App
• Add support for larger parameter models for up to 34B

Phase 3:

• Expand the state of the art in roleplay LLMs through continuous iteration and data collection
• Redefine definition of SOTA for roleplay LLMs through integrating Dippy app data

Overview of Miner and Validator Functionality

Miners would use existing frameworks, fine tuning techniques, or MergeKit, to train, fine tune, or merge models to create a unique roleplay LLM. These models would be submitted to a shared Hugging Face pool.

Validators would evaluate the and assess model performance via our protocol and rank the submissions based on various metrics (empathy, conciseness etc). We will provide a suite of testing and benchmarking protocols with state-of-the-art datasets.

Running Miners and Validators

Running a Miner

Important: Please carefully read through the FAQ and Detailed Miner Documentation. These contain critical information about model requirements, evaluation criteria, and best practices that will help ensure your submissions are valid and competitive.

Running a Validator

Requirements

• Python 3.9+

Setup

To start, clone the repository and cd to it:

git clone https://github.com/impel-intelligence/dippy-bittensor-subnet.git
cd dippy-bittensor-subnet
pip install -e .

To run the evaluation, simply use the following command:

python neurons/validator.py --wallet.name WALLET_NAME --wallet.hotkey WALLET_HOT_NAME

To run auto-updating validator with PM2 (recommended):

pm2 start --name sn11-vali-updater --interpreter python scripts/start_validator.py -- --pm2_name sn11-vali --wallet.name WALLET_NAME --wallet.hotkey WALLET_HOT_NAME [other vali flags]

If you wish to use a local subtensor node, the additional flags required are --local in additional to the typical arguments. Example:

python neurons/validator.py \
--wallet.name coldkey \
--wallet.hotkey hotkey \
--local \
--subtensor.network local --subtensor.chain_endpoint ws://chain_endpoint

Please note that this validator will call the model validation service hosted by the dippy subnet owners. If you wish to run the model validation service locally, please follow the instructions below.

Subnet Incentive Mechanism

The general structure of the incentive mechanism is as follows:

1. Every miner has a model registered per UID

2. Each miner's model submission is scored, with details outlined below

   • The scoring mechanism is constantly evolving according to SOTA model bechmark data

3. The validator compares each miner's score against all the other miners, and calculates a win rate

   • Note that there are some modifiers for a miner's score such as their submission age in relation to other miner submissions (aka time penalty) to combat blatant model copying

4. Given each miner's win rate, weights are assigned sorted by highest win rate

Model Evaluation Criteria

Model Size

A smaller model will score higher than a big model. Model size is the disk space occupied by the model repo from HF. The max model size is limited to 72GB.

Latency

A faster model will score higher than a slow model.

Output Similarity

Evaluated against datasets, a model that generates similiar resposne to groundtruth will score higher. There is an additional creativity component that utilizes the

Post Evaluation

After initial evaluation, a model will be selected for post evaluation after some time. The current process for this is a proprietary solution that is based on judging criteria from SOTA model benchmarking approaches. In the future, the details for this will be available on https://research.dippy.ai

Acknowledgement

Our codebase was originally built upon Nous Research's and MyShell's Subnets. At the time of this writing, we have deviated significantly from these subnet architectures, providing more efficiency and capability.

License

The Dippy Bittensor subnet is released under the MIT License.

Project Structure Overview

Core Components

1. Main Application

• neurons/ - Core neural network components
  • miner.py - Mining node implementation
  • validator.py - Validation node implementation
  • model_queue.py - Queue management for model processing

2. Model Management

• model/ - Model-related functionality
  • data.py - Data structures and model definitions
  • scores.py - Scoring system implementation

3. Worker API (for internal use)

• wokrer_api/ - API for model validation. Only validators and subnet operators require usage of this API. Miners do not need to set this up in 99% of cases

4. Utilities

• utilities/ - Common utility functions
  • repo_details.py - Repository management utilities
  • validation_utils.py - Validation helper functions

5. Documentation

• docs/ - Project documentation
  • miner.md - Miner setup and usage guide
  • validator.md - Validator setup and usage guide
  • FAQ.md - Frequently asked questions

Configuration Files

• pyproject.toml - Project metadata and dependencies
• requirements.txt - Main project dependencies
• requirements.api.txt - Worker API dependencies
• requirements.miner.txt - Miner-specific dependencies
• requirements.eval.txt - Evaluation-specific dependencies used for docker based evaluation
• min_compute.yml - Minimum compute requirements specification

Docker Configuration

• evaluator.Dockerfile - Docker configuration for evaluator (scoring worker)
• worker_api/vapi.Dockerfile - Docker configuration for worker API