Subnet 13 Validating

Introduction

The Validator is responsible for validating the data delivered by Miners and scoring Miners according to the Incentive Mechanism.

Validators run multiple concurrent threads:

Component	Purpose
Main Loop	Evaluation cycles, weight setting
Miner Evaluator	Score calculation and validation
On-Demand Processor	Data Collection job validation
API Server	External query interface (FastAPI/uvicorn)
W&B Logger	Metrics logging (rotates every 3 hours)
Metagraph Syncer	Network state updates

System Requirements

Validators require at least 32 GB of RAM but do not require a GPU. We recommend a decent CPU (4+ cores) and sufficient network bandwidth to handle protocol traffic. Must have python >= 3.10.

Getting Started

Prerequisites

1. Data Universe supports Twitter and Reddit scraping via Apify so miners have to setup their Apify API token. Validators will default to using the recommended reddit account for reliability but this can be changed editing the PREFERRED_SCRAPERS map in validator.py locally. Data Universe also supports YouTube Scraping via a official youtube api.

2. Clone the repo

git clone https://github.com/RusticLuftig/data-universe.git

1. Setup python virtual environment or Conda environment.

2. Install the requirements. From your virtual environment, run

cd data-universe
python -m pip install -e .

1. Make sure you've created a Wallet and registered a hotkey.

This will prompt you to navigate to https://wandb.ai/authorize and copy your api key back into the terminal.

## Running the Validator

### With auto-updates

We highly recommend running the validator with auto-updates. This will help ensure your validator is always running the latest release, helping to maintain a high vtrust.

Prerequisites:
1. To run with auto-update, you will need to have [pm2](https://pm2.keymetrics.io/) installed.
2. Make sure your virtual environment is activated. This is important because the auto-updater will automatically update the package dependencies with pip.
3. Make sure you're using the main branch: `git checkout main`.

From the data-universe folder:
```shell
pm2 start --name net13-vali-updater --interpreter python scripts/start_validator.py -- --pm2_name net13-vali --wallet.name cold_wallet --wallet.hotkey hotkey_wallet [other vali flags]

This will start a process called net13-vali-updater. This process periodically checks for a new git commit on the current branch. When one is found, it performs a pip install for the latest packages, and restarts the validator process (who's name is given by the --pm2_name flag)

Without auto-updates

If you'd prefer to manage your own validator updates...

From the data-universe folder:

pm2 start python -- ./neurons/validator.py --wallet.name your-wallet --wallet.hotkey your-hotkey

Configuring the Validator

Flags

The Validator offers some flags to customize properties.

You can view the full set of flags by running

python ./neurons/validator.py -h

.env

Your validator .env should look like the following after setup for all data sources:

APIFY_API_TOKEN="your_apify_token"
REDDIT_CLIENT_ID="your_reddit_client_id"
REDDIT_CLIENT_SECRET="your_reddit_client_secret"
REDDIT_USERNAME="your_reddit_username"
REDDIT_PASSWORD="your_reddit_password"
YOUTUBE_API_KEY="your_youtube_api_key"

Please see docs on Apify, Reddit, and Youtube for more information on the environment variables above.