> For the complete documentation index, see [llms.txt](https://docs.macrocosmos.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.macrocosmos.ai/subnets/subnet-13-data-universe/data-universe-validating.md).

# Data Universe Validating

### Introduction

The Validator is responsible for validating the data delivered by Miners and scoring Miners according to the [Incentive Mechanism](https://docs.macrocosmos.ai/subnets/subnet-13-data-universe/subnet-13-incentive-mechanism).

Validators run multiple concurrent threads:

| **Component**       | **Purpose**                                |
| ------------------- | ------------------------------------------ |
| Main Loop           | Evaluation cycles, weight setting          |
| Miner Evaluator     | Score calculation and validation           |
| On-Demand Processor | Data Collection job validation             |
| API Server          | External query interface (FastAPI/uvicorn) |
| W\&B Logger         | Metrics logging (rotates every 3 hours)    |
| Metagraph Syncer    | Network state updates                      |

### System Requirements

Validators require at least 32 GB of RAM but do not require a GPU. We recommend a decent CPU (4+ cores) and sufficient network bandwidth to handle protocol traffic. Must have python >= 3.10.

### Getting Started

#### Prerequisites

1. Data Universe supports Twitter and Reddit scraping via Apify so miners have to [setup their Apify API token](https://github.com/macrocosm-os/data-universe/blob/main/docs/apify.md). Validators will default to using the recommended reddit account for reliability but this can be changed editing the PREFERRED\_SCRAPERS map in validator.py locally. Data Universe also supports YouTube Scraping via a [official youtube api](https://github.com/macrocosm-os/data-universe/blob/main/docs/youtube.md).
2. Clone the repo

```
git clone https://github.com/RusticLuftig/data-universe.git
```

3. Setup python [virtual environment](https://docs.python.org/3/library/venv.html) or [Conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands).
4. Install the requirements. From your virtual environment, run

```
cd data-universe
python -m pip install -e .
```

5. Make sure you've [created a Wallet](https://docs.learnbittensor.org/keys/working-with-keys) and [registered a hotkey](https://docs.learnbittensor.org/local-build/mine-validate#1-register-the-neuron-hotkeys).

````

This will prompt you to navigate to https://wandb.ai/authorize and copy your api key back into the terminal.

## Running the Validator

### With auto-updates

We highly recommend running the validator with auto-updates. This will help ensure your validator is always running the latest release, helping to maintain a high vtrust.

Prerequisites:
1. To run with auto-update, you will need to have [pm2](https://pm2.keymetrics.io/) installed.
2. Make sure your virtual environment is activated. This is important because the auto-updater will automatically update the package dependencies with pip.
3. Make sure you're using the main branch: `git checkout main`.

From the data-universe folder:
```shell
pm2 start --name net13-vali-updater --interpreter python scripts/start_validator.py -- --pm2_name net13-vali --wallet.name cold_wallet --wallet.hotkey hotkey_wallet [other vali flags]
````

This will start a process called `net13-vali-updater`. This process periodically checks for a new git commit on the current branch. When one is found, it performs a `pip install` for the latest packages, and restarts the validator process (who's name is given by the `--pm2_name` flag)

#### Without auto-updates

If you'd prefer to manage your own validator updates...

From the data-universe folder:

```
pm2 start python -- ./neurons/validator.py --wallet.name your-wallet --wallet.hotkey your-hotkey
```

## Configuring the Validator

### Flags

The Validator offers some flags to customize properties.

You can view the full set of flags by running

```
python ./neurons/validator.py -h
```

### `.env`

Your validator `.env` should look like the following after setup for all data sources:

```
APIFY_API_TOKEN="your_apify_token"
```

Please see docs on [Apify](https://github.com/macrocosm-os/data-universe/blob/main/docs/apify.md), [Reddit](https://github.com/macrocosm-os/data-universe/blob/main/docs/reddit.md), and [Youtube](https://github.com/macrocosm-os/data-universe/blob/main/docs/youtube.md) for more information on the environment variables above.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.macrocosmos.ai/subnets/subnet-13-data-universe/data-universe-validating.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.