Subnet 37: Competitions

Subnet 37's competition system

Competition B7_MULTICHOICE

Goal

The purpose of this competition is to finetune the top models from our pretraining subnet to produce a chatbot.

Evaluation

Models submitted here are evaluated on a set of tasks, where each is worth a sub-portion of the overall score. The current evaluations are:

1. SYNTHENTIC_MMLU: In this task, the model is evaluated on a synthetic MMLU-like dataset from subnet 1. This is a multiple choice dataset with a large array of questions, spanning a domain of topics and difficulty levels, akin to MMLU. Currently, the dataset is generated using Wikipedia as the source-of-truth, though this will be expanded over time to include more domain-focused sources.

2. WORD_SORTING: In this task, the model is given a list of words and are required to sort them alphabetically. See the code here.

3. FINEWEB: In this task, the model's cross entropy loss is computed on a small sample of the fineweb dataset. See here for details.

4. IF_EVAL: In this task, the model is evaluated on a sythentic version of the IFEval dataset. The prompt contains a list of rules the response must follow. The full list of possible rules is listed in rule.py

Definitions

See here for more information on definitions.

Competition INSTRUCT_8B

The goal of this competition is to train a SOTA instruct 8B model. This competition provides more freedom to miners than others: there are no restrictions on the tokenizer used and miners are allowed to use a wider range of architectures.

The evaluation tasks are the same as the B7_MULTICHOICE competition

See the code for more information.

Deprecated Competitions

Competition 1: SN9_MODEL

This was the competition for the finetuning subnet.

Its purpose was to finetune the top models from subnet 9 to produce a chatbot.

Models submitted to this competition were evaluated using a synthetic Q&A dataset from the cortex subnet. Specifically, models were evaluated based on their average loss of their generated answers.