Subnet 37 Competitions
Subnet 37's competition system
Competition B7_MULTICHOICE
Goal
The purpose of this competition is to finetune the top models from our pretraining subnet to produce a chatbot.
Evaluation
Models submitted here are evaluated on a set of tasks, where each is worth a sub-portion of the overall score. The current evaluations are:
SYNTHENTIC_MMLU: In this task, the model is evaluated on a synthetic MMLU-like dataset from subnet 1. This is a multiple choice dataset with a large array of questions, spanning a domain of topics and difficulty levels, akin to MMLU. Currently, the dataset is generated using Wikipedia as the source-of-truth, though this will be expanded over time to include more domain-focused sources.
WORD_SORTING: In this task, the model is given a list of words and are required to sort them alphabetically. See the code here.
FINEWEB: In this task, the model's cross entropy loss is computed on a small sample of the fineweb dataset. See here for details.
IF_EVAL: In this task, the model is evaluated on a sythentic version of the IFEval dataset. The prompt contains a list of rules the response must follow. The full list of possible rules is listed in rule.py
Definitions
See here for more information on definitions.
Competition INSTRUCT_8B
The goal of this competition is to train a SOTA instruct 8B model. This competition provides more freedom to miners than others: there are no restrictions on the tokenizer used and miners are allowed to use a wider range of architectures.
The evaluation tasks are the same as the B7_MULTICHOICE competition
See the code for more information.
Deprecated Competitions
Competition 1: SN9_MODEL
This was the competition for the finetuning subnet.
Its purpose was to finetune the top models from subnet 9 to produce a chatbot.
Models submitted to this competition were evaluated using a synthetic Q&A dataset from the cortex subnet. Specifically, models were evaluated based on their average loss of their generated answers.
Last updated