# Subnet 1 Apex Subnet 1 — Apex. A routing layer for intelligence. ## Quickstart * Visit the [Apex website](https://apex.macrocosmos.ai/) * Miners: * Read through: * [Current Competitions](/subnets/subnet-1-apex/subnet-1-current-competitions) * [Miner Setup](/subnets/subnet-1-apex/subnet-1-base-miner-setup) and [Apex CLI](/subnets/subnet-1-apex/subnet-1-base-miner-setup/apex-cli) guides * [Incentive Mechanism](/subnets/subnet-1-apex/incentive-mechanism) * Validators: * Read through the [Validator Setup](/subnets/subnet-1-apex/validating) guide ## Introduction **Apex is a routing layer for intelligence.** You provide a problem. Apex provides research, iteration, improvement, and real results. * A competition structures a well-defined problem and a way to score solutions. * A global network of independent developers and agentic agents competes to solve it. * The platform returns the best-performing solution. Apex is not a chatbot, a model marketplace, or a one-off prize platform. It is infrastructure for converting clearly-specified problems into working solutions — on demand, at scale, and without the need to recruit, manage, or evaluate researchers yourself. The premise is simple: if you can define what "better" looks like, this gets fitted down to a scoring function that Apex measures success on. ### Who Apex is for Apex is built for organizations that have a measurable challenge but don't want to staff or wait on an internal research team to solve it: * **Enterprises** with a quantifiable bottleneck — a forecasting model that needs to be more accurate, a compression ratio that needs to be tighter, a trading strategy that needs better risk-adjusted returns. * **Research labs and foundations** that want to crowdsource sustained progress on an open benchmark instead of running a single bounty. * **Product teams** that need a working algorithm as a component — not a paper, not a prototype, but code that runs. * **Domain experts** who can specify what "better" looks like in their field but don't have the ML or systems-engineering depth to build it themselves. You don't have to know *how* the solution will be built. Apex and its miners source the innovation. ### How Apex works Every competition on Apex follows the same lifecycle, regardless of whether the underlying task is compression, control, a head-to-head game, or a systems-design problem: 1. **Problem specification.** A task, a dataset or simulation environment, a scoring function, and any constraints (runtime, model size, allowed dependencies) are clarified. 2. **Competition launch.** The competition is published to the network. Contributors on the network see the spec, the leaderboard, and the reward structure. 3. **Submission.** Miners build solutions independently and submit them. Submissions can be source code, model weights, or both — whatever the task requires. 4. **Sandboxed evaluation.** Every submission runs in an isolated, secure sandbox against identical inputs and identical resource limits. Scores are deterministic and reproducible. 5. **Continuous ranking.** Leaderboards update in real time. Miners iterate, resubmit, and compete. Better solutions displace weaker ones. Miners vie for the top spot — it's winner takes all. 6. **Reward distribution.** Emissions flow to the top-ranked miner. If a better submission arrives, the new leader takes over. See the [Incentive Mechanism](https://docs.macrocosmos.ai/subnets/subnet-1-apex/incentive-mechanism) page for how originator protection and emission burning keep the network pushing forward. 7. **Delivery.** The customer receives the top-ranked solution(s) along with evaluation metadata, ready to deploy, integrate, or study. **You bring the problem. You describe what makes a good solution. Everything else — the algorithms, the experimentation, the engineering — is handled across a competitive, decenralized network of miners that are paid only for measurable progress.** ### What's Next Apex is built to host many competitions at once. New tasks are added as customers, partners, and the community bring novel problems to the platform. Miner emissions are split across active competitions so that each one carries genuine incentive weight, and the same submission, sandboxing, scoring, and reward infrastructure backs every one of them. If you have a problem that is measurable, well-specified, and worth solving — Apex is built for you. ### Collaborators Want to transform your idea into a competition on Apex? Reach out to us through email at . ### Related Resources * [Website](https://apex.macrocosmos.ai/) * [Apex X (Twitter)](https://x.com/Apex_SN1) * [GitHub](https://github.com/macrocosm-os/apex) * [Substack](https://macrocosmosai.substack.com/t/language-models) * [Bittensor Discord](https://discord.com/channels/799672011265015819/1161764867166961704) * [Macrocosmos Discord](https://discord.com/channels/1238450997848707082) * [Cosmonauts - Macrocosmos Telegram](https://t.me/macrocosmosai) * [Macrocosmos X (Twitter)](https://x.com/MacrocosmosAI) # Subnet 1 Incentive Mechanism Subnet 1 incentive and operation The core idea behind the subnet is to share problem-solving code openly to enhance competition results, round by round. Open-sourcing these solutions widens impact and helps tackle real-world challenges. In the Matrix Compression competition — the first event on the subnet — the winning solution will not only directly enhance Subnet 9’s output but may also contribute to optimising data centres and cloud infrastructure. ### General Operations * **Solution Submission**\ Miners submit their solutions to the competition to the submission endpoint API. * **Execution and Metrics**\ Each solution runs inside an isolated secure sandbox, generating its evaluation metrics. * **Evaluation**\ The subnet validator reviews these metrics and assigns a score to each submitted solution. * **Scoring and Selection**\ Validators retrieve the scores from the subnet orchestrator to determine the winning miner. * **Rewards**\ The miner with the highest score receives the full reward. If that solution is replaced or improved, the new best submission takes over and earns the rewards of the competition. All the competitions are "winner takes all" for subnet 1.
### Incentive Challenges #### 1. How does SN1 prevent successful solution code from being stolen and resubmitted for reward? To ensure miners are fairly rewarded for their best solutions, the submitted code remains hidden for a set period. Once evaluation is complete and rewards are distributed, the code is made public for others to explore.
The winner’s code is revealed with a delay. The duration of the hidden code period depends on the competition. Logs are published only after the completion of a competition round. This approach guarantees creators to fully exercise the submission rewards, while still allowing others to learn from and improve existing solutions — tapping into the power of community innovation.
{% hint style="info" %} ### Warning: Running Miner Submission Code Running miner submission code may involve executing third-party code that is **not authored, audited, or controlled by Macrocosmos (**[**see Disclaimer**](#disclaimer-of-responsibility)**)**. Miner submissions can contain bugs, malicious logic, or unsafe configurations. **Only run miner submission code in isolated, sandboxed, or disposable environments.** {% endhint %} #### 2. How are miners driven toward continuous improvement? The emission burning mechanism was created to push successful miners toward improving their code. Once a top-performing solution is submitted, its associated emissions gradually start to burn. The longer the solution remains unchallenged, the higher the burn rate becomes — motivating continuous innovation and competition.
#### 3. How are code submissions validated? The Code Executor is a part of the subnet where miner code is ran and evaluated. Evaluation metrics depend on the competition. In case of the Matrix Compression competition such metrics are: * Compression ratio - the max level of compression provided by solution * Compression speed - the task\_time, required to run a compression and decompression processes After solutions run in the Code Executor, metrics are converted into score in accordance with the given competition's scoring mechanism. ### Disclaimer of Responsibility Macrocosmos **does not review, verify, endorse, or guarantee** the safety, correctness, or security of miner submission code provided by third parties. By running miner submission code, you acknowledge and agree that: * You are solely responsible for reviewing and evaluating the code before execution * You assume **all risks** associated with running third-party miner submissions * Macrocosmos is **not responsible or liable** for: * System damage * Data loss * Security breaches * Financial loss * Network or infrastructure compromise * Any other direct or indirect damages Miner submissions are executed **at your own risk**. # Subnet 1 Mining Your guide to setup and participation as a miner. ### Introduction **Apex** drives algorithmic innovation across diverse problem domains. Each pursuit of the best solution takes place within a **Competition**, which consists of multiple **Rounds** of evaluation. Participants, known as miners, join Competitions by submitting their solutions through the [Apex CLI](/subnets/subnet-1-apex/subnet-1-base-miner-setup/apex-cli), and earn rewards based on their performance. ### Prerequisites To setup a miner on Apex you will need the following: * A registered [Bittensor wallet](https://docs.bittensor.com/working-with-keys) * If your chosen competition has a submission fee, the wallet must contain enough funds per submission to pay for it. ### Setting Up the Environment To get started, [clone the repo](https://github.com/macrocosm-os/apex) and run `./install_cli.sh`. Then, activate the environment created by running `source .venv/bin/activate` . ### Creating Your Solutions: Miners are encouraged to review the baseline solutions for available competitions before submitting their own. All baseline solutions and general submission templates can be found in their respective competition folders within `shared/competition/src/competition`. Keep all function signatures identical to the baseline to ensure that your submission can be evaluated properly. To understand the current competitions and their scoring guidelines visit the [Current Competitions](/subnets/subnet-1-apex/subnet-1-current-competitions) page. ### Submitting Your Solutions To use the CLI to submit miner solutions, pull previous winning submissions, and view the submission dashboard, visit the [CLI Usage](/subnets/subnet-1-apex/subnet-1-base-miner-setup/apex-cli) page. # Apex CLI Instructions on using the Apex CLI. The Apex CLI is a miner's interface with the subnet: linking wallets, submitting competition solutions, and viewing the dashboard. The dashboard contains all miner submissions to past and current competitions - use it to view others' code submissions and logs to compare your individual performance against the subnet. To use the CLI, you must have a [registered wallet on subnet 1.](https://docs.learnbittensor.org/miners#miner-registration) ### Setup Before using the Apex CLI, make sure you have ran `./install_cli.sh`. For further instructions, see the [mining](/subnets/subnet-1-apex/subnet-1-base-miner-setup) page. * Make sure you have activated your `.venv` prior to using the CLI. Do so with: * `source .venv/bin/activate` ### Link Wallet Link your registered wallet with the CLI - required for most CLI commands: `apex link` This will prompt you to enter your Bittensor wallet location, which defaults to the default Bittensor wallet path: `/Users/{USERNAME}/.bittensor/wallets` . * To select the default wallet location, press *ENTER/RETURN*. * For custom wallet locations, enter the path from the root directory. Then, use the arrow keys to select your registered coldkey and hotkey from the list provided. NOTE: Some previously created wallets using earlier versions of btcli may not a have a private key configured. If this is the case, regenerate this hotkey before linking. ### View Competitions #### To view the currently active competition: `apex competitions` This will return a list of all active competitions and their associated competition IDs. #### To view competitions and their status in detail: `apex dashboard` Example Output:

Competition 1, Round 1

### View Submissions To view a submission, first open the dashboard `apex dashboard` and press *ENTER/RETUR*N when hovering the competition of interest.
Your own submissions will be viewable immediately, other's submissions will be viewable after a delay. ### Submit a Solution `apex submit` This will prompt you to enter the file path to your solution path and competition ID. Or, in one line: ``` apex submit -c ``` * View the current \ and \ via the dashboard. * Submissions are limited to **4 submissions per day,** **per hotkey**. # Subnet 1 Validating ### Prerequisites To setup a validator on subnet 1 Apex, you will need the following: * A Registered [Bittensor wallet](https://docs.bittensor.com/working-with-keys) with at least 35k Alpha staked. ### Getting Started To get started, [clone the repo](https://github.com/macrocosm-os/apex), setup your .env file, and run `./start_validator.sh` , activating it with `source .venv/bin/activate` Clone your validator keys, and add the following to your .env file. ``` WALLET_NAME="" WALLET_HOTKEY="" BITTENSOR=True ``` The default subtensor network is Finney. If you would like to use a local subtensor, add the following to your `.env`: ``` NETWORK="" ``` ### Auto-updates with PM2 A validator auto-update script is located in the `scripts` folder. The script spawns a PM2 process which runs the validator, restarting when the repo has a new release. To run this script: ``` ./scripts/start_autoupdater_pm2.sh ``` # Subnet 1 Current Competitions Registry of competitions currently active on SN1 APEX.
Cover image
Energy Arbitrage/files/O0aHybQY6z3q5v3Nacdn
Text Clustering/files/FVxWAomWmLYCsUrDvETa
Humanoid Parkour/files/K91NnGf8E3Esc0mjahKR
Humanoid Olympics/files/8urUarc8bP3NZMJdifVb
## # Humanoid Olympics Competition ## Humanoid Olympics Competition **Athletics is a breadth test.** A sprinter that cannot corner, a hurdler that cannot jump, or a jumper that cannot run are all narrow controllers; a real athlete is one body under one policy across every discipline. Learned locomotion is usually trained per skill, and the specialisation shows the moment the task changes. The Humanoid Olympics competition challenges miners to enter a **Unitree G1** humanoid into a six-discipline meet in simulation — sprints, hurdles, and jumps — with **one** submitted policy. Miners submit a trained ONNX controller that outputs joint targets at 50 Hz. The robot has twelve actuated leg joints; its arms and upper body are mass and collision geometry, but are **not actuated**, so every event is solved with the legs. The event geometry is fixed and public, but each attempt runs under its own surface friction and wind, **neither is observable, and both are re-drawn every round** from a seed you never see. A policy has to feel the slip or the push and adapt rather than memorise a condition set, which is why the interface carries recurrent state. ### Evaluation Overview Each evaluation runs the miner's policy through a full meet: **four attempts of each of the six events, 24 attempts in total**. Every attempt draws a different point on the same public condition lattice: | Condition | Range, and whether it moves between rounds | | ---------------- | ---------------------------------------------------------------------------------------------------------------- | | Surface friction | Band µ ∈ \[0.30, 1.25]. Four strata per event, 0.2375 µ apart, ±0.076 per-slab jitter. **Re-drawn every round.** | | Wind speed | Cap 8 m/s. Four strata per event, 2 m/s apart, steady for the attempt. **Re-drawn every round.** | | Wind direction | Uniform over the full circle, horizontal. **Re-drawn every round.** | | High-jump bar | 1.00, 1.10, 1.20, 1.30 m above the deck — one per attempt. **Fixed.** | * **Friction and wind are not static. They are re-drawn every round from a seed you never see.** There is no fixed set of 24 conditions to learn: a policy that is tuned to the exact operating points of one round meets different ones in the next. What is stable is the *shape* of the meet — six events, four attempts each, four strata per event — not the values. * Every event samples **four friction strata** in each round, so a policy sees near-full grip and a genuinely slippery track within the same meet. Friction is drawn once per attempt and applied to the whole route, with a small per-slab jitter. * The bottom stratum is a **real slip regime**, and how slippery it is depends on the round. Across 3000 round seeds a round's hardest attempt lands anywhere in **µ 0.30–0.48**, and its grippiest in **µ 1.06–1.25**. So some rounds bottom out well below the point where a foot can push hard enough to sprint or convert an approach into a take-off without managing traction, and others do not go as low. Expect the low strata to cost time, and to foul jumps taken as if the track were dry. * Wind acts through MuJoCo's fluid model at air density 1.204 kg/m³, so drag scales with the robot's velocity relative to the air — a headwind costs more than a tailwind, and it costs disproportionately more at the top of the wind band. * The four strata are paired with **opposing wind directions**, so a lap or a runway is not systematically favoured. * The meet's **structure** is fixed and public, but its **conditions are drawn per round**. Every round runs 24 attempts in the same shape — six events, four attempts each, four strata per event — and the platform round seed phase-shifts where in each band those strata land, independently per event, so the six disciplines do not shift together. * Be precise about what is equal between rounds: the **spacing** is, the **window** is not. Within an event the four strata are always exactly 0.2375 µ apart (one quarter of the band) and 2 m/s apart in wind, but the whole set slides. Measured over 3000 round seeds, a round covers 0.75–0.95 of the 0.95-wide friction band. Every round is therefore a broad sweep, but not an identically hard one — and a policy tuned to one round's exact 24 conditions does not carry over. * Within a round, **every submission faces identical conditions**. The seed is the round's, not the submission's, and the incumbent is re-scored on it alongside the challengers. * **The round seed is never published.** It is not in your observations, not in your reset call, not in your result, and not in your history files. Post-round you are told the conditions you actually ran — friction level and µ, wind speed and direction, per attempt — because you need to know what you were scored on. You are not told the value that generated them, and you cannot get the next round's conditions from the ones you were given. * Because conditions move, **absolute scores are only comparable within a round**. Measured on the baseline artifact, the same policy spans roughly 4% of its mean score across 20 round seeds (CV \~1.1%) — see the takeover note under Scoring. #### The Events Six disciplines, equally weighted. All routes are a **1.7 m lane** on a raised deck 0.8 m above the floor — the floor is visible to the terrain scan and to the renderer, but is never a support, so a gap is a real fall. | Event | Layout | Skill tested | | --------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------ | | 100 m sprint | Straight lane, finish at 100 m, 24 s cap (≈4.25 m/s average) | acceleration and top speed | | 400 m circular sprint | A true 400 m circle, radius ≈63.66 m, 72 s cap (≈5.56 m/s average) | sustained pace, route following, cornering | | 100 m hurdles | 10 barriers 8.5 m apart from 12 m to 88.5 m, rising 0.55 → 1.15 m, 38 s cap | repeated clearance at speed | | High jump | Runway, then a physical bar at 12 m, 1.00–1.30 m above the deck, 18 s cap | one clean vertical clearance and crossing | | Long jump | Runway, 0.40 m take-off board at 15 m, a **6 m real void**, sand from 21 m, 20 s cap | approach, take-off, flight, safe landing | | Triple jump | Board at 12 m, hop pad 14.0–15.5 m, step pad 18.0–19.5 m, sand from 25 m, 28 s cap | an ordered hop, step, and final landing | * The 400 m is a **genuine 400 m route**, not a scaled lap. Progress is accumulated forward route distance around the circle, and the lap only completes after 400 m travelled inside the lane, so cornering is part of the task rather than something to be cut. * The hurdles form a within-run curriculum: the first barrier is a step, the last is at the frontier of what a legs-only G1 should clear at speed. Each hurdle overhangs the lane, so it cannot be skimmed around the end. * The triple jump's gaps are real: 1.95 m from the board to the hop pad, 2.5 m from hop to step, and 5.5 m from step to sand. * Throwing events and the pole vault intentionally belong to a future competition — they need arms. #### Step by Step At each of the 50 Hz control steps, the miner's policy receives a **104-float observation**, in the robot's yaw frame: * Proprioception: projected gravity, base angular and linear velocity, 12 joint angles and velocities, previous action. * Task: gait clock (sin/cos of a 0.8 s cycle), heading error against the **route tangent**, signed cross-track offset, distance remaining, pelvis height above the surface below. * Terrain: a 9 × 5 height scan (45 rays) reaching **6 m ahead**, plus 7 overhead-clearance samples out to 4 m. * Downward channels report **walkable surfaces only** — the high-jump bar and the hurdles appear in the overhead channels and nowhere else. * The 6 m horizon is deliberate: at 6 m/s the inherited 1.6 m scan was only a quarter of a second of warning, which is not enough to set up a hurdle or a take-off. * The policy returns **12 floats**: joint position targets as offsets from the default pose, driven by a PD loop. This is a position target, **not** a torque. * The policy also threads an opaque **256-float recurrent state**, zeroed at the start of every attempt. Friction and wind are not in the observation, so remembering that you just slipped is the only way to adapt. Physics runs at 500 Hz (2 ms), ten substeps per control action. Every legality gate — lane boundary, hurdle and bar contact, take-off, flight, and landing — is sampled at **each physics substep**, not once per action, so a fast scrape cannot slip between control frames. #### Constraints * Termination gates shared by all events, each surfaced to the miner as a `terminal_reason`: * `fell` — pelvis drops below 0.45 m of clearance above the surface below it, or the torso tilts past \~66°. * `out_of_bounds` — cross-track offset from the route centreline exceeds **0.85 m**, half the lane width. No running around the obstacles. * `physics_glitch` — NaN/Inf state or |qvel| > 100. Glitch-surfing scores 0. * `timeout` — the event's step budget elapsed. * Event-specific outcomes: * Races: `completed` at the finish line; `hurdle_hit` if the robot touches any barrier. * High jump: `cleared` requires the pelvis to cross the bar plane at least 0.08 m above the bar after both feet have been unsupported for 40 ms, without touching the bar, followed by a supported landing 0.75 m past it. Otherwise `bar_hit`, `bar_missed`, or `high_foul`. * Long and triple jump: `landed` requires a one-foot take-off from the 0.40 m board (0.35 m before the line to 0.05 m after), a real flight, and sustained first foot support on sand. Triple jump additionally requires a same-foot hop landing and an opposite-foot step landing, in order, each with its own flight. Anything else — a side scrape, a non-foot contact, a premature pad contact, a body-first landing, overrunning the board — is a `jump_foul`. * Only **top-face foot contacts** count as support, classified by contact normal and impulse. Landing distance is latched from the first legal sand contact point, not from a later pose. * Episode length, per attempt: **1,200** control steps for the 100 m, **3,600** for the 400 m, **1,900** for the hurdles, **900** for the high jump, **1,000** for the long jump, and **1,400** for the triple jump — at most 40,000 control calls for a full meet. * Timeouts: * Per-`/act` deadline = 500 ms. * Referee (scorer) timeout = 900 seconds for the whole 24-attempt meet, with an internal 840-second scheduling budget so a result is always persisted. Any attempt that the budget does not reach is retained as a zero-scored row in the fixed denominator. * Player timeout = 1200 seconds, deliberately longer so the policy server outlives the referee. #### Scoring Every attempt returns a bounded score in `[0, 1]`. A completed attempt scores from **0.25** upward; an incomplete attempt stays **below 0.25**, so no amount of partial progress can outrank a legal finish. ``` races (100 m, 400 m, hurdles) completed 0.25 + 0.75 * (1 - steps / max_steps) -> (0.25, 1.0] fell / timeout / hurdle hit 0.24 * progress -> [0.0, 0.24] high jump cleared 0.25 + 0.75 * (bar - 1.00) / 0.30 -> [0.25, 1.0] bar hit / missed / fell 0.24 * best_clearance / bar -> [0.0, 0.24] long jump / triple jump landed 0.25 + 0.75 * (distance - min) / (target - min) long jump min 6 m, target 12 m triple jump min 13 m, target 18 m no legal landing 0.20 * progress -> [0.0, 0.20] foul, out of bounds, physics glitch, invalid action, player fault 0.0 Where progress is the furthest route distance reached, as a fraction of the event's finish. ``` * Quality within a legal result is **pace** for races, **selected bar height** for the high jump, and **legal distance** for the horizontal jumps. * Partial progress still scores, so a policy that does not finish an event gets a training gradient and a meaningful ranking — but a foul does not. Invalid contacts earn exactly nothing. * The round result is the **mean of the six event means**, each event mean taken over its four attempts: ``` raw_score = mean(event_mean[100m, 400m, hurdles, high jump, long jump, triple jump]) ``` * Macro-averaging is what keeps the meet balanced: the 400 m has three times the steps of the high jump but exactly the same weight, so a policy cannot buy a rank with one long event. * To surpass the current winner, a miner must earn a raw score > 1% higher than the current top raw score. * If there is no current winner, the miner must beat the baseline raw score by at least 1%. `baseline_raw_score` is **0.0** by design, so round 1 goes to anything scoring above zero. * At the start of each round the incumbent is automatically re-submitted and re-scored, so the comparison is always like-for-like. **This is what makes per-round conditions safe.** The incumbent's stored score from an easier draw is never what a challenger has to beat: both are scored on the same round seed, so the condition draw is common-mode in the comparison. Note the margins involved — cross-round condition spread on a fixed policy is CV \~1.1%, the same order as the 1% takeover threshold, so a *stale* incumbent score would be roughly a coin flip rather than a bar. * The `score_to_beat` is displayed in the Apex CLI dashboard under competition information. The launch preset is deliberately severe — a 24 s 100 m, a 72 s circular 400 m, hurdles to 1.15 m, bars to 1.30 m, a 6 m long-jump void. **A first complete all-round performance is intended to be a meaningful breakthrough**, and partial credit exists so that the climb toward it is measurable. #### Miner Submissions * Miners submit a single **ONNX graph** with this exact tensor signature: * inputs — `obs [batch, 104]`, `state_in [batch, 256]` * outputs — `action [batch, 12]`, `state_out [batch, 256]` * all `float32` * The **architecture is not constrained**; only the signature is. A feed-forward policy can ignore `state_in` and return zeros, but will struggle to adapt to unobservable friction and wind. * Maximum submission size: **15 MB**. This is a compute limit as well as a storage one — inference cost is linear in artifact size, and the cap pairs with the 40,000-call meet to keep the worst case inside the referee's 900 s timeout. * Evaluation runs on **2 CPU / 2 GiB / no GPU**. * Default round length: **1 day**. * Submission fee: **$20 USD**. * 1% `raw_score` threshold to beat the current top scorer. * Miners' models are revealed **1 day** after evaluation. * Logs are opened after the current round is completed. Every one of the 24 attempts also produces a replayable history file, delivered to the miner post-round. * The submission rate limit is 4 submissions per hotkey within 24 hours, across all competitions. * The full environment — all six events, physics, scoring, and a reference baseline policy — is public at [apex-competition-humanoid-olympics](https://github.com/macrocosm-os/apex-competition-humanoid-olympics). Train against the real referee, not a reimplementation. * Local tools in that repo: `tools/local_eval.py` scores a policy across the meet in-process, `tools/preview.py` renders or films a single event, and `tools/replay.py` films a recorded run. A complete seed-1 baseline meet is committed under `docs/example-histories/` as a worked example. # Humanoid Parkour Competition Physics-simulated legged locomotion and obstacle traversal **Legged locomotion over rough terrain** is one of the hardest open problems in robotics: a biped has to stay upright while climbing, dropping, leaping and balancing, on surfaces whose grip it cannot see and against disturbances it cannot predict. Traditional controllers are hand-tuned per obstacle; learned policies have to generalise. The Humanoid Parkour competition challenges miners to drive a **Unitree G1** humanoid through a 51 m obstacle course in simulation. Miners submit a trained ONNX locomotion policy that outputs joint targets at 50 Hz. The course geometry is fixed and public, but every evaluation instance draws its own surface friction and wind from a per-round seed — and **neither is observable**. A policy has to feel the slip or the push and adapt, which is why the interface carries recurrent state. #### Evaluation Overview Each evaluation runs the miner's policy across **24 course instances**. The course is identical in every instance; what changes is the conditions: | Condition | Range, drawn per instance | | ---------------- | -------------------------------------------------- | | Surface friction | µ ∈ \[0.35, 0.50] course-wide, ±8% per-slab jitter | | Slick patch | µ ∈ \[0.08, 0.14] (near-ice → wet tile) | | Wind speed | 0–14 m/s, steady for the episode | | Wind direction | Uniform over the full circle, horizontal | * Friction is drawn **once per instance** and applied to the whole course, so a run happens on one surface family rather than a patchwork. * Wind acts through MuJoCo's fluid model at air density 1.204 kg/m³, so drag scales with the robot's velocity relative to the air — a headwind costs more than a tailwind. * 14 m/s is Beaufort 7 measured *at the robot*, worth 35.1 N of drag against the G1's 315 N weight (11.1% of body weight). * Each round's evaluation set is seeded to ensure determinism. * This seed changes from round to round, so a memorised suite is worthless. **The Course** 51.14 m, one fixed layout, 1.68 m of vertical range: | Obstacle | Detail | | ------------ | --------------------------------------------------------------- | | On-ramp | 15.38° incline, then a sheer 0.55 m drop | | Stairs up | 5 steps, 0.20 m rise | | Void leap | 1.0 m gap — a real hole in the deck, so a missed leap is a fall | | Drop-down | 0.6 m | | Hurdle | 0.62 m barrier to step over (the robot has no usable arms) | | Step-up | 0.55 m platform, then back down | | Duck-under | Bar at 1.05 m, forcing a \~0.2 m squat-walk on a 1.26 m robot | | Balance beam | 3.5 m long, 0.32 m wide | | Slick patch | 3.0 m of the lowest-friction surface in the round | | Stairs down | 6 steps, 0.18 m drop | **Step by Step** At each of the 50 Hz control steps, the miner's policy receives a **104-float observation**, in the robot's yaw frame: * Proprioception: projected gravity, base angular and linear velocity, 12 joint angles and velocities, previous action. * Task: gait clock (sin/cos of a 0.8 s cycle), heading error, lateral offset, distance to the finish line, pelvis height above the surface below. * Terrain: a 9 × 5 height scan (45 rays) of walkable surface height relative to the pelvis, plus 7 overhead-clearance samples ahead. * Downward channels report **walkable surfaces only** — the duck bar appears in the overhead channels and nowhere else. * The policy returns **12 floats**: joint position targets as offsets from the default pose, driven by a PD loop. This is a position target, **not** a torque. * The policy also threads an opaque **256-float recurrent state**, zeroed on reset. Friction and wind are not in the observation, so remembering that you just slipped is the only way to adapt. **Constraints** * Termination gates, each surfaced to the miner as a `terminal_reason`: * `completed` — pelvis past the finish line. * `fell` — pelvis drops below 0.45 m of clearance above the surface below it, or the torso tilts past \~66°. * `out_of_bounds` — lateral offset |y| > 1.2 m. No walking around the course. * `physics_glitch` — NaN/Inf state or |qvel| > 100. Glitch-surfing scores 0. * `timeout` — the step budget elapsed. * Episode length: up to **3000 control steps** per instance (60 s of simulated time). * Timeouts: * Per-`/act` deadline = 500 ms. * Referee (scorer) timeout = 900 seconds for the whole 24-instance suite. * Player timeout = 1200 seconds, deliberately longer so the policy server outlives the referee. **Scoring** Per instance, higher is better: ``` completed 1.0 + (max_steps - steps) / max_steps -> (1.0, 2.0] fell / timeout / out_of_bounds progress (fraction of course covered) -> [0.0, 1.0) physics_glitch / invalid action 0.0 Where progress is the furthest distance reached, as a fraction of 51.14 m. ``` * Any completion outranks any non-completion, and faster completions outrank slower ones. * Partial progress scores, so a policy that does not finish still gets a training gradient and a meaningful ranking. * The miner's `raw_score` is the **mean across all 24 instances**. * To surpass the current winner, a miner must earn a raw score > 1% higher than the current top raw score. * If there is no current winner, the miner must beat the baseline raw score by at least 1%. `baseline_raw_score` is **0.0** by design, so round 1 goes to anything scoring above zero. * At the start of each round the incumbent is automatically re-submitted and re-scored on the new round's conditions, so the comparison is always like-for-like. * The `score_to_beat` is displayed in the Apex CLI dashboard under competition information. **Miner Submissions** * Miners submit a single **ONNX graph** with this exact tensor signature: * inputs — `obs [batch, 104]`, `state_in [batch, 256]` * outputs — `action [batch, 12]`, `state_out [batch, 256]` * all `float32` * The **architecture is not constrained**; only the signature is. A feed-forward policy can ignore `state_in` and return zeros, but will struggle to adapt to unobservable conditions. * Maximum submission size: **15 MB**. This is a compute limit as well as a storage one — inference cost is linear in artifact size, and the cap pairs with the 3000-step episode to keep the worst case inside the referee's 900 s timeout. * Default round length: 1 **day**. * Submission Fee: $20 USD. * 1% `raw_score` threshold to beat current top scorer. * Miners' models are revealed **1 day** after evaluation. * Logs are opened after the current round is completed. Each instance also produces a replayable history file, delivered to the miner post-round. * The submission rate limit is 4 submissions per hotkey within 24 hours, across all competitions. * The full environment — course, physics, scoring, and a reference baseline policy — is public at [apex-competition-humanoid-parkour](https://github.com/macrocosm-os/apex-competition-humanoid-parkour). Train against the real referee, not a reimplementation. * Local tools in that repo: `tools/local_eval.py` scores a policy against the course in-process, `tools/preview.py` renders the layout, and `tools/replay.py` films a recorded run. # Ended: Aurelius Steering Competition Large Language Model Steering ## Aurelius Steering Competition A partnership between **Aurelius (SN37)** and **Macrocosmos (SN1)** to crowdsource the discovery of *concept directions* inside a language model — vectors that, when added to the model's activations, reliably steer its output toward a target concept. ### What this is In 2024 Anthropic released [Golden Gate Claude](https://www.anthropic.com/news/golden-gate-claude): by amplifying a single internal feature they made the model obsess over the Golden Gate Bridge, mentioning it in nearly every reply regardless of the prompt. That demo showed something powerful — a model's behavior can be *steered* by manipulating a direction in its internal representation space, not by retraining or prompting. This competition turns that idea into an open, incentivized search. Miners submit a steering direction for a hidden concept; the validator adds that direction to the model's residual stream during generation and measures how strongly the resulting completions exhibit the concept. The miner whose direction steers the model most effectively wins the round. The goal is to explore — at scale and across many independent miners — how well a model can be steered toward a *specified* concept using a single learned direction. ### How it works at a glance 1. Each round targets one **concept** (e.g. *positive sentiment*). Prompts are never revealed in advance, but the concept name itself is published so miners know what to steer toward. 2. A miner crafts a single **steering direction** (a unit vector in the model's hidden space) plus a scalar **alpha** (how hard to push), and submits it as a small `.safetensors` file. 3. At the **end of the round**, all submissions are scored **in a batch** on GPU: the validator loads the model, adds `alpha × direction` to the residual stream at a fixed layer, generates completions for a held-out prompt sample, and scores how strongly each completion expresses the concept. 4. A completion only counts if it is also **coherent**: the unsteered base model judges every completion, and incoherent ones score zero (see Coherence gate). Steering so hard that the model degenerates into concept-spam does not pay. 5. Scores are **baseline-adjusted and normalized** so the unsteered model scores 0.0 and a perfect steer scores 1.0. The highest score wins the round (winner-take-all, standard Apex incentive). ### The model and the steering mechanism * **Model:** `google/gemma-3-12b-it` at a pinned revision, loaded **4-bit NF4** with **bf16** compute. All miners calibrate against this identical configuration — the steering directions are specific to it. * **Hidden size:** `3840`. A direction is a vector of this dimension. * **Steer layer:** `32` (fixed for the whole competition). The validator registers a forward hook that, at **every token position** during generation, does: ``` hidden_states[layer 32] += alpha * direction ``` * **Decoding:** greedy (`do_sample=False`, `num_beams=1`), fixed per-round seed, short completions. Greedy + fixed seed makes scoring reproducible across machines on the same GPU architecture and pinned image. Because the direction is **L2-normalized to unit norm**, `alpha` is the *sole* knob controlling steering strength, and submissions are directly comparable across miners. ### Concepts The scorer is concept-agnostic; each round names one active concept. The four supported concepts: | Concept | What a steered completion looks like | | -------------------- | ------------------------------------------------------------------------------- | | `birthday_cake` | Mentions birthday cake, candles, parties, "happy birthday" | | `medical_disclaimer` | Adds disclaimers like "consult a healthcare professional", "not medical advice" | | `positive_sentiment` | Skews strongly positive in tone (AFINN-style net valence) | | `hedging` | Uses hedging language — "perhaps", "it might", "arguably", "it depends" | Each concept is scored by a deterministic, version-pinned lexicon/detector, so results are reproducible. **Which concept goes in your file:** every submission declares its target concept in the safetensors `concept` metadata key (one of the four strings above — e.g. `"birthday_cake"`). You must set it to the **round's currently-active concept** — the one published for the round you are submitting into — *not* whichever concept you happened to train your direction on. The active concept is announced per round and changes manually over time (see Round structure); a direction built for last round's concept must be re-declared (and ideally re-derived) for the new one. A submission whose declared `concept` does not match the round's active concept is **rejected, not scored 0** — and this is now caught at submission time by a screener, so you find out immediately instead of at end-of-round. ### Submission format A single `.safetensors` file (\~15 KB) containing exactly one tensor plus three metadata keys: | Field | Requirement | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | | tensor `direction` | shape `(3840,)`, dtype `float32`, finite, **L2 unit-norm** (tolerance `1e-3`) | | metadata `alpha` | float in `[-32000, 32000]` — the steering strength; `0` is a valid unsteered baseline | | metadata `layer` | must equal `32` | | metadata `concept` | one of the four concept strings — must equal the **round's active concept** (the published concept for this round), not the concept you trained on | Every failed check is a *typed rejection* (bad shape/dtype, non-unit-norm, non-finite, wrong layer, concept mismatch, alpha out of bounds), not a silent zero. The reference generator in scripts/example\_submission\_generator.py documents each rule inline. > **Tuning `alpha`:** for gemma-3-12b at layer 32 the residual L2 magnitude is \~57k. Empirically, `alpha` below \~16k tends to be inert, the useful band is roughly **12k–16k**, and above \~20k the model degenerates into repetition. Start around `alpha=12000` and sweep from there. Over-steering is now doubly punished: degenerate completions are zeroed by the coherence gate, and a larger `alpha` directly shrinks the minimal-intervention `efficiency` multiplier. ### Evaluation Evaluation runs against a **pinned scorer image** built from Aurelius' open-source [`concept-competition`](https://github.com/Aurelius-Protocol/concept-competition/tree/main) repository. The image hosts an API server that loads the model on a GPU and exposes a `/score` endpoint; the Apex validator provisions a GPU fleet, posts each submission to the scorer, and collects the resulting score and completions. The scorer image is **digest-pinned** in the backend (`aurelius-steering-scorer`, see `DEFAULT_SCORER_IMAGE`), and the eval fleet runs on the same GPU architecture and image that the baseline was scored on, so all scores in a round are comparable. #### Batched, end-of-round evaluation Unlike per-submission competitions, **evals are done in a batch at the end of the round**. The runner receives the full round's submissions in one job, provisions one GPU per mini-batch (`evals_per_gpu` submissions each), and scores them. Every submission produces exactly one result, error or not. #### Baseline and normalization 1. **Baseline:** at round generation, the validator scores an **unsteered** submission (a unit vector with `alpha=0`, so the model output is unaffected) on a freshly drawn round seed. This `baseline_score` is the concept's natural hit rate with no steering. 2. **Raw score:** each submission is posted to the scorer, which returns the concept score (a hit-fraction in `[0, 1]`). The eval pipeline takes the returned number as the submission's `raw_score` (stored as `eval_raw_score`) — *raw* meaning **before normalization**. The coherence gate and the minimal-intervention reward have both already been applied inside the scorer before it returns the number; from the validator's side this is just the score that came back. 3. **Headroom normalization:** the `eval_score` we store in the DB removes the baseline and rescales by the available headroom, so every concept maps the unsteered model to 0.0 and a perfect steer to 1.0: ``` eval_score = clip((raw_score - baseline_score) / (1 - baseline_score), 0, 1) ``` This makes scores comparable across concepts: a hard concept (high baseline) and an easy one (low baseline) are judged purely on how much of the *remaining* headroom the miner captures. #### Coherence gate (on by default) As of the **v0.1.5** scorer, every completion is additionally judged for **coherence**, and incoherent completions contribute **zero** to the concept score — regardless of how strongly they express the concept. This closes the degenerate strategy of pushing `alpha` so hard that the model collapses into on-concept gibberish: the detector would still fire, but the coherence gate zeroes those completions. How it works, per `/score` call: 1. After the steered completions are generated, the scorer runs a **second, unsteered pass** of the same base model as a judge: for each `(prompt, completion)` pair it asks whether the output coherently follows the instruction, answering `True`/`False`. (An unparseable judge answer counts as coherent — benefit of the doubt.) 2. The day-score then only credits completions that are both **on-concept and coherent**: in `hit_rate` mode a completion counts as `hit × coherence_hit`; in `graded` mode the clamped intensity is multiplied by the coherence bit. Consequences worth knowing: * **`hit_count` in the response stays the raw detector count** (coherence-blind); `coherence_hit_count` reports how many completions the judge passed. The score can therefore be *lower* than `hit_count / total`. Each completion's own `coherence_hit` is in the history file. * **The judge is imperfect.** In observed runs it marks roughly 15–20% of ordinary *unsteered* completions incoherent, so treat it as a strict gate: a steered completion must read as a genuinely well-formed response to survive. * **The baseline is judged too.** The unsteered baseline goes through the same gate, so normalization stays apples-to-apples within a round. * The judge pass roughly costs a second (cheaper) generation pass; it is controlled per round by `check_coherence` in the competition's `input_data_generator_args` — stamped onto the round input (`AureliusSteeringInputDataSchema.check_coherence`) and sent on **every** `/score` call, baseline included. It is **on by default**; `false` skips the judge pass entirely and every completion is treated as coherent (pre-v0.1.5 scoring). #### Minimal-intervention reward (enabled in current rounds) As of the **v0.1.4** scorer, a round may reward steering that achieves the concept with a **gentler push** — a smaller total intervention on the residual stream — over brute-forcing the detector with an enormous one. (This replaces the experimental Hoyer-sparsity penalty from v0.1.3, which is gone in v0.1.4.) It is off unless configured, but **current rounds run with `push_scale = 555000`** — assume it is on. This all happens **inside the scorer**, before the eval pipeline sees a number. When enabled, the scorer multiplies its pre-reward concept score by an `efficiency` factor in `(0, 1]`, derived from the submission's total absolute steering `push`, to produce the `score` it returns: ``` push = |alpha| * sum(|direction|) # total absolute steering applied efficiency = exp(-push / push_scale) # in (0, 1]; larger push_scale = gentler discount ``` A bigger `push_scale` tolerates more steering before discounting; as `push_scale → ∞` the factor → 1 (no discount). The day-score gates it (a submission that generates nothing on-concept stays at 0), and `efficiency ∈ (0, 1]` with `raw_score ∈ [0, 1]`, so the product stays in `[0, 1]`. It is enabled per round, by precedence **request > per-concept config > off**: * **From the competition (the apex path):** set `push_scale` in the competition's `input_data_generator_args`. Round generation stamps it onto the round input (`AureliusSteeringInputDataSchema.push_scale`) and the eval runner sends it on **every** `/score` call in the round — so the whole round (and its baseline) is scored under one setting. Omit it (or `null`) to leave the reward off. A recommended starting value is **\~555000**. * **In the scorer config:** a non-null `push_scale` for a concept in the scorer's `config/competition.yaml` (used when the request omits it). `push` is **always computed and reported**, even when the reward is off, so it can be calibrated against the real distribution before being switched on. The `/score` response (and each submission's history file) carries: | Scorer response field | Meaning | | --------------------- | --------------------------------------------------------------------------------------- | | `score` | the concept score the scorer returns, **after** the reward (`= raw_score × efficiency`) | | `raw_score` | the scorer's score **before** the reward — already coherence-gated | | `push` | total absolute steering, \` | | `push_scale` | the scale in effect for this request (`null` = off) | | `efficiency` | the multiplier applied: `exp(-push / push_scale)` (`1.0` when off) | | `check_coherence` | whether the coherence judge ran for this request | | `coherence_hit_count` | completions the judge passed (`null` when the judge was skipped) | > **⚠️ `raw_score` means two different things across the two layers.** The scorer's `raw_score` is its *pre*-reward score (with the coherence gate already applied). The eval pipeline's `raw_score` (what we store as `eval_raw_score`) is the scorer's *post*-reward **`score`** field. The full chain, with both features on: > > ``` > raw_score (scorer) = coherence-gated day-score (incoherent completions count 0) > score (scorer) = raw_score (scorer) × efficiency > raw_score (eval / eval_raw_score) = score (scorer) > eval_score (DB) = clip((raw_score − baseline_score) / (1 − baseline_score), 0, 1) > ``` > > With the reward off, `efficiency = 1.0`, so the scorer's `raw_score` and `score` are equal and the distinction collapses. > **Note on the baseline.** The unsteered baseline submission has `alpha = 0`, so `push = 0` and `efficiency = exp(0) = 1` for any `push_scale` — the baseline is never discounted. So even with the reward on, a (discounted) miner score is normalized against an undiscounted `baseline_score`. This is intended — the baseline is the no-steer reference point — but worth being aware of before enabling it. ### Round structure and defaults | Setting | Default | | ----------------------------- | --------------------------------------------------------------------------------------------------------- | | Round length | **1 day** (`default_round_length_in_days = 1`) | | Submission code visible after | **1 day** (`default_submission_reveal_days = 1`) — submissions stay hidden until then, then are published | | Prompt sample size per round | `150` — configured per round | * **No logs:** this competition produces **no execution logs**. There is nothing to inspect mid-round. * **History file:** at the **end of the round**, each scored submission has a **history file** — the full scorer response, including the **prompt and the resulting inference (completion) for each prompt**. This is the artifact miners use to see exactly how their direction steered the model. * **Concept cycling:** the active concept is **changed manually after X rounds**, by a joint decision between **Macrocosmos and Aurelius**. It does not rotate automatically on a fixed schedule. ### Tips for miners #### Building a steering direction * A direction is a single unit vector in the model's 3840-dim hidden space at layer 32. A common, effective approach is **diff-of-means**: collect layer-32 activations on text that *does* express the concept and text that does not, take the difference of the two means, and L2-normalize it. That difference vector points from "not the concept" toward "the concept". * Other approaches: probing classifiers (use the learned weight vector as the direction), sparse-autoencoder features, or contrastive activation pairs. Anything that yields a meaningful direction at layer 32 is fair game. * **Always work against the canonical config** (gemma-3-12b-it, 4-bit NF4, bf16, layer 32). Directions derived from a differently-quantized or different-layer setup will not transfer cleanly. * **Then tune `alpha`** against real steered completions. The direction sets *what* to push toward; `alpha` sets *how hard*. Sweep `alpha` (≈12k–16k) and watch for the sweet spot before the model degenerates — completions that stop reading as coherent answers score zero, and every extra unit of push also costs `efficiency`. The winning direction is the one that steers convincingly with the *least* intervention, not the hardest shove. * Normalize before submitting: `direction = direction / ||direction||`. #### Regenerating the baseline and a random steer This folder ships a generator and two reference files you can reproduce: * scripts/example\_submission\_generator.py — writes a valid `.safetensors` submission, with each field annotated by the rule it must satisfy. * scripts/baseline-zero-steer-positive\_sentiment.safetensors — an unsteered (`alpha=0`) baseline submission. The filename's `positive_sentiment` suffix is the concept stamped in its metadata; for a different round, regenerate with that round's `--concept`. * scripts/sample-random-steer-positive\_sentiment.safetensors — a random-direction steered submission for the `positive_sentiment` concept (a sanity-check / lower bound, not a real strategy). Regenerate them from the `scripts/` directory. Pass `--concept` to stamp the round's active concept into the file's metadata — it defaults to `positive_sentiment`, so set it explicitly for any other round or your submission will be rejected with a concept mismatch. The commands below reproduce the committed `positive_sentiment` reference files; swap in the active concept (and filename) for your round: ```sh # Unsteered baseline (alpha = 0) python scripts/example_submission_generator.py --concept positive_sentiment --alpha 0 --out baseline-zero-steer-positive_sentiment.safetensors # Random steered direction (deterministic via --seed), alpha = 12000 python scripts/example_submission_generator.py --concept positive_sentiment --alpha 12000 --seed 0 --out sample-random-steer-positive_sentiment.safetensors ``` The `--concept` value must be one of the four supported concepts and must equal the round's active concept. The random/baseline files are starting points and references — a competitive submission replaces the random `direction` with one you've actually derived for the active concept. ### How to submit 1. Build your `direction` for the round's active concept and tune `alpha`. 2. Write the `.safetensors` submission (use the generator as a template). Confirm it has the single `direction` tensor (`(3840,)`, `float32`, unit-norm) and the `alpha` / `layer=32` / `concept` metadata. 3. Submit it through the standard Apex miner flow for the `aurelius_steering` competition before the round closes. 4. All submissions are scored together when the round ends; your submission's code is revealed after the reveal window, and the history file lets you review the per-prompt completions your direction produced. ### References * Anthropic — [Golden Gate Claude](https://www.anthropic.com/news/golden-gate-claude) * Aurelius — [concept-competition (scorer source)](https://github.com/Aurelius-Protocol/concept-competition/tree/main) # Ended: RL Tron Competition Reinforcement Learning with TRON In this head-to-head reinforcement learning competition for Apex, miners train RL agents to play Tron on their own machines and submit their models in TorchScript (`.pt` files) for evaluation. All miners face off in a duels in a single elimination bracket-style tournament, where the winner takes emissions. ### Tron Settings * Be sure the model loads with `torch.jit.load()` before submitting. * Submissions must be in `.pt` files to be accepted. * Max submission size is 100 MB. * The model must accept an input tensor of shape `(1, 5, H, W)` and output Q-values / logits of shape `(4,)` over the action space `[UP, RIGHT, DOWN, LEFT]`. ### Round Structure A round is run as a **single-elimination bracket**. Miners can make submissions while the round is `OPEN`. Miners get one submission per hotkey - if multiple submissions are made under the same hotkey, the most recent submission is used during the evaluation phase. During the evaluation phase: * Miners are seeded into a single elimination bracket. * Every match is a head-to-head duel between two miners. * The winner advances to the next round of the bracket. * The last surviving miner is the round winner and receives all competition emissions, annealing with the burn. ### Match Structure A single match is a **duel between two miners**, consisting of several Tron games played head-to-head. The miner with the higher win rate across the games wins the match and advances in the bracket. * The default is **3 games per match**. Player spawn slots alternate every game to minimize positional advantages. * Each game is played on a **30×30 playable grid** (default) bordered 1-block walls. * Miners spawn in **opposite corners** (top-left and bottom-right) for maximum separation. * Each game runs for at most **500 ticks**. * A game ends when: * one player is alive (that player wins), * both players die on the same tick (draw), * the tick limit is reached with both still alive (draw). * Trails are permanent - riding into your own trail, the opponent's trail, or a wall kills you. * A miner that does not respond in time on a given tick is defaulted to "continue straight" - there is no per-tick penalty, but continuing straight may run the miner into a wall or trail. #### Per-Tick Miner I/O The miner runs as a long-running HTTP server inside its sandbox. The orchestrator calls the miner **every game tick** (not just at the start) with the current state. Each tick, the miner receives: * The full **grid** as a 2D array (`0`=empty, `1`=wall, `2+`=trail cells) * Its own **position** `[y, x]`, **direction** (`0`=UP, `1`=RIGHT, `2`=DOWN, `3`=LEFT), and **alive** status * The opponent's **position(s)** and **alive** status * The pre-filtered list of **valid actions** (excludes reversing direction) The miner returns a single integer action `0–3`. #### Timing * **Per-tick move timeout**: 0.1 seconds. Exceeding this defaults the miner to its current direction. * **Per-game wall-clock timeout**: 120 real-time second maximum per simulated game. ### Evaluation #### Game Score Every game produces a per-game score for each miner based on a **death-cause cascade**. Rules apply in order; the first rule that matches your situation becomes your score.
#Your situationScore
1You killed your opponent (their killed_by is you) and you're still alive1.00
2You're still alive and your opponent self-destructed (hit a wall or their own trail)0.80
3You killed your opponent but also died on the same step (head-on, mutual trail-kill, or you wall-died while your trail killed them)0.40
4Both players alive at max_steps (timeout draw)0.25
5Your opponent killed you and you did not kill them0.10
6You died alone (wall or your own trail), no kill credit0.00
* If a game fails to start due to model load failures, both miners receive 0.0 for that game. * The scoring rewards aggression: a clean kill (`1.00`) is worth more than waiting for your opponent to crash (`0.80`). #### Match Score A match's score for each miner is the **average** of per-game scores across all games in the match: ``` match_score = sum(per_game_scores) / num_games ``` So a match win rate is bounded in `[0.0, 1.0]`. #### Match Outcome The miner with the higher match score **wins the match** and advances in the bracket. A losing miner is eliminated from the bracket. Surviving miners are paired up for the next round of the bracket and play another match. This continues until one miner remains. * Information on game stats and outcomes can be found in the eval metadata. #### Aggregate Stats * **`eval_raw_score`** = the number of rounds this submission has survived. * **`eval_score = rounds_survived / total_rounds`** These numbers do not determine the bracket winner - they are tracking stats. The round winner is the **last surviving miner in the bracket**. ### Additional Details * View results on the website's [competition dashboard.](https://apex.macrocosmos.ai/competitions/8) * Miner code is revealed 2 days after evaluation. * Round length: 2 days. * 1% `raw_score` threshold to beat current top scorer. * Submission fee: $1.40 USD, converted to the current TAO price. * Logs are opened after the current round is completed. * Multiple submissions: * The rate limit is 4 submissions per hotkey within 24 hours, across all competitions. * A guide on training a baseline model can be found in the [`train` folder](https://github.com/macrocosm-os/apex/tree/main/shared/competition/src/competition/tron/train). * Information on the RL Tron Player API can be found in [launch\_tron\_rl.py](https://github.com/macrocosm-os/apex/blob/main/shared/competition/src/competition/tron/launch_tron_rl.py). * See `requirements.txt` for information on allowed packages. * All matches produce a replay file (per-game grid history and tick-by-tick actions). # Energy Arbitrage Competition Algorithmic electric grid optimization **Energy storage arbitrage** is a core problem in modern electricity markets: a battery operator can profit by purchasing power when prices are low and selling it back when prices are high, but must act under uncertainty as real-time prices deviate from day-ahead forecasts due to weather, demand shocks, and transmission congestion. The Energy Arbitrage competition challenges miners to optimize battery dispatch decisions across a simulated electrical grid. Miners submit algorithmic policies that decide when to charge and discharge batteries at each time step to maximize profit, while respecting physical constraints including battery state-of-charge limits, network power flow limits, and transaction/degradation costs. ### Evaluation Overview Each evaluation runs the miner's policy across 100 challenge instances. Instances cycle through 5 scenarios of increasing difficulty: | Scenario | Nodes | Lines | Batteries | Time Steps | Duration | | --------- | ----- | ----- | --------- | ---------- | -------- | | Baseline | 20 | 30 | 10 | 96 | 1 Day | | Congested | 40 | 60 | 20 | 96 | 1 Day | | Multiday | 80 | 120 | 40 | 192 | 2 Days | | Dense | 100 | 200 | 60 | 192 | 2 Days | | Capstone | 150 | 300 | 100 | 192 | 2 Days | * Each time step represents 15 minutes. Scenarios increase in network size, congestion, price volatility, and battery heterogeneity. * Each rounds's evaluation set is seeded to ensure determinism. * This seed changes from round to round. #### Step by Step At each time step, the miner's policy function receives: * The current state: battery state-of-charge levels, real-time nodal electricity prices, exogenous grid injections, feasible action bounds per battery, and accumulated profit. * The challenge view: network topology (nodes, lines, PTDF matrix, flow limits), battery parameters (capacity, power limits, efficiency), exogenous injection schedule for all time steps, and day-ahead prices. * The policy returns a list of actions (MW), one per battery. Negative values charge; positive values discharge. * Actions must stay within the provided bounds. * Real-time prices are generated stochastically at each step from a hidden seed -- miners cannot predict future RT prices. * Day-ahead prices are known in advance for the full horizon. #### Constraints * Battery SOC: Must remain between 10% and 90% of capacity. Starts at 50%. * Charge/discharge efficiency: 95% each direction. * Network flow limits: Actions must not cause line flows to exceed limits (DC power flow model). Violations cause the step to fail. * Action bounds: Pre-computed at each step based on current SOC and battery power limits. * Timeouts: * Per-step timeout = 30 seconds. * Total evaluation timeout = 1200 seconds. #### Profit Calculation At each time step, per battery: ``` profit = revenue - transaction_cost - degradation_cost revenue = action * rt_price * dt transaction_cost = 0.25 * |action| * dt degradation_cost = 1.00 * (|action| * dt / capacity)^2 Where dt = 0.25 hours (15 min). Total profit is the sum across all batteries and all time steps. ``` #### Scoring * Each instance is scored by comparing the miner's total profit against a baseline (the better of two built-in heuristic policies -- greedy and conservative): * `quality = (miner_profit - baseline_profit) / (baseline_profit + 1e-6)` * `quality_int = round(clamp(quality, -10, +10) * 1,000,000)` * The miner's final score is the average quality across all 100 instances. * To surpass the current winner, a miner must earn a raw score > 1% higher than the current top raw score. * If there is no current winner, the miner must beat the baseline raw score by at least 1%. * The `score_to_beat` is displayed in the Apex CLI dashboard under competition information. #### Miner Submissions * Miners submit a single .py file implementing: * `def policy(challenge: PolicyView, state: State) -> list[float]:` * Maximum submission size: 50,000 characters. * Default round length: 1 day. * Submission Fee: $10.00 USD. * 1% `raw_score` threshold to beat current top scorer. * Miners code is revealed 1 day after evaluation. * Logs are opened after the current round is completed. * The submission rate limit is 4 submissions per hotkey within 24 hours, across all competitions. * An example of baseline solver implementations can be found in the [energy\_arbitrage/python](https://github.com/macrocosm-os/apex/tree/main/shared/competition/src/competition/energy_arbitrage/python) folder. * The information about enabled packages is in [requirements.txt](https://github.com/macrocosm-os/apex/blob/main/shared/competition/src/competition/energy_arbitrage/dockerfiles/requirements.txt). Only numpy is available beyond the standard library. # Ended: iota Simulator Competition Algorithmic distributed training optimization The `iota` Simulator models a distributed compute network where activations flow through layers of miners. Miners submit routing and load-balancing algorithms to guide activations through the network as fast as possible. The top-performing algorithms from this competition will be considered for use in subnet 9 `iota`'s orchestration layer. ### Simulator Details A detailed simulator, submission, and log file description can be found in the **info doc** within the [iota\_simulator folder](https://github.com/macrocosm-os/apex/tree/main/shared/competition/src/competition/iota_simulator). The simulator code is currently **proprietary**. * 96 simulated miners (by default, number may change from round to round) are distributed across layers in a simulated network with bandwidth, latency, queuing, and caching. * Each activation travels forward through layers 0→N-1, then backward N-1→0. * An activation completes when it has finished all forward and backward passes, ending back at layer 0. * If an activation takes too long to complete, it will become stale and is dropped, removing it from all applicable caches. * The default timeout for staleness is 60 simulated seconds. * Miners have forward and backward queues. * Processing backward activations take priority over forward activations. * Miners cache activations after forward processing; cache pressure can stall forward queues * A miner, by default, has a cache size of 5, meaning it can hold space for up to 5 in-flight activations at a time. * An activation is added to a simulated miner's cache after it has been forward-processed. * An activation is removed from a simulated miner's cache after it has been backward-processed. * An epoch completes when 500 activations (by default) finish their full round-trip. * Between epochs, a merge phase occurs: queues clear, your `/balance-orchestrator` is called, and miner properties may drift slightly. * All time is simulated — HTTP latency to your server does not count toward your score. * Simulated miners may have different drop out and rejoin probabilities, bandwidth, latency, etc. #### **Simulator Diagram Examples - Queues and Caches** NOTE: * These images don't reflect actual time steps and are just to illustrate the general sequence of events for specific activations and nodes. In the actual simulation, all nodes are active. * During the simulation, different nodes may process activations at different rates. Some may drop out unexpectedly, some may rejoin, and some have slower latency and bandwidths than other nodes.
Example 1: 2 Layers, 2 Miners In this example, 1 activation is routed between 2 miners in 2 layers from start to finish.

Miner A in Layer 0 receives the red activation in the forward queue.

Miner A in Layer 0 is processing the red forward activation.

Miner A in Layer 0 has finished processing the forward activation and passes it to Miner B in Layer 1. Miner A's cache is increased by 1.

Miner B in Layer 1 is processing the forward activation.

Miner B in Layer 1 has finished processing the forward activation, and it moves to its backward queue, as Layer 1 is the final layer. Miner B's cache increments after the forward activation has finished processing.

Miner B in Layer 1 is processing the backward activation.

Miner B in Layer 1 has finished processing the backward activation and sends it back to Miner A at Layer 0, from whom it received the activation. The activation joins Miner A's backward queue.

Miner A in Layer 1 is processing the backward activation.

Miner A in Layer 1 has finished processing the backward activation, and its cache is then decreased by 1. The activation has completed.

Example 2: 3 Layers, 6 Miners In this example, we follow activations within an in-progress network.
In this image: * Miner A in Layer 0 just received the red backward activation from miner C in Layer 1. Miner C's cache was decreased from 2 to 1. * Note: Miner A in layer 0 cannot process any more forward activations until it has finished processing the backward activation in its queue, because its cache is full (at 5). * Miner B in Layer 0 is processing the green forward activation. * Miner E in Layer 2 is currently processing the yellow backward activation. * Miner F in Layer 2 is currently processing the blue forward activation.
In this image: * Miner A in Layer 0 is currently processing the red backward activation. * Miner B in Layer 0 finished processing the green forward activation. Its cache is increased by 1, and it sends the activation to the Miner D's forward queue in Layer 1. * At the same time, Miner E in Layer 2 finished processing the yellow backward activation. Its cache is decreased by 1, and it and sends the activation to Miner D's backward queue in Layer 1. * Miner C in Layer 1 is currently processing the purple forward activation. * Miner F in Layer 2 is continues processing the blue forward activation.
In this image: * Miner A in Layer 0 finished processing the red backward activation. Its cache is decreased by 1 and the activation is completed. * The forward queue is no longer stalled - this node can process 1 more forward activation from the queue before its cache is filled again. * Miner C in Layer 1 is still processing the purple forward activation. * Miner D in Layer 1 is currently processing the yellow backward activation. * The backward activation was prioritized over the forward activation, and thus processed first. * Miner F in Layer 2 has finished processing the blue forward activation, and has added the activation to its backward queue.
### Evaluation Miners implement an HTTP server with two endpoints — `/route` and `/balance-orchestrator` — that control how activations are routed through a multi-layer miner network. Each evaluation task runs a simulation, each with a different random seed and number of layers (3-8) simulating 5 epochs of 500 activations traversing the network forward and backward through all layers. * `/route` is called once per activation routing decision. * 0.5 second timeout per call. * `/balance-orchestrator` is called once per epoch, between epochs. * 0.5 second timeout per call. {% hint style="info" %} An example of a submission implementing **random** routing and balancing can be found in the [iota simulator folder](https://github.com/macrocosm-os/apex/tree/main/shared/competition/src/competition/iota_simulator). {% endhint %} #### **Scoring** Score is calculated by: ``` task_score = clamp(1 - (total_epoch_time / max_epoch_time), 0.0, 1.0) final_score = median(task_scores) # median across 5 tasks ``` * total\_epoch\_time: sum of all epoch durations (simulated seconds), excluding merge phases. * max\_epoch\_time: analytically computed time ceiling with a safety multiplier. * To surpass the current winner, a miner must earn a raw score at least 1% higher than the current top raw score. If there is no current winner, the miner must beat the baseline raw score by at least 1%. * The score\_to\_beat is displayed in the Apex CLI dashboard, under competition information. #### Miner Submissions * Miners submit a single `.py` file. * Maximum submission size: 50,000 characters. * Submission Fee: $1.00 USD. * Default round length: 1 day. * 1% `raw_score` threshold to beat current top scorer. * Standard [Incentive mechanism](/subnets/subnet-1-apex/incentive-mechanism). * Miners code is revealed 1 day after evaluation. * Logs are opened after the current round is completed. * Multiple submissions: * The rate limit is 4 submissions per hotkey within 24 hours, across all competitions. * An example of a submission implementing **random** routing and balancing can be found in the [iota simulator folder](https://github.com/macrocosm-os/apex/tree/main/shared/competition/src/competition/iota_simulator). * The information about enabled packages is in [requirements.txt](https://github.com/macrocosm-os/apex/blob/main/shared/competition/src/competition/iota_simulator/dockerfiles/requirements.txt). * All matches produce a history file, with activation logs and simulation timestamps detailing miner metrics at the given point in the simulation. # Text Clustering Competition The Text Clustering competition challenges miners to build fast, CPU-only clustering algorithms that approximate industry-standard NLP pipelines (sentence embeddings + UMAP + HDBSCAN) on real social media data from Bittensor Subnet 13's data scraping product, Gravity. Clustering messy real-world text by topic is the engine behind every trend-detection and feedback-analysis pipeline — this competition crowdsources the best version of it. Each round, miners receive fresh slices of real X and Reddit posts collected through SN13 and must return cluster assignments. Submissions are scored against pre-computed embedding-based ground truth using Adjusted Rand Index and Normalized Mutual Information. ### Settings * Competition Dashboard * Submissions must be a single `.py` file implementing a FastAPI server. * Maximum submission size: 50,000 characters. * CPU-only sandbox — no GPU, no internet at runtime. ### Round Structure * A round consists of **four clustering tasks (subsets)**: three on fresh batches of real social media posts and one on a sample of **arXiv paper titles** — a deliberately different domain that rewards algorithms which generalize rather than overfit to social text. The round score is the mean across all four. * Each round, the platform samples texts from its sources (X and Reddit via Gravity/SN13, and a curated arXiv-titles pool) and pre-computes ground-truth cluster labels using a sentence-transformer + UMAP + HDBSCAN pipeline. Ground truth is baked in an isolated sandbox — miners never see it, and it never leaves the platform. For the arXiv subset, ground truth comes from the same pipeline on our random sample — **never** from arXiv's public category labels. * Round data is disjoint by construction: a text used in a recent round is excluded from sampling, so memorizing revealed rounds does not help. * The miner's task is unchanged: receive raw texts via HTTP and return integer cluster assignments — without seeing the ground truth, knowing the number of clusters (it varies per subset), or having internet access at runtime. * The miner that produces the clusterings closest to ground truth wins and receives all competition emissions, annealing with the burn. ### Evaluation Miners implement an HTTP server with two endpoints — `/health` and `/cluster` — that receive raw texts and return cluster assignments. * `/health` is polled during sandbox startup until the miner returns 200 OK. * `/cluster` is called once per subset with the full text batch — typically **\~5,000 texts** (batch size may vary between rounds) — with a **90-second** clustering time budget per call. An example of a baseline implementation using TF-IDF + MiniBatchKMeans can be found in the `text_clustering` folder. ### Data Source Rounds mix two independent text sources: **Social media (subsets 1-3).** Real posts collected by Bittensor Subnet 13 miners through the Gravity decentralized data network. * Platforms: X (Twitter) and Reddit. * Topics vary each round: AI, crypto, politics, sports, science, gaming, etc. * Quality gates applied before a text can enter a round: a content filter (slurs/obscenity never appear in served data), English-language filter, and a minimum length of \~80 characters (anchorless one-liners are excluded). **arXiv paper titles (subset 4).** Titles of research papers harvested from arXiv's public metadata feed, spanning 150+ scientific categories (ML, physics, math, biology, ...). This is the same domain used by standard academic clustering benchmarks (e.g. MTEB). Each round samples \~5,000 previously-unused titles; the pool is refreshed continuously from arXiv's \~2,000 new papers/day. * Volume: \~5,000 texts per subset, four subsets per round (may vary). * Disjointness: texts are excluded from re-use across recent rounds — every round is new material. You can explore the kind of social data your algorithm will cluster using the Macrocosmos Dataverse CLI: ``` cargo install dataverse-cli dv auth # key from https://app.macrocosmos.ai/account?tab=api-keys dv -o json search x -k AI -l 100 # 100 X posts about AI dv -o json search reddit -k MachineLearning -l 50 ``` For the arXiv domain, any public arXiv title listing is representative of the distribution (the exact round sample is never predictable). ### API Interface Health check: ``` GET /health Response: {"status": "healthy"} ``` Clustering endpoint: ``` POST /cluster Request: {"texts": ["text1", "text2", ...]} Response: {"cluster_ids": [0, 1, 0, 2, ...]} ``` ### Scoring A round runs 4 subsets; the miner's round score is the mean of the 4 per-subset combined scores (range 0 to 1). Note that the arXiv subset is substantially harder than the social subsets for classical methods — low absolute scores there are expected and affect all miners equally; relative performance is what decides the round. * **Adjusted Rand Index (ARI)**, range −1 to 1: similarity between the two clusterings, adjusted for chance. * **Normalized Mutual Information (NMI)**, range 0 to 1: how much information the miner's assignments and the ground truth share, normalized. The per-subset score clamps ARI at 0 and averages it with NMI: ``` ari_normalized = max(0.0, ari) # negative ARI (random/degenerate) -> 0 combined = (ari_normalized + nmi) / 2 ``` A round runs 3 subsets; the miner's round score is the mean of the 3 per-subset combined scores (range 0 to 1). To surpass the current winner, a miner must earn a raw score at least 1% higher than the current top raw score. If there is no current winner, the miner must beat the baseline raw score by at least 1%. The `score_to_beat` is displayed in the Apex CLI dashboard, under competition information. ### Constraints * **CPU only** — no GPU available in the sandbox. * **No internet** — cannot download models, embeddings, or external data at runtime. Bake everything into your submission or `requirements.txt`. * **Sandbox time limit** — 90 seconds per `/cluster` response (plus 30 seconds for server startup). * **Restricted environment** — submissions run in a locked-down sandbox. Anything that reaches outside it (network access, escaping the workspace, spawning external processes, or executing code dynamically) is rejected by the screener at submission time. Keep your solution to self-contained, in-sandbox computation. ### Miner Submissions * Miners submit a single `.py` file implementing the FastAPI server above. * Maximum submission size: 50,000 characters. * Submission fee: $20.00 USD. * Default round length: 1 day. * Miner code is revealed 1 day after evaluation. * Logs are opened after the current round is completed. * Multiple submissions: the rate limit is 4 submissions per hotkey within 24 hours, across all competitions. * The information about enabled packages is in `requirements.txt`. # Apex Support and FAQs Frequently asked questions and Apex support channels {% columns %} {% column %} ![](/files/7tF4ewLP5RF4CWhIvXv2) **Get started** [Current Competitions](/subnets/subnet-1-apex/subnet-1-current-competitions) [Mining Guide](/subnets/subnet-1-apex/subnet-1-base-miner-setup) [Validation Guide](/subnets/subnet-1-apex/validating) {% endcolumn %} {% column %} ![](/files/LCYRWYUZsNHtG4DvvZZn) **Contact us for support** [Macrocosmos Discord](https://discord.gg/vRTaAXpRcd) [Bittensor Discord](https://discord.com/channels/799672011265015819/1162768567821930597) [Support Email](mailto:support@macrocosmos.ai) {% endcolumn %} {% endcolumns %} ### Mining
Where do I start? Read the docs related to the [Apex subnet](/) and [current competitions](/subnets/subnet-1-apex/subnet-1-current-competitions). Have a look at the [Subnet 1 Mining](/subnets/subnet-1-apex/subnet-1-base-miner-setup) page for setup guidance.
How to test my miner locally? Visit each of the [competition folders](https://github.com/macrocosm-os/apex/tree/main/shared/competition/src/competition) to see example submission formats as well as a baseline submission. Inside each competition-specific folder is also a ReadMe guide to get started.
Where can I see my scores and other metrics? Each competition has its own dashboard demonstrating relevant metrics. You can find them on the Apex website at [apex.macrocosmos.ai/](https://apex.macrocosmos.ai/) in the competition section. If you are registered on the subnet, you may also view your results and download files from the [Apex CLI](/subnets/subnet-1-apex/subnet-1-base-miner-setup/apex-cli).
I scored higher than the current top scorer, why am I not receiving emissions? To surpass the current top scorer and claim emissions, your `raw_score` must be **at least 1% higher** than the current top scorer. You can find your `raw_score` under the Submission Details page on the CLI.
### Submission Fees
How much are submission fees? Submission fees are variable per competition. If a competition is not listed here, then it's free to submit to. * Energy Arbitrage: $10 USD * Text Clustering: $20 USD * Humanoid Parkour: $20 USD
My submission was rejected. What happens to my fee? If your submission was rejected, you can reuse the fee payment information on another submission. Resubmit with: `apex submit --payment-block-hash [YOUR BLOCK HASH] --payment-extrinsic-index [YOUR EXTRINSIC INDEX]`
Something went wrong with my submission fee payment. What can I do? If your payment went through successfully, you will receive a receipt including the payment block hash and extrinsic index. You can resubmit your solution with the unused payment with: \ `apex submit --payment-block-hash [YOUR BLOCK HASH] --payment-extrinsic-index [YOUR EXTRINSIC INDEX]`
I don't have my payment receipt log. What can I do? If you do not have this receipt log, you can retrieve your block hash and extrinsic index from [taostats](https://taostats.io/account/5EtauUg5ZyHYuRN8MP1hBSejvFjXsKoCKcDr3FJrdy8dZepK/transfers). * The extrinsic index is the number after the `-` in the "Extrinsic" column, without any leading zeros. * i.e. Extrinsic: 8077100-0020 -> extrinsic index = 20. * You can find the block number by clicking on the transaction details for an extrinsic. \ To convert block number to block hash: ``` import bittensor as bt print(bt.subtensor('wss://archive.chain.opentensor.ai:443').substrate.get_block_hash(YOUR_BLOCK_HASH)) ```
# Subnet 9 IOTA Bittensor can build the best models ### Introduction In August 2024, Bittensor’s Subnet 9 (SN9) demonstrated that a distributed network of incentivized, permissionless actors could each pre-train large language models (LLMs) ranging from 700 million to 14 billion parameters, while surpassing established baselines. While that work validated blockchain-based decentralized pretraining as viable, it contained core issues: every miner had to fit an entire model locally, and “winner-takes-all” rewards encouraged model hoarding. Here we introduce IOTA (Incentivised Orchestrated Training Architecture), an architecture that addresses these limitations by transforming SN9’s previously isolated competitors into a single cooperating unit that can scale arbitrarily while still rewarding each contributor fairly. IOTA is a data- and pipeline-parallel training algorithm designed to operate on a network of heterogeneous, unreliable devices in adversarial and trustless environments. The result is a permissionless system that is capable of pre-training frontier-scale models without per-node GPU bloat, and tolerates unreliable devices and aligns participants through transparent token economics. Various solutions attempt to solve key technical hurdles regarding distributed training but lack an incentive model, while others provide economic incentives but have yet to achieve the training performance of a coordinated cluster. IOTA bridges this gap by combining novel techniques that jointly tackle all three limitations.

Centralised vs decentralised LLM training

### Foundational Research The technical primer doc [INCENTIVISED ORCHESTRATED TRAINING ARCHITECTURE\ (IOTA)](https://www.macrocosmos.ai/research/iota_primer.pdf) provides a detailed view of our pre-training efforts. Have a look at the [Miners Dashboard](https://iota.macrocosmos.ai/) to get the updates on the training process.
For more details on how to contribute you can have a looks at [**mining instructions**](https://app.gitbook.com/o/eu9Z3qt7ycTIHIJGObFB/s/JDlWdmSC3GnzBPSkAiBM/~/changes/165/subnets/subnet-9-pre-training/subnet-9-iota-mining-setup-guide), and [**validating** **instructions**](https://app.gitbook.com/o/eu9Z3qt7ycTIHIJGObFB/s/JDlWdmSC3GnzBPSkAiBM/~/changes/165/subnets/subnet-9-pre-training/subnet-9-validating). If you have any questions or require support, please message us in the [Bittensor Discord](https://discord.com/channels/799672011265015819/1162768567821930597) channel for subnet 9, or our own [Macrocosmos Discord](https://discord.gg/vRTaAXpRcd) server. ### Training at Home Have a look at [Training At Home (TAH)](/product-and-services/tah) application for permissionless contribution to distributed foundational model training.
Read substack and other articles: * [Why IOTA is different: Comparing Pretraining, Federated Learning, and Swarm](https://macrocosmosai.substack.com/p/why-iota-is-different-comparing-pretraining) - The Cosmonaut Substack, August 2025 * [Swarm Intelligence Is Reshaping How AI Gets Trained](https://www.forbes.com/sites/torconstantino/2025/06/02/swarm-intelligence-is-reshaping-how-ai-gets-trained/) - Forbes, July 2025 * [IOTA: Bittensor's biggest pretraining breakthrough is here](https://macrocosmosai.substack.com/p/iota-bittensors-biggest-pretraining) - The Cosmonaut Substack, June 2025 ### Other related resources * [Website](https://iota.macrocosmos.ai/) * [IOTA X (Twitter)](https://x.com/Iota_SN9) * [Dashboard](https://iota.macrocosmos.ai/) * [GitHub](https://github.com/macrocosm-os/iota) * [Substack](https://macrocosmosai.substack.com/t/pre-training) * [Bittensor Discord](https://discord.com/channels/799672011265015819/1162768567821930597) * [Macrocosmos Discord](https://discord.com/channels/1238450997848707082) * [Cosmonauts - Macrocosmos Telegram](https://t.me/macrocosmosai) * [Macrocosmos X (Twitter)](https://x.com/MacrocosmosAI) # Scoring, Rewards & Kicking How IOTA scores miners, distributes emissions, and removes underperforming or misbehaving nodes. ### Introduction IOTA turns many independent, unreliable machines into a single cooperating training run. To keep that swarm honest and productive, the network has to answer three questions continuously: 1. **How good is each miner's contribution?** (scoring) 2. **How should emissions be shared based on that contribution?** (rewards) 3. **When should a miner be removed from the swarm?** (kicking) This page explains the flow from raw work to on-chain rewards, why some nodes earn more than others, and the conditions under which a miner is removed. It stays at a conceptual level — the exact thresholds and configuration knobs live in the code and can change per training run. ### The miner lifecycle at a glance Every miner moves through the same lifecycle. Scoring and kicking are not one-off events; they repeat on every training epoch for as long as the miner participates. ```mermaid flowchart TD A[Register for a run and layer] --> B[INITIALIZING
warming up: downloading weights,
catching up — exempt from kicks] B --> C[IDLE
training and scored on every epoch] C -->|scores accumulate over a rolling window| D[Score share becomes on-chain weight,
weight becomes emissions] C -->|end of each epoch: kick policy applied| E{Bottom of the pack,
or below the bar?} E -->|no| C E -->|yes| F[KICKED
removed from the layer,
moved to history] C -->|crash or disconnect| G[Reset with back-off
a recovery step, not a kick] G --> C C -->|operator action or tier ban| F ``` {% hint style="info" %} A newly registered miner starts in an **initializing** state while it downloads the current model weights and catches up with the swarm. During this warm-up it is **not eligible to be kicked** — a node is never penalized for the time it spends getting ready. {% endhint %} ### How contributions are scored Miners do not score themselves, and they are not scored on how much raw compute they claim to have spent. Instead, **validators** independently measure the **forward-pass activations** each miner produces during training and assign a score for the work done in each validation window. Rather than a single "quality" number, scoring combines several independent checks that each look at the miner's activations from a different angle, for example: * **Consistency over time** — does the miner's output behave coherently across successive steps, rather than jumping around randomly? * **Agreement with peers** — do the miner's activations line up with what other honest miners on the same layer are producing? * **Expected signal shape** — does the energy and structure of the activations match what genuine training produces? * **Anti-gaming checks** — are there tell-tale patterns of repetition or shortcuts that suggest the miner is faking work rather than actually training? Because several independent signals feed the score, a miner cannot do well by optimizing one trick — it has to actually contribute useful training work that holds up under all of the checks. Scores are collected continuously in short validation windows and are aggregated over a **rolling recent window** (on the order of a day), so a miner's standing reflects its recent behavior rather than one lucky or unlucky moment. ### How scores become emissions Scores are converted to on-chain **weights**, and weights determine each miner's share of emissions. The path from work to reward is: {% stepper %} {% step %} #### Aggregate recent scores Each miner's scores from the recent window are summed into a single total for the run it is participating in. Miners can carry an individual multiplier that scales this total up or down; a contribution whose adjusted total comes out negative is dropped entirely rather than allowed to drag things down. {% endstep %} {% step %} #### Take a share within the run Within a training run, a miner's weight is its **share of the total score** produced by all miners in that run. Earn a larger slice of the run's verified work, earn a larger slice of that run's rewards. {% endstep %} {% step %} #### Scale by the run's allocation Each run controls a portion of the subnet's total emissions and may set aside a fraction of its own rewards to be burned. A run that only ran for part of the reward period contributes proportionally less. A miner's final weight is its in-run share scaled by these run-level settings. {% endstep %} {% step %} #### Publish weights on-chain All miners' weights are combined into a single vector that sums to one, normalized, and written to the Bittensor chain on a regular cadence — independent of how long an epoch takes. Emissions then flow to each miner in proportion to its published weight. {% endstep %} {% endstepper %} ### The reward formula, and why layers are not equal Putting those steps together, the on-chain **weight** for a miner *m* in run *r* — which its emissions are proportional to — is: $$ \text{weight}*m = \underbrace{\frac{S\_m}{\sum*{k \in r} S\_k}}*{\text{share within run } r} \times (1 - b\_r) \times \underbrace{w\_r \cdot a\_r}*{\text{effective run allocation}} $$ Over the recent scoring window: * $$S\_m$$ — miner *m*'s **total score**: the sum of the validation scores it earned, multiplied by any individual multiplier. If that multiplied total is negative, the miner is dropped entirely. * $$\sum\_{k \in r} S\_k$$ — the total score of **every miner in the run**, summed across all of its layers. This is the denominator each miner competes for a share of. * $$b\_r$$ — the run's **burn factor**: the fraction of the run's rewards that is burned rather than paid out. * $$w\_r$$ — the run's **incentive weight** (its slice of the subnet's emissions), and $$a\_r$$ the fraction of the reward window the run was actually active. Their product $$w\_r \cdot a\_r$$ is the run's *effective* allocation. **Why the number of miners per layer is a key factor.** Work in IOTA is not spread evenly across the model. IOTA is pipeline-parallel: the model is split into layers, and each layer is staffed by a configurable number of miners (its `miners_per_layer`). Just as importantly, scoring is done on **forward-pass activations** — the outputs a miner produces on the forward pass — **not** on backward/gradient work. Because different layers run different volumes of forward passes and hold different numbers of miners, the **total amount of validated work — and therefore the total score available — genuinely differs from layer to layer.** The weighted number of miners on each layer is what makes the shared denominator $$\sum\_{k \in r} S\_k$$ a fair basis for comparison: a miner's share has to be read against the work its layer actually produced, rather than assuming every layer generated the same amount of scorable work. ### Why some miners earn more than others Two miners in the same run can earn very different amounts. The differences come down to a few factors, in order of importance: * **Quality and quantity of verified work.** The single biggest driver. A miner that consistently passes the validators' checks accumulates a larger $$S\_m$$, and therefore a larger slice of emissions. * **The layer they are on.** Because layers differ in their miner count and in how much forward-pass work they produce (see the formula above), the same effort can translate into a different score depending on where in the pipeline a miner sits. * **Being in an active, well-allocated run.** Rewards are shared within a run and scaled by that run's effective allocation ($$w\_r \cdot a\_r$$), so the same score contributes differently depending on the run. * **Individual multipliers.** Operators can apply a per-miner multiplier to adjust earnings without touching the scoring system itself. The chart below is an illustration of how emissions concentrate toward the highest-scoring miners in a run — higher score share, disproportionately higher reward — rather than being split evenly. ```mermaid xychart-beta title "Illustrative: share of a run's emissions by miner rank" x-axis "Miner rank (by recent score share)" [1, 2, 3, 4, 5, 6, 7, 8] y-axis "Share of run emissions (%)" 0 --> 30 bar [26, 21, 16, 12, 9, 7, 5, 4] ``` {% hint style="info" %} There is no "winner-takes-all." Every contributor with positive, verified work earns a proportional share — but the share tracks contribution, so stronger miners are rewarded meaningfully more than weaker ones. {% endhint %} ### Safeguards: caps and burning Not every emission is paid to miners. Whatever is **not** earned by miners is **burned** (sent to the subnet owner), and several guardrails protect the system from misconfiguration or runaway payouts: * **A ceiling on miner emissions.** There is a hard cap on the total share of emissions that can go to miners in a period. If contributions would push past that ceiling, the excess is burned rather than paid. * **An over-allocation tripwire.** If the runs in a period ever claim more than 100% of emissions between them (a sign of a configuration error or a race during a run rotation), the system refuses to pay anyone for that period and burns everything, rather than paying out corrupted numbers. * **The remainder is always burned.** The weight vector always sums to one; whatever miners collectively did not earn goes to the burn. This keeps the accounting transparent and auditable. These are deliberately conservative: when in doubt, the network burns rather than mis-pays. ### How and when miners are kicked Being kicked means a miner is removed from its layer, stops earning, and is moved to a history record (kept for audit — kicked miners are never silently deleted). There are three distinct ways this happens. {% hint style="warning" %} **Low scores do not automatically kick a miner.** Kicking is driven by a run's configured **policy**. A run can be set to never kick, and in that case even a poorly-scoring miner stays. Scores only matter for kicking when a policy says they do. {% endhint %} #### 1. Performance kicks (automatic, per epoch) At the end of **every epoch**, once scoring for that epoch is complete, the run applies whichever kick policy it is configured with: * **No-kick** — no one is removed on performance grounds. * **Bottom-by-score** — the lowest-scoring *N* miners on the layer are removed each epoch, regardless of their absolute scores. Even in a strong field, the tail gets trimmed. * **Score-threshold** — every miner at or below a set score bar is removed. Two protections apply to performance kicks: * **Warm-up is exempt.** Miners still initializing are never performance-kicked. * **A layer is never emptied or gutted.** If a policy would remove so many miners that the layer can't keep functioning, the kick is skipped for that epoch. The time-series below illustrates the score-threshold case: a miner's recent score drifts down over successive epochs and, once it crosses below the bar, it is removed at the next epoch boundary. ```mermaid xychart-beta title "Illustrative: a miner's score decays below the kick threshold" x-axis "Epoch" [1, 2, 3, 4, 5, 6, 7, 8] y-axis "Recent score" 0 --> 100 line [82, 76, 63, 51, 40, 31, 24, 18] line [35, 35, 35, 35, 35, 35, 35, 35] ``` The upper line is the miner's score; the flat line is the threshold. The miner drops below it around epoch 5 and is removed shortly after — note that removal happens **at an epoch boundary**, not the instant a score dips. #### 2. Manual kicks (operator action, immediate) An operator can remove a specific miner at any time through the orchestrator, independent of scores or epochs. This is used for clear-cut cases and records a reason for the audit trail. #### 3. Tier bans (immediate, operator-level) Operators belong to tiers, and a ban at the operator level removes **all** of that operator's active miners at once and blocks their pending registrations. This is the fastest lever for dealing with abuse. ### Recovery vs. removal: resets and back-off Not every failure is a kick. When a miner crashes, disconnects, or falls out of sync, the orchestrator can **reset** it so it re-initializes and rejoins the same run and layer — this is recovery, not removal. To keep a flaky node (or a fleet-wide hiccup) from hammering the system, resets use an **escalating, jittered back-off**: the first reset is immediate, and each subsequent reset waits longer — roughly doubling from a few seconds up to a cap of a few minutes — with random jitter so that many miners recovering at once don't all retry in lockstep. ```mermaid flowchart LR R1[Reset #1
~0s] --> R2[Reset #2
~5s] --> R3[Reset #3
~10s] --> R4[Reset #4
~20s] --> RN[...doubling...] --> RC[Capped
~5 min] ``` A miner only leaves the swarm through one of the three kick paths above. Resets are designed to keep good-faith nodes participating through transient trouble. ### In summary * Miners are **scored by validators** on the quality of verified forward-pass work, combining several independent checks and aggregated over a recent rolling window. * **Emissions follow score share** within a run: $$\text{weight}*m = \frac{S\_m}{\sum*{k \in r} S\_k} \times (1 - b\_r) \times w\_r a\_r$$ — scaled by the run's burn and effective allocation, then published on-chain. The number of miners per layer matters, because layers do different amounts of forward-pass work. * **Guardrails burn** anything above the miner-emissions ceiling, and refuse to pay at all if allocations are over-committed. * **Kicking is policy-driven**, evaluated per epoch: bottom-by-score, below-threshold, or disabled — with warm-up and layer-health protections. Operators can also kick manually or ban an entire tier. * **Resets with back-off** recover struggling nodes; only a kick actually removes a miner, and every removal is recorded. # Auditor Payout System The auditor payout system is a set of public endpoints that allow anyone to observe why specific compute was allocated to an IOTA run. The information includes decision logic, pricing, location, provider details, and more. It is recommended that your agent ingest the docs below to build your own custom introspection tool. ### Verify provisioning decisions The provisioning decision trail is exposed read-only and unauthenticated, so anyone can check why each GPU was bought or dropped — the price we paid, its provenance, and the alternatives we rejected. Two GET endpoints, no credentials. **Base URL** ``` https://liquid-compute.api.macrocosmos.ai ``` > ⏱ Rate-limited to **1 request / minute per IP** — for spot-checks, not bulk pulls. *** ### 1. The decision trail Per run, newest first: the machine we chose, the ones we rejected (each with a reason), the pillar scores it was ranked on, and the price. `run_id` is required; `limit` ≤ 32. ```bash # decisions for a run (realized compute only, by default) curl -s "https://liquid-compute.api.macrocosmos.ai/audit/placement/decisions?run_id=INSERT_RUN_ID&limit=5" | jq ``` Read `chosen.price_per_hour_usd` alongside `price_is_dynamic` / `price_source_system` — a live market quote vs a static catalog price — and `rejected[].reject_reason` to see why cheaper options were passed over. *** ### 2. Filter to what you're checking If desired, you can narrow by decision type or widen to include picks that never came up. Same endpoint, extra query params. ```bash # only skips (wanted to buy, every candidate was gated) curl -s "https://liquid-compute.api.macrocosmos.ai/audit/placement/decisions?run_id=INSERT_RUN_ID&decision=skip" | jq ``` ```bash # include decided-but-never-provisioned picks curl -s "https://liquid-compute.api.macrocosmos.ai/audit/placement/decisions?run_id=INSERT_RUN_ID&include_pending=true" | jq ``` A `create` with `provisioned_at` set = realized compute; `null` = decided but never ran (hidden unless `include_pending=true`). Page with `&offset=`. *** ### 3. The glossary — `/audit/placement/schema` Self-documenting: this endpoint returns the field definitions and the meaning of every pillar score and reject reason, so a decision row is verifiable without insider context. ```bash curl -s "https://liquid-compute.api.macrocosmos.ai/audit/placement/schema" | jq ``` The response has three keys: * `entry_schema` — JSON Schema of one decision * `field_notes` * `reject_reasons` #### Pillar scores — how a candidate is ranked | Field | Meaning | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `composite` | The score the strategy sorts on — lower is better. `price_per_runner(on-demand) × capability_fit × stability_churn`. A unitless heuristic, not dollars; computed on the on-demand price even for spot runs. | | `price_per_runner_hour` | Effective (billed) $/runner-hr = `price_per_hour_usd / runners_per_vm`. The money. | | `capability_fit` | `gpu_memory_gb / min_gpu_memory_gb`. 1.0 = exact fit; >1 = paying for memory over the requested floor. | | `stability_churn` | `1 / √(accelerator_count)`. Lower = a larger multi-GPU VM (fewer VMs to manage). | | `quality_reliability` | 0..1 supplier reliability; 1.0 = the (cell, sku) failure bucket is cool (no recent failures/throttles). | #### `reject_reasons` — why an alternative was passed over | Reason | Meaning | | ------------------ | ----------------------------------------------------------------------------------------------------------- | | `scored_worse` | Valid + available, but ranked below the chosen option (composite margin in `reject_detail`). | | `capacity_capped` | Cloud advertised 0 remaining for the cell or (cell, sku) this tick (per-cell instance quota folds in here). | | `bucket_cooldown` | The (cell, sku) failure bucket is still cooling after recent failures. | | `excluded_sku` | Operator excluded this SKU/accelerator (`params.excluded_skus`). | | `slot_unavailable` | No free (group, slot) left this tick. | | `quota_exhausted` | Reserved — currently surfaces as `capacity_capped`. | | `throttled` | Reserved — recent provider throttle (not emitted by the elastic core today). | *** ### Provenance Provenance on every entry pins it to exact config + code — `params_sha256` (hash of the effective strategy params) + `git_sha` (the commit the image was built from). That's what makes a row reproducible. > 💡 Only runs created with `auditing: true` record a trail. A run with no rows either had auditing off or doesn't exist — the endpoint returns **404**, not an empty list. # Subnet 13 Data Universe Bittensor absorbs the most data ## What is SN13 — Data Universe **Subnet 13** is Bittensor’s decentralized data layer, focused on the **collection, and distribution of fresh, desirable data**. Its incentive mechanism rewards miners for gathering content that is desirable, based on the following: * Source of the data * Specific categories of data within that source * The age of the data * Data uniqueness Currently, miners are incentivized to scrap data from platforms like **Reddit** and **X (Twitter).** Keeping the focus on rapidly shifting trends , because we all know in today’s world, stale data just doesn’t cut it. API Documentation is available at [Macrocosmos API](/subnets/subnet-13-data-universe/readme). Data Universe Marketplace and Data Collection sign up. ## Why It Matters In fast-moving markets, **up-to-date data** is critical. Data provided by subnet 13's product [Data Universe](https://datauniverse.macrocosmos.ai/) allows businesses to: * Track brand sentiment and market shifts in real time * Make data-driven decisions based on the latest insights * Refine strategy and stay competitive with fresh intelligence As machine learning becomes more focused on the output, the **quality of training data** will become a key differentiator. SN13 aims to make **data one of Bittensor’s most valuable commodities.**
## Architecture Subnet 13's decentralized design allows data to be distributed across miners and queried by validators, showcasing Bittensor's scalability. Macrocosmos is expanding data sources, developing a queryable API ([Broken mention](broken://pages/Wfdi9DT48sFFhsxsSNXj)), and is currently one of the largest data providers with access to over 55 billion scraped posts and comments. By providing the raw material for pre-training and inference, subnet 13 is poised to supercharge the next stage of AI model development within the Bittensor ecosystem. ## Macrocosmos API Using the `GravityClient` in the **Macrocosmos API**, you can easily launch scraping tasks and get structured datasets built by miners. API Documentation is available at [Macrocosmos API](/subnets/subnet-13-data-universe/readme). You'll also need an API key. Visit [Data Universe](https://app.macrocosmos.ai/account) to get started. ## Want a deeper dive? Have a look at the Social Media Data use cases in [Data Universe Use Cases](/product-and-services/gravity/data-universe-use-cases) page. For more details about the subnet 13 R\&D and Data Science work, take a look at our Substack articles: * [Beyond dashboards: Why social listening SaaS, AI teams and growth marketers are starving for better data](https://macrocosmosai.substack.com/p/beyond-dashboards-why-social-listening) * [From TAO price to flow: emissions upgrade through the lens of sentiment analysis](https://macrocosmosai.substack.com/p/from-tao-price-to-flow-emissions) * [TAOLOR: Building a subnet-native AI agent with distributed RAG](https://macrocosmosai.substack.com/p/taolor-building-a-subnet-native-ai) * [Building the future of authentic data: how subnet 111 leverages Macrocosmos' Gravity](https://macrocosmosai.substack.com/p/building-the-future-of-authentic) * [Result: SN44, Score, partners with SN13’s Data Universe](https://macrocosmosai.substack.com/p/result-sn44-score-partners-with-sn13s) * [SN13's Election experiment: A decisive victory for data scraping?](https://macrocosmosai.substack.com/p/sn13s-election-experiment-a-decisive) * [Dynamic desirability: Launching Gravity, SN13’s democratization of data source selection](https://macrocosmosai.substack.com/p/dynamic-desirability-launching-gravity) Related resources * [Website](https://www.macrocosmos.ai/sn13) * [Data Universe X (Twitter)](https://x.com/Data_SN13) * [GitHub](https://github.com/macrocosm-os/data-universe) * [Substack](https://macrocosmosai.substack.com/t/data-scraping) * [Bittensor Discord](https://discord.com/channels/799672011265015819/1185617142914236518) * [Macrocosmos Discord](https://discord.com/channels/1238450997848707082) * [Cosmonauts - Macrocosmos Telegram](https://t.me/macrocosmosai) * [Macrocosmos X (Twitter)](https://x.com/MacrocosmosAI) # Data Universe MCP ## dv — Dataverse CLI A fast Rust CLI for querying real-time social media data from X/Twitter and Reddit, powered by the [Bittensor SN13](https://docs.macrocosmos.ai) decentralized data network. > \[NOTE] Dataverse CLI is currently in Beta. We'd love your feedback — please open an [issue](https://github.com/macrocosm-os/dataverse-cli/issues) or submit a PR. Dataverse CLI ### Features at a Glance * **Real-Time Search** — Query X/Twitter and Reddit posts by keyword, username, or URL via decentralized Bittensor miners * **Large-Scale Collection** — Gravity tasks collect data continuously for up to 7 days across the miner network * **Dataset Export** — Build downloadable Parquet datasets from collected data * **Multiple Output Formats** — Table, JSON, and CSV output for terminal, scripting, and analysis * **Agent/LLM Friendly** — `dv commands` emits a full JSON schema of all commands for tool integration * **Dry-Run Mode** — Preview exact API requests without executing or consuming credits * **Secure Config** — API keys stored with 0600 permissions, masked in output *** ### Install #### Cargo (Rust) ```sh cargo install dataverse-cli ``` #### From Source ```sh git clone https://github.com/macrocosm-os/dataverse-cli cd dataverse-cli cargo install --path . ``` #### Manual Download the binary for your platform from [Releases](https://github.com/macrocosm-os/dataverse-cli/releases), and place `dv` in your `$PATH`. *** ### Setup Get a free API key at [app.macrocosmos.ai](https://app.macrocosmos.ai/account?tab=api-keys), then: ```sh # Interactive setup (recommended — input is masked) dv auth # Or via environment variable export MC_API=your-api-key # Verify configuration dv status ``` API key resolution order: `--api-key` flag > `MC_API` env > `MACROCOSMOS_API_KEY` env > config file. *** ### Global Flags ```sh # JSON output (for scripting and agents) dv -o json search x -k bitcoin -l 10 dv -o json search x -k bitcoin -l 100 | jq '.[0].tweet.like_count' # CSV export dv -o csv search x -k bitcoin -l 1000 > bitcoin_posts.csv # Dry-run mode (shows the API request without executing it) dv --dry-run search x -k bitcoin -l 10 # Custom timeout dv --timeout 180 search x -k bitcoin -l 500 ``` All data commands support `-o json` and `-o csv`. Diagnostics go to stderr; stdout is always clean data. *** ### Commands #### `dv search` — Real-Time Social Data Search X/Twitter or Reddit posts in real-time via the Bittensor SN13 miner network. ```sh # Search X by keyword dv search x -k bitcoin -l 10 dv search x -k bitcoin,ethereum -l 50 --from 2025-01-01 # Search by username (X only) dv search x -u elonmusk -l 20 # Multiple keywords with AND mode dv search x -k bittensor,subnet --mode all -l 50 # Search Reddit dv search reddit -k r/MachineLearning -l 25 # Search by URL dv search x --url "https://x.com/user/status/123456" ``` | Flag | Default | Description | | ----------------- | ------- | ------------------------------------------------------------------------ | | `source` | — | **Required.** `x`, `twitter`, or `reddit` | | `-k, --keywords` | — | Keywords, comma-separated (up to 5). For Reddit, first item is subreddit | | `-u, --usernames` | — | Usernames, comma-separated (up to 5, X only) | | `--from` | 24h ago | Start date (YYYY-MM-DD or ISO 8601) | | `--to` | now | End date (YYYY-MM-DD or ISO 8601) | | `-l, --limit` | 100 | Max results (1–1000) | | `--mode` | any | Keyword match mode: `any` (OR) or `all` (AND) | | `--url` | — | Search by URL instead of keywords | Search results *** #### `dv gravity create` — Start Data Collection Create a Gravity task that collects social data from the Bittensor miner network for up to 7 days. ```sh dv gravity create -p x -t '#bittensor' -n "TAO tracker" dv gravity create -p x -k bitcoin -n "Bitcoin collection" dv gravity create -p reddit -t 'r/MachineLearning' -k transformer dv gravity create -p x -t '$BTC' --email me@example.com ``` | Flag | Default | Description | | ---------------- | ------- | ------------------------------------------------------------------ | | `-p, --platform` | — | **Required.** `x`, `twitter`, or `reddit` | | `-t, --topic` | — | Topic to track. X: `#hashtag` or `$cashtag`. Reddit: `r/subreddit` | | `-k, --keyword` | — | Additional keyword filter | | `-n, --name` | — | Task name | | `--email` | — | Notification email on completion | *** #### `dv gravity status` — Monitor Tasks List all tasks or check a specific task. **Always use `--crawlers`** to see record counts and data sizes. ```sh # List all tasks with collection stats dv gravity status --crawlers # Check a specific task dv gravity status multicrawler-abc123 --crawlers ``` | Flag | Default | Description | | ------------ | ------- | ------------------------------------ | | `task_id` | — | Omit to list all tasks | | `--crawlers` | false | Include record counts and data sizes | Gravity status *** #### `dv gravity build` — Build Dataset Build a downloadable Parquet dataset from a crawler. > **Warning:** This stops the crawler and deregisters it from the network. Only build when you have enough data. ```sh dv gravity build crawler-0-multicrawler-abc123 dv gravity build crawler-0-multicrawler-abc123 --max-rows 50000 ``` | Flag | Default | Description | | ------------ | ------- | ------------------------ | | `crawler_id` | — | **Required.** Crawler ID | | `--max-rows` | 10000 | Maximum rows in dataset | *** #### `dv gravity dataset` — Dataset Status Check dataset build progress and get download links. ```sh dv gravity dataset dataset-abc123 dv -o json gravity dataset dataset-abc123 ``` *** #### `dv gravity cancel` / `dv gravity cancel-dataset` ```sh dv gravity cancel multicrawler-abc123 dv gravity cancel-dataset dataset-abc123 ``` *** #### `dv auth` — Configure API Key ```sh dv auth ``` Interactive setup that validates your key against the SN13 network and saves to config. *** #### `dv status` — Check Connection ```sh dv status ``` Shows API key source and tests connectivity to the SN13 network. *** ### Agent / LLM Integration Dataverse CLI is designed for use by AI agents and LLMs. ```sh # Full JSON schema of all commands, flags, types, and examples dv commands ``` The hidden `dv commands` outputs a machine-readable catalog for tool integration. See AGENTS.md for the full integration guide including response schemas, workflow tips, and common patterns. *** ### Gravity Workflow ``` 1. Create task → dv gravity create -p x -k bitcoin -n "my task" 2. Monitor → dv gravity status --crawlers 3. Wait → Let miners collect data (hours to days) 4. Build dataset → dv gravity build crawler-0-multicrawler-... --max-rows 50000 5. Check progress → dv gravity dataset dataset-... 6. Download → Parquet files with download URLs ``` > **Tip:** Don't build too early. If a task has very few records, the dataset will be empty. Let it collect for at least a few hours. *** ### Development ```sh cargo build cargo test cargo build --release ``` *** ### Tech Stack | Crate | Purpose | | ---------------------------------------------------- | ------------------------------------ | | [clap](https://github.com/clap-rs/clap) | CLI argument parsing with derive API | | [request](https://github.com/seanmonstar/reqwest) | Async HTTP/2 client with rustls | | [serde](https://serde.rs) | JSON serialization/deserialization | | [tokio](https://tokio.rs) | Async runtime | | [tabled](https://github.com/zhiburt/tabled) | Terminal table formatting | | [colored](https://github.com/mackwic/colored) | Terminal colors | | [dialoguer](https://github.com/console-rs/dialoguer) | Interactive prompts | # Data Universe Incentive Mechanism Subnet 13 incentive overview Subnet 13 is Bittensor’s decentralized data layer, focused on the collection, and distribution of fresh, desirable data. Incentive mechanism rewards miners for providing successfully validated, non-duplicate, fresh data. There are two main parts that make up a miner’s total scaled score: their raw score and their credibility. To maximise their raw scores, miners either scrape data according to their own preferences (with a value equal to our default scale factor, which is currently set 0.075), or scrape data from dynamically specified labels that validators can submit with the [validator API](https://github.com/macrocosm-os/data-universe/tree/main/vali_utils/api). Desirable content, based on the following: * Source of the data, * Specific categories of data within that source, such as hashtags, keywords, etc, * The age of the data, * Data uniqueness. Validator voting power is proportional to the amount they have currently staked on subnet 13, and indicates the amount the custom label will be incentivised for. From this incentivised list, miners can choose labels to scrape and receive significantly higher reward for returning data with required labels. This mechanism allows validators with significant subnet stake to leverage their bandwidth and request large amounts of fresh data for use in analytics, training, and more. To maximise their credibility, miners must provide data that matches a real-time scrape. This is determined by an exponential moving average based on the percentage of successfully validated bytes. The scaled score is heavily reliant on credibility– raw scores are multiplied by credibility of 2.5, so this is key to miner success on the network. Credibility is such an important mechanism on the network because unreliable data is worthless. Keeping miners accountable for failed validation keeps data on SN13 fresh and trustworthy. Miners upload scraped data to S3 via presigned URLs obtained from an auth server. Validators retrieve and validate this data through pagination-supported S3 access, performing comprehensive checks on format, content, and quality. For more details see [S3 Storage & Validation](https://github.com/macrocosm-os/data-universe/blob/main/docs/s3_validation.md). Final Score Formula ``` Final Score = (Raw Data Score + S3 Boost + OnDemand Boost) × Credibility^2.5 ``` This formula combines three revenue sources and applies an exponential credibility multiplier (exponent = 2.5) that severely penalizes unreliable miners. **Raw Data Score Calculation (P2P Verification)** The raw data score is calculated during p2p verification by the DataValueCalculator class, which sums the value of all data entity buckets in a miner’s index. Data source weights are applied only to this p2p raw score component, not to S3 or OnDemand boosts. For each bucket, the score contribution is: ``` bucket_score = data_source_weight × job_weight × time_scalar × effective_scorable_bytes ``` The Data Source Weights sum to 1 and currently include Reddit and X. Label Scale Factors determine the weight placed on certain subreddits and hashtags for Reddit and X respectively, ranging anywhere from 0-1. Age Scale Factors depend on content age, with data over 30 days old valued at 0. Data Source Weight (P2P Only): Each platform has a fixed weight applied during p2p verification scoring: | Source | Weight | | --------- | ---------- | | Reddit | 0.65 (65%) | | X/Twitter | 0.35 (35%) | Note: These weights are applied only to the raw p2p data score. S3 and OnDemand boosts are not weighted by data source. S3 upload validation details can be found in [S3 Validation](https://github.com/macrocosm-os/data-universe/blob/main/docs/s3_validation.md). For more information, see our [reward model in GitHub](https://github.com/macrocosm-os/data-universe/tree/main/rewards), and our [miner reward evaluation details](https://github.com/macrocosm-os/data-universe/blob/main/rewards/miner_scorer.py#L131). # Data Universe Mining ### Introduction Miners scrape data from various Data Sources and get rewarded based on how much valuable data they have, see the [Incentive Mechanism](https://github.com/macrocosm-os/data-universe/blob/main/README.md#incentive-mechanism) for the full details. The incentive mechanism does not require a Miner to scrape from all Data Sources, allowing Miners to specialize and choose exactly what kinds of data they want to scrape. However, Miners are scored, in part, based on the total amount of data they have. So Miners should make sure they are scraping sufficient amounts of data. The Miner stores all scraped data in their local database, and uploads it to a shared S3 bucket. ### System Requirements Miners do not require a GPU and should be able to run on a low-tier machine, as long as it has sufficient network bandwidth and disk space. Must have python >= 3.10. ### Getting Started #### Prerequisites 1. As of Dec 17th 2023, we support X (Twitter) and Reddit scraping via Apify. You can [setup your Apify API token here](https://github.com/macrocosm-os/data-universe/blob/main/docs/apify.md), or use official APIs from X (Twitter) and Reddit (recommended). The data delivered by miner have to be compliant with [Miner Data Compliance Policy (v1.0, March 2025)](https://github.com/macrocosm-os/data-universe/blob/b24723d18770f413a4d501de2122e4314370a0c5/docs/miner_policy.md?plain=1#L4). 2. Clone the repo ``` git clone https://github.com/RusticLuftig/data-universe.git ``` 3. Setup your python [virtual environment](https://docs.python.org/3/library/venv.html) or [Conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands). 4. Install the requirements. From your virtual environment, run ``` cd data-universe python -m pip install -e . ``` 5. (Optional) Run your miner in [offline mode](https://github.com/macrocosm-os/data-universe/blob/main/docs/miner.md#offline) to scrape an initial set of data. 6. Make sure you've [created a Wallet](https://docs.learnbittensor.org/keys/working-with-keys) and [registered a hotkey](https://docs.learnbittensor.org/local-build/mine-validate#1-register-the-neuron-hotkeys). #### Running the Miner For this guide, we'll use [pm2](https://pm2.keymetrics.io/) to manage the Miner process, because it'll restart the Miner if it crashes. If you don't already have it, install pm2. **Online** From the data-universe folder, run: ``` pm2 start python -- ./neurons/miner.py --wallet.name your-wallet --wallet.hotkey your-hotkey ``` **Offline** From the data-universe folder, run: ``` pm2 start python -- ./neurons/miner.py --offline ``` Please note that your miner will not respond to validator requests in this mode and therefore if you have already registered to the subnet you should run in online mode. #### Configuring the Miner **Flags** The Miner offers some flags to customize properties, such as the database name and the maximum amount of data to store. You can view the full set of flags by running ``` python ./neurons/miner.py -h ``` **Configuring** The frequency and types of data your Miner will scrape is configured in the [scraping\_config.json](https://github.com/RusticLuftig/data-universe/blob/main/scraping/config/scraping_config.json) file. This file defines which scrapers your Miner will use. To customize your Miner, you either edit `scraping_config.json` or create your own file and pass its filepath via the `--neuron.scraping_config_file` flag. By default `scraping_config.json` is setup use both the apify actor and the personal reddit account for scraping reddit. If you do not want to use Apify you should remove the sections where the `scraper_id` is set to either `Reddit.lite` or `X.microworlds` or `X.apidojo`. If you do not want to use a personal Reddit account you should remove the sections where the `scraper_id` is set to either `Reddit.custom`. If either of them is in the configuration but not setup properly in your `.env` file then your miner will log errors but still scrape using any configured scrapers that are properly setup. For each scraper, you can define: 1. `cadence_seconds`: to control how frequently the scraper will run. 2. `labels_to_scrape`: to define how much of what type of data to scrape from this source. Each entry in this list consists of the following properties: 1. `label_choices`: is a list of DataLabels to scrape. Each time the scraper runs, **one** of these labels is chosen at random to scrape. 2. `max_age_hint_minutes`: provides a hint to the scraper of the maximum age of data you'd like to collect for the chosen label. Not all scrapers provide date/time filters so this is a hint, not a rule. 3. `max_data_entities`: defines the maximum number of items to scrape for this set of labels, each time the scraper runs. This gives you full control over the maximum cost of scraping data from paid sources (e.g. Apify) Let's walk through an example to explain how all these properties fit together. ``` { "scraper_configs": [ { "scraper_id": "X.apidojo", "cadence_seconds": 300, "labels_to_scrape": [ { "label_choices": [ "#bittensor", ], "max_age_hint_minutes": 1440, "max_data_entities": 100 }, { "label_choices": [ "#decentralizedfinance", "#btc" ], "max_data_entities": 50 } ] } ] } ``` In this example, we configure the Miner to scrape using a single scraper, the "X.microworlds" scraper. The scraper will run every 5 minutes (300 seconds). When it runs, it'll run 2 scrapes: 1. The first will be a scrape for at most 100 items with #bittensor. The data scrape will choose a random [TimeBucket](https://github.com/macrocosm-os/data-universe/blob/main/README.md#terminology) in (now - max\_age\_in\_minutes, now). The probability distribution used to select a TimeBucket matches the Validator's incentive for [Data Freshness](https://github.com/macrocosm-os/data-universe/blob/main/README.md#1-data-freshness): that is, it's weighted towards newer data. 2. The second will be a scrape for either #decentralizedfinance or #tao, chosen at random (uniformly). The scrape will scrape at most 50 items, and will use a random TimeBucket between now and the maximum data freshness threshold. You can start your Miner with a different scraping config by passing the filepath to `--neuron.scraping_config_file.` On Demand request handle As described in [on demand request handle](https://github.com/macrocosm-os/data-universe/blob/main/docs/on_demand.md) #### Choosing which data to scrape As described in the [incentive mechanism](/subnets/subnet-13-data-universe/subnet-13-incentive-mechanism), miners are, in part, scored based on their data's desirability and uniqueness. We encourage miners to tune their Miners to maximize their scores by scraping unique, desirable data. For desirability, the [DataDesirabilityLookup](https://github.com/RusticLuftig/data-universe/blob/main/rewards/data_desirability_lookup.py) defines the exact rules Validators use to compute data desirability.
# Data Universe Validating ### Introduction The Validator is responsible for validating the data delivered by Miners and scoring Miners according to the [Incentive Mechanism](https://docs.macrocosmos.ai/subnets/subnet-13-data-universe/subnet-13-incentive-mechanism). Validators run multiple concurrent threads: | **Component** | **Purpose** | | ------------------- | ------------------------------------------ | | Main Loop | Evaluation cycles, weight setting | | Miner Evaluator | Score calculation and validation | | On-Demand Processor | Data Collection job validation | | API Server | External query interface (FastAPI/uvicorn) | | W\&B Logger | Metrics logging (rotates every 3 hours) | | Metagraph Syncer | Network state updates | ### System Requirements Validators require at least 32 GB of RAM but do not require a GPU. We recommend a decent CPU (4+ cores) and sufficient network bandwidth to handle protocol traffic. Must have python >= 3.10. ### Getting Started #### Prerequisites 1. Data Universe supports Twitter and Reddit scraping via Apify so miners have to [setup their Apify API token](https://github.com/macrocosm-os/data-universe/blob/main/docs/apify.md). Validators will default to using the recommended reddit account for reliability but this can be changed editing the PREFERRED\_SCRAPERS map in validator.py locally. Data Universe also supports YouTube Scraping via a [official youtube api](https://github.com/macrocosm-os/data-universe/blob/main/docs/youtube.md). 2. Clone the repo ``` git clone https://github.com/RusticLuftig/data-universe.git ``` 3. Setup python [virtual environment](https://docs.python.org/3/library/venv.html) or [Conda environment](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands). 4. Install the requirements. From your virtual environment, run ``` cd data-universe python -m pip install -e . ``` 5. Make sure you've [created a Wallet](https://docs.learnbittensor.org/keys/working-with-keys) and [registered a hotkey](https://docs.learnbittensor.org/local-build/mine-validate#1-register-the-neuron-hotkeys). ```` This will prompt you to navigate to https://wandb.ai/authorize and copy your api key back into the terminal. ## Running the Validator ### With auto-updates We highly recommend running the validator with auto-updates. This will help ensure your validator is always running the latest release, helping to maintain a high vtrust. Prerequisites: 1. To run with auto-update, you will need to have [pm2](https://pm2.keymetrics.io/) installed. 2. Make sure your virtual environment is activated. This is important because the auto-updater will automatically update the package dependencies with pip. 3. Make sure you're using the main branch: `git checkout main`. From the data-universe folder: ```shell pm2 start --name net13-vali-updater --interpreter python scripts/start_validator.py -- --pm2_name net13-vali --wallet.name cold_wallet --wallet.hotkey hotkey_wallet [other vali flags] ```` This will start a process called `net13-vali-updater`. This process periodically checks for a new git commit on the current branch. When one is found, it performs a `pip install` for the latest packages, and restarts the validator process (who's name is given by the `--pm2_name` flag) #### Without auto-updates If you'd prefer to manage your own validator updates... From the data-universe folder: ``` pm2 start python -- ./neurons/validator.py --wallet.name your-wallet --wallet.hotkey your-hotkey ``` ## Configuring the Validator ### Flags The Validator offers some flags to customize properties. You can view the full set of flags by running ``` python ./neurons/validator.py -h ``` ### `.env` Your validator `.env` should look like the following after setup for all data sources: ``` APIFY_API_TOKEN="your_apify_token" ``` Please see docs on [Apify](https://github.com/macrocosm-os/data-universe/blob/main/docs/apify.md), [Reddit](https://github.com/macrocosm-os/data-universe/blob/main/docs/reddit.md), and [Youtube](https://github.com/macrocosm-os/data-universe/blob/main/docs/youtube.md) for more information on the environment variables above. # Data Universe API The Data Universe API - the bridge to decentralised AI services ## Overview Data Universe API makes it easy to bring fresh, structured social data into your product, pipeline, or analysis. Through [Data Universe](https://app.macrocosmos.ai/gravity/marketplace) platform you can access high-signal content from X (Twitter) and Reddit, and turn it into action right away powering market intelligence, brand monitoring, trend tracking, lead research, and AI training with clean outputs that fit directly into your workflow. Data Universe is built for teams that need both speed and scale: * use On-Demand API requests for real-time queries, * or run dataset jobs to collect large batches over time, delivered in clean, consistent formats with rich metadata. Compared to traditional data providers, Data Universe is designed to be high-volume, low-friction, and cost-efficient, so you can go from a quick test to production workloads with a single API key. Macrocosmos API provides extensive documentation and built-in support for authentication to ensure a smooth experience from prototyping to production.
## Why Build with Macrocosmos? Whether you are a developer, data scientist, researcher, growth/marketing team, or agency, Macrocosmos API helps you turn live social platforms into reliable inputs for products, analytics, and automation. Choose the workflow that matches your goal: * On-demand queries for immediate lookups and quick experiments * Dataset builds for high-volume collection and repeatable pipelines * Streaming for continuous monitoring and real-time signals Macrocosmos makes it accessible to: * **Monitor**\ Track keywords, brands, topics, creators, and communities in near real time, so you can spot spikes, sentiment shifts, competitor moves, and emerging narratives early. * **Research** Collect high-signal posts, threads, and conversations from social media to validate ideas, map audiences, measure demand, and uncover the “why” behind trends. * **Enrich** Turn raw social activity into structured fields you can attach to leads, accounts, companies, or segments powering better targeting, scoring, personalization, and reporting in your tools. * **Train LLMs** Build fresh, domain-specific datasets for training, fine-tuning, and evaluation, so AI agents and LLMs perform better on real-world language, up-to-date topics, and niche communities. ## Key Capabilities
Data CollectionRequest filtered data to collect relevant insights, structured information, real-time updates, hidden patterns, and competitive signals across multiple sources.
Build DatasetBuild comprehensive datasets to enable precise retrieval, targeted analysis and intelligent automation, driving smarter decisions and scalable AI-driven capabilities.
OnDemand APICreate a data flow for real-time updates and identify predictive signals to drive timely decisions, automation, and strategic forecasting across operations.
## Find out more \>> [Data Universe](/product-and-services/gravity) platform \>> [Subnet 13 Data Universe](/subnets/subnet-13-data-universe) \>> Data Universe Marketplace and Data Collection Tool sign up # Getting Started Gravity is a decentralized data collection platform powered by SN13 (Data Universe) on the Bittensor network. ## Get Started To get started using Macrocosmos API you should: 1. Generate your API key using the instruction from the [API Keys](https://docs.macrocosmos.ai/developers/readme/api-keys) page 2. Ensure that you are using Python 3.9+ or Typescript **📎 Supported Platforms** * `reddit` * `twitter` (X) More platforms will be supported as subnet capabilities expand. 3. Install the Macrocosmos API using pip or npm: {% tabs %} {% tab title="Python" %} ```python pip install macrocosmos ``` {% endtab %} {% tab title="Typescript" %} ```javascript npm install macrocosmos ``` {% endtab %} {% endtabs %} 4. Macrocosmos API should be * version 3.0.0 for Python * version 2.1.1 for Typescript For upgrade use the command {% tabs %} {% tab title="Python" %} ```python pip install -U macrocosmos ``` {% endtab %} {% tab title="Typescript" %} ```javascript npm install macrocosmos==2.1.1 ``` {% endtab %} {% endtabs %} 5. Choose `GravityClient` for sync tasks. Use `AsyncGravityClient` if async fits better.\ Check [examples/gravity\_workflow\_example.py](https://github.com/macrocosm-os/macrocosmos-py/blob/main/examples/gravity_workflow_example.py) for a complete working example of a data collection CLI you can use for your next big project or to plug right into your data product. ### Demo Video {% embed url="" %} ## Data Universe API Endpoints ### Create a task for Data Collection The task after the launch gets registered on the network within 20 min. The data is starting to be collected and delivered by miners from the moment of the registration on the Blockchain. The task stays live for 7 days to allow the most data to be collected. After that, the dataset gets built automatically. If you provided an email you’ll get a notification with a download link. To check the status of the task and the amount of data collected at any time use the endpoint [**Get status of the task**](#get-status-of-task)**.** To start building the dataset prior the 7 days completion, use the endpoint [**Build dataset**](#build-dataset). {% tabs %} {% tab title="Typescript" %} {% code overflow="wrap" %} ```javascript import { GravityClient } from 'macrocosmos'; // Initialize the client const client = new GravityClient({ apiKey: 'your-api-key' }); // Create a new gravity task const task = await client.createGravityTask({ gravityTasks: [ { platform: 'x', topic: '#ai' }, { platform: 'reddit', topic: 'r/MachineLearning' } ], name: 'My Data Collection Task', notificationRequests: [ { type: 'email', address: 'user@example.com', redirectUrl: 'https://example.com/datasets' } ] }); ``` {% endcode %} {% endtab %} {% tab title="Python" %} ```python import macrocosmos as mc client = mc.GravityClient(api_key="your-api-key") gravity_tasks = [ {"platform": "x", "topic": "#ai"}, {"platform": "reddit", "topic": "r/MachineLearning"}, ] notification = { "type": "email", "address": "user@example.com", "redirect_url": "https://app.macrocosmos.ai/", } response = client.gravity.CreateGravityTask( gravity_tasks=gravity_tasks, name="My First Gravity Task", notification_requests=[notification] ) # Print the gravity task ID print(response) ``` {% endtab %} {% tab title="Constellation API: curl" %} ```bash curl -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '{ "gravity_tasks": [ { "topic": "#ai", "platform": "x" }, { "topic": "r/MachineLearning", "platform": "reddit" } ], "name": "My First Gravity Task", "notification_requests": [ { "type": "email", "address": "user@example.com", "redirect_url": "https://app.macrocosmos.ai/" } ] }' \ https://constellation.api.cloud.macrocosmos.ai\ /gravity.v1.GravityService/CreateGravityTask ``` {% endtab %} {% tab title="Constellation API: grpcurl" %} ```bash grpcurl -H "Authorization: Bearer your-api-key" \ -d '{ "gravity_tasks": [ { "topic": "#ai", "platform": "x" }, { "topic": "r/MachineLearning", "platform": "reddit" } ], "name": "My First Gravity Task", "notification_requests": [ { "type": "email", "address": "user@example.com", "redirect_url": "https://app.macrocosmos.ai/" } ] }' \ constellation.api.cloud.macrocosmos.ai:443 \ gravity.v1.GravityService/CreateGravityTask ``` {% endtab %} {% endtabs %} **Body** | Name | Type | Description | | ---------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------- | | `gravityTasks` | List of `GravityTask` objects | List of task objects. Each must include a `topic` and a `platform` (`x`, `reddit`, etc.) | | `name` | string | Optional name for the Gravity task. Helpful for organizing jobs. | | `notificationRequests` | List of `NotificationRequest` objects | List of notification configs. Supports `type`, `address`, and `redirect_url`. | **Response** {% tabs %} {% tab title="200" %} ```json { "gravityTaskId": "multicrawler-9f518ae4-xxxx-xxxx-xxxx-8b73d7cd4c49" } ``` {% endtab %} {% tab title="400" %} ```json { "error": "Invalid request" } ``` {% endtab %} {% endtabs %} ### Get status of task To check the status of the task and the amount of data collected at any time use the endpoint Get status of the task. If you wish to get further information about the crawlers, you can use the `include_crawlers` flag or make separate `GetCrawler()` calls since returning in bulk can be slow. {% tabs %} {% tab title="Typescript" %} ```javascript import { GravityClient } from 'macrocosmos'; // Initialize the client const client = new GravityClient({ apiKey: 'your-api-key' }); / List all gravity tasks const tasks = await client.getGravityTasks({ includeCrawlers: true }); // Get a specific crawler const crawler = await client.getCrawler({ crawlerId: 'crawler-id' }); ``` {% endtab %} {% tab title="Python" %} {% code overflow="wrap" %} ```python import macrocosmos as mc client = mc.GravityClient(api_key="your-api-key") response = client.gravity.GetGravityTasks(gravity_task_id="your-gravity-task-id", include_crawlers=False) # Print the details about the gravity task and crawler IDs print(response) ``` {% endcode %} {% endtab %} {% tab title="Constellation API: curl" %} ```bash curl -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '{ "gravity_task_id": "your-gravity-task-id", "include_crawlers": false }' \ https://constellation.api.cloud.macrocosmos.ai\ /gravity.v1.GravityService/GetGravityTasks ``` {% endtab %} {% tab title="Constellation API: grpcurl" %} ```bash grpcurl -H "Authorization: Bearer your-api-key" \ -d '{ "gravity_task_id": "your-gravity-task-id", "include_crawlers": false }' \ constellation.api.cloud.macrocosmos.ai:443 \ gravity.v1.GravityService/GetGravityTasks ``` {% endtab %} {% endtabs %} **Body** | Name | Type | Description | | ------------------ | ------ | ------------------------------------------------------------------------------- | | `gravity_task_id` | string | The unique identifier of the Gravity task you want to inspect. | | `include_crawlers` | bool | Whether to include details of the associated crawler jobs. Defaults to `False`. | **Response** {% tabs %} {% tab title="200" %} ```json { "gravityTaskStates": [ { "gravityTaskId": "multicrawler-9f518ae4-xxxx-xxxx-xxxx-8b73d7cd4c49", "name": "My First Gravity Task", "status": "Running", "startTime": "2025-05-30T15:56:20.201500586Z", "crawlerIds": [ "crawler-0-multicrawler-9f518ae4-xxxx-xxxx-xxxx-8b73d7cd4c49", "crawler-1-multicrawler-9f518ae4-xxxx-xxxx-xxxx-8b73d7cd4c49" ] } ] } ``` {% endtab %} {% tab title="400" %} ```json { "error": "Invalid request" } ``` {% endtab %} {% endtabs %} ### Build dataset No need to wait 7 days until the task is complete. If you already collected enough data, you can request your dataset early. Add a notification to get alerted when the dataset is built. Once built, the task gets completed and de-registered. {% tabs %} {% tab title="TypeScript" %} ```javascript import { GravityClient } from 'macrocosmos'; // Initialize the client const client = new GravityClient({ apiKey: 'your-api-key' }); // Build a dataset from a crawler const dataset = await client.buildDataset({ crawlerId: 'your-crawler-id', notificationRequests: [ { type: 'email', address: 'user@example.com', redirect_url: 'https://app.macrocosmos.ai/' } ], maxRows: 100, }); ``` {% endtab %} {% tab title="Python" %} ```python import macrocosmos as mc client = mc.GravityClient(api_key="your-api-key") notification = { "type": "email", "address": "user@example.com", "redirect_url": "https://app.macrocosmos.ai/" } response = client.gravity.BuildDataset( crawler_id="your-crawler-id", notification_requests=[notification], max_rows=100 ) # Print the dataset ID print(response) ``` {% endtab %} {% tab title="Constellation API: curl" %} ```bash curl -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '{ "crawler_id": "your-crawler-id", "notification_requests": [ { "type": "email", "address": "user@example.com", "redirect_url": "https://app.macrocosmos.ai/" } ], "max_rows": 100 }' \ https://constellation.api.cloud.macrocosmos.ai\ /gravity.v1.GravityService/BuildDataset ``` {% endtab %} {% tab title="Constellation API: grpcurl" %} ```bash grpcurl -H "Authorization: Bearer your-api-key" \ -d '{ "crawler_id": "your-crawler-id", "notification_requests": [ { "type": "email", "address": "user@example.com", "redirect_url": "https://app.macrocosmos.ai/" } ], "max_rows": 100 }' \ constellation.api.cloud.macrocosmos.ai:443 \ gravity.v1.GravityService/BuildDataset ``` {% endtab %} {% endtabs %} **Body** | Name | Type | Description | | ---------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `crawlerId` | string | The ID of the completed crawler job you want to convert into a dataset. | | `notificationRequests` | List of `NotificationRequest` objects | A list of notification objects (e.g., email or webhook). Includes `type`, `address`, and `redirect_url`. | | `maxRows` | int | The maximum number of rows to include in the dataset | **Response** {% tabs %} {% tab title="200" %} ```json { "datasetId": "dataset-71e97cfa-xxxx-xxxx-xxxx-33cd91be9028", "dataset": { "crawlerWorkflowId": "crawler-0-multicrawler-b56179b1-xxxx-xxxx-xxxx-0ffd616ad830", "status": "Running", "statusMessage": "Initializing", "totalSteps": "10" } } ``` {% endtab %} {% tab title="400" %} ```json { "error": "Invalid request" } ``` {% endtab %} {% endtabs %} ### Build All Datasets If your Gravity task consists of multiple crawlers (i.e. multiple sets of request parameters), the `BuildAllDatasets` call allows you to build the corresponding datasets simultaneously by supplying the `build_crawlers_config` . {% tabs %} {% tab title="Python" %} ```python import macrocosmos as mc client = mc.GravityClient(api_key="your-api-key") task_id = "your-task-id" response = client.gravity.BuildAllDatasets( gravity_task_id=task_id, build_crawlers_config=[ { "crawler_id": f"crawler-0-{task_id}", "max_rows": 200, }, { "crawler_id": f"crawler-1-{task_id}", "max_rows": 200, }, ] ) print(response) ``` {% endtab %} {% tab title="Constellation API: curl" %} ```bash curl -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '{ "gravity_task_id": "your-task-id", "build_crawlers_config": [ { "crawler_id": "crawler-0-your-task-id", "max_rows": 200 }, { "crawler_id": "crawler-1-your-task-id", "max_rows": 200 } ] }' \ https://constellation.api.cloud.macrocosmos.ai\ /gravity.v1.GravityService/BuildAllDatasets ``` {% endtab %} {% tab title="Constellation API: grpcurl" %} ```bash grpcurl -H "Authorization: Bearer your-api-key" \ -d '{"gravity_task_id": "your-task-id", "build_crawlers_config": [ {"crawler_id": "crawler-0-your-task-id", "max_rows": 200 }, {"crawler_id": "crawler-1-your-task-id", "max_rows": 200 } ] }' \ constellation.api.cloud.macrocosmos.ai:443 \ gravity.v1.GravityService/BuildAllDatasets ``` {% endtab %} {% endtabs %} **Body** | Name | Type | Description | | ----------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `gravity_task_id` | string | The task ID of the gravity task you want to collect data from. | | `build_crawlers_config` | dict | A configuration dictionary which allows you to specify the maximum number of rows that you will collect for each individual crawler within the task. | **Response** {% tabs %} {% tab title="200" %} ```json { "gravityTaskId": "your-task-id", "datasets": [ { "crawlerWorkflowId": "crawler-1-your-task-id", "createDate": "2026-01-15T17:56:40.524749Z", "status": "Running", "statusMessage": "Initializing", "steps": [ { "stepName": "Initializing" } ], "totalSteps": "8" } ] } ``` {% endtab %} {% tab title="400" %} ```json { "error": "Invalid request" } ``` {% endtab %} {% endtabs %} ### Get status of a build Watch your dataset build with `GetDataset()`. Once built, the task gets completed and de-registered. {% tabs %} {% tab title="TypeScript" %} ```javascript import { GravityClient } from 'macrocosmos'; // Initialize the client const client = new GravityClient({ apiKey: 'your-api-key' }); // Get a dataset const datasetStatus = await client.getDataset({ datasetId: 'your-dataset-id' }); ``` {% endtab %} {% tab title="Python" %} ```python import macrocosmos as mc client = mc.GravityClient(api_key="your-api-key") response = client.gravity.GetDataset(dataset_id='your-dataset-id') # Print the details about the gravity task and crawler IDs print(response) ``` {% endtab %} {% tab title="Constellation API: curl" %} ```bash curl -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '{ "dataset_id": "your-dataset-id" }' \ https://constellation.api.cloud.macrocosmos.ai\ /gravity.v1.GravityService/GetDataset ``` {% endtab %} {% tab title="Constellation API: grpcurl" %} ```bash grpcurl -H "Authorization: Bearer your-api-key" \ -d '{ "dataset_id": "your-dataset-id" }' \ constellation.api.cloud.macrocosmos.ai:443 \ gravity.v1.GravityService/GetDataset ``` {% endtab %} {% endtabs %} #### Body | Name | Type | Description | | ----------- | ------ | --------------------- | | `datasetId` | string | The ID of the dataset | #### Response {% tabs %} {% tab title="200" %} ```json { "dataset": { "crawlerWorkflowId": "crawler-0-multicrawler-b56179b1-xxxx-xxxx-xxxx-0ffd616ad830", "createDate": "2025-06-04T10:31:38.747918Z", "expireDate": "2025-07-04T10:31:38.747933Z", "files": [ { "fileName": "x_ai_0.parquet", "fileSizeBytes": "261100", "lastModified": "2025-06-04T10:31:28.770Z", "numRows": "478", "s3Key": "example-s3-key", "url": "example-url" } ], "status": "Completed", "statusMessage": "Dataset ready for download", "steps": [ { "progress": 1, "step": "1", "stepName": "Registering dataset" }, { "progress": 1, "step": "2", "stepName": "Collecting crawler information" }, { "progress": 1, "step": "3", "stepName": "Collecting available data sources" }, { "progress": 1, "step": "4", "stepName": "Validating data sources" }, { "progress": 1, "step": "5", "stepName": "Collating data" }, { "progress": 1, "step": "6", "stepName": "Creating dataset path" }, { "progress": 1, "step": "7", "stepName": "Extracting data" }, { "progress": 1, "step": "8", "stepName": "Consolidate dataset" }, { "progress": 1, "step": "9", "stepName": "Publish dataset" }, { "progress": 1, "step": "10", "stepName": "Cleaning up" } ], "totalSteps": "10", "nebula": { "fileSizeBytes": "795061", "url": "example-url" } } } ``` {% endtab %} {% tab title="400" %} ```json { "error": "Invalid request" } ``` {% endtab %} {% endtabs %} ### Cancel requests Use `CancelDataset()` to stop a build. If it's done, that call will purge the dataset. {% tabs %} {% tab title="TypeScript" %} ```javascript import { GravityClient } from 'macrocosmos'; // Initialize the client const client = new GravityClient({ apiKey: 'your-api-key' }); // Cancel a gravity task const cancelResult = await client.cancelGravityTask({ gravityTaskId: 'your-gravity-task-id' }); // Cancel a dataset build // const cancelDataset = await client.cancelDataset({ // datasetId: 'your-dataset-id' // }); ``` {% endtab %} {% tab title="Python" %} ```python import macrocosmos as mc client = mc.GravityClient(api_key="your-api-key") # Cancel a gravity task response_grav = client.gravity.CancelGravityTask( gravity_task_id='your-gravity-task-id' ) # Cancel a dataset # response_data = client.gravity.CancelDataset( # dataset_id='your-dataset-id' #) # Print the dataset ID print(response_grav) ``` {% endtab %} {% tab title="Constellation API: curl" %} ```bash curl -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '{ "gravity_task_id": "your-gravity-task-id" }' \ https://constellation.api.cloud.macrocosmos.ai\ /gravity.v1.GravityService/CancelGravityTask ``` {% endtab %} {% tab title="Constellation API: grpcurl" %} ```bash grpcurl -H "Authorization: Bearer your-api-key" \ -d '{ "gravity_task_id": "your-gravity-task-id" }' \ constellation.api.cloud.macrocosmos.ai:443 \ gravity.v1.GravityService/CancelGravityTask ``` {% endtab %} {% endtabs %} #### Body | Name | Type | Description | | ----------------------------- | ------ | ------------------------- | | `gravityTaskId` (`datasetId`) | string | Gravity task (dataset) Id | #### Response {% tabs %} {% tab title="200" %} ```json { "message": "success" } ``` {% endtab %} {% tab title="400" %} ```json { "error": "Invalid request" } ``` {% endtab %} {% endtabs %} ### Streaming API ( On Demand Data API) Run precise, real-time queries using the synchronous `Sn13Client` to query historical or current data based on users, keywords, and time range on platforms like X (Twitter) and Reddit. The Streaming API is limited to 1000 posts per request. As of the latest data-universe [release](https://github.com/macrocosm-os/data-universe/releases): * Users may select two post-filtering modes via the `keyword_mode` parameter: * `"any"` : Returns posts that contain any combination of the listed keywords. * `"all"` : Returns posts that contain all of the keywords (default, if field omitted). * For Reddit requests, the first keyword in the list corresponds to the requested subreddit, and subsequent keywords are treated as normal. * URL mode is mutually exclusive with `usernames` and `keywords` fields. If `url` is provided, `usernames` and `keywords` must be empty. {% tabs %} {% tab title="Typescript" %} ```typescript import { Sn13Client } from 'macrocosmos'; // Initialize the client const client = new Sn13Client({apiKey: 'your-api-key'}); // Get the onDemandData response const response = await client.onDemandData({ source: 'X', // or 'Reddit' usernames: ['nasa', 'spacex'], // Optional, up to 5 users keywords: ['photo', 'space', 'mars'], // Optional, up to 5 keywords startDate: '2024-04-01', // Defaults to 24h range if not specified endDate: '2025-04-25', // Defaults to current time if not specified limit: 3, // Optional, up to 1000 results keywordMode: 'any', // Optional, 'any' or 'all' }); ``` {% endtab %} {% tab title="Python" %} ```python import macrocosmos as mc client = mc.Sn13Client(api_key="your-api-key") response = client.sn13.OnDemandData( source='X', # or 'Reddit', 'YouTube' usernames=["nasa", "spacex"], # Optional, up to 5 users keywords=["photo", "space", "mars"], # Optional, up to 5 keywords start_date='2024-04-01', # Defaults to 24h range if not specified end_date='2025-04-25', # Defaults to current time if not specified limit=3, # Optional, up to 1000 results keyword_mode='any' # Optional, 'any' or 'all' ) print(response) ``` {% endtab %} {% tab title="Constellation API: curl" %} ```bash curl -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '{ "source": "X", "usernames": ["nasa", "spacex"], "keywords": ["photo", "space", "mars"], "start_date": "2024-04-01", "end_date": "2025-04-25", "limit": 3, "keyword_mode": "any" }' \ https://constellation.api.cloud.macrocosmos.ai\ /sn13.v1.Sn13Service/OnDemandData ``` {% endtab %} {% tab title="Constellation API: grpcurl" %} ```bash grpcurl -H "Authorization: Bearer your-api-key" \ -d '{ "source": "X", "usernames": ["nasa", "spacex"], "keywords": ["photo", "space", "mars"], "start_date": "2024-04-01", "end_date": "2025-04-25", "limit": 3, "keyword_mode": "any" }' \ constellation.api.cloud.macrocosmos.ai:443 \ sn13.v1.Sn13Service/OnDemandData ``` {% endtab %} {% endtabs %} #### Body | Name | Type | Description | | ------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `source` | string | Data source (`X` or `Reddit`). | | `usernames` | Array of strings |

Default: \[]

Number of items: <= 10 items

List of usernames to fetch data from. Searches for posts from any of the given usernames.

If usernames are not included, they will not be constrained in the search parameters.

| | `keywords` | Array of strings |

Default: \[]

Number of items: <= 5 items

List of keywords to search for. Searches for posts where all given keywords are present.

If keywords are not included in the query, they will not be constrained in the search parameters.

For Reddit:
The first keyword indicates the subreddit (r/all for cross-subreddit queries), and subsequent keywords are text matches.

| | `startDate` | string |

\[Optional]

Start date or datetime (ISO format).

Defaults to 24 hours prior to the request time if not specified.

Datetimes without time information will be set to midnight (00:00:00) by default.

Datetimes without timezone information will be set to UTC by default.

| | `endDate` | string |

\[Optional]

End date or datetime (ISO format).

Defaults to the request time if not specified.


Datetimes without time information will be set to midnight (00:00:00) by default.

Datetimes without timezone information will be set to UTC by default.

| | `limit` | integer |

\[Optional]

Default: 100

Options: \[1,...,1000]

Maximum number of items to return.

| | `keywordMode` | string |

\[Optional]

Default: all

Options: all , any

Selects the post-filtering mode:

  • "any": Returns posts that contain any combination of the listed keywords.
  • "all": Returns posts that contain all of the keywords
| | `url` | string |

\[Optional]

Single url for URL search mode for X (Twitter)

If url is provided, usernames and keywords must be empty or omitted.

| #### Response {% tabs %} {% tab title="200" %} {% code fullWidth="false" %} ```json { "data": [ { "content": "Falcon 9 launches the Bandwagon-3 rideshare mission to orbit from Florida", "datetime": "2025-04-22T03:00:38+00:00", "label": null, "media": [ { "type": "photo", "url": "https://pbs.twimg.com/media/GpG2kuBagAADw92.jpg" }, { "type": "photo", "url": "https://pbs.twimg.com/media/GpG2kuDa4AEr1RV.jpg" }, { "type": "photo", "url": "https://pbs.twimg.com/media/GpG2kuBbUAAU7Rd.jpg" }, { "type": "video", "url": "https://pbs.twimg.com/amplify_video_thumb/1914512114154409984/img/lD1axdjW7cRnRol6.jpg" } ], "source": "X", "tweet": { "conversation_id": "1914514653763584254", "hashtags": [], "id": "1914514653763584254", "is_quote": false, "is_reply": false, "is_retweet": false, "like_count": 10689, "quote_count": 76, "reply_count": 677, "retweet_count": 2058 }, "uri": "https://x.com/SpaceX/status/1914514653763584254", "user": { "display_name": "SpaceX", "followers_count": 39073448, "following_count": 121, "id": "34743251", "username": "@SpaceX", "verified": false } }, { "content": "Falcon 9 launches NROL-145 from California, completing our first of the new national security missions awarded in October 2024", "datetime": "2025-04-20T17:05:09+00:00", "label": null, "media": [ { "type": "photo", "url": "https://pbs.twimg.com/media/Go_mYDJbIAA9mbK.jpg" }, { "type": "video", "url": "https://pbs.twimg.com/amplify_video_thumb/1914001831661084672/img/ydKPVd7KoS6B6U_l.jpg" } ], "source": "X", "tweet": { "conversation_id": "1914002408545615936", "hashtags": [], "id": "1914002408545615936", "is_quote": false, "is_reply": false, "is_retweet": false, "like_count": 8190, "quote_count": 71, "reply_count": 495, "retweet_count": 1802 }, "uri": "https://x.com/SpaceX/status/1914002408545615936", "user": { "display_name": "SpaceX", "followers_count": 39073448, "following_count": 121, "id": "34743251", "username": "@SpaceX", "verified": false } } ] } ``` {% endcode %} {% endtab %} {% tab title="400" %} ```bash { "error": "Invalid request" } ``` {% endtab %} {% endtabs %} #### Request Examples ``` import macrocosmos as mc client = mc.Sn13Client(api_key="your-api-key") response = client.sn13.OnDemandData( source='X', # Searches X keywords=["AI"], # For posts about AI start_date='2024-08-01', # From midnight 2024-08-01 UTC # To the time this request was made. limit=10 # For 10 items maximum ) print(response) ``` ``` import macrocosmos as mc client = mc.Sn13Client(api_key="your-api-key") response = client.sn13.OnDemandData( source='Reddit', # Searches Reddit keywords=["r/astronomy", "space"], # For posts/comments mentioning 'space', in the r/astronomy subreddit # In the default time range of the past 24 hours limit=50 # For 50 items maximum ) print(response) ``` ``` import macrocosmos as mc client = mc.Sn13Client(api_key="your-api-key") response = client.sn13.OnDemandData( source='Reddit', # Searches Reddit keywords=["r/all", "space"], # For posts/comments mentioning 'space', across all subreddits start_date='2025-04-01', # From midnight 2025-04-01 UTC end_date='2025-04-02', # To midnight 2025-04-02 UTC limit=50 # For 50 items maximum ) print(response) ``` # API Keys How to Create a Macrocosmos API Key To interact with Macrocosmos subnets programmatically, you'll need a personal API key. ### Step 1: Organise your credits 1. Head to [app.macrocosmos.ai](https://app.macrocosmos.ai) and login. 2. Click to Macrocosmos logo icon at the top left corner of the screen\ ![](/files/saNmKah1qCCkWdzNDKU5) 3. In the dropdown menu, select **`Payments and credit`.** 4. Provide your credit card details in order to access the account management and **Constellation API Key Dashboard.** ### Step 2: Generate Your Personal API Key 1. From the home menu and select **`Account Settings`.**
2. Click to the API Keys tab and create a new key. 3. Copy the generated API key immediately and save it in a secure location , you **won’t be able to view it again**. You can now use this key to interact with **all Macrocosmos services** from your preferred coding environment. ### You’re Ready! With your API key and credits in place, you’re ready to build with Macrocosmos! # Data Universe MCP Using Data Universe MCP with Claude Desktop or Cursor ## Data Universe MCP Using Data Universe MCP with Claude Desktop or Cursor *** Data Universe MCP (Model Context Protocol) allows you to integrate with Data Universe APIs directly into Claude for Desktop, Cursor, or your custom LLM pipeline. Query **X (Twitter)** and **Reddit** data on demand from your AI environment! ### Prerequisites * Python 3.10+ * `uv` package manager * Claude Desktop or Cursor installed ### Install UV Package Manager ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Or via pip: ```bash pip3 install uv ``` ### Quickstart 1. Get your API key from [Macrocosmos](https://app.macrocosmos.ai/account?tab=api-keys). There is a free tier with $5 of credits to start. 2. Install `uv` using the command above or see the [uv repo](https://github.com/astral-sh/uv) for additional install methods. *** ### Configure Claude Desktop Run the following command to open your Claude configuration file: ```bash code ~/Library/Application\ Support/Claude/claude_desktop_config.json ``` Update with this configuration: ```json { "mcpServers": { "macrocosmos": { "command": "uvx", "args": ["macrocosmos-mcp"], "env": { "MC_API": "" } } } } ``` Open Claude Desktop and look for the **hammer icon** — this confirms your MCP server is running. You'll now have SN13 tools available inside Claude. *** ### Configure Cursor #### Option 1: Via UI (Recommended) 1. Go to **Cursor Settings** 2. Navigate to MCP settings and select **Add New Global MCP Server** 3. Enter the configuration details #### Option 2: Manual JSON ```bash code ~/Library/Application\ Support/Cursor/cursor_mcp_config.json ``` Add the same configuration: ```json { "mcpServers": { "macrocosmos": { "command": "uvx", "args": ["macrocosmos-mcp"], "env": { "MC_API": "" } } } } ``` > ⚠️ **Note:** In some cases, manually editing this file doesn't activate the MCP server in Cursor. If this happens, use the UI method above for best results. #### Use Agent Mode In Cursor, make sure you're using **Agent Mode** in the chat. Agents have the ability to use any MCP tool — including custom ones and those from SN13. *** ### Available Tools #### Quick Query Tool **`query_on_demand_data` - Real-time Social Media Queries** Fetch real-time data from X (Twitter) and Reddit. Best for quick queries up to 1,000 results. | Parameter | Type | Description | | -------------- | ------ | ------------------------------------------------------------------------------------ | | `source` | string | **REQUIRED**. Platform: `'X'` or `'REDDIT'` (case-sensitive) | | `usernames` | list | Up to 5 usernames. For X: `@` is optional. Not available for Reddit | | `keywords` | list | Up to 5 keywords/hashtags. For Reddit: subreddit names (e.g., `'r/MachineLearning'`) | | `start_date` | string | ISO format (e.g., `'2024-01-01T00:00:00Z'`). Defaults to 24h ago | | `end_date` | string | ISO format. Defaults to now | | `limit` | int | Max results 1-1000. Default: 10 | | `keyword_mode` | string | `'any'` (default) or `'all'` for strict matching | **Example prompts:** * "What has @elonmusk been posting about today?" * "Get me the latest posts from r/bittensor about dTAO" * "Fetch 50 tweets about #AI from the last week" *** #### Large-Scale Collection Tools (Gravity) Use Gravity tools when you need to collect large datasets over 7 days (more than 1,000 results). **`create_gravity_task` - Start 7-Day Data Collection** | Parameter | Type | Description | | --------- | ------ | ------------------------------------ | | `tasks` | list | **REQUIRED**. List of task objects | | `name` | string | Optional name for the task | | `email` | string | Email for notification when complete | **Task object structure:** ```json { "platform": "x", "topic": "#Bittensor", "keyword": "dTAO" } ``` > ⚠️ **Important:** For X (Twitter), topics **MUST** start with `#` or `$` (e.g., `#ai`, `$BTC`). Plain keywords are rejected! *** **`get_gravity_task_status` - Monitor Collection Progress** | Parameter | Type | Description | | ------------------ | ------ | ---------------------------------------------------- | | `gravity_task_id` | string | **REQUIRED**. The task ID from create\_gravity\_task | | `include_crawlers` | bool | Include detailed stats per crawler. Default: `True` | **Returns:** Task status, crawler IDs, records\_collected, bytes\_collected *** **`build_dataset` - Build Dataset from Collected Data** | Parameter | Type | Description | | ------------ | ------ | ------------------------------------------------- | | `crawler_id` | string | **REQUIRED**. Get from get\_gravity\_task\_status | | `max_rows` | int | Max rows to include. Default: 10,000 | | `email` | string | Email for notification when ready | > ⚠️ **Warning:** Building a dataset will **STOP** the crawler and de-register it from the network. *** **`get_dataset_status` - Get Download Links** | Parameter | Type | Description | | ------------ | ------ | ------------------------------------------------ | | `dataset_id` | string | **REQUIRED**. The dataset ID from build\_dataset | **Returns:** Build status, download URLs for Parquet files when complete *** **`cancel_gravity_task` - Stop Data Collection** | Parameter | Type | Description | | ----------------- | ------ | ----------------------------------- | | `gravity_task_id` | string | **REQUIRED**. The task ID to cancel | *** **`cancel_dataset` - Cancel Build or Purge Dataset** | Parameter | Type | Description | | ------------ | ------ | -------------------------------------------- | | `dataset_id` | string | **REQUIRED**. The dataset ID to cancel/purge | *** ### Example Workflows #### Quick Query (On-Demand) ``` User: "What's the sentiment about $TAO on Twitter today?" → Uses query_on_demand_data to fetch recent tweets → Returns up to 1,000 results instantly ``` #### Large Dataset Collection (Gravity) ``` User: "I need to collect a week's worth of #AI tweets for analysis" 1. create_gravity_task → Returns gravity_task_id 2. get_gravity_task_status → Monitor progress, get crawler_ids 3. build_dataset → When ready, build the dataset (stops crawler) 4. get_dataset_status → Get download URL for Parquet file ``` *** ### Example Prompts #### On-Demand Queries * "What has the president of the U.S. been saying over the past week on X?" * "Fetch me information about what people are posting on r/politics today" * "Please analyze posts from @elonmusk for the last week" * "Get me 100 tweets about #Bittensor and analyze the sentiment" #### Large-Scale Collection * "Create a gravity task to collect data about #AI from Twitter" * "Start a 7-day collection of $BTC tweets with keyword 'ETF'" * "Check how many records my gravity task has collected" * "Build a dataset with 10,000 rows from my crawler" *** ### Supported Platforms | Platform | Username Filtering | Keyword Search | Subreddit Filtering | | ----------- | ------------------ | -------------- | ------------------- | | X (Twitter) | ✅ Yes | ✅ Yes | N/A | | Reddit | ❌ No | ✅ Yes | ✅ Yes | *** ### Troubleshooting If you encounter any issues: 1. **Ensure you're using Python 3.10+** 2. **Verify uv is installed:** Run `uv --version` 3. **Check your API key:** Ensure `MC_API` is set correctly 4. **Restart the application:** After config changes, restart Claude Desktop or Cursor For more on MCPs, refer to the [official MCP documentation](https://modelcontextprotocol.io/). # Data Universe OpenClaw Skills Fetch real-time social media data from X (Twitter) and Reddit by keyword, username, date range, and filters with engagement metrics via Macrocosmos SN13 API on Bittensor. Fetch real-time social media data from X (Twitter) and Reddit by keyword, username, date range, and filters with engagement metrics via Macrocosmos SN13 API on Bittensor. Link to the ClawHub: ### Tips & Known Behaviors #### What works reliably * **High-volume keyword searches**: Popular terms like "bittensor", "AI", "iran", "lfg" return fast * **Wider date ranges**: Setting `start_date` further back (e.g., weeks/months) improves results * **`keyword_mode: "all"`**: Great for finding intersection of two topics (e.g., "chutes" AND "bittensor") #### What can be flaky * **Username-only queries**: Can timeout (DEADLINE\_EXCEEDED). Adding `start_date` far back helps * **Niche/low-volume keywords**: Very specific terms may timeout if miners don't have data indexed * **No `start_date`**: Defaults to last 24h which can miss data; set explicitly for best results #### Best practices for LLM agents 1. **Always set `start_date`** — don't rely on the 24h default. Use at least 7 days back for user queries 2. **Prefer keywords over usernames** — keyword searches are more reliable 3. **For username queries, always include `start_date`** set weeks/months back 4. **Use `keyword_mode: "all"`** when combining a topic with a subtopic (e.g., "bittensor" + "chutes") 5. **Handle timeouts gracefully** — if a query times out, retry with broader date range or switch to keyword search 6. **Parse engagement metrics** — `view_count`, `like_count`, `retweet_count` help rank relevance 7. **Check `is_reply` and `is_quote`** — filter for original tweets vs replies depending on use case # Training at Home What is Training at Home (TAH)?
Today, access to large-scale compute is concentrated in a few organizations, limiting innovation and slowing progress across the ecosystem. By enabling anyone to contribute hardware and collectively train powerful models, we lower the barrier to participation, unlock underutilized global compute, and empower a wider community of builders and researchers. This creates a more resilient, transparent, and scalable foundation for advancing open-source AI - one that grows stronger as more people join and contribute. TAH is a decentralized, community-driven, and efficient pre-training platform. It is built on top of IOTA, a data- and pipeline-parallel training algorithm designed to operate on a network of heterogeneous, unreliable devices in adversarial and trustless environments. For more information on IOTA, please see the [IOTA documentation](/subnets/subnet-9-iota). You can plug in any hardware you have access to and earn rewards for powering decentralized AI model training. We aim to solve one of the biggest challenges in open-source AI today: the absence of a decentralized, community-driven, and highly efficient pre-training platform. #### Demo Video {% embed url="" %} #### Learn More * [TAH User Guide](/product-and-services/tah/tah-user-guide) * [Hardware & OS Requirements](/product-and-services/tah/hardware-requirements) * [FAQs](/product-and-services/tah/faqs) # TAH User Guide Plug in to power decentralised AI model training The Training at Home (TAH) application allows you to connect and partake in our decentralized AI training platform [IOTA](/subnets/subnet-9-iota). You can plug in any hardware you have access to and earn rewards for powering decentralized AI model training. ## Installation To start training at home, visit and click "Download" in the Train at Home section. Currently TAH only supports MacOS, with Linux support coming soon. Look at [Hardware & OS Requirements](/product-and-services/tah/hardware-requirements) for more details. If you already have access, you can double click a `.dmg` file to install the app. Video below demonstrates installation process: {% embed url="" %} ## Operating TAH Once the app is installed you should be able to see the main dashboard: ![TAH dashboard showing Start training and Connect controls](/files/dYXR6ovbY3gWD0Z6l8dk) You’ll see metrics for your model, and in the center there will be a visualization of all the other participants in the training. #### Main controls (top right of the screen)
* **Start training**: begins contributing your GPU/CPU to the current run. Click again to stop/exit the run. * **Connect**: optionally paste your wallet coldkey to receive rewards to that address. You can train without connecting; if you connect later, rewards accrue to the provided coldkey from that point forward. * **Status pill**: shows whether your hardware is ready (e.g., “Connected • Your GPU is ready to contribute”). Resolve any warnings before starting. #### Start a session 1. Launch the app and wait for the status pill to read **Connected**. 2. (Optional) Click **Connect**, paste your wallet coldkey, and save. 3. Click **Start training**. The run ID, model size, and layer count populate in the right panel; progress and tokens/hour graphs begin updating. 4. To stop, click **Start training** again (it toggles off) or quit the app. 5. A TAH tray icon is also available that allows you to stop or start training and to quit the app:
#### Monitor while training * **Run panel**: shows run ID, model size, layers, cumulative progress, and tokens/hour graph.
*** * **Network view**: visualizes peers and traffic; switch tabs (Network/Layer/Miner) for different views.
*** * **Loss chart**: training loss over time; toggle Log/Linear and tokens/time axes.
*** * **Run Log** (bottom right): per-epoch/step logs; lock icon indicates read-only when idle.
#### Resource behavior * The app uses available GPU/CPU once training starts. If you need to pause, toggle **Start training** off. * Keep the laptop powered and on a stable network for steady rewards. Thermal throttling may reduce contribution; use a cooler or lower-power profile if needed. #### Updates * If an update is available, install the latest build before starting a new session for compatibility with the active run. * If you do not update, you will not be able to join the current run. ## Rewards Earnings will be calculated and paid as described below. #### How Rewards Are Calculated Rewards are primarily based on: * The amount of tokens you contribute during training. * How many times you contribute your model weights back to the network. * Additional factors such as uptime, quality, and current network parameters. Note: Exact weighting can evolve as the system updates. #### Payout Cadence * Payouts occur approximately every 24 hours; this window may be extended in the future. * Minimum payout is subject to a variable network threshold (approximately 0.4 alpha currently). Balances below the threshold roll over to the next payout. For details see the [FAQs](/product-and-services/tah/faqs) "Earnings and Payouts" section. * Payouts may be frozen if the active run is not improving. For details see the [FAQs](/product-and-services/tah/faqs) "Earnings and Payouts" section. * Network transfer costs are deducted from the amount you receive. * To receive payouts, optionally connect a wallet by pasting your public coldkey (via the **Connect** button in the app). If you don’t connect, training still runs but rewards won’t be delivered to you; unconnected earnings are not paid retroactively. #### Viewing Earnings * In the app, click the **Miner** button (top left) to view your current earnings and contribution stats.
* *Total Earned*: Total amount credited to you, including both completed and pending payments. * *Total Paid:* The amount paid to your wallet. * *Total Pending:* Pending amount credited to you, but not yet paid. ## Accessing Log data for T\@H To document how the Train at Home app is running, operation logs are stored on your computer. You can access these via the following steps: 1. Open the Terminal app. 2. Navigate to the T\@H logging directory by typing `cd ~/Library/Logs/IOTA\ Train\ at\ Home` in the Terminal window, then press "enter". 3. Type `ls` and press "enter". This will show you your current log files. The file which provides diagnostic data on your T\@H activity is named `YYYY-MM-DD-cli.log`. 4. Choose the relevant date and open the file by typing `cat YYYY-MM-DD-cli.log` and pressing "enter". This will show the file contents in the terminal, allowing you to scroll through the T\@H events. 5. You can copy this log to another directory using `cp YYYY-MM-DD-cli.log /Users//Documents/logs` for example.

Commands used to access the T@H logs, with a print out of initial logs

#### Taxes & Compliance * You are responsible for complying with local tax and regulatory requirements related to rewards. {% columns %} {% column %} For more questions use [FAQs](/product-and-services/tah/faqs) {% endcolumn %} {% column %} If you have any issues use [TAH Support](broken://pages/sOuXmk7Tjf1tKjUMyPCT) {% endcolumn %} {% endcolumns %} # Hardware & OS Requirements Supported platforms and resource guidance for TAH #### Supported Platforms * macOS Apple Silicon. M‑class chips are recommended for efficiency but not required. * Linux support is planned; Windows is not yet supported. #### Minimum Specs (participate) * CPU/GPU: modern Apple Silicon capable of sustaining training loads. * Memory: **16 GB RAM minimum**. * Disk: At least 10 GB free for binaries, caches, and logs; up to **30 GB** may be required depending on the current model size being trained. * Network: Fast, stable internet connection is imperative; training will stall on intermittent links. * Power/uptime: Keep the machine plugged in and awake to maintain contribution. #### Recommended Specs (best performance) * Apple Silicon **M‑class chips** (M1/M2/M3) for best performance-per-watt. * 16–32 GB RAM to avoid swapping under load. * Wired Ethernet or strong Wi‑Fi 6 for steady throughput. * Use a cooling pad or elevated stand to reduce thermal throttling during long runs. #### Network & Security * Requires a reliable internet connection throughout the session; reconnects may drop your rewards. * Run on a trusted network; avoid captive portals or VPNs that frequently re-authenticate. {% columns %} {% column %} For more questions use [FAQs](/product-and-services/tah/faqs) {% endcolumn %} {% column %} If you have any issues use [TAH Support](broken://pages/sOuXmk7Tjf1tKjUMyPCT) {% endcolumn %} {% endcolumns %} # TAH Support and FAQs Frequently asked questions and TAH support channels {% columns %} {% column %} ![](/files/7tF4ewLP5RF4CWhIvXv2) **Get started** [Training At Home (TAH) User Guide](/product-and-services/tah/tah-user-guide) [Hardware & OS Requirements](/product-and-services/tah/hardware-requirements) {% endcolumn %} {% column %} ![](/files/LCYRWYUZsNHtG4DvvZZn) **Contact us for support** [Macrocosmos Discord](https://discord.gg/vRTaAXpRcd) [Bittensor Discord](https://discord.com/channels/799672011265015819/1162768567821930597) [Support Email](mailto:support@macrocosmos.ai) {% endcolumn %} {% endcolumns %} ## Getting Started
What is Training at Home? Training At Home lets you contribute your computer’s spare power to help train models on the IOTA network. You can participate for free or earn rewards.
How do I get access to Training At Home (TAH)? The TAH app is now available to everyone, and can be downloaded by clicking "Download" in the Train at Home section of .
What platforms are supported by TAH? MacOS at launch, Linux following soon after. Windows is not on the roadmap yet. For more details have a look at [Hardware & OS Requirements](/product-and-services/tah/hardware-requirements).
Do i need to create an account? No account is required. Your setup is tied to your device and, if you choose to earn rewards, a wallet address.
## Wallets & Rewards
Do i need a wallet to use Training at Home? No. A wallet is only required if you want to receive rewards. You can train and contribute without one.
I want to earn rewards. What do i need? To earn rewards you only need one thing: * A Bittensor coldkey address. A coldkey is simply the address where rewards are sent. You do not need crypto experience, tokens, or an exchange to start.
What is a Bittensor wallet? A Bittensor wallet is just where your rewards are sent if you choose to earn money from Training At Home. Training At Home runs on the Bittensor network, and Bittensor does not use accounts or usernames. Instead, it sends rewards directly to a wallet address, similar to how a bank transfer needs an IBAN. If you connect a wallet, Training At Home knows where to pay you. If you don’t connect a wallet, you can still train and contribute, you just won’t receive rewards. The app only ever uses the coldkey address, which is a public payout address. It never asks for passwords, private keys, or recovery words, and it never has access to your funds or your computer. In short: Bittensor wallet = optional payout address for rewards.
Do I need to connect a Bittensor wallet to use Train at home? No. Connecting the wallet is optional. If you'd like to receive rewards paste your public coldkey. Support for popular wallet managers is coming soon. See the [Rewards](https://github.com/macrocosm-os/macrocosmos-content/blob/main/main-content/product-and-services/tah/rewards.md) page for payout details.
What is a Coldkey? A coldkey is the address where your rewards are paid. When you earn rewards in Training At Home, the system needs to know *where to send them*. The coldkey is simply that destination. It works like a bank account number, you can receive money through it, but it does not give anyone permission to spend it. Training At Home only uses the coldkey to send rewards. It cannot see your balance, move funds, access your wallet, or control your computer.
You never share passwords, secret words, or private keys. You only paste the coldkey address, nothing else.
In short: Coldkey = safe, read only payout address for rewards.
How do i setup a Bittensor Coldkey on MacOS? #### **Step 1, open Terminal** 1\. Open Finder 2\. Go to Applications 3\. Open Utilities 4\. Click Terminal #### **Step 2, install Python 3.12** If you already have Python 3.12, skip this. 1\. Go to python.org and install Python 3.12 2\. Restart your computer 3\. In Terminal, check it worked by typing: `python3.12 --version`\
You should see Python 3.12.x.\ \
#### **Step 3, create a safe “wallet folder”** This keeps everything tidy.\ \ Copy and paste:\ `python3.12 -m venv bt-env`\ `source ~/bt-env/bin/activate`\ \ You will see (bt-env) in Terminal. That means it is working. #### **Step 4, install the wallet tool** Copy and paste: `python -m ensurepip --upgrade`\ `python -m pip install --upgrade pip`\ `python -m pip install bittensor bittensor-cli` If macOS asks to install developer tools, click Install. If it asks for your Mac password, that is normal. #### **Step 5, create your wallet** Copy and paste: `btcli wallet create`
It will ask: • Wallet directory, press Enter • Wallet name, choose any name, for example AlmasWallet, this is not public • Coldkey name, type default **Then it will show you secret words.** #### **Step 6, write down your secret words** This is the most important part. • Write the words down on paper • Do not screenshot • Do not store them on your computer • Do not share them **The app will never ask for these words.** \ **Step 7, find your coldkey address** Copy and paste:\ `btcli wallet list`
You will see: • Coldkey address • Hotkey address You only need the coldkey address.
Coldkey looks like a long string, for example: 5EHRFDz...
**Hotkey looks similar but is not used for payouts.**\ \ Step by step, connect your coldkey in the IOTA app 1\. Open the Training At Home app 2\. Click Connect wallet 3\. Select Specify a Bittensor wallet coldkey for payouts 4\. Paste your coldkey address 5\. Click Connect 6\. Click Start training That is it.
I cannot install a Coldkey wallet tool, what should i do? The most common fixes are: • Install Python 3.12 • Allow macOS developer tools when prompted • Install OpenSSL if your Mac asks for it
If you still get stuck, contact support and include the error text.
Does the app ever need my secret words or password? No. If any screen asks you for: • secret words • mnemonic • private key **Stop. Do not continue. That is not required for Training At Home.**
Do I need an exchange account to earn? No. You can earn rewards without an exchange account. An exchange is only needed later if you want to convert TAO into cash, and availability depends on your region.
Can I change my wallet later? Yes. You can disconnect or replace your coldkey at any time.
How do I create a wallet if I’m not technical? We provide a [step by step guide ](#how-do-i-setup-a-bittensor-coldkey-on-macos)that walks you through creating a wallet safely on macOS. You only need to do this once. If you get stuck, you can: * Ask someone technical to help once * Or run Training At Home without rewards
Can I train on multiple devices using the same payout address? Yes. While the miner address (hotkey) must be different on each device, your payout address (coldkey) can be the same across all. I.e. payouts from all devices can be delivered to your specified payout address.
## Earnings & Payouts
How are rewards calculated? Rewards are based on how much useful training work your computer contributes. This includes:\ • How many tokens your computer processes during training\ • How often your computer sends updates back to the network\ • Additional factors like uptime, reliability, and current network settings The exact weighting can change over time as the system improves.
How often are rewards paid? When rewards are active: * Payouts happen approximately every 24 hours * This timing may change in the future
Is there a minimum payout? Yes. * The minimum payout is determined by a dynamic network threshold, presently around 0.4 alpha * If your balance is below this amount, it rolls over to the next payout * Network transfer costs are deducted from the amount you receive
Are there other reasons why I have not been paid out? Yes. If your balance is above the minimum threshold, but the model being trained hasn't improved over that period, the payout entitlement for users can be frozen. In more detail * The performance of the model being trained is measured by a metric called "Global Loss". When this decreases, the model is improving. Conversely, increasing loss means the model is deteriorating. * If the loss is not at the global minimum (the smallest value in the run's history) in the current billing period, the payout entitlements for that period are currently frozen. * These frozen entitlements may paid out at a later time, subject to future eligibility criteria. We have an economic safeguard that delays rewards - a part of the layered validation stack. It is triggered automatically due to a loss fluctuation. In this case the rewards may not to be distributed on time. If after the check next day the state of the system is green, the rewards will be payed the next night.
Are there taxes or legal requirements? Yes. You are responsible for understanding and complying with any local tax or regulatory requirements related to rewards you receive.
Do I need to connect a wallet to earn rewards? You only need a wallet to receive payouts.\ • If you connect a wallet, rewards are sent to your coldkey\ • If you don’t connect a wallet, training still runs but rewards are not paid Rewards earned while no wallet is connected are not paid retroactively.
Where can I see my earnings? In the app: 1. Click the Miner button in the top left 2. View your current earnings and contribution stats
Why do my earnings show 0? The earnings may be showing 0 in case the machine contribution is not sufficient for the current training period. This can happen due to several reasons: 1. If it runs not long enough or the connection with the network is failing for any reason. In this case machine: * didn't submit weights * didn't do enough activations * didn't merge partitions * or fell asleep and had to rejoin multiple times 2. If the machine failed validations. For example, due to hardware or software compatibility or not sufficient activation files quality due to the network issues. 3. If any of the machines is in the chain layer 1 to layer 3 failed to transfer their results, therefore the loss of the current run is not at the global minimum 4. We don’t exclude the probability of bugs in our current Beta version of the system. We are continuously looking into the issues and fixing them.
## Training & Usage
What does “Start training” do? When you click Start training, your computer connects to the network and begins receiving small training tasks. It uses a portion of your computer’s resources to process those tasks and sends the results back to the network. Until you click Start training, the app is idle and does not use your computer. Once training starts, your contribution and activity will appear in the app. You can stop training at any time, and nothing continues running in the background when training is stopped.
Why does my contribution change over time? Your contribution is not a fixed score. It changes because the network is shared with many other people. As others join or leave, as training phases change, or if your computer or internet speed changes, your contribution number can go up or down. This is normal and does not mean something is wrong.
Can i pause or stop training? Yes. You are always in control. You can stop training at any time, and your computer will immediately stop doing training work.
What happens if I close the app, restart my computer, or my laptop sleeps? Training stops automatically. Nothing continues running when the app is closed or your computer goes to sleep. When you open the app again, you can start training again.
I have been training, but my contribution hasn't changed? What is wrong? Train at Home and activation processing in general depend on various factors, such as hardware performance, internet speed, and the presence of other participants. Since these factors can change over time, there may be periods when your training session results in no activation submissions (no contribution). In machine learning, training consists of a forward pass (the inference stage) followed by a backward pass (the learning stage). The model learns based on how close its prediction (inference) was to the correct answer. If T\@H allowed an unlimited number of predictions before performing any learning, those predictions would likely be of lower quality, which would hinder overall learning. It is therefore better to allow the model to make a limited number of predictions, eight for example, and then perform a learning step before continuing. This significantly improves training quality. Once you have completed your allocated eight predictions, you must wait for other participants to return the corresponding learning updates. Without this coordination, the work performed would not contribute effectively to the model. Therefore, if you have been training for a long time without seeing a change in contribution, it is likely that there is no issue, you may simply be waiting to receive the necessary learning updates. This highlights the importance of maintaining a stable connection and continuous training over extended periods, so that activations can be passed back and forth successfully among participants.
I am receving no (or lower) contribution on Layer 3, why is this happening? Last layer miners have a more intensive role in T\@H, as they need to calculate the model’s loss. This means it’s imperative that both internet-connection and machine capabilities are at a very high level. If you were performing well on other layers, but are struggling on this layer, it could mean your device or your internet is struggling with the workload. As there’s a little more heavy-lifting on the last layer, we try to compensate miners who are there.
I have a strong machine and fast internet, why am I performing poorly on T@H? While it's paramount that you have a powerful (and compatible) machine along with fast internet, there are scenarios where this does not yield the desired benefits on T\@H. Some contributors who have ideal setups perform poorly while training the model. This isn't due to their configuration, but rather it could be related to the other contributors that they've been matched with during parts of their training run. As this is a swarm training architecture, you're working as a collective, where your work relies on the work of others. If some of the participants you've been matched with have poor internet, or a poorly optimised machine, it can have a negative effect on your work.
What are T@H logs, and how do I access mine? Your T\@H logs act as a snapshot of your training history. They document your machine's activity on the network. To access your logs, follow [our steps here](/product-and-services/tah/tah-user-guide#accessing-log-data-for-t-h).
## Hardware & internet
What hardware do i need? Most modern computers can participate. More powerful computers usually contribute more, but you do not need special hardware to get started.\ \ A minimum of 16 GB RAM is recommended. More capable hardware generally contributes more, but you can still participate without having the best setup. If your device struggles, you can still help by training for shorter sessions
Will this slow down my computer? It can, depending on what else you are doing. Training uses compute resources, so you may notice:\ • Slower performance when multitasking\ • Higher fan activity\ • Faster battery drain if unplugged If you need full performance for work, you can pause training temporarily.
Will this overheat or damage my computer? It should not damage your computer. Your device has built in safety controls, and you can always stop training if it feels too warm or loud. If you notice persistent high heat, reduce usage by running fewer other apps while training, improve airflow, or train in shorter sessions.
How important is my internet connection? A stable internet connection helps your computer stay connected to the network and send results back. If your connection drops often, your contribution may be lower, which is normal.
Can I stop using Training At Home whenever I want? Yes. You can stop training or uninstall the app at any time. There are no penalties or commitments.
How can I improve performance/heat? Keep the machine plugged in, use a cooling pad/stand, and close heavy GPU/CPU apps. You can pause anytime by toggling **Start training** off.
What hardware do I need to contribute to TAH? 16 GB RAM minimum; M‑class chips recommended but not required. See [Hardware & OS Requirements](/product-and-services/tah/hardware-requirements) for full specification.
## Privacy & Security
Can the app access my files or personal data? No. The app does not read your files, see your screen, or monitor what you do. It only runs training tasks and sends technical results back to the network.
What data do you store? We store your public coldkey to pay you, and may store location and hardware metadata to allocate runs and monitor performance. Some training artifacts are stored locally on your machine.
## Updates
How do updates work? The app auto-updates on startup or prompts when an update is available. Staying current is essential to avoid errors or security issues.
# Gravity Power your business with real-time insights from the latest data with our data collection tool ### **What is** Gravity**?** In a world where consumer preferences shift overnight, businesses battle to stay relevant— outdated, incomplete data is no longer an option. Gravity is a data collection tool which **solves this problem** by enabling rapid access to fresh, relevant data in real-time. Gravity was created to make large-scale social data accessible, affordable, and instantly usable for teams who rely on real-time online conversations. Instead of dealing with unreliable scrapers, constant platform changes, API limitations, or high data-provider costs, users get a single interface that delivers fresh, structured data from major platforms like X, Reddit, and YouTube. Whether the goal is market intelligence, audience research, trend tracking, brand monitoring, or training AI models, Gravity provides clean, comprehensive, enriched social content that users can plug directly into their workflows. Gravity is available as {% columns %} {% column %}

Data Collection

Sign in

{% endcolumn %} {% column %}

Marketplace

Sign in

{% endcolumn %} {% column %}

API

Docs

{% endcolumn %} {% endcolumns %} Explore [Use Cases](#data-universe-use-cases) or stories from our recent collaborations in [Cases Studies](#case-studies) to see real examples of how teams use Gravity and the full range of benefits it can deliver. ### Gravity Use Cases
Cover image

How to understand how people feel about my brand?

Learn how to turn messy online conversations into a heartbeat for your brand, combining social listening, surveys, and simple dashboards to track feelings, fix issues, and amplify what customers love.Learn more.../files/OnlmslGR6CTYhccsArga

What is the best way to find out what topics people connect with my brand and competitors?

Turn scattered online mentions into a living association map, revealing where competitors dominate, and which untapped narratives your brand should claim next.Learn more.../files/cEtilMkwQEETnHsnMqCB

How can I identify emerging trends in my market before they show up in traditional surveys?

Turn online conversations into your early-warning radar, spotting weak signals, frustrations and opportunities on X, Reddit and YouTube long before surveys, so you can respond faster and outgo key competitors.Learn more.../files/PkWy2eGYQO12CZxGDe4g

How can I track changes in public perception after a product launch or major campaign?

Make every launch a learning loop by tracking perception shifts across social channels and reviews, revealing what changed in people’s minds so you can refine messaging and de-risk campaigns.Learn more.../files/AOZNaOxnIuZbYCUPxHWK
For more use cases >> [Data Universe Use Cases](#data-universe-use-cases) ### Data Universe Data Collection Demo {% embed url="" %} ### Case Studies ![](/files/Ghq0roud04Rjre9kTWTY)[TAOLOR: Building a subnet-native AI agent with distributed RAG](https://macrocosmosai.substack.com/p/taolor-building-a-subnet-native-ai) ![](/files/Yz0j3Fuxt21C0HgnrilP)[Building the future of authentic data: how subnet 111 leverages Macrocosmos' Gravity](https://macrocosmosai.substack.com/p/building-the-future-of-authentic) ![](/files/Gx4SjOasKRDXMS9O1FuF)[Global weather, social context: How SN13 is powering SN57's weather data innovation](https://macrocosmosai.substack.com/p/global-weather-social-context-how) ![](/files/RkwXSoqQDmac85mU6A0P) [Result: SN44, Score, partners with SN13’s Data Universe](https://macrocosmosai.substack.com/p/result-sn44-score-partners-with-sn13s)
# Data Collection and Marketplace Social Media Data Collection Solution from Macrocosmos Navigate to [app.macrocosmos.ai](https://app.macrocosmos.ai/) to access Data Universe. #### Choose between options * **Search** for any available data from the Marketplace * **Create a new scraping task** for the new customised data collection * or **Tasks** to see your data library.
#### Use the Macrocosmos icon to navigate the menu
#### **Create a New Task to collect data** With customised data collection request get only the data that truly matters. Understand audiences with confidence and turn social data into actionable insights for your brand or product. How it works: 1. Name your task. For example, Top 10 stock assets US. 2. Set your date range. 3. Choose your platforms X (Tweeter), Reddit or YouTube. 4. Add keywords, topics, or Youtube channels. 5. Click **Launch data collection** 6. Receive fresh, structured data collected specifically for your request in hours, or wait for a couple of day to get more data. You also can use the **AI Assistant** to help you:\ Simply type a topic into your chat with **Mission Commander**, then review the suggested labels, hashtags, and keywords that appear on the right side of the task collection request. You can easily customise them by adding new ones or deleting any that don’t fit. No complex setup. No scraping headaches. Just fast, precise social data delivered straight to you.
#### Mission Commander Choosing the right topics for data collection is harder than it looks. Social conversations shift fast, keywords evolve, and each platform has its own language. Miss the right hashtag, subreddit, or phrasing, and you risk collecting irrelevant data or missing the conversations that matter most. Many teams spend hours guessing which keywords to include, testing variations, and manually exploring communities just to ensure their dataset is meaningful. **Mission Commander** removes that guesswork. As a specialised AI assistant, it automatically suggests high-relevance keywords, trending hashtags on X, the most suitable subreddits on Reddit, and topic-aligned YouTube channels. It also explains *why* each choice fits your objective and recommends the best tactics for structuring your query. Whether you're tracking sentiment, analysing competitors, researching trends, or preparing training data, Mission Commander helps you choose smarter topics, faster, ensuring your collection is accurate, comprehensive, and aligned with your real goal. #### Demo of Mission Commander {% embed url="" %} **You can also interact with Data Universe through API at** [**docs.macrocosmos.ai**](https://docs.macrocosmos.ai/)**.** Our pricing plans are available at the [app.macrocosmos.ai](https://app.macrocosmos.ai/) in your account section or on at [datauniverse.macrocosmos.ai](datauniverse.macrocosmos.aihttps://datauniverse.macrocosmos.ai/). # Gravity Use Cases # How to understand how people feel about my brand? Social listening and consumer intelligence SaaS If you don’t know how people feel about your brand, you’re basically flying blind with a very expensive logo. Feelings drive word of mouth, renewals, recommendations, and whether prospects even give you a chance. When you can clearly see what delights, frustrates, or confuses people, every decision in marketing, product, and CX gets sharper—and your brand stops being a guess and starts being a measurable asset. Start by defining what you actually mean by “how people feel”: awareness, trust, satisfaction, and likelihood to recommend. Then map where those feelings show up: social media, reviews, support tickets, sales calls, communities, and search queries about your brand. [Set up a social listening workspace](https://app.macrocosmos.ai/gravity/tasks/create) on the platforms where your audience is most active (X, Reddit, YouTube, TikTok, forums): 1. Start by creating keyword rules for your brand name, product names, key people, and main competitors. As new mentions come in, tag each post with sentiment (positive, negative, neutral) and, if possible, a topic label such as “praise,” “bug,” “pricing,” “support,” or “feature request.” Aim for consistent rules so different teammates would tag the same post in the same way. 2. Then, review your tagged data weekly: sort by negative posts to spot recurring complaints, and by positive posts to identify what people love most. Save representative screenshots and direct quotes into a shared document or slide deck - these become powerful “voice of the customer” examples that explain why your metrics look the way they do and help teams prioritize fixes and improvements. 3. Combine this with direct feedback: short surveys, NPS (“How likely are you to recommend us?”), and one-to-one interviews with customers and churned users. Ask what nearly stopped them buying, what delighted them, and what they tell friends about you. 4. Finally, quantify and track over time. Build a simple sentiment dashboard that shows: volume of mentions, sentiment trend, top topics, and key issues by segment (e.g., agency vs. brand vs. SMB). Revisit monthly. The goal isn’t a perfect score, but a living picture of how people feel and what you should fix or amplify next. **The platforms specifics:** * **X**: real-time short posts mentioning your brand, product names, executives, hashtags. * **Reddit**: longer discussions in relevant subreddits (e.g., industry, product, tech, local communities). * **YouTube**: video reviews, unboxings, “first impression” videos + comments. **What to do:** * [Collect mentions](https://app.macrocosmos.ai/gravity/tasks/create) of your **brand, key products, and competitors** across all three. * Run **sentiment analysis** (positive/negative/neutral, emotion tagging). * Create a **dashboard** tracking sentiment over time: spikes, drops, volume. * Segment by **topic** (price, quality, service, UX) and **channel** (X vs Reddit vs YouTube). Understanding how people feel about your brand becomes powerful when you translate vague impressions into concrete signals, sources, and trends over time. By combining social listening, reviews, and direct feedback, you get both the numbers and the stories behind them. Make that view visible in a simple, recurring dashboard and it becomes a shared truth that aligns marketing, product, and CX around what to fix, protect, and amplify next. # What is the best way to find out what topics people connect with my brand and competitors? Social listening and consumer intelligence SaaS Understanding what topics people naturally connect with your brand and your competitors shows you what you actually stand for in the market versus what you *wish* you stood for. It helps you see which themes you own, where rivals are winning the conversation, and where there are gaps no one has claimed yet. With that map, you can sharpen your positioning, adjust messaging, and invest in the narratives that matter most to your audience. Identify the channels where your audience is active the most: * Start with your existing data: UTM tags, referral sources, CRM fields, and “How did you hear about us?” answers. * Combine this with audience interviews and competitor analysis—where do they post, run ads, and get engagement? * Prioritize channels where your ideal customers both **talk about problems** and **react to brands**. Define clear, organized sets of search terms your tool should look for in order to filter, compare, and analyze mentions consistently instead of searching ad hoc. This includes brand names, products, people, competitors, hashtags, and common misspellings, grouped by purpose, for example brand vs. competitor. Identify the social media sources where your audience would be present the most: * **X**: co-mentioned hashtags and phrases near your brand name. * **Reddit**: common themes in threads where your brand and competitors appear. * **YouTube**: titles/description keywords + recurring words in comments. [Collect the data](https://app.macrocosmos.ai/gravity/tasks/create) related to the defined keywords and tag or auto-classify mentions by **topic**. Start with a short list like: “pricing,” “UX/usability,” “features,” “performance,” “support,” “values/ethics,” “community,” “competitor comparison.” As new posts come in, apply one or more topic tags. Review and refine the list so it matches how people actually talk. Next, **analyze volume and co-occurrence**: * For your brand and each competitor, see which topics appear most often. * Look at “brand + topic” combos (e.g., “{Brand} + pricing” vs. “{Competitor} + pricing”). Layer in **search and content data**: * Build **co-occurrence maps**: what words/topics frequently appear with your brand (e.g., “expensive,” “fast,” “buggy,” “support”). * Compare your map with **competitor topic maps**. * Identify **white spaces** (topics competitors own that you don’t, or vice versa) to guide positioning and messaging. Finally, turn this into a simple report: top 5 topics for your brand vs. each competitor, with example quotes. That becomes your living “association map” for positioning and messaging.\ \ When you regularly map which topics cluster around your brand and competitors, you replace gut feel with a living, evidence-based view of what you stand for in people’s minds. This lets you double down on the associations that support your strategy, systematically repair the ones that don’t, and deliberately move into white-space territories others are ignoring. Over time, that “association map” becomes a practical steering wheel for brand, PR, content, and product decisions. # How can I identify emerging trends in my market before they show up in traditional surveys? Social listening and consumer intelligence SaaS Spotting emerging trends and issues early lets you react while competitors are still blind, shaping the narrative instead of responding to it. If you can see weak signals before they show up in surveys or analyst reports, you can protect your brand, de-risk campaign launches, and use new opportunities while they’re still undervalued. Start identifying emerging trends by treating public online conversations as an **early-warning radar**. 1. **Define signals.** List \~30–50 seed keywords: brand, competitors, category, known pain points, upcoming tech, regulations. Include slang and abbreviations. 2. [**Continuously collect data**](https://app.macrocosmos.ai/gravity/tasks/create)**.** * **X (Twitter):** track new phrases, memes, complaints, and unusual spikes in niche hashtags. * **Reddit:** new recurring questions or complaints in expert/enthusiast subreddits. * **YouTube:** new content formats or trending angles in your category. 3. **Track volume & sentiment over time.** Use dashboards to monitor: * Mention volume per topic. * Sentiment and emotion, such as worry, anger, excitement. * First-time or rapidly growing phrases, like“suddenly everyone says X”. 4. **Detect anomalies & new clusters.** Apply basic trend and clustering models to spot: * Sudden spikes in negative conversations about a feature or practice. * New use cases, needs, or complaints that weren’t present before. * New communities starting to talk about your category. 5. **Close the loop.** Review anomalies weekly with product, CX, and PR; flag issues to investigate and test with more structured methods (quick surveys, interviews, A/B tests). This way, social data becomes your “ahead of the survey” insight layer. By turning always-on online conversations into your early-warning system, you stop relying solely on slow, backward-looking surveys. Patterns in X, Reddit and YouTube give you real-time clues about shifting needs, rising frustrations and unexpected opportunities. When you review these signals regularly and feed them into product, CX and PR decisions, you don’t just react to the market - you quietly get ahead of it! # How can I track changes in public perception after a product launch or major campaign? Social listening and consumer intelligence SaaS After a launch or major campaign, the real question isn’t “Did we ship?” but “Did anything actually change in people’s minds?” If you can track shifts in public perception, you’ll see whether your message landed, what unintended narratives emerged, and which audiences reacted in ways you didn’t expect. That turns every launch from a one-off event into a learning loop that makes the next one smarter and less risky. Start by defining **what “perception change” means** for you: fewer complaints, more positive mentions, better reviews, higher NPS, or specific narratives. For example, “easy to use” instead of “too complex”. Then, set a **baseline** at least a few weeks before your launch: measure sentiment, share of voice vs. competitors, and top topics associated with your brand. After launch, use [**social listening**](https://app.macrocosmos.ai/gravity/tasks/create) to continuously track brand, product, and campaign mentions across key channels. Tag mentions by **sentiment** and **theme**, such as pricing, UX, bugs, support, values. Compare pre- and post-launch: * Volume of mentions * Ratio of positive/negative/neutral * Dominant themes in each sentiment bucket Add **structured feedback signals** too: survey responses, NPS, app-store or G2 reviews, support tickets, sales call notes. Look for alignment: do qualitative quotes match the trend in your metrics? Review weekly in a simple dashboard and highlight shifts in narratives (e.g., “confusing onboarding” declining, “fast support” rising). Share concrete examples with product, marketing, and leadership so they can respond quickly. **Suggest using:** * **X**: volume of mentions of your campaign slogan, branded hashtags, launch keywords. * **Reddit**: threads discussing the launch, early adopter feedback. * **YouTube**: “review,” “is it worth it,” “unboxing” videos and comments right after launch. **What to do:** * Define **time windows**: pre-launch, launch week, post-launch, for example, 4–6 weeks. * Compare **sentiment, volume, and key themes** across windows. * Track **share of voice vs competitors** around the launch topic. * Export insights into a simple **before/after perception report** for internal stakeholders. By treating launches as the start of a measurement process—not the finish line—you turn perception tracking into a core part of how you go to market. Consistently comparing pre- and post-launch data across social channels, reviews, and structured feedback gives you an honest view of what actually changed and why. Over time, those before/after reports become a feedback engine that sharpens your messaging, de-risks future launches, and helps every campaign land closer to the mark. # Gravity Support and FAQs Frequently asked questions and Data Universe support channels {% columns %} {% column %} ![](/files/7tF4ewLP5RF4CWhIvXv2) **Get started** [Data Universe Platform](/product-and-services/gravity) [Build with API](/subnets/subnet-13-data-universe/readme) [User Guides](/product-and-services/gravity/scraping-data) {% endcolumn %} {% column %} ![](/files/LCYRWYUZsNHtG4DvvZZn) **Contact us** [Macrocosmos Discord](https://discord.gg/hSuasW9q) [Bittensor Discord](https://discord.gg/a2h4JYTS) [Support Email](mailto:support@macrocosmos.ai) {% endcolumn %} {% endcolumns %} ### **Payments**
I have provided a credit card in my Data Universe Account. How do I get credits? Click on the **Payments and subscription** from the Gravity Home Navigation, then click Add credits button. If you’d like to discuss special price policies for your needs or having some issues, reach us out at .
Why my credit card is not accepted by the Data Universe Payment System? Your credit card may not be accepted because the Gravity Payment System is powered by Stripe, and Stripe has certain limitations on which cards, countries, and issuing banks are supported. If your card falls outside those supported criteria, the payment will be declined, please try a different card or contact your bank/card issuer.
Do credits roll over from month to month? Credits for "pay as you go" plan do not expire over time. Once you’ve purchased or received credits, they’ll remain in your account indefinitely until you decide to use them.\ \ Monthly allowance of posts to scrape for plans with subscription fee are expiring by the end of the month.
What payment methods does Data Universe support? Can I pay in $TAO, SN13 alpha, or even BTC? Currently we accept payments in fiat currencies through the payments system. Crypto token payments automation are in the shirt-term plan. Meanwhile it is possible to pay by digital currencies by invoice.
Where can I find Data Universe prices? The Gravity plans are listed at the [Data Universe Page](https://datauniverse.macrocosmos.ai/).
Is it the same cost to collect data from X and Reddit? Yes, the payment is taken per collected post and does not depend on the source.
### **Data Collection**
How much data can I collect? Gravity does not have limits to the amount of posts to collect. It collects data based on your selected hashtags or subreddits, and updates the quantity in real time as miners process your task.
How is data delivered? Once a task is complete or actively running, you can download it in **CSV** or **Parquet** format for easy use in data pipelines or LLM training.
Where the social media data can be used? There are multiple use cases for social media data exist. You can have a look at the Data Universe Use Cases section for inspiration. A couple of examples are * Fine-tuning LLMs for specific use-cases * Market research and brand tracking * Social Listening, and etc.
### **Mining**
Where do I start with mining on the subnet? Have a look at the guides for miners provided at [Subnet 13 Mining](/subnets/subnet-13-data-universe/data-universe-mining) section and contact us, if any additional questions.
### **Validating**
Where do I start with validating on the subnet? Have a look at the guides for validators provided at [Subnet 13 Validating](/subnets/subnet-13-data-universe/data-universe-validating) and contact us, if any additional questions.
# Managing And Collecting Your Data ### Track and Manage Your Tasks in Gravity To monitor your scraping progress and access results, go to your **Task Library**.\ This is where you’ll see the current status of each task you’ve submitted. **Pending**: Your task is waiting in the queue.
**Running**: Miners are actively working on your request.
### Build Your Dataset To build a dataset for a **single topic**, click **“Build dataset”** next to it. To fulfil your entire task, click **“Build all”**. If you chose to build a dataset, the scraping task for the relevant topics/ labels will be completed.
After clicking "Build all" Gravity begins collecting and preparing data across for the selected labels across X and reddit. The task moves through a few key processing stages: **Validating Data Sources, Collecting Available Data Sources** and **Collecting Crawler Information** **Validating Data Sources**\ Gravity is checking each selected topic (like `r/tech` or `r/googlepixel`) to ensure the source is reachable, active, and properly formatted for extraction. **Collecting Available Data Sources**\ This stage begins retrieving all accessible data from the validated sources. **Collecting Crawler Information**\ Gravity’s crawlers structure the data further.
It takes several minutes for dataset to be built, once your dataset is ready, you’ll get an **email notification**.\ Return back to your task library and you’ll also see a **“View Dataset”** button in your task row.\ Click it to preview the data. ### Download Your Data To download, hit **“Download”** — files are exported in **CSV** and **Parquet** format. ### Cancel a Task Tasks are present in the request lists for **7 days** by default. This allows miners to add more data to your request. If you'd like to stop a task early, click **“Cancel.”**
**If you cancel:** You can still preview or download any completed datasets. If you'd like to get a new data for the same topics, you should launch a new task.