We incentivize the prediction of future events. We currently restrict the prediction space to binary future events listed on Polymarket. We will expand soon to new markets and providers. We are focused on judgemental forecasting rather than statistical forecasting. We hence expect the models used by miners to be LLMs.
Miners submit their predictions to validators. Each prediction has to be done early enough before the event underlying the prediction settles. Once the event settles, the validators that received the prediction score the miner.
Making predictions is a hard task that requires cross-domain knowledge and intuition. It is often limited in explanatory reasoning and domain-specific (the expert in predicting election results will differ from the one predicting the progress in rocket-engine technology) ([1]). At the same time it is fundamental to human society, from geopolitics to economics.
LLMs approach or surpass human forecasting abilities. They near on average the crowd prediction on prediction market events ([1]), and surpass humans in predicting neuroscience results ([2]). They are also shown to be calibrated with their predictions i.e confident when right. Through their generalization capabilities and unbounded information processing, LLMs have the potential to automate the prediction process or complement humans.
The value of the subnet first relies in the improvement of the efficiency of prediction markets. This value can be extracted by validators through arbitrage. The validators may obtain a better knowledge of the probability of an event settling and communicate this information to a prediction market by opening a position.
The first applications built on top of our subnet could be related to prediction markets. A trader could query our market to obtain the most up to date and relevant predictions to their portfolio based on the current news landscape (LLMs would be constantly ingressing the most up to date and relevant news articles). They could then readjust their positions accordingly or trade directly on this information.
In the long term, a validator could provide paid economic forecasts or more generally the output of any forward-looking task addressed to an LLM ([2]). A customer might then provide a series of paid sub-queries related to the information they aim at retrieving.
Miners compete by sending to the validators a dictionary where the key is a Polymarket condition id to an event
A reference providing a baseline miner strategy is the article "Approaching Human Level Forecasting with Langage Models" ([1]). The authors fine-tune an LLM to generate predictions on binary events (including the ones listed on Polymarket) which nears the performance of human forecasters when submitting a forecast for each prediction, and which beats human forecasters in a setting where the LLM can choose to give a prediction or not based on its confidence.
Validators record the miners' predictions and score them once the Polymarket events settles. At each event settlement, a score is added to the moving average of the miner's score. This simple model ensures that all validators score the miners at roughly the same time. Importantly, we implement a cutoff for the submission time of a prediction, currently set at 24 hours. This means that miners must submit their prediction for a given Polymarket event 24 hours before the settlement time.
We will launch our subnet with model 1 and then move to model 2.
Denote by
The validators directly use a quadratic scoring rule on the miners' predictions. If the miner predicted that
In this model, we discard the moving average update and validators record the scores they obtained at settlement time. The validators then all update the aggregated scores of miners at an agreed upon time.
We implement a sequentially shared quadratic scoring rule. This allows us to score
The aggregated score of a miner that a validator sends to the blockchain is the following:
We give miners a score of
In the first iteration of the model, instead of paying the miner for their delta to the previous prediction, we will pay them for their delta to the Polymarket probability at the submission time i.e
See here for a discussion of our mechanism.
We first aim at adjusting the scoring rule by updating to the model 1 described above. We will likely implement several other updates in order to make the mechanism more robust. One of them could be a commit-reveal step for the predictions submitted by miners. Some updates may be due to experimental data.
We would also possibly like to make the prediction framework more LLM specific and create mechanisms that explicitely generate data for the fine-tuning of prediction focused LLMs.
We plan to extend the set of predicted events to other prediction markets and event providers (Metacalculus, Azuro). Our goal is to obtain a continuous feed of organic events by using e.g Reuters' API or WSJ headlines.
Regarding instructions and requirements, see here for validators and here for miners.
A detailed explanation of how to set up a wallet can be found here. We also provide some indications here.
| Reference ID | Author(s) | Year | Title |
|---|---|---|---|
| 1 | Halawi and al. | 2024 | Approaching Human Level Forecasting with Langage Models |
| 2 | Luo and al. | 2024 | LLM surpass human experts in predicting neuroscience results |
This repository is licensed under the MIT License.
# The MIT License (MIT)
# Copyright © 2023 Yuma Rao
# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
# documentation files (the “Software”), to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in all copies or substantial portions of
# the Software.
# THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO
# THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.