ema, ci, api, json breakdown

pages/devs/consumers/_meta.json

@@ -1,4 +1,5 @@
 {
+ "allora-api-endpoint": "Allora API Endpoint",
  "onchain-query-existing-data": "How to Query Topic Data On-Chain",
  "consumer-contracts": "Consumer Contracts",
  "existing-consumers": "Existing Consumers",

pages/devs/consumers/allora-api-endpoint.mdx

@@ -0,0 +1,135 @@
+# Allora API: How to Query Data of Existing Topics
+
+The **Allora API** provides an interface to query real-time on-chain data of the latest inferences made by workers. Here's an explanation of how it works using the example endpoint:
+
+**Generic**: `https://allora-api.testnet.allora.network/emissions/{version_number}/latest_network_inferences/{topic_id}`
+
+**Example**: `https://allora-api.testnet.allora.network/emissions/v4/latest_network_inferences/1`
+
+Where:
+- "v4" represents the latest network version number
+- "1" represents the topic ID
+
+Sample Response:
+
+```json
+{
+ "network_inferences": {
+ "topic_id": "1",
+ "reputer_request_nonce": null,
+ "reputer": "",
+ "extra_data": null,
+ "combined_value": "2605.533879185080648394998043723508",
+ "inferer_values": [
+ {
+ "worker": "allo102ksu3kx57w0mrhkg37kvymmk2lgxqcan6u7yn",
+ "value": "2611.01109296"
+ },
+ {
+ "worker": "allo10q6hm2yae8slpvvgmxqrcasa30gu5qfysp4wkz",
+ "value": "2661.505295679922"
+ }
+ ],
+ "forecaster_values": [
+ {
+ "worker": "allo1za8r9v0st4ntfyeka23qs5wvd7mvsnzhztupk0",
+ "value": "2610.160000000000000000000000000000"
+ }
+ ],
+ "naive_value": "2605.533879185080648394998043723508",
+ "one_out_inferer_values": [
+ {
+ "worker": "allo102ksu3kx57w0mrhkg37kvymmk2lgxqcan6u7yn",
+ "value": "2570.859434973857748387096774193548"
+ },
+ {
+ "worker": "allo10q6hm2yae8slpvvgmxqrcasa30gu5qfysp4wkz",
+ "value": "2569.230589724828006451612903225806"
+ }
+ ],
+ "one_out_forecaster_values": [],
+ "one_in_forecaster_values": [],
+ "one_out_inferer_forecaster_values": []
+ },
+ "inferer_weights": [
+ {
+ "worker": "allo102ksu3kx57w0mrhkg37kvymmk2lgxqcan6u7yn",
+ "weight": "0.0002191899319465528034563075461505151"
+ },
+ {
+ "worker": "allo10q6hm2yae8slpvvgmxqrcasa30gu5qfysp4wkz",
+ "weight": "0.0002191899319465528034563075461505151"
+ }
+ ],
+ "forecaster_weights": [
+ {
+ "worker": "allo1za8r9v0st4ntfyeka23qs5wvd7mvsnzhztupk0",
+ "weight": "0.1444137067859501612197657742201029"
+ }
+ ],
+ "forecast_implied_inferences": [
+ {
+ "worker": "allo1za8r9v0st4ntfyeka23qs5wvd7mvsnzhztupk0",
+ "value": "2610.160000000000000000000000000000"
+ }
+ ],
+ "inference_block_height": "1349577",
+ "loss_block_height": "0",
+ "confidence_interval_raw_percentiles": [
+ "2.28",
+ "15.87",
+ "50",
+ "84.13",
+ "97.72"
+ ],
+ "confidence_interval_values": [
+ "2492.1675618299669694181830608795809",
+ "2543.9249467952655499150756965734158",
+ "2611.033130351115229549044053766836",
+ "2662.29523395638446190095015123294396",
+ "2682.827040221238"
+ ]
+}
+```
+
+## Breaking Down the Response
+
+Below is an explanation of important sub-objects displayed in the JSON output:
+
+### `topic_id`
+In this case, "1" represents the topic being queried. [Topics](/devs/topic-creators/how-to-create-topic) define the context and rules for a particular inference.
+
+### `naive_value`
+The **naive value** omits all forecast-implied inferences from the weighted average by setting their weights to zero. The naive network inference is used to quantify the contribution of the
+forecasting task to the network accuracy, which in turn sets the reward distribution between the inference and forecasting tasks.
+
+### `combined_value`
+The **combined value** is an optimized inference that represents a collective intelligence approach, taking both naive submissions and forecast data into account.
+
+### `inferer_values`
+Workers in the network submit their inferences, each represented by an `allo` address. For example:
+
+```json
+{
+ "worker": "allo102ksu3kx57w0mrhkg37kvymmk2lgxqcan6u7yn",
+ "value": "2611.01109296"
+}
+```
+
+Each worker submits a value based on their own models. These individual submissions contribute to both the naive and combined values. The combined value gives higher weighting to more reliable workers, based on performance or other criteria.
+
+### `one_out_inferer_values`
+These values simulate removing a single worker from the pool to see how the overall inference changes. This is a technique used to evaluate the impact of individual inferences on the combined result. 
+
+### `forecast_implied_inferences`
+The [Forecast-Implied Inference](/home/layers/forecast-synthesis/synthesis#forecast-implied-inferences) uses forecasted losses and worker inferences to produce a predicted value where each prediction is weighted based on how accurately the forecasters predicted losses in previous time steps, or epochs.
+
+### `inference_block_height`
+The specific chain block that the inference data was generated
+
+### `confidence_interval_raw_percentiles`
+Fixed percentiles that are used to generate [confidence intervals](/home/confidence-intervals)
+
+### `confidence_interval_values`
+Confidence intervals](/home/confidence-intervals) show the predicted range of outcomes based on worker inferences.
+

pages/devs/consumers/onchain-query-existing-data.mdx

@@ -2,7 +2,7 @@
 
 To query the latest inferences attributed to each worker for a specific topic:
 
-**Active Topics:**
+## Active Topics
 
 ```
 allorad query emissions active-topics [pagination] --node <RPC_URL>
@@ -12,15 +12,14 @@ allorad query emissions active-topics [pagination] --node <RPC_URL>
 
 **Example:** `'{"limit":10}'`
 
-**Latest Inferences for a Topic:**
+## Latest Inferences for a Topic
 
 ```
 allorad query emissions latest-network-inference [topic_id] --node <RPC_URL>
 ```
 
-`[topic_id]` is the integer value of the topic
-
-**Example:** `1`
+- `[topic_id]` is the integer value of the topic
+ - **Example:** `1`
 
 Visit the [`allorad` reference section](/devs/reference/allorad#query-functions) for the complete list of available on-chain query commands.
 

pages/devs/workers/_meta.json

@@ -1,5 +1,6 @@
 {
  "requirements": "System Requirements",
  "deploy-worker": "Build/Deploy a Worker",
- "walkthroughs": "Walkthroughs"
+ "walkthroughs": "Walkthroughs",
+ "query-ema-score": "Query EMA Score"
 }

pages/devs/workers/query-ema-score.mdx

@@ -0,0 +1,36 @@
+import { Callout } from 'nextra/components'
+
+# How to Query Worker EMA Scores
+
+## What is an EMA Score?
+
+The EMA score (Exponential Moving Average) reflects a worker's performance over time for a given topic, balancing recent and past achievements, and helps determine whether a participant stays in the **active set** (eligible for rewards) or remains 
+in the **passive set**. Active participants have their EMA score updated based on their current performance, which influences their ongoing 
+eligibility for rewards. In contrast, inactive participants, who do not contribute during a given epoch, receive an adjusted score using a 
+"dummy" value, which determines whether they can re-enter the active set in future epochs and qualify for rewards. This process ensures 
+fairness while allowing inactive participants the chance to rejoin the active set based on their historical performance.
+
+## Query EMA Score for a Specific Worker
+
+To query the EMA score for a specific worker (identified by the worker's allo address), run:
+
+```bash
+allorad q emissions inferer-score-ema [topic_id] [worker_address] --node https://allora-rpc.testnet.allora.network/
+```
+
+- Replace `[topic_id]` and `[worker_address]` with your specific details.
+
+## Query the Lowest Worker in the Active Set's EMA Score 
+
+To query the lowest EMA score for a worker in the Active Set, run:
+
+```bash
+allorad q emissions current-lowest-inferer-score [topic_id]
+```
+
+- Replace `[topic_id]` with your specific details.
+
+<Callout>
+To determine if your worker is in the active set and eligible for rewards, query your worker's EMA score and the lowest worker in the active set's EMA score and compare them. 
+If your worker's EMA score for a specific topic is higher than the EMA score of the lowest worker in the active set, your worker is in the active set for that topic.
+</Callout>

pages/home/_meta.json

@@ -5,6 +5,7 @@
  "participants": "Participants",
  "layers": "Layers of the Network",
  "tokenomics": "Tokenomics",
+ "confidence-intervals": "Confidence Intervals",
  "release-notes": "Release Notes",
  "whitepaper": {
  "title": "Whitepaper",

pages/home/confidence-intervals.mdx

@@ -0,0 +1,32 @@
+# Confidence Intervals
+
+Confidence intervals (CIs) help users understand how much variation there is among the predictions made by different workers. Rather than just giving one prediction number, the network also provides a **range**—a window that tells you how close or far apart the workers' predictions are. This helps gauge the **certainty** of the prediction.
+
+## How Confidence Intervals are Generated
+
+Confidence intervals are not simply derived from the raw predictions of workers. The Allora network takes a weighted approach, accounting for the reliability of individual workers to ensure that less accurate predictions do not unduly influence the final result. Here's how the process works:
+
+1. **Percentiles as Key Indicators**: 
+ The network uses specific percentiles to outline confidence intervals, mimicking the 1σ and 2σ limits found in a normal distribution. These include:
+ - **2.28% and 97.72%**: Representing the extreme ends of the spectrum, these percentiles capture nearly all predictions including outliers.
+ - **15.87% and 84.13%**: These percentiles represent a more central range, focusing on the core set of inferences with the highest levels of confidence.
+
+>While worker inferences may not follow a perfect Gaussian (bell curve) distribution, these percentiles provide a structured way to describe the uncertainty inherent in collective predictions.
+
+2. **Weighted Worker Predictions**: 
+ Not all worker inferences are treated equally. Workers with a proven track record of accuracy are assigned higher **weights**, which amplify their influence on the final confidence interval. Conversely, less reliable workers are given lower weights to ensure that their predictions don’t artificially broaden the confidence interval.
+
+3. **Building the Distribution**: 
+ Once the inferences are submitted, they are sorted from lowest to highest, and each worker’s weight is accumulated to create a **cumulative distribution function** (CDF), which allows the network to map how predictions are distributed across the worker base.
+
+4. **Determining the Confidence Interval**: 
+ Using the CDF, the network calculates the prediction values that correspond to the chosen percentiles (e.g., 2.28%, 15.87%, etc.). To increase precision, the network employs **interpolation**—a method of estimating values that fall between two known data points in the sorted list. This ensures that the confidence intervals reflect the actual spread of inferences rather than approximate ranges.
+
+## What Does This Mean for You?
+
+When you see a confidence interval in Allora, it gives you a sense of how **confident** the network is about the prediction. 
+
+- A **narrow** confidence interval means the workers mostly agree on the result.
+- A **wider** confidence interval suggests that the workers’ predictions vary more, indicating more uncertainty in the result.
+
+By factoring in these confidence intervals, users can make more informed decisions about the consensus surrounding inferences provided for a given topic.

pages/home/release-notes.mdx

@@ -41,6 +41,44 @@ The Allora Network v0.5.0 is now live! This version introduces several major upd
 #### Idempotent Payload Submission
 - Simplified submission conditions for inference payloads. Duplicate submissions are now handled in an idempotent manner, preventing them from affecting the system’s behavior or causing erroneous calculations.
 
+## v0.4.0
+
+**Release Date**: September 2024
+
+This version focuses on implementing key fixes from the June 2024 Sherlock.xyz audit, enhancing active topic management, and refining the scoring system. Below are the critical updates, new features, and fixes included in this release.
+
+### Key Features and Improvements
+
+#### Scalable Management of Active Topics
+- **Active Topic Queries**: Introduced scalable solutions for managing active topics with new queries such as `GetActiveTopicsAtBlock` and `GetNextChurningBlockByTopicId`. These additions enhance the network's ability to efficiently retrieve topic statuses at specific blocks and predict future churn events for each topic.
+
+#### Exponential Moving Average for Scores
+- **Smoother Scoring**: Transitioned to using an **Exponential Moving Average (EMA)** for score calculations, replacing the previous instantaneous score values from each epoch. This change smooths out score fluctuations and ensures a more representative scoring system over time.
+
+### Removed
+
+#### Deprecated Unpartitioned Active Topic Queries
+- As part of the new scalable topic management system, the outdated `GetActiveTopics` query and paginated versions were removed. This helps streamline how active topics are stored and queried in the system.
+
+### Bug Fixes
+
+#### Reward Conversion to cosmosInt
+- Implemented a check to prevent **zero-rewards** after conversion to `cosmosInt`, ensuring rewards are correctly handled and distributed.
+
+#### InsertPayload Error Handling
+- Improved error handling in the `InsertPayload` function, along with enhanced testing for error scenarios. This strengthens payload processing and prevents errors from affecting the overall system.
+
+#### Reputer Window Limit Fix
+- Fixed the **Reputer window upper limit** to ensure that reputation calculations do not exceed the intended thresholds.
+
+#### Worker Nonce Window Timing
+- Resolved an issue where the **worker nonce window** was prematurely closing as soon as it opened, preventing proper timing of nonce submissions.
+
+### Security Enhancements
+
+#### Signature Verification for Payloads
+- Added checks to ensure that signatures on **Worker or Reputer Payloads** match the corresponding `Inferer`, `Forecaster`, or `Reputer` inside the bundle. This prevents unauthorized manipulation of payloads and strengthens overall network security.
+
 ## v0.3.0
 
 **Release Date**: August 2024