Merge pull request #2255 from opentensor/release/7.4.0

Release 7.4.0

Author: ibraheem-abe

.github/auto_assign.yml

@@ -0,0 +1,7 @@
+addReviewers: true
+
+# A list of team slugs to add as assignees
+reviewers:
+  - opentensor/cortex
+
+numberOfReviewers: 0

.github/workflows/auto-assign.yml

@@ -0,0 +1,15 @@
+name: Auto Assign Cortex to Pull Requests
+
+on:
+  pull_request:
+    types: [opened, reopened]
+
+jobs:
+  auto-assign:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Auto-assign Cortex Team
+      uses: kentaro-m/auto-assign-action@v1.2.4
+      with:
+        repo-token: "${{ secrets.GITHUB_TOKEN }}"
+        configuration-path: .github/auto_assign.yml

.github/workflows/e2e-subtensor-tests.yaml

@@ -46,6 +46,7 @@ jobs:
   run:
     needs: find-tests
     runs-on: SubtensorCI
+    timeout-minutes: 45
     strategy:
       fail-fast: false  # Allow other matrix jobs to run even if this job fails
       max-parallel: 8  # Set the maximum number of parallel jobs

CHANGELOG.md

@@ -1,5 +1,83 @@
 # Changelog
 
+## 7.4.0 /2024-08-29
+
+## What's Changed
+* [Fix] Allow unstake below network min by @camfairchild in https://github.com/opentensor/bittensor/pull/2016
+* Tests/e2e tests staging by @open-junius in https://github.com/opentensor/bittensor/pull/1943
+* Chore: Backmerge 7.2 by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2020
+* Fix broken tests and Enforce BTCLI usage by @opendansor in https://github.com/opentensor/bittensor/pull/2027
+* Add time delay to faucet by @opendansor in https://github.com/opentensor/bittensor/pull/2030
+* Skip faucet test by @opendansor in https://github.com/opentensor/bittensor/pull/2031
+* Adds normalization for alpha hyperparams by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2035
+* Revert info logging in processing response by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2043
+* Pin numpy version to 1.26.4 in prod.txt by @rajkaramchedu in https://github.com/opentensor/bittensor/pull/2045
+* Test hot key Swap by @opendansor in https://github.com/opentensor/bittensor/pull/2044
+* Do not run Circle-CI on drafts by @thewhaleking in https://github.com/opentensor/bittensor/pull/1959
+* Enhancement: Detailed nonce information in-case of failures by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2050
+* fix bittensor not installing under Python 3.13 by @mjurbanski-reef in https://github.com/opentensor/bittensor/pull/2053
+* Enable Faucet Test by @opendansor in https://github.com/opentensor/bittensor/pull/2056
+* Add back BT_SUBTENSOR_CHAIN_ENDPOINT env variable by @bradleytf in https://github.com/opentensor/bittensor/pull/2034
+* Fix: Logging configs not being set by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2065
+* Feature/gus/liquid alpha params by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2012
+* Test Emissions E2E by @opendansor in https://github.com/opentensor/bittensor/pull/2036
+* Prevent e2e draft by @opendansor in https://github.com/opentensor/bittensor/pull/2072
+* Fix e2e to only run when PR is ready for review by @opendansor in https://github.com/opentensor/bittensor/pull/2077
+* Fix Faucet and fastblocks interaction by @opendansor in https://github.com/opentensor/bittensor/pull/2083
+* Float normalization for child hotkeys by @opendansor in https://github.com/opentensor/bittensor/pull/2093
+* Fix e2e test hanging by @open-junius in https://github.com/opentensor/bittensor/pull/2118
+* Fixes leaked semaphores by @thewhaleking in https://github.com/opentensor/bittensor/pull/2125
+* Backmerge master -> staging by @roman-opentensor in https://github.com/opentensor/bittensor/pull/2136
+* fix: coldkeypub usage instead of coldkey for arbitration_stats by @Rapiiidooo in https://github.com/opentensor/bittensor/pull/2132
+* Removes extra no_prompts in commands by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2140
+* Adds timeout for e2e tests by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2141
+* fix: updates test_axon verify body async tests by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2142
+* test: fix mocksubtensor query previous blocks by @timabilov in https://github.com/opentensor/bittensor/pull/2139
+* Adds E2E for Metagraph command by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2143
+* feat: Enhance dendrite error messaging by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2117
+* Adds E2E Tests for wallet creation commands by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2145
+* [Ledger Integration] [Feature] bump pysub to 1.7.9+ by @camfairchild in https://github.com/opentensor/bittensor/pull/2156
+* Ruff complains about an extra line by @roman-opentensor in https://github.com/opentensor/bittensor/pull/2158
+* support Wallet names with hyphens when passing password through ENV vars by @mjurbanski-reef in https://github.com/opentensor/bittensor/pull/1949
+* Fix naming convention of swap hotkey test by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2162
+* Adds E2E test for wallet regenerations + fixes input bug for regen hotkey by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2149
+* Backmerge Master -> Staging (7.4) by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2170
+* ci: auto assigns cortex to opened PRs by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2184
+* CI/E2E test improvements by @mvds00 in https://github.com/opentensor/bittensor/pull/2168
+* Fix multiprocessing POW errors and No Torch logging errors by @thewhaleking in https://github.com/opentensor/bittensor/pull/2186
+* ci: update reviewers by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2189
+* Adds updated type in timeouts dendrite by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2196
+* Bumps setuptools ~=70.0.0 by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2150
+* Bump black from 23.7.0 to 24.3.0 in /requirements by @dependabot in https://github.com/opentensor/bittensor/pull/2197
+* btlogging/loggingmachine.py: Fix bw compat API. by @mvds00 in https://github.com/opentensor/bittensor/pull/2155
+* Check for participation before nomination call by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2193
+* test: subnet list e2e by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2198
+* ensure msg is str in _concat_msg by @thewhaleking in https://github.com/opentensor/bittensor/pull/2200
+* Fixes tests depending on explicit line numbers by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2211
+* Merge streaming fix to staging by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2183
+* Multiple bittensor versions e2e workflow by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2212
+* Changes name of workflow file by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2213
+* Enhances e2e tests to contain assertions & logging  by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2192
+* Security fix: Bumps ansible and certifi  by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2214
+* Wallet List Command e2e test by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2207
+* fix Synapse base performance (more than 10x speed up) by @mjurbanski-reef in https://github.com/opentensor/bittensor/pull/2161
+* Child Hotkeys by @opendansor in https://github.com/opentensor/bittensor/pull/2071
+* Improve child hotkeys QOL by @opendansor in https://github.com/opentensor/bittensor/pull/2225
+* Child hotkeys handle excess normalization by @opendansor in https://github.com/opentensor/bittensor/pull/2229
+* Fixes chain compilation timeouts by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2238
+* Update Child Hotkey commands by @opendansor in https://github.com/opentensor/bittensor/pull/2245
+* feat: return error message instead of raising exception by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2244
+* Backmerge master to staging (7.3.1) by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2254
+
+## New Contributors
+* @bradleytf made their first contribution in https://github.com/opentensor/bittensor/pull/2034
+* @Rapiiidooo made their first contribution in https://github.com/opentensor/bittensor/pull/2132
+* @timabilov made their first contribution in https://github.com/opentensor/bittensor/pull/2139
+* @mvds00 made their first contribution in https://github.com/opentensor/bittensor/pull/2168
+* @dependabot made their first contribution in https://github.com/opentensor/bittensor/pull/2197
+
+**Full Changelog**: https://github.com/opentensor/bittensor/compare/v7.3.1...v7.4.0
+
 ## 7.3.1 / 2024-08-19
 
 ## What's Changed

VERSION

@@ -1 +1 @@
-7.3.1
+7.4.0

bittensor/__init__.py

@@ -40,7 +40,7 @@
 
 
 # Bittensor code and protocol version.
-__version__ = "7.3.1"
+__version__ = "7.4.0"
 
 _version_split = __version__.split(".")
 __version_info__ = tuple(int(part) for part in _version_split)
@@ -125,7 +125,14 @@ def debug(on: bool = True):
 # Needs to use wss://
 __bellagene_entrypoint__ = "wss://parachain.opentensor.ai:443"
 
-__local_entrypoint__ = "ws://127.0.0.1:9944"
+
+if (
+    BT_SUBTENSOR_CHAIN_ENDPOINT := os.getenv("BT_SUBTENSOR_CHAIN_ENDPOINT")
+) is not None:
+    __local_entrypoint__ = BT_SUBTENSOR_CHAIN_ENDPOINT
+else:
+    __local_entrypoint__ = "ws://127.0.0.1:9944"
+
 
 __tao_symbol__: str = chr(0x03C4)
 
@@ -200,19 +207,6 @@ def debug(on: bool = True):
                 },
             },
         },
-        "ValidatorIPRuntimeApi": {
-            "methods": {
-                "get_associated_validator_ip_info_for_subnet": {
-                    "params": [
-                        {
-                            "name": "netuid",
-                            "type": "u16",
-                        },
-                    ],
-                    "type": "Vec<u8>",
-                },
-            },
-        },
         "SubnetInfoRuntimeApi": {
             "methods": {
                 "get_subnet_hyperparams": {
@@ -318,7 +312,6 @@ def debug(on: bool = True):
     strtobool,
     strtobool_with_default,
     get_explorer_root_url_by_network_from_map,
-    get_explorer_root_url_by_network_from_map,
     get_explorer_url_for_network,
     ss58_address_to_bytes,
     U16_NORMALIZED_FLOAT,

bittensor/axon.py

@@ -31,11 +31,12 @@
 import traceback
 import typing
 import uuid
+import warnings
 from inspect import signature, Signature, Parameter
 from typing import List, Optional, Tuple, Callable, Any, Dict, Awaitable
 
 import uvicorn
-from fastapi import FastAPI, APIRouter, Depends
+from fastapi import APIRouter, Depends, FastAPI
 from fastapi.responses import JSONResponse
 from fastapi.routing import serialize_response
 from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
@@ -44,18 +45,19 @@
 from substrateinterface import Keypair
 
 import bittensor
+from bittensor.utils.axon_utils import allowed_nonce_window_ns, calculate_diff_seconds
+from bittensor.constants import V_7_2_0
 from bittensor.errors import (
+    BlacklistedException,
     InvalidRequestNameError,
-    SynapseDendriteNoneException,
-    SynapseParsingError,
-    UnknownSynapseError,
     NotVerifiedException,
-    BlacklistedException,
-    PriorityException,
     PostProcessException,
+    PriorityException,
+    SynapseDendriteNoneException,
     SynapseException,
+    SynapseParsingError,
+    UnknownSynapseError,
 )
-from bittensor.constants import ALLOWED_DELTA, V_7_2_0
 from bittensor.threadpool import PriorityThreadPoolExecutor
 from bittensor.utils import networking
 
@@ -484,17 +486,50 @@ def verify_custom(synapse: MyCustomSynapse):
 
         async def endpoint(*args, **kwargs):
             start_time = time.time()
-            response_synapse = forward_fn(*args, **kwargs)
-            if isinstance(response_synapse, Awaitable):
-                response_synapse = await response_synapse
-            return await self.middleware_cls.synapse_to_response(
-                synapse=response_synapse, start_time=start_time
-            )
+            response = forward_fn(*args, **kwargs)
+            if isinstance(response, Awaitable):
+                response = await response
+            if isinstance(response, bittensor.Synapse):
+                return await self.middleware_cls.synapse_to_response(
+                    synapse=response, start_time=start_time
+                )
+            else:
+                response_synapse = getattr(response, "synapse", None)
+                if response_synapse is None:
+                    warnings.warn(
+                        "The response synapse is None. The input synapse will be used as the response synapse. "
+                        "Reliance on forward_fn modifying input synapse as a side-effects is deprecated. "
+                        "Explicitly set `synapse` on response object instead.",
+                        DeprecationWarning,
+                    )
+                    # Replace with `return response` in next major version
+                    response_synapse = args[0]
+
+                return await self.middleware_cls.synapse_to_response(
+                    synapse=response_synapse,
+                    start_time=start_time,
+                    response_override=response,
+                )
+
+        return_annotation = forward_sig.return_annotation
+
+        if isinstance(return_annotation, type) and issubclass(
+            return_annotation, bittensor.Synapse
+        ):
+            if issubclass(
+                return_annotation,
+                bittensor.StreamingSynapse,
+            ):
+                warnings.warn(
+                    "The forward_fn return annotation is a subclass of bittensor.StreamingSynapse. "
+                    "Most likely the correct return annotation would be BTStreamingResponse."
+                )
+            else:
+                return_annotation = JSONResponse
 
-        # replace the endpoint signature, but set return annotation to JSONResponse
         endpoint.__signature__ = Signature(  # type: ignore
             parameters=list(forward_sig.parameters.values()),
-            return_annotation=JSONResponse,
+            return_annotation=return_annotation,
         )
 
         # Add the endpoint to the router, making it available on both GET and POST methods
@@ -847,6 +882,8 @@ async def default_verify(self, synapse: bittensor.Synapse):
                 The method checks for increasing nonce values, which is a vital
                 step in preventing replay attacks. A replay attack involves an adversary reusing or
                 delaying the transmission of a valid data transmission to deceive the receiver.
+                The first time a nonce is seen, it is checked for freshness by ensuring it is
+                within an acceptable delta time range.
 
             Authenticity and Integrity Checks
                 By verifying that the message's digital signature matches
@@ -893,33 +930,43 @@ async def default_verify(self, synapse: bittensor.Synapse):
             if synapse.dendrite.nonce is None:
                 raise Exception("Missing Nonce")
 
-            # If we don't have a nonce stored, ensure that the nonce falls within
-            # a reasonable delta.
-
+            # Newer nonce structure post v7.2
             if (
                 synapse.dendrite.version is not None
                 and synapse.dendrite.version >= V_7_2_0
             ):
                 # If we don't have a nonce stored, ensure that the nonce falls within
                 # a reasonable delta.
+                current_time_ns = time.time_ns()
+                allowed_window_ns = allowed_nonce_window_ns(
+                    current_time_ns, synapse.timeout
+                )
+
                 if (
                     self.nonces.get(endpoint_key) is None
-                    and synapse.dendrite.nonce
-                    <= time.time_ns() - ALLOWED_DELTA - (synapse.timeout or 0)
+                    and synapse.dendrite.nonce <= allowed_window_ns
                 ):
-                    raise Exception("Nonce is too old")
+                    diff_seconds, allowed_delta_seconds = calculate_diff_seconds(
+                        current_time_ns, synapse.timeout, synapse.dendrite.nonce
+                    )
+                    raise Exception(
+                        f"Nonce is too old: acceptable delta is {allowed_delta_seconds:.2f} seconds but request was {diff_seconds:.2f} seconds old"
+                    )
+
+                # If a nonce is stored, ensure the new nonce
+                # is greater than the previous nonce
                 if (
                     self.nonces.get(endpoint_key) is not None
                     and synapse.dendrite.nonce <= self.nonces[endpoint_key]
                 ):
-                    raise Exception("Nonce is too old")
+                    raise Exception("Nonce is too old, a newer one was last processed")
+            # Older nonce structure pre v7.2
             else:
                 if (
-                    endpoint_key in self.nonces.keys()
-                    and self.nonces[endpoint_key] is not None
+                    self.nonces.get(endpoint_key) is not None
                     and synapse.dendrite.nonce <= self.nonces[endpoint_key]
                 ):
-                    raise Exception("Nonce is too small")
+                    raise Exception("Nonce is too old, a newer one was last processed")
 
             if not keypair.verify(message, synapse.dendrite.signature):
                 raise Exception(
@@ -952,7 +999,7 @@ def log_and_handle_error(
     exception: Exception,
     status_code: typing.Optional[int] = None,
     start_time: typing.Optional[float] = None,
-):
+) -> bittensor.Synapse:
     if isinstance(exception, SynapseException):
         synapse = exception.synapse or synapse
 
@@ -1420,14 +1467,21 @@ async def run(
 
     @classmethod
     async def synapse_to_response(
-        cls, synapse: bittensor.Synapse, start_time: float
-    ) -> JSONResponse:
+        cls,
+        synapse: bittensor.Synapse,
+        start_time: float,
+        *,
+        response_override: Optional[Response] = None,
+    ) -> Response:
         """
         Converts the Synapse object into a JSON response with HTTP headers.
 
         Args:
-            synapse (bittensor.Synapse): The Synapse object representing the request.
-            start_time (float): The timestamp when the request processing started.
+            synapse: The Synapse object representing the request.
+            start_time: The timestamp when the request processing started.
+            response_override:
+                Instead of serializing the synapse, mutate the provided response object.
+                This is only really useful for StreamingSynapse responses.
 
         Returns:
             Response: The final HTTP response, with updated headers, ready to be sent back to the client.
@@ -1446,11 +1500,14 @@ async def synapse_to_response(
 
         synapse.axon.process_time = time.time() - start_time
 
-        serialized_synapse = await serialize_response(response_content=synapse)
-        response = JSONResponse(
-            status_code=synapse.axon.status_code,
-            content=serialized_synapse,
-        )
+        if response_override:
+            response = response_override
+        else:
+            serialized_synapse = await serialize_response(response_content=synapse)
+            response = JSONResponse(
+                status_code=synapse.axon.status_code,
+                content=serialized_synapse,
+            )
 
         try:
             updated_headers = synapse.to_headers()