Merge pull request #2302 from opentensor/backmerge/release-7.4.0

Backmerge/release 7.4.0

Author: ibraheem-abe

CHANGELOG.md

@@ -1,5 +1,83 @@
 # Changelog
 
+## 7.4.0 /2024-08-29
+
+## What's Changed
+* [Fix] Allow unstake below network min by @camfairchild in https://github.com/opentensor/bittensor/pull/2016
+* Tests/e2e tests staging by @open-junius in https://github.com/opentensor/bittensor/pull/1943
+* Chore: Backmerge 7.2 by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2020
+* Fix broken tests and Enforce BTCLI usage by @opendansor in https://github.com/opentensor/bittensor/pull/2027
+* Add time delay to faucet by @opendansor in https://github.com/opentensor/bittensor/pull/2030
+* Skip faucet test by @opendansor in https://github.com/opentensor/bittensor/pull/2031
+* Adds normalization for alpha hyperparams by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2035
+* Revert info logging in processing response by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2043
+* Pin numpy version to 1.26.4 in prod.txt by @rajkaramchedu in https://github.com/opentensor/bittensor/pull/2045
+* Test hot key Swap by @opendansor in https://github.com/opentensor/bittensor/pull/2044
+* Do not run Circle-CI on drafts by @thewhaleking in https://github.com/opentensor/bittensor/pull/1959
+* Enhancement: Detailed nonce information in-case of failures by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2050
+* fix bittensor not installing under Python 3.13 by @mjurbanski-reef in https://github.com/opentensor/bittensor/pull/2053
+* Enable Faucet Test by @opendansor in https://github.com/opentensor/bittensor/pull/2056
+* Add back BT_SUBTENSOR_CHAIN_ENDPOINT env variable by @bradleytf in https://github.com/opentensor/bittensor/pull/2034
+* Fix: Logging configs not being set by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2065
+* Feature/gus/liquid alpha params by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2012
+* Test Emissions E2E by @opendansor in https://github.com/opentensor/bittensor/pull/2036
+* Prevent e2e draft by @opendansor in https://github.com/opentensor/bittensor/pull/2072
+* Fix e2e to only run when PR is ready for review by @opendansor in https://github.com/opentensor/bittensor/pull/2077
+* Fix Faucet and fastblocks interaction by @opendansor in https://github.com/opentensor/bittensor/pull/2083
+* Float normalization for child hotkeys by @opendansor in https://github.com/opentensor/bittensor/pull/2093
+* Fix e2e test hanging by @open-junius in https://github.com/opentensor/bittensor/pull/2118
+* Fixes leaked semaphores by @thewhaleking in https://github.com/opentensor/bittensor/pull/2125
+* Backmerge master -> staging by @roman-opentensor in https://github.com/opentensor/bittensor/pull/2136
+* fix: coldkeypub usage instead of coldkey for arbitration_stats by @Rapiiidooo in https://github.com/opentensor/bittensor/pull/2132
+* Removes extra no_prompts in commands by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2140
+* Adds timeout for e2e tests by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2141
+* fix: updates test_axon verify body async tests by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2142
+* test: fix mocksubtensor query previous blocks by @timabilov in https://github.com/opentensor/bittensor/pull/2139
+* Adds E2E for Metagraph command by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2143
+* feat: Enhance dendrite error messaging by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2117
+* Adds E2E Tests for wallet creation commands by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2145
+* [Ledger Integration] [Feature] bump pysub to 1.7.9+ by @camfairchild in https://github.com/opentensor/bittensor/pull/2156
+* Ruff complains about an extra line by @roman-opentensor in https://github.com/opentensor/bittensor/pull/2158
+* support Wallet names with hyphens when passing password through ENV vars by @mjurbanski-reef in https://github.com/opentensor/bittensor/pull/1949
+* Fix naming convention of swap hotkey test by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2162
+* Adds E2E test for wallet regenerations + fixes input bug for regen hotkey by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2149
+* Backmerge Master -> Staging (7.4) by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2170
+* ci: auto assigns cortex to opened PRs by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2184
+* CI/E2E test improvements by @mvds00 in https://github.com/opentensor/bittensor/pull/2168
+* Fix multiprocessing POW errors and No Torch logging errors by @thewhaleking in https://github.com/opentensor/bittensor/pull/2186
+* ci: update reviewers by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2189
+* Adds updated type in timeouts dendrite by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2196
+* Bumps setuptools ~=70.0.0 by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2150
+* Bump black from 23.7.0 to 24.3.0 in /requirements by @dependabot in https://github.com/opentensor/bittensor/pull/2197
+* btlogging/loggingmachine.py: Fix bw compat API. by @mvds00 in https://github.com/opentensor/bittensor/pull/2155
+* Check for participation before nomination call by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2193
+* test: subnet list e2e by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2198
+* ensure msg is str in _concat_msg by @thewhaleking in https://github.com/opentensor/bittensor/pull/2200
+* Fixes tests depending on explicit line numbers by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2211
+* Merge streaming fix to staging by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2183
+* Multiple bittensor versions e2e workflow by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2212
+* Changes name of workflow file by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2213
+* Enhances e2e tests to contain assertions & logging  by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2192
+* Security fix: Bumps ansible and certifi  by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2214
+* Wallet List Command e2e test by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2207
+* fix Synapse base performance (more than 10x speed up) by @mjurbanski-reef in https://github.com/opentensor/bittensor/pull/2161
+* Child Hotkeys by @opendansor in https://github.com/opentensor/bittensor/pull/2071
+* Improve child hotkeys QOL by @opendansor in https://github.com/opentensor/bittensor/pull/2225
+* Child hotkeys handle excess normalization by @opendansor in https://github.com/opentensor/bittensor/pull/2229
+* Fixes chain compilation timeouts by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2238
+* Update Child Hotkey commands by @opendansor in https://github.com/opentensor/bittensor/pull/2245
+* feat: return error message instead of raising exception by @gus-opentensor in https://github.com/opentensor/bittensor/pull/2244
+* Backmerge master to staging (7.3.1) by @ibraheem-opentensor in https://github.com/opentensor/bittensor/pull/2254
+
+## New Contributors
+* @bradleytf made their first contribution in https://github.com/opentensor/bittensor/pull/2034
+* @Rapiiidooo made their first contribution in https://github.com/opentensor/bittensor/pull/2132
+* @timabilov made their first contribution in https://github.com/opentensor/bittensor/pull/2139
+* @mvds00 made their first contribution in https://github.com/opentensor/bittensor/pull/2168
+* @dependabot made their first contribution in https://github.com/opentensor/bittensor/pull/2197
+
+**Full Changelog**: https://github.com/opentensor/bittensor/compare/v7.3.1...v7.4.0
+
 ## 7.3.1 / 2024-08-19
 
 ## What's Changed

VERSION

@@ -1 +1 @@
-7.3.1
+7.4.0

bittensor/__init__.py

@@ -40,7 +40,7 @@
 
 
 # Bittensor code and protocol version.
-__version__ = "7.3.1"
+__version__ = "7.4.0"
 
 _version_split = __version__.split(".")
 __version_info__ = tuple(int(part) for part in _version_split)
@@ -131,7 +131,7 @@ def debug(on: bool = True):
 ) is not None:
     __local_entrypoint__ = BT_SUBTENSOR_CHAIN_ENDPOINT
 else:
-    __local_entrypoint__ = "ws://127.0.0.1:9946"
+    __local_entrypoint__ = "ws://127.0.0.1:9944"
 
 
 __tao_symbol__: str = chr(0x03C4)

bittensor/btlogging/loggingmachine.py

@@ -29,6 +29,7 @@
 import os
 import sys
 from logging.handlers import QueueHandler, QueueListener, RotatingFileHandler
+from logging import Logger
 from typing import NamedTuple
 
 from statemachine import State, StateMachine
@@ -55,7 +56,7 @@ class LoggingConfig(NamedTuple):
     logging_dir: str
 
 
-class LoggingMachine(StateMachine):
+class LoggingMachine(StateMachine, Logger):
     """Handles logger states for bittensor and 3rd party libraries."""
 
     Default = State(initial=True)
@@ -360,58 +361,45 @@ def __trace_on__(self) -> bool:
         """
         return self.current_state_value == "Trace"
 
-    @staticmethod
-    def _concat_msg(*args):
-        return " - ".join(str(el) for el in args if el != "")
-
-    def trace(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def trace(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps trace message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        self._logger.trace(msg, *args, **kwargs, stacklevel=2)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.trace(msg, *args, **kwargs)
 
-    def debug(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def debug(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps debug message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        self._logger.debug(msg, *args, **kwargs, stacklevel=2)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.debug(msg, *args, **kwargs)
 
-    def info(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def info(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps info message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        self._logger.info(msg, *args, **kwargs, stacklevel=2)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.info(msg, *args, **kwargs)
 
-    def success(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def success(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps success message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        self._logger.success(msg, *args, **kwargs, stacklevel=2)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.success(msg, *args, **kwargs)
 
-    def warning(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def warning(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps warning message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        self._logger.warning(msg, *args, **kwargs, stacklevel=2)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.warning(msg, *args, **kwargs)
 
-    def error(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def error(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps error message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        self._logger.error(msg, *args, **kwargs, stacklevel=2)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.error(msg, *args, **kwargs)
 
-    def critical(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def critical(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps critical message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        self._logger.critical(msg, *args, **kwargs, stacklevel=2)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.critical(msg, *args, **kwargs)
 
-    def exception(self, msg="", *args, prefix="", suffix="", **kwargs):
+    def exception(self, msg="", prefix="", suffix="", *args, **kwargs):
         """Wraps exception message with prefix and suffix."""
-        msg = self._concat_msg(prefix, msg, suffix)
-        stacklevel = 2
-        if (
-            sys.implementation.name == "cpython"
-            and sys.version_info.major == 3
-            and sys.version_info.minor < 11
-        ):
-            # Note that, on CPython < 3.11, exception() calls through to
-            # error() without adjusting stacklevel, so we have to increment it.
-            stacklevel += 1
-        self._logger.exception(msg, *args, **kwargs, stacklevel=stacklevel)
+        msg = f"{prefix} - {msg} - {suffix}"
+        self._logger.exception(msg, *args, **kwargs)
 
     def on(self):
         """Enable default state."""

bittensor/metagraph.py

@@ -25,10 +25,11 @@
 import bittensor
 from os import listdir
 from os.path import join
-from typing import List, Optional, Union, Tuple
+from typing import List, Optional, Union, Tuple, cast
 
 from bittensor.chain_data import AxonInfo
 from bittensor.utils.registration import torch, use_torch
+from bittensor.utils import weight_utils
 
 METAGRAPH_STATE_DICT_NDARRAY_KEYS = [
     "version",
@@ -649,33 +650,40 @@ def _process_weights_or_bonds(
 
                 self.weights = self._process_weights_or_bonds(raw_weights_data, "weights")
         """
-        data_array = []
+        data_array: list[Union[NDArray[np.float32], "torch.Tensor"]] = []
         for item in data:
             if len(item) == 0:
                 if use_torch():
-                    data_array.append(torch.zeros(len(self.neurons)))  # type: ignore
+                    data_array.append(torch.zeros(len(self.neurons)))
                 else:
-                    data_array.append(np.zeros(len(self.neurons), dtype=np.float32))  # type: ignore
+                    data_array.append(np.zeros(len(self.neurons), dtype=np.float32))
             else:
                 uids, values = zip(*item)
                 # TODO: Validate and test the conversion of uids and values to tensor
                 if attribute == "weights":
                     data_array.append(
-                        bittensor.utils.weight_utils.convert_weight_uids_and_vals_to_tensor(
+                        weight_utils.convert_weight_uids_and_vals_to_tensor(
                             len(self.neurons),
                             list(uids),
-                            list(values),  # type: ignore
+                            list(values),
                         )
                     )
                 else:
-                    data_array.append(
-                        bittensor.utils.weight_utils.convert_bond_uids_and_vals_to_tensor(  # type: ignore
-                            len(self.neurons), list(uids), list(values)
-                        ).astype(np.float32)
+                    da_item = weight_utils.convert_bond_uids_and_vals_to_tensor(
+                        len(self.neurons), list(uids), list(values)
                     )
+                    if use_torch():
+                        data_array.append(cast("torch.LongTensor", da_item))
+                    else:
+                        data_array.append(
+                            cast(NDArray[np.float32], da_item).astype(np.float32)
+                        )
         tensor_param: Union["torch.nn.Parameter", NDArray] = (
             (
-                torch.nn.Parameter(torch.stack(data_array), requires_grad=False)
+                torch.nn.Parameter(
+                    torch.stack(cast(list["torch.Tensor"], data_array)),
+                    requires_grad=False,
+                )
                 if len(data_array)
                 else torch.nn.Parameter()
             )
@@ -731,7 +739,7 @@ def _process_root_weights(
                 uids, values = zip(*item)
                 # TODO: Validate and test the conversion of uids and values to tensor
                 data_array.append(
-                    bittensor.utils.weight_utils.convert_root_weight_uids_and_vals_to_tensor(  # type: ignore
+                    weight_utils.convert_root_weight_uids_and_vals_to_tensor(  # type: ignore
                         n_subnets, list(uids), list(values), subnets
                     )
                 )

tests/unit_tests/test_logging.py

@@ -1,5 +1,3 @@
-import os
-import re
 import pytest
 import multiprocessing
 import logging as stdlogging
@@ -170,36 +168,3 @@ def test_all_log_levels_output(logging_machine, caplog):
     assert "Test warning" in caplog.text
     assert "Test error" in caplog.text
     assert "Test critical" in caplog.text
-
-
-def test_log_sanity(logging_machine, caplog):
-    """
-    Test that logging is sane:
-    - prefix and suffix work
-    - format strings work
-    - reported filename is correct
-    Note that this is tested against caplog, which is not formatted the same as
-    stdout.
-    """
-    basemsg = "logmsg #%d, cookie: %s"
-    cookie = "0ef852c74c777f8d8cc09d511323ce76"
-    nfixtests = [
-        {},
-        {"prefix": "pref"},
-        {"suffix": "suff"},
-        {"prefix": "pref", "suffix": "suff"},
-    ]
-    cookiejar = {}
-    for i, nfix in enumerate(nfixtests):
-        prefix = nfix.get("prefix", "")
-        suffix = nfix.get("suffix", "")
-        use_cookie = f"{cookie} #{i}#"
-        logging_machine.info(basemsg, i, use_cookie, prefix=prefix, suffix=suffix)
-        # Check to see if all elements are present, regardless of downstream formatting.
-        expect = f"INFO.*{os.path.basename(__file__)}.* "
-        if prefix != "":
-            expect += prefix + " - "
-        expect += basemsg % (i, use_cookie)
-        if suffix != "":
-            expect += " - " + suffix
-        assert re.search(expect, caplog.text)

tests/unit_tests/test_metagraph.py

@@ -124,6 +124,37 @@ def test_process_weights_or_bonds(mock_environment):
     # TODO: Add more checks to ensure the bonds have been processed correctly
 
 
+def test_process_weights_or_bonds_torch(
+    mock_environment, force_legacy_torch_compat_api
+):
+    _, neurons = mock_environment
+    metagraph = bittensor.metagraph(1, sync=False)
+    metagraph.neurons = neurons
+
+    # Test weights processing
+    weights = metagraph._process_weights_or_bonds(
+        data=[neuron.weights for neuron in neurons], attribute="weights"
+    )
+    assert weights.shape[0] == len(
+        neurons
+    )  # Number of rows should be equal to number of neurons
+    assert weights.shape[1] == len(
+        neurons
+    )  # Number of columns should be equal to number of neurons
+    # TODO: Add more checks to ensure the weights have been processed correctly
+
+    # Test bonds processing
+    bonds = metagraph._process_weights_or_bonds(
+        data=[neuron.bonds for neuron in neurons], attribute="bonds"
+    )
+    assert bonds.shape[0] == len(
+        neurons
+    )  # Number of rows should be equal to number of neurons
+    assert bonds.shape[1] == len(
+        neurons
+    )  # Number of columns should be equal to number of neurons
+
+
 # Mocking the bittensor.subtensor class for testing purposes
 @pytest.fixture
 def mock_subtensor():