Release 1.194.0

.github/workflows/build.yml

@@ -238,7 +238,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        arch: [x64]
+        arch: [arm64]
         os: [macos-latest]
         python-version: ["3.10", "3.11", "3.12"]
     defaults:

.pre-commit-config.yaml

@@ -31,12 +31,12 @@ repos:
       # - id: gitlint-ci
 
   - repo: https://github.com/codespell-project/codespell
-    rev: v2.2.6
+    rev: v2.3.0
     hooks:
       - id: codespell
         description: Checks for common misspellings.
         types_or: [python, cython, rst, markdown]
-        args: ["-L", "crate,ot,zar"]
+        args: ["-L", "crate,ot,socio-economic,zar"]
 
   ##############################################################################
   #  Rust formatting and linting
@@ -82,7 +82,7 @@ repos:
         exclude: "docs/_pygments/monokai.py"
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.4.5
+    rev: v0.4.6
     hooks:
       - id: ruff
         args: ["--fix"]

README.md

@@ -43,16 +43,16 @@ including FX, Equities, Futures, Options, CFDs, Crypto and Betting - across mult
 
 ## Features
 
-- **Fast** - Core written in Rust with asynchronous networking using [tokio](https://crates.io/crates/tokio)
-- **Reliable** - Type safety and thread safety through Rust. Redis backed performant state persistence
-- **Portable** - OS independent, runs on Linux, macOS, Windows. Deploy using Docker
-- **Flexible** - Modular adapters mean any REST, WebSocket, or FIX API can be integrated
-- **Advanced** - Time in force `IOC`, `FOK`, `GTD`, `AT_THE_OPEN`, `AT_THE_CLOSE`, advanced order types and conditional triggers. Execution instructions `post-only`, `reduce-only`, and icebergs. Contingency order lists including `OCO`, `OTO`
-- **Customizable** - Add user defined custom components, or assemble entire systems from scratch leveraging the cache and message bus
-- **Backtesting** - Run with multiple venues, instruments and strategies simultaneously using historical quote tick, trade tick, bar, order book and custom data with nanosecond resolution
-- **Live** - Use identical strategy implementations between backtesting and live deployments
-- **Multi-venue** - Multiple venue capabilities facilitate market making and statistical arbitrage strategies
-- **AI Agent Training** - Backtest engine fast enough to be used to train AI trading agents (RL/ES)
+- **Fast:** Core written in Rust with asynchronous networking using [tokio](https://crates.io/crates/tokio)
+- **Reliable:** Type safety and thread safety through Rust. Redis backed performant state persistence
+- **Portable:** OS independent, runs on Linux, macOS, Windows. Deploy using Docker
+- **Flexible:** Modular adapters mean any REST, WebSocket, or FIX API can be integrated
+- **Advanced:** Time in force `IOC`, `FOK`, `GTD`, `AT_THE_OPEN`, `AT_THE_CLOSE`, advanced order types and conditional triggers. Execution instructions `post-only`, `reduce-only`, and icebergs. Contingency order lists including `OCO`, `OTO`
+- **Customizable:** Add user defined custom components, or assemble entire systems from scratch leveraging the cache and message bus
+- **Backtesting:** Run with multiple venues, instruments and strategies simultaneously using historical quote tick, trade tick, bar, order book and custom data with nanosecond resolution
+- **Live:** Use identical strategy implementations between backtesting and live deployments
+- **Multi-venue:** Multiple venue capabilities facilitate market making and statistical arbitrage strategies
+- **AI Training:** Backtest engine fast enough to be used to train AI trading agents (RL/ES)
 
 ![Alt text](https://github.com/nautechsystems/nautilus_trader/blob/develop/docs/_images/nautilus-art.png?raw=true "nautilus")
 
@@ -63,10 +63,10 @@ including FX, Equities, Futures, Options, CFDs, Crypto and Betting - across mult
 
 ## Why NautilusTrader?
 
-- **Highly performant event-driven Python** - native binary core components
-- **Parity between backtesting and live trading** - identical strategy code
-- **Reduced operational risk** - risk management functionality, logical correctness and type safety
-- **Highly extendable** - message bus, custom components and actors, custom data, custom adapters
+- **Highly performant event-driven Python:** Native binary core components
+- **Parity between backtesting and live trading:** Identical strategy code
+- **Reduced operational risk:** Risk management functionality, logical correctness and type safety
+- **Highly extendable:** Message bus, custom components and actors, custom data, custom adapters
 
 Traditionally, trading strategy research and backtesting might be conducted in Python (or other suitable language)
 using vectorized methods, with the strategy then needing to be reimplemented in a more event-drive way
@@ -129,8 +129,8 @@ into a unified interface. The following integrations are currently supported:
 | [Databento](https://databento.com)                        | `DATABENTO`           | Data Provider           | ![status](https://img.shields.io/badge/beta-yellow)     | [Guide](https://docs.nautilustrader.io/integrations/databento.html) |
 | [Interactive Brokers](https://www.interactivebrokers.com) | `INTERACTIVE_BROKERS` | Brokerage (multi-venue) | ![status](https://img.shields.io/badge/stable-green)    | [Guide](https://docs.nautilustrader.io/integrations/ib.html)        |
 
-- `ID:` The default client ID for the integrations adapter clients
-- `Type:` The type of integration (often the venue type)
+- **ID:** The default client ID for the integrations adapter clients
+- **Type:** The type of integration (often the venue type)
 
 ### Status
 - `building` - Under construction and likely not in a usable state
@@ -196,19 +196,23 @@ Refer to the [Installation Guide](https://docs.nautilustrader.io/getting_started
 
 ## Versioning and releases
 
-NautilusTrader is currently following a bi-weekly beta release schedule.
+NautilusTrader is currently targeting a weekly release schedule, occasionally there may be experimental
+or larger features which will delay a release by several weeks.
+
 The API is becoming more stable, however breaking changes are still possible between releases.
 Documentation of these changes in the release notes are made on a best-effort basis.
 
 ### Branches
 
+We aim to maintain a stable passing build on all branches.
+
 - `master` branch will always reflect the source code for the latest released version
 - `nightly` branch may contain experimental features and is generally merged from `develop` branch daily, and also when required
-- `develop` branch is normally very active with frequent commits and may contain experimental features. We aim to maintain a stable
-  passing build on this branch
+- `develop` branch is normally very active with frequent commits and may contain experimental features
 
-The current roadmap has a goal of achieving a stable API for a `2.x` version. From this
-point we will follow a formal process for releases, with deprecation periods for any API changes.
+The current roadmap has a goal of achieving a stable API for a `2.x` version (likely post Rust port).
+From this point we will follow a formal process for releases, with deprecation periods for any API changes.
+This allows us to maintain a maximum pace of development for now.
 
 ## Makefile
 

RELEASES.md

@@ -1,3 +1,31 @@
+# NautilusTrader 1.194.0 Beta
+
+Released on 31st May 2024 (UTC).
+
+### Enhancements
+- Added `DataEngine` order book deltas buffering to `F_LAST` flag (#1673), thanks @davidsblom
+- Added `DataEngineConfig.buffer_deltas` config option for the above (#1670), thanks @davidsblom
+- Improved Bybit order book deltas parsing to set `F_LAST` flag (#1670), thanks @davidsblom
+- Improved Bybit handling for top-of-book quotes and order book deltas (#1672), thanks @davidsblom
+- Improved Interactive Brokers integration test mocks (#1669), thanks @rsmb7z
+- Improved error message when no tick scheme initialized for an instrument, thanks for reporting @VeraLyu
+- Improved `SandboxExecutionClient` instrument handling (instruments just need to be added to cache)
+- Ported `VolumeWeightedAveragePrice` indicator to Rust (#1665), thanks @Pushkarm029
+- Ported `VerticalHorizontalFilter` indicator to Rust (#1666), thanks @Pushkarm029
+
+### Breaking Changes
+None
+
+### Fixes
+- Fixed `SimulatedExchange` processing of commands in real-time for sandbox mode
+- Fixed `DataEngine` unsubscribe handling (edge case would attempt to unsubscribe from the client multiple times)
+- Fixed Bybit order book deltas parsing (was appending bid side twice) (#1668), thanks @davidsblom
+- Fixed Binance instruments price and size precision parsing (was incorrectly stripping trailing zeros)
+- Fixed `BinanceBar` streaming feather writing (was not setting up writer)
+- Fixed backtest high-level tutorial documentation errors, thanks for reporting @Leonz5288
+
+---
+
 # NautilusTrader 1.193.0 Beta
 
 Released on 24th May 2024 (UTC).
@@ -72,7 +100,7 @@ Released on 20th April 2024 (UTC).
 - Improved `modify_order` error logging when order values remain unchanged
 - Added `RecordFlag` enum for Rust and Python
 - Interactive Brokers further improvements and fixes, thanks @rsmb7z
-- Ported Bias indicator to Rust, thanks @Pushkarm029
+- Ported `Bias` indicator to Rust, thanks @Pushkarm029
 
 ### Breaking Changes
 - Reordered `OrderBookDelta` params `flags` and `sequence` and removed default 0 values (more explicit and less chance of mismatches)
@@ -111,8 +139,8 @@ Released on 22nd March 2024 (UTC).
 - Improved Binance execution client ping listen key error handling and logging
 - Improved Redis cache adapter and message bus error handling and logging
 - Improved Redis port parsing (`DatabaseConfig.port` can now be either a string or integer)
-- Ported ChandeMomentumOscillator indicator to Rust, thanks @Pushkarm029
-- Ported VIDYA indicator to Rust, thanks @Pushkarm029
+- Ported `ChandeMomentumOscillator` indicator to Rust, thanks @Pushkarm029
+- Ported `VIDYA` indicator to Rust, thanks @Pushkarm029
 - Refactored `InteractiveBrokersEWrapper`, thanks @rsmb7z
 - Redact Redis passwords in strings and logs
 - Upgraded `redis` crate to 0.25.2 which bumps up TLS dependencies, and turned on `tls-rustls-webpki-roots` feature flag
@@ -355,7 +383,7 @@ Released on 23rd December 2023 (UTC).
 - Changed `StrategyConfig.strategy_id` to type `StrategyId | None`
 - Changed `Instrument`, `OrderFilled` and `AccountState` `info` field serialization due below fix (you'll need to flush your cache)
 - Changed `CacheConfig` to take a `DatabaseConfig` (better symmetry with `MessageBusConfig`)
-- Changed `RedisCacheDatabase` data structure for currencies from hashset to simpler key-value (you'll need to clear cache or delete all curreny keys)
+- Changed `RedisCacheDatabase` data structure for currencies from hashset to simpler key-value (you'll need to clear cache or delete all currency keys)
 - Changed `Actor` state loading to now use the standard `Serializer`
 - Renamed `register_json_encoding` to `register_config_encoding`
 - Renamed `register_json_decoding` to `register_config_decoding`

docs/api_reference/index.md

@@ -34,8 +34,8 @@ from the latest NautilusTrader source code using [Sphinx](https://www.sphinx-doc
 
 Please note that there are separate references for different versions of NautilusTrader:
 
-- **Latest**: This API reference is built from the head of the `master` branch and represents the latest stable release.
-- **Nightly**: This API reference is built from the head of the `nightly` branch and represents bleeding edge and experimental changes/features currently in development.
+- **Latest:** This API reference is built from the head of the `master` branch and represents the latest stable release.
+- **Nightly:** This API reference is built from the head of the `nightly` branch and represents bleeding edge and experimental changes/features currently in development.
 
 You can select the desired API reference from the **Versions** top right drop down menu.
 

docs/concepts/data.md

@@ -199,7 +199,7 @@ from nautilus_trader.model.data import OrderBookDelta
 data_config = BacktestDataConfig(
     catalog_path=str(catalog.path),
     data_cls=OrderBookDelta,
-    instrument_id=instrument.id.value,
+    instrument_id=instrument.id,
     start_time=start,
     end_time=end,
 )

docs/concepts/overview.md

@@ -26,7 +26,7 @@ including FX, Equities, Futures, Options, CFDs, Crypto and Betting - across mult
 - **Backtesting:** Run with multiple venues, instruments and strategies simultaneously using historical quote tick, trade tick, bar, order book and custom data with nanosecond resolution
 - **Live:** Use identical strategy implementations between backtesting and live deployments
 - **Multi-venue:** Multiple venue capabilities facilitate market making and statistical arbitrage strategies
-- **AI Agent Training:** Backtest engine fast enough to be used to train AI trading agents (RL/ES)
+- **AI Training:** Backtest engine fast enough to be used to train AI trading agents (RL/ES)
 
 ![Nautilus](https://github.com/nautechsystems/nautilus_trader/blob/develop/docs/_images/nautilus-art.png?raw=true "nautilus")
 > *nautilus - from ancient Greek 'sailor' and naus 'ship'.*
@@ -36,10 +36,10 @@ including FX, Equities, Futures, Options, CFDs, Crypto and Betting - across mult
 
 ## Why NautilusTrader?
 
-- **Highly performant event-driven Python** - native binary core components
-- **Parity between backtesting and live trading** - identical strategy code
-- **Reduced operational risk** - risk management functionality, logical correctness and type safety
-- **Highly extendable** - message bus, custom components and actors, custom data, custom adapters
+- **Highly performant event-driven Python:** Native binary core components
+- **Parity between backtesting and live trading:** Identical strategy code
+- **Reduced operational risk:** Risk management functionality, logical correctness and type safety
+- **Highly extendable:** Message bus, custom components and actors, custom data, custom adapters
 
 Traditionally, trading strategy research and backtesting might be conducted in Python (or other suitable language)
 using vectorized methods, with the strategy then needing to be reimplemented in a more event-drive way
@@ -95,11 +95,13 @@ Python 3.11 offers improved run-time performance, while Python 3.12 additionally
 ```
 
 ## Domain model
+
 The platform features a comprehensive trading domain model that includes various value types such as 
 `Price` and `Quantity`, as well as more complex entities such as `Order` and `Position` objects, 
 which are used to aggregate multiple events to determine state.
 
 ### Data Types
+
 The following market data types can be requested historically, and also subscribed to as live streams when available from a venue / data provider, and implemented in an integrations adapter.
 - `OrderBookDelta` (L1/L2/L3)
 - `OrderBookDeltas` (container type)
@@ -140,6 +142,7 @@ The price types and bar aggregations can be combined with step sizes >= 1 in any
 This enables maximum flexibility and now allows alternative bars to be aggregated for live trading.
 
 ### Account Types
+
 The following account types are available for both live and backtest environments;
 
 - `Cash` single-currency (base currency)
@@ -149,6 +152,7 @@ The following account types are available for both live and backtest environment
 - `Betting` single-currency
 
 ### Order Types
+
 The following order types are available (when possible on a venue);
 
 - `MARKET`
@@ -160,4 +164,3 @@ The following order types are available (when possible on a venue);
 - `LIMIT_IF_TOUCHED`
 - `TRAILING_STOP_MARKET`
 - `TRAILING_STOP_LIMIT`
-

docs/developer_guide/adapters.md

@@ -9,10 +9,10 @@ into a unified interface.
 ## Structure of an Adapter
 
 An adapter typically consists of several components:
-1. **Instrument Provider**: Supplies instrument definitions
-2. **Data Client**: Handles live market data feeds and historical data requests
-3. **Execution Client**: Handles order execution and management
-5. **Configuration**: Configures the client settings
+1. **Instrument Provider:** Supplies instrument definitions
+2. **Data Client:** Handles live market data feeds and historical data requests
+3. **Execution Client:** Handles order execution and management
+5. **Configuration:** Configures the client settings
 
 ## Steps to Implement a New Adapter
 

docs/integrations/databento.md

@@ -161,7 +161,7 @@ The following Databento instrument classes are supported by NautilusTrader:
 | FX spot                    | `X`  | `CurrencyPair`               |
 | Bond                       | `B`  | Not yet available            |
 
-### MBO (market by order)
+### MBO (market-by-order)
 
 This schema is the highest granularity data offered by Databento, and represents
 full order book depth. Some messages also provide trade information, and so when
@@ -175,7 +175,7 @@ registered handler.
 Order book snapshots are also buffered into a discrete `OrderBookDeltas` container
 object, which occurs during the replay startup sequence.
 
-### MBP-1 (market by price, top-of-book)
+### MBP-1 (market-by-price, top-of-book)
 
 This schema represents the top-of-book only (quotes *and* trades). Like with MBO messages, some
 messages carry trade information, and so when decoding MBP-1 messages Nautilus 

docs/integrations/ib.md

@@ -56,7 +56,7 @@ print(gateway.is_logged_in(gateway.container))
 print(gateway.container.logs())
 ```
 
-**Note**: To supply credentials to the Interactive Brokers Gateway, either pass the `username` and `password` to the config dictionaries, or set the following environment variables:
+**Note:** To supply credentials to the Interactive Brokers Gateway, either pass the `username` and `password` to the config dictionaries, or set the following environment variables:
 - `TWS_USERNAME`
 - `TWS_PASSWORD`
 

docs/tutorials/backtest_high_level.md

@@ -63,7 +63,7 @@ Then we can create Nautilus `QuoteTick` objects by processing the DataFrame with
 
 ```python
 # Here we just take the first data file found and load into a pandas DataFrame
-df = CSVTickDataLoader.load(raw_files[0], index_col=0, format="%Y%m%d %H%M%S%f")
+df = CSVTickDataLoader.load(raw_files[0], index_col=0, datetime_format="%Y%m%d %H%M%S%f")
 df.columns = ["bid_price", "ask_price"]
 
 # Process quote ticks using a wrangler
@@ -141,7 +141,7 @@ data_configs = [
     BacktestDataConfig(
         catalog_path=str(ParquetDataCatalog.from_env().path),
         data_cls=QuoteTick,
-        instrument_id=instrument.id.value,
+        instrument_id=instrument.id,
         start_time=start,
         end_time=end,
     ),
@@ -152,7 +152,7 @@ strategies = [
         strategy_path="nautilus_trader.examples.strategies.ema_cross:EMACross",
         config_path="nautilus_trader.examples.strategies.ema_cross:EMACrossConfig",
         config=dict(
-            instrument_id=instrument.id.value,
+            instrument_id=instrument.id,
             bar_type="EUR/USD.SIM-15-MINUTE-BID-INTERNAL",
             fast_ema_period=10,
             slow_ema_period=20,

docs/tutorials/index.md

@@ -19,8 +19,8 @@ From basic tasks to more advanced operations, these tutorials cater to a wide ra
 
 ```{tip}
 Make sure you are following the tutorial docs which match the version of NautilusTrader you are running:
-- **Latest** - These docs are built from the HEAD of the `master` branch and work with the latest stable release.
-- **Develop** - These docs are built from the HEAD of the `develop` branch and work with bleeding edge and experimental changes/features currently in development.
+- **Latest:** These docs are built from the HEAD of the `master` branch and work with the latest stable release.
+- **Develop:** These docs are built from the HEAD of the `develop` branch and work with bleeding edge and experimental changes/features currently in development.
 ```
 
 ## Backtesting

examples/live/binance/binance_futures_testnet_market_maker.py

@@ -71,6 +71,7 @@
     # snapshot_orders=True,
     # snapshot_positions=True,
     # snapshot_positions_interval=5.0,
+    # streaming=StreamingConfig(catalog_path="catalog"),
     data_clients={
         "BINANCE": BinanceDataClientConfig(
             api_key=None,  # 'BINANCE_API_KEY' env var