Tunneled LSO/CSO Support #614
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Accidentally over-counted by the ETH header, and illumos is sending down chunks of size n×MSS. This would split one TSO packet into three (versus baseline two).
FelixMcFelix
force-pushed
the
ingot-and-offload
branch
from
fbae192 to
3042e76
Compare
Needed for older propolis (falcon), which does not preallocate sufficient headroom for us to push into.
This makes sure that if we're receiving a packet which has benefited from real/pseudo GRO, viona is able to split it up if the guest can't actually take those packets.
Goodbye CI, I knew thee well.
Back to target perf.
At the very least, this compiles. We needed to regenerate `ip.rs`, on account of the many `extern` -> `unsafe extern` block changes.
CI on recent PRs is breaking, due to rustup 1.28.0+ no longer autoinstalling the correct rust toolchain version. This hurts us immediately since we have *two* toolchains (pinned nightly and stable), and deliberately specified the nightly for some tooling. This PR changes this over to use buildomat's auto-installation for the stable variant, and the new toolchain show -> install pattern for nightly. This also lets us place `$NIGHTLY` into most of our `cargo fmt` invocations, which should reduce the busywork in future compiler bumps for XDE.
FelixMcFelix
commented
Mar 7, 2025
| @@ -0,0 +1,107 @@ | |||
| // This Source Code Form is subject to the terms of the Mozilla Public | |||
There was a problem hiding this comment.
Most of this file is moved out of port/mod.rs (formerly port.rs). The discussion of "opte:" reserved keys and the use of INTERNAL_TARGET are new.
| @@ -204,7 +204,7 @@ impl DlsStream { | |||
| /// but for now we pass only a single packet at a time. | |||
| pub fn tx_drop_on_no_desc( | |||
| &self, | |||
| pkt: MsgBlk, | |||
| pkt: impl AsMblk, | |||
There was a problem hiding this comment.
The AsMblk changes here would ordinarily support later functionality like Tx/Rx of batches of packets (blocked on the flows rework, naturally). I've pulled it forward here because, strictly speaking, mac_hw_emul can turn (1..n) packets into (0..m) packets even if we're only using it to emulate checksums one-at-a-time.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes use of in-progress illumos bits to detect common offloads between underlay devices, and advertises LSO/CSO capabilities on
opteports, relying illumos's emulation functionality to split packets and insert checksums if not oncxgbe(or other compatible NIC).This allows for a few nice things:
Ultimately, we have control over the MSS we advertise to the NIC for LSO. The useful part of this is that when we know we are using a purely rack-internal path, we can elevate the MSS up to
MTU - overheads. I've added a system of 'well-known'ActionMetaKV pairs which allow for layers likeoverlayto propagate this knowledge out. Given that the use of a larger MTU vastly reduces inbound packet rate on the receive half (the main bottleneck today), this gets us to around 8Gbps iPerf for rack-local traffic (and 17Gbps for two or more parallel streams) onglasgow.On
dublinwith a full control plane, this resolves to around 4Gbps sled-to-sled and 14-Gbps sled-local between Linux VMs – we want to investigate the drop here once this is on dogfood. illumos as a guest doesn't do much better than before since it does not advertise GRO (illumos-as-host will fragment the packets) or LSO (it will never send TCP packets > MTU).Closes #328, closes #329.
Based on #688.