Open
Conversation
Summary: Add Data Direct Interface (DDI) support to the devmem TCP path. DDI provides a direct PCIe link between NIC and GPU, bypassing the CPU root complex for higher bandwidth device memory networking. Core changes: 1. Add ndo_get_ddi_device() callback to ask the NIC driver if a DDI device is available, and returns the DDI device if available. network core will expose it to userspace via sysfs. and net_devmem_bind_dmabuf() to bind with DDI device when required 2. Add ddi_device sysfs attribute (/sys/class/net/<iface>/ddi_device) to expose DDI availability to userspace. 3. Add NETDEV_A_DMABUF_DDI netlink attribute to bind_rx and bind_tx. for userspace explicitly requests bind with DDI. 4. Add net_devmem_is_ddi() helper for drivers to check if the current page_pool binding uses a DDI device. e.g. mlx5e will create crossing mkey for DMA via DDI Design: - DDI knowledge stays in the driver -- the core networking layer only asks "is there a DDI device?" via the callback - networking core expose DDI capability via sysfs to userspace - DDI is opt-in via netlink flag -- userspace must explicitly request DDI mode Signed-off-by: Sheldon Wu <xzw2048@gmail.com>
Summary: enable Data Direct Interface (DDI) for devmem TCP on ConnectX-8 NICs. DDI device discovery: - Query NIC firmware for VUID (unique identifier) - Scan PCI bus for DDI devices (0x2100) with matching VPD VUID - Cache the DDI device on first lookup to avoid repeated PCI scans - Ideally IB and TCP devmem should share the discovery later. so the discovery process needs to move to mlx5 core at the end. DDI mkey setup: - It mostly mirror IB's process (KSM mkey + crossing VHCA mkey) to enable DMA via DDI. - Create DDI PA mkey (device-level, shared across all connections) - For RX: switch UMR from MTT to KSM mode, embed DDI PA mkey as child key in each KSM entry. Create crossing VHCA mkey wrapping the KSM UMR mkey. - For TX: create crossing VHCA mkey wrapping DDI PA mkey directly. Per-frag mkey selection by checking net_devmem_niov_is_ddi() to use crossing mkey for DDI-bound frags. Firmware limitation: PCI function 1 is skipped due to UMR syndrome 0x52 (firmware fix pending). Signed-off-by: Sheldon Wu <xzw2048@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
use this PR to discuss approach to support DDI for devmem.