|
HTML rendering created 2025-09-06 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|
|
NAME | SYNOPSIS | DESCRIPTION | USAGE | EXAMPLE | SEE ALSO | AUTHOR | COLOPHON |
|
|
|
MLX5DV_WR(3) mlx5 Programmer’s Manual MLX5DV_WR(3)
mlx5dv_wr_set_dc_addr - Attach a DC info to the last work request
mlx5dv_wr_raw_wqe - Build a raw work request
mlx5dv_wr_memcpy - Build a DMA memcpy work request
#include <infiniband/mlx5dv.h>
static inline void mlx5dv_wr_set_dc_addr(struct mlx5dv_qp_ex *mqp,
struct ibv_ah *ah,
uint32_t remote_dctn,
uint64_t remote_dc_key);
static inline void mlx5dv_wr_set_dc_addr_stream(struct mlx5dv_qp_ex *mqp,
struct ibv_ah *ah,
uint32_t remote_dctn,
uint64_t remote_dc_key,
uint16_t stream_id);
struct mlx5dv_mr_interleaved {
uint64_t addr;
uint32_t bytes_count;
uint32_t bytes_skip;
uint32_t lkey;
};
static inline void mlx5dv_wr_mr_interleaved(struct mlx5dv_qp_ex *mqp,
struct mlx5dv_mkey *mkey,
uint32_t access_flags, /* use enum ibv_access_flags */
uint32_t repeat_count,
uint16_t num_interleaved,
struct mlx5dv_mr_interleaved *data);
static inline void mlx5dv_wr_mr_list(struct mlx5dv_qp_ex *mqp,
struct mlx5dv_mkey *mkey,
uint32_t access_flags, /* use enum ibv_access_flags */
uint16_t num_sges,
struct ibv_sge *sge);
static inline int mlx5dv_wr_raw_wqe(struct mlx5dv_qp_ex *mqp, const void *wqe);
static inline void mlx5dv_wr_memcpy(struct mlx5dv_qp_ex *mqp_ex,
uint32_t dest_lkey, uint64_t dest_addr,
uint32_t src_lkey, uint64_t src_addr,
size_t length)
The MLX5DV work request APIs (mlx5dv_wr_*) is an extension for IBV
work request API (ibv_wr_*) with mlx5 specific features for send
work request. This may be used together with or without ibv_wr_*
calls.
To use these APIs a QP must be created using mlx5dv_create_qp()
with send_ops_flags of struct ibv_qp_init_attr_ex set.
If the QP does not support all the requested work request types
then QP creation will fail.
The mlx5dv_qp_ex is extracted from the IBV_QP by ibv_qp_to_qp_ex()
and mlx5dv_qp_ex_from_ibv_qp_ex(). This should be used to apply
the mlx5 specific features on the posted WR.
A work request creation requires to use the ibv_qp_ex as described
in the man for ibv_wr_post and mlx5dv_qp with its available
builders and setters.
QP Specific builders
RC QPs mlx5dv_wr_mr_interleaved()
registers an interleaved memory layout by using an indirect
mkey and some interleaved data. The layout of the memory
pointed by the mkey after its registration will be the data
representation for the num_interleaved entries. This
single layout representation is repeated by repeat_count.
The data as described by struct mlx5dv_mr_interleaved will
hold real data defined by bytes_count and then a padding of
bytes_skip. Post a successful registration, RDMA
operations can use this mkey. The hardware will scatter
the data according to the pattern. The mkey should be used
in a zero-based mode. The addr field in its ibv_sge is an
offset in the total data. To create this mkey
mlx5dv_create_mkey() should be used.
Current implementation requires the IBV_SEND_INLINE option
to be on in ibv_qp_ex->wr_flags field. To be able to have
more than 3 num_interleaved entries, the QP should be
created with a larger WQE size that may fit it. This
should be done using the max_inline_data attribute of
struct ibv_qp_cap upon its creation.
As one entry will be consumed for strided header, the mkey
should be created with one more entry than the required
num_interleaved.
In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the
reported WC opcode will be MLX5DV_WC_UMR. Unregister the
mkey to enable another pattern registration should be done
via ibv_post_send with IBV_WR_LOCAL_INV opcode.
mlx5dv_wr_mr_list()
registers a memory layout based on list of ibv_sge. The
layout of the memory pointed by the mkey after its
registration will be based on the list of sge counted by
num_sges. Post a successful registration RDMA operations
can use this mkey, the hardware will scatter the data
according to the pattern. The mkey should be used in a
zero-based mode, the addr field in its ibv_sge is an offset
in the total data.
Current implementation requires the IBV_SEND_INLINE option
to be on in ibv_qp_ex->wr_flags field. To be able to have
more than 4 num_sge entries, the QP should be created with
a larger WQE size that may fit it. This should be done
using the max_inline_data attribute of struct ibv_qp_cap
upon its creation.
In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the
reported WC opcode will be MLX5DV_WC_UMR. Unregister the
mkey to enable other pattern registration should be done
via ibv_post_send with IBV_WR_LOCAL_INV opcode.
RC or DCI QPs
mlx5dv_wr_memcpy()
Builds a DMA memcpy work request to copy data of length
length from src_addr to dest_addr. The copy operation will
be done using the DMA MMO functionality of the device to
copy data on PCI bus.
The MLX5DV_QP_EX_WITH_MEMCPY flag in
mlx5dv_qp_init_attr.send_ops_flags needs to be set during
QP creation. If the device or QP doesn’t support it then
QP creation will fail. The maximum memcpy length that is
supported by the device is reported in
mlx5dv_context->max_wr_memcpy_length. A zero value in
mlx5dv_context->max_wr_memcpy_length means the device
doesn’t support memcpy operations.
IBV_SEND_FENCE indicator should be used on a following send
request which is dependent on dest_addr of the memcpy
operation.
In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the
reported WC opcode will be MLX5DV_WC_MEMCPY.
Raw WQE builders
mlx5dv_wr_raw_wqe()
It is used to build a custom work request (WQE) and post it
on a normal QP. The caller needs to set all details of the
WQE (except the “ctrl.wqe_index” and “ctrl.signature”
fields, which is the driver’s responsibility to set). The
MLX5DV_QP_EX_WITH_RAW_WQE flag in
mlx5_qp_attr.send_ops_flags needs to be set.
The wr_flags are ignored as it’s the caller’s
responsibility to set flags in WQE.
No matter what the send opcode is, the work completion
opcode for a raw WQE is IBV_WC_DRIVER2.
QP Specific setters
DCI QPs
mlx5dv_wr_set_dc_addr() must be called to set the DCI WR
properties. The destination address of the work is
specified by ah, the remote DCT number is specified by
remote_dctn and the DC key is specified by remote_dc_key.
This setter is available when the QP transport is DCI and
send_ops_flags in struct ibv_qp_init_attr_ex is set. The
available builders and setters for DCI QP are the same as
RC QP. DCI QP created with
MLX5DV_QP_INIT_ATTR_MASK_DCI_STREAMS can call
mlx5dv_wr_set_dc_addr_stream() to define the stream_id of
the operation to allow HW to choose one of the multiple
concurrent DCI resources. Calls to mlx5dv_wr_set_dc_addr()
are equivalent to using stream_id=0
/* create DC QP type and specify the required send opcodes */
attr_ex.qp_type = IBV_QPT_DRIVER;
attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;
attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_DC;
attr_dv.dc_init_attr.dc_type = MLX5DV_DCTYPE_DCI;
ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);
ibv_wr_start(qpx);
/* Use ibv_qp_ex object to set WR generic attributes */
qpx->wr_id = my_wr_id_1;
qpx->wr_flags = IBV_SEND_SIGNALED;
ibv_wr_rdma_write(qpx, rkey, remote_addr_1);
ibv_wr_set_sge(qpx, lkey, local_addr_1, length_1);
/* Use mlx5 DC setter using mlx5dv_qp_ex object */
mlx5dv_wr_set_wr_dc_addr(mqpx, ah, remote_dctn, remote_dc_key);
ret = ibv_wr_complete(qpx);
ibv_post_send(3), ibv_create_qp_ex(3), ibv_wr_post(3),
mlx5dv_create_mkey(3).
Guy Levi ⟨guyle@mellanox.com⟩
Mark Zhang ⟨markzhang@nvidia.com⟩
This page is part of the rdma-core (RDMA Core Userspace Libraries
and Daemons) project. Information about the project can be found
at ⟨https://github.com/linux-rdma/rdma-core⟩. If you have a bug
report for this manual page, send it to
linux-rdma@vger.kernel.org. This page was obtained from the
project's upstream Git repository
⟨https://github.com/linux-rdma/rdma-core.git⟩ on 2025-08-11. (At
that time, the date of the most recent commit that was found in
the repository was 2025-08-04.) If you discover any rendering
problems in this HTML version of the page, or you believe there is
a better or more up-to-date source for the page, or you have
corrections or improvements to the information in this COLOPHON
(which is not part of the original manual page), send a mail to
man-pages@man7.org
mlx5 2019-02-24 MLX5DV_WR(3)
|
HTML rendering created 2025-09-06 by Michael Kerrisk, author of The Linux Programming Interface. For details of in-depth Linux/UNIX system programming training courses that I teach, look here. Hosting by jambit GmbH. |
|