Enabling Fast Reg MR usage during First contact in SMC-R
Publication Date: 2017-Feb-10
The IP.com Prior Art Database
There is an existing limitation with SMC-R protocol (RFC-7609) to use Fast Reg MR during first contact in a Link Group (LG). Proposing a viable modification to SMC-R protocol that will allow SMCR to use Fast Reg MR even during first contact in a Link Group (LG).
Enabling Fast Reg MR usage during First contact in SMC -R
SMC-R is IBM’s Shared Memory Communications over RDMA protocol. This protocol provides Remote Direct Memory Access (RDMA) communications to Transmission Control Protocol (TCP) endpoints in a manner that is transparent to socket applications. It further provides for dynamic discovery of partner RDMA capabilities and dynamic setup of RDMA connections, as well as transparent high availability and load balancing when redundant RDMA network paths are available. It maintains many of the traditional TCP/IP qualities of service such as filtering that enterprise users demand, as well as TCP socket semantics such as urgent data.
There is an existing limitation with SMC-R protocol (RFC-7609) to use Fast Register Memory Region (Fast Reg MR) during first contact in a Link Group (LG). Proposing a viable modification to SMC-R protocol that will allow SMCR to use Fast Reg MR even during first contact in a Link Group (LG).
Fast Reg MR is Infiniband (IB) specification compliant but needs Queue Pair (QP) to be in Ready To Send (RTS) state to post a send Work Request (WR) which will not work with first contact of a LG in SMC-R. In existing SMC-R RFC, during first contact while RKey is being created, QP will be in Ready To Receive (RTR) and INIT state on client and server sides respectively.
Suggestion is to make current SMC-R protocol to wait without creating RKey till QP moves into Ready To Send (RTS) state during First contact in a LG.
There are 2 types of Fast Memory Region. 1. Fast Memory Region (FMR) provides API which "may" be good with current SMC-R RFC but is not compliant with IB specification and is phasing out. 2. Fast Reg MR is IB specification compliant but needs QP to be in RTS state to post a send (control) WR which will not work with first contact of a LG in SMC-R RFC.
As understood, Fast Reg MR can be aggressively used for dynamic expansion and contraction of Remote Memory Buffer (RMB) pools which results in reduced CPU usage and reduced registration latency. All Fast Reg MR does is to pre-allocate data structures at adapter device driver level and some resources in adapter which will consume considerable time if it was done every time. When performed a de-register RMB with adapter during memory contraction, these pre-allocated fast REG MR's will be added to the free pool which can be reused. So there will be substantial savings compared to ib_reg_mr() when new connections (and RMBs) are coming and leaving. In summary, using Fast Reg MR will reduce CPU usage for all cases (First contact or subsequent contact). There are substantial savings of CPU usage during expansion or contra...