THREAD SWITCHING VERSUS ADAPTER SWITCHING ON PCIe* SLOTS
Publication Date: 2016-Aug-16
The IP.com Prior Art Database
Disclosed is a method to dynamically switch workloads to aid performance or for power savings by either migrating the workload between threads or altering the physical topology of the peripheral component interconnect express (PCIe*) slot(s) to create workload affinity to a CPU.
Page 01 of 4
THREAD SWITCHING VERSUS ADAPTER SWITCHING ON PCIe *
Disclosed is a method to dynamically switch workloads to aid performance and for power savings by either migrating the workload between threads or altering the physical topology of the peripheral component interconnect express (PCIe*) slot(s) to create workload affinity to a CPU. In server systems, the input/output ( IO) interface is usually done through PCIe over the past few years. PCIe offers maximum bandwidth and compatibility with different protocols and has proven to be an accepted solution. With increasing speed and expanding memory, the IO bridge may be incorporated within the processor to provide faster access to the IO world. With higher speed PCIe buses, the utilization of the bus is split using a switch. Functions that can multiplex the same bus from bandwidth perspective are grouped and presented in a switch so that they can be shared. In a system, an adapter behind a socket can have affinity towards the other processor and, hence, data transfer would be done through the PCIe bus (IO) to the other chip memory through inter-processor communication bus. Figure 1 depicts a block diagram of a 2S system showing different PCIe slots in the hosts for different adapters.
Apart from networking, due to acceleration and non-volatile memory (NVM) express (NVMe**), the requirement of data movement from both the CPUs to an adapter is unavoidable. A slot behind the DCM2 hosts a card. DCM1 has to go via inter-chip bus. Consider the following situation:
Data from DCM1 to its own slot
Data from DCM2 to the slot in DCM1 goes over the X bus
Work Load (WL) thread moves to the other DCM
If free thread not available, data occupies X bus
Current Solutions in the market: Single device multiplexed to two different CPUs through
Page 02 of 4
PCIe fabric switch.
Involves active switch.
Wastes pin out of the processor and switch.
PCIe fabric and switch are active logic.
Thread balancing and adapter balancing is not an option in current solutions.
In server systems, if the passive multiplexer (mux) allows switching between two CPUs, then
the workload assigned thread can be moved between CPUs or adapter affinity can be switched using the following approach:
Monitor the inter-chip bus traffic and the address ranges to see the affinity of an
adapter to a different CPU.
Analyze the buffer and IO affinity to the workload thread.
Analyze the power and thermal take-down due to the card affinity to the next physical
If the local chip that has an affined adapter, has threads and memory to handle
without degrading the existing performance, move the workload to the other chip. If the local chip cannot handle any more threads or the workload has an affinity
with the thread employed with adjacent chip, make the decision to move the affinity to the next chip by swit...