Extendable Address Translation Tables for use by Infinband Host Channel Adapters.
Original Publication Date: 2005-May-10
Included in the Prior Art Database: 2005-May-10
Disclosed is a mechanism for providing scalable InfiniBand* memory management. Typical schemes implement the protection and address translation tables in a separate memory bank attached to the Host Channel Adapter (HCA), which imposes limitations in terms of the number of memory regions and their size that may be supported. This invention defines a scheme for implementing these structures in system memory, which provides a much more scalable solution.
Extendable Address Translation Tables for use by Infinband Host Channel Adapters .
The following describes a mechanism that allows an InfiniBand* (IB) Host Channel Adapter (HCA) to determine the real address(s) in system memory that can be used to DMA data to/from, based on a virtual address that is provided in a work request on a send or receive queue, or in an RDMA packet received on an IB link. The structures are fully extendable to support large numbers of memory regions and memory regions that vary in size from bytes to multiple terabytes. Because the structures are stored in 4K pages, the task of allocating and restoring the memory used to contain them is based on existing, well-understood techniques.
A Protection Table Entry (PTE) defines the characteristics of a memory region. The HCA accesses a PTE by using the index portion of an L_Key (Local Key) or R_Key (Remote Key). The PTE contains the starting virtual address of the memory region, the length of the memory region, access controls for the memory region (e.g. remote write access allowed) and one or more address translation pointers. The number of address translation pointers stored in the PTE is determined based on the size of the protection table, being a convenient value that is easily indexed using the L_Key/R_Key, and if any optimizations are required for a particular size memory region.
Address Translation Tables (AT_Tables) are used by the HCA to determine the real address of a physical page that is referenced by a virtual address. The AT_Tables are stored in system memory, which allows great flexibility in terms of the number of memory regions that may be supported. There can be multiple AT_Tables for a given memory region, depending on the size of the memory region and the page size. The larger the memory region, the greater the number of levels of AT_Tables required. Smaller memory regions with less levels of AT_Table are more efficient requiring less fetches from system memory. In the following descriptions it is assumed that the AT_Tables are stored in 4K pages, although other pages sizes could be used. It is also assumed that the AT_Tables contain 64-bit addresses, although 32-bit addresses could be used, whereby the number of entries in each AT_Table would be doubled.
The HCA uses the address translation process to determine the re...