Modern applications in information technology are rapidly evolving and setting the stage for the development of high-end data communication technologies that not only meet the demands of the changing computing environment but also enable users to successfully accomplish projects not achievable before. The InfiniBand architecture (IBA) emerged as one of the most effective network infrastructures providing solutions to areas in server communications and I/O requirements that IP and Ethernet had not previously addressed (IBTA, 2001).
IBA is an industry-level-architecture standard focusing on inter-server and server I/O communication. The InfiniBand Trade Association, with about 200 member companies developed the IBA. An IBA infrastructure can be a smaller server with a single processor featuring a minimal number of I/O devices or a massive parallel application featuring multitudes of processors and a higher number of I/O devices. Unlike the traditional system-bus architecture, IBA is much more advanced in terms of scalability, reliability, performance and availability for modern and emerging server systems (Joe, 2000).
The specification defines a many outstanding functions in data communication networks’ management addressed by IBA. The architectural development was driven by two industry-focused objectives: future I/O and next generation I/O (Kashyap, 2006). General Overview of the IB Architecture The specification associates the size of the IB architecture to the two objectives driving the developmental process: (i) providing enough room for vendor differentiation and new invention; and (ii) to advance to broader and much more robust enterprise-level facilities while enhancing the shift into smaller and economical server systems (Gregory, 2010).
According to IBTA, the IBA specification is a description of a high-end technology that facilitates the development of System Area Networks (SAN) by enhancing the interconnection of I/O nodes and processor nodes without relying on the processor platform or operating system (IBTA, 2001). The IB architecture is a perfect solution for mass application as the architecture allows multiple connections per server hence addressing the problem associated with the shrinking form factors of the server.
The IBA I/O specification has been found to be effective in high volume severs such as PCI which include network and storage adapters essential for attachment to peripheral buses such as Ethernet, and fiber channel. The architecture offers a lot in cluster server computing in terms of high bandwidths and lower latency enhanced by the new platforms in programming and communication semantics (IBTA, 2001). Enterprises therefore benefit from a higher performance, interoperability and a reduced cost.
However, the adoption of the technology also comes with a set of challenges regarding the method of developing and incorporating server products as well as the development of enterprise architectures. Unlike other network technologies, IBA addresses a wide range of server markets such as networking, mainframe computing, storage, server platforms and communications. IB architecture facilitates host’s interprocessor communication enabling network managers to incorporate additional features for the purpose of enhancing separation and achievement of deterministic behaviour.
The architecture also allows flexible management mechanisms that make it possible for single hosting without depending on expensive fabric managers. Unanticipated burden in the facilitation of high-end SAN can therefore be avoided (Mellanox Technologies Inc). The IBA SAN infrastructure is focused on management and communication and is dedicated to enhancing interprocessor and I/O communication for a single or multiple computer systems. IBA features IP supportive characteristics that enhance the interconnection to the intranet, internet or link to isolated computer systems (Kashyap, 2006).
Its fabric enhances end node communication through a number of paths and through the many ports. It helps solve the problem of high over ahead arising from the use of communicating protocols. Two key environments are defined: (i) chassis-to-chassis, characterized by the interconnection of computers to the external WAN/LAN devices for access; (ii) module-to-module, referring to computers systems based on add-in slots of the I/O modules (Joe, 2000). Why InfiniBand?
Following the rapid evolution of computing technologies; enterprise computing, data servers and application processing have been subject to excessive burden especially due to the growth of the internet (IBTA, 2001). High-order computing technologies in the context of fail-safe, full time availability and clustering demand a high ability of moving data between I/O devices and processor nodes or between processor nodes themselves. In this regard, a lower latency and higher bandwidth is paramount since more functionality is directed to the I/O devices.
Moreover, these developments have increased the demand for higher isolation, service quality and more protection. The IB architecture provides solution to these and other issues. Working of the InfiniBand Architecture IBTA (2001) described in details how the IB architecture works to accomplish its objectives. The IBA specification defines IB architecture as a network constituting of subnets that are usually interconnected to each other by the use of IB routers. Every IB switch or node connects to each other directly or else to one or more switches.
Therefore, channel adapters (CAs) facilitate interface between the link and each of the IB unit (Gregory, 2010). It is imperative to note that for each unit, numerous CAs; each featuring one or multiple ports connected to the fabric, are supported by the architecture. In the context of the IB fabric, each CA is treated as a node. Data is usually sent to the endpoints (ports). The ports have many Queue pairs (QPs) which can be addressed directly from a remote peer. The QP number is usually considered part of the address when transferring data. The architecture supports both datagram and connection-oriented service in between the ports.
The port identifiers and QP number do identify the peer. However, there normally are some exceptions: (i) QP numbers cease to be applied in situations of multicast packets; and (ii) QP numbers are also not used in the case of raw datagram mode. The local or global identifier then identifies the ports in a data packet. The global identifier (GID) only does so when communication occurs across a subnet. The global identifier features an extended unique identifier (EUI) 64 compliant element: global unique identifier (GUID) which persistently identifies the port and cannot be used as a packet address (IBTA, 2001).
On the other had, the local identifier is a 16-bit value normally assigned following the activation of the port. Although not mandatory, the subnet manager works in maintaining the values of the local identifier the same across reboot. Therefore, the subnet manager does the work of both the local and global identifiers. Each IB subnet constitutes of not less than one element of the subnet manger (Mellanox Technologies Inc). The subnet manager programmes the switches in order enable packet routing from one end to the other. IB routers route packets between the subnets of the architecture as influenced by the GIDs.
The local identifier is always essential in a packet since it plays an immense role in determining the destination in subnet (Kashyap, 2006). Like switches, CAs has numerous. Unlike a switch port, a CA port is addressed by a single or a range of local identifiers assigned to it. A switch port is open to other end nodes. Every port has its own buffers with the virtual lanes controlling the buffering operation. There is about sixteen virtual lanes. The virtual lanes offer a mechanism that makes it possible to create numerous virtual links in a physical link (IBTA, 2001).
However, it is important that each of the ports be compatible with the virtual lane number fifteen, which is reserved only for the subnet management datagram. The subnet manager configures the virtual lane normally used by the packet and this occurs in the channel adapter tables. This configuration is determined according to every service level packet. Important aspects such as partitioning are defined, in that, all packets apart from raw datagram contain the partition key. Q_keys are useful in the enforcement of access rights for both the unreliable and reliable datagram connections.
Multicast, referring to packet replication across many output ports is also described. Data Transfer over IB Architecture The Reliable connection (RC) service refers to a process where messages are relayed reliably between end nodes which have an established connection (IBTA, 2001). The service is a key requirement for host CA but optional for device CA. Assembly and segmentation is taken care of by the implementation of the channel adapter. Like the RC, the Unreliable connected (UC) service allows message to be relayed between end nodes once there is established connection.
However, RC is optional and does not guarantee transmission. The Unreliable datagram (UD) service allows a single packet message destined for an end node to be sent although it does not guarantee transmission. The Ether type raw datagram and the Raw IPv6 datagram refer to services that transfer only the local transport header message. The service allows traversing of an IB architecture network by packets that use transport layers that are not the IB architecture. This service is applied in network interfaces or routers for the purpose of sending packets (with very minimal modification) to other media.
The reliable and unreliable datagram and the reliable and unreliable connected techniques are known as IB transports while the raw IPv6 and the raw Ether type are referred to as raw datagram (Gregory, 2010). Components of the IB Architecture The specification defines the IBA subnet which is the smallest but a complete unit of the architecture. This is expandable into an IB network by manner of connecting numerous subnets through routers (Joe, 2000). A subnet encompasses of various elements including, links, switches, a subnet manager and end nodes. On the other hand, links are defined as the communication cannels of the IB architecture.
They are of either optical fiber or copper nature and are point-to-point bidirectional channels. The v1. 0 specification gives the signaling rate of links as 2. 5 Baud with later releases expected to demonstrate faster speeds (Mellanox Technologies Inc). Link widths include: 1X, 4X and 12X. Switches, which serve in routing messages from their origin to their destination according to routing tables, are also defined. The architecture’s vendor determines the routing table content as well as the organization format. The specification defines Channel adapters (CA), as the interface that exists between the link and an end node (Joe, 2000).
There is a profound difference among channel adapters for a wide range of devices and that of the host since unlike the device CA; the host CA includes a defined software interface. HCA’s position within a host system is usually vendor based. End nodes are defined as the core origin and sink of IB architecture communication. End nodes encompass of devices or host systems. The specification defines many form factors including standard, tall, wide and tall wide for end node devices. Subnet management defines the management of the IB architecture as executed through agents and managers (Mellanox Technologies Inc).
Agents encompass of passive elements that act in response to manager messages while managers refers to the active elements. Each IB architecture subnet has one major subnet manager normally contained in a switch or an end node responsible for the discovery and initialization of the network and determination of path MTUs, loading the routing tables’ switches, as well as the assignment of the local IDs to each element. The specification describes various layers featured in the architecture and which operate independently from each other.
The link layer together with the transport layer, form the basis for the IB architecture. It includes pint-to-point connection features, packet layout and subnet switching (Kashyap, 2006). Physical layers encompass of receptacles and cables for copper and fiber media, hot swap features and backplane connectors. The network layer facilitates packet routing between subnets. The transport layer facilitates channel multiplexing, partitioning, reliable and unreliable datagram, reliable and unreliable connection, raw datagram as well as segmentation of transport data when sending and assembly when receiving.