Network Tracing in a Dynamic Network Multi-Nodal System
Original Publication Date: 2009-Oct-20
Included in the Prior Art Database: 2009-Oct-20
Network tracing in a dynamic network multi-nodal system is described.
Virtualization is a hot topic in the IT industry today. Everything is being virtualized. CPUs, hardrives, systems and even entire networks. Case in point is the new Blue Gene*/Q machine in the works for the DOE. Early Blue Gene systems have five different physical networks connecting the components of the system. Each network has unique characteristics making it more suitable for different types of task. As a whole, the group of networks offer ideal optimization options as well as redundancy for fault tolerance,
In the future Blue Gene/Q system, the multiple physical networks are being replaced by a single five dimensional (5d) torus. To maintain the advantages of having a multiple optimized networks in a single machine, some of the other networks, like the tree, are being virtualized over the 5d torus. In this configuration, each node shares its network bandwidth with traffic routed to other nodes, potentially reducing the effective bandwidth. The effect will be passed on to any networks virtualized over this physical network.
Debugging distributed and parallel applications often involves taking traces of communications between processes. This can be done with tools such as tcpdump/wireshark. This is a daunting task on massively parallel, multi network systems such as Blue Gene. In such systems, there are multiple physical and/or virtual network and they are running many user programs. Deciding what traffic to capture from where is very challenging.
Consider a simple, single node example: A user wished to trace all communications traffic to and from job. Current technology would involve the user invoking a tool such as tcdump specifying a filter string. The filter string causes tcpdump to include/exclude certain packets, i.e,
tcpdump -i 1 tcp port not 22
This causes all network traffic on interface 1, which is not destined to tcp port 22, to be printed out. This can be a huge amount of data, and it is the data over the entire interface, not
just for a specific job.
The problem is even worse on a multinodal/multi-network system. In addition to having much more traffic, tracing the data will have an effect on the performance of the system. If there are multiple physical networks, and traffic can be dynamically routed to any of the networks, then no tracing tools currently available can correctly perform the trace.
What is needed are ways to enable communications tracing on multi nodal, multi network systems with multiple physical and/or virtual networks. This disclosure covers two ideas:
The first invention is a method and apparatus to provide communications trace information for a given job running on a cluster style system that has multiple physical or virtual networks. This system will enable tracing the network infrastructure (physical and/or virtual networks), extracting only the communications information related to the selected job or job...