The High Resolution Client displays one of the three-dimensional bifurcations. Isofurfaces show the pressure within the artery while arrows show the direction and magnitude of the velocity of blood flow. © 2006 The University of Chicago/Argonne National Laboratory

"We want to be able to study diseases like atherosclerosis in more detail," said George Karniadakis from Brown University. "What's the effect on the arteries in the brain if there's atherosclerosis, or plaque buildup, in the carotid artery? Individual arteries have been studied by many research groups, but they have to specify artificial conditions at the edges of each artery." With a global blood circulation model such guesses at conditions are not necessary, and scientists can see the effect of a disturbance in blood flow anywhere in the body. Such simulations may impact many areas of medical research.

Over the past 10 years, Karniadakis and his research group developed a computer program that simulates the largest 55 arteries and 27 bifurcations, or places where an artery splits into two, in the tree. Until last year, the simulations were run on only one computing resource at a time, modeling at most two or three branching sites at once. But in early 2005, the collaboration proposed harnessing the combined power of the TeraGrid's many computing resources to simulate, and visualize in real time, more than a dozen branching sites.

The Low Resolution Client displays a model of the 55 major arteries colored by the data from the one-dimensional simulation. On the left are low resolution models of each of the three-dimensional bifurcations being simulated. Selecting one of these models will cause a high resolution model of that bifurcation to be displayed in the High Resolution Client. © 2006 The University of Chicago/Argonne National Laboratory

"MPICH-G2 takes the MPICH library—an implementation of the MPI standard—and puts it together with the Globus Toolkit," explained Karonis. "From the application scientist's point of view, it looks like the program is running on one enormous machine."

Once the simulation program was working with MPICH-G2, the next step was to create a real-time, remote visualization of the blood flow through the arterial tree. Michael Papka's group at Argonne National Laboratory and The University of Chicago was tasked with developing a visualization that would educate visitors to SC|05 both about the arterial tree and the inner workings of the TeraGrid.

"What we showed at SC|05 is not the way the scientists will typically work with the data," said Papka. Just because the visualizations were educational didn't mean they weren't useful to the researchers, however. The TeraGrid behind-the-scenes visualization gave the collaboration vital, up-to-the-minute information about the status of all the resources and network connections used in the demonstration.

"We put a lot of network monitoring in so we could see where the jobs were starting," noted Papka. "We could instantly see what sites were running and where bottlenecks were without going through a bunch of log files."

The visualization application running on the tiled display at SC|05. The center tile displays the Low Resolution Client; above that is a display of the network and performance monitoring information; and the remaining tiles display instances of the High Resolution Client. © 2006 The University of Chicago/Argonne National Laboratory

The blood flow part of the visualization combined a one-dimensional picture of the whole arterial tree with low- and high-resolution three-dimensional images of each bifurcation. When a certain low-resolution bifurcation was selected, a high-resolution image was computed and displayed if available.

"In the future we'd like to make it so that the visualization is part of the scientific analysis as well," added Papka. For this to become commonplace, he noted, the computational environment needs to become more robust. The average scientist doesn't have the nine-panel tiled LCD display, 10 Gb network link and group of dedicated experts that was available at SC|05.

One additional hurdle will need to be surmounted before the arterial tree simulation, and other computations using multiple TeraGrid resources, can become routine. Scheduling jobs to run at the same time on multiple sites, or co-scheduling, is a technical and policy challenge just now being faced by the project. Programs such as the arterial tree simulation have to run at the same time on multiple resources so that the different instances can communicate with each other. But each resource on the TeraGrid is scheduled separately, so if one computation is submitted to multiple sites, they all must wait until all sites are ready to begin. As researchers are only allocated a certain number of hours on each resource, long delays create big problems. For the SC|05 demonstration special arrangements were made, but the TeraGrid is now addressing this issue for the future.

Steve Dong from Brown University and Joseph Insley from Argonne National Laboratory and The University of Chicago also collaborated on the demonstration. Computing resources from the Pittsburgh Supercomputing Center, the Texas Advanced Computing Center, the National Center for Supercomputing Applications, the San Diego Supercomputer Center and the UK's Computer Services for Academic Research were used. Visualization processing was completed at the University of Chicago and Argonne National Laboratory.

Learn more at the TeraGrid Web site.