Distributed computing—which harnesses the power of multiple CPUs—grew out of scientists’ and academics’ needs for processing power, but it is rapidly developing commercial applications.

One of the hottest tickets in distributed computing is grid computing. According to Ian Foster, associate division director, senior scientist, and head of the distributed systems lab at Argonne National Laboratory, the scientific community realised in the early 1990s that high-speed networks presented an opportunity for resource sharing.

This would allow interpersonal collaboration, distributed data analysis, or access to specialised scientific instrumentation.

This led to the Globus Project, which defines grids as “persistent environments that enable software applications to integrate instruments, displays, computational, and information resources that are managed by diverse organisations in widespread locations.” Foster qualifies this by identifying three distinguishing features of grid computing:

• Coupling resources that span multiple administrative domains;
• Doing that in a way that provides well-defined and non-trivial qualities of service, whether it’s performance, security, or something else; and
• Doing it with standard, open protocols so resources can be incorporated into other systems.
• The Globus Toolkit created at Argonne “is used by essentially all the big science projects across the US and Europe. In Australia as well there’s a lot of interest [and] Japan is just starting a big program,” says Foster.

The idea attracted broader attention when Foster, Carl Kesselman of The University of Chicago, and Steven Tuecke of The University of Southern California, published a paper called The Anatomy of the Grid which showed how this idea of resource sharing could be generalised. IBM picked up the idea and the two organisations collaborated on the Open Grid Services Architecture (OGSA) “which is basically grid meets Web Services”.

According to Foster, an OGSA version of Globus Toolkit will appear late this year, and IBM, Microsoft, Sun, and other companies have committed to OGSA-compliant versions of relevant software.

One of the goals of OGSA is to provide the building blocks required for the construction of autonomic (self-managing) systems. “We’re taking WSDL (Web Services Description Language) and building on that, defining standard WSDL interfaces for things like manageability of resources, lifetime management of services, service data access—so that every resource has a standard format description of its capabilities, standard WSDL interfaces for subscribing to notification events,” says Foster. The next version of the Globus Toolkit will define WSDL interfaces.

The big grid
The first components of a Grid described as “the world’s largest, fastest, most comprehensive, distributed infrastructure for open scientific research” are expected to go online early next year. Funded by the US National Science Foundation, the US$53 million TeraGrid will include 13.6 teraflops of computing power, over 450 terabytes of data storage, and high-resolution visualisation systems, interconnected by a 40Gbps network. The nodes will comprise Linux clusters of Intel-based IBM computers. Sun and Oracle are also involved in the project, and the research partners are the National Center for Supercomputing Applications at the University of Illinois, the San Diego Supercomputer Center at the University of California, Argonne National Laboratory, and the Center for Advanced Computing Research at the California Institute of Technology.

IBM is also involved in several other major Grid projects including the UK National Grid, the North Carolina Bioinformatics Grid, and the University of Pennsylvania Grid.

SGI is another company closely involved with the development of grid computing. The Globus Toolkit was developed entirely on SGI systems, and the company’s hardware ran the first public demonstration of grid technology by Argonne and the University of Southern California. The NASA Information Power Grid is powered exclusively by SGI, and the company claims its systems are used in almost all of the major grid installations in Europe, North America, Japan, and Australia.

Australian universities and other organisations are collaborating to create GrangeNet (GRid And Next GEneration Network), which will connect Melbourne, Canberra, Sydney, and Brisbane with a high-speed backbone network that will provide a platform for grid and other services.

Where’s it going?
Grid computing has been driven by technical areas such as life sciences, automotive design and testing, and electronic design, says Kevin Mayo, Asia-Pacific government technologist at Sun. His company uses the technology internally for chip design, with a campus grid covering three centres. “The tools are mostly there now,” says Mayo, including open source and public domain software. Over 3000 new grids were created with Sun’s software last year, he says. The average size is 40 CPUs, but some have over 1000.

Despite that initial emphasis, Foster says that vendors such as IBM seem to be looking to apply the technology to the notion of sharing heterogenous resources within organisations in order to make more effective use of them.

Early examples of this might be seen in hosting centres. “If you look at most hosting centres they’re incredibly inefficient with all these dedicated stacks of machines for each user. The promise of grid computing is you can start being much more dynamic with resource provision at the back end, and then at the front end you make it possible for people to dynamically acquire resources from different locations rather than rely on the static resources they have,” says Foster.

“And then companies like Sun, IBM, and HP want to move towards utility computing, where you’re starting to break down this tight coupling between producing and consuming of services, and you can achieve economies of scale and hopefully reduce the cost of acquiring and using various kinds of computing services. But that’s maybe some time out, two or three years,” he adds.

Security must also be addressed, and allowing the use of resources without increasing risk is “a challenging issue”.

Grid services could be accessed through a portal, suggests Mayo. Users would be able to indicate the type of hardware needed for their problem and an approximation of the time required, and the portal would act as a broker, discovering the available resources, scheduling the jobs, and handling security issues.

Don’t DIY
Jim Miller is lead program manager, common language runtime (kernel) at Microsoft, but spoke to Technology & Business in a private capacity as someone with long experience of distributed computing, a topic he describes as a “recurrent theme” in IT.

Miller is concerned by the research establishment’s tradition of custom building their own distributed systems, when much is available commercially with broadly similar capabilities, he says. For example, researchers shouldn’t worry about creating new wire formats and protocols, since they can deliver only marginal performance improvements. Instead, existing and emerging standards such as Web Services can provide these functions. Although Web Services uses a text protocol rather than binary, this incurs a performance penalty perhaps as small as 10 percent. It would be useful if commercial and research work was aligned this time, he suggested.

But Miller understands the reluctance to throw away infrastructure that’s been created over a period of years, even if that means ongoing costs escalate.

The focus of distributed computing has shifted from parallelism to the sharing of equipment for computing, he suggests, for example, to take advantage of processing time outside normal hours. Although it is now viable on a global scale, “I don’t think it’s going to make the commercial cut this time round,” he says, due to the need for security and a mechanism to support accounting for payment.

According to Kim Branson, a computational biochemist at CSIRO Health Sciences and Nutrition, grids already have security features to prevent eavesdropping, but they are only as secure as the individual machines doing the work.

Back to time-sharing
Work is underway locally to address the need for a market in grid services. The flagship project of the Grid Computing and Distributed Systems (GRIDS) Laboratory at The University of Melbourne is called GridBus (for grid computing and business). Its initiatives include the development of an architecture for a grid economy (including brokers that can dynamically trade for grid resources in an open market), and GridBank, a system to handle accounting and payments for grid computing.

Branson says such an approach is likely to work in two ways: organisations that donate processing time will receive credits that can be used to buy time on other systems (obviously, time on a modest PC cluster would be less valuable than time on a real supercomputer), and people who cannot afford the capital cost of high-performance computing will be able to purchase time on the grid. He hopes this will allow researchers in countries with limited resources to find cures for local diseases that are not of commercial interest.

But why trade time on one system for another? Branson explains that certain problems are best solved on particular types of system. Some problems benefit from hardware with vector maths capabilities, others need large amounts of memory. A heterogenous grid provides the flexibility to attack different types of problems, he says. Another aspect of the grid economy is the ability to juggle time and cost. Existing scheduling algorithms usually enable jobs to be completed within the requested time period and notional budget, he explains.

Branson works in computer aided drug design. This involves testing compounds to see if they will usefully bind to protein receptors associated with particular diseases. The virtual screening process compares the shapes of the compound and the receptor to ensure a sufficiently sticky fit to block the normal action of the protein. What makes this job difficult is that the receptors are tested against databases of six to eight million compounds.

The size of the database “begins to pose some interesting problems” in a grid environment, he says. It is too large to store copies at every site, so a few copies are located around the world. When a particular set of calculations are scheduled, the system concerned collects information about the relevant compounds from the closest copy of the database. Once this virtual process is completed, promising candidate compounds are tested in the laboratory.

Clustering

Such clusters provide a substantial amount of processing power at comparatively low cost, and are most often found in research environments. Much of the current activity can be traced to NASA’s Beowulf project in the mid-1990s, which harnessed commodity hardware and Linux to create a system capable of processing large data sets. A recent, high-end example of this type of system is the one recently installed at Swinburne University of Technology and described in our case study. Also in Melbourne, CSIRO Health Sciences and Nutrition operates a more modest cluster of 64 Athlon CPUs (a mixture of 1.3 and 1.6GHz chips) for drug design. Branson explains that some of the protein receptors being tested are considered proprietary, and they are tested on the cluster rather than on a grid to ensure security.

Cluster or grid
The appropriateness of clustering depends on the nature of the problem being addressed, as it needs to be capable of being broken down into smaller chunks and the results reassembled, says Mayo. Determining in advance which problems are amenable is largely a case of “holding a wet finger in the air,” he jokes, but anything that is non-symmetric can be a problem. That said, clusters are being used successfully for a variety of applications such as jet engine design and data analysis. For example, a cluster of 1024 Sparc CPUs is used to analyse data collected at the Stanford Linear Accelerator.

Traditional high-performance computing (HPC) vendors went out of business because the market was not big enough to sustain development, says Mayo, so people want to use off-the-shelf components. What’s changing is that whereas universities and other organisations would build their own clusters, they now look to vendors to help build and support the systems. Issues such as availability, security, and upgrades are becoming more important than finding out whether the technology works.

This concept is becoming mainstream. Forrester Research has started talking about Organic IT, which it defines as “computing infrastructure built on cheap, redundant components that automatically shares and manages enterprise computing resources—software, processors, storage, and networks—across applications.” Clearly, this definition can encompass grids and clusters, though Forrester doesn’t predict widespread adoption of Organic IT until 2004.

Cluster-funk
Not everyone is so positive about the benefits of cluster computing. In a reference to the processing power achieved by NEC’s Earth Simulator supercomputer, Bob Bishop, chairman and CEO of SGI says “the world of clustered PCs as a substitute for supercomputing is in tatters right now.” The US has spent US$1 billion on clustering, and “failed miserably”.

Clusters have “fallen behind the leading edge by a factor of 100,” he adds. “It’s called B-grade science . . . causing Australians to think they have high-performance computing when they don’t,” says Bishop, himself an Australian whose first degree was in mathematical physics. He says there are two main problems with using clusters. Firstly, it is hard to administer the software across multiple nodes. Secondly, and more fundamentally, most HPC jobs require a single, large memory space or else communication between the processes becomes a bottleneck.

SGI’s multiprocessor systems use an architecture called NUMAflex, which allows each computer to access memory attached to other nodes with very little increase in latency, thanks to a system of crossbar switches and high-speed cabling.

The cluster model works well where the data is uniform across the space, he says, but that is not true of problems in physics and chemistry, he says. A cluster gives you theoretical horsepower, but you cannot achieve good utilisation of those processor cycles: “You bought them cheaply, but you can’t use them.” Under-funded scientists have been pushed into using clusters, but that’s trapped them in a dead end, suggests Bishop.

But that doesn’t mean he doesn’t see a role for some forms of distributed computing: “Grid computing is definitely a requirement to provide the horsepower for the [scientific and engineering] community,’ he says. SGI’s view is that HPC is not just about number crunching. Visualisation and collaboration with other researchers around the world are both critical. “It’s got to be a visual grid—you need an overlay to provide the immersion,” says Bishop. That will allow Australian scientists and engineers to overcome the tyranny of distance and participate in the large northern hemisphere markets without leaving the country.

“We know how to be globally competitive in sport . . . we need to be globally competitive in science and engineering,” says Bishop, adding that Australian industry will lag behind if we do not build the necessary IT capabilities. Some Australian organisations in the oil and gas, defence, and education sectors are moving in the right direction, but “we haven’t rallied around the history of science [and] mobilised our young scientists,” Bishop asserts. Australia was a world leader in remote connectivity 50 years ago, he says, but we didn’t follow through on that lead.

“Sun gets involved in a lot of collaborative stuff,” says Mayo, such as work being done by the Defence Science and Technology Organisation on next-generation command and control systems. The project involves giving distributed command headquarters knowledge management tools and the ability to share information. “It’s not changing the decision making process, it’s assisting it,” he says.

Foster sees grid computing and clusters as complementary. “Grid computing technology allows you to coordinate and manage multiple distributed resources, and clustering supports a collection of resources under a single administrative domain. For grid computing to move forward, clustering needs to become more sophisticated, so that you can start to externally manage it to do this dynamic provisioning.”

He questions whether managing resources within a cluster and across a grid can be done with the same technologies. “When you talk to IBM, they think it is the case, but when you talk to Sun, they think it’s completely different.”

Harnessing the home user
Miller is not sure about the potential for harnessing the unused power of home PCs, “so far at least no one’s made a business of that. There are security and trust issues, and also interpersonal trust issues trying to convince people.”

As for the idea of taking advantage of wasted internal CPU resources, “it isn’t obvious to me” that spare cycles could be put to good use, he says. For example, Microsoft tried using the PCs on ordinary employees’ desks to stress test software overnight, but found it was not practical as it was too hard to collect the information generated, and there was too much variation in system configurations.

But Foster believes there are some situations where spare CPU cycles can be put to work. US broker Charles Schwab is looking to improve the utilisation of its large systems, which are sized to support double the peak load. “They’ve got a lot of empty space over a 24-hour period. They want to start using that space for computation-intensive things like options pricing and portfolio management activities,” he explains.

Projects such as Folding@Home and SETI@Home have shown that computationally intensive tasks that are amenable to a divide-and-conquer approach can be distributed among a large number of volunteered machines. This is particularly appropriate to situations where huge data sets can be divided into small chunks that can be processed independently. On the pragmatic rather than technical side, key considerations appear to be that the goal is considered worthwhile by a section of the community, the sponsoring organisation is considered trustworthy, and the underlying project is being run on a non-commercial basis.

But as hardware gets faster, new software functions emerge to apply that power to the benefit of the primary user. When you’re not actively using your PC, Miller asks whether you might prefer it to reindex your files so you can find a piece of content more quickly, or work on a problem for someone else.

Rac ’em up
A different approach to clustering—and one directly applicable to business systems—can be found in Oracle’s Real Application Clusters (RAC).

New technology such as the Grid initiative and the availability of high-speed interconnects for Intel-standard computers facilitates distributed computing, says Roland Slee, director, business and technology solutions, Oracle Australia. But there are still challenges for general business computing. “Business systems almost always have . . . the desire to let a large number of people share access to a single set of information,” he says, which means the database layer is often the central component.

While clustering was widely accepted for high-performance computing in the research and scientific community—Oracle uses it internally for compiling large pieces of software—the approaches used prior to Oracle 9i RAC could not address the database issue, he says. RAC makes it possible to apply clustering to mainstream computing, “even mission-critical commercial computing,” claims Slee. Deploying a database-centric application in a distributed environment requires two characteristics, he asserts. The first is the ability to operate with no single point of failure. Since such systems are usually based on commodity hardware that is not engineered for high availability, there is a reasonable chance of one node failing. RAC handles such failures transparently, unlike DB2 or SQL Server, he says. A side effect is the ability to add more nodes without shutting down and reorganising the database.

Secondly, users want the ability to deploy a database that is physically and logically unchanged from the single node version. This allows a seamless move to a distributed environment, since no changes need to be made to the database design or to the application to make it cluster-aware. All that is necessary is to install the RAC option and restart Oracle 9i with some parameter changes.

Apart from offering high availability and allowing incremental growth, clustering promises a better price-performance ratio since four two- or four-way systems are cheaper than one eight- or 16-way server. Furthermore, RAC allows the use of blade servers at the database layer, reducing costs even further.

RAC’s shared disk architecture (using either SAN or NAS) means data is equally accessible from every node since none of the data is owned by a particular node. This means there is no replication within the cluster. “It’s been widely understood that this is impossible, so it’s difficult to convince people that it is possible . . . and now easy,” says Slee. Using Oracle technology, replication can be used to provide fail over to a backup site.

With clustering, “Linux becomes a very attractive system for running mainstream computing,” says Slee, especially as Oracle collaborated with Red Hat to create Red Hat Advanced Server for highly available, highly manageable, and highly scalable systems.

According to Slee, as few as two or three percent of Oracle 8i customers used clustering, but he expects that to increase to at least 50 percent with 9i RAC. Fully clusterable, fault-tolerant systems are needed in the database, application server, HTTP server, and cache layers, and “that’s what Oracle has today and no other vendor has,” he says.

Switch on payments

In Australia, most switches are operated by banks and other financial institutions. Coles Myer operates its own switch, as do telcos, utilities, and other organisations in various geographical markets. Although the Australian market is relatively mature, electronic payments are growing by between eight and 13 percent per year, says Rod Dew, Distra’s sales and marketing manager. Globally, the growth is between 20 and 30 percent, which means banks and other financial institutions may need to double their processing capacity every three years. This makes the scalability of distributed systems very attractive.

Switches traditionally run on small clusters of large mainframes running software created by very few suppliers and using two-phase commit in the database to ensure that each transaction is either completed or does not happen at all. (You wouldn’t want a situation where a customer’s credit card was debited, but a corresponding credit was not applied to the retailer’s account.) These vendors “haven’t cracked some of the fundamental distributed problems,” says Dew. In particular, fault tolerance is handled inelegantly by proprietary and expensive hardware, as opposed to being inherent in the switch application hardware. Additionally, the traditionally monolithic approach limits scalability. “It may be parallel, but not what I’d call massively parallel,” he says. In contrast, Distra provides fault tolerance at the application level.

Distra’s product can run on any Java Virtual Machine, and has been tested on Himalaya, Solaris, Linux, and Tru64. As for performance, an installation with four entry-level Sun servers distributed across two sites can switch 1500 EFT transactions per second with full real-time disaster recovery. This performance would satisfy the largest acquiring operation in Australia and can easily be increased to meet future needs, says Dew.

The design of the underlying transaction processing platform means it can survive “n-1” failures (processing will continue, albeit at a reduced rate, as long as at least one node is still functioning). “If you lose a server, processing continues immediately on another server.” This also means that a single logical switch can be distributed across multiple sites, with practically instantaneous failover even if one of the sites goes offline. Reliability is essential for this type of application: “They’ve got to have this running 100 percent of the time,” says Dew.

Additional benefits are that particular processes can be moved around for convenience (eg, to take a server offline for maintenance or replacement, or to handle a peak load in a particular geographical area), and that granular changes can quickly be made to suit marketing campaigns, etc.

“Early adopters of a pure Java switch will reap benefits,” says Dew. Other developers could add additional functionality around the switch, for example for ATM management or card issuing systems, he explains. Distra’s goal is to create a development community around its transaction- processing environment.

There is significant competition between banks for transaction acquisition, and offering the cheapest rates is a major inducement to retailers. Dew says that the distributed architecture used by Distra’s switch means hardware costs are reduced, software licensing and maintenance costs should be lower, and Java developers are easy to find at a relatively low cost for custom development.

Anything else?
While Distra’s current focus is on retail EFT payments, there is the potential to build other applications on its distributed transaction platform to service markets such as health, travel, wholesale banking, and stock markets.

Sun has other technologies that could be applied to distributed computing, suggests Mayo. JINI was originally floated as a way of handling hardware services such as printing and storage, but it now looks like it will be more useful for software services. In particular, it provides a mechanism that makes services self-describing to the rest of the system.

JavaSpaces provides persistent, network-based storage, allowing communication between independent but related processes. This could be used in a military context by having one system responsible for identifying targets, and another that takes care of shooting them.

The spotter application posts each target in a JavaSpace, and the shooter application watches that space for new targets, decides whether they are in range, and takes the appropriate action. More prosaically, an ATM might perform a transaction and drop the details into a JavaSpace monitored by software responsible for the back-end processing. These routines could behave selectively, for instance picking out transactions associated with a particular bank. If the number of transactions (or targets) starts to approach the limit, you simply add more back-end processes (or shooters) to keep up.

Case study: Swinburne University

Thirty more dual Xeons are located at the Parkes radio telescope, where they are primarily used for real-time data reduction before observations are stored on disk. Even though there is no high-speed link between Melbourne and Parkes, these systems can be used as part of the cluster when they would otherwise be idle.

The cluster is being used for cosmological models involving N-body and hydrodynamic simulations. Another application is engineering simulations of the Square Kilometre Arrayâ€"a proposed international project to build a radio telescope with an effective collecting area of one square kilometre.

Around 20 percent of the computing resource is used outside the Centre, says acting director Professor Brad Gibson. In the research sphere, applications include brain simulation and molecular modelling. There are also public outreach projects such as the Virtual Milky Way, and commercial ventures such as rendering 3D VR movies for museums and observatories.

The Centre has plenty of experience with cluster computing, having previously run an Alpha cluster, followed by a smaller Beowulf cluster of 32 Pentium III CPUs. -The code ran very well, [and was] very fast," says Gibson, hence the new system.

The current cluster was funded by Swinburne, and as part of the Square Kilometre Array project by the Victorian and Commonwealth governments, with help from Dell and Intel.

-The bottom line for us . . . is [the] price-performance ratio," says Gibson. -This is the best compromise and gives us great performance," he adds. The cluster gets into the teraflop range, and when the planned high-speed interconnections are installed, he expects the system to be around the 40th fastest in the world.

The cost of those interconnections is an issue. When you are linking over 100 nodes, -the price sure does go up quickly," says Gibson, even though they are readily affordable for smaller numbers.

While the cluster's primary operating system is Linux, the rendering jobs run on Windows-based software. -It may take an hour or two, but you can bring the nodes up under Windows," says Gibson, which would not be possible with a traditional supercomputer.

One of the newest additions is a quad Itanium II system, which combines 64-bit CPUs and the ability to address 64GB of memory per node. This will be especially useful for cosmological simulations, but the Centre has not ported any of its code to the new system. Gibson says the extra speed could reduce processing times from weeks to days, and the obvious upgrade path for the next few years is to this type of processor.

The code used by the Centre could be ported to any type of computer, but they have been tuned for distributed memory systems like the Beowulf cluster. The tweaking process will continue: -There's no reason why we couldn't improve things 30 to 40 percent," he says, suggesting that it is cheaper to spend a few months working on the code than it would be to get the extra speed via Rolls Royce hardware.

In recent years, the focus for clustering has moved to using large numbers of commodity PCs.

Distributed computing: an executive summary

• Grid computing enables the integration of processing power, data storage and presentation across organisational and (potentially) geographical boundaries.
• Grid computing is based on open standards, including Web Services.
• Some vendors see grid technology as a way to deliver computing as a utility, with a similar model to electricity and water supplies.
• Clustering provides a way to bring large numbers of low-cost CPUs to bear on large-scale problems.
• Not all computationally intensive problems are suited to being solved on a cluster.
• Distributed technologies may provide a way of harnessing spare capacity within an organisation.
• Companies are already delivering systems that take advantage of clustered hardware.

Subscribe now to Australian Technology & Business magazine.

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.