Scratch the surface of just about any science fiction plot and you will find some combination of aliens and massively powerful computers, with which the characters interact to save or destroy the world.

This is probably why the SETI@home project run through Berkley University in the US has inspired nearly 4 million computer owners around their world to offer up some of their processing power to scanning the cosmos for signs of intelligent life.

Four years after its launch, the SETI@home software is running across no less than 127 operating systems in 226 different countries, and has managed to spew out half a billion results.

Even before the Linux-based Beowulf project kicked-off in the mid-nineties, distributed computing was allowing researches to overcome limitations in processor size. COWs (Cluster of Workstations), NOWs (Network of Workstations) and PoPCs (Piles of PCs) have seen computers linked-up and problems broken down, so as to be solved over these distributed -grid" architectures.

However, a Grid, and the Grid are different concepts altogether, according to Argonne National Labs' Grid guru Ian Foster. Whereas distributed or cluster computing pools the processing resources of a series of computers of all shapes and sizes, grid computing takes this concept one step further, allowing for detailed scheduling, high levels of service and distributed control.

"The promise of grid computing is, in the back end, you can start doing much more dynamic resource provisioning than was possible with cluster computing, and then on the front end you can make it possible to acquire resources from different locations, rather than dealing with static operations," Foster explains.

Closer to home, Rajkumar Buyya, assistant professor and lecturer at Melbourne University makes the following distinction.

"Cluster computing is about resources aggregation in a single administrative domain," he explains. "Grid computing in about resource sharing and aggregation across multiple domains."

Having recently completed a doctoral thesis outlining the economic paradigm which might underpin such a system in the commercial environment, Buyya points to the Grid's scheduling capabilities as the most crucial difference between grid and cluster computing.

"As part of the Gridbus project, We are developing the software to support a grid bank, where resources from different organisations can become part of the cooperative environment," Buyya says. "We need to design a system where deadline, budget and service level requirements can be guaranteed."

Seeking the components

"One of our primary goals right now is getting a grip on what the technology is, and what is out there already," Abel says. "This is a niche which Australia can make its own."

Broadly speaking, work on grid computing in Australia can be divided into two areas; the high-powered networks which will enable the grids to operate, and software components which includes resource brokering systems such as Monash University's Nimrod/G resource broker as well as applications development.

David Abramson, head of the School of Computer Science and Software Engineering at Monash, traces the Australian roots of distributed computing back to a Commonwealth Government-sponsored Distributed Systems Technology Centre (DSTC), founded in 1994.

"Originally I was interested in providing an environment where scientists and engineers could use distributed computers to solve big problems faster," Abramson says. "We were using parametric computing, similar to what runs behind an Excel spreadsheet."

It was about this time that Nimrod (a precursor to Monash's NimrodG) was born. Written in Python, the software allowed large algorithms to be processed across a range of otherwise inactive computers, and complied the results.

"It caused a real paradigm shift - at that time there were just problems that were just too big," Abramson says. "We were able to solve them using idle workstations."

At about the same time this first incarnation of Nimrod was being spun-off into a company by the name of Active Tools, the Globus project started gaining the attention of the international academic community. Designed to assist in the development and running of distributed systems, Globus was quickly picked up by universities around the world.

These days, Nimrod/G provides an architecture for a resource management and scheduling system in a global computational grid, and Globus has become the accepted standard for the middleware which enables such grids to operate.

Making the connections

Once built, the GrangeNet infrastructure will provide a ten gigabit backbone running from Melbourne to Sydney, via Canberra, with a further five gigabit connection into Brisbane.

Such infrastructure will allow the integration of high-performance computing services, support distributed user communities, and enable Australian Universities to participate in Grid computing test beds running in the Asia-Pacific region, North America and Europe.

Dr Marcus Buchhorn, grid services coordinator for GrangeNet, says the resources will then be available for private and public sector groups that are interested in using the networked service to develop or utilise applications.

"Within Australia, GrangeNet is in the process of setting up working groups and getting the infrastructure together so that the researchers working on grid projects get access to the resources they need," Buchhorn says.

Creating a Grid bank

While the cutting edge development in Australia is largely being carried out in universities, cluster computing and even grid computing is already making its presence felt in the commercial sector.

Kevin Mayo, enterprise architect at Sun Microsystems, says many of the key components for the creation of a computational grid are available off-the-shelf and are already being implemented by the tertiary education and commercial sectors in Australia.

"The software component takes a bunch of machines and connects them together so that you can logically treat them as a single machine and is now accessible through what I like to call a blue curtain," Mayo says. "This is portal software that makes operations within the grid engine transparent and readily accessible."

According to Mayo, IT buyers in the corporate and government sectors are prepared to forgo a certain amount of a system's overall processing power in exchange for ease-of-use and a painless implementation.

"Grid computing is no longer a black art now that you can buy the tools and a grid together with relative ease, and handle all the scheduling requirements through a portal software," Mayo explains.

However, researchers such as Buyya and Abramson have a vision of grid computing which goes beyond amassing processing power within a single university or corporation.

"We are developing the software to support a grid bank, where resources from different organisations can become part of the cooperative environment," Buyya says. "If you are looking at taking grid computing to the global level and make it work commercially, participants will need to offer their services for a profit, so the infrastructure will allow for resource sharing and grid management."

According to such a vision, participants are linked through a resources broker which monitors how much processing power is being used, and who it is being used by. Participants are able to gain credits when their systems are utilised by others, and use these later to purchase their own processing requirements.

"The key is to offer, select and aggregate resources based on individual requirements," Buyya explains. "If grid computing is to work commercially, participants need to be able to operate within deadlines and budgets. Commercially, we need to provide access to greater levels of bandwidth and processing power at a premium."

Buyya is also looking at fast-tracking the process by which applications are implemented across the grid system, and believes Australia is well placed to crack the quality of service requirements of commercial grid systems.

Beware the cluster hog

Luckily, Dr Kopp found himself at Monash University, where his research became a simulations guinea pig for Abramson's recently-commissioned clusters.

"I wanted to buy the biggest number-crunching machine the budget could support," Kopp explains. "I started with a 20 CPU cluster, hogged it completely, and kept hogging it, no matter how many CPUs David [Abramson] got the budget to add."

As a result, Kopp was able to take his research beyond the boundary simulations he had originally planned and was eventually able to map out the problems in considerable detail.

Despite his initial scepticism, Kopp soon overcame concerns regarding cluster software fragility, becoming a self-confessed departmental cluster hog, as well as an enthusiastic evangelist of the technology.

According to the CSIRO's David Abel, Kopp's previous scepticism is not uncommon amongst researchers, and this fact continues to present a significant hurdle to the uptake and development of grid computing.

"Awareness is another issue we need to overcome by making business and the science community aware of the potential of grid computing," Able says, pointing out that heightened interest in, and understanding of grid computing should lead to an increase in applications development in the area.

Martin Sevior, a grid computing convert and associate professor at the Melbourne University School of Physics, is currently involved in the development of applications aimed at solving high-energy physics problems.

"Basically I investigated it [grid computing] closely and decided it really did work," Sevior says. "We're currently involved in two very large-scale experiments; one in Japan that has already generated in the order of ten terabytes of data, an amount which we expect to increase by a factor of ten."

Using grid computing, Sevior believes it will take approximately one year to complete computations which might have otherwise taken ten. Ultimately, he hopes it will help him to locate proof of theoretical physics measure known Higgs Boson, which is thought to provide particles with their mass.

"In order to find it we need to create particle collisions at energies greater than anything that currently exists, and examine trillions of collisions. Without grid computing we would be an order of magnitude short of time and money," Sevior explains.

At this stage, the diverse but interconnected Australian-based developments in grid computing benefit from what Able describes as a "remarkable" but largely informal degree of cooperation. Through projects such as Grangenet, and what Dr Buchhorn describes as a certain degree of "generosity" with respect to grid computing resources, Australia could well carve a niche at the forefront of applications development in this area. Not to mention convert the "bang for buck" advantages of grid computing to advances in other areas of scientific endeavour.

"We need to combine the disciplinary expertise with the core academic strengths," Able says. "Collaborations of the kind we are already seeing are essential. If we compete as a set of small teams we will get left in the dust."

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.