Dealing with unimaginable amounts of data can be extremely tough for any organisation.
The Bureau of Meteorology's Web site has approximately 50,000 pages and the amount of data it collects is almost doubling every two years.
Needless to say, with such a massive amount of information, the Bureau's storage demands are huge and complex.
Peter Gigliotti, assistant director of central operations and systems at the Bureau of Meteorology in Australia, admits it's a challenge to find new technologies which allow it to manage its data storage.
He says that because of the large amount of data which passes through the Bureau each day, it is often a case of simply keeping its head above water when it comes to storage.
Currently the Bureau of Meteoreology shares two Storage Tech silos with the Commonwealth Scientific Industrial Research Organisation (CSIRO). Each silo holds about 120TB of data.
For its operational systems the organisation stores data on a combination of storage area networks including an HP XP 256 and a 9176 Storage Tech Disk Array.
To deal with the Bureau's masses of information, Gigliotti says it's important for it to use state-of-the-art infrastructure. Because of this, he believes the organisation's storage needs are also different to those of most commercial businesses.
"In essence, most people use large silos to back up data," he explains. "We actually use them so that our staff can access the data in near real time.
"It's not just part of the back up service. It's just the size of the data and how we add and subtract to it that makes our storage facilities unique."
According to Gigliotti, the biggest storage issue the Bureau of Meteorology faces is transferring data. When doing so "communications can become almost a bottleneck," he says.
In an effort to solve this problem the bureau uses a high-speed network between its super computer and the major data storage units and servers that distribute data. "In essence we try all our data intensive operations on that network before it goes out to users," he explains.
This method isn't cheap either. "With the data explosion we've got, cost per megabyte becomes a problem and so does access time," he says. "Once you start talking about terabytes of data it becomes an even bigger problem. We are now planning for petabytes of data and it is an issue for us in terms of where technology is going to help out in increasing the capacities of our systems."
Nevertheless, Gigliotti is confident developing technologies will keep up.
"We rely on improvements in technology to make sure we don't drown in all that data," he says. "It's really all about maintaining a balance between computing capacity, communications, and data storage." In the future, Gigliotti expects data management to play a huge role in the way the Bureau stores its information.
"Because people are demanding so much data...and although the cost of data storage is decreasing, the actual amount of information is increasing. For this reason people will have to spend more money," he asserts. "What's important for people to ask is: how do you store this data long term? and if you change storage systems how do you do it?"
