It's hard to comprehend the sheer volume of information which companies with an online presence need to store, manage, and access.

Businesses have often found that there comes a crunch time when they discover the systems they've cobbled together can no longer cope with the demand being placed upon them.

But how do you find the right solution, and one which will fit within your company's budgetary constraints?

According to IDC, the storage market in Asia/Pacific has been impacted to varying degrees by the recent economic downturn, although this has varied by country. Its research has found that broadly there has been increasing interest in low-cost, shorter-term storage solutions.

"Many users still have the ongoing need for significant and regular increases in storage capacity due to the sheer growth of data," states IDC. "The limitation on expenditure budgets, however, has resulted in a short-term response to buy more low cost direct attached storage (DAS) or network attached storage (NAS) storage at the expense of deferred integration and management costs."

IDC has also found that there is greater awareness of the use of storage by organisations. Its research found that users in many organisations are beginning to see the storage resource as a competitive differentiator, rather than simply as a growing, necessary cost. "It will provide a key reason for user organisations to adopt network storage solutions that offer significant benefits in terms of data access, availability, and reliability to those users that can make a successful transition," according to the report.

Graham Penn, director of storage research for Asia Pacific at IDC says that as a company's e-commerce presence grows there becomes a need to have data available on at least three levels and replicated to minimise the latency in your system.

"You need to provide a lot of storage so whatever you are publishing is stored and accessible," he says. "These days the same data is stored anywhere between nine and 15 times."

Penn believes that for companies focusing on e-commerce it's mandatory they have a very sophisticated storage infrastructure in place.

He says one of the fundamental trends at the moment is companies introducing network storage infrastructure, to maximise availability and minimise business continuity issues.

And, there is also the need to assess what you're trying to achieve with storage, rather than just thinking about the individual products that are available.

David Templeton, marketing manager at vendor/integrator Storage Tek Systems thinks businesses need to look at some level of consolidation of their storage environment.

"Over many years a distributed environment becomes very difficult to manage," Templeton says. "What we're seeing now is it's not the storage costs that are the issue, it's the storage management costs."

As a result, Templeton believes businesses should be focusing on storage management. "When managing your entire storage environment often people look at the isolated costs of the hardware, and do not realise that's only a small part of the costs," he says.

"Unless they consolidate that, and share it, the costs to upgrade and that sort of thing are huge because it's very people intensive. People need to focus on reducing the management costs and not necessarily buying cheap disks."

Key considerations

Graham Schultz, Australia and New Zealand regional manager of SAN infrastructure manufacturer Brocade, says there's a huge uptake of storage worldwide, with e-business being one of its drivers.

"The message here that I think is important is that this is going to continue--no matter what happens with dot-coms, no matter what happens with the economy--storage is growing and companies and e-business organisations need to start putting strategies in place to address what is going to be continued huge growth in this space."

"Online applications like e-business, CRM applications, data warehousing applications, ERP applications...these are very data intensive," says Schultz. "These are all contributing to the massive upswing in the amount of storage that is going to be deployed over the next couple of years."

According to Schultz, there's also a trend away from direct storage, towards SAN and NAS. "The bottom line is that people are seeing this infrastructure and this technology as a way of managing their storage. But, as importantly, making their business more profitable--better availability, better management, requiring less people to manage the storage."

Storage Tek's Templeton thinks organisations need to start looking at more cost effective ways of storing data, simply because they now have so much of it. He also highlights the importance of storing information from your Web site, both for business and legal reasons.

"If a transaction happens then the information needs to be stored for legal reasons for a number of years," he says. "The legality needs to be looked at and the availability and protection of the associated data also needs to be looked at.

"In that environment high availability is vital. Therefore there needs to be a storage infrastructure that can support that--organisations doing business over the Web can lose millions of dollars not to mention customers, if their system is unavailable."

IDC's Penn says e-commerce companies tend to have a relationship, in one form or another, with an ISP. Although some may choose to use their own resources, those with no particular skills in managing storage will find they get a better return on investment by outsourcing.

"Either way, companies have to make sure customers have almost instantaneous access to information and products," he suggests. "Customers should be able to do transactions and put it into some sort of facility where they can do intelligent querying and profiles. Typically that means companies don't just have a single facility."

"There are lots of issues to resolved depending on how ambitious you are in e-business."

There is also a trend for companies to take a wider view about what they're trying to achieve with the storage solutions they put in place.

Storage Tek's Templeton sees businesses starting to take a more integrated approach to dealing with their storage requirements.

"Companies are taking a more holistic approach with storage rather than going out and buying a piece from Vendor A and then Vendor B," he says. "My advice would be to take a holistic approach, and not an isolated application by application approach. This is much more efficient [when] it comes to funding."

Common mistakes

Storage Tek's Templeton believes one of the common mistakes companies doing business online make is taking a piecemeal approach to storage. He says they associate storage with specific applications, rather than looking broadly across the environment to see where efficiencies and economies of scale can be found.

He believes if companies consolidate resources across their whole environment, it can help them reduce their storage management costs.

Nor are technologies such as SAN and NAS either-or solutions.

Storage vendor Veritas' senior systems engineer Simon Elisha says that they co-exist because certain problems can be assisted by using those technologies. "What I mean broadly is that if you're looking at...a high performance application that needs storage allocated to it--such as databases, e-mail--typically we see customers put in a SAN infrastructure."

He says this is because SANs are optimised for high-performance storage. Moving data over a network is very processor intensive, which can lead to performance bottlenecks. SAN and fibre channel hardware offload processing from the host on to the storage network, which shifts data at an optimum rate for storage, while leaving the host processor to serve applications.

For applications such as streaming media, Elisha sees SANs as a good option.

There are also emerging standards, like iSCSI, which involve moving data over IP networks.

NAS, Elisha believes, is useful when you want to do heavy duty file serving. He says often businesses will have to put in a separate dedicated network for the NAS, if they're shifting decent amounts of data. One of the benefits he sees of NAS is that it's easy to plug in and get going quickly, because it's an appliance. However, the downside is that it can be inflexible.

Planning essential

"Most organisations find growth is far quicker than expected. Plan from the beginning: have bigger pipes into servers and allow for the fact that you may have to do backup and recovery every day and perhaps multiple times a day. This is one way to minimise costs--prevent mistakes before they happen."

Likewise, Brocade's Schultz says that any organisation which is finding that it's buying a lot of servers, and therefore lots of storage, will find that through storage consolidation there is a direct spin off and benefit on capital expenditure.

"Because when you start consolidating your storage--be it disk and/or tape--you have the scenario where you're not buying a tape drive for each server, you're not buying a disk for each server, and in some cases having to buy new servers because you've run out of capacity for your direct attached disk."

"The actual cost of buying individual direct-attached disk per server, when you add it all up, is infinitely more expensive in the long run than buying a large storage infrastructure and being able to just add disks to that infrastructure when you need it and not link it to any particular server."

Schultz says Brocade studies carried out in conjunction with KPMG on large organisations, revealed that direct return on investment for storage consolidation were about 296 per-cent with a payback period of four months.

In the same case studies, the back up and restore applications and infrastructure had an ROI of 123 percent and a payback period of nine months. The high availability environment based on the SAN infrastructure once it was deployed at an ROI of 525 percent and a payback period of two months.

Veritas' Elisha is another advocate of carefully assessing what your storage requirements are before the implementation is started. He says back up is often a last thought.

"People will go into a new application and [think] 'we're going to implement a new ecommerce application, it's going to have all these bells and whistles' and people will be very focused on the hardware that it needs, the performance level it has to reach, the new technologies that it's using--they will not think about protection of that data."

"It's most likely they'll think about it in the last few weeks before implementation, or when something goes wrong."

Elisha says at that stage it's very difficult to retrofit an effective solution. He says if storage is considered from the outset, then storage vendors could suggest possible solutions, such as completely backing up data every night, or by tweaking infrastructure in a certain way.

It all comes back to the design process. "Your system and your data is only as good as how available it is, so if, for example, you have a major system crash and you can't get your information back, your business has a very good chance of going out of business," Elisha warns. "And no amount of high tech will save you from that if you haven't planned it in in the first place."

Elisha says businesses often don't think about the availability of their system, but that it can be very easily planned in from the start and planned in to the overall strategy. "They know at the end of the day they're going to have an effective application that's backed up and recoverable and will work in the times that they expect it to work."

Reducing costs

"You talk to an analyst and they'll tell you that 60-70 percent of the costs of a server infrastructure environment is in storage. [Businesses] need to better leverage that storage for online access--make sure it's highly available 24/7 and then apply the business continuity applications to that environment."

Nevertheless, Templeton adds it's also dependent on the application. "Some are mission critical and they require a level of disk mirroring, but that in itself is not sufficient, because any virus attack will also be mirrored. Obviously virus attacks are quite prevalent, so at the end of the day, you need some sort of consolidated backup recovery using tape."

However, businesses shouldn't forget they still have to manage the resources. Brocade's Schultz says this requires skilled staff which aren't always easy to find and are also costly. "With the right infrastructure you can keep the same head count and increase your storage."

Switched SANs

Brocade's Schultz says any organisations challenged by pressures such as large amounts of data tend to find a switched SAN a viable solution. "Any company with lots of information, and typically e-business client applications may have lots of storage requirements," he says.

The other issue is availability, where information has to be available to customers visiting your site, no matter what time of the day or night. Schultz says availability can also create other issues, such as how to carry out maintenance on the system.

Yet another critical driver for businesses to deploy a storage infrastructure that allows them to just add storage as a utility. "You don't want to have to go buy a new server because you haven't got enough space for your storage," he says.

"You just want to be able to add storage as, and when, you need it. You can have your servers just tapping in to that utility in different amounts depending on what their application may be. That is the high level value of a SAN--to be able to deploy and allocate storage resources seamlessly without interferring with any of the business applications."

He says there are different ways of providing levels of availability and performance, but you need to design the infrastructure to meet your business' needs. Schultz believes the SAN is also evolving, using the example of the core fabric switches which Brocade has recently introduced. These can sit in the middle and fan out to the smaller port count devices around them.

What drives companies to reassess their storage needs?

This is because these businesses find they need to proactively manage their growing storage needs.

The key market drivers are applications that capture, store and access vast amounts of data.

IDC sees these drivers including:

• e-business and increased Web activity;

• e-mail and collaborative computing; data replication becomes a major source of increased storage demand;

• enterprise resource management and CRM systems;

• data warehousing and decision support systems;

• new applications which store video, voice and sound, analogue and real time data, in addition to graphics and other images;

• businesses expanding their reach out towards their customers or backwards down the supply chain.

Josh Mehlman, Rebecca Gardiner, and Vivienne Fisher, WebHead Magazine contributed to this report.

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.