Jim Mileski takes no chances with his LANs. As the administrator for 100 bank branch offices spread throughout the United States, Mileski can't afford to fix network problems on the rebound when, as in some instances, he is thousands of miles away.

Consequently, Mileski, systems and communications specialist at Farm Credit Financial Partners, takes the proactive approach to delivering 24-by-7 uptime for departmental LANs. Among his arsenal of preventive measures: duplicating network equipment, building redundancy into servers, and providing round-the-clock remote management software and after-hours online backups to ensure internal systems keep running smoothly.

Mileski doesn't consider his wide-ranging backup plan to be overkill. Rather, he says he's just doing his job, keeping business going for the bank's farmer customers and the branches that Mileski manages. "Farmers rely on the banks ... and it is our goal to keep them in business," he says.

Like Mileski, a growing cadre of IT execs say that a fastidious approach toward high availability at the departmental level has become essential to keeping business going in this age of highly distributed electronic applications. The advent of ERP (enterprise resource planning) software such as SAP AG's R/3, corporate intranets, electronic commerce and electronic business applications in the form of end-to-end network data exchanges and e-mail are ushering in a new era of RAS--reliability, availability and serviceability--this time on the LAN.

RAS, an old IBM 3090 data center term, is being redefined in the client/server and Internet age. While the goal is the same--100 percent uptime for mission-critical applications and fast recovery in the case of disaster--the approach is very different because the complexity of the problem is more severe on distributed LANs. Unlike the glass house, where there might be a single mainframe to maintain, today's remote environments house disparate systems running Windows NT or Unix variants. Keeping track of the hardware configurations and finding enterprise management software that can accommodate this diversity can be a much more arduous task.

"It has gone from a data center-based, single machine [environment] ... to now [managing] a multitude of technology across a network," says Jon Oltsik, an analyst at Forrester Research. "So the scope of the problem is much harder."

LAN resilience At Farm Credit, Mileski isn't dealing with an intranet, a complex ERP system or even an e-commerce application. His application sounds simple--exchanging data between branch offices--but in fact, there is no assurance that all servers and network connections will always be stable. That's a scenario that eats away at Mileski, who has had server drives go bad on occasion, disrupting productivity. One time, he recalls, one of the branches was hit by lightning that knocked out the CSU/DSU and router. Mileski was able to ship out new equipment the following day, but work at that office was impaired 24 hours too long, he says.

Even if it is for only a few hours, "downtime is very painful," says Mileski, explaining that if even one server with pertinent loan information were to go down, business would come to a screeching halt in some branches. "Loan officers need the information on those databases," he says.

Farm Credit processes loans and provides other banking services to its customers using information stored on Microsoft SQL Server databases within each office. While each of the 100 branch offices located throughout California, Hawaii, Nevada, Arizona, Idaho, Utah, Maine, New Hampshire, Vermont and Massachusetts generally operate independently, information is continually exchanged and shared between the branches across a frame relay connection.

To keep that information flow active around the clock, Mileski is tapping into a number of technologies to neutralize the effect of a defective drive in a server or uncommunicative router, for example. To ensure the quickest recovery possible, Mileski has RAID boxes attached to each Dell Computer PowerEdge 6100 quad-processing system. By next spring, he will deploy NT server clusters, which include two or more servers running in parallel with the same operating system and applications, acting as a backup to each other through high-speed interconnect technology such as Fibre Channel.

Mileski has also opted to leave inactive coax network wiring in place to serve as a backup in the event that the new 100M-bps 10BaseT Ethernet wiring goes down. Finally, he has duplicated the Bay Networks. BayStack 350 Series switch and BayStack Advanced Remote Node routers in the company's service headquarters and at some of the larger branch offices.

While the backup switch and router are available in case of an emergency, Mileski is interested in a more foolproof plan. In the future, he will likely upgrade to Bay Networks' new BayStack 450 family of switches, introduced last month. Bay Networks and other networking vendors such as 3Com are building resilience into their equipment. Similar to a RAID storage box, these new stackable solutions include multilink trunking with at least two active paths.

"If you have a network failure, another [link] is always there, ready to take on the additional load," says Rick Lougee, senior marketing manager for Bay Networks. Conversely, a router in standby mode could take 30 seconds to 1 minute to reconverge and learn about a new path. "When you are talking about [financial] or airline reservation industries, 30 seconds of downtime can cost a half a million dollars," Lougee says.

For smaller companies, the effects of downtime can be more personal. Take Robert MacLean, the SAP project manager at a manufacturing company in Southern California who put R/3 Financial modules into production this past June for tracking purchasing and material management. "I haven't calculated what downtime would cost--probably my job," MacLean says.

As a result, before making R/3 live on his LAN running on two Compaq Computer ProLiant 7000 servers with NT, MacLean tested the ERP system at the Compaq/SAP competency center in Houston and has explicitly followed the rules set in place for proper deployment. So far, so good, MacLean reports.

The cost of business

While downtime can take a toll on the productivity of a financial institution such as Farm Credit, the consequences can be far more dire for a Web storefront such as EJ's Sunglasses, according to Erik Johnson, the company's president. "We are 100 percent on the Internet. When we are down, it is like locking the front door," he says.

The 3-year-old company has grown about 300 percent doing business over the Web. While it is difficult for Johnson to calculate just how much money is at risk if his Web site--at www.ej-sunglasses.com--is inaccessible, a telling measurement is in the amount of customers he would lose.

"We get about 500 people an hour on our Web site, and about 2 percent of them will make a purchase," says Johnson of the Minneapolis-based EJ's Sunglasses. With an outage, "we would potentially be losing future sales." Johnson, who has been working with local ISP (Internet service provider) Vector Internet Services (Visicom) to house his e-commerce site, recently took out an insurance policy in the form of his own Sun Microsystems Ultra 1 Model 170E SPARC-based systems running the Solaris operating system and stocked with 256MB of RAM and two 4GB hard drives. This way, he is assured that there is a reliable system available to handle the workload instead of renting shared space on his ISP's server and exposing the company to the possibility of a systems bottleneck, Johnson explains.

EJ's Sunglasses still counts on Visicom to service the system and keep the Web site alive via a 10M-bps connection to the Internet via two T-3 pipes. As additional protection, Johnson has the ISP back up the server every night onto digital linear tapes. Visicom is also responsible for monitoring the EJ's Sunglasses Web site on a 24-by-7 basis because Johnson doesn't currently have the staff to manage the site.

Similarly, companies in the midst of deploying large ERP suites are finding out that managing the applications and maintaining uptime can be as difficult as the actual deployment. Many are turning to outsourcing partners for help. For instance, a spokeswoman for one large global manufacturer of glass and industrial chemicals who requested anonymity recently went live with an ERP suite from Oracle called Oracle Application and picked Bell Atlantic Network Integration, a managed services organization, to watch over its WAN and LAN. For many companies, enlisting the help of an outside network management company is the only way to ensure 99 percent network availability in a distributed LAN/WAN setup. Times have changed since the days of the glass house.

"The data center wasn't moving anywhere. It was just one big room with lots of stuff in it, and if [administrators] could keep the stuff running, then users were ostensibly doing their work," says Don Montgomery, product manger for BANI's network management services. "But [in data centers] they didn't have smart desktops and SNA controllers. ... The support challenge and the way of supporting the environment is different."

As a result, BANI is often called upon to monitor ERP applications within a LAN and across a WAN from its own network operations center in Frazer. BANI uses tools such as Sun's Domain Manager, Hewlett-Packard's OpenView and router management software from Cisco Systems to alert its clients of a potential problem and allow the clients to respond within minutes.

Sometimes, just monitoring LAN availability isn't enough, however. Therefore, BANI and other network management vendors are beginning to customize services down to the availability of the applications to show how often an individual may tap into a program and what the response times were. Tools such as Compuware's EcoScope and Ganymede Software's Pegasus are among the tools that can help measure and pinpoint glitches deeper into the layers of the LAN.

"We could show on a 7x24 basis a trend when [application] response times on the network reach unacceptable levels and then make recommendations of how to avoid it," Montgomery says.

The standards battle The ongoing efforts by vendors, service providers and IT execs to make the distributed client/server and intranet environments comply with the same RAS standards common to the data center can be tedious work. However, simply maintaining standards is an often overlooked but easy way to start addressing the problem.

"You can eliminate 60 percent of the problem if you make sure there is rigid adherence to [server] standards and defined policies," says David Flawn, vice president of NT marketing for Data General.

Standardize on CPU configuration, disk space, amount of cache and specific controllers--all issues that relate to the revision levels of operating system software. Defining what a high-availability server looks like can weed out potential configuration deviations, which often make restoring an NT server problematic. For example, if an NT server went down and the backup server did not have the exact same configuration, the data would be saved, but the operating system and applications would have to be reinstalled and reconfigured for the new machine.

Farm Credit's Mileski knows that half of his reliability problems stem from trying to manage disparate resources. By standardizing on Dell servers and Bay Network routers and switches, the majority of the issues can be resolved. "If you only have three pieces of equipment, you don't need an army to take care of it," Mileski says.

Mileski keeps extra inventory of the Dell servers and Bay Networks equipment at the Agawam office. That way, in the event there is a bad drive or defective network device at another branch, he can quickly ship out a new piece of hardware without worrying if it is compatible. In addition, he has equipped each branch office with an emergency repair disk to restore the NT operating system and data back to the server. >From that point, Mileski administers the server remotely over the frame relay WAN using a tool called Remotely Possible from Enterprise International. By doubling up on key devices and keeping extra inventory, Mileski's plan might seem overindulgent and costly. But he says the price will never add up to what it would cost the bank in terms of business lost in the event of a systems failure. Another perk: It lets Mileski sleep at night, knowing his networks are keeping pace round-the-clock.

Stepping up departmental uptime Follow these best practices to wring the best performance out of your LAN * Redundancy: Build redundancy into the LAN by deploying at least two of everything, including servers, routers and switches. Duplicate network wiring if possible. * Standardize: Large companies with many branch offices should standardize on server and networking equipment. Keep extra inventory in stock in the case of a hardware failure. * Store and retrieve: Deploy RAID storage devices with departmental servers that enable hot-swapping defective drives without losing any data. Also, back up servers every night onto digital audiotape or digital linear tape drives. * Distributed management: Embrace service-level management, which combines network management tools, remote management software and business metrics to prioritize activities, including applications, performance planning and administrative support. Source: PC Week reporting

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.