Sun power proves shocking for Powercor SAP

It was one of those things that must have sounded like a good idea at the time. But facing a major upgrade of its core SAP environment, Victorian electricity distributor Powercor last year found itself facing a mountain of issues -- and riding a steep learning curve to testing management -- after an abortive go-live effort stalled its major systems upgrade.

The company had been using SAP R/3 since 1997, and it "was extremely stable", according to Stephen Comer, Powercor's manager for SAP development. Comer told recent attendees at Eventful Management's Mastering SAP Technologies conference that Powercor had "no unplanned outages for three to four years" and it was "only getting around 70 helpdesk calls per month".

To ensure the system remained new enough to stay covered by SAP support, however, Powercor realised by 2003 that they needed to upgrade their environment. Three years of lobbying finally paid off last year, when the IT team got management go-ahead for a far-reaching project to update the company's core systems from R/3 version 4.6b to SAP ECC6, which combines mySAP ERP 2005 and the mySAP NetWeaver framework.

... to try and pull out a big lot of money for something that seems to be working OK is a bit difficult at times.

James Rennick, manager of SAP Corporate Business Services

To complicate the project further, the decision to simultaneously upgrade the company's Oracle database; SAP Business Warehouse; and introduce an SAP based employee self-service (ESS) portal that would improve delivery of information to Powercor employees, and the more than 3000 contractors that regularly work for the company.

"As well as wanting to upgrade, we had strategic ideas around the portal," said James Rennick, manager of SAP Corporate Business Services with Powercor. "We had a real push within the organisation to expand our footprint beyond the firewall, and make it available to customers and employees. We've also been looking at document management aspects and other projects going on at the moment. Coupled with that, the landscape was significantly more complex than in the past."

Where the Sun doesn't shine
Powercor was also facing the looming consolidation of two geographic information systems (GIS), which was expected to consume considerable staff resources for the best part of two years. The SAP upgrade was moved forward to ensure it could be finished in time.

"We are a capital intensive organisation, and to try and pull out a big lot of money for something that seems to be working OK is a bit difficult at times," Rennick said. "Last year we managed to convince [management] we needed it."

Little did the IT team know, however, just what it was getting into. The reason: upon hearing that the SAP upgrade was finally set to go ahead, the company's infrastructure team pushed hard -- and won approval for -- a simultaneous upgrade from Sun Microsystems Solaris 8 operating system to the newer version 10. Powercor runs SAP on Sun's high-end SunFire F15K server, with development and testing handled on Sun Fire V480 Servers.

Changing operating systems is never easy. However, the team went forward with it, introducing Solaris 10 into the SAP test and development environment.

The SAP modules rolled out quite smoothly, in large part thanks to a project management mentality that was strictly focused on keeping the implementation plain-vanilla. "It's not far from correct to say you needed authorisation from the current head of state of the country before you got approval for modifications," Comer laughed.

However, it wasn't long before the project hit the wall.

"From when the project began in March through April 2006, most of our issues related to Solaris 10," Comer explained. "We would upgrade the box and couldn't get SAP to restart under Solaris 10. We had a lot of issues with it and still have issues. Whether that's lack of experience with Solaris 10 or not I can't say, but this was one of our challenges."

In the past, a strict development and testing regime had ensured that only high-quality, heavily debugged applications made it into production. Bouyed with success, the team had assumed it oculd manage the Solaris 10 and SAP rollout using the same discipline.

However, recurring issues led the team to introduce SAP Solution Manager, a suite of testing and application management tools designed to improve control over SAP deployments.

The system proved invaluable in helping the team work through the myriad problems it encountered. "It's fair to say we significantly underestimated the effort to [manage testing]," Comer conceded. "There were no [vendor support] notes around the sort of stuff we were coming across."

While the Solution Manager environment helped the team work through many of its problems, other issues persisted: Sendmail, for example, didn't work after the upgrade, so the team had to roll back to an older version and eventually adopted SAP's own SMTP functionality.

By the time the system went live on July 21, the team thought it had worked through most of the issues. However, another operating system-related problem had project managers reaching for the Panadol once again: an issue with Solaris 10 user management interrupted the entire workflow underneath the SAP ESS system.

The issue proved so stubborn that despite escalating the issue to SAP in Germany the team was forced to abort the upgrade and frantically roll back to the previous installation. "We really struggled to get it there," said Rennick. "Everyone logged in on Monday morning, but it nearly killed us."

Persistence pays off
Having waited so long and eagerly for the upgrade, the problems that the Powercor team encountered were far from welcome -- particularly since the SAP implementation went very well.

Months later, the upgraded system is up and running, delivering the benefits for which the SAP team had been pushing hard for years. The system has expanded from one to five Powercor operating entities, and has laid a foundation for projects such as a five-year plan that will build on the SAP environment to optimise Powercor's Human Resources operations.

The team has made its peace with Solaris 10, although it still has "issues" with the environment. "We've had an outage every month since then, except for the last three months," Comer explained. "Most of the issues are not related to SAP, but were related to the operating system and our approach to ESS."

While the strictly managed SAP upgrade was "easy", the need to get up to speed with Solution Manager, and issues with transferring previous Basis development skills onto the new NetWeaver architecture, kept the team on its toes. So, too, did fine-tuning of applications, some of which pushed CPU usage to 100 percent when implemented until the team could tweak the installations and bring usage down to sustainable levels.

With the system more or less ticking over, however, the company's SAP environment is now growing steadily as better information continues to swell its core database. Powercor had 428GB of data under management when it started the upgrade, and that database now measures nearly 550GB.

"We're expanding our footprint," said Rennick. "What did work well throughout the project was our approach to developing enhancements. We worked hard with the SAP contacts and resources, and it was a challenge but we got there."