The tech behind the NSW SES

It's been raining non-stop for what feels like weeks now, and it's still coming down in buckets. You bring your knees up closer to your chest for warmth, and cling onto the soaking-wet blanket you salvaged before climbing onto the roof of your house to escape the rising water. The current is moving so fast, taking cars, boats, trees and even people with it. There's shouting in the distance; something about a car underwater.

Andrew Edwards, the NSW SES chief information officer.
(Credit: ZDNet Australia; Rain, Rain and More Rain image by Trin, Royalty Free)

During very bad floods, the NSW State Emergency Service (SES) can receive a call for help every 7.2 seconds to pluck people from their roofs, cut down the tree that has fallen through their living room and perform other general relief operations. How does an organisation, powered primarily by volunteers, keep up with the immense demand? ZDNet Australia went behind the scenes with the SES to see how the organisation uses IT to race against the clock to save lives in an emergency.

At the heart of the NSW SES lies a dizzying core of numbers. Communication is key to keeping people safe. Finding out where people are, and being able to get to them fast can mean the difference between finding them alive or dead.

The SES is powered by 1200 desktops, laptops and netbook devices; 20 servers; 450 mobile broadband cards from Telstra and Optus; 250 network sites, complete with routers, switches, printers and UPS units; 4000 radios; 2000 fixed phones; 1000 mobile phones; 2000 pagers; and 170 satellite phones.

Keeping all of these pieces working is Andrew Edwards, chief information officer of the NSW SES. Edwards believes that while technology is a useful tool for helping people in danger, having accurate data — like maps, weather information and land formations — in a time of crisis is invaluable.

"Technology now touches everyone in our organisation, and whether they realise it or not. Whether it's the GPS units in vehicles, phones and radios, maps on walls, HR information, training management and networks and that type of thing.

"We do a lot of work around mapping, and I try ... where I can to present complex data in a simple way," Edwards said.

Edwards' most critical situation came in 2007, when the city of Newcastle on the New South Wales north coast was being battered by the worst storms in over a century. The storm was caused by a severe low-pressure system that slammed the New South Wales coastline for three days — 300 millimetres of rain fell in just days in some areas. The storm took the lives of 10 people, knocked out power to 4000 homes and saw thousands evacuated and rescued. Most famously, the floods brought the MV Pasha Bulker to ground off Newcastle's Nobbys Beach, where it stayed for almost a month.

The NSW SES oversaw the state's brewing disaster from its 50-seat crisis-management centre in Wollongong, and brought all hands on deck to manage the situation.

Newcastle flooded in a matter of hours, as residents sought to flee the storm-stricken area. Over 18 days, the NSW SES received 72,914 calls to its crisis hotline, and lodged over 19,000 requests for assistance.

The SES constantly mapped and remapped the disaster area to plot those in crisis, and show teams how best to navigate the flood-stricken areas. In less than a month, the SES had created 500 operations maps for its volunteers to use.

Edwards said that it was during this time, when mobile web browsing reached critical mass, that the emergency services' main site was under the most strain.

The SES posted 39 updates to its simple text- and image-based website over the course of the natural disaster, and users hit it like a hurricane. Edwards told ZDNet Australia that around 90GB of data was exchanged via the website, and 374 contact emails were received. The service's busiest day came on 9 June 2007, when the SES was hit by 10,701 website visits and 550 requests for assistance every hour (one every 7.2 seconds), and one web-contact email every 36 minutes.

It is for this reason that Edwards needs to keep the SES' IT elastic. The IT resources of the SES become stretched to breaking point during a crisis — a time that no IT manager can ever predict. Thankfully, Edwards has found a way to fix his load problems with the Amazon cloud.

"We do burst out into the cloud as a result of [the 2007 flood events]," Edwards said, when asked about lessons learned.

"We have our website in the cloud. That's being hosted through Amazon.com's cloud, and our content-management system is provided through a NSW Government platform. We had concerns with information clutter on our old website. It was getting a bit unwieldy, so we decided to cut that all back down. That's going quite well," he added.

After the website's successful push into the cloud, the SES decided to push email up in the same direction, this time with the help of Microsoft.

"We've had some fantastic support from Microsoft in that, as well. We are playing with Azure, but the first thing we moved out into the cloud was our email for our volunteers. Microsoft have given us access at no cost to their Live@Edu platform there. They provide 10,000 members with addresses," he said.

Edwards knows that while having data in a crisis is important, getting it to the teams on the ground is doubly so. Thankfully, Microsoft helped out with that, too.

"We rolled out SharePoint 2007, and called it 'Everyone's Online System' to try and open up a whole new way to do business. We set up every unit with a SharePoint site, and we can have blogs, calendar management, et cetera, and we've also built our own sites around project management we can use."

Edwards told ZDNet Australia that he recognises the need for data security when moving information into the cloud, and is always mindful about what goes where when managing critical infrastructure. For Edwards, the benefits of cloud are clear.

Click through for a complete photo gallery. (Credit: Luke Hopewell/ZDNet Australia)

"I know that there has been a lot of concern about putting things in the cloud, but we looked at it in terms of capacity and the ability to scale up [as we needed it]. What we find in our business in everything that we do, is that day to day, it's pretty static when there's nothing going on, but in an event ... our volumes and capacity of all our systems go through the roof. The cloud and what the cloud is trying to do is a really good business model. I would see a lot more moving into the cloud," he said.

As far as next-generation technology is concerned, Edwards said that the SES has been working on deploying all-conditions technology to improve the response time and ability of its volunteers, including a project to see SES boats equipped with water-resistant GPS units to navigate on submerged streets more effectively.

As with any organisation, staff members are often the biggest sticking point when it comes to the deployment of new technology. Edwards said that as the SES matures in its use of technology, it's important to ensure that staff, and, most importantly, volunteers, don't get left behind by the upgrades, or feel alienated. He says that he hasn't always succeeded in this regard.

"We have a technology divide. We have our young members, who come in and are absolutely fantastic, very tech savvy, and want to engage using technology. Then I have older members that have been in the organisation for a long time, who are used to doing business the same way we are now. So it's about working very closely with engaging all of them, and taking them on a journey with me. This stuff is here, and it's here to help us do our jobs better. It's not just telling them how they should use it; it's about knowing what it can do for you," Edwards said. He added that the SES would love to see more young, technology-savvy people join the organisation to help out.