It’s been quiet so far but nevertheless, it’s Atlantic hurricane season. Even with an August lull, it’s doubtful anyone is letting their guard down, including the wireless service providers with networks deployed in the region. Hurricane Sandy late last October hit the coast causing almost immeasurable destruction, exposing weaknesses in wireless networks along the way. The big four all experienced outages that took weeks to recover from. Steve Waken, an assistant vice president responsible for AT&T’s Business Continuity Management Program, has been guiding the carrier’s disaster response efforts since 2005, which led in 2012 to the Department of Homeland Security giving AT&T its Private Sector Preparedness certification. Wireless Week recently spoke with Waken about what’s changed in his approach in the wake of Sandy.
Wireless Week: How do you get ready for a big storm?
Steve Waken: In a lot of places, especially New Jersey and New York, we’re not the incumbent phone company. We have to work with the other carriers when one of our cell sites it out of service because we don’t have connectivity. So, a lot of coordination with them and a lot of coordination with the power companies so we know what their ability is to help us get back in service.
Before Sandy and Irene—and any other hurricane for that matter—we will take an estimate of how many generators we might need, how much fuel we might need, how many meals ready-to-eat, water and other supplies we might need and we position it just outside of the storm area.
We also sandbag our buildings. If you have a cell tower, central office, mobile switching office, you’ll sandbag the doors to make sure everything is sealed up.
The other thing we did is around the employees. We have about 24,000 employees in the area affected by Sandy. Part of this is making sure that they have an opportunity to prepare and that they have extra water and food, potentially a generator. We spend a lot of time educating our employees in what they need to do in preparation. We also have a program where after a disaster we want employees to call in and let us know you’re OK. That way, if somebody goes missing, we know about it and we can send out someone to look for them.
Wireless Week: After Hurricane Sandy, what did AT&T learn that is altering the way it readies its network for severe weather?
Waken: Coordination with the local authorities; when trying to get into the city, if you don’t have the proper credentials the highway patrol or the sheriff is there at the bridge saying “Why do you need to come into the city?” So, we’re making sure the folks we need to come into the network have the proper credentials so they know that we’re there to help recover and not to cause problems. We worked a lot with the City of New York and the State of New Jersey office of preparedness to ask the question, “How can we work with you to help recover? Where do you want us to help restore?” Because you know the police, the fire department, the first responders, they’re all depending on their cell service to be able to recover.
So one of our learnings is we want to be able to continue to foster those relationships. Particularly in New York, where there were mass power outages, we have portable wireless, Cell on Lights Trucks (CoLTs) and Cells on Wheels (CoWs). How can we bring them in so we can provide temporary service for the first responders and the mayor’s office so they can manage the bigger event?
WW: What are some advances AT&T has made in protecting things like backhaul and AC power?
Waken: We have an entire program all about pushing Ethernet out to the cell sites, most of that over fiber. That protects the backhaul if you can get a redundant fiber in there. We’ve got a lot of cell sites on fixed generators. Probably one of the most interesting things is that we’ve got around 30,000 generators that we can use for wireline or wireless and our folks are trained on how to get as many of them pre-positioned before the storm.
The most important thing during a sustained outage is getting fuel to those generators and setting up standard runs where you start at one generator, fill it up, go to the next then the next then the next and you do that on a schedule.
And then you’ve got the understanding of where the most critical cell towers are that can impact your coverage. If you’ve got towers that are relatively close together, you can lose one and the others will pick up the service. But you can’t lose of all of them, so knowing our network. The engineering group and the radio access organization have done a lot of work to understand how that works.
WW: Hurricane Sandy was an extreme case that really tested networks. What natural disaster preparedness practices already in place held up well during Sandy?
Waken: I think the network reliability, one of those things where you don’t just fix it overnight, but I think the way we do it worked very well. You can always make incremental improvements but I think our whole approach to understanding the coverage and where are the priority and critical locations along with what we just discussed about power and backhaul. Can we improve it? We absolutely can. But the improvements are incremental.
I’ll say one of the most significant improvements that started with Irene that we got in a better place with Sandy, is communicating and working with local government. Communications with the media, communications with customers, communications with local authorities, local emergency management and the state public utility commissions; we made a lot of improvements between Irene and Sandy that worked really well and we’ve actually taken that to another level where cover all that regardless the emergency. We have that repeatable process.