Tuesday, October 30, 2012

Hospital Power Outages (Updated 11/1/12, 3/28/13)

I've been getting some good questions about the New York hospitals that lost power and had to shut down. Questions such as, "why did they evacuate?", "why did they lose the generators?", "did the electrical rooms flood?"  I thought I'd try to address them here, in one place.

It's pretty wordy, and I apologize, but I tried to write it so that, if you want, you can learn a little about how an emergency power system works and what it is that I do.

For the TLDR, skip down to "so what happened in New York?".


Background: 
I'm an electrical engineer, specifically a power engineer who designs the electrical systems in buildings. I've worked on many hospital projects, including emergency departments, critical care wards, a NICU, more MRIs than I can count, and the infrastructure at the core of several hospitals.

Disclosures and disclaimers: 
  1. I'm licensed in Washington state; I am not licensed in New York. While the underlying codes and standards are the same, different states and cities adopt their own amendments to the codes. Since I'm not licensed in New York, I will not and cannot claim to be familiar with the local amendments to the codes.
  2. I'm not in New York, and I don't have any first-hand knowledge of the situation.
  3. What follows is a very general description of how hospital power systems usually work. Every building is different, based on when and where it was built.
How a hospital's power system is supposed to work

We're all familiar with the concept of generators. The power goes out, we fire up the generator, we plug in the fridge and laptop. Our office building might have a generator, but it probably just runs the lights in the stairs and maybe keeps the server room running.

In hospitals, the needs in an outage are a lot more complicated. It's not just a matter of getting out of the dark building or keeping the servers up. Now it's keeping the lights on in the operating room, keeping the life support equipment running, running the elevators for the gurneys, powering the computers with everyone's medical charts, the surgical equipment mid-surgery, or the preemie incubators.

We divide the needs up by priority, from "would be really nice to have" to "people will immediately die". The codes and standards tell us how to do this divvying. Each priority group, or "branch", is connected to the generators by its own switch. The switch is there to toggle the branch from being connected to the utility, to being connected to the generator.

So. There's a blackout. Maybe it's a storm, or an earthquake, or a fire at the substation. The hospital's system notices and tells the generators to start. Within ten seconds, the generators are up to speed and the switches are starting to connect their branches to the generators. This happens in order by priority. If the generators start to get overloaded the system stops adding more branches.

But how long can the generators last? Generally speaking, hospitals are supposed to have 3-4 days' worth of fuel. It has to be onsite, they can't just connect it to the natural gas utility and be done, or keep 12 hours' worth and expect to be resupplied by a tanker.  But even 3-4 days might not be enough. If they look around and decide that the utility power isn't going to be back on before they run out of fuel, they're required to start evacuating right away. They use the time they have before their fuel runs out to make an orderly, controlled, calm evacuation.

Where are all these generators and switches? This is the tricky part. Traditionally, the switches and all the big electrical panels are in the basement, where they're not taking up valuable (read: windowed) floor space. The generators are often in a separate building, usually connected by tunnels. This is how it's always been done, and it's how it often still is done.

After Katrina, a whole new chapter of the National Electrical Code was written. It's called COPS, and it tells us how to design the most critical buildings: police and fire stations, trauma centers and designated hospitals. COPS buildings can't have their power systems below the flood plain, they have to be up high where they're unlikely to be knocked out by the flood. But everyone else, even many hospitals (or parts of hospitals), can and do put them in the basement.

Yeah, yeah, so what happened in New York? Why did NYUH evacuate?

Ok, this is completely 100% unfounded speculation on my part, based on two sentences from a press conference last night.  It has zero basis in facts.  With that caveat, here are my guesses:
  • Guess 1: They decided the utility power wasn't going to be back before their fuel ran out, and/or they were going through fuel faster than they expected. If so, they're required to begin the evacuation process, so that it can be completed in an orderly way before their fuel runs out.
  • Guess 2: One of the switches failed. The switches are huge, they're bigger than refrigerators. They're not something you keep spares of, or that you could reasonably swap out if you did.  The switches are a point of failure. Though every truly life-critical thing has more than one branch serving it, if you lose a higher-priority branch you're going to evacuate.
  • Guess 3: Too many generators failed. If the remaining generators couldn't run the highest priority branches, they would need to move out.
Update 11/1/12: I'm seeing reports now that the generator fuel pumps failed at NYUH and Bellevue Hospital. So that would be #3. As for how one prevents that being a shut-down event: you have more generators than you need (a common practice, called "n+1"), and you do careful maintenance. This is not to say that their maintenance, planning, and design wasn't solid, I have no idea. Even with good maintenance and testing, things sometimes just break.

Hope my phonebook of an answer here helps answer your questions. Feel free to ask if there's something else electrical you're wondering about.

Update 3/28/13: The industry has begun reviewing the situation in Manhattan in order to learn from the storm and revise our codes and practices accordingly. Some early findings can be found in this article from the magazine Electrical Construction & Maintenance.

No comments:

Post a Comment