Saturday, September 17, 2011

Same storm, different state

We are in Branchburg, New Jersey. I last posted here just before Labor Day, thinking that we were very much on the downside of the operation and that I would be posting here again in perhaps another week, just as we were ready to leave. As it turned out, things took a weird turn at 8:30 Tuesday morning, and we've been running flat out ever since. This is literally the first break I've had in which to post, and our first full day off since we left Florida on August 23rd.

On Tuesday morning I learned on a conference call that the technology leadership on the New Jersey relief operation for Hurricane Irene was transitioning out on Sunday. That would leave a leadership void on an operation that was relatively large, still had lots more work in front of it than the North Carolina operation, and was in a politically sensitive area. After the conference call I volunteered that we could move up to the New Jersey operation to fill that void if it would help out, knowing that I had capable people in North Carolina who could finish out the operation there.

It took the Disaster Operations Center (DOC) less than half an hour to call me back to arrange the details. They needed to hear my plan for coverage after our departure, and brief me on the sensitivities of the New Jersey operation. We agreed that we'd remain in North Carolina through mid-day Thursday to wrap some things up and complete the transition, and arrive in New Jersey before the end of the day Friday, to give me a full day of overlap on Saturday with the individual I would be replacing.

We spent the next two days wrapping up documentation and transitioning the NC operation to our teammates there, as well as training them on the handful of arcane tasks that only come up at the very close of an operation. We had a final dinner with the team, and Thursday morning I removed the temporary wiring that had been feeding Odyssey directly from the electrical panel. By 2pm we were on the road and heading north.

We had set a goal for Thursday evening of Stafford, Virginia, just south of Quantico. There was a Wal-Mart there with easy access to a couple of dining options, and satellite imagery showed good parking. Unfortunately, when we arrived we found that they were expanding the store, and more than half the parking lot was fenced off for the construction. It was too tight and high-traffic for comfort, and we decided to press on to the next Wal-Mart north, on the other side of Quantico, in Dumfries.

It was just then that the remains of Tropical Storm Lee caught up with us. While we had been dry for the entire drive up to that point, the heavens opened just as we got back onto I-95, and we pressed forward in driving rain that eventually slowed the entire freeway down to 25mph. It took us nearly an hour to go the 20 miles to Dumfries, where we parked and then fixed ourselves dinner in the bus. Around 9:30 or so the rain let up enough for me to run into the store for a few items, including coffee for the morning, as we were completely out.

Friday we slogged through DC and Baltimore rush hour traffic, making for a nearly six hour drive to Tinton Falls, New Jersey where the relief operation was running from the regional Red Cross chapter office. En route we made a lunch stop at an Olive Garden near Newark, Delaware, and we stopped for fuel on the NJ Turnpike, so that we'd be well covered if we needed to run the generator. We were fortunate to have found a dump station in Virginia on Thursday, so that half-hour chore was already behind us. It was almost 4pm by the time we arrived.

Fortunately, the chapter had an open gravel area in back where we could park Odyssey, and a 20-amp outlet just inside the door we could use, so we were all set for the time we were in Tinton Falls. We pulled the bus around back just as the lot was emptying out for the day. There is a WaWa market just across the street from the chapter, and no fewer than a half dozen Italian restaurants within two miles, so it was really a very nice spot, if not for the mosquitoes in the grassy area next to us.

No sooner had I gotten my feet firmly planted in New Jersey than plans were afoot to move headquarters. The relief operation's role along the Jersey coast had more or less wrapped up, and the rain that Lee dumped on the state exacerbated conditions mostly in the northern and western parts of the state. So our continued operation out of the Jersey Coast Chapter would mean we'd be in their hair as well as a bit too far from the bulk of our operations. By this point, we had already established a distribution warehouse in Edison.

I took over the technology department on Sunday and by Monday morning the facilities people were already out signing a lease on some donated space here in Branchburg. By Monday afternoon we had set a move date of Tuesday evening. By 9am Wednesday we had completed the move, deploying two satellite dishes for connectivity, backing up and moving the server, and redeploying around 50 computers and a half dozen printers.

The move was not without its glitches. We took down the satellite dish, our backup connectivity in Tinton Falls, at 11am, and loaded it, along with as many other pieces of equipment as could then be packed away, onto a rented box truck, to get a head start in Branchburg. The box truck left HQ just after noon, and we sent half the team to Branchburg as well, mostly by car. Not even half an hour had passed before my phone rang -- the box truck had hit a low railroad trestle, shearing the top off.

Neither of the guys in the truck was injured, and the equipment was fine, as it was arranged along the floor and not stacked to the top. But it waylaid the equipment for hours while we scrambled to find another truck to offload the gear, and I had two people who had to wait with the truck for five hours while the rental company came with a tow truck to get it. I had been keeping the Emergency Communications Response Vehicle (ECRV) on standby at the Tinton Falls HQ, since we sent the satellite dish to Branchburg, and when the accident happened I had them stow their dish and roll over to Branchburg to get it on line. We also have some cables, power strips, extension cords, and other material on the ECRV, so the guys who were already in Branchburg waiting for the box truck could at least get started.

When all was said and done, it only cost us an hour or so, since the long pole in the tent had always been moving the server, anyway. We had scheduled an early HQ shut down in Tinton Falls, 5pm from a normal HQ closing time of 7pm, so that we could back up the server before loading it into the box truck on its second trip. By this time, though, the replacement truck was stuck in return traffic and could not make it back to Tinton Falls when the backup completed at 6:30. Louise was able to score a Chevy Traverse SUV that, by lucky coincidence, was being returned to the transportation department just at that moment, and with the seats down the entire 300-lb, two-case server setup fit in the back, along with a handful of cables, cords, and other critical items. Louise drove the server straight here, with the truck following a good hour later.

We had everything wrapped up by 2am, when I locked the building and set the alarm. Other than myself, I let the guys who had started at 8am loose around 11pm or so, and the bulk of the team, who had come in at 11am, stayed until nearly 1:30. The last thing we finished was to bring the satellite dish on line that we had positioned on the roof, which under normal circumstances would have been the first thing to be completed. I opened back up at 6am so we could finish getting all the laptops back on line, and when folks started coming in to HQ at 9am everything was working. Of course, the technology staff were all dragging by early afternoon, and we called it a day at 4pm.

While moving a headquarters like this is old hat for many of us, we had a lot of first-timers on this operation. An HQ move is a rite of passage for disaster technology staff, and it was an eye-opening experience for most of them. This is one of the things that differentiates us from the IT department of any given company with a headquarters this large -- we can plan a move in one day and execute it in a few hours, with little impact to the users. We can do (and have done) this every few days for a month or more. And we can spin up a 100-station headquarters operation from scratch in a matter of a couple of days. Our manager in DC likes to call this "speed to scale."

Now that HQ is fully operational and things have settled down, we've released the ECRV back to its home base and we're rotating staff through days off. By the end of the weekend we will have downsized considerably through the normal attrition of volunteers on fixed-length deployments, and we'll be down to the core group of die-hards who are available to close down the operation. With any luck, we'll all be out of here in two weeks, and we'll be able to make our Trawler show in Baltimore.

The "sensitivities" that I mentioned earlier have made this operation somewhat stressful for me. I've been cleaning up mistakes since I arrived and it turns out that the guy in charge on the ground has had some previous conflict with our leadership at the DOC, putting us in a precarious position. I'm hoping to be back to my normal self after a nice day off; the folks who have worked for me before have already commented, and my fear is that the folks who have not are not getting the leadership they deserve.

5 comments:

  1. we can plan a move in one day and execute it in a few hours, with little impact to the users. We can do (and have done) this every few days for a month or more.

    Wow! Talk about agile computing. That is incredibly impressive!

    ReplyDelete
  2. On a slightly less "techie" note, there are WAAY too many underpasses in North America (I'm including Canada here, too) that are in dire need of sorting out. There should be no reason on earth why a vehicle should ever come in contact with a bridge or overpass. I realise the responsibility rests with the driver to be vigilant, but I've seen far too many situations where the markings were either inaccurate or misleading.

    ReplyDelete
  3. With all that going on I don't see how you could have time to post. The Red Cross had better know what diamonds they have in you guys! Many thanks for doing what you do. Steve & Carol

    ReplyDelete
  4. Politics always complicates stuff that should be straight forward. The RC is very lucky to have you folks in their inventory of deploy-able assets. Kind of like an ERV they don't even have to pay for.....

    ReplyDelete
  5. wow ... I'm in Newark, Delaware, and I think I know the Olive Garden of which you speak! Wish I had known you ... I would have tried to come and meet you.

    ReplyDelete

Share your comments on this post! We currently allow anyone to comment without registering. If you choose to use the "anonymous" option, please add your name or nickname to the bottom of your comment, within the main comment box. Getting feedback signed simply "anonymous" is kind of like having strangers shout things at us on the street: a bit disconcerting. Thanks!