It is September 11th, 2014 today. A day that affected a huge number of people. I’m not going to tell you that I was affected more than the next guy, because I just wasn’t. I can tell you HOW I was affected though. Side-stepping for a moment the issues of human suffering, loss of life, and the sacrifice that literally thousands of Americans made on that day and the following WEEKS, I’d like to focus, if you’ll allow, on the IT administrators who were affected on September 11th, 2001.

Back on September 11, 2001, we didn’t have robust systems or technologies that allowed us to protect our data centers. Actually, that’s not true. We had them, and they were very very expensive. Windows Clustering existed, sure. Was it wide spread in 2001? Not exactly. Did it work? Well… sure. But did it work to protect the systems and the data in a way that mattered on Sept 11, 2001? Largely, no.  So, the point is, that on September 11, 2001, most of the Microsoft Windows-based systems, including Exchange Server and SQL Server, were not replicated or protected by any high availability techniques.

Since 2001, and over the past ten years, many IT suppliers have worked tirelessly to create and proliferate data protection systems. EMC, in particular, has continued to innovate, perhaps more than any other IT supplier. Back in 2005, I moved from Microsoft to EMC… Speaking for EMC, we saw our data protection portfolio erupt with new entries since 9/11. During the 2000’s, we had developed point solutions for file systems, for backup targets, for hosts, for arrays. Some arrays had multiple choices for replication and data protection. At one point in 2009, we had more than 19 different ways to replicate data. Seriously. Nineteen.  But back in 2001, most Microsoft customers were using nothing to protect their systems other than backups to TAPE.  These tapes were place in a large steel box and carried off-site every day to a storage facility several miles from the originating tape drives.

So why was Sept 12th worse than Sept 11th? It was on Sept 12th that the magnitude of the loss actually registered with the survivors. For the souls lost on Sept 11th, it was the first day experiencing the afterworld — a place filled with none of their living friends or family. For those still on Earth, it was also the first day. It was the first day without their mom, or their dad, or their son, or their daughter.  For the IT folks, not only dealing with personal loss, they were now dealing with the loss of their systems.  This wasn’t necessary a sad thing, it was however a challenging thing.

I was a Technical Account Manager at Microsoft back in 2001.  My only account was a global financial services company located, ironically, across the street from the Twin Towers in the World Financial Center (WFC) on the West Side Highway.  Most of this firm’s financial systems were based on IBM mainframes — zSeries and the like.  The storage infrastructure was all EMC Symmetrix and replicated with EMC’s Symmetrix Remote Data Facility (SRDF) to another capable data center on Staten Island.

The systems that were not protected were the global email infrastructure and dozens of SQL Servers. Because their Messaging Bridgehead was located in WFC South while the redundant servers were in WFC North.  Not a great plan, in hind site.  The reason that Sept 12th was worse (for those systems) is that at 10:00pm on Sept 11th, all power was cut to the entire downtown electrical grid.  The entire southern tip of Manhattan went black. All the uninterruptible power systems were depleted by midnight.  All computers were not only down, they could not be brought back up.

All access to WFC had been blocked due to the recovery efforts.  People are, in fact, more important than computers after all.  Somehow we have not forgotten that.  All of the backup tapes were at off-site facilities, but gaining access to them was just as difficult as getting to the servers because the retrieval queues were days deep.  It would take at least five days to retrieve the tapes, and another several days to weed through and restore them.

The “bosses” could not wait.  We set up “camp” at the data center on the other side of New Jersey — I shouldn’t indicate exactly where in an attempt to preserve the privacy of this financial firm.  Let’s just say for now that it was “near Philly”.  This data center had some spare servers and enough SAN storage to allow a complete re-installation of the worldwide email infrastructure.  This effort was expected to take several days. In the end, it required nearly six weeks of 24×7 involvement of more than a dozen people.

So, I went to the data center “near Philly” with several people from the firm’s IT department and began to rebuild their Exchange infrastructure from scratch.  All tallied, we invested nearly 13,000 hours of labor during a five week period.  Every day, it was our pleasure to do the work.  We were delighted to be helping in some way.  With the loss of more than 3,000 lives, we felt privileged to be working even if we weren’t doing anything to help those who had fallen.  The work was not, however, fun.

The effort came to a head for me about five weeks into this 140-hour — not 40-hour — 140-hour work week; this is to say that I had logged 700 hours of work in five weeks.  I was so tired that I could not make reasonable decisions any longer.  I knew I was affected on October 14th following Mass on Sunday morning. I was with my, now, wife in the church’s parking lot.  There was a flea market of sorts raising money for the parish.  I was wandering around the parking lot looking at cups and saucers when my mobile phone rang.  I think it was a Motorola clam-shell device — the StarTac, I think.  It was the Microsoft Global Account Rep for this financial firm.  He asked me “one more favor”.  His name was Rob — great guy.  He asked, “Jim, there’s this Director who deleted a mail message”.  Keep in mind, this is the new system that we had just built, but we were still trying to get remote systems in Japan, Ireland, India, etc. connected and receiving mail.  We were still completely overworked and under water.  So this director had deleted a mail message from his mailbox that was being protected by nightly tape backups managed by an overworked engineer who was manually swapping tapes during a frantic rebuilding process.  To make matters worse, the director wasn’t sure when he deleted it.  This meant that we would need to take a valued engineer off his tasks and have him rifle-through backup tapes, restoring several tapes to recovery servers in an attempt to find message that might NOT have been captured in a backup job.  Long story short, I lost control.  I didn’t become violent.  I just started sobbing.  You can imagine the impact this had on my fellow parishioners as they were also looking at lamps and shoes and nick-nacks.  I can remember saying to Rob, “this guy wants us to derail this recovery effort because he can’t figure out how to NOT destructively delete a single email message?!  There are 5,000 people missing!  He wants us to recover an email message?!  He should be focused on recovering PEOPLE!  Has he gone mad?!” So there I was, on a perfectly sunny Sunday morning, in a church parking lot, at a fund raising flea market, sobbing and yelling into my cell phone.  Not pretty.  I took some mandatory PTO later that day.

My point in documenting any of this is partly cathartic, partly to share my experiences, but mostly to provide those who were similarly affected a voice — a brother in arms — a common link.  I have spoken to dozens of people about 9/11 over the past 13 years.  Each one has a story that starts with “I remember exactly where I was…”  It’s our brain’s natural way of cataloging information — spacial pinning.  It’s the brain’s way of helping us recall an event with as many details as possible.

Unlike those inside the towers, my view was from outside the towers.  I had an awesome and dreadful view of the towers from the window of my apartment on 42nd Street.  My apartment was on the 50th floor of a new multi-tower apartment complex on the West Side Highway.  My view — through massive plate windows that opened — was of downtown.  I had a direct — line of site — view of the the two towers.  They filled most of the Window.  I saw the second plane hit.  I saw the towers fall.  First one, then some time later, the second.  The feeling of helplessness is still devastating.  The feeling that something needed to be done, but couldn’t be done, is something that still causes me to stare into the air and ponder.

For the Fire Fighters and Rescue personnel and those who ran into the towers in an attempt to simply DO SOMETHING, my heart is yours.

I made some very good friends during those weeks.  We are all damaged by the experience.  Even if we had learned anything good during the process, we cannot take any joy in the learning because it was among so much BAD.  We have not given ourselves permission, even now, to enjoy the lessons we learned.  We, instead, hold them in a vault in our souls — a place we reserve for the memories of those who gave their lives that day.

Every year, around August 15th, I begin to relive the experience.  For the weeks leading up to September 11th, I notice the number eleven everywhere.  It seems that every time I look at the clock, it’s eleven minutes past the hour.  And on the eve of September 11th, I wonder what I should do to commemorate the day.  Just like my fateful day in the church parking lot, I feel lost, alone, frustrated, but mostly just sad.  Just very, very sad.

So on this day, please take a moment to think about those who lost lives, those who lost others, and those who worked tirelessly to support the companies who were affected by the events of September 11th, 2001.

God bless you.