SQL Quiz
Evidently while I was on vacation I missed Chris Shaw ask for us to share our mistakes, however now that I see it, thought I’d jump in.
My first full time job I was employed as a System Administrator. I soon realized the purpose of this generic term was because I was the only technical guy on site supporting a company’s sales office of around 20 users and as a result I had to maintain NT 4, SQL 6.5, Exchange 4.0, Win ‘95, Office ‘97, Goldmine, Veritas Backup, etc. This office also housed the primary dns information and our website and exchange. The primary office which was around 60 people at the time was in another country with a time difference of 6 hours.
Anyway I had weekly calls with the senior sys admin who was telecommuting from his house (1 timezone away) and one of the items that had been on the to do list for awhile was upgrade the Exchange server from 4.0 to 5.5. Not liking things on my to do list for very long I kept asking about it and was told it was no big deal and when we get some time we will do it. Well being about a month out of college and wanting to make a good impression I realized I had some time one late afternoon and figured as it was no big deal, I’ll just pop in the CD and go for it. I mean it’s an upgrade right, not like a new install where I’ll have to configure a bunch of stuff.
Well suffice to say the upgrade actually crashed partway through and NT 4.0’s Dr. Watson faithfully kicked off. Not really understanding anything about anything I rebooted and tried it again at which point I received the same result. So I remembered being told we has backup tapes and I should be able to restore if there’s a problem, so I popped in the backup tape and starting restoring. Once the restore was done I tried to access my email and still no luck. Time to call someone, but as it was no big deal I wasn’t even worried at this time.
Well calling the senior sys admin they gave me a little better perspective of how big the deal was. And they were even gracious enough to inform me that I really shouldn’t have done the upgrade by myself and that the backups I had just restored weren’t of the Exchange server, they were of the file server so not only did was not have Email working for the entire company, we had also just lost the sale’s work for the entire day.
It was at this point I immediately starting doing multiple things at once. I started sweating. I started feeling sick to my stomach. I started worrying about my job, and subsequently my bills (I had just gotten my first mortgage earlier that month). I started trying to figure out what I could do to solve the problem.
The next 26 hours went by like a blur, but I can say that I ended up doing those four things a lot. I got on the phone with MS Support (paid-for incident). Finally we got Exchange Server running after using a low-level utility which investigated every record in the Exchange data store and if that record wasn’t perfect it was deleted. We went from about 3GB data store to about 650 MB data store. I quickly wrote up a post-mortem (although I didn’t know it was called a post-mortem at the time) and sent it out. I actually shared an office with the COO and co-owner of the company at the time due to space restrictions and I remember hitting the send button on my email and booking it out of the room on an ”errand”. However after a discussion of the post-mortem with management they ended up using this as a learning opportunity for me and I ended up continuing to work there for another 2 years or so.
While I have made mistakes since then, none have every been to this level and therefore I usually keep a nice cool head on my shoulders in the face of problems. 
Wow, nice story. Makes you wonder what doctors talk about when they have this same conversation.