Being that I have been involved in a project since the first of the year that pretty much drained me week-to-week, I sadly wasn’t able to keep up with my blog.  However now that I’ve moved onto to a new project I plan on getting back to posting blogs again. 

I continue to be very grateful to the SQL community for its hard work and support (I loved 24 Hours of PASS and the impromptu associated ustream – 24 hours of LaRock) and hopefully I can give back to the community a portion of what you all have given me.

Chris Shaw posted a second SQL Quiz where he asks: "What are the largest challenges that you have faced in your career and how did you overcome those?”

#1

When I was working at the University I was the primary DBA on the system for the student and Employee ID card.  This card was swiped for access to dorms, parking, buildings etc as well as it was hooked up to a student accounts for cafeteria, laundry, etc and the numbers were used for websites access.  There was a server at each of the three main campus throughout the state and the data was replicated via transactional replication.  Being it was a university it was a Database Application that had to have all the business logic in the database as we had to open it up to any platform or data access technology that the individual campuses, departments wanted to use.  However if the server was down or unavailable for any reason there were certain applications that didn’t cache data like the cafeteria services.  Our service window which allowed downtime was Sunday mornings from 2 am to 4 am.  This was run on Windows NT 4.0/SQL 7.0.  We eventually migrating to new machines running Windows 2003/SQL 2000 and then upgraded to SQL 2005.  Shortly after the upgrade to 2005 we went live with a new version which included utilization of service broker and clr and encryption.  So in a little over a year we did a platform migration a platform upgrade and an application upgrade without missing any of target dates or times or affecting out users.  Our core technical team was 4 people that was managing the entire process including communication to clients.  Documenting and scripting each step along with running multiple tests and involving our clients in the tests as well to set everyone’s expectations made the whole thing a success.

#2

I don’t think I’ll be shocking anyone by saying that in the consulting world sometimes you go onto projects in areas or technologies that you aren’t necessarily an expert in and therefore are sometimes learning on the job.  For example I knew DTS fairly well, however when I went on my first gig that was teaching and doing SSIS I wasn’t extremely knowledgeable about it.  However I crammed and learned about SSIS and the client was extremely happy with what I brought to the table.  So sometimes when I’m billed out as an expert, knowing how to get up to speed and being able to is just as important a skill to have.  And twitter has made that just more easier and fun!

Anyway that’s the two that really come to mind.

Tagging Dan English

Evidently while I was on vacation I missed Chris Shaw ask for us to share our mistakes, however now that I see it, thought I’d jump in.

My first full time job I was employed as a System Administrator.  I soon realized the purpose of this generic term was because I was the only technical guy on site supporting a company’s sales office of around 20 users and as a result I had to maintain NT 4, SQL 6.5, Exchange 4.0, Win ‘95, Office ‘97, Goldmine, Veritas Backup, etc. This office also housed the primary dns information and our website and exchange.  The primary office which was around 60 people at the time was in another country with a time difference of 6 hours.

Anyway I had weekly calls with the senior sys admin who was telecommuting from his house (1 timezone away) and one of the items that had been on the to do list for awhile was upgrade the Exchange server from 4.0 to 5.5.  Not liking things on my to do list for very long I kept asking about it and was told it was no big deal and when we get some time we will do it.  Well being about a month out of college and wanting to make a good impression I realized I had some time one late afternoon and figured as it was no big deal, I’ll just pop in the CD and go for it.  I mean it’s an upgrade right, not like a new install where I’ll have to configure a bunch of stuff.

Well suffice to say the upgrade actually crashed partway through and NT 4.0’s Dr. Watson faithfully kicked off.  Not really understanding anything about anything I rebooted and tried it again at which point I received the same result.  So I remembered being told we has backup tapes and I should be able to restore if there’s a problem, so I popped in the backup tape and starting restoring.  Once the restore was done I tried to access my email and still no luck.  Time to call someone, but as it was no big deal I wasn’t even worried at this time.

Well calling the senior sys admin they gave me a little better perspective of how big the deal was.  And they were even gracious enough to inform me that I really shouldn’t have done the upgrade by myself and that the backups I had just restored weren’t of the Exchange server, they were of the file server so not only did was not have Email working for the entire company, we had also just lost the sale’s work for the entire day. 

It was at this point I immediately starting doing multiple things at once.  I started sweating.  I started feeling sick to my stomach.  I started worrying about my job, and subsequently my bills (I had just gotten my first mortgage earlier that month).  I started trying to figure out what I could do to solve the problem.

The next 26 hours went by like a blur, but I can say that I ended up doing those four things a lot.  I got on the phone with MS Support (paid-for incident).  Finally we got Exchange Server running after using a low-level utility which investigated every record in the Exchange data store and if that record wasn’t perfect it was deleted.  We went from about 3GB data store to about 650 MB data store.  I quickly wrote up a post-mortem (although I didn’t know it was called a post-mortem at the time) and sent it out.  I actually shared an office with the COO and co-owner of the company at the time due to space restrictions and I remember hitting the send button on my email and booking it out of the room on an ”errand”.  However after a discussion of the post-mortem with management they ended up using this as a learning opportunity for me and I ended up continuing to work there for another 2 years or so.

While I have made mistakes since then, none have every been to this level and therefore I usually keep a nice cool head on my shoulders in the face of problems.