Wait, let’s back up:

Sweat mars your brow as you start to feel panic set in.  You reach for your mountain dew can and quickly put it down so no one will see your hand shaking (plus don’t want to spill a good dew).

Still need to back up further:

You were minding your own business watching a great SQLPASS webcast and thinking of how you were going to implement those cool things you learned.  At least, that was until you got THE PHONECALL.  Your boss calls and says, “Hey there’s this project going on that is going to need some database help, I’ve told them you’re just the person to do it.  So I’ve scheduled you in a conference call tomorrow morning first thing.  Sounded pretty straight forward so should be no problem.” You then start downloading your brain to notepad and internet links so you don’t forget the SQLPASS presentation.

You come in the next day, get your morning Mountain Dew (hey it’s my story and I’m not much of a coffee drinker) and jump on the conference call.    That is when you hear this:

“Glad you could join us; your boss said you were the best, so we’re expecting great things from you.  What we need is a database to store the Excel documents we’ve been working on for the past year.  We’ve just been emailing them around, so we’ll need to import the data and be able to create reports.  I already spec’d it out in Access and don’t think it should take more than a week, but with you on here, it should take less time!  This is good because we’re already committed to deadlines in about a month. I will email you the Excel documents, once you look and understand them we will talk later this afternoon to answer any questions you have. K, thanks, bye”

You start seeing visions of SSIS, SSRS and SQL go through your head and you’re wrapping your brain around everything that needs to be done, when you get the email: “Here’s a couple of the first type of spreadsheets, there are actually 3 different types of spreadsheets but it’s all related.  Should be no problem for a wiz like you, huh? When you’re ready for the rest, let us know and we can create a CD of them – think there’s about 300 or so.  I forgot to mention on the call that while we have a deadline in a month, we promised to start getting reports out next week, so I’m really looking forward to seeing the reports this week.”

Sweat mars your brow as you start to feel panic set in.  You reach for your mountain dew can and quickly put it down so no one will see your hand shaking (plus don’t want to spill a good dew).   Then a thought creeps into your head, you initially reject it, and look around to make sure no one heard it, but as you think about it, the thought starts making sense and you start to smile.  You now have the way to meet or possibly exceed the expectations and you can look like the hero.  So you say to yourself, “Don’t worry, I’ll just import manually, create the code as I go to be quick, then I can go back and clean it up later”

———————

The above scenario is not that uncommon in our field.  We are brought in as the experts and expected to produce things quickly.  The business group does not know and possibly does not even care about everything we do. But, really, it is not their job, if they did, you would not have one.  Things like code quality, code coverage, and security are your job, not theirs.

However, I am still seeing too many people try to be the hero by working around the clock or reducing code quality to meet the requirements as soon as possible.  Let me refute some more common reasons:

I will have plenty of time…

The idea that you will all of a sudden have this huge amount of free time to do performance improvements, proper error handling, better naming conventions, etc. is a myth.  More likely, you will be pulled onto another project and then someone will knock on your door in a year or two asking you to explain this really bad code they found.  If on the other hand you are experiencing large blocks of free time, please send me a note as I might have some work for you.  Companies pay you to work, which is the opposite of having free time.

I’ll just do it this once…

If you put that kind of code out there, you will probably end up doing a lot of manual process/manipulation.  This means when it is time to actually go-live, you scramble to get it all working, again reducing quality.  I have often seen “While it’s in development I don’t have to worry about getting it scheduled.  Oh that process didn’t work; I’ll just manually fix it.”  If you have to fix it manually in development, you will have to fix it manually in prod.

They just don’t understand

Our job should not just be seen as a cost center it should be seen as an asset.  That does not mean we just produce great code/functionality, it means we help the business achieve the business goals.  In other words, we work “with” not “for” the business. 

Back in the Exchange 5.5 days, I did some work as a systems administrator.  We discovered that some people were using mailboxes that had up to 500 MB in them, while the average was around 100; I researched best practices and did all this analysis.  I finally brought the suggestion to my boss, “We should limit everyone’s mailbox to XX size that way we don’t have space issues” His response “This is obviously the way our people work and if it is, why we shouldn’t just invest in more space so they can continue to work that way?” 

I had never considered the business, only the technology.  When you just have technology on the brain and not the business or the people you come up with solutions for the technology. “My system would be perfect except for all those users”

Take the time to explain what you need to accomplish so you can create what they want you to accomplish.  They want it sooner rather than later? Then reflect some of those decisions back to them, even empowering them to make some.  “Yes I can get that really cool function out there this week, but everyone in the world will have access to it,  With a couple more days I can also have it secure.  Which would you like?”  Of course, this means that you have to think more strategically about the project, which can be difficult at times.  We all know how easy (and fun) it is to just jump in there and start producing code”.  Transparency is the key.

But I’m always having to change things

Realize that some refactoring will always be necessary. The days of going away for 6 months to create something and come back to do your big “ta-da” are long gone.  People want things faster with more agility so things can change as we go along.    This means more pressure to produce faster, which means refactoring and adding additional functionality as you go.  However ever line of code you write should be quality.  I don’t know how many times I have seen Proof-of-Concepts (POCs) turn into the actual production system.  Just because it is a POC from a business perspective does not mean that it should be a POC from a code quality perspective.  I actually saw someone code up a textbook T-SQL injection (put the stored procedure parameter straight in the where clause without any validation), with the intention of fixing it later.  If it is wrong to do later, it is wrong to do now.

In the development world, there are many tools for code coverage, unit testing, refactoring, etc.  While there are some tools out there like VSDBPro unit testing, VSDBPro and Red Gate refactoring, etc., the tools in the database world are not as mature.  There is no IDE that forces white space or automatically documents for you.  We do not have tools where we can just do a few clicks and have end-to-end tests.

However, that is no excuse to be a gunslinger and just shoot from the hip.  We are professionals and should be creating professional code that is readable, documented and have a standard of quality.

If you are struggling with that, adopt a Test Driven Development (TDD) philosophy or something else that will force you to think about or do things better. 

Or maybe it is taking you 5 minutes to figure out what is going on in a stored procedure you just wrote a couple days ago.  If so, let’s be honest you failed in either code readability and/or documentation. No excuses, there are even good free tools out there. Whitespaceisreallyagoodthing.

Code Quality, Code Readability and Code Documentation is Important

Now that I’m typing and you’re reading quietly, on to the reintroduction: here I am, over here to, also mentioned here and here.

As to why no activity lately, to be honest I have had a lot to say and I have a whole list of ideas for blog posts however as soon as I start to write one, my perfectionism comes into play and I never have time to create what I think would be a perfect post, so I don’t.  I look up at what I don’t know and decide I don’t know enough, instead of looking down or around at what I do know to talk about.  This is the beginning of the effort to change and just put some thoughts out there (for better of worse . 

For those keeping score at home I’m still at Magenic and while a couple good friends have left, we’re hiring.  In fact in the Chicago office we have had 3 people start in the last 2 weeks (including here)! So if you are looking, hit me up and we’ll see what can be done.

Other than that, I’m still doing all things SQL Server from SSIS to Service Broker to CDC to Replication to CLR to encryption to SSRS to data modeling and of course VSDBPro.  Basically the entire stack and oh by the way, still loving it!

For review this post had 3 purposes:

1. Begin Momentum
2. Remind everyone I’m still around
3. Fulfill my promise to Ted

Mission Accomplished!

• Intro
• First Problem

Now that I have the StartLSN and EndLSN I was ready to go.  Hit my breakpoint and the variables were set appropriately.  Woo-hoo!

Then…wait a minute…what?

vSQLQuery.Value =
    "SELECT " +
    "    c.__$operation " +
    "    , c.Column1 " +
    "    , c.Column2 " +
    "    , ColumnN " +
    "FROM " +
    "    cdc.fn_cdc_get_net_changes_<instance_name>( " + sStartLSN + ",  " + sEndLSN + ", 'all') c";

That statement wasn’t changing the value of vSQLQuery.  huh?

Ok, I don’t by any stretch of the imagination claim to be an expert at C# but that looked pretty straight forward to me, however it wasn’t changing the value.

So now I start looking in earnest, google, bing, Stackoverflow, Serverfault, all to no avail.  This just wasn’t making sense.  I was about to call over an expert C# person (Magenic is full of them ) when I remembered that my previous problem was not a problem with the code but a problem with the configuration of the task.  So I start looking into the configuration of the script task and the variables that I was using and that’s where I saw it. 

vSQLValue was set to evaluate as an expression.  I had done this so I could copy and paste the SQL query I had developed in SSMS as opposed to making the query 1 line and pasting it into the value of the variable (the variable box in SSIS can’t interpret multi-line values, it only pastes the first line).

Turned off the fact that it was an expression and since it had already been evaluated it set the variable correctly (score!) and now I could manipulate the vSQLQuery.Value via the script task.

Now as to why I couldn’t modify a variable that is set to evaluate as an expression, I don’t know I’m sure there’s some sort of order of operations or locking catch here, but I still haven’t been able to find anything official about this.  Best I saw was this which would indicate to me that it would work.

Onto the next problem.

The first problem I noticed was that the SQL query that was running still had nulls in on the StartLSN and EndLSN

so I was querying:

cdc.fn_cdc_get_net_changes_<instance name> (null, null, ‘all’)

Which as I said last post, returns an error. So throw a break point into the Script task and figure out why it’s not setting properly.  This took me awhile, here’s the code:

var vStartLSN = Dts.Variables["User::sStartLSN"];
var vEndLSN = Dts.Variables["User::sEndLSN"];
var vSQLQuery = Dts.Variables["User::sSQLQuery"];

string sStartLSN;
string sEndLSN;

Dts.VariableDispenser.LockForRead("User::sStartLSN");
Dts.VariableDispenser.LockForRead("User::sEndLSN");
Dts.VariableDispenser.LockForWrite("User::sSQLQuery");

sStartLSN = vStartLSN.Value.ToString();
sEndLSN = vEndLSN.Value.ToString();

vSQLQuery.Value =
    "SELECT " +
    "    c.__$operation " +
    "    , c.Column1 " +
    "    , c.Column2 " +
    "    , ColumnN " +
    "FROM " +
    "    cdc.fn_cdc_get_net_changes_<instance_name>( " + sStartLSN + ",  " + sEndLSN + ", 'all') c";
Dts.Variables.Unlock();

So hitting my first breakpoint, I first realized that the StartLSN variable was receiving a null.  That wasn’t supposed to happen, so I looked at the SQL code I used to set the variables in the SQL Task:

SELECT sys.fn_cdc_get_min_lsn(<instance_name>) as @StartLSN

A-ha, that’s it. The function returns a record set and I had the task to just get a return value, not a record set.

So changed to:

SELECT @StartLSN = sys.fn_cdc_get_min_lsn(<instance_name>);

SELECT @StartLSN as StartLSN

That got me past the first problem, now onto the next problem.

You can see Change Data Capture since Last Request Package Sample or Change Data Capture for Specified Interval Package Sample for the official MS examples which is the basis of what I was trying to accomplish, although in the specified interval sample they cheated by using a database snapshot to get the starting LSN and then just ran continuously from there keeping the LSNs in variables, they didn’t address the concept of true interval run that would be come to a stopping point and then start again after an interval passes.

So to give a little context to this post you need to understand some basics of SSIS and CDC.  CDC utilizes Log Sequence Numbers (LSNs) to know the start and stop point of the changes you are trying to identify.  CDC is table based and logs those changes to an “instance” of the table that you specify, which is basically just a copy of the table DDL with a few added columns, the new table exists within the cdc schema and the table name is what you specify via the “instance name” parameter. You can track changes on the entire table or just specific columns, and there are functions to get all the changes or just the net changes.

Within CDC you have some very helpful system functions to assist with figuring out what LSNs you should be selecting, including:

• sys.fn_cdc_get_max_lsn
  sys.fn_cdc_get_min_lsn
  sys.fn_cdc_map_time_to_lsn
  sys.fn_cdc_map_lsn_to_time

The other thing to remember is that LSNs are binary.  In SSIS there is no variable type binary.  The first odd thing I noticed (and this should have been a clue to the difficulties that lie ahead) was that the sys.fn_map_time_to_lsn has a strange parameter called the ‘relational operator’ and I kid you not these are the potential values:

• largest less than
  largest less than or equal
  smallest greater than
  smallest greater than or equal

Yes the actual text strings are what is needed to be entered there.  The idea being that LSNs aren’t directly time related to so you need to specify the relationship in relation to the datetime your passing. 

The next  thing I noticed that is strange was that sys.fn_cdc_get_min_lsn must be instance specific, while sys.fn_cdc_get_max_lsn is the entire server.  So hopefully this will be a little more straight forward in the next version. <TangentRant>although that’s what I thought about Service Broker which I love, but sadly still isn’t that straight forward, however that’s for another post</TangentRant> 

So to cover all bases this was my process:

1.) The parent package would get the LSN boundaries based upon the last successful run of the package (StartLSN) and the current run (EndLSN), passing those variables to the children packages.

2.) The child package which is instance specific, would then verify that that StartLSN was not less than the minimum LSN for that instance, if it was, than use the MinLSN instead. 

3.) Utilize the TVF to get the net changes cdc.fn_cdc_get_net_changes_<capture_instance> (to get all changes you would use cdc.fn_cdc_get_all_changes_<capture_instance>)

A couple other issues here when using with SSIS here:

1.) It’s a TVF so if you pass bad parameters as start and end LSNs, you get an error in addition to the column header information:

“An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ … .”

2.) It’s a TVF so you can’t use it as an OLEDB data source, instead you have to use an ssis variable as the data source, which mean the query needs to be set with the proper query with null values for the Start and End LSNs (remember even though it brings back an error it will bring back column information).  This also means the script task is needed to change the alter the query string variable to substitute the proper start and end LSNs at runtime.

So now that we have the basis of what I was doing, now onto the debugging…