Having spent a significant time in SSIS already and even had thanked a few people, including Andy, a couple years ago when I was learning how to load data warehouses, I was really looking forward to this session.

Agenda:

• Frameworks
• Using Metadata
• Centralized Logging
• Monitoring & Analytics
• Logging in SQL Server “Denali”
• Configurations
• Expressions
• Deployment
• Data Flow Internals
• Measuring Performance
• Data Flow Best Practices
• Scripting
• Design Patterns

First was the de facto disclaimer – This is a level 300 session and it’s difficult to stay at 300 through everything – Configurations, deployment, expressions, etc. They’re covering many areas some of which we might be more expert at than others, but they were looking at being comprehensive.

Andy took the stage first and presented his work on SSIS Frameworks that he’s been honing and presenting for awhile.  It’s some great stuff and if you’ve never looked at it, do yourself a favor and go there now, I’ll wait.  While he was presenting he said the best line, which I ended up tweeting – “You can’t avoid SSIS Pain, only pick when you feel it.”  Matt then talked about some of the upcoming features in Denali and plugged his session later in the week where he’ll be making some “announcements”.

Next up, Tim took the stage and talked about the configurations, expression and deployment at which point Matt chimed in with some of the new deployment options in Denali and since we were running early, he went on to talk about Data Flow Internals before breaking for lunch.  After lunch Matt finished discussing the date flow task and then Tim did scripting.

The day ended with Matt presenting Design Patterns and listed out design patterns he “could” talk about:

• Using the DB Engine
• Lookups
• Parallel Processing
• Avoiding Transactions
• Surrogate Key Generation
• Partitioning
• CDC
• Null Value Substitution
• Balanced Data Distribution
• Using Multiple Transforms
• Late Arriving Facts
• Work Pile Pattern – Scale Out
• Slowly Changing Dimensions
• Change Detection
• XML Patterns

Course we only had time to look at 5 of them (great presentations leave you wanting more, right?).

Overall all three presenters were excellent – they were both knoweldgable passionate about their material, the slide decks were very professional, demo’s were also well prepared and they handled questions while staying on schedule very well.  They were even monitoring twitter to get questions during the session as well as honoring requests to just keep going and not worry about the time.  As with any session this long they were right about there being some things that I already knew and some things I didn’t.  However even the things I knew, it was good to have a refresher on the why and even get the re-assurance that I do something right once in awhile.

Now onto some stuff I learned:

• Got pretty deep into how the Data Flow worked, talking about the Buffers, Execution Trees and Backpressure.
• Discussed when to use synchronous, asynchronous and blocking transformations
• Discussed SSIS strategy for performance – Operations Volume Application Location (OVAL)
• All LOB columns are paged to disk, no exceptions, but you can change the location
• Currency/money data types are more efficient than doubles
• Don’t use the SQL Destination (I’ve never used this but had seen the 30% faster claims)
• There is an actual good use case to use the SCD Wizard.
• Cascading lookups
• How to design for scale out scenarios – this is really cool and can even utilize Service Broker, I’m really pumped to try this one out.
• When not to use SSIS to move data
• Perf Counter – Buffers Swap to Disk should always be 0, otherwise you’re using too much (or don’t have enough) memory
• Why SQL 2005 expressions run time values trump design time and why SQL 2008 is the reverse
• Don’t use SSIS Transactions (i.e., transaction property within the packages)

Thanks to Andy, Tim and Matt for a great session.  They obviously spent a lot of time preparying, in fact they were even tweeting Sunday about practicing.  Andy later said they were making tweaks during their practice session and even decided to drop talking about source control in favor of additional Design Patterns.  Very good content, thanks to the three presenters!

Finished the day at a networking event at Lowell’s.  Looking forward to  Tuesday’s “Advanced T-SQL for SQL Server 2008 and Denali” by Itzik Ben-Gan (blog)

• Intro
• First Problem

Now that I have the StartLSN and EndLSN I was ready to go.  Hit my breakpoint and the variables were set appropriately.  Woo-hoo!

Then…wait a minute…what?

vSQLQuery.Value =
    "SELECT " +
    "    c.__$operation " +
    "    , c.Column1 " +
    "    , c.Column2 " +
    "    , ColumnN " +
    "FROM " +
    "    cdc.fn_cdc_get_net_changes_<instance_name>( " + sStartLSN + ",  " + sEndLSN + ", 'all') c";

That statement wasn’t changing the value of vSQLQuery.  huh?

Ok, I don’t by any stretch of the imagination claim to be an expert at C# but that looked pretty straight forward to me, however it wasn’t changing the value.

So now I start looking in earnest, google, bing, Stackoverflow, Serverfault, all to no avail.  This just wasn’t making sense.  I was about to call over an expert C# person (Magenic is full of them ) when I remembered that my previous problem was not a problem with the code but a problem with the configuration of the task.  So I start looking into the configuration of the script task and the variables that I was using and that’s where I saw it. 

vSQLValue was set to evaluate as an expression.  I had done this so I could copy and paste the SQL query I had developed in SSMS as opposed to making the query 1 line and pasting it into the value of the variable (the variable box in SSIS can’t interpret multi-line values, it only pastes the first line).

Turned off the fact that it was an expression and since it had already been evaluated it set the variable correctly (score!) and now I could manipulate the vSQLQuery.Value via the script task.

Now as to why I couldn’t modify a variable that is set to evaluate as an expression, I don’t know I’m sure there’s some sort of order of operations or locking catch here, but I still haven’t been able to find anything official about this.  Best I saw was this which would indicate to me that it would work.

Onto the next problem.

The first problem I noticed was that the SQL query that was running still had nulls in on the StartLSN and EndLSN

so I was querying:

cdc.fn_cdc_get_net_changes_<instance name> (null, null, ‘all’)

Which as I said last post, returns an error. So throw a break point into the Script task and figure out why it’s not setting properly.  This took me awhile, here’s the code:

var vStartLSN = Dts.Variables["User::sStartLSN"];
var vEndLSN = Dts.Variables["User::sEndLSN"];
var vSQLQuery = Dts.Variables["User::sSQLQuery"];

string sStartLSN;
string sEndLSN;

Dts.VariableDispenser.LockForRead("User::sStartLSN");
Dts.VariableDispenser.LockForRead("User::sEndLSN");
Dts.VariableDispenser.LockForWrite("User::sSQLQuery");

sStartLSN = vStartLSN.Value.ToString();
sEndLSN = vEndLSN.Value.ToString();

vSQLQuery.Value =
    "SELECT " +
    "    c.__$operation " +
    "    , c.Column1 " +
    "    , c.Column2 " +
    "    , ColumnN " +
    "FROM " +
    "    cdc.fn_cdc_get_net_changes_<instance_name>( " + sStartLSN + ",  " + sEndLSN + ", 'all') c";
Dts.Variables.Unlock();

So hitting my first breakpoint, I first realized that the StartLSN variable was receiving a null.  That wasn’t supposed to happen, so I looked at the SQL code I used to set the variables in the SQL Task:

SELECT sys.fn_cdc_get_min_lsn(<instance_name>) as @StartLSN

A-ha, that’s it. The function returns a record set and I had the task to just get a return value, not a record set.

So changed to:

SELECT @StartLSN = sys.fn_cdc_get_min_lsn(<instance_name>);

SELECT @StartLSN as StartLSN

That got me past the first problem, now onto the next problem.

You can see Change Data Capture since Last Request Package Sample or Change Data Capture for Specified Interval Package Sample for the official MS examples which is the basis of what I was trying to accomplish, although in the specified interval sample they cheated by using a database snapshot to get the starting LSN and then just ran continuously from there keeping the LSNs in variables, they didn’t address the concept of true interval run that would be come to a stopping point and then start again after an interval passes.

So to give a little context to this post you need to understand some basics of SSIS and CDC.  CDC utilizes Log Sequence Numbers (LSNs) to know the start and stop point of the changes you are trying to identify.  CDC is table based and logs those changes to an “instance” of the table that you specify, which is basically just a copy of the table DDL with a few added columns, the new table exists within the cdc schema and the table name is what you specify via the “instance name” parameter. You can track changes on the entire table or just specific columns, and there are functions to get all the changes or just the net changes.

Within CDC you have some very helpful system functions to assist with figuring out what LSNs you should be selecting, including:

• sys.fn_cdc_get_max_lsn
  sys.fn_cdc_get_min_lsn
  sys.fn_cdc_map_time_to_lsn
  sys.fn_cdc_map_lsn_to_time

The other thing to remember is that LSNs are binary.  In SSIS there is no variable type binary.  The first odd thing I noticed (and this should have been a clue to the difficulties that lie ahead) was that the sys.fn_map_time_to_lsn has a strange parameter called the ‘relational operator’ and I kid you not these are the potential values:

• largest less than
  largest less than or equal
  smallest greater than
  smallest greater than or equal

Yes the actual text strings are what is needed to be entered there.  The idea being that LSNs aren’t directly time related to so you need to specify the relationship in relation to the datetime your passing. 

The next  thing I noticed that is strange was that sys.fn_cdc_get_min_lsn must be instance specific, while sys.fn_cdc_get_max_lsn is the entire server.  So hopefully this will be a little more straight forward in the next version. <TangentRant>although that’s what I thought about Service Broker which I love, but sadly still isn’t that straight forward, however that’s for another post</TangentRant> 

So to cover all bases this was my process:

1.) The parent package would get the LSN boundaries based upon the last successful run of the package (StartLSN) and the current run (EndLSN), passing those variables to the children packages.

2.) The child package which is instance specific, would then verify that that StartLSN was not less than the minimum LSN for that instance, if it was, than use the MinLSN instead. 

3.) Utilize the TVF to get the net changes cdc.fn_cdc_get_net_changes_<capture_instance> (to get all changes you would use cdc.fn_cdc_get_all_changes_<capture_instance>)

A couple other issues here when using with SSIS here:

1.) It’s a TVF so if you pass bad parameters as start and end LSNs, you get an error in addition to the column header information:

“An insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_net_changes_ … .”

2.) It’s a TVF so you can’t use it as an OLEDB data source, instead you have to use an ssis variable as the data source, which mean the query needs to be set with the proper query with null values for the Start and End LSNs (remember even though it brings back an error it will bring back column information).  This also means the script task is needed to change the alter the query string variable to substitute the proper start and end LSNs at runtime.

So now that we have the basis of what I was doing, now onto the debugging…

• Top 10 SQL Server Integration Services Best Practices
• SCD (Slowly Changing Dimension) optimization in SSIS
• SSIS – Decrease your fact table loading time up to 40%
• SSIS: Custom Logging Using Event Handlers
• SSIS Design Pattern – ETL Instrumentation