1. Required 9′s & Protection – Integration Services is now Integration Server; HA for StreamInsight; AlwaysOn
2. Blazing-Fast Performance – Column Store Indexes; improvements across the board
3. Rapid Data Exploration – Power View + PowerPivot; Admin from SharePoint; Reporting Alerts
4. Managed Self-Service BI – same as #3
5. Credible Consistent Data – BI Semantic Model (BISM); Data Quality Services (DQS);Master Data Services (MDS)
6. Organize Compliance – Expanded Audit (user defined and filtering); User Defined Server Roles
7. Peace of Mind – Production-simulated distributed testing; System Center Advisor & Management Packs; Expanded Support – Premier Mission Critical
8. Scalable Data Warehouse – SQL Server Appliances; HW + SW + Support – Just add Power; choice of hardware
9. Fast Time to Solution – Same as #8
10. Extend Any Data, Anywhere – Greater Interoperability, New Drivers for PHP, Java and Hadoop; ODBC Drivers for Linux & CDC for SSIS and Oracle; Beyond Relational: FileTable, 2D Spatial, Semantic Search
11. Optimized Productivity – SQL Server Data Tools (SSDT); Unified Across Database & BI; Database & Targeting Freedom
12. Scale on Demand – AlwaysOn; Deployment across Public & Private; Elastic Scale

We saw demos of AlwaysOn, Column Store Index, Data Quality, and Semantic Search, plus they showed off the SQL Server Appliances.  The thing that really stood out to me in the keynote was the Semantic Search Demo.  The presenter took two pdf files and utilizing TSQL was able to relate them conceptually.  Looked very cool, you can go here for more info.

After that I attended Matt Masson’s (twitter | blog) session on Top 10 Things You Didn’t Know About SSIS in SQL Server Code Name “Denali”.  These were items that are post CTP3.  To kill some time while waiting for the session to start Matt went through a mock scenario of utilizing SSIS to create virtual friends to party with him and his cat (Mr. Wiggles) on Facebook using Facebook Services and StreamInsight Transform – it was absolutely hilarious!

1. Change Data Capture – as someone who has done CDC in SSIS this was very cool
2. ODBC Support
3. Connections Managers – Shared In-Memory Cache Across Packages using the shared project level connection manager
4. Flat File Source Improvements – supports varying number of columns and can parse embedded qualifiers
5. Package Format Changes – doesn’t persist default values; doesn’t use lineage id’s, instead uses ref id’s; sorted by name
6. Visual Studio Configurations
7. Scripting Improvements – can now set a break point in a script task/component and see data
8. Troubleshooting & Logging – Built-in reporting; Component Timing & Row Counts; Server level logging
9. Data Taps – can write out data to a csv location without code change (doesn’t log binary)
10. Server Management using PowerShell
11. Honorable Mention – Project Barcelona Metadata Discovery – crawler that crawls entire MS stack to determine Impact Analysis and Data Lineage

These were on top of all the other things we new about for SSIS in SQL 2012, and I have to say that I’m really looking forward to it!

Next I went to the Lighting Talks which is a cool idea.  The premise is to have a number of speakers, this one has seven.  Each speaker however is limited to 5 minutes and there is someone utilizing a timer and the crowd is encouraged to clap people off stage that go to long.  It was a lot of fun.  The two that really stood above the rest (in my opinion anyway) were Grant Fritchey’s (twitter | blog) “Backup Testing, The Rant ” and Niko Neugebauer’s (twitter | blog) “Build Up”.

Last was Database Development with SQL Server Data Tools Code-Named “Project Juneau” by Gert Drapers (blog).  As someone who has used Database Professional extensively, I was pretty skeptical going in since in that Unit Tests aren’t in SSDT.  As you would expect one of the big features is that it has support for Azure.  However some of the stuff that I did see was really cool:

• A True Debugger with break points and a Call Stack, with the ability to start in the .NET app
• Refactor can now reach into the application
• True Language Services – can right click to go to reference, etc.
• Drift Detection – if working in a connected stated it will poll database to see if any changes happened behind you and you can just drag and drop changes into project
• Easier way to save specific deployed versions
• Table designer where you see both code and the designer window
• SQL CLR is within the same project, no longer have to create a separate CLR project

Some things I still have to think if I like or not:

• Renamed Deploy to publish so that Projects are now Published
• Multiple objects are now in a single file, for example the indexes, keys are now defined in the create table script file

And the things I don’t like, until vNext Visual Studio + 3 months we’re missing:

• Data comparison
• Data Generation
• Unit Testing

Now for the great news.  It’s completely FREE!!  It will be delivered as part of the SQL Server install as well as part of the Visual Studio install or you can get it through  the Web Platform Installer to get it!  We’ve come a long way from requiring TFS 2005 Team System.  Looks like we are getting closer to a standard SQL Server development environment.  That was it for day 2, next up is the 3rd and final day.

I have been using Visual Studio Database Projects since DataDude beta’s came out in 2005.  A while ago, I was tasked with a database project that had a CLR dependency and working through the dependency.  Here’s how to have a database project that depends on a CLR project and works for Deployment. 

Let’s create a simple Database Project with just 1 table for Customers.  In the Customer table  we’ll store an email that will have a constraint for proper email format written in CLR.  We’ll also build the dependency between the projects and configure proper deployment.

So first we need to create the Database Project called CLRDependency:

And a simple Customers table:

CREATE TABLE [dbo].[Customers]
(
    pkCustomerID int NOT NULL IDENTITY(1,1),
    FirstName varchar(25) NULL,
    LastName varchar(25) NOT NULL,
    EmailAddress varchar(50) NOT NULL
);

Next let’s add a new CLR project called CLRFunctions:

When creating this, it will ask for a database to reference.  Since we haven’t created the database yet you can just click Cancel.  Next Create a UserDefinedFunction called ValidateEmailAddress.cs with the following code:

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Text.RegularExpressions;

public partial class UserDefinedFunctions
{
    [Microsoft.SqlServer.Server.SqlFunction]
    public static SqlBoolean ValidateEmailAddress(SqlString emailAddress)
    {

        return new SqlBoolean(Regex.IsMatch(emailAddress.Value, @"^([\w-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([\w-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)$"));
    }
};

In Solution Explorer, Right-click on References in the database project and click on Add Reference, then select the project :

You’ll see CLRFunctions appear under References:

Now you only have to build the CLRFunctions Project and the Assembly will appear in your Database Schema View:

Now we need to create the UserDefinedFunction in the Database Project so Right-Click on Functions and Add a Scalar Function:

CREATE FUNCTION [dbo].[ValidateEmailAddress]
(
    @EmailAddress nvarchar(50)
)
RETURNS BIT
AS
    EXTERNAL NAME [CLRFunctions].[UserDefinedFunctions].[ValidateEmailAddress];

Lastly let’s create the Check Constraint:

ALTER TABLE [dbo].[Customers]
    ADD CONSTRAINT [CK_ValidateEmailAddress]
    CHECK  ([dbo].[ValidateEmailAddress](EmailAddress) = 1);

So now we have the two projects working together and everything is great.  However if you build it, you’ll see this:

------ Build started: Project: CLRFunctions, Configuration: Debug Any CPU ------
  CLRFunctions -> C:\AMLProjs\Local\CLRDependency\CLRFunctions\bin\Debug\CLRFunctions.dll
------ Build started: Project: CLRDependency, Configuration: Debug Any CPU ------
  CLRDependency -> C:\AMLProjs\Local\CLRDependency\CLRDependency\sql\debug\CLRDependency.dbschema
------ Deploy started: Project: CLRFunctions, Configuration: Debug Any CPU ------
Error: Cannot deploy. There is no database connection specified. To correct this error, add a database connection using the project properties.
------ Deploy started: Project: CLRDependency, Configuration: Debug Any CPU ------
    Deployment script generated to:
C:\AMLProjs\Local\CLRDependency\CLRDependency\sql\debug\CLRDependency.sql 

    The deployment script was generated, but was not deployed. You can change the deploy action on the Deploy tab of the project properties.
========== Build: 2 succeeded or up-to-date, 0 failed, 0 skipped ==========
========== Deploy: 1 succeeded, 1 failed, 0 skipped ==========

That’s not good, we have a failure.  However this is easily solved.  Just uncheck the deployment option on the CLR Project in the solution properties:

And now when you deploy you will see:

------ Build started: Project: CLRFunctions, Configuration: Debug Any CPU ------
  CLRFunctions -> C:\AMLProjs\Local\CLRDependency\CLRFunctions\bin\Debug\CLRFunctions.dll
------ Build started: Project: CLRDependency, Configuration: Debug Any CPU ------
    Loading project references...
    Loading project files...
    Building the project model and resolving object interdependencies...
    Validating the project model...
    Writing model to CLRDependency.dbschema...
  CLRDependency -> C:\AMLProjs\Local\CLRDependency\CLRDependency\sql\debug\CLRDependency.dbschema
------ Skipped Deploy: Project: CLRFunctions, Configuration: Debug Any CPU ------
Project not selected to build for this solution configuration
------ Deploy started: Project: CLRDependency, Configuration: Debug Any CPU ------
    Deployment script generated to:
C:\AMLProjs\Local\CLRDependency\CLRDependency\sql\debug\CLRDependency.sql

    Creating CLRDependency...
    Creating [CLRFunctions]...
    Creating [dbo].[Customers]...
    Creating [dbo].[ValidateEmailAddress]...
    Creating CK_ValidateEmailAddress...
    Checking existing data against newly created constraints
========== Build: 2 succeeded or up-to-date, 0 failed, 0 skipped ==========
========== Deploy: 1 succeeded, 0 failed, 1 skipped ==========

Something to remember is that you can actually deploy this to a server that does not have CLR Enabled, however when you go to insert anything into that table, it will fail with this error: 

Msg 6263, Level 16, State 1, Line 1
Execution of user code in the .NET Framework is disabled. Enable "clr enabled" configuration option.

So make sure you enable CLR on the Server.  Now onto testing (you do test your work right?)

USE CLRDependency;
GO
INSERT INTO dbo.Customers
    (FirstName, LastName, EmailAddress)
VALUES
    ('Aaron', 'Lowe', 'AaronL@AaronLowe.net');
GO
INSERT INTO dbo.Customers
    (FirstName, LastName, EmailAddress)
VALUES
    ('Aaron', 'Lowe', 'boo');
GO

Results as expected:

(1 row(s) affected)
Msg 547, Level 16, State 0, Line 1
The INSERT statement conflicted with the CHECK constraint "CK_ValidateEmailAddress". The conflict occurred in database "CLRDependency", table "dbo.Customers", column 'EmailAddress'.
The statement has been terminated.

Wait, let’s back up:

Sweat mars your brow as you start to feel panic set in.  You reach for your mountain dew can and quickly put it down so no one will see your hand shaking (plus don’t want to spill a good dew).

Still need to back up further:

You were minding your own business watching a great SQLPASS webcast and thinking of how you were going to implement those cool things you learned.  At least, that was until you got THE PHONECALL.  Your boss calls and says, “Hey there’s this project going on that is going to need some database help, I’ve told them you’re just the person to do it.  So I’ve scheduled you in a conference call tomorrow morning first thing.  Sounded pretty straight forward so should be no problem.” You then start downloading your brain to notepad and internet links so you don’t forget the SQLPASS presentation.

You come in the next day, get your morning Mountain Dew (hey it’s my story and I’m not much of a coffee drinker) and jump on the conference call.    That is when you hear this:

“Glad you could join us; your boss said you were the best, so we’re expecting great things from you.  What we need is a database to store the Excel documents we’ve been working on for the past year.  We’ve just been emailing them around, so we’ll need to import the data and be able to create reports.  I already spec’d it out in Access and don’t think it should take more than a week, but with you on here, it should take less time!  This is good because we’re already committed to deadlines in about a month. I will email you the Excel documents, once you look and understand them we will talk later this afternoon to answer any questions you have. K, thanks, bye”

You start seeing visions of SSIS, SSRS and SQL go through your head and you’re wrapping your brain around everything that needs to be done, when you get the email: “Here’s a couple of the first type of spreadsheets, there are actually 3 different types of spreadsheets but it’s all related.  Should be no problem for a wiz like you, huh? When you’re ready for the rest, let us know and we can create a CD of them – think there’s about 300 or so.  I forgot to mention on the call that while we have a deadline in a month, we promised to start getting reports out next week, so I’m really looking forward to seeing the reports this week.”

Sweat mars your brow as you start to feel panic set in.  You reach for your mountain dew can and quickly put it down so no one will see your hand shaking (plus don’t want to spill a good dew).   Then a thought creeps into your head, you initially reject it, and look around to make sure no one heard it, but as you think about it, the thought starts making sense and you start to smile.  You now have the way to meet or possibly exceed the expectations and you can look like the hero.  So you say to yourself, “Don’t worry, I’ll just import manually, create the code as I go to be quick, then I can go back and clean it up later”

———————

The above scenario is not that uncommon in our field.  We are brought in as the experts and expected to produce things quickly.  The business group does not know and possibly does not even care about everything we do. But, really, it is not their job, if they did, you would not have one.  Things like code quality, code coverage, and security are your job, not theirs.

However, I am still seeing too many people try to be the hero by working around the clock or reducing code quality to meet the requirements as soon as possible.  Let me refute some more common reasons:

I will have plenty of time…

The idea that you will all of a sudden have this huge amount of free time to do performance improvements, proper error handling, better naming conventions, etc. is a myth.  More likely, you will be pulled onto another project and then someone will knock on your door in a year or two asking you to explain this really bad code they found.  If on the other hand you are experiencing large blocks of free time, please send me a note as I might have some work for you.  Companies pay you to work, which is the opposite of having free time.

I’ll just do it this once…

If you put that kind of code out there, you will probably end up doing a lot of manual process/manipulation.  This means when it is time to actually go-live, you scramble to get it all working, again reducing quality.  I have often seen “While it’s in development I don’t have to worry about getting it scheduled.  Oh that process didn’t work; I’ll just manually fix it.”  If you have to fix it manually in development, you will have to fix it manually in prod.

They just don’t understand

Our job should not just be seen as a cost center it should be seen as an asset.  That does not mean we just produce great code/functionality, it means we help the business achieve the business goals.  In other words, we work “with” not “for” the business.

Back in the Exchange 5.5 days, I did some work as a systems administrator.  We discovered that some people were using mailboxes that had up to 500 MB in them, while the average was around 100; I researched best practices and did all this analysis.  I finally brought the suggestion to my boss, “We should limit everyone’s mailbox to XX size that way we don’t have space issues” His response “This is obviously the way our people work and if it is, why we shouldn’t just invest in more space so they can continue to work that way?”

I had never considered the business, only the technology.  When you just have technology on the brain and not the business or the people you come up with solutions for the technology. “My system would be perfect except for all those users”

Take the time to explain what you need to accomplish so you can create what they want you to accomplish.  They want it sooner rather than later? Then reflect some of those decisions back to them, even empowering them to make some.  “Yes I can get that really cool function out there this week, but everyone in the world will have access to it,  With a couple more days I can also have it secure.  Which would you like?”  Of course, this means that you have to think more strategically about the project, which can be difficult at times.  We all know how easy (and fun) it is to just jump in there and start producing code”.  Transparency is the key.

But I’m always having to change things

Realize that some refactoring will always be necessary. The days of going away for 6 months to create something and come back to do your big “ta-da” are long gone.  People want things faster with more agility so things can change as we go along.    This means more pressure to produce faster, which means refactoring and adding additional functionality as you go.  However every line of code you write should be quality.  I don’t know how many times I have seen Proof-of-Concepts (POCs) turn into the actual production system.  Just because it is a POC from a business perspective does not mean that it should be a POC from a code quality perspective.  I actually saw someone code up a textbook T-SQL injection (put the stored procedure parameter straight in the where clause without any validation), with the intention of fixing it later.  If it is wrong to do later, it is wrong to do now.

In the development world, there are many tools for code coverage, unit testing, refactoring, etc.  While there are some tools out there like VSDBPro unit testing, VSDBPro and Red Gate refactoring, etc., the tools in the database world are not as mature.  There is no IDE that forces white space or automatically documents for you.  We do not have tools where we can just do a few clicks and have end-to-end tests.

However, that is no excuse to be a gunslinger and just shoot from the hip.  We are professionals and should be creating professional code that is readable, documented and have a standard of quality.

If you are struggling with that, adopt a Test Driven Development (TDD) philosophy or something else that will force you to think about or do things better.

Or maybe it is taking you 5 minutes to figure out what is going on in a stored procedure you just wrote a couple days ago.  If so, let’s be honest you failed in either code readability and/or documentation. No excuses, there are even good free tools out there. Whitespaceisreallyagoodthing.

Code Quality, Code Readability and Code Documentation is Important

I prefer to assign permissions to roles and add users to roles which mirrors the user/groups of windows, so going down that path we have to create a role (going under Security –> Roles):

CREATE ROLE [TestRole]

Now comes the part that isn’t very intuitive.  To associate a user with a role you have to go back to the Solution Explorer and actually create a generic user script file (see here), you can call it whatever you want and place it wherever you want in the project, however if you reverse engineer the database it will be called <ProjectName>.rolememberships.sql under the schema objects folder:

EXECUTE sp_addrolemember @rolename = N'TestRole', @membername = N'TestUser';

Now we have the role defined and the users as part of the role, we can start adding permissions, this is where you’ll have loads of fun provided you enjoy manually editing xml documents.  Again in Solution Explorer under Properties you’ll see a Database.sqlpermissions.sql file:

In there you’ll see some xml with examples:

<?xml version="1.0" encoding="utf-8"?>
<?XML:NAMESPACE PREFIX = [default] urn:Microsoft.VisualStudio.Data.Schema.Permissions NS = "urn:Microsoft.VisualStudio.Data.Schema.Permissions" />
  

So as you can see while it’s not that intuitive it is pretty straight forward.

So first we take the Server project and create the login for the user (if don’t know the difference between Logins and Users go read K. Brian Kelley’s post here, I’ll wait).  If you are in Schema view (and if not, why not?) make sure to go under Server Level Objects –> Security –> Logins, not the Security –> Users path, as you want a Server Login, not a master db User.

You’ll also notice in the Add new item dialog that there is a different template for each:

• Login (Windows Auth)
• Login (Sql Server)

At this point you might also be tempted in creation of the login to assign a default database, avoid that temptation.  When you assign a default database, that user database must already exist, however when you deploy the server project (i.e., the master database) there’s a good chance the user database won’t exist yet, I promise we’ll take care of it later though.

CREATE LOGIN TestLogin WITH PASSWORD = 'P@ssw0rd1'

So now that you have a server login in the server project we need to create a database user (remember to reference the server project in the database project).  This time you can go under the Security –> Users path.

CREATE USER [TestUser]
    FOR LOGIN [TestLogin]
    WITH DEFAULT_SCHEMA = dbo;

Now we have a login and a user created, lastly we set the default database in the database project post-Deployment script:

ALTER LOGIN [TestLogin] WITH DEFAULT_DATABASE=[$(DatabaseName)]

As you see I’m using the variable of the database name since I can change the database name for each deployment:

Taking a look at pertinent parts of the deployment sql scripts we have (remember this is in SQLCMD mode):

/*
Deployment script for master -LocalServer.sql
*/
:setvar DatabaseName "master"
GO
USE [$(DatabaseName)]
GO
PRINT N'Creating TestLogin...';
GO
CREATE LOGIN [TestLogin]
    WITH PASSWORD = N'P@ssw0rd1', DEFAULT_DATABASE = master;
GO
------------------------------
/*
Deployment script for AdventureWorks - AdventureWorks.sql
*/
:setvar DatabaseName "AdventureWorks"
GO
CREATE USER [TestUser] FOR LOGIN [TestLogin];
GO
ALTER LOGIN [TestLogin] WITH DEFAULT_DATABASE=[$(DatabaseName)]
GO