Next up was board announcements such as Wayne Snyder and Rick Heiges both rolling off the board.  While they were bringing Wayne up on stage they were showing quotes from people within the community about Wayne, all of which were very moving, however the one that I thought really stood out was about (and I apologize that I didn’t catch who said it) how Wayne transformed SQLPASS from a technical conference into a family reunion.

Keynote

Then onto the final keynote done by Dr. David DeWitt (site), you can download his presentation here (this is a summary of that presentation, if you are really interested, go download it or better yet, watch it).  Dr. DeWitt was a professor for years and has mastered the ability of explaining complex concepts.  During the presentation we were able to email questions which were answered after the presentation.  Today he decided to tackle Big Data.

In 2009 there was 0.8 Zettabyte (ZB) (1 million petabytes/1 trillion terabytes/1 quadrillion GB) and by the years 2020 the expectation is to have 35 ZBs (growth factor of 44).  That’s a lot of data, much of it coming from sources such as sensors, Web2.0, Web clicks, etc.  However much of this data while valuable to store, we don’t necessarily care about ACID properties or relational integrity or other things that would utilize a traditional RDBMS. Point being as always, right tool for the right job.

So in these cases, what is the right tool?  Well with Wednesday’s announcement, looks like Microsoft is putting their weight behind Hadoop and MapReduce, which offers:

• Scalability and a high degree of fault tolerance
• Ability to quickly analyze massive collections of records without forcing data to first be modeled, cleansed and loaded
• Easy to use programming paradigm for writing and executing analysis programs that scale to 1000s of nodes and PBs of data
• Low up front software and hardware costs

So what’s the system look like:

1. Hadoop Distributed File System (HDFS) –  objectives are load balancing, fact access and fault tolerance, designed with the expectations that hardware/software failures
2. MapReduce – framework for writing/executing distributed, fault tolerant algorithms – 2 functions map which divided a large problem into smaller problems and then performs the same function on all smaller problems and reduce which then combines the results.
3. Hive & Pig – Hive was created by Facebook as a and is SQL-like, while Pig was created by Yahoo and is more procedural; both target MapReduce jobs.  However due to the complexity of MapReduce, HiveQL was created to combine the best features of SQL with MapReduce
4. Sqoop – package for moving data between HDFS and relational DB systems via command line load and unload utilities

He then showed some performance metrics of SQL PDW and stated

I assert that it is MUCH easier to add support to SQL Server PDW for unstructured data (w/o having to load it), improved scalability, and fault tolerance than it is to ever get competitive performance from a Hadoop-based system

But again the point being that both of these type of systems (RDBMS and Hadoop)  are going to be working together, it will not be a case of choosing one or the other.

Sessions

After the keynote I sat in Adam Machanic’s (twitter | blog) Query Tuning Mastery for a bit before going downstairs to host the “SSIS for all, DBAs developers, etc.” table at the Birds of a Feather luncheon with Matt Masson (twitter | blog).  Ted Krueger (twitter | blog) had to leave leave early and Mike Walsh (twitter | blog) was looking for volunteers.  Had some good conversation ranging from “What is SSIS” to “How do I do meta-driven SSIS”

After that I jumped into Rewrite Your T-SQL for Great Good by Jeremiah Peschka (twitter | blog).  Jeremiah’s slides as always excellent, and I have a similar presentation entitled Writing Professional Database Code so I figured I’d go and borrow check it out.  I was glad I did as it had a different focus and perspective.  This just underscored to me again that each person brings their own experiences and perspective which is valuable.  In all honesty it is a common excuse that I used (as well as others) as to why I couldn’t present or blog, because the content was already out there.   So lesson learned again, stop making excuses. Anyway he talked about consistency and gave a link to open source Unit Testing tools.  As a user of TFS, I haven’t had much experience with these and look forward to investigating them.  He gave a lot of query performance options and even threw down the gauntlet for everyone who uses “Distinct” in a query to explain the necessity of it as most of the times it’s used due to either a bad data model or bad joins.  Great stuff!

Last I went to Are you a Linchpin? Career Management lessons to help you become indispensable.  This was a panel discussion with Q&A at the end Linchpin: Are You Indispensable by Seth Godin as the starting point of the discussion.  The panel was made up of:

• Andy Warren (twitter | blog)
• Brent Ozar (twitter | blog | )
• Jeremiah Peschka (twitter | blog)
• Kevin Kline (twitter | blog)
• Louis Davidson (twitter | blog)
• Stacia Misner (twitter | blog)
• Thomas LaRock (twitter | blog)
• And Moderator: Andy Leonard (twitter | blog)

One of the interesting things that Kevin said (attributing credit to Brent) was that

People are going to remember you for 1, 2, maybe 3 adjectives.  What adjectives do you want to be known for?

My mother had always told me something similar:

People won’t remember what you did, but they will remember how you made them feel

Anyway it was a great conversation with some interesting Q&A, would love to see more of these type of things.

Whew, that’s it for the summit, had a great time and looking forward to next years!

1. Required 9′s & Protection – Integration Services is now Integration Server; HA for StreamInsight; AlwaysOn
2. Blazing-Fast Performance – Column Store Indexes; improvements across the board
3. Rapid Data Exploration – Power View + PowerPivot; Admin from SharePoint; Reporting Alerts
4. Managed Self-Service BI – same as #3
5. Credible Consistent Data – BI Semantic Model (BISM); Data Quality Services (DQS);Master Data Services (MDS)
6. Organize Compliance – Expanded Audit (user defined and filtering); User Defined Server Roles
7. Peace of Mind – Production-simulated distributed testing; System Center Advisor & Management Packs; Expanded Support – Premier Mission Critical
8. Scalable Data Warehouse – SQL Server Appliances; HW + SW + Support – Just add Power; choice of hardware
9. Fast Time to Solution – Same as #8
10. Extend Any Data, Anywhere – Greater Interoperability, New Drivers for PHP, Java and Hadoop; ODBC Drivers for Linux & CDC for SSIS and Oracle; Beyond Relational: FileTable, 2D Spatial, Semantic Search
11. Optimized Productivity – SQL Server Data Tools (SSDT); Unified Across Database & BI; Database & Targeting Freedom
12. Scale on Demand – AlwaysOn; Deployment across Public & Private; Elastic Scale

We saw demos of AlwaysOn, Column Store Index, Data Quality, and Semantic Search, plus they showed off the SQL Server Appliances.  The thing that really stood out to me in the keynote was the Semantic Search Demo.  The presenter took two pdf files and utilizing TSQL was able to relate them conceptually.  Looked very cool, you can go here for more info.

After that I attended Matt Masson’s (twitter | blog) session on Top 10 Things You Didn’t Know About SSIS in SQL Server Code Name “Denali”.  These were items that are post CTP3.  To kill some time while waiting for the session to start Matt went through a mock scenario of utilizing SSIS to create virtual friends to party with him and his cat (Mr. Wiggles) on Facebook using Facebook Services and StreamInsight Transform – it was absolutely hilarious!

1. Change Data Capture – as someone who has done CDC in SSIS this was very cool
2. ODBC Support
3. Connections Managers – Shared In-Memory Cache Across Packages using the shared project level connection manager
4. Flat File Source Improvements – supports varying number of columns and can parse embedded qualifiers
5. Package Format Changes – doesn’t persist default values; doesn’t use lineage id’s, instead uses ref id’s; sorted by name
6. Visual Studio Configurations
7. Scripting Improvements – can now set a break point in a script task/component and see data
8. Troubleshooting & Logging – Built-in reporting; Component Timing & Row Counts; Server level logging
9. Data Taps – can write out data to a csv location without code change (doesn’t log binary)
10. Server Management using PowerShell
11. Honorable Mention – Project Barcelona Metadata Discovery – crawler that crawls entire MS stack to determine Impact Analysis and Data Lineage

These were on top of all the other things we new about for SSIS in SQL 2012, and I have to say that I’m really looking forward to it!

Next I went to the Lighting Talks which is a cool idea.  The premise is to have a number of speakers, this one has seven.  Each speaker however is limited to 5 minutes and there is someone utilizing a timer and the crowd is encouraged to clap people off stage that go to long.  It was a lot of fun.  The two that really stood above the rest (in my opinion anyway) were Grant Fritchey’s (twitter | blog) “Backup Testing, The Rant ” and Niko Neugebauer’s (twitter | blog) “Build Up”.

Last was Database Development with SQL Server Data Tools Code-Named “Project Juneau” by Gert Drapers (blog).  As someone who has used Database Professional extensively, I was pretty skeptical going in since in that Unit Tests aren’t in SSDT.  As you would expect one of the big features is that it has support for Azure.  However some of the stuff that I did see was really cool:

• A True Debugger with break points and a Call Stack, with the ability to start in the .NET app
• Refactor can now reach into the application
• True Language Services – can right click to go to reference, etc.
• Drift Detection – if working in a connected stated it will poll database to see if any changes happened behind you and you can just drag and drop changes into project
• Easier way to save specific deployed versions
• Table designer where you see both code and the designer window
• SQL CLR is within the same project, no longer have to create a separate CLR project

Some things I still have to think if I like or not:

• Renamed Deploy to publish so that Projects are now Published
• Multiple objects are now in a single file, for example the indexes, keys are now defined in the create table script file

And the things I don’t like, until vNext Visual Studio + 3 months we’re missing:

• Data comparison
• Data Generation
• Unit Testing

Now for the great news.  It’s completely FREE!!  It will be delivered as part of the SQL Server install as well as part of the Visual Studio install or you can get it through  the Web Platform Installer to get it!  We’ve come a long way from requiring TFS 2005 Team System.  Looks like we are getting closer to a standard SQL Server development environment.  That was it for day 2, next up is the 3rd and final day.

• Provide 1 Million hours of education, 430k hours provided so far
• Grow to 250k members, 80k so far
• 5 Global regions, 1 so far

PASS Summit is the largest event focused on SQL and BI:

• 189 Sessions, 5 tracks (57 MS, 11 CAT)
• 204 speakers (93 MVPS, 11 MCM)
• SQL Server Clinic : SQLCAT architects, CSS engineers
• Expert Pods: MS Engineers, MVPs, 9 focus areas
• Over 5000 registered attendees

Ended by stating the purpose it to

Build connections that will last you a lifetime

He then introduced Ted Kummert with guests Denny Lee (blog | twitter) and Amir Netz (twitter).  Announcements:

• Denali will be released the first half of 2012 and will be called SQL 2012
• Project Juneau office title is SQL Server Data Tools
• Project Crescent official title is Power View
• Apache Hadoop-based distribution for Windows Server and Windows Azure
• ODBC Drtiver and Add-in for Excel, both for Apache Hive
• JavaScript Framework for Hadoop
• SQL Server and SQL Server PDW connectors for Apache Hadoop
• Partnering with hortonworks

Ted went on to describe the focus going forward:

• Any data, any size, anywhere
• Manage and process data of all types
• Mission-critical scale from on premises to cloud
• Common  management and development

Lastly they demoed a product codenamed “Data Explorer” which was simply described as a way to enable data discover, enrichment and publishing.  They actually demoed it utilizing the Metro UI and showed the ability to do a 5 way join between SQL Azure, Excel, Data Market and WCF calls.  The other thing that made this compelling was that it enables interactive mobile reporting with demo’s on an iPad, Samsung Table, Windows Phone and Windows Slate.

After that I went to a Hands on Lab to look at the new SQL 2012 Always on functionality (yes it is turned off by default).  Got my copy of SQL MVP Deep Dives vol 2 and ended up in Adam Jorgensen (blog | twitter) session called Zero to Cube – Fast Track to SSAS Development.  Adam is just awesome.  He’s come up to our SQLSaturday in Chicago and presented and if you have a chance to see a session or just talk to him, do it.  Here’s his picture:

Here was the Agenda:

• APPLY Magic
• Grouping Sets
• TOP / OFFSET-FETCH
• Sequences
• Windows Functions
• Intervals
• Other T-SQL Improvements in Denali

First a truism – SQL is a set based language, the concepts and logic that exist are designed for set based activities.  He also went over how the Query is written:

1. SELECT
2. FROM
3. WHERE
4. GROUP BY
5. HAVING
6. ORDER BY

vs. how it’s processed:

1. FROM
2. WHERE
3. GROUP BY
4. HAVING
5. SELECT
6. ORDER BY

The session was, as you can imagine, pretty intense.  Itzik obviously loves what he was talking about and is very passionate.  He talked about how to improve standard SQL as well for migrations between other SQL implementations.

As far as some of the new features of T-SQL in Denali that was mentioned:

• OFFSET / FETCH (think paging)
• Sequences (independent identity objects that can be referenced and keep proper seeding)

• Windows Functions (this set functions) – these have been expanded tremendously to include things like LAG, LEAD, FIRST_VALUE, LAST_VALUE, distribution functions (PERCENT_RANK, etc.), ROWS and RANGE 
• Conversion – PARSE, TRY, TRY_CONVERT (returns NULL if can’t convert), TRY_PARSE
• Date and Time – EOMONTH (end of month), DATEFROMPARTS and similar functions to build date, time and datetime data from different parts
• Logical – CHOOSE, IIF – these were added to improve Access migrations
• String – CONCAT this is used for string concatenation, similar to the + operator, however it will automatically convert NULL to to strings; FORMAT – this brings in .NET format functionality, however it’s slower than native SQL formatting (probably because it utilized CLR behind the scene)
• Mathematical – LOG now supports the ability to indicate the base
• Improved Error Handling – THROW, this finally allows us re-throw the original error to be bubbled up.  Until now we’ve had to utilize user errors
• EXECUTE WITH RESULT SETS – This allows us to call an SP and guarantee the shape of result set.  We submit an expected result set shape which if the return set doesn’t match SQL will try to implicitly convert to, however it could fail if it can’t return what is specified.
• Metadata discovery – removes the SET FMTONLY – this allows us to interrogate and return the metadata
• FORCESCAN and FORCESEEK – this allows us to force an index seek or index scan for a particular query

Another great session and it’s cool to see the advancements being made within T-SQL.  More to come later.

Having spent a significant time in SSIS already and even had thanked a few people, including Andy, a couple years ago when I was learning how to load data warehouses, I was really looking forward to this session.

Agenda:

• Frameworks
• Using Metadata
• Centralized Logging
• Monitoring & Analytics
• Logging in SQL Server “Denali”
• Configurations
• Expressions
• Deployment
• Data Flow Internals
• Measuring Performance
• Data Flow Best Practices
• Scripting
• Design Patterns

First was the de facto disclaimer – This is a level 300 session and it’s difficult to stay at 300 through everything – Configurations, deployment, expressions, etc. They’re covering many areas some of which we might be more expert at than others, but they were looking at being comprehensive.

Andy took the stage first and presented his work on SSIS Frameworks that he’s been honing and presenting for awhile.  It’s some great stuff and if you’ve never looked at it, do yourself a favor and go there now, I’ll wait.  While he was presenting he said the best line, which I ended up tweeting – “You can’t avoid SSIS Pain, only pick when you feel it.”  Matt then talked about some of the upcoming features in Denali and plugged his session later in the week where he’ll be making some “announcements”.

Next up, Tim took the stage and talked about the configurations, expression and deployment at which point Matt chimed in with some of the new deployment options in Denali and since we were running early, he went on to talk about Data Flow Internals before breaking for lunch.  After lunch Matt finished discussing the date flow task and then Tim did scripting.

The day ended with Matt presenting Design Patterns and listed out design patterns he “could” talk about:

• Using the DB Engine
• Lookups
• Parallel Processing
• Avoiding Transactions
• Surrogate Key Generation
• Partitioning
• CDC
• Null Value Substitution
• Balanced Data Distribution
• Using Multiple Transforms
• Late Arriving Facts
• Work Pile Pattern – Scale Out
• Slowly Changing Dimensions
• Change Detection
• XML Patterns

Course we only had time to look at 5 of them (great presentations leave you wanting more, right?).

Overall all three presenters were excellent – they were both knoweldgable passionate about their material, the slide decks were very professional, demo’s were also well prepared and they handled questions while staying on schedule very well.  They were even monitoring twitter to get questions during the session as well as honoring requests to just keep going and not worry about the time.  As with any session this long they were right about there being some things that I already knew and some things I didn’t.  However even the things I knew, it was good to have a refresher on the why and even get the re-assurance that I do something right once in awhile.

Now onto some stuff I learned:

• Got pretty deep into how the Data Flow worked, talking about the Buffers, Execution Trees and Backpressure.
• Discussed when to use synchronous, asynchronous and blocking transformations
• Discussed SSIS strategy for performance – Operations Volume Application Location (OVAL)
• All LOB columns are paged to disk, no exceptions, but you can change the location
• Currency/money data types are more efficient than doubles
• Don’t use the SQL Destination (I’ve never used this but had seen the 30% faster claims)
• There is an actual good use case to use the SCD Wizard.
• Cascading lookups
• How to design for scale out scenarios – this is really cool and can even utilize Service Broker, I’m really pumped to try this one out.
• When not to use SSIS to move data
• Perf Counter – Buffers Swap to Disk should always be 0, otherwise you’re using too much (or don’t have enough) memory
• Why SQL 2005 expressions run time values trump design time and why SQL 2008 is the reverse
• Don’t use SSIS Transactions (i.e., transaction property within the packages)

Thanks to Andy, Tim and Matt for a great session.  They obviously spent a lot of time preparying, in fact they were even tweeting Sunday about practicing.  Andy later said they were making tweaks during their practice session and even decided to drop talking about source control in favor of additional Design Patterns.  Very good content, thanks to the three presenters!

Finished the day at a networking event at Lowell’s.  Looking forward to  Tuesday’s “Advanced T-SQL for SQL Server 2008 and Denali” by Itzik Ben-Gan (blog)