How to interpret Disk Latency

I was analyzing IO stats in one of our SQL Servers and noticed that IO latency (Read / Write) are very high. As a rule of thumb, we know that Disk Write Latency is should be below 20 milliseconds. But in this case, Disk Write Latency was 6,000 milliseconds. I was extremely worried and I immediately checked this with the storage team and as per their stats, storage behaves normally.

See below graphs for three storage metrics, Disk Queue Length, Page Writes, and SQL Disk Write Latency.

Most of the time we need to focus on multiple metrics to understand the full picture. So just pay your attention to the red marked areas. During that period, Disk Queue Length is high (close to 7,000 IO operations), and Disk Latency also high. Even it’s the same for the 2nd time slot as well. So clearly, those two has some correlation.

If the Disk Queue Length is high means, Disk Write Latency is also high. 😊

Figure-1: IO Stats

Can you think of why Disk Write Latency is high when the Disk Queue Length is high? Well, that is because Disk Queue Length is also taken when calculating Disk Write Latency.

What is Disk Write Latency: - It is the time between when SQL Server creates an IO request and the time it is completed.

So if the IO request is waiting in the queue means, that time is also added for the latency. That is how Perfmon calculates the Disk Write Latency. Interesting huh!

Check here for more details.

Cheers!

Transitioning of SQLOS to SQLPAL

​The SQLOS​

Prior to SQL Server 2005, SQL Server runs as a user mode process like any other Windows application and it depends on Windows OS to manage hardware resources

​. The hardware resource management includes the following;

• ​
  ​Processor scheduling
• Memory management
• Network 
• Disk I/O​

In SQL Server 2005, the hardware resource management part has been incorporated into SQL Server as a separate component known as SQLOS (SQL Server Operating System). The objective of this initiative was to improve SQL Server performance by assigning database own tasks manage itself, so that SQL Server can perform better. The SQLOS then acts as a non-preemptive schedule to service database own tasks by cooperating with Windows OS. As a result, SQL Server knows how to manage the hardware resources better than Windows OS. 

The SQLPAL

SQLPAL-SQL Platform Abstraction Layer, is an extended version of SQLOS to facilitate platform independent behaviour with the introduction of SQL Server on Linux initiative. To achieve this objective Microsoft has used a research project called, Drawbridge. Drawbridge uses essentials Windows components inside SQLPAL to interact with SQL Server and then SQLPAL will interact with the underlying OS. Essentially, SQL Server on Linux still thinks SQL Server is working on Windows. See below Figure;

The SQLPAL interact with underlying OS (Windows or Linux) through its host extension implementation. So host extension has the actually platform specific implementation for Windows OS as well as Linux OS. The biggest advantage of this innovative approach is, you did not need to rewrite SQL Server where more than 20+ years of development happened on Windows and contains thousands of millions of code. 

​How about the performance?

The obvious doubt is, adding an additional abstraction layer (SQLPAL) will negatively impact the performance?​ The testing shows that SQL Server performs almost the same against both the OS. In fact, the performance of SQL Server 2017 is far better than SQL Server 2016. 

​Need to know more: Click here​

Cheers!

How to fix cardinality estimation anomalies [Video]

Use the link mentioned below to watch the presentation that I delivered for PASS DBA Virtual Chapter about Filtered Statistics. 

http://bit.ly/2tlFyNW

Understanding Skewed Data in SQL Server

My latest article. http://bit.ly/2qd4rtl

Understanding Skewed Data in SQL Server bit.ly Introduction I recently did some research to analyze skewed data distribution in SQL Server. This article is the outcome of my analysis to share with SQL Server community. SQL Server understands its data distribution using statistics. The SQL Server query optimizer then uses the statistics to calculate estimated row counts when executing the queries submitted …

How to get all SQL Server internal tasks

Use the below statement to get all SQL Server internal task details.

SELECT * FROM sys.dm_exec_sessions

WHERE is_user_process=0

The below statement too seems correct but it does not give all the internal tasks.

SELECT * FROM sys.dm_exec_requests

WHERE status='background'

SQL Server 2017 becomes the first commercial RDMS with built-in AI support

AI Into SQL Server 2017

Today (4/19/2017), Microsoft announced the release of SQL Server 2017 CTP 2.0 (formerly known as SQL Server vNext). It has pretty exciting features including deep learning AI capabilities. Deep learning model demonstrated today at Microsoft Data Amp is used some medical data to figure out the probability of getting a lung cancer for a particular patient. Faster early detection of such potential disease helps to reduce the healthcare cost and also to have a healthier life. All these happens under one roof meaning, in one single data platform which is SQL Server. All the analytics capabilities of the deep learning system are now integrated into SQL Server which makes super easy for data scientists to work and analyze data. The application has to call just a single stored procedure to do the deep learning analytics of a patient, it's that much simple.

The data, the model, and the intelligence are all built into SQL Server which will eliminate so many challenges of building today’s AI application by using so many different types of tools.

Python Integration

SQL Server 2017 has a built-in support for Python code inside T-SQL queries which gives so much powerful programming capabilities. SQL Server 2016 supported R integration as well. You can use sp_execute_external_script system stored procedure with T-SQL to call R or Python language which external to T-SQL.

Adaptive Query Processing

Adaptive Query Processing is one of the extremely powerful database engine enhancements in SQL Server 2017. With this new feature SQL Server has the intelligent to learn itself about query processing and improve itself and generate better execution plan based on past query execution with no human intervention. This is an amazing feature that I was dreamed for years.

SQL-Graph

Graph data represents many-to-many relationships and typically relational databases need to store that kind of relationships as one-to-many relationships. SQL Server now supports graph data and processing of graph data. You can create graph objects which create nodes and edges in SQL Server 2017. This essentially extends the SQL Server query processing capabilities. This query capability is fully integrated into SQL engine in SQL Server 2017. Previously, you had to use a special type of databases called graph databases to store this kind of data.

One platform which supports both Windows and Linux

All the features of SQL Server 2017 now support both Windows and Linux platforms.

In a nutshell, SQL Server is not just another database server, it is an extremely powerful data platform eco-system because it combines AI and analytics into one product. Everything is built in. So much easier than using hundreds of different tools. I can’t wait to get some hands-on of these new features.

Cheers!

How To Generate In-Memory OLTP Migration Checklist

In-Memory OLTP, code named Hakaton is a separate database engine introduced in SQL Server 2014 and it is combined with SQL Server traditional database engine. So when you install SQL Server, you will get both database engines. Modern computers/servers are getting more physical memory (RAM). Windows Server 2016 support 24 TB of physical memory. So it is clear the future is more towards In-Memory databases regardless of database type.

I’d recommend reading Microsoft published whitepaper by Kalen Delaney on In-Memory OLTP. here                                                                                                                            

In-Memory OLTP simply means you can create memory optimized tables as well as natively compiled stored procedure. In other words, you can store the schema of the entire table and data in memory. However, it obviously has some limitations too. Those limitations will be addressed in every new release of SQL Server.

It is possible to migrate the tables and other objects in existing databases to memory-optimized structures. Since it has limitations, the first step would be to identify the list of objects which are eligible for the migration. The SSMS (SQL Server Management Studio) provides a feature (wizard) that you can use for this purpose. The below steps will go through the details of the wizard against the sample database, AdventureworksDW2016CTP3 which you can download and install.

Follow the below steps to generate the checklist.

1. Connect to the server in SSMS.
2. Right-click on the database and select Generate In-Memory OLTP Migration Checklist option. (See below screenshot)

o

3. Click Next on the first page of the wizard.
4. In Choose Objects screen, you can select the tables, stored procedures or both. You also can specify a location for the checklist to be saved. Then click on Next. (See below screenshot)

5. In Summary page, you can generate the PowerShell script for this task. This is optional. Click on Finish.
6. It takes some time depending on the number of objects (tables, triggers, foreign keys, indexes, stored procedures, etc) in the database. Because it checks all the dependent objects against the In-Memory OLTP rules.

You can view the checklist generated by the Advisor in the location you specified in step #4.

It creates the subfolders under database name folder for each object. See below screenshot.

The contents of each folder look like below. Below screenshot displays the checklist for stored procedures.

Open each HTML file to see the details of the analysis.

I will write another blog post on how to analyze the checklist information.

Cheers!