How to interpret Disk Latency

I was analyzing IO stats in one of our SQL Servers and noticed that IO latency (Read / Write) are very high. As a rule of thumb, we know that Disk Write Latency is should be below 20 milliseconds. But in this case, Disk Write Latency was 6,000 milliseconds. I was extremely worried and I immediately checked this with the storage team and as per their stats, storage behaves normally.

See below graphs for three storage metrics, Disk Queue Length, Page Writes, and SQL Disk Write Latency.

Most of the time we need to focus on multiple metrics to understand the full picture. So just pay your attention to the red marked areas. During that period, Disk Queue Length is high (close to 7,000 IO operations), and Disk Latency also high. Even it’s the same for the 2nd time slot as well. So clearly, those two has some correlation.

If the Disk Queue Length is high means, Disk Write Latency is also high. 😊

Figure-1: IO Stats

Can you think of why Disk Write Latency is high when the Disk Queue Length is high? Well, that is because Disk Queue Length is also taken when calculating Disk Write Latency.

What is Disk Write Latency: - It is the time between when SQL Server creates an IO request and the time it is completed.

So if the IO request is waiting in the queue means, that time is also added for the latency. That is how Perfmon calculates the Disk Write Latency. Interesting huh!

Check here for more details.

Cheers!