MMCUG Blogs

Question:

I am building my mailbox server role for Exchange 2007 using Cluster Continuous Replication (CCR) http://technet.microsoft.com/en-us/library/bb124521.aspx. I have seen in the majority of recommendations from Microsoft in regards to disk configuration that it is better to use RAID 10 over RAID 5 for the stores. My RAID 5 configuration would use a (3+1) disk set on the SAN. If I were to configure a RAID 10 it would be in a (2x2) disk set. My disks for the Exchange 2007 stores are 300 GB 15K RPM Fibre Channel (FC) 3.5 inch disks. I have used the Microsoft Exchange 2007 Storage Calculator, but I still have a great deal of difficulty in wasting the space that I would seem to loose with over half my disk space being unutilized or lost in a RAID 10 configuration. We have 2300 Mailboxes with many users on Blackberries. We want each user to have 2 GB mailboxes in Outlook cached mode. We consider these users heavy in their mail usage. We have measured our actual user IOPS (Input / Output operations Per Second) and found each user to be using roughly 3.34 IOPS. I am guessing that our heavy IOPS usage is based on our heavy Blackberry use since I see that a regular heavy user in cached mode uses around 0.48 IOPS. I understand that a 15K RPM FC disk has roughly 180 random IOPS at the controller. All disks will be on a SAN. So here are my concerns.

1. How do I justify losing so much space with a RAID 10 configuration as opposed to a RAID 5 configuration?
2. How do I use fewer disks in a RAID 10 configuration than a RAID 5 configuration?
3. When I consider RAID 10, how does it provide more IOPS than a RAID 5 configuration when each disk is going to give me 180 random IOPS
4. Can you justify to me why I should use a RAID 5 configuration over a RAID 10 configuration?

Regards,

An Exchange 2007 Administrator

Answer:

I understand your concerns. First it is good that you are using the Exchange 2007 Mailbox Server Role Storage Requirements Calculator as it will take many of the detailed calculations out of your hands. See http://msexchangeteam.com/archive/2007/01/15/432207.aspx for the latest version. I would advise you to use a RAID 10 configuration over a RAID 5. If you use a RAID 5 disk configuration you will require significantly more disks than a RAID 10 configuration for your 2300 users to meet your IOPS requirements. Remember the most import thing to consider first in Exchange 2007 is the performance of your disk subsystem. IOPS should be considered before space requirements. Obviously space is important, but often you will meet your space requirements with fewer disks than are required to meet your performance requirements. With that all said for the sake of this discussion we are just going to worry about the storage of the databases. Use the Exchange 2007 Mailbox Server Role Storage Requirements Calculator to get all the details surrounding the storage requirements for transaction logs and the recovery storage group. Our goal here is to provide a comparison of how a RAID 10 configuration can provide you better performance with fewer disks than a RAID 5 configuration and how it may in fact waste less space once your performance requirements are met.

RAID 10 IOPS in a 4 (2x2) disk RAID set vs. a Raid 5 in a 4 (3+1) disk RAID set

Assumptions on your Exchange 2007 CCR environment:

1. The following assumes 1 CCR cluster for 2300 Mailboxes for simplicity of calculations
2. 3.34 IOPS are used for each of your user mailboxes based on your peak IOPS per mailbox as measured by you against your own system. It is assumed you are using so many IOPS per user because of your Blackberry use.
3. 300 GB 15K RPM disks are assumed in either a (2x2) RAID 10 configuration totaling 4 disks or a (3+1) RAID 5 configuration totaling 4 disks. All 15K RPM disks are assumed to reach 180 random IOPS. This is how much I/O the physical disk can sustain at the controller using a random 8K read and writes with 80% capacity utilization. Assuming cache avoidance, a 10K RPM spindle supports approximately 130 random or 150 sequential IOPS; a 15K RPM spindle supports approximately 180 random or 200 sequential IOPS.
4. The calculations here do not take into account as of yet the required IOPS for transaction logs on separate LUNS. However these would also be recommended on a RAID 10 configuration.
5. The calculations provided will show that your Exchange 2007 environment will need fewer disks to meet the Exchange 2007 database IOPS requirements with a RAID 10 configuration than a RAID 5 configuration.

Pre-Summary: So what are we going to see in the following calculations? We will show that for a single CCR cluster with LUNS configured in a RAID 10 (2x2) 4 disk configuration we would need 200 300 GB 15K RPM disks, 100 for the Active and 100 for the Passive nodes, whereas in a RAID 5 (3+1) 4 disk configuration using the same disk we would need 432 300 GB 15K RPM disks, 216 for the Active and 216 for the Passive node to meet the same IOPS requirements provided for 2300 users at 3.34 IOPS per user.

The following RAID 10 calculation will show that in a single CCR cluster configuration 100 15K RPM disks would be needed on the Active Node and 100 15K RPM disks would be needed on the passive node (200 total disks) to provide the necessary IOPS for a RAID 10 disk configuration for 2300 mailboxes at 3.34 IOPS each.

Calculations using RAID10:

• When reading data from a RAID 10 disk configuration the read is done only once, however when writing, because of the mirroring, the write must happen twice. RAID10 therefore has a 2:1 Write/Read penalty.
• 3.34 IOPS per user * 2300 mailboxes = 7682 IOPS needed for the Database IOPS
• Database IOPS requirement therefore = 7682
• Exchange 2007 has a 1:1 Read/Write Ratio as provided by Microsoft. http://msexchangeteam.com/archive/2007/01/15/432199.aspx
• Based on the Read/Ratio of Exchange 2007 3841 IOPS are Reads and 3841 IOPS are Writes (3841 + 3841 = 7682).

How to calculate the Raid 10 penalty calculations

Now let's calculate the Raid 10 penalty using the 2:1 Write/Read penalty. When we put in the figures above, we have the following IOPS requirements at the SAN level using the penalty:

• Writes (3841* 2) + Reads (3841*1) = 11523 SAN IOPS. These IOPS are needed in a RAID 10 configuration at the SAN level based on 2:1 Write/Read penalty for the database LUNS

How to calculate the Raid 10 rebuild overhead

Now if we calculate for the rebuild overhead the following IOPS are needed just for the database LUNS

The Raid 10 rebuild overhead as recommended by Microsoft is .35 or 35%. Reference the Exchange 2007 v15.6 Storage Calculator on the "Storage Design" tab under the "RAID Rebuild Overhead" section

The formula for Raid 10 rebuild overhead:

SAN IOPS / (1 – 35% RAID 10 rebuild overhead) = Required RAID 10 SAN IOPS with RAID Rebuild Overhead.

Using the formula we get the following:

11523 SAN IOPS / (1 - 35% RAID 10 rebuild overhead) = 17728 Required RAID 10 SAN IOPS with RAID Rebuild Overhead

Now let's determine the number of disks required for RAID 10 SAN configuration:

The number of RAID 10 DISKS formula is:

RAID 10 # of Disks PER NODE = (Required RAID 10 SAN IOPS with RAID rebuild Overhead / (180 Random Disk Spindle IOPS * 100% Maximum Disk IOPS Capacity)

Using the formula we get the following:

RAID 10 # of Disks = (17728 Required Raid 10 SAN IOPS / (180 Random Disk Spindle IOPS * 100% Maximum Disk IOPS Capacity) = 98.48 disks which we round up to 100 disks based on needing to be evenly divisible by 4 (2x2 RAID 10 configuration). Since a CCR configuration has the storage multiplied by 2, 200 300 GB 15K RPM disks are needed for the databases in this single CCR cluster scenario with 100 15K RPM 300 GB disks assigned to the Active and 100 15K RPM 300 GB disks assigned to the Passive.

Ok so what about the statement I made earlier that using the same 300 GB 15K RPM disks in a RAID 5 configuration would requires 432 total disks with 216 disks for the Active node and 216 of the Passive node? This means we need 232 more disks in a RAID 5 configuration to support the IOPS on the databases than in a RAID 5 configuration. Let's now look at the calculations to see why.

The following RAID 5 calculation will show that in a single CCR cluster configuration 216 15K RPM disks would be needed on the Active Node and 216 15K RPM disks would be needed on the passive node (432 total disks) to provide the necessary IOPS for a RAID 5 disk configuration for 2300 mailboxes at 3.34 IOPS each.

Calculations using RAID 5:

• When reading data from the RAID 5 disks this is done only once, however when writing, because of the calculation of parity the following happens;
  • First one disk block read occurs, then a parity read occurs, next a disk block is written and then a parity write occurs. This means that RAID 5 has a 4:1 Write/Read penalty.
• 3.34 IOPS per user * 2300 mailbox = 7682 IOPS are needed for the Database IOPS
• Database IOPS requirement therefore = 7682
• Exchange 2007 has a 1:1 Read/Write Ratio as provided by Microsoft. http://msexchangeteam.com/archive/2007/01/15/432199.aspx
• Based on the Read/Ratio of Exchange 2007 3841 IOPS are Reads and 3841 IOPS are Writes (3841 + 3841 = 7682).

How to calculate the Raid 5 penalty calculations

Now let's calculate the Raid 5 Penalty using the 4:1 Write/Read penalty. When we put in the figures above, we have the following IOPS requirements at the SAN level using the penalty:

• Writes (3841 * 4) + Reads (3841 * 1) = 19205 SAN IOPS. These IOPS are needed in a RAID 5 configuration at the SAN level based on 4:1 Write/Read penalty for the database LUNS

How to calculate the Raid 5 rebuild overhead

The Raid 5 rebuild overhead as recommended by Microsoft is 1.00 or 100 %. Reference the Exchange 2007 v15.6 Storage Calculator on the "Storage Design" tab under the "RAID Rebuild Overhead" section.

The formula for RAID 5 rebuild overhead:

SAN IOPS + (SAN IOPS * 100% RAID 5 rebuild overhead) = Required RAID 5 SAN IOPS with RAID Rebuild Overhead.

Using the formula we get the following:

19205 SAN IOPS + (19205 * 100% RAID 5 rebuild penalty) = 38410 Required RAID 5 SAN IOPS with RAID Rebuild Overhead.

Now let's determine the number of disk required for RAID 5 SAN configuration:

The number of RAID 5 disks formula is:

RAID 5 # of Disks = (Required RAID 5 SAN IOPS with RAID Rebuild Overhead / (180 Random Disk Spindle IOPS * 100% Maximum Disk IOPS Capacity)

Using the formula we get the following:

Raid 5 # of Disks = (38410 required RAID 5 IOPS / (180 Random Disk Spindle IOPS * 100% Maximum Disk IOPS Capacity) = 213.39 which we round up to 216 disks based on needing to be evenly divisible by 4 (3+1 RAID 5 configuration). Thus 432 15K RPM disks are needed for the databases in this single CCR cluster scenario with 216 15K RPM 300 GB disks assigned to the Active and 216 15K RPM 300 GB disks assigned to the Passive.

Post-Summary: We have thus shown that for a single CCR cluster with 2300 users each using 3.34 IOPS with LUNS configured in a RAID 10 (2x2) 4 disk configuration one would need 200 300 GB 15K RPM disks. This is 232 less disks than one would need with the same disk type in a RAID 5 (3+1) configuration.

So what is the end story "Exchange Administrator" for your RAID configuration to get your necessary IOPS? In order to use the least number of disks you should use RAID 10 as opposed to RAID 5 to meet your performance or IOPS requirements. As a side note how much disk space is actually wasted in your RAID 10 configuration vs. the RAID 5 configuration once the IOPS necessary for performance is met. It is more complicated than this but let's assume a 2 GB mailbox is really only equal to 2 GB of data on the Exchange stores. Based on the disk required to provide the necessary IOPS for performance in a RAID 10 vs. RAID 5 the following applies.

1. Each 300 GB Disk has an approximate formatted disk capacity of 275 GB
2. 100 300 GB disks in a RAID 10 (2x2) configuration would thus provide
   1. ( [100 * 2/4] * 275 GB) = 13750 GB or 13.4 TB
   2. In other words half of the 100 drives in a (2x2) are available in the RAID 10 or [100*2/4] = 50 drives. Since each drive has a formatted disk capacity of 275 GB, we get 50 *275 GB which is roughly 13.4 TB.
3. 216 300 GB disks in a RAID 5 (3+1) configuration would thus provide
   1. ( [216 * ¾ ] * 275 GB) = 44550 GB or 43.5 TB
   2. In other words 3 out of 4 drives are available in each (3+1) RAID 5 disk set or (216 * ¾) = 162. Since each drive has a formatted disk capacity of 275 GB, we get 162 *275 GB which is roughly 43.5 TB.
4. 2 GB Mailboxes * 2300 Users = 4600 GB or 4.5 TB for mailboxes not counting deleted item recovery, and disk space overhead
5. In a RAID 10 configuration 13.4 TB is provided – 4.5 TB needed for mailboxes = 8.9 TB of wasted space per node
6. In a RAID 5 configuration 43.5 TB is provided – 4.5 TB needed for mailboxes = 39 TB of wasted space per node

So in a RAID 5 configuration once performance is met we wasted 39 TB per node, while in a RAID 10 we wasted 8.9 TB per node. Thus for both performance and the optimization of space a RAID 10 configuration meets your needs the best and would result in less cost overall.

Other Useful References:

Exchange Server 2007 High Availability Storage Considerations:
http://msexchangeteam.com/archive/2006/10/05/429103.aspx

Configuring, validating and monitoring your Exchange 2007 storage:
http://msexchangeteam.com/archive/2007/01/15/432199.aspx

Standard RAID levels:
http://en.wikipedia.org/wiki/Raid_5#RAID_5

RAID 1+0:
http://en.wikipedia.org/wiki/RAID_10#RAID_1.2B0

Storage Technology:
http://technet.microsoft.com/en-us/library/bb738146(EXCHG.80).aspx

Forrest McDuffie
Senior Consultant
Project Leadership Associates