Need Help with understanding the true performance capabilities of my hardware

[ScottCorley]

So I recently built two whiteboxes so I could rescue some old servers that were literally on the brink of spontaneously combusting. So at the time I put together my parts list that I knew was compatible with VMWare and configured it best I could and got it running as quickly as possible because I had such an urgent need to P2V our core servers. That worked out relatively well however now im not seeing the performance that I was hoping for and I am trying to determine if it is in deed my datastore performance or the fact that my Guest Servers just have a messed up file system which is all possible.

I built my server around the ASUS Z9PE-D16 Motherboard and I equipped it with a PIKE 2108 LSI RAID card. Oddly enough even though everywhere on earth says RAID 60 requires 8 drives, this RAID card specifically describes how to build it with 6 drives which could be my problem but at this point I know i have a whole bunch of "Could be's" and its driving me nuts so i could really use some input.

So its running a RAID 60 with 6 Hitachi 2TB 7200rpm 6GBS drives. My datastore in the host is setup with a 1mb block size as a single large datastore.

I'm trying to figure out if I should change from a RAID 60 to a 10 or if I should change the host datastore to be smaller datastores with a different block size. I am not sure where I can pick up better performance being that I am using a single RAID card. I am leaning toward changing my datastore configuration lowering my store size and lowering the block size but I could really use some recommendations on this.

I originally set things up on a single datastore because i literally did not know or have time to figure out my servers drive size. I came into the company with a long gap between me and my predecessor and every server I had literally had at least one failed part, either power supply or HD or both. So now that i have breathing room and a identical vmware host to move live servers around on for changes, I really need to find ways to optimize.

Any help is much appreciated! Thank you!

[Argyle]

For writes RAID10 will be faster. You have a lot of write iops penalty with options like RAID 5 / 6 / 60.

Your raid card seem to support battery backed cache. I would look into that and configure the cache for 100% write if possible.

More datastores could potentially be better since each LUN gets its own disk queue. But not sure if you will notice with 6 drives.

[sparrowangelste]

raid 10 will speed up peformance.

also does your raid card come with battery backed cache?

if its isnt you are doing write through which can cause your problems

[ScottCorley]

Thank you for your input, it does come with the connector and capability for the BBU I just hadn't looked into getting that part. So Basically I have some hardware options and some software considerations.

I'm just getting frustrated with 9ms read latency and 12ms write latency on the datastore itself. Somehow im able to get 1112 IOPS which for my drives and config seems high by comparison to what I have been reading about online though maybe im reading something wrong.  My FileServer though is having spikes of 104ms in latency and my exchange server is spiking  around 150ms for writes and just under 100ms for reads.

Does it seem like a RAID 10 would help enough to make the difference over a RAID 60? What do I lose in the change am I down to single drive failure tolerance again?

[sparrowangelste]

Does it seem like a RAID 10 would help enough to make the difference  over a RAID 60? What do I lose in the change am I down to single drive  failure tolerance again?

with raid 60 and 6 drives you should be able to lose 2 disks and still ahve your data.

with raid 10 you might getaway with loosing 2 disks, only if they were on the same mirror. realistically 1.

read this for the iops penalty of raid 5, its more for 6

http://rickardnobel.se/raid-5-write-penalty/

[Argyle]

For writes it would improve performance when talking iops. Someone made examples around raid penalties here due to parity caluclations http://sudrsn.wordpress.com/2010/12/25/iops-raid-penalty-and-workload-characterization/

The "negative" thing with RAID10 is that you loose capacity. Basically it's striped RAID1 (mirrors). So if you RAID10 six disks you can only utilize half of the storage. The benefit is performance. Other benefits are faster raid rebuild times when you swap a disk.

Redundancy is also good since with 6 disks its three mirrors. In theory you can loose 3 disks without issues. Worst case you loose the same two disks in the same mirror though.

[markpattersones]

Are you suffering poor performance or just poor metrics?

[Josh26]

ScottCorley wrote:

Oddly enough even though everywhere on earth says RAID 60 requires 8 drives, this RAID card specifically describes how to build it with 6 drives

The problem with RAID is that, although there are "concepts", the reality of implementation is incredibly vendor-centric and heavily related to how things are marketed.

Consider the "fact" that a RAID5 provides capacity for n-1 disks worth of data. Then you get on an EVA platform and the RAID5 formula is long and not actually public.

Many of the "facts" about performance are overcome by how vendors design cache.

In short, the real answer here will be down to your product, and you may need to test a few options yourself to determine what works best.

[ScottCorley]

I am having poor performance. Statistics are just numbers that are relatively useless in production until you have an issue to troubleshoot or to anticipate. Its the poor performance that is my concern.  I would like to understand the statistics better at this point and find a way to do more specific capability testing unfortunately I don't really have a machine I can take out of production right now to test with.

Can someone enlighten me on Datastore Block Size and how to optimize it? In my haste I configured the single main datastore with a 1mb block size. to me that seems rather large but before I start migrating to my other server to tweak this one I want to understand block size optimization in a vmware environment better.

I know the general best practice is to set your block size as close to your most commonly written or accessed file size roughly to optimize space and seek time. However block size determines your overall size of a volume so then there is the factor of what sort of continuous space requirements you have. Then at your guest level your looking at a nested nested HD configuration. So in theory you could go with smaller datastore block size, then span as needed in the guest but how would that possibly effect the guest in the event of a physical HD failure? How does host and guest block size interact? I remember reading somewhere about sub-sectoring but I havent been able to perform tests on that. In theory though your host could be set at an 8mb Block Size and your host could be set at a 64k block size but would that actually make a difference? I mean, would the sub-blocking or sectoring enhance or degrade performance??

Then of course if you do adjust the block size and create seperate datastores your still accessing the same RAID card so would it even matter. Along the same lines, If you created additional arrays on the RAID card would that even help since it is still a single card?

I know a lot of this depends on RAID card manufacturer implementations, though i do know my card is based on a LSI chipset though i dont know if that makes much difference other than compatability.

I just asked a ton of questions. Im sorry. I really appreciate the feedback that I am getting from all of you and I am just trying to figure out how to maximize what i have with limited resources and I'm kinda going crazy here.

Thank you

[markpattersones]

I can't answer two of your questions:

1) Would changing your RAID improve level performance?

2) Could your block size setting be causing problems. Could this be causing mismatching problems with your RAID or your HDDs or your guest OS drives?

These could possibly be the problem but maybe not.

I would suggest that you can do other testing to see if the problem is caused by something else.

1) I'd power down half my VMs and see how it performs then. Then I'd boot them back up and shut down the other half. Then granularise it and see if you can find which ones the bad boy.

Where does the 2nd host you built come into this? I'm  assuming it can't access the local datastore on your main host but  manually moving across your suspects could be a good option.

2) Are you swapping out much on either the host level or inside the guest OS? That will cause massive IO problems any time. Depending on how your guest OSes are configured they might be natively using their own swap files which will impact your whole datstore. Take for example a Windows server. It has 4GB allocated then a swap file roughly the same. If it gets busy with 6GB (system + a database of some kind) 2GB of that will be swap file activity. If anything will bring an otherwise capable storage system to its knees its rampant swapping.

You may actually want to give some of them more RAM allocation. You will have to look inside the OS for better RAM breakdowns as the VM performance monitor will only show you how much host RAM is being used not how much guest swap is being used.

3) Does your host have any other storage configured? Ie. NFS or ISCSI somewhere? I'd remove any of these you aren't using. On that note I'd be very wary of any VMs that are providing these kind of services. Not to mention other resource hogs. Doing a blanket P2V in this way is really letting the lepers into the monastery and now you have to isolate them one by one to find which is a problem and then go through all its services etc and find out whats causing problems.

[jrmunday]

From a datastore perspective, VMFS5 has a unified 1MB block size and is much more efficient than VMFS3. Upgraded VMFS3 to VMFS5 volumes (which can be done online) retain the original block size, so if you have a VMFS3 volume that was created with a larger block size, then it would be best advised to re-provision this datastore as VMFS5 volume which uses a 1MB block size.

From the guest perspectice, make sure your disks are aligned properly (Windows 2008 does this automatically), as this makes a huge difference to performance (use fsutil and diskpart to verify this). For SQL volumes as an example, create the partition with an offset of 1024 and format NTFS with a 64k allocation unit size (to match the SQL IO).

[ScottCorley]

So you have me intrigued by this alignment concept. I hadn't thought about it in this level of detail before.

I found this site: http://www.tcpdump.com/kb/os/windows/disk-alignment/into.html

which had some good explanations and I used their little utility. It came up with some weird results but also said my drives are aligned.

One VM Server came up with this:

Microsoft (R) Windows Script Host Version 5.6

Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

Device ID: Disk #0, Partition #0

=============================================

  Number Of Blocks:  129024

  Block Size:        512 KB

  Total Volume Size: 0 GB

  Type:              Unknown

  Starting Offset:   1048576 (1024 KB)

  Disk Alignment:    Aligned

Device ID: Disk #0, Partition #1

=============================================

  Number Of Blocks:  83886080

  Block Size:        512 KB

  Total Volume Size: 40 GB

  Type:              Installable File System

  Starting Offset:   67108864 (65536 KB)

  Disk Alignment:    Aligned

Device ID: Disk #0, Partition #2

=============================================

  Number Of Blocks:  1048578048

  Block Size:        512 KB

  Total Volume Size: 500 GB

  Type:              Extended w/Extended Int 13

  Starting Offset:   43016781824 (42008576 KB)

  Disk Alignment:    Aligned

Press any key to continue . . .

The other came up with this:

Microsoft (R) Windows Script Host Version 5.6

Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

Device ID: Disk #0, Partition #0

=============================================

  Number Of Blocks:  104857600

  Block Size:        512 KB

  Total Volume Size: 50 GB

  Type:              Installable File System

  Starting Offset:   1048576 (1024 KB)

  Disk Alignment:    Aligned

Device ID: Disk #0, Partition #1

=============================================

  Number Of Blocks:  629145600

  Block Size:        512 KB

  Total Volume Size: 300 GB

  Type:              Installable File System

  Starting Offset:   53688139776 (52429824 KB)

  Disk Alignment:    Aligned

Device ID: Disk #1, Partition #0

=============================================

  Number Of Blocks:  838860800

  Block Size:        512 KB

  Total Volume Size: 400 GB

  Type:              Installable File System

  Starting Offset:   1048576 (1024 KB)

  Disk Alignment:    Aligned

Press any key to continue . . .

None of my VM's came from older VM's, they were all P2V. All of my servers are Win2K3 (not R2) one is a PDC with exchange 2003 sp2 and the other is a BDC and file server. I have shut down all non-essential VM's and verified that my linux servers are not generating the load/lag.

[ScottCorley]

Both servers are using less than a third of their RAM allotted to them. They both have 3.5 gig of RAM.

Unfortunately the Lepers were already in the monastery. I just put them in a room that was less likely to collapse.

Leprosy has a cure, I am not sure I can say the same for my exchange server. :smileysilly:

The second host is basically going to be as close to a hot backup as I can have with out having the funding to make it an actual hot backup. I'm using VEEAM backup to backup to the other host. Its not ideal but at least that restore will take less time than most of my other options.

So part of my conclusions so far is:

1. RAID 10 may provide some improvement but at potentially some loss of failure absorbsion. So at the moment RAID 60 seems worth while.

2. Its worth considering looking more intently at this alignment and possibly block size matchups.

3. I need to dig further into my page file system and ensure my guests are not trying to swap too much.

4. I hate Exchange Servers.

[Argyle]

RAID10 with a battery backed up write cache is the brute force way to go and will get you far. Still with 6 disks there is a physical limit on performance.

With 7.2k disks you might average 100 iops per disk. For read performance you will not get more than 600 iops whatever you do unless there is cache somewhere to assist. Your card has cache so you might see higher but its only for read unless it's battery backed up.

Looking at write penalites depending on raid level you end up with 300 write iops on RAID10 with 6 disks but only around 100 write iops with RAID5/6 if writes are in "write-through" mode (not cached) on the controller.

An alternative is to add even more disks to increase the spindle count.

[ScottCorley]

Thank you all for your input.

I believe my original question was in an effort to chase down a symptome and in researching a new type of deployement I believe I discovered a large part of my problem performance wise potentially.

I found this article here:

http://community.spiceworks.com/topic/228073-software-raid-in-a-guest-vm-on-top-of-a-vmware-host-har...

which made me realize that I overlooked the performance impact of doing a P2V on servers with existing hardware RAID's specifically in light of how old their raid configurations were. I am still doing research on this because while my physical servers were not running a software raid, they were running a hardware raid. I dont know how this would translate over in the p2v process. At first I thought it would be a non issue assuming that the p2v is duplicating the logical drives. But do I need to consider how a RAID card on the old server abstracted or interacted with the logical drives it created for the operating system to use?

Does anyone know anything about this question? I am sure it is somehow addressed since a large portion of P2V processes are done on servers versus workstations but this is something i would like to understand better.

I agree that a BBU would make a big difference but based on my configurations of VM Hosts before with the same hardware just with out bringing dieing servers over I have not ever had this poor of performance. If I could afford to turn off the main VM's in question then I would but that is not an option for me right now. Maybe over thanksgiving.

My Quest Continues...

[ScottCorley]

Also I would like to mark all of your remarks as helpfull but apparently it will only allow me to mark two of them as helpful.   *doh*

[jrmunday]

Hey Scott, I have no science to back this up, but I don't believe that there should be any difference in the VMDK's that original on a RAID vs Non-RAID set in the P2V process. My logic in this (and I could be wrong) is that a new disk is being created and the construct (block size, partition offset, alignment etc) of this disk will determine the characteristics of how it performs in a virtual environment.

In relation to fixing disk alignment, check out this FREE tool;

http://nickapedia.com/2011/11/03/straighten-up-with-a-new-uber-tool-presenting-uberalign/

... I'm trailing it now, and will post some results later in the week

Cheers,

Jon

[markpattersones]

The RAID array is completely invisible to anything interacting with it as one logical volume. This includes cloning and copying. So your old RAID will have no affect on the converted vms.

[ScottCorley]

@JRMunday -

     Thank you for pointing me to that tool. I will look at that a little more later when I get some more thorough backups completed. I have had to resort to setting up FreeNAS on both hosts so that I can share an iSCSI store from either to the other. This is the only way I have been able to use built in windows 2003 ntbackup.exe to backup my exchange server correctly with VSS. (Since Windows 2003 doesnt like backing up to an "actual" network drive which seems stupid but what ever.) (If your wondering, I had to do this because I couldnt get the boss to shell out the cash for an enterprise level tool to accomplish this. He says it always worked before but he couldnt understand how taking data off of an old RAID 5 server and putting the backup on a external usb hard drive from best buy was actually defeating the whole "data reliability" thing. I digress, sorry.

@Mark -

     Thank you, that makes sense and it was what i was hoping to hear but you know how it goes when you have dug through an IT problem thats like an onion sometimes you get lost in all the layers and start to question everything.

So progressively I have been simplifying my installs but My file server and my exchange/AD server are my most difficult to deal with in addition to the fact that we are a 24hr shop here so taking either down for any amount of time is tricky at best. (this is a situation where vmware essentials and veeam essentials would really help if I could afford them)

So at this point I'm looking for some ways to isolate my bottlenecks and identify the real culprit better. If anyone has some good free tools to suggest for this hunt I would really appreciate it. Thank you!

[jrmunday]

I've started another discussion for Disk alignment here (interesting comments so far);

http://communities.vmware.com/thread/425590

RE ntbackup --- I think I just threw up

Why don't you just create a share for your backup, map this on the Windows 2003 system so that you have a drive letter and then back to this?

A quick test shows this works perfectly;

Backup Status
Operation: Backup
Active backup destination: File
Media name: "Backup.bkf created 15/11/2012 at 18:42"

Volume shadow copy creation: Attempt 1.
Backup (via shadow copy) of "C: System"
Backup set #1 on media #1
Backup description: "Set created 15/11/2012 at 18:42"
Media name: "Backup.bkf created 15/11/2012 at 18:42"

Backup Type: Normal

Backup started on 15/11/2012 at 18:43.
Backup completed on 15/11/2012 at 18:43.
Directories: 4
Files: 1
Bytes: 526
Time:  1 second

----------------------