Manual Automation

Storage Transport Performance

Virtual_JTW — Wed, 12 Nov 2008 19:28:21 GMT

To give you a little background, I now have 6 ESX hosts with 58 VMs. Each host has dual-iSCSI HBAs with 1GbE connections. All Exchange 2007 roles have been virtualized, however we currently only have 1 out of 5 mailbox servers running as a virtual machine. We have a number of other workload types virtualized including file, print, SQL, web servers, etc.

Management has decided to stop virtualizing Exchange servers. Why? Fear generated by the FUD that surrounds the performance characteristics of various storage transports - in this case iSCSI via GbE. The only way to fight FUD is with facts. Towards this effort I have performed some calculations in an attempt to answer 2 questions:

1. How well is our storage transport performing given current virtualized workloads?

2. How much "performance capacity" do we have remaining?

I added up the average bandwidth utilization of all 6 of my ESX hosts which totaled 11008KBps. This converts to 0.09Gbps out of 2Gbps or 4.5% bandwidth utilization. I then added up the maximum utilization of all 6 ESX hosts. This would be the high-point of the peaks or bursts in utilization. The result was 0.48Gbps.

Assuming we can get 800Mbs of actual bandwidth per connection we have 1.6Gbps useable bandwidth remaining. Note that based on VMware's testing we should be able to reach near wire-speed (2Gbps) if the environment is configured correctly making 1.6Gbps a conservative assumption.

So even if I use the maximum bandwidth measurement of 0.48Gbps, that leaves 1.1Gbs useable. Another way to state it is that my environment is reaching a max of 30% bandwidth utilization.

The results seemed unbelievable to me at first so I digged a little deeper:

I found this in a EqualLogic presentation from 2005: "With 2 iSCSI connections and free NIC teaming, payload equals approx. 234 MB/s (1.96Gb/s) or 823GB/Hour. We found 2Gb FC delivers 196 MB/s which equals approx. 689GB/Hour payload." http://communities.vmware.com/servlet/JiveServlet/downloadBody/1806-102-1-1554/VMUG.ppt
I found this in an iSCSI Virtualization whitepaper from 2007: "For high-performance, mission-critical servers, the cost of Fibre Channel is often justified, because Fibre Channel provides higher bandwidth (4 Gbps vs. 1 Gbps) and lower latency than IP networks. However, many environments are over-served by 4Gbps Fibre Channel links. This is particularly true for hosts running applications characterized by random traffic, such as database applications and Exchange."
http://www.dell.com/downloads/global/products/pvaul/en/iscsi_virtualization.pdf
And here's one from Netapp: "...based on deployments, Netapp has proven over the past 3 years that a scalable, simple to use array with enterprise class reliability can safely be the iSCSI platform for mission-critical applications. Exchange is a perfect
example of a mission critical application that is routinely deployed over iSCSI these days."
http://storagefoo.blogspot.com/2006/05/iscsi-performance-and-deployment.html
Finally, VMware's own testing of storage protocols and their corresponding physical medium from this year: "This paper demonstrates that the four network storage connection options available to ESX Server are all capable of reaching a level of performance limited only by the media and storage devices."
http://www.vmware.com/files/pdf/storage_protocol_perf.pdf

It's important to note that I'm leaving 2 things out of this consideration:

1. I typically read how FC has lower latency than IP. My somewhat empirical belief is that IP's additional latency will not be a big factor when added to the equation.
2. I've read different sources that state disk IOPS are more important with regards to system performance than storage transport bandwidth utilization.

I'm still looking for a way to quantify these factors to better predict the performance characteristics of our IP storage implementation. This is the first part of what I'm sure will be an on-going investigation. It sure would be nice to have a tool that did all of this for me! I have yet to find something that's comprehensive enough on any given storage platform I've managed (IBM DS, EMC Celerra, et al).

Also note that I've been monitoring my bandwidth utilization more closely using Vkernel's Capacity Analyzer and can safely say that 11008KBps is high. It's dropped 30-40% over the last two months for various reasons.

Next month I hope to enable jumbo frames in this environment and expect to see some additional performance gain at some level. I'm considering capturing before/after snapshots of various performance metrics and posting the results in a future blog.

In conclusion, this analysis makes me even more confident about the performance of our ESX hosts and virtual infrastructure backend storage transport even if/when I get to virtualize the remaining Exchange mailbox servers.

Site Recovery Manager is a Hit!

Virtual_JTW — Wed, 08 Oct 2008 04:25:00 GMT

A Little History

In a previous article (http://communities.vmware.com/blogs/ManualAutomation/2008/05/15/the-big-plan-business-continuity) I discussed why I was looking closely at SRM and what I needed to get done before I could implement the product. Now that I've successfully tested the product I'd like to give an update.

The Celerra Code Upgrade

The code of both of my Celerras was upgraded to 5.6 in mid July. It wasn't pretty - no fun being in the data center until 3:00am. To EMC's credit, their CE hung in there with me, got the problems escalated and ultimately we got the VMware data stores working again. We were bit by the LUN resignaturing "bug". EMC knows the code upgrade causes this but for some reason we were surprised and found out the hard way at about 12:00am.

It took another month to recover other services such as CIFS and iSCSI replication. When I was young, my father insisted was that when I handled someone else's property, I should always return it in the same or better condition than when I first received it. My main problem with EMC in this respect is that they left me with a system that didn't work like it did before they upgraded it. I'm past the CIFS and iSCSI replication problems now, but I'm still experiencing problems with CAVA that didn't exist before. Luckily, I don't think it's anything too difficult or serious and I will be calling EMC support to get this last problem resolved.

While I've given feedback on this event to EMC support, note that I still am a fan of their unified storage product. It's not right for all companies or all situations but it is for my environment. Also, to be fair, many Celerra customers may never need to experience a code upgrade event. The only reason to do this is if you need some feature or improved capability that the upgrade provides. I've had an EMC CE tell me that they have retired EMC hardware that had the original code installed making it over three years old! This says volumes about the code's stability and reliability. In my case, I needed the expanded functionality of iSCSI LUN replication and compatibility with VMware Site Recovery Manager.

The Evaluation

Anyone semi-familiar with installing VMware products will have no problem getting SRM installed. Note that you'll need to obtain the Storage Replication Adapter (SRA) from your storage OEM and install it in the proper sequence per the documentation. In my case I used documentation from EMC and VMware to install and configure the product. See the "Additional Resources" section at the end of this article.

One of things that's awesome about VMware is the amount of attention they've given me regardless of whether I was working for a large $3 billion enterprise or a mid-sized $500 million dollar company. In this case, my sales rep offered to have a local VMware systems engineer (we'll call him "Dave") come out on-site and work with me to complete a proof-of-concept.

I had SRM and the SRA components installed. I wanted a technical resource in case I needed it while performing that first test. Well, I needed it and got it. Keep in mind I hadn't purchased the product yet(!). Dave was able to help me work through a couple of issues we ran into during that first session such as file system sizing and licensing issues. It only took 2-3 hours but when finished, I had 4 VMs running in my remote data center 325 miles away! (Thanks to Dave and Ken!)

Another tip I learned during this session: review the SRA log. In the case of the Celerra's SRA, it documents every command it executes and the results. It's a great way to learn what SRM is really doing behind the scenes with your storage in order to get the LUN(s) setup and ready to be used as a data store by ESX.

Subsequent Test Results
I have more testing to do but can report that I'm starting 4 VMs from a single replicated LUN in 8 minutes. And I'm not talking about from the time of just powering on, I'm talking about pressing the "big red (test) button" - powering-up the VMs - starting the Windows services - and the recovery plan completion. Try that using physical servers! Sorry, but even restoring servers from a B2D solution that's replicated to your DR site won't be as fast.

I demonstrated SRM for the DR team and initially got a "that's all?" kind of reaction. I quickly realized that SRM, with the combination of array-based replication, +worked too well+! Meaning, it did such a good job of hiding the complexity and number of steps required to get from A to Z that my non-technical DR teammates didn't understand what SRM was really bringing to the table. If there's only one thing you take away from this article, make sure it's that you're better off explaining in simple terms the steps SRM is executing in the background before running a demonstration.

Talking about the virtues of SRM is one thing (the recovery run book, the steps it automates, the testing capabilities (which are awesome by-the-way), etc.), demonstrating these product features for your DR team is another. If your experience is like mine, you'll find it dramatically influences the discussions on the project plan. In my case, we will be significantly changing the testing phases - actually streamlining those thanks to SRM.

I wouldn't declare SRM to be a perfect specimen of engineering excellence; I reserve that title for Windows ME (yes that's a joke). But there are a couple of things that could be improved. I would like finer-grained control over when my VMs are powered on - I'd like to be able to specify dependencies between VMs. It seems like VMware is bent on specifying everything as "High", "Medium" and "Low". What if I want six groupings instead of just three? There are also a number of folks complaining about the lack of fail-back. Yes, there's no "big red button" to press to perform a fail-back but most storage OEMs including EMC are providing documentation describing how to get this done. Finally, I'd like VMware to consider non-array-based replication capabilities. I don't think you'll replicate 20 VMs this way, but it sure would be nice for those one or two one-offs for which you don't want to replicate an entire LUN. I can also image customers with smaller implementations or those with non-supported back-end storage using this feature.

Because the POC exercise was a success it was easy to convince management to purchase the product. I think purchasing Site Recovery Manager is the best endorsement I can give it and VMware. Now I can't wait to see what the next version brings!

Additional Resources

SRM Product Site: http://www.vmware.com/products/srm
SRM Product Documentation: http://www.vmware.com/support/pubs/srm_pubs.html (The Getting Started PDF is particularly useful and pay attention to the compatibility matrix.)
SRM VMTN Forum: http://communities.vmware.com/community/vmtn/mgmt/srm
SRM Book: http://www.rtfm-ed.co.uk/?p=584 (Mike's blog is also a good one to watch.)
Storage OEM Docs: The EMC documentation can be obtained by registering on their Powerlink (http://powerlink.emc.com/) site and searching for "Site Recovery Manager". For other OEMs, contact your sales representative, search their web site or call support.

The Big Plan: Business Continuity

Virtual_JTW — Fri, 16 May 2008 03:04:00 GMT

For my employer, this is the year of disaster recovery. Almost all of our major projects tie-in to the goal of performing a successful DR test by the end of the year. Besides the standard IT things that have to get done on a regular basis (asset management, corporate application TLC, etc.), this goal is really driving the work we’re doing.

Sometime before I was hired, the company purchased two EMC Celerra NS352s for NAS/IP storage and two EMC Centeras for file and email archiving. We’re an HP shop so were using DL380 G5s with the little USB key inside running ESX 3i or ESXi or whatever it’s called today. We mostly use Cisco gear for networking and have dedicated switches for iSCSI and VMotion traffic. We have two of everything – our company’s IT services are split across two datacenters. Each datacenter will be a hot recovery site for the other.

So what does our DR solution entail? Well, some fairly advanced technologies:
Virtualization: VMware VI3, release 3.5
DR Automation: VMware Site Recovery Manager
Replication: EMC Celerra Replicator V2
Snapshot consistency: EMC Replicator

Oddly enough, looking at the list, only one of the technologies is shipping and in my possession today (VMware VI3). Hmmm… can you say, “Project risk”?

I first saw VMware Site Recovery Manager at a VMworld 2007 presentation. If it works, it will be impressive. Automating the steps to configure and power on VMs and a central place to store the DR “run book” will be sweet, to say the least.

One of the advantages of having EMC storage equipment is that they own VMware (or at least the controlling interest). This means there’s a pretty good chance that their storage platforms will be among the first certified to work with VMware products. Sure enough, my Celerras will work upon release of SRM with a firmware/code upgrade. The code is shipping on new product; however, EMC has a policy that delays certification by 90 days for installing/upgrading to current product. That puts us in the June time-frame.

The Celerra code upgrade provides a new version of Celerra Replicator that replicates iSCSI LUNs. To ensure application consistency for applications such as Exchange and SQL Server, EMC Replicator must be used.

This is the high-level plan: application-consistent snapshots, SAN/IP storage-based replication and SRM to run it all at the end. Yes, we have some physical servers = HP-UX, AIX, etc. and too bad their story won’t be interesting.

So while I’m waiting for product to GA, I’m trying to get our VI3 platform up and stable. Stay tuned for progress on that front.