help decide HP EVA4400 vs. Falconstor or Lefthand virtual appliance (virtual SAN) with TIER 0 from Intel SSDs?

[radimf]

Hi to all.

I am solving storage / architecture dilema. (It is a partial crosspost to my post in unofficial SAN performace thread.)

I am looking for solid and fast solution for our 14 VMs consolidation on 2 ESX maschines in HA config (probably HP 385p) Both equipped with 48-64GB RAM.

Only 1-2 VMs will be demanding - our production SQL 2005

64bit servers (ERP and controlling).

At least one is planned as 4 vCPUs (overkill in vmware?), 32GB RAM, 120 GB DB for ERP. Second will be max. dual cpu 8-16GB RAM for controlling

DB - very IO intensive.

Question is - should we go with traditional FC SAN (HP EVA 4400, 16 FC

15krpm disks in 2 trays - my maximum budget) , or with Falconstor or LEFTHAND virtual appliances on 2 ESX servers and 10gbit crossover for data HA.

I plan not to use dedicated maschines for iSCSI SAN purpose. I plan to deploy between our 2 ESX hosts all of our virtual machines.

One ESX - Production SQL, 2nd ERP APP server (only cpu intensive) + Active virtual SAN appliance.

Second ESX - 1st ERP APP server + rest of VMs (including 1vCPU 3GB ram Exchange 2003 server with 30GB DB) + secondary Pasive virtual SAN appliance

Between ESX servers 10gbit crossover dedicated only for data replication for HA purposes.

I plan to move one ESX to nearby DR site.

One more possibility is to use those 2 ESXs with virtual storage

appliances as "iSCSI SAN" for external physical main production SQL server

physical 2xQC 32-64GB RAM.

Only difference I see is 4 vs. 8 CPUs and no IO overhead if we use FC SAN of HP EVA 4400 x iSCSI overhead nad loosing 10gbit connection to virtual SAN.

How about 2-4 bonded 1gbit links for this scenario?

PERFORMANCE : for me is very important performance of our main SQL 2005 - I see only limiting factors amount of RAM and disk subsystem speed (we are not cpu bound that much)

Yes I know SQL performance , and vmware - why not to keep it physical?

Answer : I want to avoid clustering and still have some degree of HA.

How about of using RAIDed intel X25-E SSDs as tier 0 storage for Falconstor, with rest of our storage needs fitted with 300GB Raided SAS disks?

*Can I assume, that 4x raid5 (yes raid 5 works superb with SSDs

even for writes) Intel x25-E will smoke my max. EVA possible config

(for only storage critical 4cpu SQL VM)?*

I was investigating into FUSION IO PCI-E cards (I like this device, thus it still has some serious downsides), but their vmware support is allegedly planned, but won´t arrive for at least another 6 months.

planned maschines:

HP DL 385p can take up to 128GB RAM and up to 16 2,5 SAS disks.

Question if it could support intel 2,5inch SATAs - in case not - Supermicro makes very nice maschines. With RAID adapter of your choice.

Other vendors (SUN,DELL) are also an option.

ANY comments welcommed.

Here are my first RESULTS for 1 INTEL X25-E 32GB SSD

MAX.

THROUGHPUT is not interesting it is just one SATA disk - random numbers

and access time is

imagine 4 SSDs id RAID5 ....

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TABLE oF RESULTS

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

SERVER TYPE: Win XP 64bit VM on ESXi 3.5

CPU TYPE : intel QC 6600

/ NUMBER: VCPU / 2 (6 other low IO, low CPU VMs running on this ESXi 3.5)

HOST TYPE: Asus WS PRO x38 ICH9-R, 8GB RAM; 1x intel 6600, 2,40 GHz, QC

STORAGETYPE/ DISK NUMBER / RAID LEVEL: none/ 1x Intel X25-E SSD 32GB NO PARTITION

VMFS 32 GB LUN , 1 MB

##################################################################################

TEST NAME--

---

Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----

##################################################################################

Max Throughput-100%Read........___11.32___..........__5183.93__.........___161.99__ __CPU 20.25%!!!

RealLife-60%Rand-65%Read......__16.02__..........___3646.58__.........__28.48_____CPU 19.51%

Max Throughput-50%Read..........___30.67___.........._1939.25__.........____60.60____CPU 12.47%

Random-8k-70%Read.................____14.11______..........__4167.31_........._32.55____CPU 17.97%

EXCEPTIONS: CPU Util.-20,19,12,18%!!!

Message was edited by: radimf

INTEL SSD numbers update physical drive - even higher scores

[radimf]

Hi, anybody any ideas?

[williambishop]

I do wish to correct a mistake you were operating under. "I know sql and vmware, why not stick to a physical?"

I can tell you with confidence, and you can go look up performance numbers for huge exchange servers. In my company we have huge IO boxes of both oracle and sql that run equally well on virtual compared to physical, and I never understood that old statement, which I have heard thousands of times.....and it is woefully wrong. So I just wanted to correct you on it.

That said, I've never been a fan of HP storage except their high end arrays, which they don't build...Hitachi does. Their low and mid tier devices have a reputation in the storage world...Shouldn't take you long to find it.

[radimf]

Hi, thank you for input.

Correct me if I am wrong - so you think, that SQL with 32GB RAM 4vCPU should not be a problem in ESX?

I searched internet for HP arrays info - found numerous negative posts on HP MSA , but no problems mentioned with EVA4400.

In my region it is problem to find quallified support for more exotic vendors - SUN, virtualization products etc. (Falconstor, Lefthand)

Service support will be under HP (any other brand) contract, but to find people who have knowhow in certain products is more difficult.

As I started my research into FC SANs in my budget limit - I encountered HP EVA 4400 very early - through our local vendor.

Only thing which bothers me is potential support of SSDs - register.co.uk published, that EVA 6400 and 8400 will support 72GB SSDs - no mention about little 4400.

What do you think about iSCSI vs. FC for SQL? I encountered problem with iSCSI - how to scale it easilly beyond 1x 1gbit link in virtualized SAN scenarios.

The only solution I know of within ESX enviroment is MPIO with Microsoft initiator. Plus I do not know if there is reasonable (cpu overhead, latencies etc.) option to go beyond 2x1gbit.

More bandwith better for our SQL - we have ERP queries, which have to read 3-4 GB of data - users do not want to waste time waiting.

All of our other virtualized servers (including MS Exchange) are non storage issue for me.

thanks

[williambishop]

Good lord no, I'd even start at 2 vcpu's....you'll find that it's better to start low and work your way up, that way you don't waste resources. I've got a sql database that's a terabyte, went from a 4 way IBM server with 8 cores to a 2vcpu virtual machine and actually performs at the same or better spec. This is a high critical, high IO box. To put it in a frame of reference, my CE told me that I hit my symmetrix harder than NASA does, so that should give you some indication of the environment. It will work fine as long as you configure properly. Test, test, then test again. Start low, build up. Trust me, you put HP storage in the mix, it's not going to be esx that slows you down. What region do you live in?

To storage, my preference is always to avoid iscsi...and I know it will start a flame war, but I prefer FC. It's solid, and the payload per packet is much better. It is simply a better method. Switches have come down to about the same price as good ethernet switches.

But, and that's a big but...iscsi will work in most instances. If absolute performance is your goal, infiniband or fc. If your needs are less, other options will work. Your prediliction for ssd's says that you believe you will need ultimate performance, so skimping on the san at this point is probably unwise.

[radimf]

Thanks for response.

Actually we have very limited amount of data we need to move fast - max.

about 100GB.

It is interesting, that 2cpus can handle very heavy SQL loads - I will

try to use 2 cpu maschine at the start.

4 vCPU for sure if we go iSCSI route - for iSCSI overhead )))

Our SQL is not that much write intensive during normal hours, so I

though, that we might get away with some virtual SAN product - idealy

with support to tear storge and move data around...

Unfortunatelly there is no way how to use FC as target for Falconstor

VA, so IF I go with Falconstor (or Lefthand) in form of virtual

appliance I will be stuck with iSCSI.

There is one thing I would really apretiate on any SAN setup - NFS

support.

I use it on my test rig with opensolaris ZFS based Nexenta appliance for

non-critical VMs.

ESX vmdks are very easy to manage, backup etc. that way.

FC target support is alegedly imminent for this appliance, so may be I

could postpone SAN purchase and go this route.

I will try to test SSDs as L2ARC cache - how they impact our SQL

performance - so relativelly few SAS drives with SSD cache could do the

trick performance wise.

Sun demos are impresive.

[java_cat33]

I've implemented ESX clusters on EVA 4400's with no problems at all. All disks were FC, excellent performance and easy to configure. Definitely no hesitation in recommending the 4400.

[williambishop]

EVA's are acceptable for low usage systems--and considering it's only 100g that needs fast performance? Probably fine for his application. IO's certainly aren't in the top categories by a mile, I'd burn it into oblivion in less than an hour, but my needs are for much higher IO and much higher volumes of data.

[radimf]

Hi, thanks to all for input.

If we go with EVA - we have to use RAID10 - RAID5 is out of the

question.

How well handles EVA logs and data files on a same spindles?

As I understand it - EVA "virtualizes" storage and stripes all data

acrross all available spindles, so you can not separate logs from data?

Thanks

[DSeaman]

I'm a long time HP customer, but their EVA line is getting long in the tooth. It hasn't been refreshed in quite a while, and is lacking new key features (for some) such as thin provisioning. No idea what your budget is, but companies like 3PAR, Compellent, and NetApp all offer mid-range arrays with a better feature set. They may be out of your budget, though. You might talk to your HP rep and see if/when HP is going to refresh the EVAs if you don't have to buy immediately. Compellent does some really neat stuff with their various performance tiers, and has won many awards lately. 3PAR has what looks like a great controller design and even beat the HDS USP in SPEC benchmarks. NetApp is widely used, and seems to stay on the leading edge of features.

[frankdenneman]

The eva has 3 levels of redundancy, vraid0, vraid1 and vraid 5.

EVA's spans their vdisks over the diskgroup it is created in.

A vdisk is HP jargon for what most people call a LUN.

The best thing to do is to place at much disks in a diskgroup as possible.

Actually above 110 disk you won't see any added performance, but a 4400 cannot handle that much disks I think.

A little background how an EVA is storing its data in a vdisk;

In a vdisk is created in a diskgroup. All disks in the diskgroup will participate in a vdisk.

This means you vdisk will take advantage of the combined spindle spead.

A vdisk is divided into rstores, which manage a 8mb piece of the vdisk and will be distributed across the disks of the diskgroup.

An rstore is an 8MB unit, so when you create a small vdisk of 10GB the vdisk is divided into 1249 rstores

An Rstore is a 8mb chunk, which itself is chopped up into 4 psegs of 2mb each.

If you choose vraid 5 another pseg is created and this is the parity pseg.

So a rstore of a vraid 5 disk contains 4 pseg's of userdata and 1 pseg of parity. (10mb)

These 5 psegs are spread out over a RSS set, a RSS set is a collection of max 8 disks.

What I'm interested in, is the performance of your SQL machine, did you monitor it?

What are the paterns exactly and how much I/O does it really read and write.

Be aware that your vm cannot communicate will the storage array continuously, the vmkernel will distribute I/O time to other VM's.

Even if you do not have any other VM's running.

This is why 99% of the time, a storage array will receive random I/O from an ESX host.

[radimf]

Hallo,

Thanks a lot for indepth perspective on EVA line disk config.

You mentioned:

"This is why 99% of the time, a storage array will receive random I/O

from an ESX host. "

Do I understand it correctly - this ESX IO behavior could negativelly

impact performance? (sequential operations tend to use random like

pattern?)

Thanks

[frankdenneman]

A sequential pattern originating from a VM stays a sequential pattern, but the ESX server controls the IO to the storage array.

It is impossible to get both high performance and "fairness" when multiple machines wants to send IO to a LUN.

You always need to compromise, this is normal behaviour, nothing wrong with ESX.

If two VMs are sending IO, this can be sequential, and random, or even both are sending sequential, you always end up with random IO behaviour.

This is because the IO is comming from one HBA, send to the array.

The storage array is not aware it is for a VM, it only receives all that data from one Host.

There is some feature in the VMkernel which tries to estimate the proximity of the next IO of the VM.

If two VM's want to send IO, and VM X is already sending, the VMkernel tries to estimate the "proximity" of the next IO of VM X.

If its within this limit, the next IO slot will be given to the VM X. Because the IO will be handled by the storage array quicker, than giving the other VM a change to read and write.

If the next IO of VM X is not within this limit, the IO slot is given to VM Y to assure fairness.

I'm sure someone will advise you to adjust the QueueDepthLength (QDL)from the default (32) to a minimum of 64, but please

be aware that if you place your SQL vm on the same VMFS datastore with other vm's, it will share the QDL with the other vm's.

This means that ESX can queue up to 32 IO's in the queue for that LUN.

If you do change the QDL, adjust the Disk.SchedNumReqOutstanding as well.

I suggest that you do not adjust the QDL right from the bat, please monitor the QDL before adjusting and altering the default config.

I haven't seen that many applications that can fill a queue constantly.

(if this answer was helpfull, please award the points)

[radimf]

WOW thanks for info.

If we go other route than HP EVA line (NETapp, SUN ZFS etc.), I was

considering NFS for all, but SQL server VMs.

This way I could avoid VMFS qeue sharing betwean VMs, and get very

convenient method of vmdk file administration, and backup.

[StevoIBM]

Hi Rad,

I work for IBM and I was just reading your post here and I would suggest looking into our offerings for this project. Our storage solutions have a lot

of options and features that would give you what you are looking for. You'll have the HA, redundancy, high performance, and FC support these features tend to be

available across the board but we try to keep ahead of the game with features and options for the end user. I would definitely like to talk to you about this project and

at least see what we can advice or assist with on this project.

If you'd like to discuss this in further detail you can contact me at sversa@ca.ibm.com, I look forward to hearing from you.

Cheers,

Steven

[BUGCHK]

> that EVA 6400 and 8400 will support 72GB SSDs - no mention about little 4400.

According to the (now) official specifications all 3 arrays support SSDs (4400 will have to upgrade to latest controller firmware XCS09501000)

http://h18000.www1.hp.com/products/quickspecs/12893_div/12893_div.html

[cfranke]

-

[radimf]

Hallo,

thanks a lot for your input. It clarified for me some technological aspects of EVA storage. We ended up buying HP EVA 4400 to serve as our main and reliable (redundant paths, IO modules, cooling, FC switches etc.) storage for vsphere4 cluster. We have only one shelf 12x 15k discs at the moment. I plan to buy another full shelf ASAP.

I tested EVA vraid1 - it has read cache ON, it has writeback cache on. I am unable to get any sort of cache effect - even on 20mb dataset. I am using IOmeter 8k pages 70/30 % read/write FULL random. EVA delivers 25MB/s throughput (basically maximum what those 12 HDDs can deliver)

Why there is no caching involved? Is it by design, or may be some config glicht?

Btw for high IO controlling DB we are using Nexentastor based Supermicro appliance filled with intel SSDs - basically SUN 7xxx line on a very tight budget.

It ROCKS and delivers extreme performance compared to EVA4400 which is using "only" 12 15k FC disks. On a SINGLE 1gbit ethernet we are getting 115MB/s in that full random test.

Waiting for 10Gbit CX-4 cables to connect it via 10gbit - that should speed things up a little bit.

Thanks for any input

[AnalogKid99]

Hi

What a great read - even two years on.

I face the same situation:

FalconStor NSS-VA or dedicated hardware SAN.

I see in the end you went HP EVA.

What was it that made you head in the direction of a traditional SAN appliance versus the 'Virtual' SAN Appliance.