iSCSI SAN too slow, need opinions on direct storage options

[Karl_P]

We have a single VI3.0.2 server (Sun 4200) with 5 guests that is attached to an AX150i iSCSI SAN. There is a variety of workloads that center mostly around file serving with some light database use as well as some file replication software that adds some overhead to the system. While this setup is currently stable, it is proving to be too slow as we increase the disk I/O needs of the VM's. We are about two years out from being able to budget for a proper FC SAN. In the mean time we want to go to direct attached storage to handle our current needs. I need about 2TB of useable space. What do you guys recommend?

Can I use a "simple" U320 SCSI external raid box like the promise VTrak M610p loaded with 150GB Raptor Drives and an LSI22320SE HBA?

Do I use an HP storageworks smart array 30 chassis with U320 drives and a LSI00008 SCSI RAID Controller?

Any other ideas for something in the 4~6K$/TB that will give us reasonable performance.

Thanks for any help you guys can give.

Karl P

[MrZorry]

Karl;

I am getting ready to implement ESX 3.5 with iSCSI. So I am curious to learn a little more about your iSCSI network. Is it on it's own network? are you using jumbo frames? Spanning tree? Flow control? WHat kind of initiators are you using (software vs. hardware)? NICs, TOE or HBA?

I did found this or you;

Mr Zorry

[Karl_P]

The iSCSI is on its own network, you can't use jumbo frames with VMWare 3.0.X (and I don't think you can with 3.5.X either) No Tree or flow control, Software initiators.

The issue isn't iSCSI, there are lots of successful mid size installations using iSCSI, the issue is the performance of the low end EMC offerings.

Thanks for the link, I have seen that before. The AX150 isn't bad, it's just not the right tool for the job I (now) have at hand.

Karl P

[cmanucy]

We have a couple of the Promise boxes. They don't work with ESX.... at least not with the software initiator.

I've been able to present volumes to ESX from the Promise box, but only by proxing with another linux host that presents the storage as NFS to ESX. Complicated, but it works. I wouldn't use it for production - we use it for templates and such.

Problem is that the Promise box, even if you get it working via HBA, isn't a supported configuration, so if you ever have issues, you'll have to get off of it in order to call VMWare. The other problem is that the Promise box only has 1 NIC available at any one time (units with 2 NICs are for failover only), so you're limited to a single 1Gps link for however many hosts you have.

On top of all this, Promise's SATA implementation isn't going to be nearly as robust as something from EqualLogic. I don't know what you budget provides for, but EQ gives damn good performance over SATA drives - no need for 15K Raptors.

As a matter of fact, I doubt you'd get much improvement using 15K Raptors in a Promise box due to the fact that it really can't push that much I/O - once you've got 8-16 drives, that's a lot of bandwidth to fill. We use 1TB 7.2K drives in our units - and an 8-drive array beats the snot out of a 4-drive local array w/Raptors hooked up to an Aacra RAID controller.

I wouldn't put too much faith in FC either. Personally, I think the technology has run its course, and things like iSCSI will take over.

----

Carter Manucy

[MrZorry]

You obiously have more experiance than me, so what you have said brings concern to my mind (not hat there wasen't any). Furtunatelly I am the backup Engeeneer for thsi project. A week before last the lead engenner brifed me in the project we will be working. He spesifically pointed out to the CISCO 2810 "becasue it supports jumbo frames". To me this implied the Jumbo frames was not going to be an issue with ESX. In you enviroenment might be with the AIX-150. We are going to be suing Equallogic PS3800XV.

I did a quick search and found this http://www.vmware.com/support/vi3/doc/vi3_esx35_vc25_rel_notes.html. Could it be that ESX does or did not suport jumbo framaes on it's virtual network (there realy is no need for them). But the NICs, TOE or HBAs are phisically outside the virtual wolrd.

I do know this, if you are using software initiators, all the IP and TPC assemble / disassemble is handle by the CPU. Using TOE cards or even better iSCSi HBAs, you will increas eperformance by offloading CPU prossesing. maybe the cost of implementing DAS will turn up to be more expensive than implementing iSCSI HBAs. have you looked in to that?

Sorry of I am asking more questions than the ones I am answering. I have only been at this job for a short while (allthough WI Sys Admin for a very long, long time), and traying to learn as much as I can as quickly as a can. However, I just went trough bout and ESX 3.5 class and an Equallogic class. Equallogic promices better performance on their iSCSI products than FC. Even on their lower end products.

[cmanucy]

First off, yes, TOE/HBA iSCSI cards do offload the CPU. If you will ever see that impact, however, depends on your setup. If budget is a concern, I'd put my money into the SAN and hold off on the HBA purchase.

As for the jumbo frames, yes, it is recently supported for the network (as you point out). It is NOT supported for iSCSI (where it's more needed, but perhaps they're still testing all of that). So JF support in the switch doesn't mean jack-all at the moment. What matters more for the switch is the backplane and how the interconnects are. Generally speaking, if the switch is good enough for iSCSI, jumbo frames is one of many features that's going to be supported.

----

Carter Manucy

[Karl_P]

Well, if you read my post a little more carefully, you would see I am not talking about the promise iSCSI or FC boxes which I know are not the correct answer, I was talking about their U320 SCSI box.

Equalogic is fine, but way to expensive for what I am currently looking for. I was quoted close to 30K for a PS100E, I need something closer to 15K.

FC is certainly not dead and is still the preferred large scale storage system.

iSCSI will continue to make inroads, but it isn't positioned to take over FC in the immediate future.

Karl P

[Karl_P]

Accidental Double Post

Message was edited by: Karl P

[Karl_P]

Again, as the other poster noted, Jumbo frames mean squat for iSCSI and VMWare right now. That will probably change soon, but don't build plans on it at the moment.

My performance problem has nothing to do with my ESX server and everything to do with the 150i, it is just a slow low performance box and I am hitting the performance limitations of that box. In order to get around that limit without paying for the next step of SAN performance in the 30~60K$ range I want to use direct storage.

Equallogic, while they do have a fine product, does not fall in the price range of what I am looking for right now. Also, while the product is fine, they are obviously delusional if they try to sell themselves as the end all be all storage products.

Realize that I am looking to move away from iSCSI to Direct Attached Storage. This means a bunch of fast hard drives with a direct high capacity (SCSI maybe?) link to the ESX Server. I don't actually care how I do it so long as it will be stable and supported. If your saying the promise boxes aren't acceptable even in U320 configuration, then thats fine, do you have any other ideas?

What I really need and am looking for is a recommendation for an external Direct Attached Storage idea that is in the 4'000~6'000$/TB range.

Karl P

[Karl_P]

Taking a look at my options, maybe I want to take a look at an HP StorageWorks 1500cs with both a SATA and SCSI shelf? Use the SCSI for my heavier I/O VM's and use the SATA for my lighter I/O VM's / Backups?

Any thoughts on this?

Karl P

[MrZorry]

Karl;

You are absolutely correct. I spun off this conversation on “is iSCSI tuned to peek performance”, when you question is about using the correct DAS. As long as what you pick is on VMware’s HCL I do not see a problem.

In the EWSX class I just took we had DAS, iSCSI and FC. All connected to the same host. Since you only have one host server, things like VMotion. DRS (Distributed Resource Scheduling), HA (High Availability) do not come to play. This option require that the VM files are visible by all hosts, which implies a shared data store.

You did mention having to wait a couple of years to get a proper SAN, and in the mean time spend 4 to 6K to get you trough. My suggestion would be to spend 1K to 2K or so on a good iSCSI HBA and dedicated switch.

Software initiators do not support jumbo frames, but hardware base initiators HBA do. This is what makes the difference.

This is from Equallogics class;

Gigabit Ethernet is used as the data link level to transport large frames and perform flow control

over a gigabit switched network:

• Jumbo frames

ƒ 1500 Byte MTU

− Over 80,000 frames per second processed at line rate

− 1200 CPU cycles per frame required (typical server)

− Headers consume network and CPU bandwidth (6:1)

ƒ 9000 Byte MTU

− 14,000 frames per second processed

− Average 50% throughput increase; 50% CPU utilization decrease

− 8k NFS block will fit without fragmentation

− Fewer frames = fewer issues with out of order delivery

• Flow Control - 802.3x

ƒ Flow Control allows the receiver to instruct the sender to “throttle back.” The

receiver does this by sending “pause frames” to the sender, which causes the

sender to slow packet transmission for a short period of time.

ƒ This prevents TCP from having to retransmit packets if packets are dropped due to

congestion within the network.

− TCP Retransmit time is significantly higher than back pressuring time that is

used at the 803.3x.

‰ Gigabit Ethernet and Fibre Channel comparisons:

• Transmission rate

ƒ 1 GB Ethernet: 115 MB/sec = 414 GB/hour

ƒ 1 GB Fibre: 98 MB/sec = 353 GB/hour

• Fibre Channel and Gigabit Ethernet Frames

ƒ Fibre Channel has a total frame of 2138 bytes with 36 bytes of overhead or 1.7%

overhead on the wire.

ƒ Gigabit Ethernet with a frame size of 1518 bytes has an overhead of 18 bytes (or,

worst case, counting preamble and SFD of 26 bytes) for an overhead of 1.2% or

1.7%.

ƒ Gigabit Ethernet with a frame size of 9000 bytes has an overhead of .2% (18 bytes)

or .28% (26 bytes counting preamble and SFD); so it is much lower overhead on

the wire.

• Ethernet is 17% faster with the worst case the line rate. Differences at 17% in favor of Gb

E are real with overhead. Gb E with jumbo frames can be even much better throughput

wise than 1Gbs FC

MrZorry

[mike_laspina]

Hello,

Any other ideas for something in the 4~6K$/TB that will give us reasonable performance.

I would call the EMC sales rep and find out if you can replace the iSCSI module with an FC module like the AX150 and then pick up some FC adap's and go FC since thats is were you are headed if the future.

You can connect directly and the performance will be much much better on that box over FC.

If they will not do the modules the AX150 box is currently @ $4K (w/3 .25GB Drives but you have the drives.) then add two $400 QLE220 FC cards. All that for about $5K.

Then you can even split the load across the two boxes.

Hope this helps

Mike

[Karl_P]

Zorry,

While I appreciate your enthusiasm I need you to read my posts a little closer please :).

The problem is not that I don't have iSCSI HBA's, the problem is the 150i is slow, and everyone who has ever touched one knows this. Putting iSCSI HBA's won't help my issue. I also already have dedicated switches if you read above.

As far as EQL is concerned, there products are good, even if they are overpriced (At least in my opinion). At such a point in time as EQL comes down to the 6ishK$/TB in price, I think they will have a much more interesting product in their marketspace.

As for your stats about ethernet versus FC, please stop ignoring the fact that nothing uses 1GB FC anymore with 2 or 4 GB and multiple ports being standard issue these days.

iSCSI is certainly a viable and reliable option these days, but the price/performance ratio just isn't there yet. At least not that I have found.

As a final note, if you will read my note on budget a bit closer, you will see that I am looking for something between 4 and 6 K$ per TB, and being that I need about 2TB that would put my budget between 8 and roughly 16K$

Thanks,

Karl P

[cmanucy]

Ahh yes, the MSA1500.

We have one of these, with 2x MSA-30 shelves, and 1xMSA-20 shelf.

It's a piece of junk. I/O is about the same as our Promise vtrack over NFS (if that tells you anything).

I do realize you were talking about the Promise DAS device, but I was trying to give you options (as that's what you asked for).

Keep in mind that while you might save a few bucks here or there, you're going to end up paying for it one way or another. Either in that time you have to call VMWare and they won't support you due to the configuration, or in the overall lack in performance.

You're asking for something that's cheap and fast - trust me, if such a device existed out there, it'd be plastered all over the place.

You might want to review some of the options for some of the software driven iSCSI solutions. SAN Melody might fit, but I don't know its price range.

The HP MSA-30 might be an option on a single controller, but I would NOT suggest you invest in the 1500 - it's given just about everyone who's used it headaches, and it's not a high-performance I/O device. Keep in mind that nothing P-SCSI is going to give you great performance compared to SAS.

And oh, as for the iSCSI vs. FC - I don't disagree with what you said, but I think it's foolish to invest in FC when you already have an iSCSI solution in place. For you to get what you want out of FC, you'd have to spend a ton of money, and it doesn't sound like that's an option for you. You can get the same performance of a high-end FC cluster out of iSCSI, for a fraction of the price.

----

Carter Manucy

[Karl_P]

I know that I cannot change the SP's to FC, although the concept of just buying another frame is a viable one.

My only concern is that from those I have talked to, the performance on the FC version is not that much better than the iSCSI version.

Also, we have another use for the 150i as some nearline storage, so I am not worried about moving it out of this use.

Karl P

[cmanucy]

In thinking a little more about your problem, I think you might want to look at something like this:

http://www.promise.com/product/product_detail_eng.asp?product_id=176

If you fill this thing with SAS drives, you should get nice performance out of it. You can also chain up to 4 of them together for added performance (although I don't know if you can split LUNs across them). As I said, I've been plesantly surprised at the performance of our iSCSI Promise box, and this should be even better.

The only question I don't have the answer to is what SCSI controller you can use on the ESX side to work with this. I'm sure one exists, but I've never used the external SAS connectors with ESX.

Of course... the $10K question is - will it really work and give you what you're looking for?

----

Carter Manucy

[Karl_P]

Thanks for the very helpful post. You have confirmed my suspicions on the promise box and given me some valuable input on the MSA box.

I fully realize I am looking for something cheap and fast, which is why I had ruled out any type of SAN at the moment because I knew I wasn't going to be able to do get it. That said, I still have a budget of as much as 6K per TB, which while nothing special, is over twice the retail price of SAS or high performance SCSI per TB.

Now, as for SANMelody, Wow, I think you may have got me somewhere.

SANMelody is not bad for pricing.

In fact, I can get a box with 2K3 and a load it up with SAS drives on a 3ware SAS array for my budget.

Have any of you guys ever used this before?

What about iSCSI HBA's - are there any HBA's for 3.0.2?

Thanks for the great post,

Karl P

[Anders_Gregerse]

When you say to slow, do you then mean I/O's per second, throughput, latency, etc? How are you sure that I/O is the problem? There are different solution to each problem. I would also recommend talking to some techies at EMC, they might have a solution on hand that is within your budget. With a tight budget you want to be certain that they are spend on the real problem.

[Karl_P]

The issue is one of IOPS and throughput.

In talking with other users of this box, with EMC, with my dealers, etc, the end conclusion is that everything is ok, it just no longer meets my needs. We have looked at the EQL products, but they are out of our range at the moment and a bit pricey for what they are.

In looking around, Adaptec does make a SAS raid 31605 controller that is supported in ESX. I could use this with a 4x SAS multilane connector kit and attach it to a 16 bay external SAS JBOD enclosure over multilane SAS.

This would give me up to 16 spindles of 10K or 15K SAS local to the VM Server.

The other option would be a SanMelody Installation using the QLogic HBA on the VM server and a bunch of SAS in the sanmelody server.

I like SanMelody as it still retains the ability for us to do a second VM Server in the near term, but I would like to here if any of the rest of you guys are using this here.

EDIT: If using sanmelody, would you use an adaptec, 3ware, or areca SAS raid card with its huge 2GB cache?

Thanks!

Karl P

Message was edited by: Karl P

[cmanucy]

The only supported HBA's in 3.0.2 are the QLA4050/4052 ( see p. 22 @ http://www.vmware.com/pdf/vi3_io_guide.pdf ). I'm still waiting on PCI-E support before I jump off into the HBA testing.

I haven't used SANMelody, but I've seen plenty of results in the performance thread (have you looked at it? Lots of good information - http://communities.vmware.com/thread/73745 ). I wasn't too keen on running my SAN off of Windows, but that's just me.

I've got both 3Ware and Aareca controllers - the Aareca's are faster, but I've found our Promise box (8x 7.2K SATA 1TB drives) is almost 2x's as fast than a 4-drive Aareca array. If you'd like, I can give you some of the performance #'s off both arrays, as I've tested just about everything in my shop with the config in the performance thread.

----

Carter Manucy