ESXi 6.0 Unhappy with iSCSI DataStore presented by VM using Virtual Disks on Same Host

[PaulWestNet]

I'm hoping someone has run into this and figured out what's going on.

My scenario is in a test lab - wouldn't deploy this setup in production, even if it worked, but why it fails is a bit of a mystery.

The setup:

I have 2 VMs - one running QuadStor using VMDKs for both boot and storage for an iSCSI export, and the other is a test VM that has been placed on the iSCSI storage presented by the QuadStor VM. This is all on a single host, so ESXi 6.0 has an iSCSI storage adapter connected to a VM it's hosting.

The problem is that when I turn on that test VM that's using the iSCSI storage provided by the QuadStor VM, I run into write issues almost immediately. The test VM will start to crawl and the host becomes sluggish and virtually unresponsive. I also see a ton of errors in QuadStor's logs about unresponsive disks. However, if I don't turn on the test VM and just browse the iSCSI datastore, I can copy large ISOs to it with no issues, move data around on it, copy data off of it - it all behaves fine. The problems show up when a VMDK is being written to the iSCSI presented datastore only.

This isn't just ESXi 6.0, either. I happens on ESXi 5.5, too. It's not hardware - my ESXi 6.0 host is a little Gigabyte BRIX i3 with a Samsung 850 Pro SSD, and the ESXi 5.5 host is an Intel server with dual Xeon 5500 series processors, an LSI RAID controller, and Seagate 10k spindles. Totally different hardware.

This isn't just QuadStor, either. It happens with StarWind's Virtual SAN, too - though it's less pronounced. Given that QuadStor lives in top of CentOS (I tried 6.6 and 7.0 with no difference), and StarWind lives on top of Windows (I used Windows 2012 in my testing), it doesn't seem to be a guest OS related issue.

I can get around the problem if I slap a SAS HBA into the Intel host and pass it through to QuadStor and tell QuadStor to put the VMDK it's going to use for the iSCSI export on it (it doesn't seem to be a problem to leave the boot VMDK on the hosts disks, however - just the iSCSI export's VMDK needs to be on the storage attached to the HBA it's been given). Once I do that, all my problems go away and everything runs as you'd expect.

It smells like a bug in ESXi that's been there for some time, doesn't it? Could it be the nesting is causing some kind of I/O storm that is completely consuming the host? Should that happen with dual XEONs, 20GB of RAM, and only the 2 VMs on the host to contend with? It almost feels like the storage equivalent of plugging a network switch into itself and creating a network loop that kills the network with packets running in circles.

Any ideas? I can see people running into this frequently, now that "Hyper Convergence" is the buzz phrase of the moment.

[RichardBush]

Hi Paul,

I ve had something similar this week actually. My Synology NAS box died on me, so I decided to setup a 3 node HP Store Virtual ISCSI set, in a 3 way mirror. IF the active ISCSI node is running on the same physical host as a VM I get awful latency, I flip it to being active on another host and the problem goes away. I ve tried multiple configurations to reduce the hit and the only way I can do it is to have a single NIC for the VM's and a single Nic for ISCSI. But it still doesn't totally resolve the issue. If I trigger something like a large file copy the ISCSI path will time out and drop.

I even changed the switch in my home lab to confirm I wasn't hitting a throughput issue. Im running 6.0 and haven't tried it on any of the older versions. The Datastore I ve used to create the ISCSI volumes are SSD so I cant see the disks being an issue, The VM will sometimes see 1000+ms latency, but the Host is seeing sub 10, and very low NIC throughput.

Sorry it doesn't help you but will be interested to see what other people have found with this type of config.

Rich

[PaulWestNet]

Hi, Rich,

Yes, I saw the same thing. I was able to connect my second host to my iSCSI storage that was using VMDKs, and that was happy. It was a little slow, since the second host was reaching over to the host with the storage via a 1Gbps link, but that was to be expected. Neither host became sluggish or unresponsive, however, implying that ESXi was happy, as long as the VMDK the storage was being presented from was anywhere but on the same host as the VM using the storage. As soon as they're both on the same host, boom (which is entirely possible, if you set up a 2-node (or in your case, 3-node) replicated storage configuration and VMWare HA or DRS moves your VM to the host with the storage VM on it).

[RichardBush]

Ok so I thought id spend a bit of time googling this, and have found a few people mention disabling delayed ACK resolved this issue.

Here is the KB VMware KB: ESX/ESXi hosts might experience read or write performance issues with certain storage arr...

I think you said this was a Lab environment ? If so might be worth a test ?

Rich

[PaulWestNet]

That's some good Googling, Rich. I Googled for ages and couldn't find a whole lot.

I will give it a shot and see what happens, then report back.

[PaulWestNet]

Well, after 40 minutes of reconfiguring, testing, and eating pizza, no luck. Turning off delayed ACK on the iSCSI adapter didn't help, sadly.

I also figured it might be the vmxnet 3 adapter, since I read in another thread that it was causing problems for some people on ESXi 6.0. I'm testing on the ESXi 5.5 host, but thought I'd switch it to the e1000 adapter on the storage VM, just in case. No help there, either.

[RichardBush]

Dam thats a shame;

How are your NICs configured ? do you have the ability to separate the ISCI Initiator and the Storage VM onto different vswitches to force the traffic over the physical network ?

[PaulWestNet]

Indeed it is.

Being a lab, I just threw the iSCSI adapter on the VM Network vSwitch, and the QuadStor iSCSI NIC on the same vSwitch. I think the Intel host has 2 NICs, so I should be able to force the traffic out one NIC and into the other. I'll give that a try and see what happens.

[PaulWestNet]

Nope. Forcing the iSCSI traffic out onto the physical LAN didn't help, either. Same as before - the VM halts during writes (I'm actually just having it try to install CentOS 7 as a test), the iSCSI datastore eventually becomes inactive, and that's all she wrote. At that point, the whole host is on it's knees. Everything is uber slow. It even takes a few minutes to shut off the test VM so I can get the host responsive again.

At least the problem is easy to replicate for testing *grin*.

[RichardBush]

Hi Paul,

Did you make any headway with this ? I tried the fix at the weekend and did see some improvement, unfortunately it doesn't seem to have helped you. The other thing that made a different for me was to use E1000 nic on the VSA, im not sure if this is due to the driver or the fact that the VSA was seeing 10GB nics with VMXNET3.

Either way with both of the above its not as quick as I would expect, I can easily push the latency, but at least now it doesn't seem to go to all paths down state.

R

[PaulWestNet]

Hi, Rich,

No, I didn't make any headway - though I didn't play with it much beyond what we tried a few days ago. I got focused on Exchange 2013 cluster setup and pining for the 10Gbps NIC driver for the new XEON 1500-D series chips, instead.

I pretty much put it down to ESXi losing it's nut for whatever reason and just decided it didn't matter much because who'd deploy storage on virtual disks anyway? In a lab, sure, and it should work fine in theory - but in a production environment, it makes sense to just pass the storage and HBA directly to the VM and let it manage everything without ESXi's involvement.

I know this will come up again from someone, though, since it's an obvious issue that can be easily reproduced.

The funny thing is, I did deploy VSAN in a nested ESXi setup, which of course was using virtual disks - and that had no problem at all. Though, that's not a VM's virtual disk trying to store itself on another VM's virtual disk, since VSAN is native to the ESXi kernel, so it's a different animal. Maybe people don't land on the idea of a passed through HBA will just shell out for VSAN, and that suits VMWare just fine (conspiracy theory!)

I'm glad you got some improvement from the changes in your environment, though. Dedicated HBAs would almost certainly solve the issue entirely, but failing that, the tweaks seem to have helped you a fair bit.

[RichardBush]

Ah no problem. I agree, mine is just a stop gap until my synology device is repaired, and had to make the best of our the hardware I had.

As you say, I can see this coming up again, so this post my at least help someone I the future.

Rich

[efishta]

I just wanted to add my experience with this issue on ESXi 6.0. I haven't tested previous versions in this scenario, but would be inclined to think they're all susceptible to this slowdown.

My home lab setup was recently upgraded with an additional SSD, and so now with two HUGE 250 GB SSDs, I decided to create a half terabyte deduplicated iSCSI share, and pretty much take care of all my storage needs at this time.

I remembered having read on a VMWare blog post that there is very little overhead in VMFS5 over passing through disks, so I took that route. Created a datastore on each SSD disk that matched the total capacity of the disk it was created on, then created a VMDK on each to pass through to the iSCSI host (Windows Server 2012 R2 in this case). Just as in your experience, benchmarks and performance were great. Very happy with the outcome actually.

When I tried installing the VCSA 6.0 appliance, I noticed this behavior that you are describing. Everything ground to a halt, I would keep losing connectivity to the ESXi host (via Windows client) while trying to deploy the VCSA 6 onto the datastore being shared internally through VMXNET3 adapters.

After some Google-fu, I ran across this thread, which gave me the idea to pass the disks through as RDMs manually. I did that and everything is working fine.  Thank you for the suggestion.

This does make me wonder what kind of edge case is this that has not been tested by VMWare... or it could be intentional, of course.

[vlho]

Hi guys,

maybe help you disable VAAI ATS heartbeat, look here:

VMware KB: Enabling or disabling VAAI ATS heartbeat

[JasonVK]

Hello, Paul. Did you manage to solve your issue? I’m currently running free version of Starwind and while I don’t see anything you complain about.

I’m just not willing to have any surprises in the future..

[CollinChaffin]

Have you tried this:

Go into advanced settings and set:

Disk.UseLunReset = 1

Disk.UseDeviceReset = 0

Curious if this helps your issue!

-Collin