Been running about 5 ESXI5 hosts for a couple of months now.
During a network backup on one of them, something bizarre happened: Both machines I/O latencies skyrocketed making both extremely slow/laggy. While I was able to restart the hosts, one of them now comes up with this:
Device mpx.vmhba2:C0:T0:L0 performance has deteriorated. I/O latency increased from average value of 11364 microseconds to 382956 microseconds. warning
Both hosts have had this message before the reboot. The odd thing is that this network backup has been running since installation with no issues. I don't think it's a coincidence of it happening during the network backup as both hosts did the same thing at the same time.
While the other host has been running normally for a few hours now, I would like to investigate this matter further on the host that this problem still persists on.
I have noticed that it can be slow to do operations on virtual machines on the affected host. Logging on and off takes a little longer, for example. We're not talking hours though - maybe 30-40 seconds.
Services are still running properly on the affected host.
Anyone have any tips or ideas? There's two gigabit ports between each host, so I suppose it's possible that the bus was saturated during that time. The problem is I've had to restart each host and of course now it doesn't let me set a time range for storage adapter latency (the option is grayed out.)