General VMs briefly lose iSCSI storage connection question?

[romatlo32]

I am a new systems administrator here and we have 1 dedicated network switch (single power supply) for our iSCSI connections to our vSphere 6.5 hosts.

This switch is currently not connected to a UPS protected power source.  We've had two power cycles in this switch over the past two weeks, causing our Nimble storage to fail over to other controller, loss of iSCSI connections to hosts/VMs, etc.

Once the switch power is restored, the storage, hosts, and VMs recover and have encountered only minor issues.

Looking for comment/feedback:

I plan to power off this switch and move its power connection to a protected UPS one night next week.

I've suggested to management that would be better to gracefully power off VMs and power back on after the temporary brief iSCSI network outage.

Getting some pushback from management, especially considering the  minor impact from the previous two power offs of this switch.

Anyone have general feedback here and share their experience and consideration for this maintenance job?

[daphnissov]

Yes, some general feedback. If you have a single switch running only one iSCSI fabric with a single power connection that isn't UPS protected, and you're getting pushback from management on (rightly) shutting down VMs to correct the power situation, they as management have failed. This is literally the worst possible way (just about) that you could provide IP storage network connectivity via iSCSI and they should be told so.

But specifically on the procedure you outline, yes, that would be far preferable rather than to just yank the power and move it over to a UPS thus rebooting the one and only switch providing traffic for storage.

[romatlo32]

Thank you for that.  I figured that was a no brainer as well.  Of course you gracefully power off VMs so not to yank their network storage connections out from under them.  Who know what could happen, especially with database VMs, etc.

I am considering only power off VMs.  I would leave hosts and storage running.  Network storage may initiate a controller failover, but prepared to live with that.  Considering powering off VMs, powering off switch, moving its power cord, power back on switch, and then powering on VMs once network storage is restored.

Sound ok?

[daphnissov · Accepted solution]

Considering powering off VMs, powering off switch, moving its power cord, power back on switch, and then powering on VMs once network storage is restored.

In your situation, that's really all you can do since you don't have another path to storage. There's no need to shutdown ESXi hosts or storage as it should recover after the switch comes back up.