Maintenance mode enough to shutdown SAN and networking?

[twotech]

VMWare Community,

We will be performing some maintenance on your SAN and LAN network connecting our HP C-7000 blade system with around 14 ESXi 5.0 hosts and 3 HP Lefthand P4000 Series clusters(6 nodes), 3 VM Clusters. As a result the whole C-7000 chassis as well as the SAN will lose network connectivity. We don't have a second VM or SAN enviroment to migrate the guests to.

Our plan is to:

1. Shutdown all the VM guests.

2. Place all the hosts in maintenance mode.

3. Temporarily disable HA and DRS to prevent vMotion when the host are brought back up from maintenance mode.

4. Shutdown the SAN.

5. Shutdown/Perform maintenance on the network LAN/SAN switches.

6. Power on network switches.

7. Power on SAN.

8. Turn maintenance mode off for VM hosts.

9. Turn on VM guests.

10. Turn on HA and DRS.

Does this sound like a correct plan? Do you see anything wrong with it? I'm curious whether maintenance mode on the hosts will be enough to prevent any problems and whether the SAN datastore and paths will reconnect. I'm afraid that if I shut down the hosts completely it will create networking problems such as vmNIC renumbering etc.

Your help and thoughts would be greatly appreciated.

[spravtek]

Hi,

Your plan looks ok ... I'm guessing by now you already performed your maintenance though ...

Just make sure you leave enough time between powering up your SAN and your VMware cluster, that way your ESXi hosts will not be searching for storage when they are started up earlier then your storage.

I've performed some UPS tests on VMware setups lately, and between all problems I encountered, I didn't encounter any vmnics being changed or storage path issues ...

[twotech]

Spravtek,

Thanks for your answer. No, I actually have not performed the maintenance yet. Will do so in a couple of hours. I was actually hoping that I could perform everything without shutting down the hosts. From your answer it seems that you think I will be shutting them down. Just wanted to see if people thought that everything could be performed in VM Host "Maintenance Mode".

I'm not sure what type of environment your running but I'm concerned with the HP blade system's networking virtual nature which might appear to ESXi as if hardware changed.

Thanks

[spravtek]

Ah, I see ... Sorry misunderstood that ...

I think you will be ok in maintenance mode... But I would still opt for a shutdown, especially when you are shutting down your SAN.

I have multiple sites/customers with HP Blades (C3000, C7000) ... Shutting down the hosts/blades never caused problems for me ...

Besides that I also changed around some networking while the hosts where in maintenance mode and even sometimes live (which I don't recommend by the way ) ... Also didn't have any issue with VMware thinking hardware got changed.

What kind of changes are you planning exactly?

[twotech]

No problem. Did you ever shut down ALL the hosts in a particular chassis?

We're upgrading the code on the SAN/LAN top of the rack switches as well as uplinks from 1G to 10G from those switches back to the core. As a precaution we decided to shut down the whole VM enviroment. So.. no changes to HP chassis networking but it will also lose it's uplinks in the process.(they will be off for the duration of the maintenance) BTW we are using HP VC and VDS's.

[spravtek · Accepted solution]

Yes, when we perform UPS tests, we take the whole environment down, let the UPS shut down completely and everything ...

Besides that, I also take down the hosts when performing maintenance on the blade chassis as well ...

Haven't had issues ... Knock on wood

If you're not changing anything fundamental on the chassis or the hosts, you should be ok ... And when the hosts are down (if) then it won't matter if the uplinks are gone ... If you leave them in maintenance you'll obviously will see some, if not a lot, of alarms

[twotech]

Thanks again Spravtek. I have a couple of more hours before the maintenance but I'm leaning towards the shutting down of the hosts vs maintenance mode only. This is what VMWare recommends anyhow. Alarms aren't really of concern since I would know what caused them.

If anyone has any other opinions please respond.

[spravtek]

Yes ... Shutting the hosts down is the best option ...

Best of luck!