A stable VMWare ESXI 6.7.0 Host is now unreachable after power outage.
same as me
VMWare ESXI Host crashed overnight during unattended 2-hour power outage that exhausted all network UPS after backup generator failed to kick in.
same as me
VMWare Host NIC and Switch Port show link and activity, Host can ping itself, but every network ping to/from VMWare Host shows 100% packet loss.
same as me
Shell commands from VMWare Host console “esxcli network nic status” and “esxcli network ip interface ipv4 get” show proper status and IP address, yet the Host’s ARP table cache shows no neighbors.
same as me
Troubleshooting includes swapping in multiple Cat5 patch cables, using alternate ports (on both switch and host) and even swapping out the Host’s switch, all LED link and system statuses continue to appear normal yet VMWare Host cannot ping, be pinged or see its NAS. Original cable/port/switch have been restored to pre-failure status, exactly as everything was before.
same as me
It’s a small network, for design and connectivity, core network includes Firewall, the usual switch gear elements, but this network also relies on a Proxy VM that lives on the failed VMWare Host. So I did wonder if this issue could be a paradox that the firewall/network needs the Proxy, but the Proxy doesnt exist because VMWare can’t reach that boot image on NAS. My guess however would be a No, as the Proxy is primarily an internet focused resource anyway.
Not same as me, on my end, any access to Public facing "playground" is all physical. Investigation and TAP tools are VM's which would be bad to loose but would have no impact on potential "unwanted" loss of power.
everything else also runs on NSX
This may be a big ask for my first question, but I’m entirely stumped, it just doesn’t make sense. I’m capable at some things, but I‘m still just programming and help desk … and we lost our IT and Security expert … so I’m IT until there’s a replacement.
You know... sh... happen when its not the time. Also pretty sure that the little line at the bottom of your employment contract (the one that says "and all other relevant task" is biting your a.... now... Been there... just be 100% certain to take notes... .like ... a SH.... load of notes of everything you do, talk about, with who, where, when... like... i am dead serious... DO NOT ... DO ANYTHING IF YOU DONT TAKE NOTES... that could save your "reputation"... You talked about a production env... have everyone see you treat it like if it was your own life... BEEN THERE AS WELL...
Like... its my good friend -->Murphy<-- that had me learn the hard way.
VMware is pretty good up until you cut power without doing the proper shutdown but that... well.. you already know.
Remember that vCenter is the one used to create and manage vDS's... your next potential real challenge will be to re-instate vCenter so it can still manage those vDS's. If you re-create vCenter without a backup... you will have to add the hosts to vCenter which wont use the same vDS's and therefore... be problematic.
From what i have experience, vSAN should still be up in your case (except the host that went "**bleep**... up"). that is a good thing for you who works in your favor. for now.
i would, at the same time that you work on this... have someone else make backup of EVERYTHING and every VM's, if not already done, on your end, you can make backup of the host's configuration by logging in with ssh and "vim-cmd hostsvc/firmware/backup_config". by having backup's, you will not loose everything. (you can even run that command on the faulty host if you have an iDRAC/iLO/IPMI. it will save the configuration locally... and you will have a small amount of time to get it out but.. you can log in and transfer it into another location up until you have access back on Mgmt Interface.
if you have a backup of vCenter, and a copy of the ISO of the version it was running at the moment it died... Like.. up to the version of patch...
mount it on a workstation that has access to a good Host on the network. (like if you would mount an ISO on a Win10 VM). then... proceed with the fresh install on a regular datastore. if any patch were applied to vCenter, apply them first, after its done, restore the backup of vCenter...
at that point, you will be in a much better posture to work toward putting back vCenter on the vDS.
again i will look for my doc and come back a little later.
cheers.
