We bought a new Dell R730 this summer preloaded with ESXi 5.5.0 2068190 and dual 16gb SSD flash boot. I am using it in free licensing mode.
Everything worked fine with it at first, migrated two critical service VMs onto it, everything seemed normal except for the migrated Backup Exec, Quantum LTO-5 SAS tape drive, and LSI controller. At the time I was not aware that tape devices and VMWare do not get along.
I think I may have screwed up something in the ESXi hypervisor config, trying to get the tape drive to work, where now it fails to save the hypervisor configuration, though there are no errors displayed in the vSphere console log to indicate this.
The problems first began when I was trying to understand the tape drive going offline and tried to enable DirectPath for the dedicated SAS controller. I don't know if free licensing supports that, but the options exist in the vSphere client so... it should work? Although I could go into the DirectPath configration for the hypervisor and select the SAS controller for DirectPath, on reboot of the hypervisor, the configuration disappears and nothing is configured.
After Tape Offline problems after every Windows Update and VM reboot, I have given up, and am "de-virtualizing" the tape backup system back onto physical hardware. I don't want any more backup unreliability problems due to VMWare not giving a **** about tape support.
So this last weekend I made changes to the inventory, and hypervisor config settings, shut it all down, moved the tape controller... and on restart, all the hypervisor changes disappeared, and it boots in maintenance mode, which it never did before, so now it no longer powers up by itself after a power outage. I now have to remotely access vSphere to take it out of maintenance mode and manually start the VMs.
The new VMs are gone from the inventory, but are still there in the datastore and can be re-added. Next I restart the R730, and again poof the inventory changes are gone and we're back in maintenance mode.
The really odd bit here is that the original VMs installed in August 2015 are still present in the inventory, so it's not a total configuration loss but just it can't save changes now.
Also I've found that I can edit the VM configurations and that saves just fine, but it's the hypervisor itself that isn't saving configs.
The boot menu recovery mode is useless. "No prior hypervisors found."
The hypervisor console screen also doesn't work for some reason, though it did when the system was originally built this summer. I was also trying to enable USB xHCI support for a USB 3.0 external backup drive, and I suspect that may be playing a part here too since there's apparently some known weirdness regarding 5.5.0 and USB 3.0 on the Dell R730.
So.... now what?
Apparently I need to either wipe the R730 flash boot SD cards and reinstall a fresh new ESX 5.5.0 again, or I need to figure out what is wrong with the current configuration and manually remove whatever settings are making the current hypervisor choke to death and fail to save its settings.
I tested enabling SSH via vSphere and that does work, I am able to SSH in with PuTTY and look at /var/log on the hypervisor. But I have no idea what to look for, to figure out why it's not saving its configs [anymore].