Through additional experimentation I have determined that this is related to having PCIe devices attached via Thunderbolt. On my test server for this cluster, connecting an Apple thunderbolt to ethernet adapter does not throw this error. Attaching a thunderbolt to PCIe enclosure with an Intel X540 dual 10Gb NIC in it will throw this error. This approach has worked on ESXi 6-6.7 until U1.
Well, here is the kludgy work around which for some reason is posted in a comments thread on Github. Get an EFI shell using a USB device, then you can get a single working boot. Vmware guy says they've verified it and it will be fixed in a future release. It sure would have been nice if the vSphere update manager had known not to do this upgrade since it is a known bug on a supported hardware platform...