We're doing a PoC in our lab and have everything up and running. At this point we're testing resiliency, and have come up with a problem:
Servers are booting using PXE every time after reboot, as expected. They are set to do stateless caching. We were able to kill DHCP, reboot a server and it booted back using cached image, as expected.
However, once DHCP was brought back online, the server continues to boot from cached image and does not revert back to booting from PXE image. We've tried a few things to prove PXE / DHCP works fine with another fresh server and it does.
All our servers have default boot order in BIOS of floppy > disk > ... PXE. This hasn't changed at any point.
So our questions are:
1. Should the host revert to booting from PXE once DHCP / PXE components are back online? How to force this?
2. Should BIOS boot order be changed to 1.PXE > 2.disk in order to facilitate this? Bear in mind everything was working fine with default boot order, server was booting from PXE every time until we simulated DHCP failure.
3. Why exactly does the server boot from PXE every time even though BIOS boot order says 1.disk > 5.PXE? I understand on the very first boot there is nothing on the local disk so it must boot from PXE, but if boot image is cached to local disk upon first successful boot then after reboot it will be there for the host to boot from? Why is is still going to PXE instead of using cached image?