I am posting this as I couldn't find anything relating to this issue anywhere and wanted to make sure people knew about it and the way I have found to fix it.
I have a small test farm which has been upgraded to ESX 4 using the Host Update utility. The upgrade worked perfectly and usage through VirtualCentre is great, however, some (and only some) of the upgraded bladeservers stopped root from logging in either by SSH (which we could do previously) or by the service console. An attempt to reset the password by rebooting to single user mode failed with the error:
passwd: Module is unknown
After some digging around, the problem appears to be in the /etc/pam.d/system-auth-generic. On a working box, the /lib/security/...... parts of each line have:
/lib/security/$ISA/pam_env.so
However, on the non-working boxes, the line would read:
/lib/security/pam_env.so
This goes for every line, no matter what module was being used. When changed to include the $ISA, logging in either by SSH or the service console works. To be able to change this line, you need to boot the server into single-user mode which drops you into a root shell.
I just hope this helps other people who may come across the same issue.
