Unknown ESXi root password and lost vCS access

[Przemko1]

Hi Dears,

We have production environment consisting of 2 ESXi  5.5 hosts with 12 VMs on it. It was managed by Windows vCS installed on physical HP server. On this server was also veeam installed and backups of VM's was done everyday evening. I used vCS Windows Client to connect to vCS. (I get this environment from someone IT guy before me) This server is connected backbone with our SAN. I don't have any spare servers. This server with veeam and vCS crashed and can't boot Windows, so also lost vCS and Veeam access. All VMs are fine and still running but can't be managed. I can connect to individaul VM remotely. After this I discovered that I don't have valid root passwords to these 2 ESXi hosts (and noone from past knows it) so can't even shut down them gracefully (because it ask me for root password) Even if I would reinstall vCS from scratch then as I assume it will ask me for ESXi  root password while adding these 2 ESXi hosts to datacenter during initial configuration - as I right ?

What is best path to get out of this issue ?

[scott28tt]

@Przemko1 

Moderator: Moved to vSphere Discussions

This thread created yesterday sounds very similar: https://communities.vmware.com/t5/VMware-vSphere-Discussions/Lost-vCenter-Server-and-access-to-VMs/m...

Both you and the user who created that thread are visiting the VMTN site from the same IP address. Are you asking about the same environment?

[Przemko1]

it was my co-worker from job who first found this server issue and described it but didn't  added  additional info to initial post so I created this one with more details. If possible previous can be moved as less descriptive. Thank you.

[scott28tt]

@Przemko1 

Moderator: The duplicate thread has been archived.

Copying my reply from that post: (which was the only reply)

Wish I had some good news, read this KB about a forgotten root password: https://kb.vmware.com/s/article/1317898

Also understand that 5.5 is no longer supported or updated, you need to be on at least 6.5 if you are running anything like production workloads.

Might be time to build a new environment and look to move your VMs over...

[Przemko1]

Thank you but either I don't understand something in this KB or something is messed: mentioned article is divided on sections based on version:

In ESXi 3.5 and above

ESX 3.x and 4.x

ESX Server 2.x

so our scenario is in first section - In ESXi 3.5 and above  which states that Reinstalling the ESXi host is the only supported way to reset a password on ESXi

so to do this I must be able to move my VMs to another host but I can't because dont have vCS just now. Later they mention: From ESXi 4.1 onwards, the host profile feature was introduced.
If the host is managed by vCenter and is still connected,......        but again I can not use any feature from vCS (any host profile or whatever) because lost vCS.

To sum it up: I have running VMs (healthy and fine)  but lost vCS and no ESXi root passwords...

[e_espinel]

Hi.

Many times customers stay in very old versions due to lack of support or knowledge to update the versions. Another limitation would be the Hardware that also usually old.

Reinstalling the ESXi host is the only supported way to reset a password on ESXi.

I suggest you have the vCenter Server and Veeam reconfigured and configured as soon as possible to have it ready for any problem that requires recovering the VMs from a backup.

As you still have remote access to the VMs, you should turn them off before proceeding to reinstall ESXi.

The method is quite safe and the steps are simple:

1. remotely shutdown the VMs of one of the ESXi hosts

2. then shut down the ESXi Host and reboot it with the CD (ISO image) with the same version you had.

3 Choose the same disk where the ESXi was installed, the VMware will tell you that it found an ESXi and VMFS, you must choose the 2nd option: Install ESXi, preserve VMFS Store

4 continue with the installation by entering the same IP and name that the ESXi Host had.

5 Configure virtual switches and Nics.

6. Finally you will have to register the VMs in the ESXi host and verify that they work normally.

You must have at hand all the information about the configuration of the ESXi and the VMs.

I assume that since you have version 5.5 you should have 2 network cards, one for administration and one for data. Check how many network cables you have connected to the server.

If the installation was carried out by a supplier, a good idea would be to contact him and request all the information about his installation and configuration.

Usually a technical guide is provided with all the details about the installation and configuration.

You can google for other ways to reset password in VMware vSphere, but be careful and be ready to reinstall ESXi if something goes wrong.

[Przemko1]

Hi Enrique,

Good words from you. I have some thoughts about points you described. In reality I must reinstall besides ESXi hosts also vCS - is here any suggested order, what should be installed first (hosts or vCS) ?

Yes both hosts have 2x  4 port 1Gbit in it - same as physical server on which vCS and Veeam is/was installed. It is connected by FC with SAN (dual controller A&B)

Installation was done by supplier some years ago and unfortunately they don't have any documentation about it, engeener left that supplier as I know.

So It will be painfull and must be done very carefully to don't lost anything. The worst is that I can install Veeam on another machine but as I assume it will ask me for credentals to add vCS (which is down now) to infrastructure or even particular ESXi hosts. So Im curious if is there any way / other method /software instead of Veeam which can make backup before operation (before shut down VMs) and will not ask me for ESXi credentials.

Thank you for any help.

[e_espinel]

Hello.
Not having documentation on configuring ESXi Hosts and VMs further complicates the recovery process.

It is recommended that you hire a VMware service provider to perform these jobs. However, if you have experience and a good knowledge of VMware, you should have no difficulty performing these jobs.

The vCenter Server and the Veeam should be installed and fully configured (that has to be done first) and since you are talking about a SAN it is assumed that the VMs backups are there, you have to make sure you have access to them.

[Lalegre]

Hey @Przemko1,

As I can see you do not have to much options to restore the ESXi root password, however there is still one more procedure to follow that is described in many post that consist on booting the ESXi with an Ubuntu Live CD and using Gparted to modify the configuration files of the ESXi.

With this procedure you will be able to reset to blank the root password without any issues and once you have of the ESXi up and running then you can Add the remaining VMs to the inventory of that ESXi without powering them on in case you do not have enough resources.

Here is one of the post that applies for ESXi 5.5 and is pretty clear and straight forward to follow: https://www.top-password.com/knowledge/reset-esxi-root-password.html

In addition to what @e_espinel said, try to re-install Veeam first and connect to the existing Backup Repositories to scan the healthy .vbr and .vib files (Veeam backup files). In the worst scenario, you will be able to restore the Virtual Machines. 

From my perspective, installing vCenter Server before doing this is completely worthless as you do not have any ESXi available so your only option would be installing it on a Physical Server which does not make sense for me at least as you do not gain any capability, you just add complexity to your environment and in the future if you want to update the vCenter Server will be even harder as physical vCenter is only supported on Windows which is deprecated since version 7. 

Even if you have the choice I would recommend you to install vCenter Server Appliance in version 6.5 which is supported by VMware and you can still add the ESXi 5.5 hosts. However this will be only possible if your Veeam solution supports it.

[Przemko1]

Hey @Lalegre and Enrique 

thank you for your input, I really appreciate it.

Yes I know that procedure and affraid that must go this way (reset using Live CD, Gparted  and console commends)

About your advice: to re-install Veeam first and connect to the existing Backup Repositories to scan the healthy .vbr and .vib files (Veeam backup files). In the worst scenario, you will be able to restore the Virtual Machines.  

According to this post: https://domalab.com/manage-vmware-infrastructure-veeam/

I can install Veeam but later in "backup infrastructure" it will ask me for credentals to add particular hosts (to which currently I don;t know root passwords) or whole vCS (to which I don't have access because of crash of physical server on which this vCS was installed. So is it "vicious circle"

Sorry if I messed something but want to prepare good for this operation to not mess running VMs.  Backups of all VMs were created every evening on partition D  on this crashed Windows Server (backup repository set in Veeam) - so in case of any failure of particular VM  I was able to recover it via Veeam from D  - but fortunately in past I had no need to use it. Just now these backups residing there (on D of this crashed server) are older and older every day....     I also have set in Veeam "backup copy" on external place - Synology NAS server to which I have access but last backup there was done last evening before crash. So even if I install new Veeam instance somewhere and search these backups from NAS then: they are older and older every day, and also I can not restore them in Veeam because can not add vCS in "backup infrastructure"  to restore. Or Im wrong ??

So it looks for me that when I want reset passwords on these 2 ESXi hosts , earlier I must shut down all VMs and try to crack theses password - keeping in head that in case of any failure I can't power-on later these VMs (because lack of vCS used to start VM)   "vicious circle" ?

[Przemko1]

Hey @Lalegre and Enrique

thank you for your input, I really appreciate it.

Yes I know that procedure and affraid that must go this way (reset using Live CD ,Gparted  and console commends)

About your advice: to re-install Veeam first and connect to the existing Backup Repositories to scan the healthy .vbr and .vib files (Veeam backup files). In the worst scenario, you will be able to restore the Virtual Machines.    -  according to this post: https://domalab.com/manage-vmware-infrastructure-veeam/

I can install Veeam but later in "backup infrastructure" it will ask me for credentals to add particular hosts (to which currently I don;t know root passwords) or whole vCS (to which I don't have access because of crash of physical server on which this vCS was installed. So is it "vicious circle"

Sorry if I messed something but want to prepare good for this operation to not mess running VMs.  Additionally backup jobs of VMs was created every evening on partition D on this crashed server (backup repository set in Veeam) - so in case of any failure of particular VM  I was able to recover it via Veeam from D - but fortunately in past I had no need to use it. Just now these backups residing there (on D of this crashed server) and are older and older every day....     I also have set in Veeam "backup copy" on external place – Synology NAS server to which I have access but last backup there was copied last evening before crash. So even if I install new Veeam somewhere and add these backups from NAS they are older and older every day, and also I can not restore them in Veeam because can not add vCS in "backup infrastructure" because it’s lost.

So it looks for me that when I want reset password on these ESXi hosts , earlier I must shut down all VMs and try to crack this password - knowing that in case of any failure I can't power-on these VMs (because lack of vCS used normally to start VM)   "vicious circle" ?

[Przemko1]

Hey @Lalegre and Enrique 

thank you for your input, I really appreciate it.

Yes I know that procedure and affraid that must go this way (reset using Live CD ,Gparted  and console commends)

About your advice: to re-install Veeam first and connect to the existing Backup Repositories to scan the healthy .vbr and .vib files (Veeam backup files). In the worst scenario, you will be able to restore the Virtual Machines.    -  according to this post: https://domalab.com/manage-vmware-infrastructure-veeam/

I can install Veeam but later in "backup infrastructure" it will ask me for credentals to add particular hosts (to which currently I don;t know root passwords) or whole vCS (to which I don't have access because of crash of physical server on which this vCS was installed. So is it "vicious circle"

Sorry if I messed something but want to prepare good for this operation to not mess running VMs.  Additionally backup jobs of VMs was created every evening on partition D on this crashed server (backup repository set in Veeam) - so in case of any failure of particular VM  I was able to recover it via Veeam from D - but fortunately in past I had no need to use it. Just now these backups residing there (on D of this crashed server) and are older and older every day....     I also have set in Veeam "backup copy" on external place – Synology NAS server to which I have access but last backup there was copied last evening before crash. So even if I install new Veeam somewhere and add these backups from NAS they are older and older every day, and also I can not restore them in Veeam because can not add vCS in "backup infrastructure" because it’s lost.

So it looks for me that when I want reset password on these ESXi hosts , earlier I must shut down all VMs and try to crack this password - knowing that in case of any failure I can't power-on these VMs (because lack of vCS used normally to start VM)   "vicious circle" ?

[Przemko1]

Hey @Lalegre and Enrique

thank you for your input, I really appreciate it.

Yes I know that procedure and affraid that must go this way (reset using Live CD ,Gparted  and console commends)

About your advice: to re-install Veeam first and connect to the existing Backup Repositories to scan the healthy .vbr and .vib files (Veeam backup files). In the worst scenario, you will be able to restore the Virtual Machines.  

According to this post: https://domalab.com/manage-vmware-infrastructure-veeam/

I can install Veeam but later in "backup infrastructure" it will ask me for credentals to add particular hosts (to which currently I don't know root passwords) or whole vCS (to which I don't have access because of crash of physical server on which this vCS was installed. So is it "vicious circle"

Sorry if I messed something but want to prepare good for this operation to not mess running VMs.  Additionally backups of VMs were created every evening on partition D on that crashed server (backup repository set in Veeam) - so in case of any failure of particular VM  I was able to recover it via Veeam from D - but fortunately in past I had no need to use it. Just now these backups residing there (on D of this crashed server) and are older and older every day....     I also have set in Veeam "backup copy" on external place – Synology NAS server to which I have access but last backup there was copied last evening before crash. So even if I install new Veeam somewhere and add these backups from NAS they are older and older every day, and also I can not restore them in Veeam because can not add vCS in "backup infrastructure" because it’s lost.

So it looks for me that when I want reset password on these ESXi hosts , earlier I must shut down all VMs and try to crack this password - knowing that in case of any failure I can't power-on these VMs (because lack of vCS used normally to start VM)   "vicious circle" ?

[Przemko1]

Hey @Lalegre and Enrique

thank you for your input, I really appreciate it.

Yes I know that procedure and affraid that must go this way (reset using Live CD, Gparted  and console commends)

About your advice: to re-install Veeam first and connect to the existing Backup Repositories to scan the healthy .vbr and .vib files (Veeam backup files). In the worst scenario, you will be able to restore the Virtual Machines.    -  according to this post: https://domalab.com/manage-vmware-infrastructure-veeam/

I can install Veeam but later in "backup infrastructure" it will ask me for credentals to add particular hosts (to which currently I don;t know root passwords) or whole vCS (to which I don't have access because of crash of physical server on which this vCS was installed. So is it "vicious circle"

Sorry if I messed something but want to prepare good for this operation to not mess running VMs.  Backups of VMs was created every evening on partition D on this crashed server (backup repository set in Veeam) - so in case of any failure of particular VM  I was able to recover it via Veeam from D volume. Just now these backups residing there (on D of this crashed server) and are older and older every day....     I also have set in Veeam "backup copy" on external place – Synology NAS server to which I have access but last backup there was copied last evening before crash. So even if I install new Veeam somewhere and add these backups from NAS they are older and older every day, and also I can not restore them in Veeam because can not add vCS in "backup infrastructure" because it’s lost.

So it looks for me that when I want reset password on these ESXi hosts , earlier I must shut down all VMs and try to crack this password - knowing that in case of any failure I can't power-on these VMs (because lack of vCS used normally to start VM)   "vicious circle" ?

[Lalegre]

Hey @Przemko1,

Just to clarify and close one of your doubts, the first step is 100% to shutdown the VMs that you currently have running to avoid any issue. From there you need to recover the ESXi as first point, there is no other choice and you can use whatever method was previously described.

And yes, you are absolutely right, for restoring the Virtual Machines you need the password of the ESXi or vCenter for adding them as endpoints in Veeam. 

However as you have your virtual machines in a SAN that is external, I suggest you to go with the re-install option and preserve the VMFS as it is the quickest and cleanest one. Once you recover at least on ESXi, you can re-register all the VMs on that ESXi, connect it to Veeam and do a quick backup just to make sure you have the latest copy available.

I suggest you to follow the next procedure as it is not complex at all.

[Przemko1]

hey @Lalegre 

yes i know, before all operations I must shut down all VMs on these hosts but.....

1) how can I make backup of these VMs - I lost access to Veeam console to backup my VMs - is there any way ?

2) If I have get root password back  in my hands - I suppose then I can in first step (even before installing vCS) install Windows vSphere Client to connect to each ESXi hosts individually to power-on particular VMs - correct ?

3) If point 2 above is NOT true - then can I in any way power-on VMs directly from ESXi hosts ? - in this moment I have access because I have root access.

I want to ensure because its not usual steps for me. Thank you for support.

[Lalegre]

Hey @Przemko1,

So wait, you are saying in point 3 that you have root access to your ESXi which was the issue of this blog. Do you have access or not?

You can do backup of your VMs by using Veeam Windows and Linux agent and that will copy the data from inside the Guest OS. After achieving that you can restore to vSphere in case needed as shown here: https://helpcenter.veeam.com/docs/agentforwindows/userguide/integration_instant_restore_vsphere.html...

[e_espinel]

Hello.
If you do not have the ESXi Hosts root password the only option recommended by VMware is to reinstall.
Since you also don't have the ESXi Hosts configuration details you will have to reconfigure everything from the beginning.

The reinstallation with the option: Install ESXi, preserver VMFS Datastore should not touch you data (VMs) only the VMware vSphere as operating system. But you have to be prepared for everything even to use their VMs backup.

This is an unauthorized option that I found on Google with which you could try to remove the password on the ESXi Host, but if something goes wrong the ESXi may not turn on and you would have to reinstall. This option is under your responsibility and risk.

https://www.top-password.com/knowledge/reset-esxi-root-password.html

Once you turn off the VMs you will not be able to turn them on if you do not have access (root password) to the ESXi Host

[karols47]

hey, we have signicifient turn in situation:  I ordered suspected failed card designed for this server model and after changing it on motherboard my physical server suddenly SEEs drives, blink LEDs on front panel and can boot !!! - now its online and Im waiting for job backup this evening :))))

So I started this Windows and backed up my Veeam configuration and also made screens from vCS which started 🙂 Next step is plan maintenance  break on weekend, shut down all VMs via vCS or directly via RDP, move them offline ( I dont have VMotion lic) from #1 host to #2 and reinstall host #1 from scratch. Then same opeartion but from #2 host move all VMs to #1 host and reinstall #2 host. After these operation I should have access to root account because will create NEW ONE ant set it in stone, write and store in safe, can SSH into these hosts and are ready to migrate whole environment to new generation of servers. Any caveats in above plan ? My hosts can not run all VMs running at once (not enough resources) but as I understand there is NOT any limit how many VMs can I move offline to 1 host (to reinstall another ESXi host in that time) Is it correct ? And hypothetical question: as I read when root passwords is in my hands then I can power up  VMs even without GUI (without vCS) using cli commands directly from ESXi console ?

[Lalegre]

Hey @karols47,

Yes your procedure is correct and of course you can move the VMs powered-off and they will not consume CPU and RAM resources till you power them on so during the whole procedure you can do that.

However if you have vCenter and you have Enterprise Plus licenses you can use the Host Profiles to restart the root password without the need of re-installing the ESXi again and do not even need to reboot the host: https://kb.vmware.com/s/article/68079

Regarding your question, is it possible to power-on the virtual machines from the ESXi by invoking the next commands:

1. vim-cmd vmsvc/getallvms (Record the ID of the VM you want to power on)
2. vim-scmd vmsvc/power.on vm_id

For your reference: https://kb.vmware.com/s/article/1038043

These commands needs to be run on the ESXi where the VMs are located as it is no cross-host.