VMware Cloud Community
lkomarci
Contributor
Contributor
Jump to solution

blackout + lost vsphere + vcenter passwords

Hi everyone,

could use a little help here. I'm working in a new environment and we just had a blackout. The passwords and ip addresses are on a network drive which I can't access as the domain controllers are offline so I can't access the vcenter or vsphere to boot the virtual machines.

Whan can I do??

Thanks in advance! I'm in quite deep

Luke

Reply
0 Kudos
1 Solution

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

... and plug it directly into the back of the ESX ...

you may use the DCUI to find out the correct port(s). However, a direct connection may be tricky, especially if the ESXi host's Management network uses VLAN tagging, and if it has more that one vmnic. Anyway, assuming that the physical network (switches, routers, etc.) are up, and working correctly, and if the ESXi host can reach/ping its default gateway, you should be able connect to it over the network.

Another option is to enable ESXi shell access and power on the important systems (e.g. your DC/DNS/DHCP server) from the command line.

André

View solution in original post

Reply
0 Kudos
24 Replies
TheBobkin
Champion
Champion
Jump to solution

Hello Luke,

Are the DCs VMs or physical? Either way, getting them up so you can access everything else seems like the logical first step - if they DCs are VMs do you know what hosts they were registered on? You will require at least root credentials for that/a host to access either the Host Client or SSH to power on the VMs.

Bob

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

Hi Bob,

unfortunately the DCs are VMs, which is why none of the users can connect with their credentials. None of the PCs are getting IP addresses so naturally the DHCP is on one of those virtual DCs. I don't know the vcenter nor the ESX root passwords. Everything here was extremely well planned for disaster events - all IP addresses and credentials are stored on the network which I naturally can't access. I do know the ESX ip addresses though.

You have any ideas?

Cheers,

Luke

Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

Hello Luke,

"I don't know the vcenter nor the ESX root passwords."

This is the major hurdle here - if you had these it would just be a case of Host Client/SSH and powering stuff up - does anyone else in your organisation have these and/or might they be stored somewhere you can access?

If this is not possible then the only feasible way to change the root password of an ESXi host is to re-install ESXi - if you can re-install a host and configure access to the required datastores then you can register and power on the VMs required to access everything else.

"Everything here was extremely well planned for disaster events"

I wouldn't really be saying that at this point as you appear to be locked out of everything :smileyconfused:

Bob

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

Hello Bob,

the claim that of being very well prepared for disaster was a sarcasm, imagine my shock.

Just read VMware Knowledge Base  it's supposed to be possible to reset the ESX password but haven't ever done this - is it a safe procedure?

thanks,

L

Reply
0 Kudos
StephenMoll
Expert
Expert
Jump to solution

I imagine you are using a fairly recent version of ESXi, v6? From your linked KB:

ESXi 3.5, ESXi 4.x, ESXi 5.x and ESXi 6.x

Reinstalling the ESXi host is the only supported way to reset a password on ESXi.

I would think doing this without proper access to the network resources required would be very difficult. I think exhausting all avenues to finding someone with the required passwords would be the best approach at the moment.

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

Oh... damn. Missed that part.

But yes, it's the ESXi 6.5.0. Bad news

Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

Hello Luke,


Explore every angle while trying to work out whom/where root passwords may reside - even just one ESXi root with the necessary datastores attached should be sufficient. If not then start working out what you need to be able to re-install a host (or install/use a spare one you have on hand) and connect it to the necessary storage e.g. are these iSCSI or FC which will require zoning or NFS which will require permissions. You mentioned "- all IP addresses and credentials are stored on the network which I naturally can't access"  - Do you mean that everything is stored somewhere you cannot access over the network (and thus inaccessible with no network)? Or is this stored on a site that you can physically access?

Bob

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

all IP addresses and credentials are stored on the network

Just a thought. Are you able to access the backup? Maybe you can restore the documents with the credentials from there!?

André

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

Hi André,

not sure how to be exact. Network resourceaccess is managed by AD/group policy rules and considering that the DCs are all offline I'm not sure how to be exact.

Another thing that's bugging me is reinstalling the EXSi. Normally for reinstallation I'd use vsphere - insert the ESX OS image into the virtual drive and then upon reboot I'd be able to install it. But considering that I can't access vsphere or anything else for that matter

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

K guys,

I have the ESX root password!

Now, I can't ping the ESX so I could try to connect via vsphere. I tried giving my PC an IP address from the ESX's subnet but won't go. What can I do?

cheers!

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

I tried giving my PC an IP address from the ESX's subnet but won't go. What can I do?

Can you confirm that your PC is connected to a physical port in the proper VLAN?

Are you able to ping out (e.g. the gateway address) from the host's DCUI (console)?

André

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

I'm thinking about putting a laptop on the ESX's subnet and plug it directly into the back of the ESX to figure out which one is the management port but all of the ports are taken. Two ports are on fiber and two on patch

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

... and plug it directly into the back of the ESX ...

you may use the DCUI to find out the correct port(s). However, a direct connection may be tricky, especially if the ESXi host's Management network uses VLAN tagging, and if it has more that one vmnic. Anyway, assuming that the physical network (switches, routers, etc.) are up, and working correctly, and if the ESXi host can reach/ping its default gateway, you should be able connect to it over the network.

Another option is to enable ESXi shell access and power on the important systems (e.g. your DC/DNS/DHCP server) from the command line.

André

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

DCUI reached, I'll try to fire up the VMs now.

thanks André I really appreciate your help!

Reply
0 Kudos
StephenMoll
Expert
Expert
Jump to solution

You know how to get to the console from the DCUI?

If not this might help : Using ESXi Shell in ESXi 5.x and 6.x (2004746)

Once in there:

vim-cmd vmsvc/getallvms to list all VMs registered to the host.

You need the VMID from this list of any VMs you want to start.

vim-cmd vmsvc/power.on {VMID}

To power on a VM.

With any luck you'll find the host with one of your DCs and that will allow you get everything back up more easily.

srwsol
Hot Shot
Hot Shot
Jump to solution

Wow.   This is a text book example of why one should never have anything that ESXi is dependent upon running as a virtual machine under ESXi.  

Reply
0 Kudos
StephenMoll
Expert
Expert
Jump to solution

Let me correct that for you...

"This is a text book example of why one should never have anything that ESXi is dependent upon running as a virtual machine under ESXi without a clearly defined process for it running when things go wrong."

Smiley Wink

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

Hi Stephen,

yea I managed to find those commands. This is exactly what I've used.

For now as a temporary solution I've set the DC VMs to start automatically.

Cheers,

Luka

Reply
0 Kudos
lkomarci
Contributor
Contributor
Jump to solution

I couldn't agree more with you guys.

I'm going to think about setting the core switch as a DHCP server for certain VLANs - in case if it happens again everything should be reachable without major complications.

Reply
0 Kudos