VMware Cloud Community
ciuly
Contributor
Contributor
Jump to solution

Cannot reach ESXi through LAN, but it's up and running just fine

No idea what happened, if I've done something or it just melted or what not.

I have this ESXi instance v 6.0.0 which I've set up years ago and is running fine (I've done a few updates from the web client but the last one was quite some time ago). I remember some weeks ago I tried running the update from UI but I can't remember why it was failing (the download I think; it was in the early stages as I would start it then operate on some VMs, so most likely the download).

All VMs power up when started as expected, they operate fine, they are accessible all is good.

I wanted to change something on a VM the other day and vsphere client failed to connect. Tried ssh, same. Http client, same. They all used to work. Again, LAN is operating fine. Ping replies.

nmap however shows no open ports.

I know for sure I didn't fiddle with the firewall on it.

esxcli network firewall ruleset allowedip list

shows ALL for every rule.

Obviously tried to reboot solution, no game.

Any ideas, clues what to check, what to try? Getting tired of managing the VMs from command line.

This is a single/standalone esxi 6.0 free install with a bunch of VMs, no fancy stuff. Well, there is vmdirectpath passthrough if it makes a difference.

Thanks

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Tags (2)
Reply
0 Kudos
1 Solution

Accepted Solutions
a_p_
Leadership
Leadership
Jump to solution

Hypervisor2: 6.0.0-2.34.3620759

Did you try this one? It's likely the state before you installed the latest patches, which I understand went wrong.

There are several log files which may contain helpful hints. I'd probably start with the vmkernel.log, and the hostd.log files.


André

View solution in original post

Reply
0 Kudos
12 Replies
a_p_
Leadership
Leadership
Jump to solution

Without any details, it's quite hard to help.

Anyway, two option that come in mind are:

  • reboot the ESXi host, and see whether it offers "Shift-R" to roll back to a previous build in the boot screen
  • reinstall the ESXi host selecting the "Install, preserve VMFS datasatore" option

The second option will require some configuration, but should result in a clean installation.

André

Reply
0 Kudos
ciuly
Contributor
Contributor
Jump to solution

What kind of details would you need?

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

You mentioned that you cannot remember the error that happened, so it's basically only the log files which may help finding out twhat's wrong.

Anyway, one of the options that I posted earlier should also help.

André

Reply
0 Kudos
ciuly
Contributor
Contributor
Jump to solution

The SHIFT+R option shows 2 entries

Hypervisor1: 6.0.0-2.34.3620759 (default)

Hypervisor2: 6.0.0-2.34.3620759

they both seem to be the same version. So I guess nothing to rollback.

Before going the reinstall path, which will require some setup to get right, including the passthrough which I know I had a lot of trouble with, I'd like to explore other options.

You mentioned logs. Can you detail which ones and what to look for?

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Hypervisor2: 6.0.0-2.34.3620759

Did you try this one? It's likely the state before you installed the latest patches, which I understand went wrong.

There are several log files which may contain helpful hints. I'd probably start with the vmkernel.log, and the hostd.log files.


André

Reply
0 Kudos
ciuly
Contributor
Contributor
Jump to solution

Ok, I will try that tonight. I suppose if I do that the logs will be lost. Any idea what files I need to save to have those handy? So I can back them up before I do the restore.

Thank you.

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Reply
0 Kudos
rschmid
Enthusiast
Enthusiast
Jump to solution

are all vSwitches configured, host interfaces up and configured well?

VLAN ? GW ?

kind regards,

Roland

Reply
0 Kudos
ciuly
Contributor
Contributor
Jump to solution

I suppose so, as I could RDP to each VM just fine. And all network services on each VM were accessible. The only issue was connecting to esxi itself.

I rolled back now and it's fixed.

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Reply
0 Kudos
ciuly
Contributor
Contributor
Jump to solution

I scrolled all the logs from the esxi UI/schell but nothing popped up to me. Of course I didn't know what to look for so I was just hoping to see some errors or rejected connections or something.

Eventually I used the rollback as you suggested and that cured it. Trouble next was that 1 of the storages got missing. So the VMs on it were no longer accessible. I manually added that back but for some reason there are still 2 VMs grayed out and in unknown status, saying vmx inaccessible. I am looking at the files on the volume though and they seem fine. Access rights and everything look good.

Not sure on policies on this forum: should I create a new question for this issue?

Thanks

LE: I see some of the VMs from the previously existing datastore were no longer in the inventory either. They may have been added after the restore point was created? Just guessing.

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Not sure on policies on this forum: should I create a new question for this issue?

There are no strict polices which require a new discussion. However, creating  dedicated discussions for different issues may help other users with similar issues.

Trouble next was that 1 of the storages got missing. So the VMs on it were no longer accessible. I manually added that back but for some reason there are still 2 VMs grayed out and in unknown status, saying vmx inaccessible.

What type of storage is it (Local, NAS, SAN)? And how did you add it back (mount, resignature)?

Are other VMs - located on the same datastore - accessible?

Internally, ESXi uses a VMFS datastore's UUID, rather than its friendly name. Resinaturing a datastore will assign a new UUID, which makes it necessary, to remove the unknown VMs from the inventory, and add them again by e.g. right clicking their configuration (.vmx) file in the datastore browser.


André

Reply
0 Kudos
ciuly
Contributor
Contributor
Jump to solution

>>What type of storage is it (Local, NAS, SAN)?

Local

>> And how did you add it back (mount, resignature)?

Just the "Add Storage..." link/button from the configuration\Storage (I also did a resscan but that didn't help any)

>> Are other VMs - located on the same datastore - accessible?

Well, as soon as I added the storage, the VMs on it were seen and recognized.

I see now the 2 grayed out VMs are from the first storage that was loaded up "fine" including the other VMs on it. I will do a reboot in the evening and see what happens.

Thanks

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Reply
0 Kudos
ciuly
Contributor
Contributor
Jump to solution

The reboot didn't help. I double checked and both VMs are from 2 other storages that were up just fine after the recovery/restore with other VMs on them showing up just fine.

So the VM shows up in the vsphere client like
/vmfs/volumes/<GUID>/folder/file.vmx (inaccessible)

the exact path (checked letter by letter) is accessible via the esxi ssh shell. Access on the vmx file is -rwxr-xr-x, and matches other visible/accessible VMs on the same storage.

I can do "less filename" on the vmx and see it ok. Doesn't look to be damaged/corrupted (compared the format/output with other (running) vmx file)

Without removing the inaccessible VM, I have added it to the inventory again. It added just fine. Powered up fine as well. So I went ahead and removed the inaccessible "link".

Gremlins, I tell you.

VMDirectPath with ATI GPU document https://docs.google.com/spreadsheet/ccc?key=0Aqp_xYBwP_Y7dE5EclhtaDdIV09lNWxfODd1alRUTlE
Reply
0 Kudos