supremedalek
Contributor
Contributor

Yet another "Failed - Transport (VMDB) error -45" issue after updating esxi host. No (ramdisk) issues

Like others (Transport (VMDB) error -45: Failed to connect to peer process , Re: " Failed - Transport (VMDB) error -45: Failed to connect to peer process. " after reinstall , Transport (VMDB) error -45: Failed to connect to peer process ), I have upgraded a esxi host from 6.0 to 6.5 and now cannot run any guest.

[root@vmhost:~] vim-cmd  vmsvc/power.on 30 Powering on VM: Power on failed [root@vmhost:~]

Here is the relevant excerpt from the log file:

018-06-22T14:50:42.910Z info hostd[E5C2B70] [Originator@6876 sub=vm:VigorExe 
cVMXExCommon: Exec()'ing /bin/vmx
/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/RHCE/RHCE.vmx
opID=vim-cmd-03-1411 user=root]
2018-06-22T14:50:42.914Z info hostd[E5C2B70] [Originator@6876 sub=Libs opID=v
im-cmd-03-1411 user=root] Vigor: VMKernel_ForkExec(/bin/vmx, detached=1): sta tus=0 pid=75385
2018-06-22T14:50:42.916Z info hostd[C2F8B70] [Originator@6876 sub=Libs] SOCKE T 20 (37)
2018-06-22T14:50:42.916Z info hostd[C2F8B70] [Originator@6876 sub=Libs] recv detected
client closed connection
2018-06-22T14:50:42.916Z info hostd[C2F8B70] [Originator@6876 sub=Libs] Vigor
TransportClientProcessError: Remote disconnected
2018-06-22T14:50:42.916Z info hostd[C2F8B70] [Originator@6876
sub=vm:/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/RHCE/RHCE.vmx] 
VMX did not report err via stderr
2018-06-22T14:50:42.918Z info hostd[E9AAB70] [Originator@6876 sub=Hostsvc]
Decremented SIOC Injector Flag2
2018-06-22T14:50:42.932Z warning hostd[E9AAB70] [Originator@6876
sub=Vmsvc.vm:/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/RHCE/RHCE.vmx]
Failed operation
2018-06-22T14:50:42.932Z verbose hostd[E9AAB70] [Originator@6876 sub=Property
Provider] RecordOp ASSIGN: latestEvent, ha-eventmgr. Applied change to temp m ap. 
# The error message
2018-06-22T14:50:42.932Z info hostd[E9AAB70] [Originator@6876 sub=Vimsvc.ha-eventmgr]
Event 244 : Cannot power on RHCE on vmhost.domain.com. in ha-datacenter. A
general system error occurred:
2018-06-22T14:50:42.932Z info hostd[E9AAB70] [Originator@6876 sub=Vmsvc.vm:/v
mfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/RHCE/RHCE.vmx] State Transition
(VM_STATE_POWERING_ON -> VM_STATE_OFF)

2018-06-22T14:50:42.932Z verbose hostd[E9AAB70] [Originator@6876 sub=Property

Provider] RecordOp ASSIGN: disabledMethod, 30. Sent notification immediately.

2018-06-22T14:50:42.933Z info hostd[E9AAB70] [Originator@6876 sub=Vimsvc.Task

Manager] Task Completed : haTask-30-vim.VirtualMachine.powerOn-189254357 Stat

us error

Following the replies to the threads mentioned above I checked if I was running out of disk space:

[root@vmhost:~] df -h 
Filesystem   Size   Used Available Use% Mounted on
NFS        147.6G 110.7G     36.9G  75% /vmfs/volumes/public
VMFS-5     414.2G 288.5G    125.7G  70% /vmfs/volumes/datastore1
VMFS-5      39.8G  10.0G     29.8G  25% /vmfs/volumes/Test
vfat       285.8M 208.1M     77.7M  73% /vmfs/volumes/52a08b49-9fa64986-10bf-d067e51ce7b7
vfat       249.7M 162.4M     87.3M  65% /vmfs/volumes/0e7ac47f-85c20205-e914-af5132fe5673
vfat       249.7M 150.4M     99.3M  60% /vmfs/volumes/01cdaf72-2d3f8757-77fe-274ef8565255
vfat         4.0G 110.3M      3.9G   3% /vmfs/volumes/56306a5f-4068abd1-ddcb-00224d98ad4f
[root@vmhost:~]

What about ramdisk? Even though there is no mention in the log file (as suggested in VMware Knowledge Base​ ) that the ramdisk is full,  I still ran vdf -h to see the ramdisk:

----- 
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        1M       30M   6% --
etc                        28M      960K       27M   3% --
opt                        32M        0B       32M   0% --
var                        48M      412K       47M   0% --
tmp                       256M       16K      255M   0% --
iofilters                  32M        0B       32M   0% --
hostdstats                204M        2M      201M   1% --
snmptraps                   1M        0B        1M   0% --

What about running out of inodes? Per  I ran

[root@vmhost:~]

stat -f /vmfs/volumes/52a08b49-9fa64986-10bf-d067e51ce7b7

  File: "/vmfs/volumes/52a08b49-9fa64986-10bf-d067e51ce7b7"

    ID: 236a5549191e0c58 Namelen: 127     Type: vfat

Block size: 8192

Blocks: Total: 36586      Free: 9943       Available: 9943

Inodes: Total: 0          Free: 0

[root@vmhost:~]

But honestly do not know if Inodes: Total: 0          Free: 0 means out of inodes. What else should I be looking for?

I di dread in another thread about just reinstalling esxi back, but that does not explain why this is happening to begin with.

0 Kudos
9 Replies
SupreetK
Commander
Commander

Hello,

Do you have only one host? If not, are you seeing the same issue on all the hosts? Did you check the vmkernel.log for any errors with the storage?

To start of with, create a new VM and attach the existing disks from the VM. Then attempt to power it on.

If it still fails, share the hostd.log, vmkernel.log and vmware.log as soon as the power on fails.

Cheers,

Supreet

0 Kudos
supremedalek
Contributor
Contributor

Do you have only one host? If not, are you seeing the same issue on all the hosts?

Only one host

Did you check the vmkernel.log for any errors with the storage?

Yep. Nothing besides what I have mentioned already.

To start of with, create a new VM and attach the existing disks from the VM. Then attempt to power it on.

If it still fails, share the hostd.log, vmkernel.log and vmware.log as soon as the power on fails.

Ubuntu, default everything else. vm guest created without a problem using gui; I was even able to setup the proper vlan. When I started it up, it crashed the same way as the existing ones. vmkernel.log had no entries. I do not seem to have a vmware.log file.

0 Kudos
daphnissov
Immortal
Immortal

Attach the log files for this VM, please, found in its home directory. Also, please share the VMX file.

0 Kudos
supremedalek
Contributor
Contributor

daphnissov, I cannot attach any log files because none are available:

[root@vmhost:/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/testubuntu] ls 
testubuntu-flat.vmdk  testubuntu.vmsd
testubuntu.vmdk       testubuntu.vmx
[root@vmhost:/vmfs/volumes/52a08b50-984b4bf0-219f-d067e51ce7b7/testubuntu]

However, I can attach testubuntu.vmx

0 Kudos
SupreetK
Commander
Commander

Checked the attached vmx file and there is no entry that explicitly disables VM logging. Not sure why there are no vmware.log files :smileyplain:

Cheers,

Supreet

0 Kudos
supremedalek
Contributor
Contributor

Am I wrong to assume that if the guest startup process crashes before the guest has an opportunity to start up, there would be no log?

On a related note, this is behaving too much like https://www.virtualmvp.com/vsphere-6-5-transport-vmdb-error-45-failed-to-connect-to-peer-process/  and that required a rather Microsoft-style solution.

0 Kudos
SupreetK
Commander
Commander

Can you follow the below steps and share the log files? I'm assuming you would be performing these steps on the testubuntu VM.

1) Enable verbose logging for hostd - VMware Knowledge Base

2) Power on the VM directly from the host (not from the vCenter).

3) Once it fails to power on, share the hostd.log and vmkernel.log files.

Cheers,

Supreet

0 Kudos
supremedalek
Contributor
Contributor

, I take you have not read the original post in this thread.

But, as asked, here are the files.

0 Kudos
SupreetK
Commander
Commander

Below error has been reported in the vmkernel.log -->

2018-09-15T01:00:18.983Z cpu4:417094)WARNING: User: 4530: vmx: Error in initial cartel setup: Failed to open /bin/vmx: Operation not permitted

Going by it, looks like the MacOS Unlocker is blocking the powering on of any VMs. Uninstalling it should address this. Refer to the thread - upgrade esxi from 6.0 to 6.5,VM can't power on​.

Also, search for the above event in the article - https://www.virten.net/2016/11/homelab-will-esxi-6-5-run-on-intel-nuc/​. Several users are talking about the same error in the comments section.

Cheers,

Supreet

0 Kudos