Strange problems on ESXi 5.0.0 (build 469512)

zpnd · ‎04-11-2012

Hello,

I just installed ESXi on a server and there are many strange problems.

At first install I got boot image is corrupted error. So, I re-installed the system and now;

* I can't access to http://IP_ADDRESS/
* I can enable to SSH access but it's not permanent. It restored to disable state after reboot.

* I can't access to system via vSphere.

* I can't get prompt via CTRL+ALT+F1. It stucks just before prompt.

Are these normal ?

I can ping the ESXi server and it can ping outside. There is no firewall or port blocking between server and me.

What can you suggest ?

Thank you.

helltejas · ‎04-11-2012

Did you installed via CD, USB or by network using script??:smileyconfused:

JimKnopf99 · ‎04-11-2012

Is your Hardware on the HCL?

http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software&testConfig=16

Frank

If you find this information useful, please award points for "correct" or "helpful".

robpou · ‎04-12-2012

> * I can't access to http://IP_ADDRESS/

> * I can't access to system via vSphere.

These are network errors; you don't use bonding, or similar, do you? How many NICs are configured? Use just one.

Check routing and DNS settings, too.

> * I can't get prompt via CTRL+ALT+F1. It stucks just before prompt.

Troubleshooting Mode Options / Enable ESXi Shell - should work afterwards

> Are these normal ?

No They're probably network errors and after fixing the gateway/routing problems most of the problems will go away.

But if it still fails you could use the latest build (623860), but I don't think it'll be better.

zpnd · ‎04-12-2012

But I can ping the server and connect to it via SSH and there is no firewal, port blocking etc ... So, I doubt that because of network configuration.

Server from Hetzner. There is just one NIC and I don't know it is competible or now but it works. They also using mac binding and giving IP addresses via DHCP. I tried both of them, grabbing from DHCP and entering manually, the results are same. Machine connected to internet, it can ping, I can connect to machine via SSH but vCenter client can't.

I don't know this information will be useful but ; I tried to find open ports via esxcli on the server but it said "cannot connect to localhost".

nielse · ‎04-12-2012

Just to make sure: double check your gateway for any typo. Did you try restarting the network agents ?

@nielsengelen - http://foonet.be - VCP4/5

zpnd · ‎04-12-2012

Yes. A few times. I rebooted the server a few times also.

zpnd · ‎04-12-2012

I just figure out, I can't start hostd and vpxa services with "undefined symbol" error. Do you have any idea or is it related with my problem ?

Running vpxa restart

[74570] Begin '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swap,group=vpxa -D /etc/vmware/vpxa', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000, bg_pid_file = ''

/usr/lib/vmware/vpxa/bin/vpxa: symbol lookup error: /lib/libnfc-types.so: undefined symbol: _ZTv0_n48_N5Vmomi8StubImpl7_InvokeEPNS_13ManagedMethodERN7Vmacore9RefVectorINS_3AnyEEERNS3_3RefIS5_EE

helltejas · ‎04-12-2012

i also thinks that it is a network config. issue but if not then

from SSH console try services.sh restart command.

and attach hostd.log and vpxa.log files from ssh its in var/log/ dir

zpnd · ‎04-12-2012

Strangely, nothing.

/var/log # services.sh restart

Running sfcbd stop

This operation is not supported.

Please use /etc/init.d/sfcbd-watchdog stop

Running wsman stop

Stopping openwsmand

Running sfcbd-watchdog stop

Running usbarbitrator stop

watchdog-usbarbitrator: Terminating watchdog process with PID 74667

usbarbitrator stopped

Running vpxa stop

vpxa is not running

Running vobd stop

watchdog-vobd: Terminating watchdog process with PID 74534

vobd stopped

Running cdp stop

watchdog-cdp: Terminating watchdog process with PID 74510

Running dcbd stop

watchdog-dcbd: Terminating watchdog process with PID 74491

Running memscrubd stop

memscrubd is not running

Running slpd stop

Stopping slpd

Running hostd stop

hostd is not running.

Running sensord stop

sensord is not running

Running vprobed stop

watchdog-vprobed: Terminating watchdog process with PID 74260

vprobed stopped

Running lbtd stop

watchdog-net-lbt: Terminating watchdog process with PID 74241

net-lbt stopped

Running storageRM stop

watchdog-storageRM: Terminating watchdog process with PID 74216

storageRM stopped

Running DCUI stop

Disabling DCUI logins

VobUserLib_Init failed with -1

Running DCUI restart

Enabling DCUI login: runlevel =

VobUserLib_Init failed with -1

Running storageRM restart

storageRM started

Running lbtd restart

net-lbt started

Running vprobed restart

vprobed started

Running sensord restart

sensord started

Running hostd restart

[77154] Begin 'hostd ++min=0,swap,group=hostd /etc/vmware/hostd/config.xml', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000, bg_pid_file = ''

hostd: symbol lookup error: /lib/libvmwauthProxy-types.so: undefined symbol: _ZTv0_n48_N5Vmomi8StubImpl7_InvokeEPNS_13ManagedMethodERN7Vmacore9RefVectorINS_3AnyEEERNS3_3RefIS5_EE

Unable to verify hostd started after 10 seconds

hostd started.

Running slpd restart

Starting slpd

Running memscrubd restart

The checkPages boot option is FALSE, hence memscrubd could not be started.

Running dcbd restart

dcbd started

Running cdp restart

cdp started

Running vobd restart

vobd started

Running vpxa restart

[77319] Begin '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swap,group=vpxa -D /etc/vmware/vpxa', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000, bg_pid_file = ''

/usr/lib/vmware/vpxa/bin/vpxa: symbol lookup error: /lib/libnfc-types.so: undefined symbol: _ZTv0_n48_N5Vmomi8StubImpl7_InvokeEPNS_13ManagedMethodERN7Vmacore9RefVectorINS_3AnyEEERNS3_3RefIS5_EE

Unable to verify vpxa started after 10 seconds

Running usbarbitrator restart

usbarbitrator started

Running sfcbd-watchdog restart

Running wsman restart

Starting openwsmand

Running sfcbd restart

This operation is not supported.

Please use /etc/init.d/sfcbd-watchdog start

/var/log #

/var/log # cat hostd.log

/var/log # cat vpxa.log

/var/log #

helltejas · ‎04-12-2012

After restarting try to connect to host using vSphere client and then view log

hostd agent is used to connect vSphere Client and vpxa agent used to connect with vCenter so i think these both agents are corrupted

Try to restart host or if its not working then reinstall it again with good media

zpnd · ‎04-12-2012

Unfortunatelly I can't.

When I rebooted the server, I'm loosing SSH connection. I have to request and pay for KVM access and re-open the SSH access manually. Since it is not my server I can't do it even I willing to pay.

It is really annoying.

But I can say that, last night, after many reboots I can't connect to server via vSphere and it said continously "an unknown connection error occured. (The request failed because of a connection failure. ( Unable to connect to the remote server))

helltejas · ‎04-12-2012

i think reinstalletion is the only option bro..

nielse · ‎04-12-2012

Either that or collect the logs via SCP and open a case at VMware support 🙂

@nielsengelen - http://foonet.be - VCP4/5

zpnd · ‎04-12-2012

Thank you. I'll try both.

One last question.

Is Realtek Realtek 8168 Gigabit Ethernet compatible with ESXi ? I can't find anything on Compatibilty Guide.

JimKnopf99 · ‎04-12-2012

That's what I am talking about at the beginning....

I know that it looks like that I do not want to help, but that's what you have to check first.

If its not on the list, it's not supported and I guess your network issues related to that.

Frank

Am 12.04.2012 um 15:02 schrieb <communities-emailer@vmware.com<mailto:communities-emailer@vmware.com>>:

VMware Communities<http://communities.vmware.com/index.jspa>

Strange problems on ESXi 5.0.0 (build 469512)

reply from zpnd<http://communities.vmware.com/people/zpnd> in VMware ESXi 5 - View the full discussion<http://communities.vmware.com/message/2024913#2024913

If you find this information useful, please award points for "correct" or "helpful".

zpnd · ‎04-12-2012

This is really driving me crazy.

I'm trying to re-install ESXi but installer can't find any local storage. There are two disks as RAID 1. Some posts says that ESXi doesn't support RAID 1 but I've already install on this system ESXi twice. Why now ?

Also, I can boot old ESXi on this server. So, RAID 1 shouldn't be problem, right ?

helltejas · ‎04-12-2012

ESXi supports RAID 1. You should check your RAID controller in HCL...

Or remove RAID.. My suggestion is not to configure RAID 1 because it’s slow in write performance.

and ESXi is a very stable product hardly it crash hypervisor so remove RAID. and use SAN device to store VM

zpnd · ‎04-13-2012

After 24 hours debugging and installation marathon, finally I got an IO error and sent it to datacenter. I was told that in first two hours but ... words are really meaningless.

Thanks for your help and effort. I hope they will change the hardware and I'll be able to install a proper system.

--

Tim

xpulse · ‎06-28-2012

Ok.

I found root cause for issue with vpxa service and error "Begin '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swap,group=vpxa -D /etc/vmware/vpxa', min-uptime = 60, max-quick-failures = 1, max-total-failures = 1000000, bg_pid_file = ''
Unable to verify vpxa started after 10 seconds".

This happened because ram-disk is full 100%.

Check log under path /var/log/vmkernel.log

Cannot extend visorfs file /var/log/wtmp because its ramdisk (root) is full.

Resolution is:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=201902...

Symptoms

Large number of log in attempts to the ESXi host cause the /var/log/wtmp to grow to a large size and consume 100% of root ramdisk.
In the vobd.log. you see the following message:

vobd.log:2012-02-01T11:16:28.798Z: [VisorfsCorrelator] 4132132662503us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /var/log/wtmp because its ramdisk (root) is full.

Resolution

VMware is aware of this issue and is working to resolve this in a future release.

To workaround this issue:

Connect to the ESXi host directly using SSH. For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910).
Run these commands to remove and recreate wtmp:

rm /var/log/wtmp
touch /scratch/log/wtmp
ln -s /scratch/log/wtmp /var/log/wtmp
To ensure that the changes persist after rebooting the host:
1. Open the /etc/rc.local file using a text editor.
2. Add these entries to the file:
  
  rm /var/log/wtmp
  touch /scratch/log/wtmp
  ln -s /scratch/log/wtmp /var/log/wtmp

All