pkam
Contributor
Contributor

vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?

Hi,

I have several ESXi 5.0 boxes. I use virtualized openindiana on them. I always install vmware-tools.

When ESXi 5.1 came out, I upgraded one of my boxes with 5.1 and vmware-tools inside it (from 8.5 to 9.0).

I was not happy, as OI machine always crashed on shutdown or reboot - either issued by shutdown command or by vmtools.

After some time, I reverted back to 5.0u1.

Yesterday I upgraded the same box to ESXi 5.0u2 and vmware-tools to 8.6. I got the same behaviour with OI vm.

I tried the same on completely different box - HP ML350G6. Result was exactly the same - kernel crash on every reboot/shutdown.

I believe something is terribly wrong and OI151a7 and vmware-tools 8.6 or 9.0 do not work properly.

Is it working properly for anyone?

Regards,

Piotr

0 Kudos
13 Replies
a_nut_in
Expert
Expert

Hi Piotr,

The VMware HCL for Host/Guest OS does not list OpenIndiana as a supported guest operating systems

http://www.vmware.com/go/hcl

VMware Tools contain drivers that are specific to tested builds of OS's and as such, may have issues with the build you are trying to install. Have you tried contacing OpenIndiana and checking supportability of the Guest with the VMware tools for 5.x builds?

Regards

Do remember to mark my post as "helpful" or "correct" if I've helped resolve or answer your query!
0 Kudos
geppi
Contributor
Contributor

It works for me under ESXi 5.1 with OI151a7 desktop edition in a Solaris 10/11 64-bit VM.

0 Kudos
pkam
Contributor
Contributor

That is interesting!

Could you please tell

  • was it clean install or upgrade from ESXi 5.0
  • can you verify vmware-tools version?
  • how did you install vmware-tools?

Thanks inadvance.

0 Kudos
geppi
Contributor
Contributor

It was a clean install and the tools version installed is 9.0.0.15210 (build-782409).

I did invoke the installation from the VM/Guest menu in the console view of the OI VM.

That did automatically mount the tools CD and I extracted the archive to a local folder on the VM.

As root I changed into the vmware-tools-distrib directory and issued the command "perl ./vmware-install.pl".

I answered all questions with the default.

Are you using the desktop or the server version of OI ?

I've seen several posts stating that there seem to be issues with the server version (text only, no GUI) of OI and VMware-Tools.

Again, I'm using the desktop distribution that includes the Xserver and therefore all the GUI stuff.

pkam
Contributor
Contributor

Thanks for your detailed explanation.

Indeed, I am having problems with text-only, server-version of OI.

I will try to find some time to reinstall ESXi 5.1 and then install both OI versions (server and gui) as separate VMs.

Anyway, I am surprised to hear GUI version working OK and text version having problems. I would assume otherwise if I were to guess.

Anyone willing to explain that?

I will post my findings when I am ready,

regards

Piotr

0 Kudos
pkam
Contributor
Contributor

I need to ask you one more question.

I have installed OI 151a7 with defaults and vmware-tools with default options on my latest 5.0 updated (VMkernel esxi 5.0.0 #1 SMP Release build-914586 Nov 18 2012 11:14:47 x86_64 unknown) ESXi host.

And it is the same wrong - it does crash on vmware-tools issued reboot or shutdown.

Please note it seems to be working OK, as vmware-tools install correctly, vmware-tools work and you can see tools installed from vSphere Client. When you use GUI, you will not see any difference on reboot wheter OI has shutdown correctly or it has crashed and has been reset. After reboot, it is hardly visible on the text console (before X starts) that it claims sth about coredump crash file etc.

So: Are you sure it works properly? Are you able to shutdown the machine? Could you please test it observing console of the OI machine from vSphere Client?

Thanks in advance.

Piotr

0 Kudos
geppi
Contributor
Contributor

After I hit the stop button for the Openindiana VM this is what I get on the text console:

svc.startd: The system is coming down.  Please wait.

svc.startd: 105 system services are now being stopped.

Jan  8 10:54:51 Openindiana syslogd: going down on signal 15

svc.startd: Killing user processes.

Jan  8 10:54:58 The system is down.   Shutdown took 18 seconds.

syncing file systems... done

This looks like a proper shutdown to me.

However, I further investigated /var/adm/messages and indeed found the following lines:

savecore: [ID 570001 auth.error] reboot after panic: BAD TRAP: type=e (#pf Page fault) rp=ffffff00102bf9b0 addr=ffffffffc037f198

savecore: [ID 367237 auth.error] Panic crashdump pending on dump device but dumpadm -n in effect; run savecore(1M) manually to extract. Image UUID 3c483701-280d-eaea-f95e-debac7800450.

OK, that means that there is a crashdump on my dump device but due to my dump configuration it is not automatically saved into a vmdump file when the system is powered up. You can check the dump configuration with:

>> dumpadm

which in my case delivered:


      Dump content: kernel pages
       Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/<hostname>
  Savecore enabled: no
   Save compressed: on

Reconfigure to run savecore on startup:

>> dumpadm -y

Now I rebooted the system and found the following lines in /var/adm/messages:

savecore: [ID 570001 auth.error] reboot after panic: BAD TRAP: type=e (#pf Page fault) rp=ffffff00102bf9b0 addr=ffffffffc037f198

savecore: [ID 136035 auth.error] Saving compressed system crash dump in /var/crash/<hostname>/vmdump.0

savecore: [ID 160137 auth.error] Decompress the crash dump with 'savecore -vf /var/crash/<hostname>/vmdump.0'

Indeed there was a vmdump.0 file now in /var/crash/<hostname> (you have to substitute <hostname> with the appropriate value).

>> savecore -vf /var/crash/<hostname>/vmdump.0

gave me the following output:

savecore: System dump time: Sat Jan  5 18:31:03 2013

Ahh, this was a dump from 3 days ago !

Now that this old dump was processed by "savecore" and the dumpdevice has been "cleaned" I don't see any more messages about crashdumps in /var/adm/messages after powering off the VM and restarting.

Is this similar to your problem ?

0 Kudos
pkam
Contributor
Contributor

Again thank you for the anwer.

No, my problem is not similar.

I have gone thru your dumpadm and savecore steps, but they produce some files I cannot understand.

What I see in /var/adm/messages - when I try to do "VM shutdown" or "VM reboot" with vmware-tools is:

Jan 10 12:13:31 oi-gui savecore: [ID 570001 auth.error] reboot after panic: mutex_enter: bad mutex, lp=ffffffffc037dd38 owner=ffffff0153d2a7c0 thread=ffffff0004c13c40
Jan 10 12:19:33 oi-gui vmmemctl: [ID 978362 kern.info] VMware memory control driver unloaded
Jan 10 12:19:33 oi-gui unix: [ID 836849 kern.notice]
Jan 10 12:19:33 oi-gui ^Mpanic[cpu0]/thread=ffffff0004ccbc40:
Jan 10 12:19:33 oi-gui genunix: [ID 103648 kern.notice] mutex_enter: bad mutex, lp=ffffffffc037dd38 owner=ffffff0152a14480 thread=ffffff0004ccbc40
Jan 10 12:19:33 oi-gui unix: [ID 100000 kern.notice]
Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccba40 unix:mutex_panic+73 ()
Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbaa0 unix:mutex_vector_enter+446 ()
Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbb10 genunix:cv_timedwait_hires+fd ()
Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbba0 genunix:cv_timedwait_sig_hires+336 ()
Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbbe0 genunix:cv_timedwait_sig+4c ()
Jan 10 12:19:33 oi-gui genunix: [ID 802836 kern.notice] ffffff0004ccbc20 fffffffff819f9c5 ()
Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbc30 unix:thread_start+8 ()
Jan 10 12:19:33 oi-gui unix: [ID 100000 kern.notice]
Jan 10 12:19:33 oi-gui genunix: [ID 672855 kern.notice] syncing file systems...
Jan 10 12:19:33 oi-gui genunix: [ID 904073 kern.notice]  done
Jan 10 12:19:34 oi-gui genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
Jan 10 12:19:45 oi-gui genunix: [ID 100000 kern.notice]
Jan 10 12:19:45 oi-gui genunix: [ID 665016 kern.notice] ^M100% done: 89735 pages dumped,
Jan 10 12:19:45 oi-gui genunix: [ID 851671 kern.notice] dump succeeded

So, I press "VM shutdown, then we have the above - my oi generates this dump, and restarts.

It happens always on my oi-text and oi-gui machines under ESXi 5.0u2 and 5.1.

Why...?

0 Kudos
geppi
Contributor
Contributor

Sorry, I have no idea.

0 Kudos
Druber
Contributor
Contributor

For what it's worth, it is not an OI-specific bug. I just tried with an omnios (illumos kernel, just like OI) install and same exact behavior. e.g. either a shutdown or 'init 5' panics.

0 Kudos
dd3
Contributor
Contributor

You can reproduce this bug on esxi 5.5 also. Configure new vm as Solaris 10 (64 bit), give it only 1 vcpu and install OmniOs and vmtools. After shutdown quest (or reboot quest) quest kernel panics. More info also in https://www.illumos.org/issues/3326

When the has 2vcpus configured shutdown and reboot are working correctly.

Message was edited by: dd3 - added 2vcpu

0 Kudos
dd3
Contributor
Contributor

Workaround for shutdown/reboot bug is editing poweroff-vm-default script (in /etc/vmware-tools) and delete all lines plus add poweroff command. Esxi logs an error after quest shutdown but its working.

0 Kudos
geppi
Contributor
Contributor

dd3 wrote:

>> When the has 2vcpus configured shutdown and reboot are working correctly. <<

That might be the explanation why I don't see this problem. My OmniOS virtualized server has 2 vCPUs.

0 Kudos