13 Replies Latest reply on Oct 24, 2013 6:10 AM by geppi

    vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?

    pkam Novice

      Hi,

       

      I have several ESXi 5.0 boxes. I use virtualized openindiana on them. I always install vmware-tools.

      When ESXi 5.1 came out, I upgraded one of my boxes with 5.1 and vmware-tools inside it (from 8.5 to 9.0).

      I was not happy, as OI machine always crashed on shutdown or reboot - either issued by shutdown command or by vmtools.

      After some time, I reverted back to 5.0u1.

      Yesterday I upgraded the same box to ESXi 5.0u2 and vmware-tools to 8.6. I got the same behaviour with OI vm.

      I tried the same on completely different box - HP ML350G6. Result was exactly the same - kernel crash on every reboot/shutdown.

      I believe something is terribly wrong and OI151a7 and vmware-tools 8.6 or 9.0 do not work properly.

       

      Is it working properly for anyone?

      Regards,

      Piotr

        • 1. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
          a_nut_in Expert

          Hi Piotr,

           

          The VMware HCL for Host/Guest OS does not list OpenIndiana as a supported guest operating systems

           

          http://www.vmware.com/go/hcl

           

          VMware Tools contain drivers that are specific to tested builds of OS's and as such, may have issues with the build you are trying to install. Have you tried contacing OpenIndiana and checking supportability of the Guest with the VMware tools for 5.x builds?

           

          Regards

          • 2. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
            geppi Novice

            It works for me under ESXi 5.1 with OI151a7 desktop edition in a Solaris 10/11 64-bit VM.

            • 3. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
              pkam Novice

              That is interesting!

              Could you please tell

              • was it clean install or upgrade from ESXi 5.0
              • can you verify vmware-tools version?
              • how did you install vmware-tools?

               

              Thanks inadvance.

              • 4. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                geppi Novice

                It was a clean install and the tools version installed is 9.0.0.15210 (build-782409).

                 

                I did invoke the installation from the VM/Guest menu in the console view of the OI VM.

                That did automatically mount the tools CD and I extracted the archive to a local folder on the VM.

                As root I changed into the vmware-tools-distrib directory and issued the command "perl ./vmware-install.pl".

                I answered all questions with the default.

                 

                Are you using the desktop or the server version of OI ?

                I've seen several posts stating that there seem to be issues with the server version (text only, no GUI) of OI and VMware-Tools.

                Again, I'm using the desktop distribution that includes the Xserver and therefore all the GUI stuff.

                1 person found this helpful
                • 5. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                  pkam Novice

                  Thanks for your detailed explanation.

                  Indeed, I am having problems with text-only, server-version of OI.

                  I will try to find some time to reinstall ESXi 5.1 and then install both OI versions (server and gui) as separate VMs.

                   

                  Anyway, I am surprised to hear GUI version working OK and text version having problems. I would assume otherwise if I were to guess.

                  Anyone willing to explain that?

                  I will post my findings when I am ready,

                  regards

                  Piotr

                  • 6. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                    pkam Novice

                    I need to ask you one more question.

                    I have installed OI 151a7 with defaults and vmware-tools with default options on my latest 5.0 updated (VMkernel esxi 5.0.0 #1 SMP Release build-914586 Nov 18 2012 11:14:47 x86_64 unknown) ESXi host.

                     

                    And it is the same wrong - it does crash on vmware-tools issued reboot or shutdown.

                    Please note it seems to be working OK, as vmware-tools install correctly, vmware-tools work and you can see tools installed from vSphere Client. When you use GUI, you will not see any difference on reboot wheter OI has shutdown correctly or it has crashed and has been reset. After reboot, it is hardly visible on the text console (before X starts) that it claims sth about coredump crash file etc.

                     

                    So: Are you sure it works properly? Are you able to shutdown the machine? Could you please test it observing console of the OI machine from vSphere Client?

                     

                    Thanks in advance.

                    Piotr

                    • 7. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                      geppi Novice

                      After I hit the stop button for the Openindiana VM this is what I get on the text console:

                       

                      svc.startd: The system is coming down.  Please wait.

                      svc.startd: 105 system services are now being stopped.

                      Jan  8 10:54:51 Openindiana syslogd: going down on signal 15

                      svc.startd: Killing user processes.

                      Jan  8 10:54:58 The system is down.   Shutdown took 18 seconds.

                      syncing file systems... done

                       

                      This looks like a proper shutdown to me.

                       

                      However, I further investigated /var/adm/messages and indeed found the following lines:

                       

                      savecore: [ID 570001 auth.error] reboot after panic: BAD TRAP: type=e (#pf Page fault) rp=ffffff00102bf9b0 addr=ffffffffc037f198

                      savecore: [ID 367237 auth.error] Panic crashdump pending on dump device but dumpadm -n in effect; run savecore(1M) manually to extract. Image UUID 3c483701-280d-eaea-f95e-debac7800450.

                       

                      OK, that means that there is a crashdump on my dump device but due to my dump configuration it is not automatically saved into a vmdump file when the system is powered up. You can check the dump configuration with:

                       

                      >> dumpadm

                       

                      which in my case delivered:


                            Dump content: kernel pages
                             Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
                      Savecore directory: /var/crash/<hostname>
                        Savecore enabled: no
                         Save compressed: on

                       

                      Reconfigure to run savecore on startup:

                       

                      >> dumpadm -y

                       

                      Now I rebooted the system and found the following lines in /var/adm/messages:

                       

                      savecore: [ID 570001 auth.error] reboot after panic: BAD TRAP: type=e (#pf Page fault) rp=ffffff00102bf9b0 addr=ffffffffc037f198

                      savecore: [ID 136035 auth.error] Saving compressed system crash dump in /var/crash/<hostname>/vmdump.0

                      savecore: [ID 160137 auth.error] Decompress the crash dump with 'savecore -vf /var/crash/<hostname>/vmdump.0'

                       

                      Indeed there was a vmdump.0 file now in /var/crash/<hostname> (you have to substitute <hostname> with the appropriate value).

                       

                      >> savecore -vf /var/crash/<hostname>/vmdump.0

                       

                      gave me the following output:

                       

                      savecore: System dump time: Sat Jan  5 18:31:03 2013

                       

                      Ahh, this was a dump from 3 days ago !

                       

                      Now that this old dump was processed by "savecore" and the dumpdevice has been "cleaned" I don't see any more messages about crashdumps in /var/adm/messages after powering off the VM and restarting.

                       

                       

                      Is this similar to your problem ?

                      • 8. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                        pkam Novice

                        Again thank you for the anwer.

                        No, my problem is not similar.

                        I have gone thru your dumpadm and savecore steps, but they produce some files I cannot understand.

                         

                        What I see in /var/adm/messages - when I try to do "VM shutdown" or "VM reboot" with vmware-tools is:

                         

                         

                        Jan 10 12:13:31 oi-gui savecore: [ID 570001 auth.error] reboot after panic: mutex_enter: bad mutex, lp=ffffffffc037dd38 owner=ffffff0153d2a7c0 thread=ffffff0004c13c40
                        Jan 10 12:19:33 oi-gui vmmemctl: [ID 978362 kern.info] VMware memory control driver unloaded
                        Jan 10 12:19:33 oi-gui unix: [ID 836849 kern.notice]
                        Jan 10 12:19:33 oi-gui ^Mpanic[cpu0]/thread=ffffff0004ccbc40:
                        Jan 10 12:19:33 oi-gui genunix: [ID 103648 kern.notice] mutex_enter: bad mutex, lp=ffffffffc037dd38 owner=ffffff0152a14480 thread=ffffff0004ccbc40
                        Jan 10 12:19:33 oi-gui unix: [ID 100000 kern.notice]
                        Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccba40 unix:mutex_panic+73 ()
                        Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbaa0 unix:mutex_vector_enter+446 ()
                        Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbb10 genunix:cv_timedwait_hires+fd ()
                        Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbba0 genunix:cv_timedwait_sig_hires+336 ()
                        Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbbe0 genunix:cv_timedwait_sig+4c ()
                        Jan 10 12:19:33 oi-gui genunix: [ID 802836 kern.notice] ffffff0004ccbc20 fffffffff819f9c5 ()
                        Jan 10 12:19:33 oi-gui genunix: [ID 655072 kern.notice] ffffff0004ccbc30 unix:thread_start+8 ()
                        Jan 10 12:19:33 oi-gui unix: [ID 100000 kern.notice]
                        Jan 10 12:19:33 oi-gui genunix: [ID 672855 kern.notice] syncing file systems...
                        Jan 10 12:19:33 oi-gui genunix: [ID 904073 kern.notice]  done
                        Jan 10 12:19:34 oi-gui genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
                        Jan 10 12:19:45 oi-gui genunix: [ID 100000 kern.notice]
                        Jan 10 12:19:45 oi-gui genunix: [ID 665016 kern.notice] ^M100% done: 89735 pages dumped,
                        Jan 10 12:19:45 oi-gui genunix: [ID 851671 kern.notice] dump succeeded

                         

                         

                        So, I press "VM shutdown, then we have the above - my oi generates this dump, and restarts.

                        It happens always on my oi-text and oi-gui machines under ESXi 5.0u2 and 5.1.

                        Why...?

                        • 9. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                          geppi Novice

                          Sorry, I have no idea.

                          • 10. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                            Druber Lurker

                            For what it's worth, it is not an OI-specific bug. I just tried with an omnios (illumos kernel, just like OI) install and same exact behavior. e.g. either a shutdown or 'init 5' panics.

                            • 11. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                              dd3 Novice

                              You can reproduce this bug on esxi 5.5 also. Configure new vm as Solaris 10 (64 bit), give it only 1 vcpu and install OmniOs and vmtools. After shutdown quest (or reboot quest) quest kernel panics. More info also in https://www.illumos.org/issues/3326

                               

                              When the has 2vcpus configured shutdown and reboot are working correctly.

                               

                              Message was edited by: dd3 - added 2vcpu

                              • 12. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                                dd3 Novice

                                Workaround for shutdown/reboot bug is editing poweroff-vm-default script (in /etc/vmware-tools) and delete all lines plus add poweroff command. Esxi logs an error after quest shutdown but its working.

                                • 13. Re: vmware-tools and openindiana - broken in 5.0u2 and 5.1 ?
                                  geppi Novice

                                  dd3 wrote:

                                  >> When the has 2vcpus configured shutdown and reboot are working correctly. <<

                                   

                                  That might be the explanation why I don't see this problem. My OmniOS virtualized server has 2 vCPUs.