Workstation Installer Hang 6.5.3 and Workaround

[BitOBear]

I just encountered a hang installing Workstation on Ubuntu Karmic.

Symptom: you are running a an install using 6.5.3 bundle and you get to "Configuring VMWare Player" and then everything stops indefinitely.

Cause: the installer runs vmware-modconfig-console which produces enough output with some kernels that the python engine deadlocks is command piplining during the compile.

Technical Details: It is always dangerous for a linux/unix program to use the pipe-based "run a sub-command" library calls if it isn't ready to read all the output in one thread while waiting for the command to complete in another. Normally such commands should either be reading the result text or redirecting that result text to /dev/null. The naieve version just runs the command and waits for the result. But if the result text is larger than the system pipe buffer size, the command will block while trying to send its output, and the command never finishes. With the command waiting for the parent to read the data, and the parent waiting for the command to finish, the whole shebang just comes to a stop.

Why this is happening: The current driver patches for the 2.6.31 kernels produce a lot of warning messages about the symbols MSVCVER and __FREE_BSD__ being not defined. This never happened before, and whoever wrote the installer never tried it against a long stream of output. So with the new warning stream the compile and the installer deadlock as above.

Workaround: There are three phases to the work around; It is necessary to catch and kill the compile step; this, in turn would cause the installer to back out the installation, so you have to stop that from happening; then you have to compile the modules manually.

Steps:

1) open two command windows.

2) become root in one of those windows. i.e. "su -". (If you havn't set your root password in Ubuntu use "sudo su -" and your login password)

3) in the root-ed window run "while true; do killall -9 vmware-modconfig-console; sleep 1; done" this wil try, once each second, to kill the module compile step. Ignore the stream of not-found messages.

4) in the non-root window run the install with sudo and the --ignore-errors argument; i.e. "sudo ./VMware-Workstation-6.5.3-185404.i386.bundle --ignore-errors". This will take a little longer than usual because of the continuous killall above, don't sweat it.

5) once the installer finishes go back to the root window and do a control-C to stop the kill loop.

6) In the root window run "vmware-modconfig --console --install-all" to install the modules.

Caveats and Tips:

a) Once you have a hung installer session you have to kill the python command that is running the installer. Think of that as a step-0 if it comes up. 8-)

b) If the installer still hangs, the kill loop may not be fast enough; start over but leave the 'sleep 1;' out of the kill loop. It will slow down the install even more, but it is more likely to catch the command in time.

ASIDE: It woudl be nice if the installer had a don't compile the modules step...

Hopefully the installer will be fixed soon.

[guycook]

Here's what I do.

Terminal 1 and 2 get opened as root.

#1 I start the install of the rpm, and when the install asks me to

approve the license, I then start #2

#2 terminal I enter mv -v /etc/vmware/database /tmp/ that is done back

to #1

#1 accept the license, and then let it install until it says

"Installation Successful" then to #2 terminal

#2 terminal enter mv -v /tmp/database /etc/vmware/ to move the database

back

#2 I then in the same #2 terminal type "vmware-modconfig --console

--install-all

that's it, when the modconfig is all done your vmware install is done.

[canuck75]

I followed the steps you provided. Do I not need to do the while loop any longer? Also, I am using a bundle not an RPM.

I never get to where the installer says "Installation Successful". If fails as it did before.

Any ideas?

[guycook]

The linux distro, or the 64bit versions may be the variable that makes

my instructions not work for you, sorry about that. I haven't any

idea, I would ask vmware for assistance OR try the IRC channel for the

linux distro and ask there perhaps there is a tutorial or some other

help. Best of luck on that, and do post your solution as there always

is a solution.

Guy Cook

[mmalc]

Thanks Guy,

Worked a treat when installing from the .rpm

Cheers

[zeugma]

Your solution worked great for FC 11 on 2.6.30.9-90.fc11.i686.PAE. Thanks!

[vm_dude123]

I tried this on a fresh install of Karmic (Ubuntu 9.10) and while it worked, I could only run "vmware" as root else I got an error about not being able to read "modules.dep".

Based on another suggestion, I tried Workstation 7.0 and it installed and ran like a champ...with no tweaking.

[johnwheaton2]

I'll join the crowd of people thanking you for this lovely workaround! I now have other issues to sort out, but this cleared the most significant of not being able to install at all.

Kubuntu 9.10 x86_64 -- kernel 2.6.31-14

[martyfelker]

Incredible! Thank you so much. Following the instructions to the letter I installed WS 6.5.3 on karmic 64-bit kernel 2.6.31-15!

Marty Felker

[glunsforddavis]

This was a lifesaver! The instructions work exactly as written on Ubuntu 9.10 with Workstation 6.5.3. I had the same problem with OpenSUSE 11.2, but I tried a different distro before I found this page.

[icthy]

Thanks, I never would have figured that out. (I now have 6.5.3 working with Fedora 11 and kernel 2.6.29).

[rbaldauf]

The only thing i can say: Thank You Very Much

[tgarson]

Yahoo,

Thank you, thank you , thank you!

This worked for me on my Suse 11.2.

It's 12/30/09 and VMWare has yet to release a "fixed" install package for 6.5.3 for us Linux useing customers..........

[jprog]

thanks Peter and guycook.

I had the same issue with Workstation 6.5.4 install today (upgrade from 6.5.3.).

vmware-installer logfile show not activity after

Using UI type gtk

Opening database file /etc/vmware/database

...

this workaround helps me!

Thanks

Thomas

[jcottier]

Yes, it appears that VMware have not fixed this yet. The problem is still on version 6.5.4 with Ubuntu 8.04. Hope it works ok on Lucid wheh it arrives as I am upgrading. I have reported this issue to VMware, so hopefully they will act on it.

[gdmix]

Not only is this problem still here but it now fails to install most of the modules (network, etc.) on newer kernels (using Ubuntu 10.04 -- 2.6.32-21). I guess we are lucky ESX is not written so poorly.

[Eiríkr]

Attempting to install 6.5.4 on fresh Ubuntu 10.04. GUI installer hangs. Workaround posted in this thread fails out, ending with:

-

---

Jun 03 00:20:18.120: app| Extracting the sources of the vmci module.

Jun 03 00:20:18.136: app| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmci-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.32-22-generic/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.3

Jun 03 00:20:19.307: app| Copying Module.symvers from the vmci build failed.

-

---

So I try running the command directly to see what output I get:

-

---

root@Boreas:/tmp/vmware-root/modules/vmci-only# /usr/bin/make -C /tmp/vmware-root/modules/vmci-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.32-22-generic/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.3

Using 2.6.x kernel build system.

make: Entering directory `/tmp/vmware-root/modules/vmci-only'

make -C /lib/modules/2.6.32-22-generic/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \

MODULEBUILDDIR= modules

make[1]: Entering directory `/usr/src/linux-headers-2.6.32-22-generic'

CC /tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.o

In file included from /tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:47:

/tmp/vmware-root/modules/vmci-only/./include/vm_basic_types.h:108:7: warning: "__FreeBSD__" is not defined

In file included from /tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:48:

/tmp/vmware-root/modules/vmci-only/./include/pgtbl.h: In function ‘PgtblVa2MPN’:

/tmp/vmware-root/modules/vmci-only/./include/pgtbl.h:301: error: dereferencing pointer to incomplete type

/tmp/vmware-root/modules/vmci-only/./include/pgtbl.h: In function ‘PgtblVa2Page’:

/tmp/vmware-root/modules/vmci-only/./include/pgtbl.h:373: error: dereferencing pointer to incomplete type

In file included from /tmp/vmware-root/modules/vmci-only/./include/vmci_queue_pair.h:36,

from /tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:56:

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:329:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:333:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:401:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:407:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:460:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:506:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:551:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:595:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:640:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:684:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:729:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:773:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:775:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:816:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:860:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:862:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:903:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:945:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:947:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:986:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1028:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1030:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1069:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1223:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1227:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1313:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1536:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1663:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/./include/vm_atomic.h:1796:7: warning: "_MSC_VER" is not defined

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c: In function ‘VMCIHost_SignalCall’:

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:320: error: ‘TASK_NORMAL’ undeclared (first use in this function)

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:320: error: (Each undeclared identifier is reported only once

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:320: error: for each function it appears in.)

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c: In function ‘VMCIHost_WaitForCallLocked’:

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:362: error: dereferencing pointer to incomplete type

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:362: error: ‘TASK_INTERRUPTIBLE’ undeclared (first use in this function)

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:370: error: implicit declaration of function ‘schedule’

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:378: error: dereferencing pointer to incomplete type

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:378: error: ‘TASK_RUNNING’ undeclared (first use in this function)

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:382: error: implicit declaration of function ‘signal_pending’

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c: In function ‘VMCI_SignalEvent’:

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:674: error: ‘TASK_NORMAL’ undeclared (first use in this function)

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c: In function ‘VMCI_WaitOnEvent’:

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:704: error: dereferencing pointer to incomplete type

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:704: error: ‘TASK_INTERRUPTIBLE’ undeclared (first use in this function)

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:716: error: dereferencing pointer to incomplete type

/tmp/vmware-root/modules/vmci-only/linux/vmciKernelIf.c:716: error: ‘TASK_RUNNING’ undeclared (first use in this function)

make[2]: *** Error 1

make[1]: *** Error 2

make[1]: Leaving directory `/usr/src/linux-headers-2.6.32-22-generic'

make: *** http://vmci.ko Error 2

make: Leaving directory `/tmp/vmware-root/modules/vmci-only'

root@Boreas:/tmp/vmware-root/modules/vmci-only#

-

---

Not so good. Any ideas if/when the installer might be fixed? I've paid up through Workstation 6.5 and would like to get another few months of use out of it before upgrading to 7, but now I'm finding I can't even install the 6.5 series on the new Ubuntu distro. This doesn't exactly fill me with confidence that 7 will work either...

Cheers,

-- Eiríkr

[jcottier]

Workstation 7 works perfectly on 8.04 and 10.04. I contacted vmware about this bug and they gave me a temporary license to try 7.0 and the problem is not there. Now upgraded to 7.1 and it is working fine. No USB mods or anything needed, perfect out of the box!

[Eiríkr]

Good to know, that eases my worries. I downloaded 6.5.2 last night, the last build I had properly installed on Ubuntu 9.04, and if that fails to install on 10.04 I may just bite the bullet and upgrade to 7.

Incidentally, I forgot to mention that I'm on x86_64 -- jcottier, are you on a 64- or 32-bit host?

Cheers,

-- Eiríkr

[jcottier]

Hi Eiríkr,

No sorry only tested on 32 bit. I use an AMD 64bit CPU, but I have so far chickened out of installing a 64 bit OS on it due to some driver and legacy plugin issues.

Regards

John.