workstation 6.5.5 doesn't compile with kernel 2.6.37 (os11.4)

[cki201110141]

Hi all,

I have to stay with VMware Workstation 6.5.5 because of my prcessor lacks of PAE. Workstation 6.5.5 worked fine with a patch on openSUSE 11.3. Now I'd like to migrate to oS 11.4. But I couldn't find any patch until now. If anybody knows anything about a patch or how to resolve the following error (ioctl at the end), then please send me the infos:

Stopping VMware services:
   Virtual machine communication interface                             done
   Virtual machine monitor                                             done
   Blocking file system                                                done
Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-root/modules/vmmon-only'
make -C /lib/modules/2.6.37.1-1.2-default/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
  MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/linux-2.6.37.1-1.2-obj/i386/default'
make -C ../../../linux-2.6.37.1-1.2 O=/usr/src/linux-2.6.37.1-1.2-obj/i386/default/. modules
  CC [M]  /tmp/vmware-root/modules/vmmon-only/linux/driver.o
In file included from /tmp/vmware-root/modules/vmmon-only/./include/vmware.h:38:0,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.c:100:
/tmp/vmware-root/modules/vmmon-only/./include/vm_basic_types.h:108:7: warning: "__FreeBSD__" is not defined
In file included from /tmp/vmware-root/modules/vmmon-only/./common/vmx86.h:32:0,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.h:29,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.c:102:
/tmp/vmware-root/modules/vmmon-only/./include/x86msr.h:164:0: warning: "MSR_THERM2_CTL" redefined
/usr/src/linux-2.6.37.1-1.2/arch/x86/include/asm/msr-index.h:235:0: note: this is the location of the previous definition
In file included from /tmp/vmware-root/modules/vmmon-only/./include/vcpuset.h:103:0,
                 from /tmp/vmware-root/modules/vmmon-only/./include/modulecall.h:37,
                 from /tmp/vmware-root/modules/vmmon-only/./common/vmx86.h:33,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.h:29,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.c:102:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:329:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:333:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:401:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:407:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: In function ‘Atomic_And’:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:506:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: In function ‘Atomic_Or’:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:595:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: In function ‘Atomic_Xor’:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:684:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: In function ‘Atomic_Add’:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:773:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:775:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: In function ‘Atomic_Sub’:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:860:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:862:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: In function ‘Atomic_Inc’:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:945:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:947:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: In function ‘Atomic_Dec’:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:1028:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:1030:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h: At top level:
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:1223:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:1227:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:1536:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_atomic.h:1663:7: warning: "_MSC_VER" is not defined
In file included from /tmp/vmware-root/modules/vmmon-only/./include/vm_basic_asm.h:46:0,
                 from /tmp/vmware-root/modules/vmmon-only/./include/rateconv.h:45,
                 from /tmp/vmware-root/modules/vmmon-only/./include/modulecall.h:40,
                 from /tmp/vmware-root/modules/vmmon-only/./common/vmx86.h:33,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.h:29,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.c:102:
/tmp/vmware-root/modules/vmmon-only/./include/vm_basic_asm_x86.h:62:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_basic_asm_x86.h:177:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_basic_asm_x86.h:346:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_basic_asm_x86.h:453:7: warning: "_MSC_VER" is not defined
In file included from /tmp/vmware-root/modules/vmmon-only/./include/vm_asm.h:43:0,
                 from /tmp/vmware-root/modules/vmmon-only/linux/driver.c:104:
/tmp/vmware-root/modules/vmmon-only/./include/vm_asm_x86.h:486:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_asm_x86.h:779:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_asm_x86.h:820:7: warning: "_MSC_VER" is not defined
/tmp/vmware-root/modules/vmmon-only/./include/vm_asm_x86.h:922:7: warning: "_MSC_VER" is not defined
In file included from /tmp/vmware-root/modules/vmmon-only/linux/driver.c:120:0:
/tmp/vmware-root/modules/vmmon-only/./common/hostif.h:53:7: warning: "WINNT_DDK" is not defined
/tmp/vmware-root/modules/vmmon-only/linux/driver.c: In function ‘init_module’:
/tmp/vmware-root/modules/vmmon-only/linux/driver.c:427:15: error: ‘struct file_operations’ has no member named ‘ioctl’
make[4]: *** [/tmp/vmware-root/modules/vmmon-only/linux/driver.o] Error 1
make[3]: *** [_module_/tmp/vmware-root/modules/vmmon-only] Error 2
make[2]: *** [sub-make] Error 2
make[1]: *** [all] Error 2
make[1]: Leaving directory `/usr/src/linux-2.6.37.1-1.2-obj/i386/default'
make: *** [vmmon.ko] Error 2
make: Leaving directory `/tmp/vmware-root/modules/vmmon-only'
Unable to install vmmon

04-15-2011

I composed a patch for kernel 2.6.37.1-1.2. Now all modules compile without an error. But the application doesn't start!

Error message:

/usr/bin/vmware: line 31:  1372 Segmentation fault      "$BINDIR"/vmware-modconfig --appname="VMware Workstation" --icon="vmware-workstation"

Has anybody any ideas? Any help is apprecated!

The patch file and the compiler output are attached.

[Noel]

I don't know if anyone has done a patch for that kernel with 6.5.5.  I did it for WS 7.1.3.

[guest]

POR FAVOR, COMO SE CANCELA, OU QUEM PODE FAZER ISSO,

RECEITA-SE ALGUM EMAIL...

PLEASE HOW TO CANCEL THIS MESSAGES TO ME, ANY EMAIL SECURY....

Valmir Passero.

[guest]

HOW TO RESPONSE ...

TAMBÉM COMO RESPONDER...

Valmir Passero.

[cki201110141]

Hi Noel,

is there a 'how to' on the web to show a non system programmer how to compile those vmware modules?

Do you think vmware itself will develop a patch?

thanks for your reply.

[moep123 · Accepted solution]

Thanks a lot for the patch! Works great with 6.5.5 on suse 11.4 (2.6.37.1-1.2-desktop)

Check your environment variables for VMWARE_USE_SHIPPED_GTK=yes

When this was set i had the same problem (segfault). Set to =no or just unset completley

Hope this works for you, as your patch worked for me

[cki201110141]

Sehr geehrte Damen und Herren,

vielen Dank für Ihre email.

Ich bin zum 02.05.2011 wieder im Büro erreichbar. Die email wird nicht weitergeleitet.

In dringenden Fällen wenden Sie sich bitte an die IT-Hotline, Tel. 0 20 56 / 912-2100

Man wird Ihnen gerne weiterhelfen.

Mit freundlichen Grüßen aus Heiligenhaus

HELBAKO GmbH

Christof Kihm

Abteilung IT

Tel.: 0 20 56 / 912 - 21 01

Fax.: 0 20 56 / 912 - 21 99

christof.kihm@helbako.de

HELBAKO GmbH, Heiligenhaus

Amtsgericht Wuppertal HRB 18474

Geschäftsführung Ralf Burmester

Dear Sir or Madam,

thank you for your email.

I will be back in the office on May 02. 2011. Your email will not be forwarded.

In the meantime please contact the IT hotline, phone: +49 20 56 912 - 21 00

They will be happy to assist you.

Kind regards

HELBAKO GmbH

Christof Kihm

dept. IT

Tel.:+49 20 56 912 - 21 01

Fax.: +49 20 56 912 - 21 99

christof.kihm@helbako.de

HELBAKO GmbH, Heiligenhaus

competent county court Wuppertal HRB 18474

CEO Ralf Burmester

[cki201110141]

Hi moep123,

thanks for your hint.

I just updated to kernel 2.6.37.6-0.5 (openSuse 11.4), applied my patch, set two sym-links

*************

ln -s /usr/src/linux-headers-`uname -r`/include/generated/autoconf.h /usr/src/linux-headers-`uname -r`/include/linux/autoconf.h
ln -s /usr/src/linux-headers-`uname -r`/include/generated/utsrelease.h /usr/src/linux-headers-`uname -r`/include/linux/utsrelease.h

*************

and added

export VMWARE_USE_SHIPPED_GTK=no

to /usr/bin/vmware

after compiling by applying: vmware-modconfig --console --install-all

vmware could be started!!!

[goldbug888]

What is the patch referred to above, I have had no success following instructions above.

The sym-links fail with no such directory I am using 64bit OpenSuse

error

linux-pnv7:/usr/bin # uname -r
2.6.37.6-0.5-desktop
linux-pnv7:/usr/bin # ln -s /usr/src/linux-headers-`uname -r`/include/generated/autoconf.h /usr/src/linux-headers-`uname -r`/include/linux/autoconf.hln: failed to create symbolic link `/usr/src/linux-headers-2.6.37.6-0.5-desktop/include/linux/autoconf.h': No such file or directory
linux-pnv7:/usr/bin # ln -s /usr/src/linux-headers-`uname -r`/include/generated/utsrelease.h /usr/src/linux-headers-`uname -r`/include/linux/utsrelease.hln: failed to create symbolic link `/usr/src/linux-headers-2.6.37.6-0.5-desktop/include/linux/utsrelease.h': No such file or directory
linux-pnv7:/usr/bin #

I tried adding

export VMWARE_USE_SHIPPED_GTK=no

to /usr/bin/vmware

config fails with the following - I have the kernel sources and gcc etc.

linux-pnv7:/usr/bin # vmware-modconfig --console --install-all

gcc and kernel headers must be installed

linux-pnv7:/usr/bin #

I used to use the following script to compile on 11.3

[START]

#!/bin/bash
cd
cp -prn /usr/lib/vmware/modules/source /usr/lib/vmware/modules/source-backup
mkdir fixed
cd fixed
find /usr/lib/vmware/modules/source -name "*.tar" -exec tar xf '{}' \;

## add missing header on kernels
sed -i '/#include "compat_wait.h"/a\#include "compat_sched.h"' vmnet-only/vnetUserListener.c
sed -i '/#include "compat_page.h"/a\#include "compat_sched.h"' vmci-only/include/pgtbl.h
## comment out an invalid definition
sed -i 's|\(extern void poll_initwait(compat_poll_wqueues \*);\)|//\1|' *-only/include/compat_wait.h

tar cf vmblock.tar vmblock-only
tar cf vmci.tar vmci-only
tar cf vmmon.tar vmmon-only
tar cf vmnet.tar vmnet-only
tar cf vmppuser.tar vmppuser-only
tar cf vsock.tar vsock-only

cp -p *.tar /usr/lib/vmware/modules/source

cd /lib/modules/`uname -r`/build/include/linux
ln -s ../generated/autoconf.h
ln -s ../generated/utsrelease.h
cd

vmware-modconfig --console --install-all

# clean up
cd
rm -rf fixed
cd /lib/modules/`uname -r`/build/include/linux
rm autoconf.h utsrelease.h
cd /usr/lib/vmware/modules/
rm -rf source-backup
cd

[END]

Any help appreciated

[cki201110141]

first try to create the sysm-links, now by hand as su:

cd /lib/modules/2.6.37.6-0.5-desktop/build/include/linux
ln -s  ../generated/autoconf.h
ln -s ../generated/utsrelease.h

Now: vmware-modconfig --console --install-all

should start to compile, but stop with errors.

Now you shoult apply my patch (download the patch from my origin post) to unchanged freshly installed, untared files.

Now you tar them, put them to the right directory. Compiling should now be successfull.

[goldbug888]

Thanks for your reply, in a moment of madness I upgraded to 7.1 I figured it was time to upgrade it works fine on 11.4 kernel 2.6.37 latest kernel