VMware Communities
Noel
Expert
Expert

VMware Workstation 7.1.3 Linux kernel 2.6.37 (and 2.6.36) patch

UPDATE: These patches are obsoleted by Workstatioin 7.1.4, for which only a BKL-less kernel needs any patch (to vmnet).


Attached is the patch for VMware Workstation 7.1.3 on Linux Kernel 2.6.37.  Tested against Ubuntu Natty 11.04 2.6.37-2-generic.

The patch also includes the necessary change for 2.6.36, and should work with both kernels, since the 2.6.37 change is protected by a version check.  The plain 2.6.36 patch, which does not support 2.6.37 is available here.

List of changes:

  • define VMW_HAVE_UNLOCKED_IOCTL in vmmon's driver.c as required for 2.6.36

  • the semaphore patch for 2.6.37, applied only to those modules that actually need it: vmci, vmnet, and vsock.

[UPDATE: The attached patch is updated to work through 2.6.37-11]

Updated to reflect Workstation 7.1.4.

Reply
0 Kudos
109 Replies
Noel
Expert
Expert

No, it appears that your kernel is configured differently from Ubuntu and Fedora.  I'll try to get to it soon.  It isn't a lot of code to fix, I just need to get some time to do it.

Reply
0 Kudos
claudioseri
Contributor
Contributor

Hi guys,

i have the same problem with VMware Workstation 7.1.3 Linux kernel 2.6.37:

XXX@XXX ~/Scaricati $uname -a
Linux XXX.pc 2.6.37-12-generic-pae #26~maverick1-Ubuntu SMP Wed Jan 5 21:20:21 UTC 2011 i686 GNU/Linux

gen 20 17:04:44.274: app-3076138688| Extracting the sources of the vmmon module.
gen 20 17:04:44.306: app-3076138688| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmmon-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.37-12-generic-pae/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.5
gen 20 17:04:48.157: app-3076138688| Failed to compile module vmmon!

Reply
0 Kudos
Noel
Expert
Expert

claudioseri wrote:

i have the same problem with VMware Workstation 7.1.3 Linux kernel 2.6.37:

Linux XXX.pc 2.6.37-12-generic-pae #26~maverick1-Ubuntu SMP Wed Jan 5 21:20:21 UTC 2011 i686 GNU/Linux

From where did you get that kernel?  As far as I cai see, the most recent 2.6.37 from Ubuntu for Maverick, out of the Mainline PPA, is 2.6.37-020637rc2-generic, and that kernel is working here.

The build command that we post with the patch procedure would show us more details about the compile failure.

     --- Noel

Reply
0 Kudos
michelemase
Contributor
Contributor

The patch should work with the BKL support:

Error:

/tmp/vmware-root/modules/vmmon-only/linux/driver.c: In function ‘LinuxDriver_Ioctl’:
/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1736: error: implicit declaration of function ‘unlock_kernel’
/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1742: error: implicit declaration of function ‘lock_kernel’

Explanation:

vmmon-only/linux/driver.c
2540-2551
/*
  *-----------------------------------------------------------------------------
  *
  * LinuxDriver_UnlockedIoctl --
  *
  *      Main path for UserRPC.
  *
  *      Unfortunately our LinuxDriver_Ioctl needs some surgery before it can
  *      run without big kernel lock, so for now we do not use this
  *      as unlocked_ioctl handler.  Though we could, it does not matter
  *      who'll obtain big kernel lock, whether we or our caller...
  *
  * Results:
  *
  * Side effects:
  *      None.
  *
  *-----------------------------------------------------------------------------
  */
Suggested Solution:
Try recompiling you custom kernel with Big Kernel Lock (BKL) support (under Kernel hacking); it should work.

Regards

Michele Masè

Reply
0 Kudos
dibl
Enthusiast
Enthusiast

Apparently this is an "early adopter" issue -- I believe the situation with aptosid's 2.6.37 kernel is that the BKL option has already been removed, in anticipation of its removal from the Linux kernel entirely in 2.6.39:

http://kerneltrap.org/mailarchive/linux-kernel/2010/11/17/4646981

So, to the extent that VMware is dependent on the BKL being enabled, this is going to have to change sometime prior to support of 2.6.39 kernels, anyway. Meanwhile the 2.6.36 aptosid kernel patches correctly and runs VMware with no issue.

Reply
0 Kudos
Kodaxo
Contributor
Contributor

I am trying to recompile the kernel modules under Ubuntu 10.04 (Lucid) running the 2.6.37-12 kernel for Lucid from the Ubuntu kernel dev team's PPA which already includes legacy BKL.

I successfully applied the patch (vmware-7.1.3-2.6.37-9-generic.patch.zip) but I was still recieving errors related to sk_sleep while compiling af_vsock.c

vmware-modules/vsock-only/linux/af_vsock.c: In function 'VSockVmciStreamConnect':
vmware-modules/vsock-only/linux/af_vsock.c:3233: error: 'struct sock' has no member named 'sk_sleep'
vmware-modules/vsock-only/linux/af_vsock.c:3256: error: 'struct sock' has no member named 'sk_sleep'
vmware-modules/vsock-only/linux/af_vsock.c:3268: error: 'struct sock' has no member named 'sk_sleep'

It appears that the patch does not touch af_vsock.c. So I went and grabbed the vsock kernel module patch that addresses this particular issue for 2.6.35 kernels from here ( http://www.linuxinsight.com/persuading-vmware-workstation-7.1-to-cooperate-with-linux-kernel-2.6.35....). With both patches applied all of the kernel modules compiled successfully but the vsock kernel module is not working properly. I am not getting a /dev/vsock device.

Does anyone know why af_vsock.c is not being patched by vmware-7.1.3-2.6.37-9-generic.patch?

Does anyone know why the 2.6.35 patch which seems to address the sk_sleep problem would compile but not result in a functional vsock kernel module?

Has anyone successfully compiled and utilized the vsock.ko module with a 2.36.7 kernel?

Thanks in advance...

Reply
0 Kudos
Noel
Expert
Expert

Kodaxo wrote:

I am trying to recompile the kernel modules under Ubuntu 10.04 (Lucid) running the 2.6.37-12 kernel for Lucid from the Ubuntu kernel dev team's PPA which already includes legacy BKL.

I have not tried that particular combination (Lucid + 2.6.37-12), but I have vmlinuz-2.6.37-12-generic and vmlinuz-2.6.38-1-generic both working with Natty using the updated patch attached to the first message of this thread.

Apparently, we need a more involved patch for when the BKL is removed, which some other distros have done.  It is on my list, but spare time has been in short supply lately.

Reply
0 Kudos
Kodaxo
Contributor
Contributor

The Ubuntu Kernel team's 2.6.37-12 kernel already has Legacy BKL enabled so that should not be an issue. There would probably be difficulties with more than just the compilation of vsock.ko if that were the case.

The kernel modules all compile and function properly under the current patch, except for vsock.ko which generates the errors I listed in my previous post all related to sk_sleep in "af_vsock.c".  The patch for the sk_sleep issue that was developed for 2.6.35 fixes the compilation issues but the kernel module will not load. I have yet to try troubleshooting the reason why the resulting vsock.ko kernel module will not load.

Did you confirm that vsock.ko was loading  when you tested with vmlinuz-2.6.37-12-generic and vmlinuz-2.6.38-1-generic? If vm-tools was functioning properly (automatic input device grab/ungrab, automatic guest os resizing with window size change events, etc) then I think that would confirm vsock.ko was loading since vm-tools relies on it to accomplish a lot of that functionality.

If it was working properly with vmlinuz-2.6.37-12-generic then I will have to look at differences in the kernel options for the Ubuntu kernel...??

Reply
0 Kudos
bounceerr
Contributor
Contributor

Hi,

It seems that patch provided in the first post was done on modules patched for 2.6.35 first as whenever i tried to apply that for vanila vmware modules i was not able to compile them. Here is a patch which works on vanila modules. Tested for ubuntu's linux-image-2.6.37-12-generic.

@Kodaxo

I ran into the issue which you mentioned. After applying patches for 2.6.35 everything works flawlessly.

Reply
0 Kudos
Kodaxo
Contributor
Contributor

Thanks Bounceerr,

   I tested the patch you posted and it compiled all of the kernel modules but with warnings. The vsock module still fails to load with an "invalid argument" error. However, I recalled similar issues in the past and applied the fix for vsock dependencies on vmci and that seems to have done the trick. The vsock module now compiles without warnings and loads successfully with the Lucid 2.6.37-12 kernel from the Ubuntu Kernel Dev teams PPA. I have yet to test it extensively, I will post again if I find any issues.

So in summary, to get it working the following steps are necessary:

Reply
0 Kudos
martyfelker
Enthusiast
Enthusiast

I installed VMware WS on Debian Squeeze (stable release today) 64 bit.  Installed the bundle and immediately ran patch-modules_v61.sh with

vmware-7.1.3-2.6.37-9-generic.patch in the same directory.  Compiled with no error messages at all.  WS 7.1.3 works perfectly.
Marty
Marty Felker
Reply
0 Kudos
matthewls
Enthusiast
Enthusiast

Hello Noel,

Have you started looking at the 2.6.38 RCs?

--Matthew

Reply
0 Kudos
renatoyamane
Contributor
Contributor

Something is not working:

./vmware-7.1.3-2.6.37-9-generic.patch
diff: original//vmci-only/include/compat_semaphore.h: File or folder not found
diff: patched//vmci-only/include/compat_semaphore.h: File or folder not found

./vmware-7.1.3-2.6.37-9-generic.patch: line 2: ---: command not found
./vmware-7.1.3-2.6.37-9-generic.patch: line 3: +++: command not found

./vmware-7.1.3-2.6.37-9-generic.patch: line 4: @@: command not found

./vmware-7.1.3-2.6.37-9-generic.patch: line 8: sintax error close to `token' not espected `('
./vmware-7.1.3-2.6.37-9-generic.patch: line 8: `-#if defined CONFIG_PREEMPT_RT && LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 31)'

I'm running Kernel 2.6.37.1 from kernel.org over Debian Squeeze.

Reply
0 Kudos
michelemase
Contributor
Contributor

For renatoyamane:

the usage is:

patch -p1 < patchfile

and NOT

./patchfile

in your case patchfile is vmware-7.1.3-2.6.37-9-generic.patch

Regards

Michele

Reply
0 Kudos
renatoyamane
Contributor
Contributor

$ patch -p1 < vmware-7.1.3-2.6.37-9-generic.patch

patching file vmci-only/include/compat_semaphore.h
Hunk #1 FAILED at 28.
Hunk #2 FAILED at 41.
2 out of 2 hunks FAILED -- saving rejects to file vmci-only/include/compat_semaphore.h.rej
patching file vmmon-only/linux/driver.c
Hunk #1 FAILED at 145.
Hunk #2 FAILED at 170.
2 out of 2 hunks FAILED -- saving rejects to file vmmon-only/linux/driver.c.rej
patching file vmnet-only/compat_semaphore.h
Hunk #1 FAILED at 28.
Hunk #2 FAILED at 41.
2 out of 2 hunks FAILED -- saving rejects to file vmnet-only/compat_semaphore.h.rej
patching file vsock-only/shared/compat_semaphore.h
Hunk #1 FAILED at 28.
Hunk #2 FAILED at 41.
2 out of 2 hunks FAILED -- saving rejects to file vsock-only/shared/compat_semaphore.h.rej

I'm using vmware-7.1.3-2.6.37-9-generic.patch and VMware-Workstation-Full-7.1.3-324285.i386.bundle

Reply
0 Kudos
martyfelker
Enthusiast
Enthusiast

Possible cause for this error.

Did you completely remove a previous install of WS?  All rm -fr /usr/lib/vmware.  I have found that /usr/bin/vmware/vmware-installer -u vmware-workstation does not do this.

Thereinsatall   vmware  workstation bundle and run sudo patch-modules_v61.sh immediately after  the "successful install".  The script will call patch directly.  Looks like you  have  installed build-essential which includes patch I believe. This method has always worked for me perfectly

Marty

Marty Felker
Reply
0 Kudos
renatoyamane
Contributor
Contributor

I think I missed something.

This is what I did:

# vmware-installer --uninstall-product vmware-workstation

# ./VMware-Workstation-Full-7.1.3-324285.i386.bundle

I get patch-modules.sh from here [1] and patch from here [2]:

[1] http://communities.vmware.com/servlet/JiveServlet/download/1674972-52660/patch-modules_2.6.37.sh

[2] http://communities.vmware.com/servlet/JiveServlet/download/1650363-51266/vmware-7.1.3-2.6.37-9-gener...

Unzip patch and run:

./patch-modules_2.6.37.sh

I try run vmware-workstation and it appear to install some modules, I type the root password and VM-WS show me some warnings and told me to read the log:

Fev 20 17:52:07.867: app-3076458176| Log for VMware Workstation pid=26890 version=7.1.3 build=build-324285 option=Release
Fev 20 17:52:07.867: app-3076458176| The process is 32-bit.
Fev 20 17:52:07.867: app-3076458176| Host codepage=UTF-8 encoding=UTF-8
Fev 20 17:52:07.867: app-3076458176| Logging to /tmp/vmware-root/setup-26890.log
Fev 20 17:52:07.945: app-3076458176| modconf query interface initialized
Fev 20 17:52:07.946: app-3076458176| modconf library initialized
Fev 20 17:52:07.962: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:07.966: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:07.973: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:07.981: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:07.988: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:08.008: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.011: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.013: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.016: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.018: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.030: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.033: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.036: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.038: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.040: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.044: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:08.051: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:08.131: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.133: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.136: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.138: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.140: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.144: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:08.151: app-3076458176| Your GCC version: 4.4
Fev 20 17:52:08.179: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.182: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.185: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.187: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.190: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.800: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:08.800: app-3076458176| Building module vmmon.
Fev 20 17:52:08.800: app-3076458176| Extracting the sources of the vmmon module.
Fev 20 17:52:08.809: app-3076458176| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmmon-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.37.1/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.5
Fev 20 17:52:12.785: app-3076458176| Installing module vmmon from /tmp/vmware-root/modules/vmmon.o.
Fev 20 17:52:12.786: app-3076458176| Registering file: /usr/lib/vmware-installer/1.1/vmware-installer --register-file vmware-player-app regular /lib/modules/2.6.37.1/misc/vmmon.o
Fev 20 17:52:13.027: app-3076458176| Registering file: /usr/lib/vmware-installer/1.1/vmware-installer --register-file vmware-player-app regular /lib/modules/2.6.37.1/misc/vmmon.ko
Fev 20 17:52:13.297: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:13.297: app-3076458176| Building module vmnet.
Fev 20 17:52:13.298: app-3076458176| Extracting the sources of the vmnet module.
Fev 20 17:52:13.305: app-3076458176| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmnet-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.37.1/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.5
Fev 20 17:52:16.308: app-3076458176| Failed to compile module vmnet!
Fev 20 17:52:16.315: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:16.315: app-3076458176| Building module vmblock.
Fev 20 17:52:16.315: app-3076458176| Extracting the sources of the vmblock module.
Fev 20 17:52:16.323: app-3076458176| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmblock-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.37.1/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.5
Fev 20 17:52:19.305: app-3076458176| Installing module vmblock from /tmp/vmware-root/modules/vmblock.o.
Fev 20 17:52:19.306: app-3076458176| Registering file: /usr/lib/vmware-installer/1.1/vmware-installer --register-file vmware-player-app regular /lib/modules/2.6.37.1/misc/vmblock.o
Fev 20 17:52:19.539: app-3076458176| Registering file: /usr/lib/vmware-installer/1.1/vmware-installer --register-file vmware-player-app regular /lib/modules/2.6.37.1/misc/vmblock.ko
Fev 20 17:52:19.808: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:19.809: app-3076458176| Building module vmci.
Fev 20 17:52:19.809: app-3076458176| Extracting the sources of the vmci module.
Fev 20 17:52:19.817: app-3076458176| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmci-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.37.1/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.5
Fev 20 17:52:20.338: app-3076458176| Failed to compile module vmci!
Fev 20 17:52:20.344: app-3076458176| Trying to find a suitable PBM set for kernel 2.6.37.1.
Fev 20 17:52:20.344: app-3076458176| Building module vmci.
Fev 20 17:52:20.344: app-3076458176| Extracting the sources of the vmci module.
Fev 20 17:52:20.352: app-3076458176| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmci-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.37.1/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.4.5
Fev 20 17:52:20.867: app-3076458176| Failed to compile module vmci!

Reply
0 Kudos
martyfelker
Enthusiast
Enthusiast

I got messages about PBM sets quite awhile ago. Forget how it was resolved. Since you patched the source you should not be getting these messages from WS - WS tries to compile again when the kernel version has changed. It succeeds if the kernel upgrade is only between two versions of the same kernel (2.6.37.X) after the patch is successfully installed. I still don't understand why VMware, Inc hasn't solved this kernel compile problem. VirtualBox has no such problem but I realize the sources are far different. Also I am using the 64-bit version of WS and kernel so my results may be different (doubt this very much).

Marty

Marty Felker
Reply
0 Kudos
jeffreyknight
Contributor
Contributor

This worked for me on 11.04 (x64) with VMware-Workstation-Full-7.1.3-324285.x86_64:

- Using a fresh install of 11.04 Alpha 2 from DVD

- Install vmware:

  sudo bash VMware-Workstation-Full-7.1.3-324285.x86_64.bundle

- Download:

   - http://communities.vmware.com/servlet/JiveServlet/download/1674972-52660/patch-modules_2.6.37.sh

   - http://communities.vmware.com/servlet/JiveServlet/download/1650363-51266/vmware-7.1.3-2.6.37-9-gener...

- Update patch-modules_2.6.37.sh to point to the right patch:

   line 6: fpatch=vmware-7.1.3-2.6.37-9-generic.patch

- Patch it:

  sudo bash patch-modules_2.6.37.sh

- Result:


jknight@home:~/Desktop$ sudo bash patch-modules_2.6.37.sh
patching file vmci-only/include/compat_semaphore.h
patching file vmmon-only/linux/driver.c
patching file vmnet-only/compat_semaphore.h
patching file vsock-only/shared/compat_semaphore.h
Stopping VMware services:
   VMware USB Arbitrator                                               done
   VM communication interface socket family                            done
   Virtual machine communication interface                             done
   Virtual machine monitor                                             done
   Blocking file system                                                done
Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-root/modules/vmmon-only'
make -C /lib/modules/2.6.38-1-generic/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. \
      MODULEBUILDDIR= modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.38-1-generic'
  CC [M]  /tmp/vmware-root/modules/vmmon-only/linux/driver.o


[...etc ...]

Built vsock module
Starting VMware services:
   VMware USB Arbitrator                                               done
   Virtual machine monitor                                             done
   Virtual machine communication interface                             done
   VM communication interface socket family                            done
   Blocking file system                                                done
   Virtual ethernet                                                    done
   Shared Memory Available                                             done


All done, you can now run VMWare WorkStation.

Reply
0 Kudos
zx5000
Enthusiast
Enthusiast

I ditched vmware 6 in favor of virtualbox over a year ago because of this kernel patching hell. I see that vmware is still favoring M$ Windoze. VMware doesn't get my money for version 7.

VirtualBox just plain works. On kernel updates you run a script and re-start.

Reply
0 Kudos