Krellan
Enthusiast
Enthusiast

Here is patch for Workstation 6.5.1 on Linux 2.6.29 kernel

Here is something I'm proud of: a complete patch for getting VMware Workstation 6.5.1 to correctly compile its kernel modules against the new Linux 2.6.29 kernel.

It's attached.

I read the other threads here:

http://communities.vmware.com/message/1210084

http://communities.vmware.com/message/1141306

There were also patches posted to these threads, but after looking at them, I was concerned. There seems to be a serious error, as cap_raise() has been blindly changed to cap_raised(), which is actually a different API that has a completely different effect (it's a query instead of a set). Linux 2.6.29 changed capabilities a lot, evidently, and broke a lot of old code. I had to put in wrapper functions to call the capabilities change, because the new kernel shuffled things around quite a bit.

Also, the Makefile itself was failing to find the Linux kernel headers in many situations. This was a bug in VMware's Makefile: it wasn't including the $(HEADER_DIR) in its check for various features, which was causing misdetection of a lot of things earlier (the poll_initwait() problem).

I made what I believe are the minimum possible changes in order to get it to compile and load cleanly against 2.6.29. It's also in a single, unified, patch, that covers all 5 of the known working VMware modules (ignore vmppuser, it's pretty much dead code, I don't know why VMware still ships it, since it's not used by vmware-modconfig in any way that I could find).

To use this patch, download it from the attachment, gunzip it, then go to your /usr/lib/vmware/modules/source directory. Untar all of the files. Apply the patch with "patch -p1". After the patch has applied, re-tar all the files. Then, run "vmware-modconfig --console --install-all" again, under Linux 2.6.29, and it should now work. Please let me know your results, if it doesn't work. I tested it on an x86_64 machine with VMware Workstation 6.5.1 and it worked, but anything else is untested.

Exercises left to the reader:

1) Make a script to help people install this patch easier. It's error-prone to have to untar and re-tar files by hand.

2) Add in the KERNEL_VERSION checks to the code. Right now, I changed it just for 2.6.29. It could be made portable to other kernel versions as well by isolating the 2.6.29 changes in their own #if blocks.

Hope this helps!

Josh

0 Kudos
43 Replies
depping
Leadership
Leadership

Cool, thanks! I will write a blog article about this DIY patch this afternoon. Been looking for a fix for a while but never had the time to actually write one myself! Much appreciated!!

Duncan

VMware Communities User Moderator

-


Blogging:

Twitter:

If you find this information useful, please award points for "correct" or "helpful".

0 Kudos
Krellan
Enthusiast
Enthusiast

Thanks!

Interestingly, I learned that somebody else had been working on the same patch, and ironically, we both released it the same night. So, you have your choice of patch to apply.

Here's the other patch:

http://www.linuxinsight.com/vmware-workstation-6.5.1-build-126130-linux-kernel-2.6.29.html

I inspected the code and found it's almost identical.

Josh

0 Kudos
Krellan
Enthusiast
Enthusiast

Here's an update to my patch.

I completed the things that were mentioned earlier.

Now, it should have all of the KERNEL_VERSION checks in it, so the code is now good to run on older versions of the kernel also, not just 2.6.29. That should help when testing on other kernels.

Also, I made an "easy install" shell script. You run this file, as root, and it will automatically untar, apply the patch, and re-tar all of the module files. That will help prevent mistakes and frustration from having to do it all manually.

Here it is, attached.

Download both files, the patch and the script, to the same directory. Run the script with the bash command:

bash vmware-modules-2.6.29-fix.sh

Please try it, let me know if it works. I've tested on 2.6.29 but not 2.6.28 (another version that is known to have some problems with VMware).

Josh

0 Kudos
h2-2
Contributor
Contributor

The first patch worked perfectly on 32 bit 2.6.29 kernel, and the patch executes fine on 64 bit, no errors, but fails to build the module with the following errors on 64 bit:

::::

sux

password

vmplayer

Logging to /tmp/vmware-h2/setup-6847.log

modinfo: could not find module vmmon

modinfo: could not find module vmnet

modinfo: could not find module vmblock

modinfo: could not find module vmci

modinfo: could not find module vsock

modinfo: could not find module vmmon

modinfo: could not find module vmnet

modinfo: could not find module vmblock

modinfo: could not find module vmci

modinfo: could not find module vsock

/usr/bin/vmplayer: line 31: 6847 Segmentation fault "$BINDIR"/vmware-modconfig --appname="VMware Player" --icon="vmware-player"

::::

This is confirmed by another person too. I haven't tried the new script though the patch and first method was integrated into svmi vm installer script and worked fine with 32 bit. Didn't have as many checks because I figure that vmware will release a fixed version quite soon.

Thanks for your work.

By the way, another person said that the first rough patch, the one you fixed, worked on 64 bit, or at least appears to work.

0 Kudos
Krellan
Enthusiast
Enthusiast

Interesting. I'm running 64-bit now (distro is Fedora 10). I actually have never tested on 32-bit! So, that's interesting how it worked for you on 32-bit but NOT for 64-bit.

The memory limit of 32-bit gets very tight when using VMware Smiley Happy

I also haven't tried with vmplayer. Do you also have VMware Workstation? I wonder if it makes a difference whether you're using Workstation or Player?

Also, the error is kind of weird. The errors aren't from my patch or script, but instead, they're from when you are running vmplayer, later on. The modules should have already been built by this point.

Was the script able to do its job without throwing an error? If you got the "Done" message at the end of the script, then it should have worked. If you got any other messages, I'd be curious to see them.

0 Kudos
h2-2
Contributor
Contributor

Sorry for not providing more details: Both system are Debian Sid, recently upgraded.

I used the svmi script to run the patches and the main fixes from the original thread's one liner. You'd not yet put up your new stuff so I didn't check that or see it until checking back on the thread, though I can try using the script, but if it what it does is automate the original one liner basically, that's what svmi does too, download patch, do the tar stuff, run the patch, do the tar stuff, then quit. It does not try to build the module however, I leave that for the user to do when they start the vmplayer gui.

The errors came from trying to build the module using the gui module builder, to see where it failed, since the gui doesn't open at all.

Patches run fine, I assume the other parts run fine too.

Should not be a ram issue, 2 gigabytes on both system, actually 1 gigabyte on one. I've run this on two 32 bit installs with 2.6.29 kernel, works fine. Not sure what else to tell you.

And yes, it's vmplayer, never tried workstation, so that's a difference.

A sux start of vmplayer on the working 32 bit machine just shows the various things starting as expected.

0 Kudos
Krellan
Enthusiast
Enthusiast

If the modules didn't build, I'd be curious to see what broke.

Can you run vmware-modconfig with --console to make it write output to the console and not use the GUI?

vmware-modconfig --console --install-all

See if that works.

0 Kudos
h2-2
Contributor
Contributor

on 64 bit, returns: segmentation fault

Error 139

Nothing else, error is from $? return value.

0 Kudos
Krellan
Enthusiast
Enthusiast

If the vmware-modconfig program doesn't even start up, that's a sign of a deeper problem. There was another thread about a problem with the library linking that was causing strange ldd errors.

http://communities.vmware.com//message/1213672

Try running this a few times from the command line:

/usr/bin/vmware-modconfig --help

You should see the help text, every time. If it doesn't come up every time, you might have that strange intermittent loading problem that I was also having, in that other thread.

0 Kudos
dawoim
Contributor
Contributor

I tried out the patch, and am having a problem. I'm on SuSE Factory (almost current), kernel 2.6.29-rc8-5-pae. The script applied correctly, the modules built and installed OK. When I run the Player, it tries to start up, then I get a pop-up message, "Error while powering on: Cannot find a valid peer process to connect to." Here's the running processes:

# ps -ef | grep vm
root 28123 1 0 17:32 ? 00:00:00 /usr/bin/vmnet-bridge -s 6 -d /var/run/vmnet-bridge-0.pid -n 0
root 28639 1 0 17:32 ? 00:00:00 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet1/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet1/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet1.pid vmnet1
root 28644 1 0 17:32 ? 00:00:00 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet1.pid /dev/vmnet1 vmnet1
root 28655 1 0 17:32 ? 00:00:00 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet8/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet8/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet8.pid vmnet8
root 28657 1 0 17:32 ? 00:00:00 /usr/bin/vmnet-natd -s 6 -d /var/run/vmnet-natd-8.pid -m /var/run/vmnet-natd-8.mac -c /etc/vmware/vmnet8/nat/nat.conf
root 28668 1 0 17:32 ? 00:00:00 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet8.pid /dev/vmnet8 vmnet8
root 29583 1 0 17:32 pts/2 00:00:01 /usr/lib/vmware/bin/vmware-tray
root 29658 15858 8 17:40 pts/2 00:00:04 /usr/lib/vmware/bin/vmplayer
root 30435 29658 0 17:40 pts/2 00:00:00 /usr/lib/vmware/bin/vmplayer-daemon
root 30460 1 3 17:41 pts/2 00:00:00 /usr/lib/vmware/bin/vmware-unity-helper --daemon

# lsmod | grep vm
vmnet 39860 13
vmblock 13552 3
vmci 50004 0
vmmon 76972 0

What's missing?

0 Kudos
h2-2
Contributor
Contributor

I tried a few things, deleting /etc/vmware and /usr/lib/vmware after removing vmplayer, then running hte patch, which works fine, then running your install script, which ran fine, then trying to run the vmplayer module build command, vmware-modconfig --console --install-all which exits with error 139, segmenation fault.

running as normal user: vmware-modconfig --help works fine every time.

Since there's a small chance this is based on a hardware problem, and since another user got vmplayer running on 2.6.29 sid with the new patch and running your script, which I believe svmi duplicated in terms of the logic, then running hte module building command, I'm going to need to see if anyone else has issues.

I'll let you know what other people experience, for me it's uninstallable, sadly, on 2.5.1 vmplayer, but happily, the 32 bit install works just fine, old patch and new one both are fine.

0 Kudos
dawoim
Contributor
Contributor

OK, I see what is going on. I was trying to display to the XWindows server on my Mac (OS X 10.4.11). At the tail of the vmware log file is:

Apr 01 21:26:00.482: vmx| XINFO X fd is 133
Apr 01 21:26:00.483: vmx| XINFO depth 24 bpp 32 class 4
Apr 01 21:26:00.507: vmx| XINFO IO fatal error. Exiting ...

In other words, it had a beef with my Mac's X server. When I run the vmware on the native display, it works OK. I hope that gets fixed sometime, as the display on my Mac is much larger than the one on the PC...

0 Kudos
h2-2
Contributor
Contributor

Here's an update on Debian Sid 2.6.29 kernel 2.5.2 install for vmplayer:

Without running the patch:

vmon module build failed, I can't find the error output, but the module started building, no seg fault, but failed on vmon.

after running the patch, which had this hunk error, but otherwise seemed to go ok, the module built, I don't know if it's missing any features:

Patch error:

:::::::::::::::

Applying patch now...

patching file vmblock-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmci-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmmon-only/include/x86apic.h

Hunk #1 FAILED at 94.

1 out of 1 hunk FAILED -- saving rejects to file vmmon-only/include/x86apic.h.rej

patching file vmmon-only/linux/driver.c

patching file vmmon-only/linux/hostif.c

Hunk #1 succeeded at 3424 (offset 1 line).

Hunk #2 succeeded at 3538 (offset 1 line).

Hunk #3 succeeded at 3662 (offset 1 line).

patching file vmmon-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmnet-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmnet-only/netif.c

patching file vsock-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

::::::::::::::::

Now vmplayer starts fine with the patch on 64 bit. Any chance of getting a new patch for 2.5.2? The old one seems to work fine except for the bad hunk.

0 Kudos
h2-2
Contributor
Contributor

It's the same in 32 and 64 bit, so here's the error without the patch for 2.5.2 on Debian Sid 2.6.29 kernel:

::::::::::

Stopping VMware services:

Virtual machine communication interface done

Virtual machine monitor done

Blocking file system done

Using 2.6.x kernel build system.

make: Entering directory `/tmp/vmware-root/modules/vmmon-only'

make -C /lib/modules/2.6.29-0.lqx.1-liquorix-686/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules

make[1]: Entering directory `/usr/src/linux-headers-2.6.29-0.lqx.1-liquorix-686'

CC /tmp/vmware-root/modules/vmmon-only/linux/driver.o

In file included from /tmp/vmware-root/modules/vmmon-only/linux/driver.c:31:

/tmp/vmware-root/modules/vmmon-only/./include/compat_wait.h:78: error: conflicting types for 'poll_initwait'

include/linux/poll.h:67: error: previous declaration of 'poll_initwait' was here

/tmp/vmware-root/modules/vmmon-only/linux/driver.c: In function 'LinuxDriverSyncCallOnEachCPU':

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1423: error: too many arguments to function 'smp_call_function'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c: In function 'LinuxDriver_Ioctl':

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1987: error: 'struct task_struct' has no member named 'euid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1987: error: 'struct task_struct' has no member named 'uid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1988: error: 'struct task_struct' has no member named 'fsuid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1988: error: 'struct task_struct' has no member named 'uid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1989: error: 'struct task_struct' has no member named 'egid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1989: error: 'struct task_struct' has no member named 'gid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1990: error: 'struct task_struct' has no member named 'fsgid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:1990: error: 'struct task_struct' has no member named 'gid'

/tmp/vmware-root/modules/vmmon-only/linux/driver.c:2007: error: too many arguments to function 'smp_call_function'

make[2]: *** Error 1

make[1]: *** Error 2

make[1]: Leaving directory `/usr/src/linux-headers-2.6.29-0.lqx.1-liquorix-686'

make: *** http://vmmon.ko Error 2

make: Leaving directory `/tmp/vmware-root/modules/vmmon-only'

Unable to install vmmon

ERROR: 1 occurred in the Vmplayer kernel module build!

;:::::::::

I don't know how important that failed hunk was in the patch, but the module builds, and vmplayer starts fine now.

0 Kudos
Krellan
Enthusiast
Enthusiast

As for the patch hunk error, that's interesting. There should be no errors at all when patching, if you used the same VMware Workstation sources that I did.

What's your Workstation version? I'm running 6.5.1.

Also, was your modules directory already patched, by either this patch, or somebody else's patch? You can only have one at a time.

I recommend you restore the contents of your /usr/lib/vmware/modules/source directory to the original contents that came with VMware. Maybe remove and reinstall VMware if you have to, in order to get this directory back to the way it was.

Then, try it again. If you still get errors, when trying to patch, and you know for sure the directory is the original pristine contents that originally came with VMware, then please attach the contents of the directory here and send it to me. Maybe you're running a slightly different version of VMware, and they changed the directory contents slightly? Now I'm curious.

Josh

0 Kudos
avigator
Contributor
Contributor

i have no workstaton, i just using vmware-player (version 2.5.1), and since i using kernel 2.6.29, vmware-player does not work. it's show me as below picture that i attached. so is this solution also work for vmware-player?

$ cat /tmp/vmware-root/setup-13941.log

Apr 02 01:33:48.049: app| Log for VMware Workstation pid=13941 version=6.5.1 build=build-126130 option=Release

Apr 02 01:33:48.049: app| Host codepage=UTF-8 encoding=UTF-8

Apr 02 01:33:48.049: app| Logging to /tmp/vmware-root/setup-13941.log

Apr 02 01:33:49.403: app| Extracting the sources of the vmmon module.

Apr 02 01:33:49.414: app| Building module with command: /usr/bin/make -C /tmp/vmware-root/modules/vmmon-only auto-build SUPPORT_SMP=1 HEADER_DIR=/lib/modules/2.6.29-custom-3.2/build/include CC=/usr/bin/gcc GREP=/usr/bin/make IS_GCC_3=no VMCCVER=4.2.4

0 Kudos
h2-2
Contributor
Contributor

Krellan, thanks for your reply. As indicated, this new issue is with the patch running on the freshly, just released, vmplayer 2.5.2. I'll assume you will see the same patching issues on workstation x.x.2 as well when it's released.

I'll run the tests however you asked for, and provide the data.

Your patch and efforts are much appreciated.

It's fairly clear that the patch files changed slightly however, note the offsets on the 2.5.2 patching job that were not present on the 2.5.1 patching.

But I'll delete the directory after uninstalling then reinstall as requested and I'll post the fiiles.

0 Kudos
h2-2
Contributor
Contributor

I tried full cleanup, no difference, that was not the issue, in vmplayer 2.5.2 32 and 64 bit, the patch has this issue:

::::::::::

Applying patch now...

patching file vmblock-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmci-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmmon-only/include/x86apic.h

Hunk #1 FAILED at 94.

1 out of 1 hunk FAILED -- saving rejects to file vmmon-only/include/x86apic.h.rej

patching file vmmon-only/linux/driver.c

patching file vmmon-only/linux/hostif.c

Hunk #1 succeeded at 3424 (offset 1 line).

Hunk #2 succeeded at 3538 (offset 1 line).

Hunk #3 succeeded at 3662 (offset 1 line).

patching file vmmon-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmnet-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmnet-only/netif.c

patching file vsock-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

::::::::::::

I'm using the original logic for the pre and post patch processing, that seems to work fine, the module building works fine too, vmplayer 2.5.2 starts fine, no issues, works for other users too, but the patch error just means it needs to be updated, I'd check the syntax but I have trouble with doing patches, it's not one of my strong points.

attached the source directory, post patching, with orig data.

Thanks again for the time spent on this issue, it helps many more people out there.

0 Kudos
avigator
Contributor
Contributor

i copied entire folder /usr/lib/vmware/modules/source out for a test. has a error same as upstair.

=============================

$ patch -N -p1 < vmware-modules-2.6.29.patch

patching file vmblock-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmci-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmmon-only/include/x86apic.h

Hunk #1 FAILED at 94.

1 out of 1 hunk FAILED -- saving rejects to file vmmon-only/include/x86apic.h.rej

patching file vmmon-only/linux/driver.c

patching file vmmon-only/linux/hostif.c

Hunk #1 succeeded at 3424 (offset 1 line).

Hunk #2 succeeded at 3508 (offset 1 line).

Hunk #3 succeeded at 3632 (offset 1 line).

patching file vmmon-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmnet-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

patching file vmnet-only/netif.c

patching file vsock-only/Makefile

Hunk #1 succeeded at 122 (offset 9 lines).

=======================================

0 Kudos