Posting it here to share with the larger community and to also get some feedback if it can be improved.
Setting up Fedora 15/16 system
For this tutorial, we use either Fedora 15 or Fedora 16 on the x86_64 architecture (64bit). The initial release of Fedora 15 used a kernel that still had BKL and hence Rado Cotescu's wrapper scripts still worked. But the kernel changed in Fedora 15 through updates and the latest F15 kernel no longer has BKL, which broke VMware Server. The tutorial here, along with the updated code patches will fix this problem. So our first steps are:
-  Install Fedora 15/16 x86_64 on hardware with virtualization support.
-  Apply all current updates with “yum -y update”.
Ensure that your installation includes 'make', 'patch', the C compiler 'gcc', and the kernel source and headers. These packages are usually named 'make', 'patch', 'gcc', 'kernel-headers', and 'kernel-devel'. When you install those via the 'yum' utility, their dependencies will be pulled in. These are needed to patch the code and rebuild the kernel modules for VMware server. You can install these tools with:
-  Install packages with:
# yum -y install make patch gcc kernel-headers kernel-devel
I'm sure there is a way to setup SELinux to work with VMware Server, but I haven't figured that one out and that's not an obstacle I want to to deal with just yet. SELinux is enabled by default, so in the interim, we disable SELinux:
-  Disable “SELinux” by editing /etc/sysconfig/selinux and rebooting.
Next, there is a problem with VMware and the NMI watchdog feature of certain Linux kernels. (See http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=924) Check to see if you have this feature enabled by running this command:
$ cat /proc/sys/kernel/nmi_watchdog
If you get a '1', then the feature is enabled and must be turned off or VMs will mysteriously crash. To turn off nmi_watchdog is different for Fedora 15 and Fedora 16:
-  Fedora 15: Edit /boot/grub/grub.conf and add “nmi_watchdog=0” to the kernel line.
-  Fedora 16: Edit /etc/default/grub and add “nmi_watchdog=0” to the GRUB_CMDLINE_LINUX. Then run:
# grub2-mkconfig -o /boot/grub2/grub.cfg”
to rebuild the grub configuration.
Now, reboot the system and check the /proc/sys/kernel/nmi_watchdog parameter again to make sure you see a '0'.
The final step is to open up the necessary ports on the firewall or disable the firewall completely. The three ports that need to be opened are 902/TCP, 8222/TCP, and 8333/TCP.
-  Care must be taken in how you add the firewall rules. The iptables commands below are just guidelines.
# iptables -I INPUT 5 -p tcp --dport 902 -m state --state NEW -j ACCEPT # iptables -I INPUT 6 -p tcp --dport 8222 -m state --state NEW -j ACCEPT # iptables -I INPUT 7 -p tcp --dport 8333 -m state --state NEW -j ACCEPT
Installing VMware Server
VMware server is available from www.vwmare.com in both a .tar.gz and RPM format. Since we are on Fedora, download the RPM format and then install it:
-  Install VMware server RPM package:
yum localinstall /path/to/VMware-server-2.0.2-203138.x86_64.rpm
Normally, at this point, you can run 'vmware-config.pl' to finish the setup. But unfortunately, if you went ahead and did that, you would run into all sorts of problems like the Perl script not being able to detect your kernel sources and the missing “smp_lock.h” header file which was part of BKL that is no longer available. In order to fix these problems, we need to perform a few more steps.
First, download the patches (http://www.jlbond.com/downloads/vmware-server-2.0.2-203138-linux_2.6.39+.tar.bz2) and untar the archive:
-  Download the archive:
# wget http://www.jlbond.com/downloads/vmware-server-2.0.2-203138-linux_2.6.39+.tar.bz2
-  Untar the archive:
# tar jxf vmware-server-2.0.2-203138-linux_2.3.39+.tar.bz2
You will find two files: vmware-config.pl.patch and vmware-server-2.0.2-203138-linux_2.6.39.patch. The first one is to fix the vmware-config.pl script and the second one is to patch the VMware kernel module sources. The 2nd patch is based on Rado Cotescu's patch but with additional fixes for some deprecated syntax and naively replaces BKL with mutexes.
-  patch vmware-config.pl script:
# patch -d /usr/bin -p0 < /path/to/vmware-config.pl.patch
Patching the kernel modules is a bit more tedious (at some point, i may write a script to do this):
-  patch the VMware module sources:
# pushd /usr/lib/vmware/modules/sources # mkdir backups; cp -a *.tar backups/ # mkdir working; ls *.tar | xargs -n 1 tar -C working/ -xf # pushd working # patch -p1 < /path/to/vmware-server-2.0.2-203138-linux_2.6.39.patch # tar cf ../vmci.tar vmci-only # tar cf ../vmmon.tar vmmon-only # tar cf ../vmnet.tar vmnet-only # tar cf ../vsock.tar vsock-only # popd; rm -rf working/ ;popd
Next, there appears to be a problem between the bundled libxml2 with the system's zlib package. Either you can use an older zlib library or you can make vmware use the system's libxml2 which doesn't have problems with zlib. (see http://communities.vmware.com/message/1670102) We will move the bundled libxml2 out of the way so that vmware uses the native libxml2:
-  Move bundled libxml2 out of the way so vmware uses native libxml2:
# pushd /usr/lib/vmware/lib; mv libxml2.so.2 SKIP-libxml2.so.2 ; popd
Finally, run 'vmware-config.pl' and answer the questions to finish up the installation. Once VMware server starts up, you should be able to access the management console at https://<your_ip_address>:8333.
Remark 1: Certain browsers like Firefox don't always work with VMware server's HTTPS. If you experience such problems, disabling TLS (and forcing SSL3) seems to help. Or, you can try another browser like Chrome.
Remark 2: When using NAT or HostOnly networking, the interface counters show enormous numbers in the packet counters. Since I mostly use Bridged networking which seems to work fine, I haven't looked into this further.