Which Linux distro is this?
I've had solid sucess using YUM to install all the kerel devel/header items... You could be missing one component that's screwing the pooch there. Have you tried logging off, then back on and trying again?
I've not had any issue with installing the tools from ESXi 4.1 onto CentOS through the latest version 5 release (updated to the latest via YUM)... I plan on setting up a CentOS 6 VM soon to see how it compares (for my needs). I'm evaluating ESXi 5 now, so I'll be installing the tools from that release onto the VM...
I just figured out a similar issue yesterday with one of my older Debian Linux guests; not sure if that's the distro you're working with, but perhaps the steps will help you figure things out anyway:
1. Log into the console as root
2. Check the kernel version:
# uname –r
which in this case is 2.6.26-2-686
3. Install the appropriate headers for the installed kernel:
# apt-get install linux-headers-2.6.26-2-686
4. Attach the VMware tools cdrom ISO (or just choose to install it on the guest and it will automatically attach the ISO)
5. Mount the cd into linux:
# mount /dev/cdrom1
6. Copy the installer to a temporary location
# cp /media/cdrom0/VMwareTools-8.3.7-433358.tar.gz ~
7. Go to the temporary location:
# cd ~
8. Decompress and untar the installer:
# tar –xzf VMwareTools-8.3.7-433358.tar.gz
9. Delete the original file:
# rm VMwareTools-8.3.7-433358.tar.gz
10. Go to the installer location:
# cd vmware-tools-distrib
11. Link the headers file to the right location for the VMware Tools installer:
# ln -s /usr/src/linux-headers-2.6.26-2-686 /usr/src/linux
12. Change the gcc version to use:
# export CC=/usr/bin/gcc-4.1
13. Start the installer:
14. Accept all the defaults all the way through.
15. Delete the installer folder:
# rm –rf vmware-tools-distrib
16. And you’re done!
This is based on instructions I found here: http://www.visoracle.com/vm/debian40/installvmwaretools.html