Hello,
When trying to backup using Windows, what you are doing is actually more important than what you are not. So while you may think that requires a different thread, we do not. 
So let us know, then we can be of more help. However, before you mount VMFS to a Windows box you should read the VMware Backup Guide as it covers the intricacies of mounting, etc. As one poster stated there is a good chance you may have corrupted your VMFS. But actually that issue pertains more to RDMs than the actual VMFS.
If you have ESX licenses you should also have VCB licenses, and that is the tool used to backup using a Windows server.... VCB understands the VMFS and while the VMFS is not 'mounted' the VMDK within it will be using VCB. The Windows system is made aware of LUNs available, but they are NOT mounted. Let VCB handle all that for you.
Windows does not understand the VMFS, there are no tools that will understand a VMFS outside of ESX. The mount-vmfs tool you mentioned only works within ESX and NOT a standard Linux system. And as of today no one has written any tool that does understand the VMFS as VMware has yet to release the format.
You can use tools like dd or other disk duplicators to backup an entire VMFS, but the VMFS can not be mounted only the LUN should be available. However, that is not the same as understanding the VMFS. Nor do I think you could restore it easily or at all. Not sure of anyone who is doing this or supporting it. VMware definitely does not support it.
Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354
As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization
You can read vmfs volumes from an esx server without iscsi:
1) Enable root access via ssh:
go to /etc/ssh in the esx console, type "vi sshd_config" to edit this file. press "esx" to switch to edit mode and change the line "PermitRootLogin no" to "PermitRootLogin yes". Press "esc" again and type ":wq" to write & quit.
2) restart ssh service: "service sshd restart
3) enable service via firewall: "esxcfg-firewall -e sshServer"
4) install a tool like "Sftpdrive" on your windows machine, and add a new connection to your esx using its ip, root and root-password, setting directory to "show entire server". this will include the esx file system as a network drive (e.g. H:\) to your windows machine
5) go to e.g. H:\vmfs to see all storages of your esx server. if you have an storage called "mystorage", there will be a folder with this name. if you cannot access this folder, try one of the folders with a long number as name.
Using the "VMware Virtual Disk Development Kit" you can even mount a vmdk-file as windows-drive. like this you can access e.g. "H:\vmfs\mystorage\winxp\winxp.vmdk" via "J:\" and browse inside of the vmdk file.
Have fun