It uses both.
VMFS is a special file system designed for storing large files -VMs- where as ext3 is what the console os (linux) uses to load itself, and run on.
Generally, you partition your ESX system with standard linux ext3 partions (and 1 swap) and have a VMFS file system on which to store your VMs.
I am leaving much out, but that is the general idea.
