It uses both.

VMFS is a special file system designed for storing large files -VMs- where as ext3 is what the console os (linux) uses to load itself, and run on.

Generally, you partition your ESX system with standard linux ext3 partions (and 1 swap) and have a VMFS file system on which to store your VMs.

I am leaving much out, but that is the general idea.