We experienced performance issues with Windows x64 and SQL 2005 standard on physical servers, so the same would apply to VMs. Here's what we found...
On a 32 bit system, the file system cache working set is essentially limited to 1 GB. This is the maximum size that is blocked off in the kernel for the system cache working set. Since most systems have more than 1 GB of physical RAM today, having the system cache working set consume physical RAM with read I/O is less likely.
This scenario; however, is more prevalent on 64 bit systems. With the increase in pointer length, the kernel's address space is greatly expanded. The system cache's working set limit can and typically does exceed how much memory is installed in the system. It is much easier for applications and drivers to load up the system cache with read I/O. If the demand is sustained, the system cache's working set can grow to consume physical memory. This will push out other process and kernel resources out to the page file and can be very detrimental to system performance.
The work arounds for this problem are either limit your application to a set amount of physical memory (we found out after the fact that SQL 2005 Enterprise has this ability) or set the system file cache less than the amount of physical RAM. This requires programming skills utilizing some newer APIs.
You didn't say what applications your server is running, but thought I'd toss this relatively unknown "feature" of x64 out there.
