This is a very interesting subject and I think there are ways for you to dump the memory of the VM into the .vswp file provided or even into a .vmss file. The first requires you to send signals to the appropriate aspect of the VM and the second requires you to just suspend the VM, the suspension will create a memory file with the entire contents in it.
Once vSphere is released, VMsafe should do what you want to do as well.
Indeed, the suspend (and the corresponding .vmss) file is the best way. A snapshot (and the corresponding .vmsn file) also work, the file is essentially the same. The contents of memory start at an offset within the file, and are stored as raw contents of physical memory (e.g. not with virtual addresses). The .vswp files do not contain a deterministic set of pages (e.g. memory that doesn't need to get paged out won't be written to the .vswp).
And yes, VMsafe allows all this as well.
For online scanning, VMsafe is the only option (and the VMsafe API could also disallow execution based on the results of the scan) and is helpful because you can use VMsafe APIs to look up the page table and thus perform the analysis in the virtual address space as well as the physical address space. For offline scanning, the snapshot is certainly more efficient, but coming up with the current page table is considerably more difficult, so the patter scanning has to be in the physical address space only.
