I think BootCamp would be a step backwards unless Apple has added 64-bit support for XP64 or Vista64.
I believe Boot Camp supports 64-bit on recent Mac Pros.
Maybe I'll have to check out parallels
You should double check (e.g. on Parallels' forums), but I believe Parallels Desktop 3.0 is limited to 2 GB per virtual machine (wikipedia thinks so, as does searching their forums).
or crossover for this one
I'm not that familiar with Crossover's limitations, but I doubt they're better - they're geared more towards getting particular Windows apps working without Windows, but the tradeoff is compatibility; I doubt high performance computing like what you're trying to do is on their radar. I would start at http://www.codeweavers.com/support/
although I thought I read that the ESX VM servers could assign more than 8 GB of RAM so it doesn't appear to be impossible.
ESX is a totally different beast. It can do more than 8 GB RAM per virtual machine, but it's also a bare metal architecture as opposed to a hosted architecture.