Have been experiencing this very issue with ESXi 4.1 U2.
Had a look in my /var/log/messages and discovered FS3 error messages described below.
I believe in 4.x and 5.x the limit was 32 VMs that can have a file handle open. In 5.1 that increases to (I believe) 128. Also, in 4.x and 5.0, no more than 8 hosts can access a given file. In 5.1 this is increased to 32.
It is not, however, a tuneable parameter.
A better way to do what you are attempting might be PXE boot - no limit on that.