Sorry, I wasnt clear in my post, I agree I would always use 2 links for the iSCSI or NFS link, ideally to 2 different switches, but if you only have one gigabit port then we're stuck.
If money is an option you can get cheap gigabit switches but they aren't great at load-balancing and VLANing a lot of the time. For Vmware I would strongly recommend putting all ports on gigabit.
Drop me a private message if you need come consultancy help on this if you are in the UK