Hi all,
For your information:
Environment:
1 * ESX 3.5 Upodate 2
VC 2.5 Update 3
2 * Dell PE2950
1 * Dell MD3000i (with latest firmware); 2 Volumes are created; iSCSI-ports are redundant configured.
2 * PowerConnect 5*
All VM's are configured on 1 GB speed "autonigociate) as on the PowerConnect Switches.
Guest-OS on VM's is installed with Ubuntu (was Gentoo before).
I have this customer (International school) that provide students to upload and download movies (student material). A local copy of a movie of 800 GB takes ages and even makes it impossible to do anything else on other Virtual Machines.
Does anyone have experience in this area?
What do you advice to use for uploading and downloading big files in a virtualized environment?
Maybe it is not enough information but a start to get some answers.
Hope to get good advice.
Cheers,
Jim Smits
VCP
Sounds like you have networking issues. It's not related to a VM, we have VM's attached to Gig interfaces, and they don't suffer any bandwidth issues. 800 Meg is quite a large file, so how would you handle this in a physical machine?
Have you tested this thoroughly BEFORE you attempted virtualization? It seems like you came up with this solution after you started using VM's, so it's really not a VM issue, its your infrastructure.
We need more info, where are the files coming from, Internet? What kind of networks do you 10/100 10/1000? What kind of NICS. Do they share bandwidth with many other VM's ALL downloading the same 800 Meg file at once? It might be a better solution to centralize the location of the files rather than each VM downloading separately.
Hi there ,
Thanks for your reply.
Question 1:
Have you tested this thoroughly BEFORE you attempted virtualization? It seems like you came up with this solution after you started using VM's, so it's really not a VM issue, it's your infrastructure.
Answer 1:
The customer have set up VMware them selves. The hardware MD3000i is installed and configured together with an 3rd party company. So unfortunately we were not involved at all from the start.
Question 2:
We need more info, where are the files coming from, Internet? What kind of networks do you 10/100 10/1000? What kind of NICS. Do they share bandwidth with many other VM's ALL downloading the same 800 Meg file at once? It might be a better solution to centralize the location of the files rather than each VM downloading separately.
Answer 2:
The files are coming from the webserver (stored by the administrator).
All NIC's are configured on 1000 Auto negotiate.
The files are centralized.
Question 3:
We need more info, where are the files coming from, Internet? What kind of networks do you 10/100 10/1000? What kind of NICS. Do they share bandwidth with many other VM's ALL downloading the same 800 Meg file at once? It might be a better solution to centralize the location of the files rather than each VM downloading separately.
Answer 3:
The customer is using internal 1000 networking (see drawing).
Yes, they do share bandwidth with the other (about 20) VM's but they are not downloading the same file!
Types of NIC's:
NIC 0 Broadcom NetXtreme II BCM5708; driver bnx2
NIC 1 Broadcom NetXtreme II BCM5708; driver bnx2
NIC 2 Intel 82571 EB Gigabit Ethernet Controller; driver e1000
NIC 3 Intel 82571 EB Gigabit Ethernet Controller; driver e1000
NIC 4 Intel 82571 EB Gigabit Ethernet Controller; driver e1000
NIC 5 Intel 82571 EB Gigabit Ethernet Controller; driver e1000
Other information
Switches:
PowerConnect 5324 (no iSCSI optimization as on the PowerConnect 54**)
Up- and Download servers:
There are 2 Webservers, Guest OS is UBUNTU Server 8.0.4.
MD300i:
The MD3000i is re-installed and configured following the best practice.
15 * SAS 300 GB 15K
The internal copy is taken place on the same server!!! So I can imagine that on a 1GB connection with a speed of 100MB per second (read and write) can take approximately 4 hours.
It hangs up the total environment!
The 2 Webservers are having its own set of disks.
Webserver 1 has a RAIDset of 5 disk in a RAID5.
Webserver 2 has a RAIDset of 4 disks in a RAID5.
There is one hotspare and the rest of the disks being used for Virtual Machines.
Additional info:
Object Total
Shared Datastores 3
Datastores 4
Hosts 3
Virtual Machines 32
Operating System Running Total
Microsoft W2K3 SE 32-bit 5 5
Microsoft W2K3 EE 32-bit 9 10
Microsoft W2K8 32-bit 0 1
Ubuntu 11 15
Suse Linux ES 32-bits 0 1
What I am thinking about:
1. I would like to use VLAN for these WebServers.
2. Configure the PowerConnect 5324 and NIC's for these VM's for Jumbo Frames.
Is there something else from this point what you can suggest?
Looking forward to your answer.
Regards,
Jim
I agree that you should test non-virtual before you try virtual.
Then I would do the following on your Virtual Machine
Make SURE you're using the LSI scsi driver
If the movies are on a VMDK and not an RDM, make sure that you align the partition when you create it using diskpart
create partition primary align=64 (This can make a huge I/O difference)
Make sure you're using the vmxnet NIC driver
Increase your TCP window and MTU sizes
Consider using a file transfer protocol other than SMB/CIFS
Consider archiving the larg file into a collection of smaller files that are easier to resume
These are just best practices I would reccomend before you start too deep into troubleshooting.
If it was useful, give me credit
Jason White - VCP
Assuming that the 800GB isn't a typo, bandwidth to the storage could easily be the problem. iSCSI can handle IOPS just fine, but it limited by the 1Gb wirespeed. For that kind of bandwidth requirements I would recommend that the customer switch to FC based storage or 10GbE iSCSI. You can check this in esxtop or check utilization on the switches.
How does your storage look when you are doing a transfer? Disk utilization, proc and mem performance, too... I would frankly be surprised to find that your environment is being brought to its knees by NIC throughput. My guess is that it's something storage related. Someone mentioned disk alignment which is something to look into as well.
Whenever I've exerienced this kind of issue myself, it's always been storage related. We use NFS but it still uses copper like iscsi does.
It'll be interesting how this turns out...
If you found this posting to be useful, great. I didn't waste your time.