(see attached .doc as it has screen shots included)
After buying some HP7170's duel gigE cards on Ebay, I was determined to learn all I could about setting up the NICs to support the jumbo frame feature that was added to the 3.5 release of VMware ESX. I found out my test Linksys switch didn't support jumbo frames. But my goal was to make two VMs on the same vSwitch ping each other w/ jumbo frames. My VMs were both Windows 2003 Standard Edition.
Keep in mind all virtual and physical switches between the virtual or physical servers must also support jumbo frames for this to really be useful. If they are not the packets will get dropped or fragmented and the benefit of Jumbo packets will be lost.
The first thing you must do is edit the virtual switch to handle an MTU of 9000. For this you must use the service console as the VIC server has no ability to set this currently. Open a Putty session as Root and do a:
root@vm01 /# esxcfg-vswitch -m 9000 vSwitch0
To confirm the vSwitch do a "esxcfg-vswitch -l"
root@vm01 /# esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 64 8 64 9000 vmnic1,vmnic0
PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 3 vmnic0,vmnic1
Service Console 0 1 vmnic0,vmnic1
Secondly, on each VM you must add a new network interface to your VM that is of type ‘Enhanced vmxnet'. To do this the VM must be powered off. I already had 2K3 with the 2.5 VMware tools installed. This ensures the drivers for the Enhanced drivers will be pre-loaded. There is also no ability via the VIC GUI to change ‘flexible' type adapter to a ‘Enhanced vmxnet' adapter. I recorded the servers IP address (actually just a static host entry in my test network). I then removed the previously installed ‘flexible' adapter. I then attempted to add a new network card, but I only saw ‘flexible' type available to me. Why?!.
I actually called VMware for support, while very helpful, the tech and I both overlooked this fact which I later found online:
Enhanced VMXNET is supported only for a limited set of guest operating systems:
32/64-bit versions of Microsoft Windows 2003 (Enterprise and Datacenter Editions)
32/64-bit versions Red Hat Enterprise Linux 5.0
32/64-bit versions SUSE Linux Enterprise Server 10
32-bit versions Red Hat Enterprise Linux 4.0
I then changed my OS type from Windows 2003 Standard to Windows 2003 Enterprise. To do this simply click: Edit Virtual Machine> Options Tab> and change the Guest OS type to 2k3 Enterprise Edition.
I could then add the ‘Enhanced vmxnet' adapter to the VM.
I powered up the VM and the driver was loaded, I reset the IP information on the new NIC. I did get a warning the IP address was tied to the NIC that was now uninstalled/disabled (as I removed it from the VM) but this is not a problem.
The Next setup is to setup the MTU size on the virtual NIC driver within Windows 2003 VM. To do this pick Start > Control Panel > Network Connection > Local Area Network Connection 2 . Click Properties on the General tab > Configure on the "VMware PCI Ethernet Adapter" then click on the Advanced Tab and chance the MTU Property to 9000.
Repeat this on your second VM which was running on the same host.
Once done with the second VM, Finally you can test by sending custom created ping packets from virtual server to virtual server.
C:\Documents and Settings\Administrator>ping -f -l 8750 is01
Pinging is01 http://192.168.1.203 with 8750 bytes of data:
Reply from 192.168.1.203: bytes=8750 time<1ms TTL=128
Reply from 192.168.1.203: bytes=8750 time<1ms TTL=128
Reply from 192.168.1.203: bytes=8750 time<1ms TTL=128
Reply from 192.168.1.203: bytes=8750 time<1ms TTL=128
Ping statistics for 192.168.1.203:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
This is telling the ping command to make a packet size (-l) 8970 and not to fragment it (-f). IF you get a response you can be assured your 2 VMs can talk to each other using jumbo packets. I was able to send pings up to about 8970 (as some of the IP headers will take a few bytes in the MTU payload). You can try slightly larger or smaller numbers but should get a PING reply and it should represent the payload you chose. Success.
I then powered down my VMs and move the OS option type back to Windows 2003 Standard.
I assume one could edit the VM configuration file directly to do this but this way is pretty easy. Keep in mind this is NOT supported w/ Windows 2003 Standard (I have no idea why) but might aid you in getting this configured in a test environment. In my test environment this is an OK risk.
Useful links:
VMware VCP 3.5
VMware VCP 4.0
VMware VCP 5.0
