Hi All,
Like most people I have having a play with the new vSphere stuff, but i have hit a problem with the setup of iSCSI.
The background here is, we have 3 Dell R805 servers with 12 Nics. Connected to 3 Dell PS5000 SANS. What im trying to do on our test server is Configure 4 of those Nic ports for conenctivity to our san.
I have followed the vSphere iSCSI guide to the letter up until page 35, where it talks about Jumbo Frames. This is where im having the issues.
Create a Jumbo Frames-Enabled vSwitch
Configure a vSwitch for Jumbo Frames by changing the MTU size for that vSwitch.
Procedure
1 To set the MTU size for the vSwitch, run the vicfg-vswitch -m <MTU> <vSwitch> command from the
vSphere CLI.
This command sets the MTU for all uplinks on that vSwitch. The MTU size should be set to the largest
MTU size among all the virtual network adapters connected to the vSwitch.
2 Run the vicfg-vswitch -l command to display a list of vSwitches on the host, and check that the
configuration of the vSwitch is correct.
This part I have done succesfully and vSwitch1 shows as being MTU 9000 enabled.
Its this next part that is proving difficult to work out.
Create a Jumbo Frames-Enabled VMkernel Interface
If you are using ESX host, you must use the command-line interface to create a VMkernel network interface
that is enabled with Jumbo Frames.
Procedure
1 Log in directly to the console of the ESX host.
2 Use the esxcfg-vmknic command to create a VMkernel connection with Jumbo Frame support.
esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name>
3 Run the esxcfg-vmknic -l command to display a list of VMkernel interfaces and check that the
configuration of the Jumbo Frame-enabled interface is correct.
4 Check that the VMkernel interface is connected to a vSwitch with Jumbo Frames enabled.
5 Configure all physical switches and any physical or virtual machines to which this VMkernel interface
connects to support Jumbo Frames.
If i run the esxcfg-vmknic -l command, it shows the 4 nics I have as MTU 1500. I am confused as to what I am actually doing here. what is the esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name> command actually doing ?.. No matter what I put as the Port Group Name it doesnt work.. All i get when I run the command is the list of options again.
So at this point my testing has come to a stop. How do I get my NICS to show as 9000, and why does the documentation provide commands that dont work ?
I should point out, that we have been running 3.5 succesfully for 12 months, so im not a total newbie to ESX.
Sorry for the long winded post, but i thought it best to explain it properly.
Cheers
Paull
C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-vmknic.pl -l -h 10.63.2.202
Interface Port Group/DVPort IP Family IP Address Netmask MAC Address
MTU Type
vmk2 iSCSI2 IPv4 10.63.5.106 255.255.255.0 00:50:56:72:89:2e 1500 STATIC
vmk3 iSCSI3 IPv4 10.63.5.107 255.255.255.0 00:50:56:7d:22:8b 1500 STATIC
vmk4 iSCSI4 IPv4 10.63.5.108 255.255.255.0 00:50:56:71:84:f1 1500 STATIC
vmk0 iSCSI1 IPv4 10.63.5.105 255.255.255.0 00:50:56:70:85:ae 1500 STATIC
It looks like you still have vmknic as mtu = 1500. the only way to get ti to 9000 is to remvoe the port groups iscsi1~4, and readd them with "-m 9000" in the esxcfg-vmknic -a cmd.
try that, and let us know how it goes. also a esxcfg-vswitch output would be good to confirm your vswitch is 9000 mtu.
- Kun
You might have already done this but ave you enabled Jumbo frames on the physical side of your network? Jumbo Frames have to enabled on all devices end to end-
If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
Hi.. Yes, Jumbo frames are enabled on our Physical switches ( i.e the ones on our storage network ), and also on the DELL SANS.. The only place that its not enabled is as above on the NICS in the ESX Host. The vSwitch that houses the 4 Physical NICS is also Jumbo enabled.
Hope this can help! Similar setup -> http://malaysiavm.com/blog/performance-tuning-on-virtual-infrastructure-with-md3000i/
Regards,
MALAYSIA VMware Communities http://www.malaysiavm.com
CLL SYSTEMS http://www.cllsystems.com
#if you found this or any other answer useful please consider allocating points for helpful or correct answers ***
So at this point my testing has come to a stop. How do I get my NICS to show as 9000, and why does the documentation provide commands that dont work ?
Do you have the license for distributed vSwitches?
If yes is better that your set the MTU on the DVS using VIC.
Andre
**if you found this or any other answer useful please consider allocating points for helpful or correct answers
Jason... I have done that and the MTU on the vSwitch shows as 9000... But there Physical NICS still show as 1500..
Ive now done a test, and have checked the Dell Logs and yes the NICS are connecting at 1500 not 9000, so having the vSwitches MTU set to 9000 makes no difference at all..
Andre, no I dont have licencing for Distributed switches..
Surely there is a way to set the MTU on the Nics...
At physical NIC (esxcfg-nics command) there isn't a parameter to change MTU.
But what's the output of esxcfg-nics -l ?
Andre
**if you found this or any other answer useful please consider allocating points for helpful or correct answers
there is one missing step between "create vswitch" and "add vmknic", that's "create a port group". that's why you are getting error, what ever "prt group" name you are passing to
esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name>
since you already has vswitch created, try these steps:
1) esxcfg-vswitch -A pg1 vSwitch?
// useyour vswitch name
2) esxcfg-vmknic -a -i DHCP -m 9000 "pg1" "vmkernel 4"
// use static ip if you what with -i ip -n netmask
3) use esxcfg-vmknic -l to verify vmknic MTU
e.g.
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk4 pg1 IPv4 169.254.0.1 255.255.0.0 169.254.255.255 00:50:56:7d:f0:2f 9000 65535 true DHCP
4) use esxcfg-vswitch -l to verify vswitch MTU
e.g.
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch3 64 4 64 9000 vmnic3
5) use esxcfg-nics -l to verify vmnic MTU
e.g.
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic3 43:00.01 e1000e Up 1000Mbps Full 00:1b:78:59:1a:df 9000 Intel Corporation NC360T PCI Express Dual Port Gigabit Server Adapter
6) rescan for sw iscsi adapter, you should be using jumbo now.
Cheers,
- Kun
Hi All, I appreciate the help everyones given me so far, but im still stuck here... Ive completed the steps from Kun above... This is the current situation -
C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-vmknic.pl -l -h 10.63.2.202
Interface Port Group/DVPort IP Family IP Address Netmask MAC Address MTU Type
vmk2 iSCSI2 IPv4 10.63.5.106 255.255.255.0 00:50:56:72:89:2e 1500 STATIC
vmk3 iSCSI3 IPv4 10.63.5.107 255.255.255.0 00:50:56:7d:22:8b 1500 STATIC
vmk4 iSCSI4 IPv4 10.63.5.108 255.255.255.0 00:50:56:71:84:f1 1500 STATIC
vmk0 iSCSI1 IPv4 10.63.5.105 255.255.255.0 00:50:56:70:85:ae 1500 STATIC
C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-nics.pl -l -h 10.63.2.202
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic4 23:00.0 igb Up 1000Mbps Full 00:1b:21:20:3e:f8 9000 Intel Corporation 82575GB Gigabit Network Connection
vmnic5 23:00.1 igb Up 1000Mbps Full 00:1b:21:20:3e:f9 9000 Intel Corporation 82575GB Gigabit Network Connection
vmnic8 29:00.0 igb Up 1000Mbps Full 00:1b:21:20:40:08 9000 Intel Corporation 82575GB Gigabit Network Connection
vmnic9 29:00.1 igb Up 1000Mbps Full 00:1b:21:20:40:09 9000 Intel Corporation 82575GB Gigabit Network Connection
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 64 9 64 9000 vmnic9,vmnic8,vmnic5,vmnic4
PortGroup Name VLAN ID Used Ports Uplinks
iSCSI1 0 1 vmnic4
iSCSI4 0 1 vmnic9
iSCSI3 0 1 vmnic8
iSCSI2 0 1 vmnic5
PG1 0 0 vmnic9,vmnic8,vmnic5,vmnic4
Any more clues as to where the issue is ??
Cheers
C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-vmknic.pl -l -h 10.63.2.202
Interface Port Group/DVPort IP Family IP Address Netmask MAC Address
MTU Type
vmk2 iSCSI2 IPv4 10.63.5.106 255.255.255.0 00:50:56:72:89:2e 1500 STATIC
vmk3 iSCSI3 IPv4 10.63.5.107 255.255.255.0 00:50:56:7d:22:8b 1500 STATIC
vmk4 iSCSI4 IPv4 10.63.5.108 255.255.255.0 00:50:56:71:84:f1 1500 STATIC
vmk0 iSCSI1 IPv4 10.63.5.105 255.255.255.0 00:50:56:70:85:ae 1500 STATIC
It looks like you still have vmknic as mtu = 1500. the only way to get ti to 9000 is to remvoe the port groups iscsi1~4, and readd them with "-m 9000" in the esxcfg-vmknic -a cmd.
try that, and let us know how it goes. also a esxcfg-vswitch output would be good to confirm your vswitch is 9000 mtu.
- Kun
I think you misinterpretted Kun's recommendation. He wasn't recommending you recreate a new portgroup, just updating the ones you have.
2) esxcfg-vmknic -a -i DHCP -m 9000 "pg1" "vmkernel 4"
// use static ip if you what with -i ip -n netmask
should apply to the port groups you're using. You've created a new port group PG1, which is not what he was intending So,
1) esxcfg-vswitch -D PG1 <your vswitch>
to remove port group PG1
2) esxcfg-vswitch -m 9000 -p iSCSI1 <your vswitch>
esxcfg-vswitch -m 9000 -p iSCSI2 <your vswitch>
esxcfg-vswitch -m 9000 -p iSCSI3 <your vswitch>
esxcfg-vswitch -m 9000 -p iSCSI4 <your vswitch>
to change the MTU on your existing portgroups .
3) Remove and re-add your vmknics
esxcfg-vmknic -d iSCSI1
esxcfg-vmknic -d iSCSI2
esxcfg-vmknic -d iSCSI3
esxcfg-vmknic -d iSCSI4
esxcfg-vmknic -a -i 10.63.5.105 -n 255.255.255.0 -m 9000 iSCSI1
esxcfg-vmknic -a -i 10.63.5.106 -n 255.255.255.0 -m 9000 iSCSI2
esxcfg-vmknic -a -i 10.63.5.107 -n 255.255.255.0 -m 9000 iSCSI3
esxcfg-vmknic -a -i 10.63.5.108 -n 255.255.255.0 -m 9000 iSCSI4
Re-veirfy using the what Kun recommends in steps 3-6.
Let us know of any other problems.
Andy
<span class="jive-thread-reply-body-container">It looks like you still have vmknic as mtu = 1500. the only way to get ti to 9000 is to remvoe the port groups iscsi1~4, and readd them with "-m 9000" in the esxcfg-vmknic -a cmd.
No need to remove and recreate the portgroups. You can change the mtu on those (see my previous response).
You do need to remove and recreate the vmknics.
Andy
OK Guys, Im now good on all 3 fronts.. Nics are showing as 9000 in both VMKNIC and ESXCFG-NICs as is the vSwitch... The only issue now is that its still only connecting at 1500 on the SAN.. Im sure its all enabled correctly on the physical switches as well as the SAN, so ill do some more investigation... But thanks to all that helped so far... Great community !