VMware Cloud Community
bjselman
Contributor
Contributor
Jump to solution

ESX 3.5 Networking configuration

Just wanting to confirm a few slight adjustments before I continue moving on....I'm having a backtrack moment:

I have attached my current vswif -l, vmknic -l, vmswitch -l and vmnics -l configurations. Does this look ok from a pure network config standpoint? I have these thatched to two ProCurve 2900's for complete redundancy. So far, I have successfully tested connectivity between everything and VMotion and migration, HA, DRS, etc all seems to work on the few test VM's I made.

The only thing I originally did was put my ESX servers on the 10.80.2.x subnet (same as iSCSI SANs), but I am wondering if I should move it over to the 172.17.100.x. Either that or move my SANs over to another completely different subnet (say, 10.88.100.x). Anyone having issue's with routing the ESX servers and the SANs on different subnets. That is ideal to me, but I've heard yah's and nah's on both sides. ("completely isolated" not equal to "allowed routing", etc, etc....) I am much more interested in this from a performance standpoint (there are plenty other security techniques that can be applied).

Which would be best suited (for performance) in the whole environment to alter at this point (if any)? 1. Change ESX Servers to 172.17.100.x? or 2. Put SAN on new 10.88.100.x subnet? I just remember that messing up in the VI Client so many times pretty much required me to learn the ESX command lines and it took some figuring out to get it all going...

BTW, environment = VI3, 3.5, 2.5 on two DL380 (6 nics each) w/2 LH NSM 2060's w/solutions pack

Edited: forgot to mention that my production network is completely segmented from either of the subnets mentioned above.

Reply
0 Kudos
1 Solution

Accepted Solutions
kjb007
Immortal
Immortal
Jump to solution

I wouldn't put them on standby, override failover, but just reverse the top/bottom order and have them both as active. That way you'll get the most bang for you buck.

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB

View solution in original post

Reply
0 Kudos
18 Replies
Texiwill
Leadership
Leadership
Jump to solution

Hello,

The only thing I would change in your setup is to NOT place the VM Network on the same vSwitch as the main SC.

I would instead do:

vSwitch0 -> vmnic0,vmnic2

ServiceConsole

vMotion

vSwitch1 -> vmnic4,vmnic5

iSCSI

vSwitch2 -> vmnic1,vmnic3

VM Network

Do not allow your VMs to reside on the same vSwitch as your SC, that is generally a huge security flag. I would also setup a route within the Service Console to get to the iSCSI network for chap, no need for a second SC device. Your Service Console is your Administrative port and it should remain separate. It must participate in the iSCSI network but can do that through a router quite easily as all that is sent over the SC for iSCSI is CHAP for authentication purposes. Note that Administration includes migrating VMs, creating VMs, and backups. If you do all that and run your VMs on the same vSwitch you could run into bottle necks.

Can you route between your 172. and 10. networks for iSCSI CHAP traffic? This would be the safest way to go. If necessary setup an virtual router appliance that allows this using a separate vSwitch0 portgroup. An administrative VM is the exception to the rule for security reasons. Ideally you really want 8 pNICs, but 6 will work.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
bjselman
Contributor
Contributor
Jump to solution

Are you purposely placing the vMotion kernels in your setup (vmnic0,vmnic2) on separate pNic cards? (vmnic0 is on the Intel, vmnic2 is one of the onboard). That actually fares well now that I think about it, from a HA standpoint. Not sure if that was the intention, but good call.

As far putting vMotion and my ServiceConsole on the same subnet/vSwitch - hmm, I was impressioned that vMotion should be invisible - even a network administrator!! ? Completely segregated. No?

I tried setting this up with just one SC and I could never connect until I added another. I assumed it had something to do with a "heartbeat." Isn't two SC's necessary for HA, DRS, etc?

Yes I can route between any subnet. On my vSwitch with my VM's, I have the SC vLan'd out of the VM traffic, albeit on the same port. I know that five different people might do five different things, but I'm open to your idea.

Thanks for the tips!

Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

I agree with Ed on that. 1 and 2 appear to be onboard, which means if the board has a networking issue, you lose both nic's. I do the same thing. No single point of failure.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
bjselman
Contributor
Contributor
Jump to solution

ok, i swapped the vmnic uplinks for vMotion.

I added a Service Console port on vSwitch1 with vMotion. When I try to remove the ServiceConsoleVMnet from vSwitch2, it gives me this 'are you sure' comment: "Removing the VMkernel port group will make all NFS datastores mounted on it inaccessible. Do you want to continue?"

Now I'm confused because the new SC i created on the vMotion switch actually says "Service Console Port" (in VIC) whereas the SC I currently have on the VM switch says "VMkernel Port". So if I am trying to remove the VMkernel Port, then my volumes will go offline?

By adding a SC port on the vMotion vSwitch1, I now have (added SC is in bold):

root@thdal9esx2 root# esxcfg-vswif -l

Name Port Group IP Address Netmask Broadcast Enabled DHCP

vswif0 ServiceConsoleISCSI 10.80.2.102 255.255.255.0 10.80.2.255 true false

vswif1 ServiceConsoleVMnet 172.17.100.13 255.255.255.0 172.17.100.255 true false

vswif2 Service Console 172.17.100.102 255.255.255.0 172.17.100.255 true false

root@thdal9esx2 root# esxcfg-vmknic -l

Interface Port Group IP Address Netmask Broadcast

vmk2 ServiceConsoleISCSI 10.80.2.103 255.255.255.0 10.80.2.255

vmk1 ServiceConsoleVMnet 172.17.100.12 255.255.255.0 172.17.100.255

vmk0 VMotion 10.80.3.102 255.255.255.0 10.80.3.255

I know there is a command line way of doing this, but I am not that knowledeable just yet...still learning. Can I just:

esxcfg-vmknic -d vmk1

esxcfg-vmknic -a -i 172.17.100.12 -n 255.255.255.0 "Service Console"

esxcfg-vswif -d vswif1

without destroying anything? would i have to re-mount my LUNs or will they go bye bye?

Reply
0 Kudos
cmanucy
Hot Shot
Hot Shot
Jump to solution

I don't agree with the 'keep the SC on a routable interface' statement. To each their own, but in my eyes, my storage is everything. If I lose my router, I don't want to risk losing my storage (back to a single point of failure, no?). So I place SC on my iSCSI interfaces.

With Lefthand (this is one of my biggest gripes) you have to be able to get to the same iSCSI subnet for administration. I really wish they'd give another interface (I have the DL320's) for administration. So from your admin worksation(s) you'll still have to be able to route to whatever subnet you put iSCSI on. We use a ProCurve 3500zl for routing and ACL's. This kind of switch also allows for line-speed ACL's/routing, which is important if you also want to give LUNs to other machines that might not be on your ESX iSCSI subnet.

Okay, so enough of my personal opinions. Don't forget to trunk at least 2 (I have 4, just because) ports together for inter-switch communications on the 2900's. Then comes the real test... running operations, and unplugging one of the switches and watch what happens (!).

If you change the IP's on your iSCSI subnet, you're going to have to take your storage offline, or you'll have to be really creative with the LeftHands. Although you could do it hot, unless you really need to, I'd shut everything down and do it offline. Then you can make sure the LH units are talking correctly in their management group before you bring the ESX hosts and re-configure them for the new iSCSI targets. Once you rescan, you'll see all your LUNs again and be able to re-start your VM's.



----

Carter Manucy

---- Carter Manucy
Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

You will have to figure out which interface is it that you are actually using to connect to your storage. I'm not sure if LeftHand will tell you or not, which interface is making the connection. On ther other hand, if you're on the service console, you can run traceroute against your iSCSI target IP, and it should tell you which interface is being used.

The commands you added will remove your vmkernel and service console ports from vSwitch2, which you don't really need, so you're ok there. The middle command to add another vmkernel ports seems a bit redundant. Why are you trying to add another vmkernel port, and to which port group?

It looks to me now that you have:

  1. vmkernel and a service console port on your vswitch0, which will be enough for iSCSI

  2. vmkernel port for vmotion on vswitch2, and a possible service console port there also, based on your statement

  3. a vmnetwork for your vm's on vswitch3, so that's good also.

It looks like you're done. Can you post esxcfg-vswitch -l output to make sure again just to be sure of the layout?

The names are changeable, for the most part, so you have to make sure that your vswif correllate to actual service console ports and your vmk's are your vmkernel ports.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

Here is my switch config:

root@thdal9esx1 etc# esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 64 7 64 1500 vmnic5,vmnic4

PortGroup Name VLAN ID Used Ports Uplinks

ServiceConsoleISCSI 1080 1 vmnic4,vmnic5

ServiceConsoleMGMT 1001 1 vmnic4,vmnic5

VMkernelISCSI 1080 1 vmnic4,vmnic5

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch1 16 5 16 1500 vmnic0,vmnic1

PortGroup Name VLAN ID Used Ports Uplinks

VMotion 803 1 vmnic1,vmnic0

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch2 64 5 64 1500 vmnic2,vmnic3

PortGroup Name VLAN ID Used Ports Uplinks

Virtual Machine Network0 1 vmnic3,vmnic2

The only problem I'm having is not being able to ping the vMotion IP of the other ESX server from each one. I can ping the vMotion g.w. (10.80.3.5) via a "ping" command, but I can't "ping" the vMotion kernel IP (10.80.3.102) of the other server. By using "vmkping", I can neither ping the vMotion g.w. nor the other server's vMotion kernel. I tried adding route via command line. I did:

ip route change 10.80.3.0/24 via 172.17.100.5

route add -net 10.80.3.0/24 gw 172.17.100.5

esxcfg-route -a 10.80.3.0/24 172.17.100.5

(also tried routing thru VMkernel d.g.w 10.80.2.5)

Do both esx subnet (172.17.100.x) and the iSCSI subnet (10.80.2.x) need to be routed to the vMotion subnet (10.80.3.x)?

After I tried adding routes, I ended up with this (can't figure out how to remove the 10.80.3.0/24 under the "ip route show"):

(i haven't passed my linux license yet) Smiley Sad

root@thdal9esx1 etc# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

10.80.2.0 0.0.0.0 255.255.255.0 U 0 0 0 vswif3

10.80.3.0 172.17.100.5 255.255.255.0 UG 0 0 0 vswif0

172.17.100.0 0.0.0.0 255.255.255.0 U 0 0 0 vswif0

169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 vswif3

0.0.0.0 172.17.100.5 0.0.0.0 UG 0 0 0 vswif0

root@thdal9esx1 etc# ip route show

10.80.2.0/24 dev vswif3 proto kernel scope link src 10.80.2.11

10.80.3.0/24 via 172.17.100.5 dev vswif0

172.17.100.0/24 dev vswif0 proto kernel scope link src 172.17.100.101

169.254.0.0/16 dev vswif3 scope link

default via 172.17.100.5 dev vswif0

root@thdal9esx1 etc# esxcfg-route -l

VMkernel Routes:

Network Netmask Gateway

10.80.2.0 255.255.255.0 Local Subnet

10.80.3.0 255.255.255.0 Local Subnet

default 0.0.0.0 10.80.2.5

Are there files corresponding to these routes that are editable?

Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

I'm assuming that you enabled vmotin on the vmkernel port called vmotion on vSwitch2. One thing I noticed on that portgroup, is that it is configured for VLAN 803, not 1080, which tells me that either there is confusion with which VLAN is doing which, or there is a router available on your service console interface that is not on the other.

Also, you changed the names from your previous post, so I'm not sure which interface is which anymore. Can you post esxcfg-vswif -l, esxcfg-vmknic -l, and esxcfg-route -l.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

the vmotion goes to vlan 803 on switch. 1080 is iscsi subnet. they can't be on different subnets? vmotion is enabled on both vswitches.

yeah sorry about the renaming. i attached another doc with the configs pasted. what is wierd is that i can ping and vmkping the esx1 vmotion kernel (10.80.3.101) from esx2. and i can vmkping from esx1 to esx2.

the only thing i can't do is "ping" from esx1 to esx2 (but i can vmkping). i have doublechecked my switch and nothing out of order.

the procurve 2900 serves as my routerable between subnets and 801.2q.

Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

Ok, I think your vswif3 on both hosts, ServiceConsoleISCSI is redundant, and is not required. You should be able to disable that, and restest, and if successful, delete that interface.

They can all certainly be on different subents, so no worries there.

Your config looks good. Now, about the pinging, if you're ping'ing by IP address and failing, then that would be strange as your routing looks correct. If you're pinging by hostname, then check DNS and/or your hosts file, as maybe they're referencing different IP addresses.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

i will remove that redundant SC and test it out. seems like i tried that and didn't work. its not needed for HA, DRS, etc?

Reply
0 Kudos
francois_tiers
Enthusiast
Enthusiast
Jump to solution

redundant SC is not needed but recommanded...

Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

Redundant sc consoles are a good practice, to save yourself from an isolation resposne scenario in case you lose your service console connection. Since you're using 2 NICs already, you have a redundant service console IP. You can add another one, if you like, but I think you may run into routing problems in the future. Typically, you'd want to put your redundant SC connection in another VLAN to protect against VLAN failures, so I think in this case, it is not needed.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
nanair01
Enthusiast
Enthusiast
Jump to solution

Hi

I have got a small doubt. But y do we have to put both the Vmotion n/w and SC n/w connected to the same VMnic. Actually we need to connect VMKnic and Vmotion n/w together right?

If you find this post helpful/rectify your problem do not forget to award points
Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

You don't bind your portgroups to a nic, but to a switch, and you bind your switch to your NICs. I don't think I understand your question. A vmkernel port (VMKnic) is required for vmotion, and a service console (vswif) port is required for SC. You can separate everything if you want to, depends on how much I/O your servers are producing.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
bjselman
Contributor
Contributor
Jump to solution

i have redundant SC's on different VLANs, albeit on the same vSwitch. Since i have bonded nics along the entire path, the only SPoF right now is the iSCSI vSwitch. SC has to talk to iSCSI regardless and I didn't want it on the vMotion nor the VM network, so hence the layout. I actually called VMware on this and they perfect for what I've got (6 nics instead of 8). The next time dollars are thrown around, i'll sneak a few more 4 ports in there and the debate is then null. :smileycool:

ServiceConsoleISCSI 1080 1 vmnic4,vmnic5

ServiceConsoleMGMT 1001 1 vmnic4,vmnic5

I'm thinking of moving the opposite vmnic to standby on each SC.

Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

I wouldn't put them on standby, override failover, but just reverse the top/bottom order and have them both as active. That way you'll get the most bang for you buck.

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
tasonis
Contributor
Contributor
Jump to solution

With the management interface on a different network... from your pc goto the command prompt and do a ROUTE ADD X.X.X.X (the network of your SAN) MASK 255.255.255.0 (or other if yours is different) X.X.X.X (ip of router to that network).

We have an Equallogic SAN and this was their fix.

Reply
0 Kudos