VMware Cloud Community
Eudmin
Enthusiast
Enthusiast
Jump to solution

public facing network for VMs, private for service console and storage?

Hi,

Until now I've had a fairly small install with 2 servers with 4 physical NICs each running ESX 3.5, a small EqualLogic SAN box for shared storage, and a few VMs all on our regular, routed building network, not on a private one. The network config was really simple. I just put everything on real IP addresses on our building network.

Now I want to move the SAN and service console traffic onto a private network, but I'm not sure how to do that.

Right now I'm using 2 NICS on each server:

vmnic0 is configured on vSwitch0 and has the VM Network on it which all of my VMs use to talk to the outside world and it also has the Service Console which Virtual Center uses and I use to SSH to it.

vmnic1 is configured on vSwitch1 and has a VMKernel Port and also a Service Console Port for software iSCSI to talk to my SAN. (never been clear on why both are needed to talk to the SAN, but the docs say that they are)

My plan is to configure a vSwitch2 and link it to vmnic2 and set up a VMKernel Port and Service Console Port for software iSCSI on the 10.x.x.x network, set up my new (bigger) SAN box on the 10.x.x.x network and just use Storage vMotion to transfer the VMs to the new storage space. Once I do that I'd like to only use the Service Console on vSwitch2 and not have a Service Console at all on vSwitch0. Is it possible to delete the one on vSwitch0 and just use the new one on vSwitch2 for Virtual Center and ssh access?

So my proposed setup would be:

vSwitch0: VM Network only, used by VM guests only to access public facing network, no Service Console to buildling network, tied to vmnic0

vSwitch1: redundant once I do Storage vMotion of everything off my old SAN, will eventually delete and pair vmnic 1 with vmnic0, tied to vmnic1

vSwitch2: VMKernel and Service Console on 10.x.x.x network, used to access new SAN, used by Virtual Center to access ESX, used for SSH in to ESX on private network, tied to vmnic2

Should this work?

Thanks.

Reply
0 Kudos
1 Solution

Accepted Solutions
Texiwill
Leadership
Leadership
Jump to solution

Hello,

vmkernel ports can not live on the same subnet. So if you have 3 vmkernel ports say: vMotion, iSCSI, and NFS. You really need 3 subnets. 1 for each vmkernel port.

Otherwise how would it know to route everything properly?


Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill

View solution in original post

Reply
0 Kudos
11 Replies
Texiwill
Leadership
Leadership
Jump to solution

Hello,

Check out my Topology Blogs for some help with this. Basically you need to consider the security zones you need to protect and how you will do that.

Will you be using VLANs, Physical switches?

In general, SC + Storage could be one security zone. VMs the other. Are you also using vMotion?

If it was me I would add another set of pNICs if possible into your configuration as you will get much better over all security and redundancy. But this also depends on if you think you will be subject to Layer-2 Attacks in the physical network or not.


Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Eudmin
Enthusiast
Enthusiast
Jump to solution

Hi Texiwill,

I'm pretty glad you responded to this question. I had googled the question and actually came up with your blog post for 4 Physical NICs. It does sound like just what I want to do, but I figured since the post was 6 months ago, or so, it might be stale. Wasn't sure how often you'd be there looking at comments and answering questions...

I'm going to avoid using multiple VLANs because they involve negotiations with the people who run our Cisco architecture (plus I don't really know how that all works). I'm using physical switches. I'd like to use 1 switch for the VMs and 1 switch for everything else: SAN, vMotion, Virtual Center. I'm running HP bl460c blade servers, so I could add another 2-4 nics in each without much trouble to add redundancy and would then buy more switches to hook up the redundant ports to.

The question I had was with your nomenclature on your 4 physical NIC configuration. VMware's docs always say that you need a Service Console on the same vSwitch that your SAN VMKernel is on. So that must be the one Service Console you list. Also, when you set up a host it configures you a Service Console for you to ssh to and for virtual center to use. In your config is that the same Service Console that VMware insists you have for software iSCSI connections? Also, they don't talk about adding a "vmotion" portgroup. From your diagram and responses to questions I can see that on your vSwitch0 you have a service console and a vmkernel port called "Storage Network" on vSwitch0, but what's that "vmotion" portgroup on vSwitch0? Is that another vmkernel with the vmotion checkbox checked? If it is, does that mean that I shouldn't check the vMotion box for the "Storage Network"?

In general I get your diagram, I think. I'd connect the NICs associated with vSwitch0 to my private network switch and the NICs associated with vSwitch1 to a switch on our building network, but I guess I'm missing what some of the lines in your diagram mean.

Thanks,

Reply
0 Kudos
ShaneWendel
Enthusiast
Enthusiast
Jump to solution

I think you're mixing up the storage network with the vMotion network. You need an SC on your vMotion network (or working routing to and from), but not necessarily on your IP storage network.

Shane Wendel, VCP

----------------- Shane Wendel VCP: vSphere 4 VCP: VI3 http://fatalsync.wordpress.com
Reply
0 Kudos
Texiwill
Leadership
Leadership
Jump to solution

Hello,

I think you're mixing up the storage network with the vMotion network. You need an SC on your vMotion network (or working routing to and from), but not necessarily on your IP storage network.

Um... no, neither actually not with ESX 4.

SC, vMotion, and IP Storage do not have requirements to participate in each others networks (at least that was what I have seen in vSphere, VI3 is definitely different.). THere is no need to IP Storage and SC to participate in the same network (Vsphere, VI3 this requirement still exists).

There has NEVER been a need for SC and vMotion to participate in the same network.

Need to run some tests myself on this and update my Topology blogs.


Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
Texiwill
Leadership
Leadership
Jump to solution

Hello,

I will revise those for vSphere but the general layout still holds true...

The question I had was with your nomenclature on your 4 physical NIC configuration. VMware's docs always say that you need a Service Console on the same vSwitch that your SAN VMKernel is on.

for VI3, I thought that went away with ESX 4, but it may not have in that case your SC and iSCSI VMkernel ports shoudl participate in the same network.

So that must be the one Service Console you list.

Yes, you REALLY only need one. It needs to participate in the iSCSI Network (VI3 specific) but it may still be true for vSphere.

>Also, when you set up a host it configures you a Service Console for you to ssh to and for virtual center to use.

Yes. Service Console is part of your management network layer.

In your config is that the same Service Console that VMware insists you have for software iSCSI connections?

Yes, why have two, no real need.

Also, they don't talk about adding a "vmotion" portgroup. From your diagram and responses to questions I can see that on your vSwitch0 you have a service console and a vmkernel port called "Storage Network" on vSwitch0, but what's that "vmotion" portgroup on vSwitch0?

That is correct.

Is that another vmkernel with the vmotion checkbox checked?

Correct.

If it is, does that mean that I shouldn't check the vMotion box for the "Storage Network"?

Absolutely never do that. You want vMotion to run over a differnet 'wire' than IP Storage. IP Storage is IO intensive so is VMotion when you need it.

In general I get your diagram, I think. I'd connect the NICs associated with vSwitch0 to my private network switch and the NICs associated with vSwitch1 to a switch on our building network, but I guess I'm missing what some of the lines in your diagram mean.

That is correct.


Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Eudmin
Enthusiast
Enthusiast
Jump to solution

Thanks a ton for the explanation. I do have licenses for vSphere, but have to do more reading before I upgrade everything. I haven't quite sorted out if I keep everything I currently have if I upgrade my VI3 Enterprise licenses to vSphere Enterprise without paying more for "Enterprise Plus."

Anyway, just so I totally understand it, does this look right? I've put in square brackets the names that VC calls them.

pNIC0 -&gt; vSwitch0 -&gt; Portgroup0 (service console) [ service console, physically private SAN/management network]<br>
................. -&gt; Portgroup1 (VMotion) [ vmkernel, vmotion box checked, on vSwitch0, physically private SAN/management network]<br>
pNIC1 -&gt; vSwitch0 -&gt; Portgroup2 (Storage Network) [ vmkernel, vmotion box unckecked, physically private SAN/management network]<br>
pNIC2 -&gt; vSwitch1 -&gt; Portgroup3 (VM Network) [ network for virtual machines, physically public network]<br>
pNIC3 -&gt; vSwitch1 -&gt; Portgroup3 (VM Network) [ network for virtual machines, physically public network]

So once I've done all of that I'll have three IP addresses on my private subnet for the service console and two vmkernels. How does the ESX server know to use only the Storage Network vmkernel via pNIC1 for SAN traffic rather than the vmotion vmkernel via pNIC0? When I go to the "Storage Adapters" link on the Configuration page I don't see a way to set which physical NIC is used for the iSCSI Software adapter called vmhba32. The iSCSI Alias listed is the domain name of the service console that was installed by default on vSwitch0 and pNIC0 when I installed ESX, not the service console or vmkernel that I created on vSwitch1 and pNIC1 when setting up the iSCSI connection, however I have seen in the logs on my SAN box that the IP that connects to the SAN is that vmkernel port on vSwitch1.

Reply
0 Kudos
Texiwill
Leadership
Leadership
Jump to solution

Hello,

> pNIC0 -&gt; vSwitch0 -&gt; Portgroup0 (service console) [ service console, physically private SAN/management network]<br>
> ................. -&gt; Portgroup1 (VMotion) [ vmkernel, vmotion box checked, on vSwitch0, physically private SAN/management network]<br>
> pNIC1 -&gt; vSwitch0 -&gt; Portgroup2 (Storage Network) [ vmkernel, vmotion box unckecked, physically private SAN/management network]<br>
> pNIC2 -&gt; vSwitch1 -&gt; Portgroup3 (VM Network) [ network for virtual machines, physically public network]<br>
> pNIC3 -&gt; vSwitch1 -&gt; Portgroup3 (VM Network) [ network for virtual machines, physically public network]

This looks fine to me. Besure if you are using VLANs to trunk the first three virtual networks through pnic0 and pnic1.

So once I've done all of that I'll have three IP addresses on my private subnet for the service console and two vmkernels. How does the ESX server know to use only the Storage Network vmkernel via pNIC1 for SAN traffic rather than the vmotion vmkernel via pNIC0?

Its done by portgroup. You can assign a pNIC to each portgroup then assign the other one as the hardcoded failover pNIC for that portgroup.


Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
Eudmin
Enthusiast
Enthusiast
Jump to solution

OK. I'm trying to create the service console and the vmotion and ip storage vmkernels on vSwitch0 and it's not letting me make the second vmkernel. I get an error message that "You can not create two or more vmkernel nics on the same subnet. A vmkernel nic with the name vMotion already exists in that subnet: 192.168.1.0/255.255.255.0" I'm trying to put them both on the private subnet that I'm running on a switch and not routing anywhere.

I'm trying to put the service console on 192.168.1.1, vmotion vmkernel on 192.168.1.2, and storage network vmkernel on 192.168.1.3. Am I getting it wrong or do I just have to use the command line to do what I want to do or do it in some different order?

Reply
0 Kudos
howie
Enthusiast
Enthusiast
Jump to solution

I think there is a CLI to do it. The UI support is likely not in vShphere 4 yet.

Reply
0 Kudos
Texiwill
Leadership
Leadership
Jump to solution

Hello,

vmkernel ports can not live on the same subnet. So if you have 3 vmkernel ports say: vMotion, iSCSI, and NFS. You really need 3 subnets. 1 for each vmkernel port.

Otherwise how would it know to route everything properly?


Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
Eudmin
Enthusiast
Enthusiast
Jump to solution

LOL, that's what I was wondering a few posts ago. How does it know which vmkernel to use for what?

OK. I'm going to try getting this working, and if I get stuck some more I'll get some use out of my platinum support rather than continuing to ask on the forums.

What I'm trying to do is set up another SAN device, put this one on a private switch rather than just attaching it on our regular network, use ports 3 and 4 on my ESX servers (currently unused) to talk to the new SAN device and use storage vmotion to move the VMDK files to this new SAN device while keeping them running. Once they're on the new device I'll shut the old one down and move it over too. I've put my vcenter server's second network port on that one and it can talk to the SAN device...

Reply
0 Kudos