8 pNICs and 4 vSwitches per Host in the name of security seems like overkill to me.

I don't see a risk of allowing the SC, vMotion and NFS to share 2 ports, but maybe someone could explain how a possible exploit could take place.

So how are you going to cable Infiniband or 10ge in the near future? 8 infiniband/10ge ports per Host?

myblog

Hello,

I have been thinking about the 10Gbe/inifiniband issues for quite a while now and while I would use those mostly for the VM Network, and the Storage networks I am not sure I would use them for the others yet... But that is just me.. I would still keep things separate even if I was using 10Gbe....

Now let us talk about security and ESX. Currently, ESX has several issues in common with just about every system on the face of the earth. It has vulnerabilities... Yes they are not in the OS themselves but in the networking environment surrounding it.... So we need to look at the networks as the major part of security.... There are 4 distinct networks on most ESX servers, 5 if you include a DMZ.

1) Service Console or Administrative network... This network is the DOOR to your virtual data center. It will be a major attack point. Currently, certain aspects are susceptible to Man In the Middle Attacks (MiTM).... There is no real defense except vigilance as this is a fault within the network, and Humans, not ESX. Unless you do ALL your work from the Console, this can happen. Remember access to this implies Access to EVERYTHING but one... vMotion. Unless they share the same vSwitch/Physical NIC. This network should have a physical firewall as well as the basic ESX firewall. THis network should not be seen within the DMZ, or be accessible by non-administrative VMs.

2) vMotion network, is the one that access to the SC does not imply access to... vMotion sends all data over its network in a clear text protocol. Implying that a hacker can just capture the packets and thereby access credentials and other useful data for further penetration. This should be 100% private between ESX hosts. Either a private physical switch or a tightly controlled VLAN on a pSwitch.

3) Storage network, Access to this network implies access to all the IO over the wire.... Someone could grab all the bits and pieces of a virtual disk by capturing packets. Would you let someone walk in and take a disk out of your data center without permissions. Access to this network could imply that. Note that for iSCSI, an Service Console port must participate in this network. You can lock down this port even further if necessary. This includes NAS, ISCSI, SAN networks.

4) VM Network, this is the general purpose network with Windows and other VMs within it. Windows VMs are very easy to exploit and since 70% of all exploits happen from INSIDE companies, they should also be protected using normal means and perhaps some of the other means. vSwitchs have the same limitations as normal networks.... ARP Cache Poisoning, MiTM attacks, etc.

5) A special case of the VM Network is the DMZ. The DMZ network should be 100% separated from each other. Actually, I would use DMZ specific ESX hosts instead. Search the Security and Compliance Forum for details on this. The DMZ is the often attacked location from the outside but inside worries me as well.

Since it is extremely simple and maybe required to place a vSwitch in promiscuous mode or switch VMs from network to network, it is important to monitor everything to make sure nothing is bad. Providing more attacks points to the SC, vMotion, Storage is not a good idea and using physical separation as the pNIC level is best. Until IPv6/IPSEC is used 100% the network does have any intrinsic security so networks are where things often have issues. The best security for ESX is where everything is separate and redundant. So....

For 6 pNICs where I was not using a storage network I would use:

2 pNICs for SC (vSwitch0), 2 for vMotion (vSwitch1) , 2 for VM Network (vSwitch2)

With iSCS/NFS I would do:

1 pNIC for SC and 1 pNIC for vMotion (vSwitch0), 2 for VM Network (vSwitch1), 2 for Storage (vSwitch2)..... Where the SC/vMotion do share the same vSwitch, but use different portgroups and physical NICs unless there is a case of one pNIC failure.... There are plenty of discussions on this in the Security and Compliance forum....

Some people feel this is overkill, I do not... I rather be secure using physical separation than not.... Security depends on quite a few things but in general this separation at the pNIC gives the best security for now.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

Hello,

All this adds quite a bit to security.... If all you use is one data cable per Virtualization Server then I think you need to reread all of the VI3 documentation as that is just not going to work in a production environment regardless of security.

Adding Storage, Administration, vMotion networks to your system is what you have to do just to get ESX to work. To secure these from attack they need to be separate at some point in time. VLANs within a vSwitch are not always secure.... There are cases where the do not secure anything. THey are designed to use a wire for more than one network without collisions, not really for security. However, in some cases you do get some form of security. The idea behind security is to limit the attack scope. I consider most VMs to be risky. Maybe it is my Penetration testing background, and colleagues but a good penetration tester can penetrate most OS very easily. If they can then the hackers can really do a number. This is not including things like ARP Cache poisoning but some MiTM attacks and exploiting weaknesses in the system. Now given that a VM is exploitable in this form. And it is basically impossible to prevent the detection that it is a VM, could not the next attack be against the virtualization server? Would that not give a bigger win for the hacker? They could then get access to EVERYTHING in your virtual data center. If the VMs can not see the other networks like the Administrative network (Service Console), as well as vMotion, Storage)then the chance of this happening at the moment is not very high.... Yet if the VM can see these networks then they are hackable.... I rather spend a little money up front and provide some separation and NOT depend on VLANs for these networks..... As I have seen VLAN hopping take place, etc. all because the network itself is just not secure. Your ESX server should be secured better than your data center and any other host on the system because it IS a datacenter. Granted if your datacenter is accessible off your main lobby without a lock, then these notes are just not for you. Remember 70% of all attacks come from inside not outside.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

Hello,

Link aggregation within the pSwitch does not need to be setup, the vSwitch takes care of all that. Within ESX this is called NIC Teaming, which is slightly different than 802.3ad. Be sure to monitor the system for the failure case. While it is not impossible to run vMotion/SC on the same pNIC for long periods of time it is not recommended for performance and security reasons. To get this to work, you should setup VLANs on your pSwitch, eventually you will switch from EST mode to VST mode on failure of one of the links (http://www.vmware.com/resources/techresources/412 has more information).


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

Hello,

All ESX based load balancing is based on either vSwitch source port, VM source MAC, or VM source IP. Since for the SC and vMotion there is only 1 of each, there is ineffect absolutely no load balancing possible. Instead you get only failover when using VMware NIC Teaming. If you use 802.3ad there is none either just failover. So while you get redundancy for each, there is no load balancing nor is it really desireable. For your VM Network however load balancing will exist as long as you have > 1 VM on the vSwitch. VMware does not really suggest using load balancing as it has caused some issues in the past. But I use it for my VM Network with no issues. Just do not have more than 2 pNICs in the team.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074