VMware Cloud Community
liveammo
Contributor
Contributor

ESX 3.0.2 Service Console Security Issue

I have found what appears to be a fairly significant security issue, related to virtual switch isolation and multiple service consoles.

From a vanilla ESX 3.0.2 install, I created the following network topology which contains two multihomed VMs, each with vNIC1 and vNIC2:

192.168.0.0/24 -> vSwitch0 -> Service Console 192.168.0.2 -> VMkernel 192.168.0.3 -> Win2003K (vNIC1 192.168.0.10) / WinXP (vNIC1 192.168.0.11)

vSwitch0 is connected to one external pNIC.

172.16.0.0/24 -> vSwitch1 -> Service Console #2 172.16.0.2 -> Win2003K (vNIC2 172.16.0.10) / WinXP (vNIC2 172.16.0.11)

vSwitch1 is an isolated switch with no external connections.

Both VMs have IP forwarding disabled, so there shouldn't be anything being passed between the vNIC1 and vNIC2 interfaces. From the outside world, by setting an external station with an IP address within the internal 172.16.0.0/24 segment, Service Console #2 is directly accessible at least on port 902. I haven't yet done enough testing to determine how traffic is being passed through to vSwitch1, my initial thoughts are that vswif0/Service Console #1 is somehow forwarding frames through to the internal vSwitch1. Any ideas on this behavior? It looks like there are some sysctl variables that can be set for vswif0 but I haven't done any testing on that yet either.

Thanks in advance.

0 Kudos
54 Replies
Texiwill
Leadership
Leadership

Hello,

Then there is no need for an extra set of 'DMZ' iSCSI ports as the storage network is from internal and the VMs have no direct access to it, therefore there is no method to get into the storage network.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, (c) 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
biniam
Contributor
Contributor

On last question Edward,

The IP addresses that you set when you configure the SAN. Are these IP number the same when you setup vmware iSCSI?

I that possible to have the VLAN on pSwitch only?

VM VLAN on the pSwitch

vMotion VLAN on the pSwitch

SC VLAN on the pSwitch

ISCSI VLAN on the pSwitch

Regards Biniam

0 Kudos
Texiwill
Leadership
Leadership

Hello,

A VLAN to the pSwitch is called EST (External Switch Tagging) and from there you have multiple links to the ESX server. Yes this is fine as long as you also do not allow promicous mode on the pSwitch or if you do it is limited to just the IDS system.

Best regards,

Edward L. Haletky, author of the forthcoming 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', publishing January 2008, (c) 2008 Pearson Education. Available on Rough Cuts at http://safari.informit.com/9780132302074

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
biniam
Contributor
Contributor

Hi Edward and all,

I am back and I have all the kit to test what we discussed. I have some questions with regards to infrastructure and security.

I have installed the ESX hosts on 10.44.1.0/24 Vlan 10 also I have

2 pNIC for Service Console 10.44.1.0/24 Vlan 10

2 pNICs for VMs 10.44.2.0/24 Vlan 20

2 pNIC for vMotion 10.44.99.0/24 Vlan 99

2 pNICs for iSCSI 10.44.3.0/24 Vlan 30

I am Using EST (External Switch Tagging) with firewall controller access between the VLANs. Could you please tell me which VLANs need to see each, for VMware infrastructure to work in secure and efficient way?

Regards

Biniam

0 Kudos
JDLangdon
Expert
Expert

Hi Edward and all,

I have installed the ESX hosts on 10.44.1.0/24 Vlan 10 also I have

2 pNIC for Service Console 10.44.1.0/24 Vlan 10

2 pNICs for VMs 10.44.2.0/24 Vlan 20

2 pNIC for vMotion 10.44.99.0/24 Vlan 99

2 pNICs for iSCSI 10.44.3.0/24 Vlan 30

I am Using EST (External Switch Tagging) with firewall controller access between the VLANs. Could you please tell me which VLANs need to see each, for VMware infrastructure to work in secure and efficient way?

How many pNICs do you have per host?

Jason

0 Kudos
JDLangdon
Expert
Expert

Hi Edward and all,

I have installed the ESX hosts on 10.44.1.0/24 Vlan 10 also I have

2 pNIC for Service Console 10.44.1.0/24 Vlan 10

2 pNICs for VMs 10.44.2.0/24 Vlan 20

2 pNIC for vMotion 10.44.99.0/24 Vlan 99

2 pNICs for iSCSI 10.44.3.0/24 Vlan 30

Assuming you have eight pNICs per host here's what I would do.

1) I would use the same two pNICs for both the service console AND Vmotion and I would route all traffic over VLAN10.

2) I would install the iSCSI initiator on the ESX host server and use two pNICs and VLAN30 to route between the iSCSI SAN and the host server.

3) I would use two pNICs and VLAN20 to route traffic between all VM's and what I call "The outside world".

4) I would install the iSCSI initiator on the VM's and use two pNICS and VLAN99 to route iSCSI traffic between the iSCSI SAN and the VM's.

5) I would divide my iSCSI SAN into two sections. One section would be formatted as VMFS and would be used to store all .vmdk files. These .vmdk files would contain only the VM guest OS (C:\ drive). Assuming you are using Windows OS within the VM's, I would format the other section of the iSCSI SAN as NTFS and allocated this space to the individual VM's which would be used to store the data.

While this is not necessarily the correct way to configure this environment, it is what I would do. Attached is a .png file which outlines what I stated above.

Jason

0 Kudos
biniam
Contributor
Contributor

Hi Jason,

Thanks for your input, very interesting solution, I have 10 pNICs in total. Two planned to be used for DMZ. I am using EMC NS20 which has multi-protocol support (ALL-IN-ONE box NAS/SAN, ISCSI/FIBRE). So we have CIFS for simple data. We also use RAW for our exchange and SQL logs.

1. In your recommendation you have putted the VM on two groups. Are these similar to DMZ and trusted networks?

2. Is that good practice to allocated four pNIC to VM?

3. What is the purpose of separating VM as the raw and vmfs data usage? We have exchange server that use the data and log on raw and the OS on VMFS.

4. My staff went to Deploy, Secure and Analyse training and there was a lot of discussion on security on mixing vMotion with SC or ISCSI. Do you thing mixing SC with vMotion bring some security concern?

Regards

Biniam

0 Kudos
Texiwill
Leadership
Leadership

Hello,

The bottom looks great.

2 pNIC for Service Console 10.44.1.0/24 Vlan 10

2 pNICs for VMs 10.44.2.0/24 Vlan 20

2 pNIC for vMotion 10.44.99.0/24 Vlan 99

2 pNICs for iSCSI 10.44.3.0/24 Vlan 30

I am Using EST (External Switch Tagging) with firewall controller access between the VLANs. Could you please tell me which VLANs need to see each, for VMware infrastructure to work in secure and efficient way?

The Service Console must partiticpate in the iSCSI network in order for authentication protocols to work. This is required whether you use hardware iSCSI- HBAs or network cards. So your iSCSI device must see your service console and visa versa.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
JDLangdon
Expert
Expert

Hi Jason,

1. In your recommendation you have putted the VM on two groups. Are these similar to DMZ and trusted networks?

2. Is that good practice to allocated four pNIC to VM?

3. What is the purpose of separating VM as the raw and vmfs data usage? We have exchange server that use the data and log on raw and the OS on VMFS.

4. My staff went to Deploy, Secure and Analyse training and there was a lot of discussion on security on mixing vMotion with SC or ISCSI. Do you thing mixing SC with vMotion bring some security concern?

1. VLAN99 should be a trusted network in that the only devices on this VLAN should be the VM's that need access to the iSCSI, the EMCns20, and whatever PC's are used to manage this environment.

2. I haven't allocated four pNICs to a VM, I've attached two pNIC's to the vSwitch that controls VLAN99 (iSCSI) and 2 pNICs to the vSwitch that controls VLAN20. You could have just as easily routed both VLAN20 and VLAN99 over the same tow pNICs but in my environment, the iSCSI devices are on a private, non-route able network.

3. By storing the VM data on the iSCSI in native formate (non-vmdk) you have to option of attaching it to a physical server if the VM ever crashes. Also, while I have not experimented, I have been told by an iSCSI vendor that you should get better performance when using the iSCSI initatior inside of the guest OS to access the data.

4. If your staff attended the "Deploy, Secure and Analyse" training session, tell them to look at pages Module1 - Lesson 3 and Module2 - page 12. In my environment, all service console and Vmotion traffic will be on a secured VLAN that is only accessible from my laptop, an ESX host server, or the VirtualCenter server. This will remove all concerns about having SC and Vmotion on the same network.

Jason

0 Kudos
Texiwill
Leadership
Leadership

Hello,

If you want your VMs to see your iSCSI network then they also must participate in this. I however recommend that the VM access and the Host access be segregated some how. If you only have your VMs access the iSCSI server and you have no other remote storage then vMotion will not work. Consider this:

VM only access to iSCSI using iSCSI initiator.... Where does the boot disk live? On local storage.... So therefore no vMotion.

VM and Host access to iSCSI using initiator within VMs and iscsi target within Host.... Could a VM then see any of the VMFS' on the remote storage? This depends on how the LUNs are presented, how the iSCSI server segregates such traffic. Does your iSCSI server support multiple network links? If so you may want to use a set of links for hosts and a set of links for initiators. If the Host uses iSCSI then an SC connection must be part of this network. How would such access impact disk performance at the Host level. Or impact the VM networks. With only 10 pNICs and 2 assigned to a DMZ, you could do this but you compromise security or redundancy on other networks. See previous discussions. If you have another set of pNICs then this would be a safe option. iSCSI is all about performance moving through the network to have VMs using iSCSI initiators they would need the network bandwidth for disk access. You can get the same backup/restoration behavior using Raw Disk Maps as well, which can use the Host based iSCSI target code. Either method works.

SC and vMotion in a truly secure environment should never be over the same vSwitch or pNICs. Yes people do it, but it really is insecure. However, this is mostly a trust related issue. If your administrators are allowed to login to the VMs then it is a moot issue as they have the credentials however if they are not allowed to login then allowing them access to the vMotion network could allow someone to gain those credentials quite easily just by capturing the data during a vMotion. Or if your SC is hacked and you are using remote login services a hacker can do the exact same thing. For full protection make it a private link.

4. If your staff attended the "Deploy, Secure and Analyse" training session, tell them to look at pages Module1 - Lesson 3 and Module2 - page 12. In my environment, all service console and Vmotion traffic will be on a secured VLAN that is only accessible from my laptop, an ESX host server, or the VirtualCenter server. This will remove all concerns about having SC and Vmotion on the same network.

Absolutely, lock down which machines have access to the Service Console, but nothing but an ESX server should have access to vMotion. If this laptop or any other host was compromised it can be setup to grab the memory image as it travels across the network during a vMotion and thereby gather credential data. I would not risk this.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
v01d
Enthusiast
Enthusiast

You need to stop saying this...

SC and vMotion in a truly secure environment should never be over the same vSwitch or pNICs. Yes people do it, but it really is insecure. However, this is mostly a trust related issue. If your administrators are allowed to login to the VMs then it is a moot issue as they have the credentials however if they are not allowed to login then allowing them access to the vMotion network could allow someone to gain those credentials quite easily just by capturing the data during a vMotion. Or if your SC is hacked and you are using remote login services a hacker can do the exact same thing. For full protection make it a private link.

[

|http://www.astroarch.com/wiki/index.php/Virtualization]

Or back it up...

If you have your SC and VMotion in seperate port groups sharing a single vSwitch with a pair of teamed pNICs trunking those 2 VLANs to it there is no security issue. You need to stop telling people that don't know any better that this is insecure because it is NOT. Even in promicous mode the SC port group CAN NOT sniff traffic from another VLAN. The vSwitch strips off the tags and presents only the traffic for the specified VLAN to port group in an untagged form.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

How do you think vSwitch based IDS systems work when VLANs are involved? You can easily ignore the tags to see all the data on a vSwitch.... Granted you do need specialized tools to do this, but they are not that hard to find or use. VLANs offer some security when everything is running perfectly and everything agrees on how to handle 802.1q. If everything was using IPv6 with full IPSEC based security this also would not be an issue, but you are talking about IPV4 which has no protections so a promiscuous mode adapter can with the proper tool see all data including the individual tags used by 802.1q. This is an inherent flaw in IPV4 not ESX, VI3, the vmkernel, or a vSwitch. If you had a dumb physical switch the same could be said about it. The higher end physical switches try to make this impossible to do, but a vSwitch is not all that intelligent. When IPV4 is inherently unsafe the only thing you can do is mitigate the possible damage that can be done by trying to think of all the options. If you wish to discuss this further please send me a Private Message.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
v01d
Enthusiast
Enthusiast

Ok here we go...

How do you think vSwitch based IDS systems work when VLANs are involved?

They work by having a promiscuous mode vNIC attached to a port group that is NOT assigned to a VLAN.

If your SC and VMotion are in separate port groups each assigned to seperate VLANs then only way to sniff VMotion traffic off the vSwitch is with a purposely configured port group and vNIC attached to said vSwitch. Such a scenario is in no way mitigated by having SC and VMotion seperate.

The heart of this disagreement is your claim of a security issue with combining SC and VMotion. I don't see one. I've asked you to spell it out and all you do is keep trying to BS me.

Spell it out or man up that there isn't one.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Well we must agree to disagree. I do not agree with you. I know many people doing SC/vMotion on the same vSwitch as well as using 2 pNICs for both. I state clearly that there is a risk when promiscuous mode portgroups/vSwitches/devices are allowed. If they are not allowed there is no risk. You feel comfortable using both together others do not because there is a risk. Granted, I will agree that it takes a lot of misconfiguration to make it happen but there are no tools to prevent that misconfiguration or even detect it yet. Since promiscuous mode can only be allowed on a portgroup or the entire vSwitch it would take a misconfiguration, accident, or purposeful change to make it happen. Granted, I equate VLAN and portgroup together in an ESX environment but VLANs are not really the issue, the portgroup/vSwitch settings are the issue.

Having separate vSwitches alleviates this possibility. shrug If you do not allow promiscuous mode devices within the virtual network then it is also mitigated but with the newest bunch of tools available, the possibility exists. Segregation is the best defense currently.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
v01d
Enthusiast
Enthusiast

Hello

I state clearly that there is a risk when promiscuous mode portgroups/vSwitches/devices are allowed. If they are not allowed there is no risk.

Promiscuous mode really has no bearing here. The risk of VMotion traffic being sniffed off the vSwitch by a promiscuous mode vNIC on a port group with no VLAN assignment is EXACTLY the same regardless of wether SC is on the same vSwitch or not. SC + VMotion (seperate port groups/vlans) on a common vSwitch is NOT less secure. You have not defined how combining them INCREASES the risk.

P.S. I'm definitly NOT buying your book

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Whether or not to allow the SC/vMotion pair together depends entirely on the trust relationship set out by either the security policy, security administrator, or management of a company/institution. If you trust everyone then it really makes no difference, if you trust no one then it would. This is repeating statements made previously.

Trust is the key element here, as the vMotion data is unencrypted and can contain credential information. I err on the side of being more secure than less in other words I state do not trust. I rather not have any vMotion data sniffable by ANYTHING regardless of location. The only things on my vMotion network are ESX Servers vMotion portgroups and nothing else. I do not overlap SC/vMotion on the same vSwitch because of this as well.

The following questions should all be considered before putting the SC on with any other network actually, vMotion is just one of the more dangerous to access, but any network can have the same issues. The only exception to this rule is imposed by VMware themselves, and that is iSCSI authentication, an SC vswif must participate in this network. So you implicit trust your Administrators with the iSCSI authentication due to this. I rather not but hey, its a current software limitation.

I feel this increases the risk as major changes to virtual networking will be easily recognized as such (moving vMotion to another portgroup for example), but minor changes will not be (enabling Promiscuous mode on the SC or a VM Network portgroup.) By easily recognized as such, you can see almost immediately that something changed when looking at the network configuration screens in the VIC or by using the command line. But subtle changes require deeper review and could be missed easily. That is why I think the risk is elevated. But how much so depends on the Trust Relationships that exist within the organization.

1) Should an ESX administrator be able to access any running VM? If yes then go ahead and combine.

2) Does the ESX administrator have the necessary roles/permissions/classification/privileges within the organization to access a running VM? If yes then go ahead and combine.

3) Does the security policy allow this?

4) Do you trust your ESX administrators? If yes, go ahead and combine.

5) Is there auditing/monitoring for promiscuous mode virtual machines? If yes, go ahead and combine.

6) Do you trust your firewalls? If yes, go ahead and combine.

7) Do you trust your defense in depth? If yes, go ahead and combine.

If you answer no to any of the above then I would not combine my SC with any network (except iSCSI if iSCSI is in use, even so I would have non-authentication iSCSI traffic use its own vmkernel device). Now talking about vMotion and other networks gets the same questions, but I use 'Users' instead of ESX Administrators in these questions.

Only the Security Administrators know the trust relationships, I would talk to them before implementing anything within the virtual network. But if ESX Administrators do not know of the possibility they could think they are safe when they are not. Nor do they know what to look for when reviewing the systems for issues.

In the realm of 'Risk Assessment' is this a very high risk item.... Not as much as it used to be, and the risk is related to the Trust Relationships that are in play. But while this may not be even a low risk in one organization it could be a very high risk in another. This is one of those items that is really dependent on the organization.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
v01d
Enthusiast
Enthusiast

Another long winded non answer to save face.

I feel this increases the risk as major changes to virtual networking will be easily recognized as such (moving vMotion to another portgroup for example), but minor changes will not be (enabling Promiscuous mode on the SC or a VM Network portgroup.) By easily recognized as such, you can see almost immediately that something changed when looking at the network configuration screens in the VIC or by using the command line. But subtle changes require deeper review and could be missed easily. That is why I think the risk is elevated. But how much so depends on the Trust Relationships that exist within the organization.

How many times must I say this? Enabling promiscuous mode on the SC would still NOT allow it see ANY traffic outside it's own VLAN. Even in promiscuous mode it would only see what it's port group gives it. Since the port group is assigned to the SC VLAN, that's all the SC will ever see reguardless of promiscuous mode. Any attempt to change to the SC port group see beyond that will cause contact with that ESX host to be lost (not very subtle).

Throwing paragraphs at me to save face doesn't change the facts.

1. You've claimed repeatedly that SC/VMotoin in seperate port groups and VLANs on a common vSwitch on teamed+trunked pNICs is insecure. Which it is not.

2. As a result of your bad advice we have people on these forums who've configured there systems with 2 pNICs VMotion + 2 pNICs SC. Which is completely unnecessary.

All I ask it that you substantiate your claim or stop making it. If there is ANY increased risk at all (I don't see one/you haven't proved one), you have GROSSLY overstated it.

0 Kudos
JDLangdon
Expert
Expert

2. As a result of your bad advice we have people on these forums who've configured there systems with 2 pNICs VMotion + 2 pNICs SC. Which is completely unnecessary.

It's getting to the point now where people have no idea who to believe. I would love to see some official VMware whitepaper on whether or not this is actually a security concern. from my SDA training I do not recall any discussion on a potential security risk of sharing pNICs between the service console and Vmotion. I do however, recall the security issue of sharing pNIC's between the service console and your virtual machines.

I have considered the option above (2 pNICs VMotion + 2 pNICs SC) but having this many patch cables hanging out of the back of a single server is ludicrous if there is no real security concern. In my current environment we have 8 host servers with 6 network connections each for a total of 48 patch cables. Adding another 2 cables in order to separate the service console and Vmotion would bring the total up to 64 physical switch ports.

Unfortunately, the people who make the decisions and purchases do not see the 10-20 VM's that are running per host, they only see the one physical box with 8 network cards and cables that is taking up all the space in that new $20,000 switch the just bought.

Jason

0 Kudos
TomHowarth
Leadership
Leadership

Jason

At the end of the day is all about Risk and Trust as Edward says I have clients who would expect seperation of SC and VMotion traffic, I have others who will take the risk.

Tom Howarth

VMware Communities User Moderator

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410
0 Kudos
v01d
Enthusiast
Enthusiast

Since you seem to think there is a "risk", please describe it clearly.

Jason

At the end of the day is all about Risk and Trust as Edward says I have clients who would expect seperation of SC and VMotion traffic, I have others who will take the risk.

Tom Howarth

VMware Communities User Moderator

I have posed the same question multiple times to Edward and I have received a different incorrect answer everytime. In his latest responds he inadvertantly admits that there isn't an issue with that configuration by implying that somone would have to make subtle changes that might not be noticed. If there is a "good" reason to seperate them, I want to know about it. To this point Edward has not been able to sucessfully make that case.

0 Kudos