ESX 3.0.2 Service Console Security Issue

[liveammo]

I have found what appears to be a fairly significant security issue, related to virtual switch isolation and multiple service consoles.

From a vanilla ESX 3.0.2 install, I created the following network topology which contains two multihomed VMs, each with vNIC1 and vNIC2:

192.168.0.0/24 -> vSwitch0 -> Service Console 192.168.0.2 -> VMkernel 192.168.0.3 -> Win2003K (vNIC1 192.168.0.10) / WinXP (vNIC1 192.168.0.11)

vSwitch0 is connected to one external pNIC.

172.16.0.0/24 -> vSwitch1 -> Service Console #2 172.16.0.2 -> Win2003K (vNIC2 172.16.0.10) / WinXP (vNIC2 172.16.0.11)

vSwitch1 is an isolated switch with no external connections.

Both VMs have IP forwarding disabled, so there shouldn't be anything being passed between the vNIC1 and vNIC2 interfaces. From the outside world, by setting an external station with an IP address within the internal 172.16.0.0/24 segment, Service Console #2 is directly accessible at least on port 902. I haven't yet done enough testing to determine how traffic is being passed through to vSwitch1, my initial thoughts are that vswif0/Service Console #1 is somehow forwarding frames through to the internal vSwitch1. Any ideas on this behavior? It looks like there are some sysctl variables that can be set for vswif0 but I haven't done any testing on that yet either.

Thanks in advance.

[TomHowarth]

And likewise you have not explained your corner, only attacking Edward and making unneeded snide comments about "Not buying his book" information is a two way street. I personally believe if you have the oppotunity to seperate disparate data you do so. I do not know your data backgound, but in the UK CESG states data seperation is always better.

I understand your statments regarding vlan tagging etc, but a theoretical risks is still a risk. I am not going to get involoved in this discussion but suggest we agree to differ.

Tom Howarth

VMware Communities User Moderator

[v01d]

My apologies for any "snide comments". I personally get extremely annoyed with people who attempt to "baffle with BS" when asked a straight forward question.

I personally believe if you have the oppotunity to seperate disparate data you do so. I do not know your data backgound, but in the UK CESG states data seperation is always better.There are degrees to which that is valid, otherwise you shouldn't be using a virtual infrastructure in the first place. If "data seperation is always better" then why the hell are we all consolidating all our "seperate" physical servers onto a smaller number of virtual serversl. By your logic, using vmware at all is a "risk"

[biniam]

Sorry everyone I just come back from holiday and sad to see some of the comments in the forum. I personally don’t see the reason for this. Security is paramount in my organization and Edward ideas and advice is very valuable for me. Back to the real discussion, with regard to security permission I have firewall which control the access of all VLANs and I am trying to give permission between VLANS and specific devices.

Eg .. permit VLAN 10 to access VLAN 30.

Do I need to give more permission?

VLANID 10

-Service Console 10.44.1.10

-ESX HOST 10.44.1.1 and 10.44.1.2 etc ..

-VirtualCenter 10.44.1.5

Vmkernel (ISCSI)

VLANID 30

Virtual Machine Network

VLANID 20

SQL - 10.44.2.5

Exchange - 10.44.2.2

vMotion

VLANID 99

Regards

Biniam

[Texiwill]

Hello Biniam,

Other than VLAN 10 accessing VLAN 30 (for iSCSI authentication), there is only one other item to consider, will your VMs (VLAN 20) use iSCSI Initiators as well? If so then they also need access to VLAN 30. If they are not going to need such access, everything else looks just fine. If it was me, I would setup a different VLAN for that connectivity with different iSCSI presentations so that there is no chance a VM could gain access to any VMFS.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[biniam]

Hi Edward,

Currently there is no need for VMs (VLAN 20) use iSCSI Initiators. I guessthis is MS iSCSI Initiators?

Also when you say "I would setup a different VLAN for that connectivity with different iSCSI presentations so that there is no chance a VM could gain access to any VMFS" Is this with regards MS iscsi initiator?

Regards

Biniam

[Texiwill]

Hello,

Yes these refer to the iSCSI Initiators running within the VM, for Windows that could be the Microsoft iSCSI Initiator. Since you do not have this need, you do not need to allow this access.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[biniam]

Hello,

I have setup my system as below and

VLANID 10

-Service Console 10.44.1.10

-ESX HOST 10.44.1.1 and 10.44.1.2 etc ..

-VirtualCenter 10.44.1.5

Vmkernel (ISCSI)

VLANID 30 10.44.3.6

Virtual Machine Network

VLANID 20

SQL - 10.44.2.5

Exchange - 10.44.2.2

vMotion

VLANID 99 10.44.99.6

I am getting the following error evernthough i have permision VLAN 10 and VLAN 40 to communcate. Do i need to give access to VLAN 99 vMotion

“The service console network port and a VMkernet IP storage network port both must be able to communicate with your iSCSI storage for the SW iSCSI initiator to work. A convenient way to do this is to create a service console network port on the same physical adapter and network as the VMkerner IP storage network.”

Also what will be the gateway for Vmotion and iscsi and service console.

regards

Biniam

[Texiwill]

Hello,

You will have to add a route from the SC network to the iSCSI Storage network in order for software iSCSI to work. Your SC must be able to route to the isCSI server.

As for vMotion its default network can be anything you want. SC and iSCSI should have gateways that perhaps can see each other or at least route through to each other. I am not sure what your gateways should be as that depends entirely on the networks.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[biniam]

Hi Edward,

The network topology is as follows ... SC (pNIC) 10.44.1.1 gw 10.44.1.254 (firewall) and the firewall can route to the iscsi ISCSI (pNIC) 10.44.4.1 gw 10.44.4.254 (firewall). I can ping from SC network to ISCSI. My problem is if i put the SC and ISCSI on the same network i could see my LUN. However, i soon as i separate them no i lose connection?

Regards

Biniam

[Texiwill]

Hello,

Software iSCSI appears to not be routable. That is odd, but every install I have seen does not try to route it and your test have proven it does not work. Another option is that your firewall is not passing the appropriate ports through the route? In either case I would do the following: On your iSCSI vSwitch create another service console portgroup that is on the same network as the iSCSI server. This should route through the same VLAN so should be fine. This way you have a vswif1 device that is part of the iSCSI network. I would remove the route from normal SC to iSCSI.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[Texiwill]

Hello,

It appears the software iSCSI is not routable or that your firewall is not allowing specific ports necessary to allow this. I would be on the first. I would add a second SC port to the iSCSI vSwitch. The vswif1 device created should be on the same network as the iSCSI server. This will fix your issues. I would also remove the route from the normal SC VLAN to the iSCSI VLAN.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[biniam]

Hi Edward,

Thanks for your help. The issue was with firewall. Its working ok now.

regards

Biniam

[Texiwill]

Hello Biniam,

What ports did you open on the firewall to make it work?

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[biniam]

Hi Edward,

For now i have just open any access from SC vlan to iscsi vlan and viceversa.

Regards

Biniam

[Texiwill]

Hello Biniam,

Thank you for posting your firewall solution. I hope others will find it useful as well. There are also specific ports you could allow if you wanted to lock this down, but having your iSCSI server as a trusted source is sometimes a better way to go it all depends on your trust relationships.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization