How to connect to iSCSI SAN without compromising security

[jfields]

Hello:

How do I allow server OSes (VMs or physical hosts) to connect and mount iSCSI LUNs without compromising the security of our ESX hosts? We have some Microsoft servers that need to use iSCSI initiators to mount LUNs for MSCS. We cannot use the ESX initiators because VMware does not support iSCSI for virtual storage with MSCS. We have already read all the documentation and spoken with VMware support, so we know our only option is to use the iSCSI initiators inside the Microsoft servers to connect to the LUNs.

Our concern is related to security. If we let servers use their iSCSI initiators to connect to the SAN, then won't they also have access to our service consoles and vkernels via the iSCSI network? ESX requires that you have a service console port and vkernel port on the iSCSI network for each ESX box you wish to use the ESX initiator for. We are struggling to understand how to connect any machine (physical or virtual) to the iSCSI network to mount LUNs without exposing our service consoles and vkernels. I know the best practice is to keep VMs off this network for this exact reason, but obviously many organizations also have physical servers (UNIX, Windows) that will need access to their iSCSI SAN. How are people handling this? How much of a security issue is this? Is there a way to secure the service console and vkernel ports while also allowing non-ESX hosts access to the SAN? I know many of you are dealing with this exact situation in your organizations, so please assist. Obviously, it doesn't make sense that no one is using their iSCSI SANs for nothing else except for ESX hosts. Thank you much.

James

[Lightbulb]

Here is a thought, if you have spare target ports on the ISCSI SAN you could create a third network (dedicated switch) for VM ISCSI traffic. Then just uplink a NIC in the host to this network and setup a VM portgroup vswitch that the VMs would use when accessing the ISCSI SAN.

This solution would be dependent on whether you had the ports on the Target and an extra NIC in the host, but in this way your VM ISCSI traffic is segregated from all other networks.

Just a thought.

Message was edited by: Lightbulb Spelling

[Texiwill]

Hello,

Moved to the Security and Compliance forum.

Your VMs need to access a separate IP Network than the iSCSI network used by the ESX hosts and your backup hosts (if using VCB).

So two things need to be used.... 1) either use a separate iSCSI server and network including separate physical switches, i.e. keep both 100% separate from each other.... 2) Use a secondary/tertiary port on your iSCSI server and have it go through the 100% separate iSCSI network to your VMs.

Note on the 2nd option you have to be absolutely sure that the iSCSI server does not act as a gateway/router between the two iSCSI networks. I know NetApp has this level of security but unsure of any others arrays out there.

The key to this is 'isolation'. Isolate the VMs from the ESX hosts.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[jfields]

Thank you texiwill and lightbulb. Both of your suggestions sound interesting and are in a similar vein. By the way, I am not necessarily talking about VMs. The Microsoft servers might be virtual or physical. It appears the issue of SC exposure is the same either way.

I am not sure adding another iSCSI network would work for me, as I have a lower-tier EMC with four NICs. I don't know if it can handle separate network access to the LUNs, as you are both suggesting. We do not have another production iSCSI server, so we must use this one for this. Currently, all four NICs are serving the LUNs to the ESX boxes on the same iSCSI network. I am loath to reduce my network redundancies. Are there any other options that anyone can think of? My sysadmin suggesting using a firewall between the iSCSI network and the Microsoft VMs so that all iSCSI traffic from/to the Microsoft iSCSI initiators would have to pass through the firewall. That way, the firewall could be configured to only allow the Microsoft servers (physical or virtual) to access the EMC box in the event of the MS boxes being compromised. This would work from a security standpoint but would add a single point of failure, high costs, increased latency to the file servers, and much more complexity.

Does anyone have any other suggestions? Surely many ESX users share their iSCSI SANs with many different systems and OSes. Thanks again.

-J

[Texiwill · Accepted solution]

Hello,

Check out this blog

Use of firewalls is definitely a step in the proper direction for this. If you can not have separate iSCSI networks then you will need to isolate iSCSI from the NON-ESX/VCB nodes using other mechanisms. I would definitely opt for Firewalls or reduce redundancy to just 2 NICs per network and not 4 for one network.

Does anyone have any other suggestions? Surely many ESX users share their iSCSI SANs with many different systems and OSes. Thanks again.

They do, but they do not secure their iSCSI networks for ESX from their VMs/other physical systems. You have asked a very important question and that is how to connect to iSCSI SAN without compromising security. So options are currently:

1. Isolate physically

2. Isolate using Firewall

Since ESX talks in clear text and does not support IPsec for iSCSI you have very limited options available to you. The firewall you use and iSCSI load you send through it will determine if there is any latency. Yes its an extra expense but so is separate network switches/ports/etc.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[jfields]

Texiwill,

Thanks for the post. Your article was very interesting. What do you mean by the below comment?

+They do, but they do not

secure their iSCSI networks for ESX from their VMs/other physical

systems. You have asked a very important question and that is how to

connect to iSCSI SAN without compromising security.+

Are you saying that many admins are connecting other systems to their iSCSI networks but not worrying about the security? Incidentally, I contacted VMware support and spoke with an engineer about the risk factors. He agreed that the only options are a firewall or separate network. Even if I use physical Microsoft boxes to attach via iSCSI initiators, they will still have access the ESX SCs.

Does anyone out there have any experience with doing an iSCSI implementation through a firewall (virtual or physical) per our discussion? Any thoughts about if a virtual or physical firewall deployment would be better for this?

James

[Lightbulb]

I would assume a lot of admins have "isolated" ISCSI networks and do not think of them as security risks, which in fact they are. I suppose even though it is network communication a fair number of folks tend to think of storage in terms of total isolation. Someone compromises a system using Software ISCSI and they now have a backdoor to a lot of other systems that are probably firewalled off on the frontend, not to mention just sniffing traffic.

I work in a FC only shop, but I have experience with the difficulty of getting folks to view the internal network as a security threat (Which it more often is) If you can afford it you should totally isolate your ESX hosts from other ISCSI initiators.

Just a thought.

[Texiwill]

Hello,

+They do, but they do not

secure their iSCSI networks for ESX from their VMs/other physical

systems. You have asked a very important question and that is how to

connect to iSCSI SAN without compromising security.+

Many people connect their VMs to an iSCSI SAN that is in use by their ESX hosts without thought for security. THey do it because it A) may give perf improvements and B) it is there to use. Security is a secondary factor.

Are you saying that many admins are connecting other systems to their iSCSI networks but not worrying about the security?

Correct.,

Does anyone out there have any experience with doing an iSCSI implementation through a firewall (virtual or physical) per our discussion? Any thoughts about if a virtual or physical firewall deployment would be better for this?

I have done this, but I did not run a huge amount of traffic through it so can not give you any insight into it past that it does work. I think this is a case of trial and error. If it was me, I would setup a vFW using either IPcop or Smoothwall (two free ones) and see what performance you get. However, if it was me, I would most likely push for a physical firewall.

As Lightbulb stated, getting people to realize that the VMs are a Hostile Environment is very very difficult. No one can get past my DMZ is the often spoken words.... Then again I am not sure they have ever heard of pivot attacks.....

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[jfields]

Thank you both Lightbulb and Texiwill. You have been very helpful in pointing me towards some possible ways forward. I am still mulling over the physically separate network option versus the firewall. Either way, thank you for your help. I do think security should be discussed more and highlighted. I wonder how many VMware admins even read the Security Hardening paper by VMware? I know there are entry points into our system (and any system). We just try to design it so that an attacker must compromise multiple hosts/appliances before they get anywhere interesting. Thanks again.

-J

[jeffswanson_biz]

What about using VLANs on a speparate physical iSCSI network to isolate traffice between Windows hosts and ESX hosts? I'm not a networking guru but my initial tests shows this will work. I'm I missing something here?

Thanks, Jeff

[Texiwill]

Hello,

VLANs are NOT a security construct. However it is used as one. So in effect you are still commingling data from two different security zones on the wire, within the physical switches and within the virtual switches. There are 5 layer 2 attacks that can be used to access that data. Most currently do not work within the virtual environment but some work within physical environments. Granted this depends mostly on they switches you use.

If you use VLANs, that is a statement of trust. Some people TRUST VLANs, others do not. So basically it will boil down to the type of 'data' being commingled and the risk to the organization if the data was seen by others outside the organization. Before choosing to use VLANs involving multiple security zones, do a risk assessment so that you fully understand the threats involved and the cost to the organization if the threats were exploited.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[jfields]

Jeff,

Lightbulb suggested something similar. A separate iSCSI network would solve the issue, but the issue is that we don't have endless numbers of iSCSI ports on our SAN. The EMC only has four iSCSI ports and they are currently all used on our main iSCSI LAN. Creating a separate network would involve adding more iSCSI ports on the SAN, which is not an option for us. We ended up going with Texiwill's advice and are implementing physical firewalls that will sit between the iSCSI network and any non-ESX connections to that network.

J

[jfields]

Texiwill,

It seems a vast majority of ESX environments utilize VLANs. I also was concerned about VLAN security, so we decided to use them but in a very limited and isolated way. We are not trunking any VLAN traffic or using VLANs on our vSwitches. I am sure VLANs can be implemented safely, but I am not a network engineer. My advice to people considering VLANs is that they need to study how their particular implementation would affect their security and performance. If you are not sure how something will affect your environment's security, then you may not know the technology well enough to be employing it. To each their own, of course.

J

[Texiwill]

Hello,

It seems a vast majority of ESX environments utilize VLANs.

I agree. It is relatively easy to implement and allows for less collisions within switches, etc. VLANs make more use of the available bandwidth than you normally would so it is a usability issue as well as a cost issue. So we have usability and cost on one side, security on the other.... VLANs can be used securely as well.

I also was concerned about VLAN security, so we decided to use them but in a very limited and isolated way. We are not trunking any VLAN traffic or using VLANs on our vSwitches.

vSwitches could be considered the safer place to use VLANs as many of the layer-2 attacks against VLANs do not affect VLANs with in vSwitches.... So you trunk from pSwitch to pSwitch then split the traffic at the pSwitch to the vSwitch. This generally improves performance but not necessarily security. Depends on how the traffic GOT to the pSwitch in the first place. It also depends on if the pSwitch switching fabric has the protections in place to prevent the layer-2 attacks or not.

I am sure VLANs can be implemented safely, but I am not a network engineer. My advice to people considering VLANs is that they need to study how their particular implementation would affect their security and performance. If you are not sure how something will affect your environment's security, then you may not know the technology well enough to be employing it. To each their own, of course.

Absolutely. VLAN usage is not 'bad' per say, your use of them may not even pose a security risk. But understand those risks before making that conscious choice to employ them.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast