Thibault
Contributor
Contributor

VMware in the DMZ and SAN security

Jump to solution

We are considering implementing VI3 VM's in our DMZ and are considering using our corporate SAN environment. Management is concerned about security of the SAN. Up to this point our security team would not allow our SAN fabric to extend into the DMZ connected servers. Is it unusual for organizations to connect DMZ servers to their corporate SAN infrastucture? Is there a real risk? Thanks everyone.

0 Kudos
1 Solution

Accepted Solutions
Texiwill
Leadership
Leadership

Hello,

Do you mean to state that your SC network will be participating in the DMZ network, or that they will be 100% separate from the DMZ network. They are correct when talking about networks but there are 4 networks involved, they should be only dealing with each separately.

Also, is the NAS going to be participating in the DMZ?

Also, is vMotion going to be participating in the DMZ?

If the SC is inside the DMZ, stop and DO NOT Implement VMware at all, you are now horribly insecure. Refuse to do this on serious security issues that your security and networking team plainly do not understand.

If the NAS is inside the DMZ, stop and DO NOT implement VMware at all using that NAS, you are now horribly insecure. Refuse to do this on serious security issues that your security and networking team plainly do not understand. These are cleartext protocols in use here.

If vMotion is inside the DMZ, stop and DO NOT Implement vMotion at all. For the exact same reasons, these are also cleartext protocols.

A good design has no cross over of networks. THe DMZ stays within the DMZ and the SC, Storage, and vMotion are all outside the DMZ most likely their own networks that should be protected by firewalls. I have not seen the design, but it sounds like your security folks need some further education on VMware Virtual Infrastructure.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill

View solution in original post

0 Kudos
15 Replies
TomHowarth
Leadership
Leadership

Thread moved to the Security and Compliance forum

Tom Howarth

VMware Communities User Moderator

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410
0 Kudos
Texiwill
Leadership
Leadership

Hello,

In the DMZ a VM should never directly access the Storage Server unless it is also within the DMZ. An ESX Server has 4 major networks that should all be separate from each other.

Administrative Network (SC, VC, etc.) this is an internal network.

Storage Network (SAN, NAS, iSCSI, Local Storage)... This is an internal and isolated network to JUST the VMware ESX hosts.

vMotion Network (Storage VMotion, VMotion)... THis is an internal and isolated network to JUST the VMware ESX hosts.

Then there is the VM network. If you keep your VMs from using iSCSI Initiators and NPIV access to the SAN Fabric everything is separate from each other. So as you can see there are really at least 4 specific networks connected to the ESX Host at all times. Only one of which really lives in the DMZ. If you setup your systems properly the VM network will not see or know about the other networks. I would do the following myself for a DMZ... Note this level of security can appear to be overkill but alleviates any future attempts to VLAN Jump and other nasties. Each of the following are on their own vSwitch.

2 pNICs for Admin Network connected to an internal firewalled network.. Make sure VC is on the same side of the firewall as your ESX Servers.

2 pNICs for vMotion Network connected to an internal isolated network. ONLY ESX hosts should be on this network. Nothing else.

2 pNICs for DMZ VMs connected to your DMZ network.

And depending on your storage. People use SAN interchangeably with iSCSI Servers these days.

2 FC-HBAs connected to the SAN. I would even go so far as to choose HBAs that do not support NPIV to alleviate even that possibility.

or

2 pNICs connected to a NAS (NFS)... Do NOT allow the VMs to see this network. Which happens naturally but you can make it so it can happen.

or

2 pNICs connected to an iSCSI Server. Do not allow the VMs to see this network. Which happens naturally but you can make it so it can happen.

You need to consider how the SAN will be accessed within your design and consider the ESX Host to be more than just one physical server. Currently there is no known way for a VM to gain access to the Service Console or any vmkernel NIC outside of bad configurations which take some doing as it does not happen by default.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
drowland
Contributor
Contributor

We've just had this same issue come up in our environment.

After meeting with our network team, storage team, and security team (yeah it was a lot of meetings), we decided we would have to purchase a NAS solution to be used only in our DMZ. Our network teams logic is they will not connect any device (except a firewall) to a DMZ network that has a connection to a corporate network. Even our Service Console connections will go through the firewall.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Do you mean to state that your SC network will be participating in the DMZ network, or that they will be 100% separate from the DMZ network. They are correct when talking about networks but there are 4 networks involved, they should be only dealing with each separately.

Also, is the NAS going to be participating in the DMZ?

Also, is vMotion going to be participating in the DMZ?

If the SC is inside the DMZ, stop and DO NOT Implement VMware at all, you are now horribly insecure. Refuse to do this on serious security issues that your security and networking team plainly do not understand.

If the NAS is inside the DMZ, stop and DO NOT implement VMware at all using that NAS, you are now horribly insecure. Refuse to do this on serious security issues that your security and networking team plainly do not understand. These are cleartext protocols in use here.

If vMotion is inside the DMZ, stop and DO NOT Implement vMotion at all. For the exact same reasons, these are also cleartext protocols.

A good design has no cross over of networks. THe DMZ stays within the DMZ and the SC, Storage, and vMotion are all outside the DMZ most likely their own networks that should be protected by firewalls. I have not seen the design, but it sounds like your security folks need some further education on VMware Virtual Infrastructure.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
drowland
Contributor
Contributor

Very good points.

I didn't explain myself very well.

When I say DMZ I was actually meaning a network behind a firewall, but I guess that isn't really correct.

Our DMZ ESX servers will be as follows:

1 network (for VM's)

1 network (for NFS) totally isolated. non-routed network

1 network (for vMotion and Service Console). This will be isolated from the VM network, but will be accessable through the firewall so the service console/DNS/NTP... can talk to it. It will be very locked down with only required ports and IP's allowed through firewall... I know it would be advisable to have the SC seperate from any other network, but we are just limited on the number of ports we can get.

Each network will be behind a firewall, but will be isolated from each other.

Thanks,

Doug

0 Kudos
Texiwill
Leadership
Leadership

Hello,

However this clearly places your SC and vMotion within a DMZ unless you are doing something like:

Internet -> DMZ Firewall -> DMZ VMs

SC/vMotion <- Internal Firewall <- Internal Systems
NFS <- Internal Firewall <- Internal Systems

If however you are doing:

Internet -> DMZ Firewall -> DMZ VMs
Internet -> DMZ Firewall -> SC/vMotion/NFS

Then you are at risk and should basically refuse to implement this non-solution.

The SC and NFS are separate from the VMs and therefore need to be considered part of your Green network Not your Orange Network (DMZ). They could be considered part of a purple network (Not quite green) as well. But by no stretch should they live within the Orange Network. Nor share the same firewall.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
michaelv
Contributor
Contributor

Hi, just resurrecting this old thread, since we're doing something similar. We have:

Internet-&gt;DMZ firewall-&gt;DMZ VMs

and the SC has a DMZ IP, but the DMZ firewall blocks:

Internet-&gt;DMZ firewall-&gt;SC

(the SC is only accesible by going LAN-&gt;DMZ firewall-&gt;SC).

Is the above OK? We used to have the SC with a LAN IP and it was connected to a physical NIC, connected to our internal network, but we were advised that this was a security risk, since if the ESX server was compromised there would be a connection into the internal network separate from the path through the firewall.

Appreciate any advice you can provide, or pointers to the relevant documentation.

Thanks,

Michael

0 Kudos
Texiwill
Leadership
Leadership

Hello,

and the SC has a DMZ IP, but the DMZ firewall blocks:

SC should never touch, be a member of your DMZ.

Internet-&gt;DMZ firewall-&gt;SC

Not what you want at all. I would consider this:

Internet <-> DMZ <-> DMZ hosts <-> Firewall <-> SC

(the SC is only accesible by going LAN-&gt;DMZ firewall-&gt;SC).

Not if your DMZ firewall fails for some reason., you need to have your SC protected at all times from the DMZ. Also, your SC is accessible by ANY DMZ host, as it does not need to go through the firewall so in essence your SC is still within the DMZ and if your DMZ is attacked the SC is still at risk.

Is the above OK? We used to have the SC with a LAN IP and it was connected to a physical NIC, connected to our internal network, but we were advised that this was a security risk, since if the ESX server was compromised there would be a connection into the internal network separate from the path through the firewall.

So in essence they were saying if you can escape the VM then all bets are off. Yes that is true, but there has NOT been an escape of the VM on ESX. They were also saying that if some one broke into your LAN then they could possibly access the ESX SC/management appliance and all bets are off. Yes that is also true, that is why your ESX service consoles should be behind an additional firewall to form a virtualization management network where all virtualization management tools reside including vCenter, Lab Manager, etc. and items like VIMA.

Basically if someone can reach your SC from anywhere you are at risk. But placing a SC inside a DMZ to protect against this basically makes it a nice juicy target for attack. Consider it from the physical sense.... You have a very expensive sought after device and you label it as such, show it to the world through a glass window. A thief knowing this would just break the glass and take the device. Instead you want to put a solid door between that glass and your expensive sought after device. Showing it to the world would attract attention, placing it behind opaque surfaces does not attract as much attention. Basically you want to bury the SC within your LAN and leave it out of the DMZ even with firewall settings you have. I.e.

Internet <-> DMZ FW <-> DMZ <-> Internal FW <-> Internal Network <-> FW <-> SC

You really do not want the SC anywhere near the DMZ, keep it as far away as possible.

Appreciate any advice you can provide, or pointers to the relevant documentation.

Check out http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links for a list of links, specifically the DMZ whitepapers for ESX.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
michaelv
Contributor
Contributor

Hi Edward, really appreciate the fast and comprehensive reply. What you are describing is the way I had it set up previously, and we will make the necessary changes to go back to doing things that way. Those links look like they will be very helpful too, and I will review our configuration with them in mind.

Thanks again for your help,

Michael

0 Kudos
psaucourt
Contributor
Contributor

Hello Edward,

There is a lot of interesting things in this post but something is not really clear for me. Do these recommandations you provided are differents when you use ESX Installable in the DMZ ? As there is no more Service Console in this version we can imagine that there is no more threats, and in this case, let the default VMKernel port in the unsecure network.

Another question. I have defined a network architecture following your recommandations and ESX admin ports are localized in a specific VLAN (For ESX boxes in LAN and DMZ). The vCenter Server will be also connected to this VLAN. As vCenter will be a member of our Active Directory domain, I will have to connect this machine to our LAN. But in this case, I will create a bridge between the Admin VLAN and the LAN. If I remember right, you wrote that we have to isolate the vCenter Server. If we do that we will lose a lot of services available in the internal network, like centralized administration, supervision, Windows Update, etc. Security is important but we need also easy management.

What could you propose in this situtation ?

0 Kudos
Texiwill
Leadership
Leadership

Hello,

There is a lot of interesting things in this post but something is not really clear for me. Do these recommandations you provided are differents when you use ESX Installable in the DMZ ?

ESX is installed on a host. That hosts network connections can live in many networks. the Management network should NEVER be within the DMZ. Nor should your storage network. Only the network used by the DMZ VMs should be within the DMZ network.

As there is no more Service Console in this version we can imagine that there is no more threats, and in this case, let the default VMKernel port in the unsecure network.

There is a Management Appliance for ESXi and a Service Console for ESX. They are pretty much both Management Appliances and should never be within the DMZ. No vmkernel port should ever be in a DMZ only the network ports for the DMZ based VMs.

Another question. I have defined a network architecture following your recommandations and ESX admin ports are localized in a specific VLAN (For ESX boxes in LAN and DMZ). The vCenter Server will be also connected to this VLAN. As vCenter will be a member of our Active Directory domain, I will have to connect this machine to our LAN. But in this case, I will create a bridge between the Admin VLAN and the LAN. If I remember right, you wrote that we have to isolate the vCenter Server. If we do that we will lose a lot of services available in the internal network, like centralized administration, supervision, Windows Update, etc. Security is important but we need also easy management.

What could you propose in this situtation ?

Here is what I do:

Production Network <-> Firewall <-> VI Admin Network <-> ESX Hosts/vCenter/management nodes/etc.

The firewall should be able to allow systems on the VI Admin Network to make AD requests into the Production AD server and any other requests as necessary, but it should not allow just any system to connect for purposes of remote consoles, etc. I am not sure where the usability issues occur in this. You use this internal firewall to keep non VI-Admin users from getting into systems they should not access. Those systems, like the vCenter server should be able to make AD and Windows Update requests as necessary.

This is only for the VI Admin Network and not the VMs that live within the Production network etc.


Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
psaucourt
Contributor
Contributor

Thank you for your anwser, Edward. As we have no more port available on the firewall (and we won't use 802.1Q), we will configure access-lists on the switch to do the job.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

You can also use a virtual firewall between your production vSwitch and your Admin vSwitch as well to enable this depending on your virtual network setup.


Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
MichaelW007
Enthusiast
Enthusiast

I'm working on a large virtualisation project at the moment that will virtualise the DMZ and interior zones. Each zone will have separate hosts, networking and firewall infrastructure for VM's. However we will be using a common SAN, but with LUN's only zoned to a particular security zone (single initiator zoning). No NPIV, No RDM's. Management networks (ESXi embedded) all completely separate on completely separate switching infrastructure.

All the infrastructure separated for different firewalls and no access or knowledge of the management infrastructure by any of the VM's. The VM's will also have no knowledge of the underlying storage. We will not be using VLAN's for the VM networks, only physical switches and NICs. But we will be using VLANs in the management network. The biggest risk I see with this is the possibility of a DoS attack causing excessive storage resource usage. We'll be mitigating that by using SOIC.

I have other clients that have collapsed DMZ networks and storage onto the same set of hosts (within a single cluster), utilising VLAN's to keep the DMZ separate from Internal and either virtual or physical firewalls for separation, and using common SAN storage. They have had no problems. But this won't work for all organisations.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

If your 'fabric' is separate as you say, then you are fine. If you need data to sit only on specific LUNs and travel over specific wires, then it is a fabric issue more than an ESX issue. You also have to worry about other attacks so you must protect your fabric using auditing to ensure that only ESX hosts see or touch the LUNs in question.

There are a number of attacks that could not only effect DoS but MiTM as well.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos