One Esx clusTer for both Internal and DMZ VMS

[gmahanger]

I was wondering what security elements we need to understand before configuring an ESX cluster to have vswitch connected to a dedicated nic in our internal network hosting internal servers and, a vswitch connected to a dedicated DMZ nic with dedicated DMZ servers. What is the possibility that a DMZ server could become compramised and impact our internal network? is this a bad idea or could it work ?

[zippy_thewonder]

Whilst this is definitely technically possible, you probably wouldn't see this too much in a production/enterprise environment. You obviously couldn't have a hardware firewall solution between the internal and DMZ networks, so you would need a VM connected to both networks running some form of firewall software (ISA or something of the like), and very careful with the firewall ruleset.

Could it work? Yes, definitely. Is it a bad idea? Well, it could be a cost-effective idea for small businesses, but from an architecture point of view, probably not the best idea. Would be a perfect idea for test/dev though.

[HyperSprite]

There is no greater danger than you already have, there is no method of attacking your VMs and hopping to other VMs on the same host and the VMs themselves have no control over what port group they are connected to.

Basically, the more ESX servers that are group together, the more useful they become. I would do it with multiple port groups vlanned on trunked lines on as many NICs as I could spare after assigning vmotion and console networks (these should have separate networks unrelated to any of your VM traffic for security). Since there is no crossover communication between port groups, there is still strong security and the benefit of more redundancy, throughput and flexibility.

For instance run 1 port group for external DMZ net, 1 for internal DMZ net, 1 for backend net (going further, 1 for testing DMZ, 1 for internal testing DMZ, 1 for testing back end, 1 for dev etc. etc.) You still place your physical firewalls between your equipment by connecting them to the proper vlans on your physical switches. Then put the VMs in resource pools to manage priority.

Obviously careful administration and documentation should be observed in this kind of environment to avoid issues, but that should go without saying.

[Texiwill]

Hello,

Penetration testers and hackers alike will attempt to use the DMZ machines to pivot attacks against internal systems. To help prevent this you need to have a firewall before your DMZ and before your internal hosts. Other prevention is auditing and monitoring the Production and DMZ systems.

Network <-> FW <-> DMZ <-> FW <-> Production

Some even go as far as having multiple layers of production with more and more security with firewalls between them all. That is basic networking security statements. However, there are some issues regarding Virtualization that do not fit the above model.

If you use portgroups on a vSwitch it is possible that the vSwitch could be compromised using encapsulation attacks, so therefore you will want to keep your Production and DMZ vSwitches separate and not use portgroups for these on the same vSwitch. Also due to the nature of ESX and how easy it would be to accidentally or purposefully to place a Production VM in the DMZ it is recommended that you have a DMZ VI3 Cluster and a Production VI3 cluster, and not mix the two on the same host. For some people this is overkill, for others it is a necessity and depends entirely on how much you trust your internal folks. Remember that 70% of all attacks come from the inside. Lastly, depending on how the vSwitch/portgroup is secured and how the VMs are attached to the vSwitch it may be possible for a VM to access all data on a vSwitch (lots of ifs in this statement and would take a purposeful set of changes to make it possible, again it boils down to trust).

The best security is therefore to have one cluster for DMZ and one cluster for production never mixing the VMs from one cluster to another with a physical firewall between the two clusters. This solution alleviates the issues where production VMs can appear within the DMZ easily, it would take quite a bit for this to happen and would alleviate one aspect of human error.

The second best security would be to have a vSwitch for DMZ and a vSwitch for Production and thereby having separation at the pNIC and a physical firewall between the two networks (granted you could use a software firewall like smoothwall, but I find that this causes some other issues with vMotion, etc. .depending on configuration. This configuration also has less security as it is possible to very easily place a Production VM accidentally or purposefully within the DMZ (there is no security inherent within ESX to prevent this). Human error or on purpose changes can cause pretty major security issues.

The never really do option would be to use DMZ/Production on the same vSwitch using VLANs... There are just too many attacks against this configuration to make it worthwhile, you really want physical separation of these networks within ESX and preferably outside ESX as well.

In either case increased vigilance is required as with any DMZ setup.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[ODOCChuck]

It seems as though your missing an option where you have a Clustered ESX environment for your Production & DMZ, but you separate these out with Physical ports. The risk of human error would be minimal because you would actually have to change the IP address of the production server in order for it to be routable in the DMZ network. Just keep the Service console and VMotion networks out of the DMZ and there should not be any chance of being compromised.

It seems for every opinion that this is a secure method, there is someone else saying that it is not secure. Does VMware have an official stance and Best Practice on setting up with this configuaration?

[dkfbp]

We are running DMZ and Internal network on the same ESX servers. I know of a lot of big companies who is doing the same. This is a secure solution - it all depends on knowing the challenges.

[Texiwill]

Hello,

Absolutely. If you understand the risks it is a very good solution, unfortunately most people do this with out fully understanding the risks involved. The biggest risk I see to having the DMZ and Production on the same VI3/ESX cluster is that it is trivial to move a VM from one network to another. It can be done by accident or on purpose very easily. Because of this ease of moving VMs most security experts will recommend separate clusters. Once there is more default security around the ability to change networks then the recommendation will change.

But can you have one system with both DMZ and Production on the same system, of course, just be cognizant of the issues around placement of VMs.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[Charu]

One correction: you state that vSwitches could be vulnerable to encapsulation attacks. This is not true. The vSwitch automatically drops any double-encapsulated frames sent from a VM, so they are immune to this type of attack.

To see all the other types of attacks which they are not susceptible to, please see the Security chapter of the ESX Server Configuration Guide, in the section on Virtual Switch Protection and VLANs (there are several versions of this guide, but this section is the same for them. Try http://www.vmware.com/pdf/vi3_35/esx_3i_e/r35/vi3_35_25_3i_server_config.pdf)

[dpomeroy]

Isn't some of the risk from accidently putting an internal VM on the DMZ mitigated by not using DHCP on DMZ networks? We do mix DMZ and Internal on the same ESX servers, with separate pNICs and vSwitches for DMZ network, and all vSwitch security enabled, to me this is secure enough for most environments. If someone accidently put an internal VM on the DMZ vSwitch it would lose network connectivity and we would know immediatly that a mistake was made. Of course I'm not a security expert, so maybe I'm wrong

Don Pomeroy

VMware Communities User Moderator

[Texiwill]

Hello,

You stated the most important thing about security 'This is secure enough for me' and you understand that the change in networks will give you some form of warning either as an unreachable host or the host giving errors about not reaching the network. WIth this information you can determine that a mistake was made or something else happened. But if you had DHCP, or the network octets were the same would you then know that a problem occurred.

Granted quite a bit of misconfiguration would be necessary but that happens all the time as well.

Also, just having a live ethernet device on the network implies that the MAC Address can be known and if that is the case there are attacks that just target mac addresses, etc.

It is possible to design the most secure ESX server in the world but that may cause issues with usability. But everyone has a statement of What is Secure enough and understand that they have not covered all bases but can use monitoring and other tools to cover the possible issues. If you know about the issues ahead of time then 'Secure Enough' is fine.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[legacyb4]

Great reading for the ESXi beginner. Just moving into having some real virtual server capable hardware and wanted to decomission some of the dev VMware Server based hosts I have been using.

Based on what I've been reading, the following is pretty standard for using a single ESXi server to consolidate VM server hosting?

Hardware firewall (multiple NICs for external (NIC0), internal (NIC1), and DMZ (NIC2).

Firewall NIC0 &lt;-&gt; Internet

Firewall NIC1 &lt;-&gt; pSwitch &lt;-&gt; ESXi NIC1 (VMkernel) - management network

Firewall NIC1 &lt;-&gt; pSwitch &lt;-&gt; ESXi NIC2 (VM Port Group) - internal network VM servers

Firewall NIC2 &lt;-&gt; pSwitch &lt;-&gt; ESXi NIC3 (VM Port Group) - DMZ network VM servers

These are all application servers so not needing any fancy configuration for storage, etc.

Cheers.

[Shmeds]

I've read this with great interest as I have issued a design using separate vSwitches for DMZ and Internal with separate pNics going into the same pSwitch using VLANs for separation and hardware firewalls to route and separate the networks.

I think that this is a secure design, and as has been said elsewhere, it depends on how secure you need it. If I don't have control over the security requirements, does anyone know what NSA or CESG might say about this?

[Texiwill]

Hello,

Most ultra secure solutions will not mix a DMZ on the same physical switches that are used by other networks. The problem occurs in the physical network layer more than the virtual network layer. Use of VLANs as a security construct is not ideal in these situations. Mainly because 802.1q VLANs are not a security construct and depend on the TRUST that your physical switch layer would NOT be susceptible to layer 2 attacks.

Segregation on the vSwitch implies segregation in the physical realm as well. Ideally you should have your DMZ use a separate pair of physical switches. Segregate all networking for a DMZ from any other networking.

If you do not segregate your DMZ network in the physical world then segregation on the vSwitch is not for security reasons but for performance reasons. The vSwitch is not susceptible to currently known layer 2 attacks while your physical layer could be.

Best regards, Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]