How to ensure data separation of different data types for ESXi 4?

[VMwear8]

Does anyone have a reference architecture for ensure data separation from point of creation, in transit, and in storage for various types of dataexample PCI DSS data (personal credit card data), ITAR/EAR data, etc. Concern is that due to VLAN data comingling or crossover, VLANs in and of themselves should not be used as a security boundary. VMware suggests ensuring separation at the application levelif I read it correctlywell in transit even into storagethe data must be kept separate. Suggestions?

[Texiwill]

Hello,

Do you mean within the virtualization host itself, or within the virtual environment which include physical switches. This is a huge issue and has been discussed in many areas. However, to say VLANs are insecure is not really the issue, it is an issue about the data and the paths that data takes.

If you are concerned about VLANs please listen to the Virtualization Security Podcast which included Brad Hedlund from Cisco and read 'Rethinking vNetwork Security' from The Virtualization Practice (http://www.virtualizationpractice.com/blog/?p=4284) as this will give you a summary of various issues to think about with regards to 10g and other links which require VLANs.

You absolutely need separation and you can achieve this within the virtualization host to a certain extent but this depends on your level of risk. Also note that PCI is working on guidance for virtualization and may soon have some level of guidance.

Best regards,
Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009

Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security'[/url]

Also available 'VMWare ESX Server in the Enterprise'[/url]

Blogging: The Virtualization Practice[/url]|Blue Gears[/url]|TechTarget[/url]|Network World[/url]

Podcast: Virtualization Security Round Table Podcast[/url]|Twitter: Texiwll[/url]

[VMwear8]

Do you mean within the virtualization host itself, or within the virtual environment which include physical switches--yes both. Thank you Edward.

[VMwear8]

VLANs as an issueI didn't really explain that well. Some engineers think that VLANs in and of themselves are secure and are using them as security boundaries. And based on that thinking, VLANs are used pervasively around and within virtual infrastructure even as part of data paths. Researchers (and I think Cisco) published papers on data bleeding on VLANs. An example, I reviewed a design the other day that showed two separate types of data, international and US person data, in two separate ESXi 4.0 clusters with dual I/O paths to SAN storagebut separation failed when it hit the same virtual switch to get to shared SAN storage ---and there was no data separation in storage. Data is not encrypted in transmission or storage. So, the design was gently rejected due to data protection issues.

[Texiwill]

Hello,

VLANs as an issue--I didn't really explain that well.  Some engineers think that VLANs in and of themselves are secure and are using them as security boundaries.  And based on that thinking, VLANs are used pervasively around and within virtual infrastructure even as part of data paths. Researchers (and I think Cisco) published papers on data bleeding on VLANs.

VLANs within the vNetwork are actually quite secure. None of the known layer 2 attacks will work within the VMware vSwitch, Nexus 1000v, or vDS are all authoritative so are secured from layer 2 attacks. STP attacks may work with the Nexus 1000v, I am unsure. It is badly configured physical switches that are the basis for all these attacks. BTW, VLANs are NOT a security construct unless married with some form of ACL, ie. some PVLAN constructs.

An example, I reviewed a design the other day that showed two separate types of data, international and US person data, in two separate ESXi 4.0 clusters with dual I/O paths to SAN storage--but separation failed when it hit the same virtual switch to get to shared SAN storage ---and there was no data separation in storage.  Data is not encrypted in transmission or storage.  So, the design was gently rejected due to data protection issues.

Step back a second. You have distinct clusters for the data. Then you say they hit the same vSwitch... This is actually physically impossible. Each cluster has its OWN set of vSwitches and the vSwitches are NOT shared between clusters.

If you are really talking about using shared LUN storage instead of two distinct LUNs then that is definitely an issue. If you are commenting on the pSwitch that sits before the storage array then you also may have an issue. But there are two clusters, two sets of data, and at LEAST two vSwitches involved for the data transfer. There may be one Fabric or IP storage switch involved however.... Not a virtual concern more a physical concern I would say.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

[VMwear8]

Great stuff and thanks for pointing me to the article; certainly I think one can equate the Kernel network stack to a physical network switch backplane when considering whether there is any actual intermingling of data packets. I guess to me one question is what is the attack target in these environments? From a network perspective, we’re typically concerned with resources (workstations, servers, applications) attached and/or accessed via the network. The shared article shared seems to suggest that virtual technologies can be effective in maintaining separation between VLAN communication even though data packets from multiple VLANs may at times exist in a shared location (Kernel). It’s my impression that this scenario would be of limited concern since data packets can be used to attack target--and some network engineers would say that data packets don’t attack other data packets in transit. There are several data packet attacks tho' and data packets target VLAN resources. If we trust the Kernel in virtual environments and the physical switch backplane when implemented using the proper controls for VLAN separation, have we implemented sufficient network security? Do the proper controls exist? It seems to me that this is part of the dilemma.

Edward, is that the conclusion you expected me to reach? And I’m curious if you reached a similar conclusion . . . or I just missed the point of the article entirely. Thank you.

[Texiwill]

Hello,

Not sure that is the conclusion, but you are asking the proper questions. Eventually you have to 'Trust' Something. In the Hypervisor, you have to trust the kernel is doing the proper thing. Then think of any attacks that are known in the network world that could exacerbate that trust. BTW, I do not know of any. You have multiple vSwitch control planes, you have multiple VMs, etc....  The layer-2 vSwitch backplane so to speak is actually above the shared kernel bits and the driver, not part of the shared kernel bits.

Here is the rub, if YOU do not trust the Kernel, then you really have an issue with ANY computing device and NOT with virtualization in particular.... So where would you put your trust?

However, your description of why the setup was 'bad' is very confusing at best... Where is the break down in the virtual network? I do not even think the virtual network is involved. You have 2 clusters of ESX hosts talking on separate vSwitches to perhaps a single pSwitch. So is the breakdown in the virtual network or the physical network? Or is this not as you describe?

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

[VMwear8]

Sharing a message sent to me:  11 Jan 2011:  At this time VMware is not persuing a MLS implementation.  Our current assurance level is EAL4+.  MLS requires EAL6 with equivalent protection profiles for various security controls.  The fundamental technology will satisfy the isolation requirements.  It is all the other components that need to be validated.  Be this as it may, let us know if you would like to partner with VMware on creating a reference architecture.  There is a lot of work that has to be done to make this happen and the majority is not technical.  Regards, HD.

[Texiwill]

Hello,

I have been doing quite a bit of research into the problem of an Administrator being able to access data when they should not be able to do so. There are three possible approaches to this:

1) ensure any administrator that can log directly into a host, or access the file browser within vCenter is granted the proper classification to see such data.

2) disable file browser access via vCenter, and ONLY allow the use of sudo, use a root password vault for a 'break glass situation', disable vifs from vMA but in that break-glass situation see #1.

3) Somehow encrypt memory

4) Implement some form of MLS inside the vmkernel

5) TRUST your administrators but audit for abnormalities.

The issue boils down to, due you TRUST your administrators. In the cloud, I do not know my administrators so TRUST is a huge issue, but within my own environment I would hope you do have some level of TRUST in your administrators.

The big issue here, is WHY should a virtualization or cloud administrator even be able to peer into a Virtual Disk, Virtual Memory, or a Virtual Network at all? What use case could possibly exist that would require this level of access? One could claim file recovery, but that is usually handled by backup software not at the hypervisor level.

For Now we do #5 which is why HyTrust exists, as well as tools like RSA Envision and a few others. We TRUST but Audit, Monitor and implements the controls we can.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

[VMwear8]

Thank you Edward.  Excellent points.  Do I trust them?  No.  I don't know any of them.  And will probably never meet them.  I don't know my virtual admin, system admin, database admin, application admin, or network admin in the public or private Cloud or cloudless.  Do you know how to best secure a hypervisor, so no one can mess with it?  Thank you.

[Texiwill · Accepted solution]

Hello VMwear8,

The chances of anyone messing with the Hypervisor itself is pretty slim actually, what you need to be concerned about is the data being accessed within the VMs. So you need to protect administrators from accessing that Data.

At the moment that is not 100% possible, however you can limit access by implementing various controls:

1) Run all access to your ESX hosts and vCenter through the HyTrust appliance, it is a preventative in some cases.

2) Disable Datastore Browsing within the vSphere Client (it is a vCenter RBAC)

3) Disable 'root' or superuser to each ESX host (except in a 'break glass' situation). If they do need 'root' access track this via sudo.

4) Add auditing so you know who did what when where and how

5) Protect your management tools as if they were gold. Check out this post Security of Performance and Management tools within the Virtual Environment as an aid to helping with this.

So at most you can only put controls around the problem, we cannot currently definitively prevent access to someone you grant that access such as being able to login as the superuser in the ESX host.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

[VMwear8]

Hello Edward, How do you doublecheck to ensure that the hypervisor/VMkernel are really secure?

To keep the VMkernel secure, someone would have to develop tests to ensure the security of the kernel module integirty, memory hardening, and Trusted Platform Modeul (TPM).  If you have a reference to point to as a starting place, that would be godd.

And to keep admins and users from the data -- then perhaps configuring LUN masking and preventing a VM to talk to another VM would be some steps to ensure  that admins don't access virtual machine via additional protection--but how could one prove that an admin or user did not access the data?

Let's say we have VI in the UK and by law all admins must be UK citizens.  They are not allowed to access USA data . . . could tests be developed to test this to ensure that there is no data breach?  Thank you, VMwear8

[VMwear8]

"The big issue here, is WHY should a virtualization or cloud administrator even be able to peer into a Virtual Disk, Virtual Memory, or a Virtual Network at all? What use case could possibly exist that would require this level of access? One could claim file recovery, but that is usually handled by backup software not at the hypervisor level."

Proably better reasons, but typically if the storage admins are executing storage reclamation via a utility (for EMC Symmetrix VMaxes local and remote), 1) will need to intercede if anything goes south for one and backout what they've done.  And 2) to ensure data migrations from legacy arrays have been transferred 100 percent to the VMaxes . . . and 3) to ensure that the old legacy EMC Clarrions, LUNs, old resoruce pools, etc. are santized prior to destruction.  Peering into the configuration of the ecosystem for any reason is possible, but misconfiguration checks are common.  Hope that helps. Thanks again.

[Texiwill]

Hello,

Yes TPM and TXT are possible. TPM verifies the boot volume, TXT attests to the integrity of the kernel booted from the trusted boot volume as well as the launch of a vmware-vmx (at least in prototypes I have seen).

However, these are only boot time options, not related to issues during run-time.

You can verify where a host has booted, but NOT where each VM on the host has ever lived. And specifically you cannot tell where your data has ever been (UK vs US for example.)..... A host yes, a VM no, and Data definitely not without other controls in place.

Remember you need to concentrate on the data and not necessarily the VM or host....

As for tests to determine if the hypervisor has been attacked. Remember the hypervisor is the kernel launched off the trusted boot volume, everything else is pretty much management. Hypervisor's run virtual switches and virtual machine objects, plus all the availability objects. There is currently no way to determine if any of these have been impinged upon outside of boot time check. Is this coming, yes, is it here yet, no. For now you have to TRUST the hypervisor.

Most if not all hardening of a vSphere ESX/ESXi host is about the management appliance and vCenter. No elements are about the hypervisor directly. There are vSwitch elements (whcih must be done), VM elements (such as setting isolation settings), and guest OS elements (handled by gues OS hardening guidelines).

Maybe some people consider these the hypervisor, but I do not. and at the moment if I can trust the launch of the vmkernel, I am forced to trust it during runtime. That is as far as we have gotten with a hardware root of trust implementation. I hope to see this change over the next year.

Also, being able to manipulate the gross aspect of data (LUNs, files, etc.) does not mean that the administrator has the RIGHT to peer into the data for any reason. These are different aspects of the same problem. Yes, I know the files exist so that I can restore them, but I do not need to read to see the data to restore it, just the file name, LUN ID, etc. So we need to define who has rights to see what aspect of the data. I would say:

filenames, gross aspects of data: storage, virtualization administrators, backup administrators.

configuration details: application administrators

data itself: DBA, users.

I am sure there are other roles involved I have not listed.

Best regards,

Edward L. Haletky

Communities Moderator, VMware vExpert,

Author: VMware vSphere and Virtual Infrastructure Security,VMware ESX and ESXi in the Enterprise 2nd Edition

Podcast: The Virtualization Security Podcast Resources: The Virtualization Bookshelf

[mila30]

To Texiwill:

Thank you for your shared information.  I have learned a lot and will definitely use the ideas you shared in my future activities