Virtual Disks and SCSI Commands

Version 1



    When choosing what type of virtual disk to attach to your virtual machine you sometimes decide this based on your requirements for passing through SCSI commands to a storage array.


    In the majority of cases you use add the standard vmdk's to a guest, where all SCSI commands are virtualized by ESX.


    Usually folks pick a virtual-mode RDM if they want to access existing data on that LUN (eg. P2Ving a database where the OS was on a physical server and the data is on the SAN); with a virtual-mode RDM you still get the ability for vMotion, DRS etc., which you don't with RDMp.


    If you need to issue commands from the virtual machine to the storage array, you will want to use a physical-mode raw device map (RDMp) disk.  


    This document explains which SCSI commands are passed thru when using VMDK vs. virtual-mode RDM vs. physical-mode RDMp.


    Intended Audience


    VCP and Storage Administrators in the design and deployment of VI3 with storage arrays.




    1. Virtual Disk vs SCSI Command Matrix

    2. Special RDMp use case


    1. Virtual Disk vs. SCSI Command Matrix



    The following matrix shows which SCSI commands are virtualized vs. passed through (raw) to the storage array.


    When a SCSI command is "virtualized" this means the vmkernel does not send the SCSI command to the array verbatim and instead chooses what action to perform. 


    Disk Type

    How common?

    SCSI Commands Target

    How are SCSI Commands processed?


    Most common

    .vmdk file on a VMFS

    All SCSI commands are virtualized, including READ/WRITES

    virtual-mode RDM

    minority, specific use cases

    LUN via RDM map on a VMFS

    READ/WRITEs passed thru verbatim, every other command is virtualized

    physical-mode RDMp

    rare, single case (control array)

    LUN via RDM map on a VMFS

    All SCSI commands passed thru verbatim, except REPORT_LUNS


    2. Special RDMp Use Case



    Attaching a physical-mode RDMp to a guest is a rarely used option, limited to the specific use case of allowing applications in the guest to send control SCSI commands direct to a special LUN on the array called a "gatekeeper LUN".


    An example of this is allowing a guest to send SCSI 3b/3c commands to an EMC Symmetrix Gatekeeper LUN and the Timefinder software.


    Timefinder is an EMC software technology that is included in the array’s firmware.  There is no-host side component with the exception of management.  Management of the technology means the ability to establish BCVs, de-establish BCVs (i.e. a snapshot), re-establish etc.  This is done in-band using SCSI 3b/3c commands.  SCSI 3b/3c are standardized by T10, but include bitfields which indicate a vendor-proprietary behavior.  This is what EMC uses.


    Any LU exposed by a Symmetrix can be used as a gatekeeper.  More typically, small ~60MB sized LUs are exposed with the express intention of being used exclusively as a gatekeeper device.  It is just an in-band channel for which their management stack can talk to the array.


    Disks exposed to a VM as a .vmdk or an RDM will refuse these 3b/3c commands.  Only disks which are RDMp will allow these 3b/3c commands.  Yes, this is a security risk.  If a VM with an RDMp to a Symmetrix is compromised, then in theory it can potentially destroy/compromise the entire array, depending upon what the array firmware is configured to allow that particular ESX host to do.


    So, if you're thinking of attaching an RDMp to a guest, think:


    1. Does this VM need to send SCSI control commands to the LUN?  Is there an alternative method?

    2. Do you trust the Guest VM?  If not, don't let it have an RDMp

    3. Do you trust the Host ESX?  If not, you shouldn't trust any Guest VMs that run on it, and you should use SAN/Array security to restrict this host (e.g. zoning, masking and command restrictions like no 3b/3c from this host)

    4. Do you trust the Storage Array?  If not, you have problems