This article first describes a problem of VMware ESXi 3.5 (other versions may be affected as well; feedback appreciated) which causes host-side SCSI devices, when connected to a FreeBSD guest via SCSI pass-through, to disappear immediately, presents a workaround, and discusses why it happens.
VMware ESXi 3.5
Guest running FreeBSD
One or more host SCSI devices attached to the guest via SCSI pass-through
The SCSI devices are not found upon guest start-up.
When one runs "camcontrol rescan" as root in order to rescan the SCSI bus for the devices, they appear to be found and attached normally, only to be detached immediately. Kernel messages (in bright white print on the system console; also printed by running "dmesg") resembles the following sample output:
sa0 at mpt1 bus 0 scbus3 target 0 lun 0 sa0: <HP Ultrium 2-SCSI F45Z> Removable Sequential Access SCSI-3 device sa0: 320.000MB/s transfers (160.000MHz DT, offset 127, 16bit) (sa0:mpt1:0:0:0): lost device (sa0:mpt1:0:0:0): removing device entry
The first 3 lines indicate that the device has been found and attached, then the next 2 lines indicate that it has been detached.
Note that, on the first line, the SCSI bus (scbus) number is 3 and not 0. "bus 0" is really a part of "mpt1 bus 0", which means the sub-bus #0 of the SCSI controller device named mpt1; "mpt1 bus 0 scbus3" should be interpreted as "mpt1 sub-bus #0, a.k.a. scbus3".
The problem can be worked around by selectively probing LUNs that exist (typically just LUN 0) of the affected SCSI target.
If you already know which scbus has the affected device, skip steps 2-5 and go to step 6. Otherwise, continue to step 2.
If the problematic device is attached through the same virtual SCSI adapter where another device has been attached (a virtual disk or another SCSI pass-through device), runcamcontrol devlist to locate that other device and note its scbus number, then skip steps 3-5 and go to step 6. Otherwise (i.e. the problematic device is attached via a dedicated virtual SCSI adapter with no other devices on), continue to step 3.
Run camcontrol devlist -v. You will see one or more empty scbus-es with no devices. (But discount "scbus-1 on xpt0", which is not a real but a hidden "root" SCSI bus.)
For each empty scbus identified in step 3, run camcontrol rescan N where N is the bus number.
Either examine /var/log/messages or run dmesg to see the kernel log. You will find that the problematic device has been attached then removed immediately (see example output in the Symptoms section). Note its scbus number.
At this point, you know the scbus number where the problematic device is.
Consult the guest VM configuration to make sure which SCSI target number the problematic device uses.
Consult the device documentation (usually its installation manual) to figure out which LUN(s) the problematic device uses. Usually, this is just LUN 0.
Now the real thing: Selectively probe the scbus:target:lun using camcontrol(8)'s rescan command, e.g. camcontrol rescan 3:0:0 if the device is on scbus3 target 0 lun 0.
The kernel should now find and attach the device successfully.
For future reference, record which scbus the device is on. If you later place more pass-through devices on the same virtual SCSI controller, you will be able to skip steps 2-6 of this workaround.
Warning: Never initiate a full bus scan on a SCSI bus ("camcontrol rescan 3" for example; note missing target/LUN) if another SCSI pass-through device has been attached to the bus and is working properly! A full bus scan may cause existing SCSI pass-through devices to disappear, and may panic the OS kernel if the disappeared device is already in use.
Warning: Never initiate a system-wide scan ("camcontrol rescan"; note missing bus/target/LUN) either, because this will initiate a full bus scan on all SCSI buses.
It is recommended that you place non-pass-through devices on one virtual SCSI adapter, and pass-through devices on another virtual SCSI adapter so that you can safely initiate a full scan on the non-pass-through bus, e.g. to discover newly added virtual disks.
When scanning the SCSI bus for new devices, FreeBSD issues SCSI INQUIRY command to all possible SCSI targets and LUNs (logical unit numbers) sequentially on the bus, waiting a short time after sending each INQUIRY command for a positive response. For example, when scanning for all devices on mpt1 sub-bus 0 as in the preceding example, FreeBSD sends an INQUIRY command to a total of 128 possible LUNs:
To mpt1:0:0:0 (mpt1, sub-bus 0, target 0, LUN 0)
To mpt1:0:0:1 (mpt1, sub-bus 0, target 0, LUN 1)
...
To mpt1:0:0:6 (mpt1, sub-bus 0, target 0, LUN 6)
To mpt1:0:0:7 (mpt1, sub-bus 0, target 0, LUN 7)
To mpt1:0:1:0 (mpt1, sub-bus 0, target 1, LUN 0)
To mpt1:0:1:1 (mpt1, sub-bus 0, target 1, LUN 1)
...
To mpt1:0:15:6 (mpt1, sub-bus 0, target 15, LUN 6)
To mpt1:0:15:7 (mpt1, sub-bus 0, target 15, LUN 7)
Most SCSI devices respond positively only to LUN 0, and either ignore or send negative (i.e. absent LUN) to LUNs 1-7, as is the case with the HP tape drive in the sample output above:
To mpt1:0:0:0 - inquiry
From mpt1:0:0:0 - positive
To mpt1:0:0:1 - inquiry
From mpt1:0:0:1 - negative
...
To mpt1:0:0:6 - inquiry
From mpt1:0:0:6 - negative
To mpt1:0:0:7 - inquiry
From mpt1:0:0:7 - negative
(Negative responses may as well be absent - the corresponding inquiry will time out in that case and FreeBSD will assume that no device exists at that address.)
The virtual SCSI host adapter presented by ESXi 3.5, when responding negatively to an inquiry in order to indicate there exists no device at the requested address, errneously report LUN as 0 even when the inquiry requested non-zero LUN:
To mpt1:0:0:0 - inquiry
From mpt1:0:0:0 - positive
To mpt1:0:0:1 - inquiry
From mpt1:0:0:0 - negative
...
To mpt1:0:0:6 - inquiry
From mpt1:0:0:0 - negative
To mpt1:0:0:7 - inquiry
From mpt1:0:0:0 - negative
FreeBSD, having received a positive response to LUN 0 and created the device successfully, interprets the negative response to the same LUN 0 as an indication that the device is gone now, and removes the newly created device.