yoodle
Enthusiast
Enthusiast

USB datastore disappears after trying to create VM

Jump to solution

I built an ESXi instance on an 8GB Raspberry Pi using a 256Gb USB stick (plugged directly into the USB 3.0 port on the Pi).  I did a SHIFT+O at boot and enabled AutoPartition.  It automatically created a 100+ GB partition for the OS, and a 100+ GB partition for the datastore.  When it first boots up, I am able to see the space in the datastore.  But, as soon as I try to create a new VM, or upload an ISO to the datastore, it fails, and the space available in the datastore goes to 0 bytes.  I am still able to operate in the console, so I assume the USB is still accessible.  I've tried re-scanning and rebooting the PI, but it does not make the USB datastore accessible again.

 

Does anyone know what's causing this?  I'd like to be able to use the remaining space on the USB stick.

 

FYI, I was able to mount an iSCSI LUN from my Synology NAS and created another datastore on that.

 

[root@localhost:~] esxcli storage filesystem list

Mount Point                                        Volume Name                                 UUID                                 Mounted  Type            Size          Free

-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  ------------  ------------

/vmfs/volumes/5f9ecb26-4464890c-f7af-dca632c425e1  datastore2                                  5f9ecb26-4464890c-f7af-dca632c425e1     true  VMFS-6  107105746944  104672002048

/vmfs/volumes/5f4e58ba-f54b7c92-e768-dca632c425e1  OSDATA-5f4e58ba-f54b7c92-e768-dca632c425e1  5f4e58ba-f54b7c92-e768-dca632c425e1     true  VMFS-L             0             0

 

0 Kudos
1 Solution

Accepted Solutions
cyprienlaplace
VMware Employee
VMware Employee

As you ssh in, you can just type 'dmesg'. you can redirect the output to a file on a datastore and download it using the datastore browser.

I'm personally using a PNY USB SSD drive: https://www.amazon.com/gp/product/B01GQPXBQC I got only a couple of issues with it "dropping", when I was running VMs using 100% of the CPUs...

Other ones I've tried and seems to work well: https://www.amazon.com/dp/B081QS63CZ and https://www.amazon.com/gp/product/B07D7SX9NS.

View solution in original post

16 Replies
Elmiral
Contributor
Contributor

So I bought a new SSD and I moved my virtual machine -of some OS we are not supposed to mention here- files to the new SSD. It worked fine I finished what I was supposed to do closed the VM then I turned my laptop off. I opened the laptop again to find the external SSD I just bought became a RAW storage I opened the old VM files and checked the SSD it couldn't retrieve its files. I am not really sure what happened here but when I formatted the SSD it became NTFS again. Was VMware the reason it was damaged/Corrupted or was it the guest OS or was it the host OS(Windows)?

moderator: removed spam link

0 Kudos
TimSheppard
Enthusiast
Enthusiast

Have you turned off AND disabled the ISB arbitrator service?

 

https://www.virten.net/2016/11/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-5/

0 Kudos
yoodle
Enthusiast
Enthusiast

Thank you, good suggestion, but unfortunately, that does not seem to have fixed the issue.
I did a fresh install, and this is what the disks looked like right after the build:

 

[root@localhost:~] esxcli storage filesystem list

Mount Point                                        Volume Name                                 UUID                                 Mounted  Type            Size          Free

-------------------------------------------------  ------------------------------------------  -----------------------------------  -------  ------  ------------  ------------

/vmfs/volumes/5f4e5f6d-85c09401-eee0-dca632c425e1  datastore1                                  5f4e5f6d-85c09401-eee0-dca632c425e1     true  VMFS-6  114085068800  112575119360

/vmfs/volumes/5f4e5f6c-820e9f4c-982b-dca632c425e1  OSDATA-5f4e5f6c-820e9f4c-982b-dca632c425e1  5f4e5f6c-820e9f4c-982b-dca632c425e1     true  VMFS-L  128580583424  125799759872

/vmfs/volumes/7bb4a8ec-9e43033e-c681-8fbfe0ec2921  BOOTBANK1                                   7bb4a8ec-9e43033e-c681-8fbfe0ec2921     true  vfat      4293591040    4168417280

/vmfs/volumes/fdedeef3-7d4cb56f-e5bf-3f173b16cb2d  BOOTBANK2                                   fdedeef3-7d4cb56f-e5bf-3f173b16cb2d     true  vfat      4293591040    4293525504

 

I disabled the arbitrator:

[root@localhost:~] /etc/init.d/usbarbitrator stop

watchdog-usbarbitrator: Terminating watchdog process with PID 132464

stopping usbarbitrator...

usbarbitrator stopped

[root@localhost:~] chkconfig usbarbitrator off

 

Then, I tried to upload an ISO to the datastore1.  The upload got to about 13%, and then errored out.  After that, I get this message:

[root@localhost:~] esxcli storage filesystem list

Cannot open volume:

 

But, it looks like it still sees the disks:

[root@localhost:~] ls /dev/disks

mpx.vmhba32:C0:T0:L0                                          vml.010000000041413030303030303030303030343839555342204449

mpx.vmhba32:C0:T0:L0:1                                        vml.010000000041413030303030303030303030343839555342204449:1

mpx.vmhba32:C0:T0:L0:5                                        vml.010000000041413030303030303030303030343839555342204449:5

mpx.vmhba32:C0:T0:L0:6                                        vml.010000000041413030303030303030303030343839555342204449:6

mpx.vmhba32:C0:T0:L0:7                                        vml.010000000041413030303030303030303030343839555342204449:7

mpx.vmhba32:C0:T0:L0:8                                        vml.010000000041413030303030303030303030343839555342204449:8

 

 

Any other ideas?

0 Kudos
vsanjake
VMware Employee
VMware Employee

I had a very similar problem. Every time I tried to create a new VM and needed to upload an ISO file, the datastore just disappeared. I personally believe this was due to my USB storage I was using just could not handle that write operation or was overheating or something like that. I added a second USB storage device and it all worked fine and has never gone down. ESXi is still running from the original USB storage though, it was a SanDisk mini 128GB thumb drive (the super small ones) Some USB storage is just not great I think.

 

I hope that helps? Sorry I did not keep any logs or anything like that to prove it out.

cyprienlaplace
VMware Employee
VMware Employee

Hi, can you get a dmesg output? I suspect some issues with the USB support. My best suggestion today would be to try another drive.

That's sad for the 256GB one. Can you share the brand and model as well as the Vendor/Product ID (shown in lsusb)?

yoodle
Enthusiast
Enthusiast

How do I get dmesg output?  I'm not sure what you mean by that.

I do think you guys are correct though, I have a feeling it's just the USB stick that doesn't like what I'm trying to do.  It's weird that the OS works though, and it's on the same USB.  I was thinking about getting a 128Gb stick that would be dedicated to the OS and then seeing if the 256GB USB would work if it was dedicated as a datastore.  Is there a particular brand/model of USB that is recommended?  Until then, I am using an iSCSI LUN that I provisioned from my Synology NAS, and that is working well.

The stick I am using is a PNY 256GB USB3.0 that I purchased brand new from Amazon specifically for this.  I believe this is the exact model (sorry, my wife ordered, so I don't have it in my recent purchases):

PNY 256GB Turbo Attaché 3 USB 3.0 Flash Drive - (P-FD256TBOP-GE)
https://www.amazon.com/PNY-Turbo-256GB-Flash-Drive/dp/B00JN1TOHM/ref=sr_1_3?crid=2MHQW5UV35SN2&dchil...

 

0 Kudos
cyprienlaplace
VMware Employee
VMware Employee

As you ssh in, you can just type 'dmesg'. you can redirect the output to a file on a datastore and download it using the datastore browser.

I'm personally using a PNY USB SSD drive: https://www.amazon.com/gp/product/B01GQPXBQC I got only a couple of issues with it "dropping", when I was running VMs using 100% of the CPUs...

Other ones I've tried and seems to work well: https://www.amazon.com/dp/B081QS63CZ and https://www.amazon.com/gp/product/B07D7SX9NS.

Apex2
Contributor
Contributor

@vsanjake You happen to have a picture or the model number of that SanDisk mini 128GB thumb drive your using?

 

Thanks in advance 

0 Kudos
yoodle
Enthusiast
Enthusiast

This is all I see in the dmesg log.  Repeated 3000+ times:

2020-11-11T00:35:41.675Z cpu0:132045)J6: 2251: FS3_FSSyncIO failed on vol OSDATA-5f4e5f6c-820e9f4c-982b-dca632c425e1 with No connection

 

0 Kudos
Apex2
Contributor
Contributor

All,

Just bought (4) SanDisk Cruzer Glide 128GB USB 2.0 Flash Drives Model#SDCZ60-128G from Walmart and everything is working great now..

 

The only downside is i'm plugging into a USB 3.0 port on my Raspberry Pi 4 and the Flash Drives are USB 2.0 =( .... Hell.... it works until I get a NAS 😃 

 

 

yoodle
Enthusiast
Enthusiast

I rebooted my Pi today, and the USB datastore came back.  But, once again, as soon as I tried to copy an ISO to it, it disappeared.  The good news is that I got some info in the dmesg log this time (see attached).  I believe I started the copy at about 13:18.  I don't know what any of it means, but there it is.

0 Kudos
Apex2
Contributor
Contributor

@yoodle 

 

From what I can see from the logs you provided, it looks like ESXi timeout waiting on making a connection to mount your flash drive / datastore or it got hung

 

 -  'datastore1': HB at offset 3407872 - Waiting for timed out HB:2020-11-11T13:18:03.549Z cpu0:131238)

 - Waiting for timed out HB:2020-11-11T13:18:07.754Z cpu0:131238)  [HB state abcdef02 offset 3407872 gen 

 -  Long VMFS rsv time on 'datastore1' (held for 5708 msecs). # R: 1, # W: 1 bytesXfer: 16 sectors

 Take a look - https://kb.vmware.com/s/article/1002598

 

I don't think it really likes your flash drive your using for a datastore.... Might want to change that flash drive to one of the one's @cyprienlaplace mention earlier in this forum.

 

Good Luck!

 

 

2020-11-11T13:18:01.047Z cpu3:131097)Queued:52020-11-11T13:18:01.047Z cpu0:131094)ScsiDeviceIO: 3999: Cmd(0x452240ea7d80) 0x2a, cmdId.initiator=0x4302e0a91700 CmdSN 0x4ac from world 131238 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x8 D:0x0 P:0x0 Aborted at device layer2020-11-11T13:18:01.654Z cpu0:133719)HBX: 3057: 'datastore1': HB at offset 3407872 - Waiting for timed out HB:2020-11-11T13:18:01.654Z cpu0:133719) [HB state abcdef02 offset 3407872 gen 65 stampUS 2176322862 uuid 5f4e5638-e187efe5-3300-dca632c425e1 jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.1.228]2020-11-11T13:18:03.549Z cpu0:131238)HBX: 3057: 'datastore1': HB at offset 3407872 - Waiting for timed out HB:2020-11-11T13:18:03.549Z cpu0:131238) [HB state abcdef02 offset 3407872 gen 65 stampUS 2176322862 uuid 5f4e5638-e187efe5-3300-dca632c425e1 jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.1.228]-57-
 
C:\Temp\dmesg.logWednesday, November 11, 2020 9:48 AM2020-11-11T13:18:07.754Z cpu3:131806)HBX: 295: 'datastore1': HB at offset 3407872 - Reclaimed heartbeat [Timeout]:2020-11-11T13:18:07.754Z cpu3:131806) [HB state abcdef02 offset 3407872 gen 65 stampUS 2195027845 uuid 5f4e5638-e187efe5-3300-dca632c425e1 jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.1.228]2020-11-11T13:18:07.754Z cpu0:131238)HBX: 3057: 'datastore1': HB at offset 3407872 - Waiting for timed out HB:2020-11-11T13:18:07.754Z cpu0:131238) [HB state abcdef02 offset 3407872 gen 65 stampUS 2195027845 uuid 5f4e5638-e187efe5-3300-dca632c425e1 jrnl <FB 0> drv 24.82 lockImpl 3 ip 192.168.1.228]2020-11-11T13:18:07.756Z cpu3:131806)FS3Misc: 1750: Long VMFS rsv time on 'datastore1' (held for 5708 msecs). # R: 1, # W: 1 bytesXfer: 16 sectors
CaBo03
Contributor
Contributor

I know that it has been more than a year, but I a am having the exact same issue.  Please let me know if changing the flash drive was the solution for you.

0 Kudos
yoodle
Enthusiast
Enthusiast

Yes, changing out the flash drive DID fix my issues.  Sorry, I can't believe I didn't come back here and update.  I got pulled into other stuff and haven't been able to play with it since, but I did get it working.

I got the 128GB version of this drive that @cyprienlaplace mentioned earlier in the thread:
Amazon.com: SAMSUNG MUF-32AB/AM FIT Plus 32GB - 200MB/s USB 3.1 Flash Drive : Everything Else

0 Kudos
vitorinux1
Contributor
Contributor
0 Kudos
CaBo03
Contributor
Contributor

Thanks for taking the time to answer.  I will give it a try with that usb flash drive.

0 Kudos