I built an ESXi instance on an 8GB Raspberry Pi using a 256Gb USB stick (plugged directly into the USB 3.0 port on the Pi). I did a SHIFT+O at boot and enabled AutoPartition. It automatically created a 100+ GB partition for the OS, and a 100+ GB partition for the datastore. When it first boots up, I am able to see the space in the datastore. But, as soon as I try to create a new VM, or upload an ISO to the datastore, it fails, and the space available in the datastore goes to 0 bytes. I am still able to operate in the console, so I assume the USB is still accessible. I've tried re-scanning and rebooting the PI, but it does not make the USB datastore accessible again.
Does anyone know what's causing this? I'd like to be able to use the remaining space on the USB stick.
FYI, I was able to mount an iSCSI LUN from my Synology NAS and created another datastore on that.
[root@localhost:~] esxcli storage filesystem list
Mount Point Volume Name UUID Mounted Type Size Free
------------------------------------------------- ------------------------------------------ ----------------------------------- ------- ------ ------------ ------------
/vmfs/volumes/5f9ecb26-4464890c-f7af-dca632c425e1 datastore2 5f9ecb26-4464890c-f7af-dca632c425e1 true VMFS-6 107105746944 104672002048
/vmfs/volumes/5f4e58ba-f54b7c92-e768-dca632c425e1 OSDATA-5f4e58ba-f54b7c92-e768-dca632c425e1 5f4e58ba-f54b7c92-e768-dca632c425e1 true VMFS-L 0 0
As you ssh in, you can just type 'dmesg'. you can redirect the output to a file on a datastore and download it using the datastore browser.
I'm personally using a PNY USB SSD drive: https://www.amazon.com/gp/product/B01GQPXBQC I got only a couple of issues with it "dropping", when I was running VMs using 100% of the CPUs...
Other ones I've tried and seems to work well: https://www.amazon.com/dp/B081QS63CZ and https://www.amazon.com/gp/product/B07D7SX9NS.
So I bought a new SSD and I moved my virtual machine -of some OS we are not supposed to mention here- files to the new SSD. It worked fine I finished what I was supposed to do closed the VM then I turned my laptop off. I opened the laptop again to find the external SSD I just bought became a RAW storage I opened the old VM files and checked the SSD it couldn't retrieve its files. I am not really sure what happened here but when I formatted the SSD it became NTFS again. Was VMware the reason it was damaged/Corrupted or was it the guest OS or was it the host OS(Windows)?
moderator: removed spam link
Have you turned off AND disabled the ISB arbitrator service?
https://www.virten.net/2016/11/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-5/
Thank you, good suggestion, but unfortunately, that does not seem to have fixed the issue.
I did a fresh install, and this is what the disks looked like right after the build:
[root@localhost:~] esxcli storage filesystem list
Mount Point Volume Name UUID Mounted Type Size Free
------------------------------------------------- ------------------------------------------ ----------------------------------- ------- ------ ------------ ------------
/vmfs/volumes/5f4e5f6d-85c09401-eee0-dca632c425e1 datastore1 5f4e5f6d-85c09401-eee0-dca632c425e1 true VMFS-6 114085068800 112575119360
/vmfs/volumes/5f4e5f6c-820e9f4c-982b-dca632c425e1 OSDATA-5f4e5f6c-820e9f4c-982b-dca632c425e1 5f4e5f6c-820e9f4c-982b-dca632c425e1 true VMFS-L 128580583424 125799759872
/vmfs/volumes/7bb4a8ec-9e43033e-c681-8fbfe0ec2921 BOOTBANK1 7bb4a8ec-9e43033e-c681-8fbfe0ec2921 true vfat 4293591040 4168417280
/vmfs/volumes/fdedeef3-7d4cb56f-e5bf-3f173b16cb2d BOOTBANK2 fdedeef3-7d4cb56f-e5bf-3f173b16cb2d true vfat 4293591040 4293525504
I disabled the arbitrator:
[root@localhost:~] /etc/init.d/usbarbitrator stop
watchdog-usbarbitrator: Terminating watchdog process with PID 132464
stopping usbarbitrator...
usbarbitrator stopped
[root@localhost:~] chkconfig usbarbitrator off
Then, I tried to upload an ISO to the datastore1. The upload got to about 13%, and then errored out. After that, I get this message:
[root@localhost:~] esxcli storage filesystem list
Cannot open volume:
But, it looks like it still sees the disks:
[root@localhost:~] ls /dev/disks
mpx.vmhba32:C0:T0:L0 vml.010000000041413030303030303030303030343839555342204449
mpx.vmhba32:C0:T0:L0:1 vml.010000000041413030303030303030303030343839555342204449:1
mpx.vmhba32:C0:T0:L0:5 vml.010000000041413030303030303030303030343839555342204449:5
mpx.vmhba32:C0:T0:L0:6 vml.010000000041413030303030303030303030343839555342204449:6
mpx.vmhba32:C0:T0:L0:7 vml.010000000041413030303030303030303030343839555342204449:7
mpx.vmhba32:C0:T0:L0:8 vml.010000000041413030303030303030303030343839555342204449:8
Any other ideas?
I had a very similar problem. Every time I tried to create a new VM and needed to upload an ISO file, the datastore just disappeared. I personally believe this was due to my USB storage I was using just could not handle that write operation or was overheating or something like that. I added a second USB storage device and it all worked fine and has never gone down. ESXi is still running from the original USB storage though, it was a SanDisk mini 128GB thumb drive (the super small ones) Some USB storage is just not great I think.
I hope that helps? Sorry I did not keep any logs or anything like that to prove it out.
Hi, can you get a dmesg output? I suspect some issues with the USB support. My best suggestion today would be to try another drive.
That's sad for the 256GB one. Can you share the brand and model as well as the Vendor/Product ID (shown in lsusb)?
How do I get dmesg output? I'm not sure what you mean by that.
I do think you guys are correct though, I have a feeling it's just the USB stick that doesn't like what I'm trying to do. It's weird that the OS works though, and it's on the same USB. I was thinking about getting a 128Gb stick that would be dedicated to the OS and then seeing if the 256GB USB would work if it was dedicated as a datastore. Is there a particular brand/model of USB that is recommended? Until then, I am using an iSCSI LUN that I provisioned from my Synology NAS, and that is working well.
The stick I am using is a PNY 256GB USB3.0 that I purchased brand new from Amazon specifically for this. I believe this is the exact model (sorry, my wife ordered, so I don't have it in my recent purchases):
PNY 256GB Turbo Attaché 3 USB 3.0 Flash Drive - (P-FD256TBOP-GE)
https://www.amazon.com/PNY-Turbo-256GB-Flash-Drive/dp/B00JN1TOHM/ref=sr_1_3?crid=2MHQW5UV35SN2&dchil...
As you ssh in, you can just type 'dmesg'. you can redirect the output to a file on a datastore and download it using the datastore browser.
I'm personally using a PNY USB SSD drive: https://www.amazon.com/gp/product/B01GQPXBQC I got only a couple of issues with it "dropping", when I was running VMs using 100% of the CPUs...
Other ones I've tried and seems to work well: https://www.amazon.com/dp/B081QS63CZ and https://www.amazon.com/gp/product/B07D7SX9NS.
@vsanjake You happen to have a picture or the model number of that SanDisk mini 128GB thumb drive your using?
Thanks in advance
This is all I see in the dmesg log. Repeated 3000+ times:
2020-11-11T00:35:41.675Z cpu0:132045)J6: 2251: FS3_FSSyncIO failed on vol OSDATA-5f4e5f6c-820e9f4c-982b-dca632c425e1 with No connection
All,
Just bought (4) SanDisk Cruzer Glide 128GB USB 2.0 Flash Drives Model#SDCZ60-128G from Walmart and everything is working great now..
The only downside is i'm plugging into a USB 3.0 port on my Raspberry Pi 4 and the Flash Drives are USB 2.0 =( .... Hell.... it works until I get a NAS 😃
I rebooted my Pi today, and the USB datastore came back. But, once again, as soon as I tried to copy an ISO to it, it disappeared. The good news is that I got some info in the dmesg log this time (see attached). I believe I started the copy at about 13:18. I don't know what any of it means, but there it is.
From what I can see from the logs you provided, it looks like ESXi timeout waiting on making a connection to mount your flash drive / datastore or it got hung
- 'datastore1': HB at offset 3407872 - Waiting for timed out HB:2020-11-11T13:18:03.549Z cpu0:131238)
- Waiting for timed out HB:2020-11-11T13:18:07.754Z cpu0:131238) [HB state abcdef02 offset 3407872 gen
- Long VMFS rsv time on 'datastore1' (held for 5708 msecs). # R: 1, # W: 1 bytesXfer: 16 sectors
Take a look - https://kb.vmware.com/s/article/1002598
I don't think it really likes your flash drive your using for a datastore.... Might want to change that flash drive to one of the one's @cyprienlaplace mention earlier in this forum.
Good Luck!
I know that it has been more than a year, but I a am having the exact same issue. Please let me know if changing the flash drive was the solution for you.
Yes, changing out the flash drive DID fix my issues. Sorry, I can't believe I didn't come back here and update. I got pulled into other stuff and haven't been able to play with it since, but I did get it working.
I got the 128GB version of this drive that @cyprienlaplace mentioned earlier in the thread:
Amazon.com: SAMSUNG MUF-32AB/AM FIT Plus 32GB - 200MB/s USB 3.1 Flash Drive : Everything Else
This works like a charm with a SSD.
SATA3 USB adapter,
Thanks for taking the time to answer. I will give it a try with that usb flash drive.