Out of space on ESXi host usb flash - bad flash?

[natea]

Hey folks -

I have a personal server running ESXi 4.1 on a USB flash thingamajig.  I noticed a few minutes ago the storage on the host

appears to be full, though df shows plenty of space.

/var # df -h
Filesystem                Size      Used Available Use% Mounted on
visorfs                   1.3G    274.3M      1.0G  21% /
vmfs3                     2.0T    823.1G      1.2T  40% /vmfs/volumes/4e03554b-95aa2e2c-71dd-0025903a2f3d
vmfs3                     1.6T      1.6T      7.5G 100% /vmfs/volumes/4e0de540-ff46716b-75ea-0025903a2f3c
vfat                    285.9M    135.6M    150.3M  47% /vmfs/volumes/3c3693e8-f77a642a-1910-5c6bdcb26d3a
vfat                    249.7M     87.8M    161.9M  35% /vmfs/volumes/ff060de6-cecc88e5-4d14-8726d7ed0132
vfat                    249.7M     87.8M    161.9M  35% /vmfs/volumes/65092bef-de8a06b5-22db-2bbbc32dc3d2
/var # dd if=/dev/zero of=smallfile2 bs=100k count=200
dd: smallfile2: No space left on device
/var # ls -lh smallfile2
-rw-r--r--    1 root     root        19.0M Oct 25 01:35 smallfile2
/var # df -h
Filesystem                Size      Used Available Use% Mounted on
visorfs                   1.3G    293.4M   1013.9M  22% /
vmfs3                     2.0T    823.1G      1.2T  40% /vmfs/volumes/4e03554b-95aa2e2c-71dd-0025903a2f3d
vmfs3                     1.6T      1.6T      7.5G 100% /vmfs/volumes/4e0de540-ff46716b-75ea-0025903a2f3c
vfat                    285.9M    135.6M    150.3M  47% /vmfs/volumes/3c3693e8-f77a642a-1910-5c6bdcb26d3a
vfat                    249.7M     87.8M    161.9M  35% /vmfs/volumes/ff060de6-cecc88e5-4d14-8726d7ed0132
vfat                    249.7M     87.8M    161.9M  35% /vmfs/volumes/65092bef-de8a06b5-22db-2bbbc32dc3d2

Normally on a Linux box I would look at lsof and see what files may be open that were deleted, lsof doesn't behave the same way on ESXi, and there isn't anything I can imagine that would be in this state.I came across this when I tried to copy the 3ware tw_cli tool(2.2MB) to /sbin - something I do every time my system boots since for some reason it is lost on reboot. This time it failed, though smaller files could be copied. I ended up wiping out some logs in order to get it to fit. My system last booted on 9/28, though I didn't get round to setting this 3ware monitor stuff up until now. I suppose I could just run the tool directly from vmfs, but that still leaves the question as to what's going on with the flash disk space..

I don't see errors in the logs but it seems like maybe my flash is going bad? For my professional ESX systems everything is always boot from SAN, I never use USB keys, though for this personal server given it uses a 3ware RAID card I didn't have a lot of choice at the time.

While I'm here does anyone have reccomendations as to high quality USB flash drives for use with ESXi ?

any ideas?

thanks

[continuum]

Your urgent problem is
vmfs3                     1.6T      1.6T      7.5G 100%
and that has nothing to do with the USB-stick

Watch out - if you have anything running on /vmfs/volumes/4e0de540-ff46716b-75ea-0025903a2f3c you may run into serious problems any moment

[natea]

Thanks!

Yes I see that and it is intentional, I have a file server that basically has that VMFS volume dedicated to it. It just has a single VMDK on it. In any case there is still 7GB of space left, and that file system is not what is used for /var or /sbin for that matter on the ESXi host. In my tests I'm trying to write a mere 20MB worth of data.

Also the space on that VMFS volume has been unchanged for more than one year, the VMFS file is thick provisioned.

nate

[Josh26]

It could be that you have run out of inodes.

I don't know if df behaves the same as under Linux, but try:

df -i

If you're at 100%, look for a lot of small files.

[natea]

I thought about that as well, though I don't believe that is the case, the df command doesn't support the -i option, but more importantly if I write data to a single file and get it to 20MB it works fine, if I try to go to 21MB it fails, so I think that is the same number of inodes(1?).

I will replace the flash disk in a few weeks and hope that helps, I have seen no ill effects from this otherwise

thanks for all the responses.

[Kylel]

I'd say either bad flash, or a shot in the dark - I wonder if it's something to do with the scratch partition?

3.  Where does the scratch partition get placed when booting from USB?

Because USB/SD devices are sensitive to high amounts of I/O the installer will not place the scratch partition on a USB/SD device.  Instead, the installer first scans for a local 4GB vfat partition, if it doesn’t find one it will then scan for a local VMFS volume on which to create a scratch directory.  If no local vfat partition or VMFS volume is found, as a last resort the installer will put the scratch partition in “/tmp/scratch” (i.e. put scratch on the local ramdisk).  If this happens it’s a good idea to manually reconfigure the scratch partition after the install.

Kyle.

[natea]

Hey Kyle -

thanks for the idea.

I have heard about the scratch partition once or twice but that didn't occur to me. Though I just checked the scratch partition

configuration in the vSphere client and it is apparently mapped to a VMFS volume not the USB device

/vmfs/volumes/4e03554b-95aa2e2c-71dd-0025903a2f3d/.locker

That particular volume has 1.2T free

So I guess it must be bad flash

I searched high and low trying to find reccomendations for what is the highest quality USB key I can use but came up empty (I recall one specific case where I believe HP had a bad batch of flash keys so even the tier 1 guys can screw up)

The best I could come up with is this from Newegg(which I have two of now - I noticed it's not available from Newegg right now):

http://www.newegg.com/Product/Product.aspx?Item=N82E16820220252

"Patriot Xporter XT Boost 8GB Flash Drive (USB 2.0 Portable) Model PEF8GUSB"

1500 reviews, 77% of them 5 eggs.

Though it seems most people are more concerned about performance and perhaps physical durability of the thing itself rather than reliability of the flash memory inside. All of the review sites I came across it seems the only thing the sites measured was performance. I find it quite surprising that there isn't some sort of formal reccomendation from VMware as to which is best. They certainly go out of their way to certify all other parts of their platform!

Of course I'd have no problem paying a massive premium for the flash if that means I can get some real high quality stuff.

One of my co-workers mentioned to me recently that some of Dell's new servers actually offer the ability to have mirrored SD cards or something, that would be neat.

I just with the 3ware drivers were integrated with vSphere so I could just install to local storage instead of use flash!

thanks again for the ideas

[Kylel]

I've only ever used the HP ones for production but have used some other brands (Kingston maybe?) for testing and never had issues.

If you were looking for redundancy you might be able to have two identical USB keys plugged in and just change the boot order via iLo / IMM in the event of a failure. Local storage works too though!

Kyle.