I, like many before, are at that lovely point of trying to apply a profile update to a standalone ESXi 6.7 host:
[root@:~] esxcli software profile install -p ESXi-6.7.0-20190504001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
[OSError]
[Errno 28] No space left on device
Please refer to the log file for more details.
Joy. Suggested solutions include downloading the latest archive and applying that - the problem with is it's not a profile update, and won't patch everything previous, just that individual update. And of course ensuring Swap is enabled in the UI and backed by a datastore with an unreasonable amount of spare space.
I'd really like to solve this problem once and for-all, is it a bug?
The only log file that appears to have been written to is /var/log/esxcli, which just contains the same error. Taking the error at face value, here is my df output:
Filesystem Size Used Available Use% Mounted on
VMFS-5 447.0G 156.0G 291.0G 35% /vmfs/volumes/datastore2
VMFS-5 104.2G 3.9G 100.3G 4% /vmfs/volumes/datastore1
vfat 249.7M 146.9M 102.8M 59% /vmfs/volumes/e8401f77-0af539dc-3dd0-fd40b7891bcd
vfat 249.7M 141.7M 108.0M 57% /vmfs/volumes/ddbd4778-d8ee947a-2427-f7e19605a5cd
vfat 4.0G 25.4M 4.0G 1% /vmfs/volumes/5ae9a055-f7f64e90-4be7-6cb31131dc38
vfat 285.8M 174.2M 111.6M 61% /vmfs/volumes/52d45f44-fea28213-783c-000c295b0bf8
Standard partition layout for a ESXi installation, with only the out-of-the-box tiny partitions looking remotely tight on space (but what can you do about that?).
I tried running the command whilst monitoring disk usage, to see if one was steadily being filled before being rolled back, but nope, no change.
I am at a bit of a loss here to what's causing the problem which seems to impact so many people at random, and how to troubleshoot / permanently resolve this without hacks and workarounds. Are there any gurus frequenting this forum who could shed some light on this baffling issue?
Sometimes, error is caused by lack of inodes file objects. Please refer to VMware Knowledge Base
Wasn't aware of that one, doesn't seem to be the issue here though?
stat -f /
File: "/"
ID: 100000000 Namelen: 127 Type: visorfs
Block size: 4096
Blocks: Total: 621869 Free: 463352 Available: 463352
Inodes: Total: 655360 Free: 648777
[root@:~] localcli system visorfs ramdisk list
Ramdisk Name System Include in Coredumps Reserved Maximum Used Peak Used Free Reserved Free Maximum Inodes Allocated Inodes Used Inodes Mount Point
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
root true true 32768 KiB 32768 KiB 2292 KiB 2292 KiB 93 % 93 % 16704 5248 5230 /
etc true true 28672 KiB 28672 KiB 264 KiB 304 KiB 99 % 99 % 4096 1024 651 /etc
opt true true 0 KiB 32768 KiB 0 KiB 0 KiB 100 % 0 % 8192 1024 7 /opt
var true true 5120 KiB 49152 KiB 424 KiB 428 KiB 99 % 91 % 8192 704 674 /var
tmp false false 2048 KiB 262144 KiB 20 KiB 1380 KiB 99 % 99 % 8192 256 6 /tmp
iofilters false false 0 KiB 32768 KiB 0 KiB 0 KiB 100 % 0 % 10240 32 1 /var/run/iofilters
shm false false 0 KiB 1048576 KiB 0 KiB 0 KiB 100 % 0 % 512 32 1 /var/run/shm
hostdstats false false 0 KiB 116736 KiB 3176 KiB 3176 KiB 97 % 0 % 8192 32 4 /var/lib/vmware/hostd/stats
stagebootbank false false 0 KiB 256000 KiB 0 KiB 0 KiB 100 % 0 % 8192 32 1 /tmp/stagebootbank
Also check usage with df -I, also try update after enabling system swap https://www.starwindsoftware.com/blog/what-should-you-do-if-no-space-left-on-device-error-occurs-whi...
My df output is in the original post, and as noted there, I've already enabled a datastore backed swap on the host. Which is why I'm here scratching my head. I have this issue on two standalone hosts.
... the problem with is it's not a profile update, and won't patch everything previous, just that individual update
The offline patch bundle (.zip archive) offers the same installation options as to online method, i.e. you can apply a complete profile, either using the "update", or the "install" method (depending on your needs).
André