The dreaded: [Errno 28] No space left on device

[Dashers]

I, like many before, are at that lovely point of trying to apply a profile update to a standalone ESXi 6.7 host:

[root@:~] esxcli software profile install -p ESXi-6.7.0-20190504001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
 [OSError]
 [Errno 28] No space left on device
 Please refer to the log file for more details.

Joy.  Suggested solutions include downloading the latest archive and applying that - the problem with is it's not a profile update, and won't patch everything previous, just that individual update.  And of course ensuring Swap is enabled in the UI and backed by a datastore with an unreasonable amount of spare space.

I'd really like to solve this problem once and for-all, is it a bug?

The only log file that appears to have been written to is /var/log/esxcli, which just contains the same error.  Taking the error at face value, here is my df output:

Filesystem   Size   Used Available Use% Mounted on
VMFS-5     447.0G 156.0G    291.0G  35% /vmfs/volumes/datastore2
VMFS-5     104.2G   3.9G    100.3G   4% /vmfs/volumes/datastore1
vfat       249.7M 146.9M    102.8M  59% /vmfs/volumes/e8401f77-0af539dc-3dd0-fd40b7891bcd
vfat       249.7M 141.7M    108.0M  57% /vmfs/volumes/ddbd4778-d8ee947a-2427-f7e19605a5cd
vfat         4.0G  25.4M      4.0G   1% /vmfs/volumes/5ae9a055-f7f64e90-4be7-6cb31131dc38
vfat       285.8M 174.2M    111.6M  61% /vmfs/volumes/52d45f44-fea28213-783c-000c295b0bf8

Standard partition layout for a ESXi installation, with only the out-of-the-box tiny partitions looking remotely tight on space (but what can you do about that?).

I tried running the command whilst monitoring disk usage, to see if one was steadily being filled before being rolled back, but nope, no change.

I am at a bit of a loss here to what's causing the problem which seems to impact so many people at random, and how to troubleshoot / permanently resolve this without hacks and workarounds.  Are there any gurus frequenting this forum who could shed some light on this baffling issue?

[vijayrana968]

Sometimes, error is caused by lack of inodes file objects. Please refer to VMware Knowledge Base

[Dashers]

Wasn't aware of that one, doesn't seem to be the issue here though?

stat -f /
  File: "/"
    ID: 100000000 Namelen: 127     Type: visorfs
Block size: 4096
Blocks: Total: 621869     Free: 463352     Available: 463352
Inodes: Total: 655360     Free: 648777

[root@:~] localcli system visorfs ramdisk list
Ramdisk Name   System  Include in Coredumps  Reserved   Maximum      Used      Peak Used  Free   Reserved Free  Maximum Inodes  Allocated Inodes  Used Inodes  Mount Point
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
root           true    true                  32768 KiB    32768 KiB  2292 KiB   2292 KiB   93 %           93 %           16704              5248         5230  /
etc            true    true                  28672 KiB    28672 KiB   264 KiB    304 KiB   99 %           99 %            4096              1024          651  /etc
opt            true    true                      0 KiB    32768 KiB     0 KiB      0 KiB  100 %            0 %            8192              1024            7  /opt
var            true    true                   5120 KiB    49152 KiB   424 KiB    428 KiB   99 %           91 %            8192               704          674  /var
tmp            false   false                  2048 KiB   262144 KiB    20 KiB   1380 KiB   99 %           99 %            8192               256            6  /tmp
iofilters      false   false                     0 KiB    32768 KiB     0 KiB      0 KiB  100 %            0 %           10240                32            1  /var/run/iofilters
shm            false   false                     0 KiB  1048576 KiB     0 KiB      0 KiB  100 %            0 %             512                32            1  /var/run/shm
hostdstats     false   false                     0 KiB   116736 KiB  3176 KiB   3176 KiB   97 %            0 %            8192                32            4  /var/lib/vmware/hostd/stats
stagebootbank  false   false                     0 KiB   256000 KiB     0 KiB      0 KiB  100 %            0 %            8192                32            1  /tmp/stagebootbank

[vijayrana968]

Also check usage with df -I, also try update after enabling system swap https://www.starwindsoftware.com/blog/what-should-you-do-if-no-space-left-on-device-error-occurs-whi...

[Dashers]

My df output is in the original post, and as noted there, I've already enabled a datastore backed swap on the host.  Which is why I'm here scratching my head.  I have this issue on two standalone hosts.

[a_p_]

... the problem with is it's not a profile update, and won't patch everything previous, just that individual update

The offline patch bundle (.zip archive) offers the same installation options as to online method, i.e. you can apply a complete profile, either using the "update", or the "install" method (depending on your needs).

André