Dashers
Enthusiast
Enthusiast

The dreaded: [Errno 28] No space left on device

I, like many before, are at that lovely point of trying to apply a profile update to a standalone ESXi 6.7 host:

[root@:~] esxcli software profile install -p ESXi-6.7.0-20190504001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

[OSError]

[Errno 28] No space left on device

Please refer to the log file for more details.

Joy.  Suggested solutions include downloading the latest archive and applying that - the problem with is it's not a profile update, and won't patch everything previous, just that individual update.  And of course ensuring Swap is enabled in the UI and backed by a datastore with an unreasonable amount of spare space.

I'd really like to solve this problem once and for-all, is it a bug?

The only log file that appears to have been written to is /var/log/esxcli, which just contains the same error.  Taking the error at face value, here is my df output:

Filesystem   Size   Used Available Use% Mounted on

VMFS-5     447.0G 156.0G    291.0G  35% /vmfs/volumes/datastore2

VMFS-5     104.2G   3.9G    100.3G   4% /vmfs/volumes/datastore1

vfat       249.7M 146.9M    102.8M  59% /vmfs/volumes/e8401f77-0af539dc-3dd0-fd40b7891bcd

vfat       249.7M 141.7M    108.0M  57% /vmfs/volumes/ddbd4778-d8ee947a-2427-f7e19605a5cd

vfat         4.0G  25.4M      4.0G   1% /vmfs/volumes/5ae9a055-f7f64e90-4be7-6cb31131dc38

vfat       285.8M 174.2M    111.6M  61% /vmfs/volumes/52d45f44-fea28213-783c-000c295b0bf8

Standard partition layout for a ESXi installation, with only the out-of-the-box tiny partitions looking remotely tight on space (but what can you do about that?).

I tried running the command whilst monitoring disk usage, to see if one was steadily being filled before being rolled back, but nope, no change.

I am at a bit of a loss here to what's causing the problem which seems to impact so many people at random, and how to troubleshoot / permanently resolve this without hacks and workarounds.  Are there any gurus frequenting this forum who could shed some light on this baffling issue?

0 Kudos
5 Replies
vijayrana968
Virtuoso
Virtuoso

Sometimes, error is caused by lack of inodes file objects. Please refer to VMware Knowledge Base

0 Kudos
Dashers
Enthusiast
Enthusiast

Wasn't aware of that one, doesn't seem to be the issue here though?

stat -f /

  File: "/"

    ID: 100000000 Namelen: 127     Type: visorfs

Block size: 4096

Blocks: Total: 621869     Free: 463352     Available: 463352

Inodes: Total: 655360     Free: 648777

[root@:~] localcli system visorfs ramdisk list

Ramdisk Name   System  Include in Coredumps  Reserved   Maximum      Used      Peak Used  Free   Reserved Free  Maximum Inodes  Allocated Inodes  Used Inodes  Mount Point

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

root           true    true                  32768 KiB    32768 KiB  2292 KiB   2292 KiB   93 %           93 %           16704              5248         5230  /

etc            true    true                  28672 KiB    28672 KiB   264 KiB    304 KiB   99 %           99 %            4096              1024          651  /etc

opt            true    true                      0 KiB    32768 KiB     0 KiB      0 KiB  100 %            0 %            8192              1024            7  /opt

var            true    true                   5120 KiB    49152 KiB   424 KiB    428 KiB   99 %           91 %            8192               704          674  /var

tmp            false   false                  2048 KiB   262144 KiB    20 KiB   1380 KiB   99 %           99 %            8192               256            6  /tmp

iofilters      false   false                     0 KiB    32768 KiB     0 KiB      0 KiB  100 %            0 %           10240                32            1  /var/run/iofilters

shm            false   false                     0 KiB  1048576 KiB     0 KiB      0 KiB  100 %            0 %             512                32            1  /var/run/shm

hostdstats     false   false                     0 KiB   116736 KiB  3176 KiB   3176 KiB   97 %            0 %            8192                32            4  /var/lib/vmware/hostd/stats

stagebootbank  false   false                     0 KiB   256000 KiB     0 KiB      0 KiB  100 %            0 %            8192                32            1  /tmp/stagebootbank

0 Kudos
vijayrana968
Virtuoso
Virtuoso

Also check usage with df -I, also try update after enabling system swap https://www.starwindsoftware.com/blog/what-should-you-do-if-no-space-left-on-device-error-occurs-whi...

0 Kudos
Dashers
Enthusiast
Enthusiast

My df output is in the original post, and as noted there, I've already enabled a datastore backed swap on the host.  Which is why I'm here scratching my head.  I have this issue on two standalone hosts.

0 Kudos
a_p_
Leadership
Leadership

... the problem with is it's not a profile update, and won't patch everything previous, just that individual update

The offline patch bundle (.zip archive) offers the same installation options as to online method, i.e. you can apply a complete profile, either using the "update", or the "install" method (depending on your needs).

André

0 Kudos