VMware Cloud Community
mr_root_
Contributor
Contributor
‎06-23-2019 08:16 AM

ESXi Update: [OSError] [Errno 28] No space left on device

Hi,

I am trying to do an update of my ESXi 6.7.0 U2 host with the command

esxcli software profile update -p ESXi-6.7.0-20190604001-standard \

-d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

but I'm getting the error message

[OSError]

[Errno 28] No space left on device

Please refer to the log file for more details.

I have already rebooted the ESXi host in order to free some memory the ramdisks could be using.

My vdf -h looks like this:

...

-----

Ramdisk                   Size      Used Available Use% Mounted on

root                       32M        2M       29M   7% --

etc                        28M      196K       27M   0% --

opt                        32M        0B       32M   0% --

var                        48M      352K       47M   0% --

tmp                       256M        8K      255M   0% --

iofilters                  32M        0B       32M   0% --

shm                      1024M        0B     1024M   0% --

hostdstats                204M        2M      201M   1% --

snmptraps                   1M        0B        1M   0% --

My df -h looks like this:

Filesystem   Size   Used Available Use% Mounted on

VMFS-5     223.5G 196.0G     27.5G  88% /vmfs/volumes/0_OCZ_DARC_240GB

VMFS-5     931.2G 877.9G     53.3G  94% /vmfs/volumes/0_WD_1TB_RED

VMFS-5     465.5G 461.0G      4.5G  99% /vmfs/volumes/0_SAMS_500GB

VMFS-5     465.5G 461.0G      4.5G  99% /vmfs/volumes/1_SAMS_500GB

VMFS-6     465.5G 391.7G     73.8G  84% /vmfs/volumes/SAMS_850EVO_500G_0

VMFS-6     465.5G 172.4G    293.1G  37% /vmfs/volumes/SAMS_850EVO_500G_1

VMFS-6     458.2G 175.1G    283.2G  38% /vmfs/volumes/SAMS_850EVO_500G_2_esxiboot

vfat       249.7M 147.4M    102.3M  59% /vmfs/volumes/ddc4e178-95bc9fd2-2875-f12db19fbedd

vfat       249.7M 146.8M    102.9M  59% /vmfs/volumes/c8bcc07e-bfadfa2f-dbdd-6da6bb17b2c7

vfat         4.0G  21.2M      4.0G   1% /vmfs/volumes/5c433dad-57e1863f-b50b-ac1f6b7a4ea2

vfat       285.8M 172.9M    112.9M  60% /vmfs/volumes/5c433da5-ece0b664-e9b8-ac1f6b7a4ea2

I have already enable swapping to disk with mountpoint /vmfs/volumes/SAMS_850EVO_500G_1.

The /var/log/esxupdate.log says:

2019-06-23T15:08:27Z esxupdate: 2099595: imageprofile: INFO: Adding VIB VMware_locker_tools-light_10.3.5.10430147-12986307 to ImageProfile (Updated) ESXi-6.7.0-20190404001-standard

2019-06-23T15:08:27Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-U', 'host-acceptance-level', '-G']', outfile = 'None', returnoutput = 'True', timeout = '30.0'.

2019-06-23T15:08:28Z esxupdate: 2099595: Transaction: INFO: Downloading esx-update vib from https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_...

2019-06-23T15:08:28Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk add -m 0 -M 20 -n esx-update-2099595 -p 755 -t /tmp/esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2019-06-23T15:08:28Z esxupdate: 2099595: downloader: INFO: Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_... to /tmp/esx-update-2099595/VMware_bootbank_esx-update_6.7.0-2.60.13981272

2019-06-23T15:08:31Z esxupdate: 2099595: root: DEBUG: Verifying VIB VMware_bootbank_esx-update_6.7.0-2.60.13981272 signature #2

2019-06-23T15:08:31Z esxupdate: 2099595: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2099595/esxupdt-2099595 in ramdisk /tmp/esx-update-2099595

2019-06-23T15:08:31Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = '/usr/lib/vmware/secureboot/bin/secureMount.py ramdiskMount /tmp/esx-update-2099595/VMware_bootbank_esx-update_6.7.0-2.60.13981272 esxupdt /tmp/esx-update-2099595/esxupdt-2099595 esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2019-06-23T15:08:31Z esxupdate: 2099595: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2099595/weaselin-2099595 in ramdisk /tmp/esx-update-2099595

2019-06-23T15:08:31Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = '/usr/lib/vmware/secureboot/bin/secureMount.py ramdiskMount /tmp/esx-update-2099595/VMware_bootbank_esx-update_6.7.0-2.60.13981272 weaselin /tmp/esx-update-2099595/weaselin-2099595 esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2019-06-23T15:08:32Z esxupdate: 2099595: Transaction: DEBUG: Added /tmp/esx-update-2099595/usr/lib/vmware and /tmp/esx-update-2099595/lib64/python3.5/site-packages/vmware to sys.path

2019-06-23T15:08:32Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list | grep /esx-update-2099595 && localcli system visorfs ramdisk remove -t /tmp/esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2019-06-23T15:08:32Z esxupdate: 2099595: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/esxupdt-2099595

2019-06-23T15:08:32Z esxupdate: 2099595: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/weaselin-2099595

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR: Traceback (most recent call last):

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 470, in <module>

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:     main()

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 461, in main

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:     ret = CMDTABLE[command](options)

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 213, in ProfileUpdateCmd

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:     nohwwarning=opts.nohwwarning)

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13644319/bora/build/esx/release/vmvisor/esxupdate/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 376, in UpdateProfileFromDepot

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/tmp/esx-update-2099595/usr/lib/vmware/weasel/util/upgrade_precheck.py", line 2161, in cliUpgradeAction

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/tmp/esx-update-2099595/usr/lib/vmware/weasel/util/upgrade_precheck.py", line 997, in _parseVmwareVersion

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 514, in getoutput

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 495, in getstatusoutput

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 316, in check_output

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 383, in run

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 676, in __init__

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 1228, in _execute_child

2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR: OSError: [Errno 28] No space left on device

I don't see which mountpoint is running out of space and how to mitigate it.

Any ideas?

Thanks in advance.

Best regards,

mr_root_

Tags (2)
0 Kudos
9 Replies
mr_root_
Contributor
Contributor
‎07-02-2019 08:32 AM

Noone an idea?

0 Kudos
Vikas_VM
Contributor
Contributor
‎07-03-2019 08:35 AM

Hello Mr_Root,

Hope you are doing well.

I hope below links can help you to resolve the issue.

https://www.starwindsoftware.com/blog/what-should-you-do-if-no-space-left-on-device-error-occurs-whi...

VMware Knowledge Base

0 Kudos
mr_root_
Contributor
Contributor
‎07-03-2019 09:43 AM

Hi @Vikas_VM,

I have checked the inode count as per your article, but I have enough inodes free for the update. I do not run out of inodes during running the update command.

I wont do find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -l '{}' \; since this would delete my .vmdk files.

I still get the above error message.

What I found out is that the host cache is enabled on a datastore with enough free space, but not active.

Command: esxcli sched swap system get

Output:

   Datastore Active: true

   Datastore Enabled: true

   Datastore Name: SAMS_850EVO_500G_1

   Datastore Order: 1

   Hostcache Active: false

   Hostcache Enabled: true

   Hostcache Order: 0

   Hostlocalswap Active: false

   Hostlocalswap Enabled: true

   Hostlocalswap Order: 2

Any ideas how I could make ESXi to activate the host cache?

Thanks in advance.

Best regards,

mr_root

0 Kudos
Vikas_VM
Contributor
Contributor
‎07-03-2019 10:48 AM

Hello Mr Root,

I've found two VMware article for this, these are below.

Configure Host Cache with VMFS Datastore

Configuring Host Swap Cache

I've never faced this issue so taking help from VMware articles.

Hope this can help you as well.

Thanks,

Vikas

0 Kudos
mr_root_
Contributor
Contributor
‎07-03-2019 11:16 AM

Hi Vikas_VM,

thanks for the VMWare articles.

Though your article Configure Host Cache with VMFS Datastore refers to VMWare Vsphere but I'm running standalone ESXi. I don't have the Configure->Storage-> Host Cache Configuration setting in the ESXi GUI. And your article only refers to flash host cache storage, but I want to bring out host cache to a local SSD storage.

And your second article Configuring Host Swap Cache refers to flash memory as well. Maybe this is the problem, why my host cache is not active (though enabled), since I am not caching to flash memory?

Thanks in advance.

Best regards,

mr_root

0 Kudos
Vikas_VM
Contributor
Contributor
‎07-03-2019 11:37 AM

Hello Mr Root,

Hope this can help.

https://www.experts-exchange.com/articles/10844/HOW-TO-Configure-Host-Cache-Configuration-for-your-V...

Thanks,

Vikas

0 Kudos
mr_root_
Contributor
Contributor
‎07-03-2019 11:50 AM

Hi Vikas_VM,

this article refers to ESXi up to version 6.5, since it describes how to enable Host cache with the VSphere client. My machine is ESXi 6.7 U2, and the VSphere client is not supported there anymore.

I only have the web interface and the esxcli to configure it. As already said, my host cache is enabled in Host->Manage->Swap and there is also a sysSwap-ds-5c433ca2-1513-dd2b-55a2-ac1f6b7a4ea2.swp file on the selected datastore, but is has a modification time in the past (23/06/2019), shouldn't it then have a modification time of today when swapping would work?

Best regards,

mr_root

0 Kudos
mr_root_
Contributor
Contributor
‎07-05-2019 10:18 AM

Noone an idea?

I have already tried the suggestions from other threads where the esx-update caused No space left on device, but it doesn't mitigate my error.

Best regards,

mr_root

0 Kudos
mr_root_
Contributor
Contributor
‎07-17-2019 02:23 PM

I upgraded from 16GB to 32GB memory and now the update works. Weird that I need 32GB memory for the update Smiley Happy

0 Kudos