VMware Cloud Community
Mustibh
Contributor
Contributor
Jump to solution

Vmware Esxi Update No Space Left On Device Error

Hello,

I'm trying to install updates from a zip file but always gives the same error. "No Space Left On Device"

But there is a plenty of space on Device.

When i try to run

esxcli software vib update -d /vmfs/volumes/datastore1/ESXi600-201703001.zip

it gives that error:

[InstallationError]

[Errno 28] No space left on device

       vibs = VMware_bootbank_misc-drivers_6.0.0-3.57.5050593

Please refer to the log file for more details.

And in the log file:

2017-04-02T00:13:50Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list |            grep /stagebootbank &&            localcli system visorfs ramdisk remove -t /tmp/stagebootbank', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2017-04-02T00:13:50Z esxupdate: BootBankInstaller.pyc: WARNING: Cannot remove staging directory: [Errno 16] Device or resource busy: '/tmp/stagebootbank'

2017-04-02T00:13:50Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list |            grep /stagebootbank &&            localcli system visorfs ramdisk remove -t /tmp/stagebootbank', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2017-04-02T00:13:51Z esxupdate: root: ERROR: Traceback (most recent call last):

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 462, in <module>

2017-04-02T00:13:51Z esxupdate: root: ERROR:     main()

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 453, in main

2017-04-02T00:13:51Z esxupdate: root: ERROR:     ret = CMDTABLE[command](options)

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 362, in VibInstallCmd

2017-04-02T00:13:51Z esxupdate: root: ERROR:     checkacceptance=checkacceptance)

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 250, in InstallVibsFromSources

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 356, in _installVibs

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 399, in _validateAndInstallProfile

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 710, in Stage

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 483, in _download_and_stage

2017-04-02T00:13:51Z esxupdate: root: ERROR: InstallationError: ('VMware_bootbank_vsanhealth_6.0.0-3000000.3.0.3.58.5224738', '[Errno 28] No space left on device')

"vdf -h" Output

Ramdisk                   Size      Used Available Use% Mounted on

root                       32M      244K       31M   0% --

etc                        28M      204K       27M   0% --

opt                        32M        0B       32M   0% --

var                        48M      428K       47M   0% --

tmp                       256M       20K      255M   0% --

iofilters                  32M        0B       32M   0% --

hostdstats                303M        2M      300M   0% --

As you can see %0 is used on /tmp i dont know why gives that error.

How can i fix this problem ? I cant update my esxi 3620759 to 5224934.

Tags (1)
47 Replies
mveras1972
Enthusiast
Enthusiast
Jump to solution

METOO​

I copied all the direct links to the VIBs and also the locker I did it manually as instructed.

Thanks!

Reply
0 Kudos
Dfitz
Contributor
Contributor
Jump to solution

How did you know the url to download the individual VIB?

Reply
0 Kudos
Mazeno00
Contributor
Contributor
Jump to solution

Just enabling swap on a datastore fixed the issue for me as well. BTW the swap setting is in Manage, System (not Host, System)

Reply
0 Kudos
heronmb2013
Contributor
Contributor
Jump to solution

Thank you so much, it works for me too.

Reply
0 Kudos
tasikas
Enthusiast
Enthusiast
Jump to solution

I have same problem and different solution, Pleas see my reply in this post

Re: [Errno 28] No space left on device

Reply
0 Kudos
secconse
Contributor
Contributor
Jump to solution

I got this problem when upgrading from 6.7u1 to 6.7u2 and solved it with the manual vib installation as well. Thanks so much for that tip - saved me lots!

[user@host:~] cd /tmp

[user@host:/tmp] wget http://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_to...

[user@host:/tmp] esxcli software vib install -f -v /tmp/VMware_locker_tools-light_10.3.5.10430147-12986307.vib

Installation Result

   Message: Operation finished successfully.

   Reboot Required: false

   VIBs Installed: VMware_locker_tools-light_10.3.5.10430147-12986307

   VIBs Removed: VMware_locker_tools-light_10.3.2.9925305-10176879

   VIBs Skipped:

pbraren
Hot Shot
Hot Shot
Jump to solution

I also attempted an ESXCLI upgrade, failing, with swap file set work-arounds not getting resolving the error. What's interesting about your approach is that it seems to indicate it's not always a storage/swap issue. I'm wondering if anybody has found the root cause, and/or a more elegant (simpler) work-around yet?

TinkerTry.com
Reply
0 Kudos
harrypnyce
Contributor
Contributor
Jump to solution

I have been experiencing a near identical issue on my HP Z420 (Xeon E5-1620, 64GB RAM), trying to apply 6.7 U2 using the one-liner while currently on Version: 6.7.0 Update 1 (Build 11675023) when attempting to issue the one-liner update, the --dry-run appears to finish successfully, without issue, however I'm still unable to apply this update, even after attempting to follow along with the various suggestions.  I've added one of my local datastores as /swap, however this has still failed to resolve the issue and I'm not really sure what I'm doing wrong at this point.  Original image profile:

(Updated) ESXi-6.7.0-20190104001-standard (Hewlett Packard Enterprise)

> vdf -h

-----

Ramdisk                   Size      Used Available Use% Mounted on

root                       32M        2M       29M   7% --

etc                        28M      220K       27M   0% --

opt                        32M       40K       31M   0% --

var                        48M      504K       47M   1% --

tmp                       256M       32K      255M   0% --

iofilters                  32M        0B       32M   0% --

shm                      1024M        0B     1024M   0% --

hostdstats                303M        3M      299M   1% --

And the exact issue(s) that seem to continually pop up, no matter what I attempt to resolve this:

[root@esxi2:~] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.7.0-20190402001-standard

[InstallationError]

[Errno 28] No space left on device

       vibs = VMware_locker_tools-light_10.3.5.10430147-12986307

Please refer to the log file for more details.

I appear to have plenty of space available:

[root@esxi2:~] stat -f /

  File: "/"

    ID: 100000000 Namelen: 127     Type: visorfs

Block size: 4096

Blocks: Total: 641306     Free: 447708     Available: 447708

Inodes: Total: 655360     Free: 648545

Didn't have any issues updating my primary ESXi host (Cisco UCS-C240-M3S), which is using the custom Cisco image profile: (Updated) ESXi-6.7.0-20190402001-standard (Cisco)

Version: 6.7.0 Update 2 (Build 13006603)

Guess I'll continue to go bash my head against the wall some more -- at this point I'd even consider re-rolling the bootable USB media with a fresh install of a 6.7 U2 ISO, but there doesn't appear to be the custom HPE variant I'm looking for.  Perhaps this HPE Gen9 Plus Custom Image for ESXi 6.7 U2 Install CD 2019-04-18 would work, but for some reason I thought i used a different ISO when I refreshed things back in February with v6.7 -- thanks to all who have contributed to this thread, I've obviously still got a lot to learn.

EDIT:  Thanks, pbraren -- i've found a LOT of helpful information on your page(s) and it's been the primary place I go for up to the minute reference material(s).  Cannot thank you enough for the contributions you continue to make to this wonderful community, sir.

Reply
0 Kudos
pbraren
Hot Shot
Hot Shot
Jump to solution

A simple reboot got the swap file fix working just fine, with the 2nd ESXCLI attempt working just fine, session transcript here. Another similar host didn't need the swap fix, not sure what the difference between the two was. Glad to now have my VCSA at 6.7U2 and my ESXi hosts at 6.7U2 now, looking good so far.

TinkerTry.com
Reply
0 Kudos
pbraren
Hot Shot
Hot Shot
Jump to solution

Wow, harrypnyce​, I really appreciate the kind words, thank you!

TinkerTry.com
harrypnyce
Contributor
Contributor
Jump to solution

Finally resolved this after taking a (somewhat lengthy) break and managed to find some time this afternoon to poke around at it again, and appears all is now resolved.  Was running ESXi host, version: 6.7.0 Update 1 (Build 11675023) -- update to v6.7 U2 fails with the following error message:

[OSError]

[Errno 28] No space left on device

Please refer to the log file for more details.

Unfortunately adjustments to the swap file / location didn't seem to resolve my issue(s), even after placing the host in Maintenance Mode and even rebooting between adjustments.  I could see the swap file moving from one datastore to the next as I tried both SSDs and a larger 1TB spinning rust with gobs of free space, but ultimately the fix was to manually apply the specific VIB in question, at which point I was able to rerun the original update command and things seem to have been applied successfully.  Waiting on the host to come back up now.

> [root@esxi2:/tmp] wget http://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_to...

Connecting to hostupdate.vmware.com (184.29.84.35:80)

VMware_locker_tools- 100% |*****************************************************************************************************************************************************|   153M  0:00:00 ETA

> [root@esxi2:/tmp] esxcli software vib install -f -v /tmp/VMware_locker_tools-light_10.3.5.10430147-12986307.vib

Installation Result

   Message: Operation finished successfully.

   Reboot Required: false

   VIBs Installed: VMware_locker_tools-light_10.3.5.10430147-12986307

   VIBs Removed: VMware_locker_tools-light_10.3.2.9925305-10302608

   VIBs Skipped:

And here is where I re-ran the original update command, which applied successfully:
> [root@esxi2:/tmp] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p  ESXi-6.7.0-20190504001-standard

Update Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

After failing with moving swap file(s) around to various datastores and checking inode values & free space, eventually my error message morphed to display the following, at which point (see above) i manually installed the mentioned VIB:

[InstallationError]

[Errno 28] No space left on device

       vibs = VMware_locker_tools-light_10.3.5.10430147-12986307

Please refer to the log file for more details.

Voila, now on: v6.7.0 Update 2 (Build 13644319)

As always, your site is an immensely valuable resource, Mr. Braren, sir!~  My apologies for the lack of chronology contained within this post.  Hope someone else is spared the research efforts once they've stumbled upon thi

(cc: pbraren​)

Neutro
Enthusiast
Enthusiast
Jump to solution

Thank you i had the same problem and your solution worked perfectly

Reply
0 Kudos
psed
Contributor
Contributor
Jump to solution

Hi, the same problem

update vmware 6.7up1 -> up2

[root@olvmware01:/tmp] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p  ESXi-6.7.0-20190504001-standard

[OSError]

[Errno 28] No space left on device

Please refer to the log file for more details.

[root@olvmware01:/tmp] df -h

Filesystem   Size   Used Available Use% Mounted on

VMFS-6     924.0G  13.3G    910.7G   1% /vmfs/volumes/datastore01_SATA_1TB_Seagate

VMFS-6     931.2G 701.4G    229.8G  75% /vmfs/volumes/datastore02_SATA_1TB_Seagate

vfat       285.8M 172.9M    112.9M  60% /vmfs/volumes/5c6bd88a-81d56d1c-4eed-4c5262099845

vfat       249.7M 146.7M    103.0M  59% /vmfs/volumes/970d7e7f-244c6e5c-de39-3887b52dde70

vfat       249.7M 146.7M    103.0M  59% /vmfs/volumes/21f8f578-b9d58c62-3141-5767f4693b98

vfat         4.0G  17.2M      4.0G   0% /vmfs/volumes/5c6bd892-9a69c4fe-0692-4c5262099845

Navigator > Host > Manage > System > Swap > Edit Settings >

Enable - Yes

Datastore - "datastore01_SATA_1TB_Seagate"

Local swap - Yes

Host Cache -Yes

+reboot

[root@olvmware01:/tmp] tail -30 /var/log/esxupdate.log

2019-07-03T13:06:55Z esxupdate: 2099827: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-U', 'host-acceptance-level', '-G']', outfile = 'None', returnoutput = 'True', timeout = '30.0'.

2019-07-03T13:06:55Z esxupdate: 2099827: Transaction: INFO: Downloading esx-update vib from https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_...

2019-07-03T13:06:55Z esxupdate: 2099827: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk add -m 0 -M 20 -n esx-update-2099827 -p 755 -t /tmp/esx-update-2099827', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2019-07-03T13:06:55Z esxupdate: 2099827: downloader: INFO: Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_... to /tmp/esx-update-2099827/VMware_bootbank_esx-update_6.7.0-2.55.13644319

2019-07-03T13:06:56Z esxupdate: 2099827: root: DEBUG: Verifying VIB VMware_bootbank_esx-update_6.7.0-2.55.13644319 signature #2

2019-07-03T13:06:56Z esxupdate: 2099827: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2099827/esxupdt-2099827 to /tardisks.noauto/

2019-07-03T13:06:56Z esxupdate: 2099827: Ramdisk: INFO: Attach tardisk esxupdt-2099827 to ramdisk esx-update-2099827

2019-07-03T13:06:56Z esxupdate: 2099827: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2099827/weaselin-2099827 to /tardisks.noauto/

2019-07-03T13:06:56Z esxupdate: 2099827: Ramdisk: INFO: Attach tardisk weaselin-2099827 to ramdisk esx-update-2099827

2019-07-03T13:06:56Z esxupdate: 2099827: Transaction: DEBUG: Added /tmp/esx-update-2099827/usr/lib/vmware and /tmp/esx-update-2099827/lib64/python3.5/site-packages/vmware to sys.path

2019-07-03T13:06:56Z esxupdate: 2099827: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list | grep /esx-update-2099827 && localcli system visorfs ramdisk remove -t /tmp/esx-update-2099827', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2019-07-03T13:06:56Z esxupdate: 2099827: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/esxupdt-2099827

2019-07-03T13:06:56Z esxupdate: 2099827: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/weaselin-2099827

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR: Traceback (most recent call last):

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 470, in <module>

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:     main()

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 461, in main

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:     ret = CMDTABLE[command](options)

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 213, in ProfileUpdateCmd

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:     nohwwarning=opts.nohwwarning)

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/build/mts/release/bora-10764712/bora/build/esx/release/vmvisor/esxupdate/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 375, in UpdateProfileFromDepot

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/tmp/esx-update-2099827/usr/lib/vmware/weasel/util/upgrade_precheck.py", line 2161, in cliUpgradeAction

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/tmp/esx-update-2099827/usr/lib/vmware/weasel/util/upgrade_precheck.py", line 997, in _parseVmwareVersion

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/build/mts/release/bora-10302608/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 514, in getoutput

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/build/mts/release/bora-10302608/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 495, in getstatusoutput

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/build/mts/release/bora-10302608/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 316, in check_output

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/build/mts/release/bora-10302608/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 383, in run

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/build/mts/release/bora-10302608/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 676, in __init__

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR:   File "/build/mts/release/bora-10302608/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 1228, in _execute_child

2019-07-03T13:06:56Z esxupdate: 2099827: root: ERROR: OSError: [Errno 28] No space left on device

[root@olvmware01:/tmp] stat -f /

  File: "/"

    ID: 100000000 Namelen: 127     Type: visorfs

Block size: 4096

Blocks: Total: 580901     Free: 383584     Available: 383584

Inodes: Total: 655360     Free: 648814

Install VMware_locker_tools-ligh ok

[root@olvmware01:/tmp] esxcli software vib install -f -v /tmp/VMware_locker_tools-light_10.3.5.10430147-12986307.vib

Installation Result

   Message: Operation finished successfully.

   Reboot Required: false

   VIBs Installed: VMware_locker_tools-light_10.3.5.10430147-12986307

   VIBs Removed: VMware_locker_tools-light_10.3.2.9925305-10176879

   VIBs Skipped:

How can i fix this problem ?

Thank you
Pavel

ivanerben
Enthusiast
Enthusiast
Jump to solution

Same here, unable to update. And hitting bug with OCFlush cpu load

High CPU usage by system process after 49 days of uptime (OCFlush)

high OCFlush %USED with no VMs running. Esxtop %USED sort not working once in world 

which is maybe solved...

[root@esxi:~] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.7.0-20190604001-standard

[OSError]

[Errno 28] No space left on device

Please refer to the log file for more details.

VMware ESXi 6.7.0 build-10302608

VMware ESXi 6.7.0 Update 1

Reply
0 Kudos
GeraldVM
Contributor
Contributor
Jump to solution

Very frustrating. I just spent more than an hour trying all kinds of ways to update a fresh install of

"VMware-VMvisor-Installer-6.7.0.update02-13006603.x86_64.iso"

to the latest version using

"esxcli software profile update -p ESXi-6.7.0-20190604001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml"

and it always failed with: [Errno 28] No space left on device

I tried everything, except manually installing the VIB. Nothing which usually helps (setting swap space, changing the locker directory, removing files from the locker directory, installing the -no-tools instead of standard profile) helped. I might have been able to fix this by installing the vib manually but...

I just wiped the machine and installed VMware ESXi 6.5u2 (VMware-VMvisor-Installer-6.5.0.update02-8294253.x86_64.iso), and the update to ESXi-6.7.0-20190604001-standard just worked without any hassle. (I did set the swap space before trying, but did not even reboot after doing so. It just worked)

Something is seriously wrong with the 6.7u2 installation media. Guess quality control was too expensive and so is fixing the bug.

Neutro
Enthusiast
Enthusiast
Jump to solution

And you're bitching about a free software.

If you were using the paid version of ESXi you would have updated through vcenter without having to update through SSH.

Not saying that you should pay for it but since most VMware users are entreprises using Vcenter only a few people actually have to update through SSH like you and me. So it's understandable that since only a few people use it, when bugs are present they are detected and fixed less faster.

In any case, crying like you do wont change anything... Maybe it could make VMware stop giving free licences because too much people like you complain after.

Reply
0 Kudos
GeraldVM
Contributor
Contributor
Jump to solution

>And you're bitching about a free software.

This will come as a shock to you, but VMware is not giving away the free version out of the goodness in their hearts. I'm sorry to burst your bubble.

This is a business decision, where giving away a somewhat limited edition draws people into their ecosystem and then upsells them into a paid version once they eventually require a feature beyond what is offered in the free version.

What does make sense in this context (contrary to what you are thinking) is making the free version bad and buggy.

Users of VMware which are not yet heavily invested will not say

"Oh, the update is not working AGAIN, I guess everything will be fine in the paid version."

instead they will say

"Oh, the update is not working AGAIN, I guess I need to have a closer look at Proxmox."

tymchenko
Contributor
Contributor
Jump to solution

Thanks it helps me update 6.7 U2 -> 6.7 U3

Reply
0 Kudos
stephenyu05
Enthusiast
Enthusiast
Jump to solution

i cannot upgrade from 6.7u2 to u3/

no space left

Reply
0 Kudos