Mustibh
Contributor
Contributor

Vmware Esxi Update No Space Left On Device Error

Jump to solution

Hello,

I'm trying to install updates from a zip file but always gives the same error. "No Space Left On Device"

But there is a plenty of space on Device.

When i try to run

esxcli software vib update -d /vmfs/volumes/datastore1/ESXi600-201703001.zip

it gives that error:

[InstallationError]

[Errno 28] No space left on device

       vibs = VMware_bootbank_misc-drivers_6.0.0-3.57.5050593

Please refer to the log file for more details.

And in the log file:

2017-04-02T00:13:50Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list |            grep /stagebootbank &&            localcli system visorfs ramdisk remove -t /tmp/stagebootbank', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2017-04-02T00:13:50Z esxupdate: BootBankInstaller.pyc: WARNING: Cannot remove staging directory: [Errno 16] Device or resource busy: '/tmp/stagebootbank'

2017-04-02T00:13:50Z esxupdate: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list |            grep /stagebootbank &&            localcli system visorfs ramdisk remove -t /tmp/stagebootbank', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

2017-04-02T00:13:51Z esxupdate: root: ERROR: Traceback (most recent call last):

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 462, in <module>

2017-04-02T00:13:51Z esxupdate: root: ERROR:     main()

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 453, in main

2017-04-02T00:13:51Z esxupdate: root: ERROR:     ret = CMDTABLE[command](options)

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 362, in VibInstallCmd

2017-04-02T00:13:51Z esxupdate: root: ERROR:     checkacceptance=checkacceptance)

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 250, in InstallVibsFromSources

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 356, in _installVibs

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 399, in _validateAndInstallProfile

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 710, in Stage

2017-04-02T00:13:51Z esxupdate: root: ERROR:   File "/build/mts/release/bora-3620759/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 483, in _download_and_stage

2017-04-02T00:13:51Z esxupdate: root: ERROR: InstallationError: ('VMware_bootbank_vsanhealth_6.0.0-3000000.3.0.3.58.5224738', '[Errno 28] No space left on device')

"vdf -h" Output

Ramdisk                   Size      Used Available Use% Mounted on

root                       32M      244K       31M   0% --

etc                        28M      204K       27M   0% --

opt                        32M        0B       32M   0% --

var                        48M      428K       47M   0% --

tmp                       256M       20K      255M   0% --

iofilters                  32M        0B       32M   0% --

hostdstats                303M        2M      300M   0% --

As you can see %0 is used on /tmp i dont know why gives that error.

How can i fix this problem ? I cant update my esxi 3620759 to 5224934.

Tags (1)
1 Solution

Accepted Solutions
FreeAgent
Contributor
Contributor

@FirstS0ul

Thanks for your solution.  We tried this but encountered memory errors. The solution was to enter the host webui and Go to Host > System > Swap and activate swap on our datastore vmfs. Once the setting was enabled the esxcli software profile update command worked normally.

Thanks for posting very helpful.

View solution in original post

45 Replies
RajeevVCP4
Expert
Expert

Please try this KB

ESXi/ESX error: No free space left on device (1007638) | VMware KB

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you
0 Kudos
Mustibh
Contributor
Contributor

Thanks for your reply,

I tried what said in this KB but it did not helped. All it shows these files.

[root@myserver:/tmp] find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -lh '{}' \;

-r--r--r--    1 root     root      139.5M Apr  1 23:53 /tardisks/sb.v00

-r--r--r--    1 root     root      315.7M Apr  1 23:53 /tardisks/s.v00

-rw-------    1 root     root      439.0G Apr  2 08:38 /dev/lvm/5878b981-2bb79012-ec3d-0cc47ac67b50

-rw-------    1 root     root        1.8T Apr  2 08:38 /dev/lvm/587641e3-53d93890-7ae2-0cc47ac67b50

-rw-------    1 root     root        1.8T Apr  2 08:38 /dev/disks/naa.600605b000a029f02008fd5307f25822

-rw-------    1 root     root        1.8T Apr  2 08:38 /dev/disks/naa.600605b000a029f02008fd5307f25822:1

-rw-------    1 root     root      446.6G Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb

-rw-------    1 root     root        2.5G Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb:9

-rw-------    1 root     root      286.0M Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb:8

-rw-------    1 root     root      110.0M Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb:7

-rw-------    1 root     root      250.0M Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb:6

-rw-------    1 root     root      250.0M Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb:5

-rw-------    1 root     root      439.2G Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb:3

-rw-------    1 root     root        4.0G Apr  2 08:38 /dev/disks/naa.600605b000a029f02000e7f407fec1fb:2

Here is the stat -f output too;

[root@myserver:/tmp] stat -f /

  File: "/"

    ID: 100000000 Namelen: 127     Type: visorfs

Block size: 4096

Blocks: Total: 398288     Free: 249397     Available: 249397

Inodes: Total: 524288     Free: 518408

Please help how can i fix this problem ?

0 Kudos
a_p_
Leadership
Leadership

To rule out issues with the downloaded "ESXi600-201703001.zip", please ensure that the .zip file isn't corrupted by verifying it's checksum.

André

0 Kudos
Mustibh
Contributor
Contributor

Thanks for your reply,

I already checked the file but it does not seems with the file issue.

Sometimes on log shows "cannot allocate memory" and i event cant write ls -la command always says '-sh cant fork'

I didnt understand whats going on with this server 😕

0 Kudos
Mustibh
Contributor
Contributor

Nobody knows any fix about this ?

Problem still exists

0 Kudos
Mustibh
Contributor
Contributor

Problem still exists ? No one had this problem ?

0 Kudos
vijayrana968
Virtuoso
Virtuoso

Did you check space on the /locker directory ? Have a look on this : ESXi 6 Update error – No Space left on device /locker » vPRH

0 Kudos
YaZi
Contributor
Contributor

I have the same issue on 2 different ESXI servers, both running ESXi-6.5.0-20170104001-standard.

Have you made any progress? Like you I have tried pretty much all KB's that seems related and made no progress.

File: "/"

ID: 100000000 Namelen: 127     Type: visorfs

Block size: 4096     

Blocks: Total: 356432     Free: 164084     Available: 164084

Inodes: Total: 655360     Free: 648922

Ramdisk                    Size           Used      Available      Use% Mounted on

root                            32M            24K       31M             0% --

etc                             28M           164K       27M             0% --

opt                             32M             0B       32M             0% --

var                             48M           320K       47M             0% --

tmp                            256M             8K      255M             0% --

iofilters                       32M             0B       32M                  0% --

hostdstats                 112M             1M      110M             1% --

stagebootbank          250M           146M      103M            58% --

Filesystem        Size        Used           Available      Use%      Mounted on

VMFS-5          225.2G      158.9G          66.3G       71%      /vmfs/volumes/datastore1

vfat                 285.8M      128.0K         285.7M        0%      /vmfs/volumes/58de5208-936557fc-18e3-f44d30665179

vfat                   4.0G         640.0K           4.0G           0%      /vmfs/volumes/58de520e-00e0262a-3724-f44d30665179

vfat                 249.7M      143.7M         106.0M       58%      /vmfs/volumes/8ce13377-7f367bc5-634a-9b0ea248875f

vfat                 249.7M      146.5M         103.2M       59%      /vmfs/volumes/b2a84977-623ad144-c31a-02c96d03d2ed

0 Kudos
msripada
Virtuoso
Virtuoso

If it is cannot allocate memory, then can you try restarting the ESXi host after keeping it in maintenance mode and then try to update it after reboot.

Thanks,

MS

YaZi
Contributor
Contributor

Thanks, tried your suggestion but still get the same ...

[InstallationError]

[Errno 28] No space left on device

       vibs = VMware_locker_tools-light_6.5.0-0.23.5969300

Tried using script from here, worked for previous updates perfectly, but not this time.

https://esxi-patches.v-front.de/ESXi-6.5.0.html

esxcli network firewall ruleset set -e true -r httpClient

esxcli software profile update -p ESXi-6.5.0-20170701001s-standard \

-d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

esxcli network firewall ruleset set -e false -r httpClient

Not sure what else to check or try?

0 Kudos
FirstS0ul
Contributor
Contributor

Hey!

i have the same issue and tested all the suggestions...

Any update?

UPDATE:

I've solution!

1) Install the locker vib manually:

cd /tmp

wget http://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_to...

esxcli software vib install -f -v /tmp/VMware_locker_tools-light_6.5.0-0.23.5969300.vib

2)

Update:

esxcli software profile update -f -p ESXi-6.5.0-20170702001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-

depot-index.xml

Reboot and youre done 🙂

FreeAgent
Contributor
Contributor

@FirstS0ul

Thanks for your solution.  We tried this but encountered memory errors. The solution was to enter the host webui and Go to Host > System > Swap and activate swap on our datastore vmfs. Once the setting was enabled the esxcli software profile update command worked normally.

Thanks for posting very helpful.

View solution in original post

sphensilhagen
Contributor
Contributor

this worked for me!

0 Kudos
YaZi
Contributor
Contributor

Thanks, this worked for me also.


I tested just enabling SWAP and skipping all the other steps, which also worked, so I guess only that is needed.

dferner
Contributor
Contributor

I hit this issue when attempting to update to 6.5.0 U1 and swap was enabled, but not set to use my datastore. This was an easy fix. Thank you! I previously have had to update the ESXi environments using a thumb drive which extended downtime. With 6.5.0 U1 only being an update at this point, I was unable to move to this version using my previous method and was stuck not being able to use the CLI update method either.

0 Kudos
possamai
Contributor
Contributor

Just when I ended up installing them like this:

esxcli software vib update \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-base/VMware_bootbank_es... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-tboot/VMware_bootbank_e... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-ui/VMware_bootbank_esx-... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-xserver/VMware_bootbank... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/i40en/VMW_bootbank_i40en_1.... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/igbn/VMW_bootbank_igbn_0.1.... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/ixgben/VMW_bootbank_ixgben_... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/lsu-hp-hpsa-plugin/VMware_b... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/lsu-lsi-lsi-mr3-plugin/VMwa... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/lsu-lsi-lsi-msgpt3-plugin/V... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/lsu-lsi-megaraid-sas-plugin... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/lsu-lsi-mpt2sas-plugin/VMwa... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/misc-drivers/VMW_bootbank_m... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/ne1000/VMW_bootbank_ne1000_... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/ntg3/VMW_bootbank_ntg3_4.1.... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/nvme/VMW_bootbank_nvme_1.2.... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/pvscsi/VMW_bootbank_pvscsi_... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/qlnativefc/VMW_bootbank_qln... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/sata-ahci/VMW_bootbank_sata... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_t... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/usbcore-usb/VMW_bootbank_us... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vmkata/VMW_bootbank_vmkata_... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vmkusb/VMW_bootbank_vmkusb_... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vmw-ahci/VMW_bootbank_vmw-a... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vmware-esx-esxcli-nvme-plug... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vsan/VMware_bootbank_vsan_6... \

-v https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vsanhealth/VMware_bootbank_...

I read the part about the swap location....

Oh well, in case the swap thing doesn't work for someone, the above worked for me.

I got all the urls to the individual patches from https://esxi-patches.v-front.de/ESXi-6.5.0.html

0 Kudos
Computerpathway
Contributor
Contributor

Thanks, guys solved my issue too. working

0 Kudos
vivithemage
Contributor
Contributor

This resolved it for me, thanks FirstSOul.

0 Kudos
Masterplatt
Contributor
Contributor

THANK YOU!

This worked for me 100% I have added the commands below for the 2018 update which includes the fix for the Spectre vulnerability;

FIX:

- cd /tmp

- wget http://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-light/VMware_locker_to...

- esxcli software vib install -f -v /tmp/VMware_locker_tools-light_6.5.0-1.33.7273056.vib

2018 Update:

- esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.5.0-20180104001-standard

0 Kudos