Bad file descriptor during update

[toxadi]

Hello all,

I am trying to patch an ESXi 5.0.0.0-474610 but when I run

esxcli software vib install -d /vmfs/volumes/...

I get

[InstallationError]
 Failed to clear bootbank content /altbootbank: [Errno 9] Bad file descriptor: '/altbootbank/state.167901'
 Please refer to the log file for more details.

When I go to /altbootbank I see several state.XXXXXX folders and their contents, but the specific state.167901 is having problems. I cannot cd into it, I cannot delete it even with rm -Rf.

~ # cd /altbootbank/ /vmfs/volumes/295d3aaa-e07d5aaa-6aaa-2a6e97710aaa # ls -l

some other files... drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.167901 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.173542 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.1905743 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.2006991 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.2044903 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.2075069 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3439559 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3814560 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3869077 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3999265 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4036776 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4059918 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4137343 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4998361 -rwx------    1 root     root              23225 Aug  1 17:43 state.tgz some other files...

/vmfs/volumes/295d38eb-e07d5e9d-6f8e-2a6e97710fdd # cd state.167901/ sh: getcwd: No such file or directory (unknown) #

and

# rmdir state.167901 rmdir: 'state.167901': Bad file descriptor # rm -Rf state.167901 rm: cannot remove 'state.167901': Bad file descriptor

dosfsck finds no errors in that partition, so I am pretty stuck.

Ideas anyone? 🙂

[sysxperts]

I had similar problem and unfortunately found no workaround other than to rebuild the server.

[toxadi]

I opened a support ticket to DELL and this is exactly what they told me to do:-(

There must be something wrong in the SD boot card

[Ajsharma]

This issue could be due to file system errors on the altbootbank partition. Can you VMware KB -  VMware KB: Remediating an ESXi 5.x host with Update Manager fails with the error: There was an error...

[nE0sIghT]

To delete broken filesystem entries you must use dosfsck tool with "-d" key and short (8.3 format) filename.

For an example let's assume you have 4 broken "state" entries in altbootbank:

~ # ls -l /altbootbank/
-rwx------    1 root     root          1323 Aug  2 10:02 boot.cfg
-rwx------    1 root     root           197 Aug  2 10:02 imgdb.tgz
drwxr-xr-x    1 root     root             8 Jan  1  1980 state.19851
drwxr-xr-x    1 root     root             8 Jan  1  1980 state.29714
drwxr-xr-x    1 root     root             8 Jan  1  1980 state.72154
drwxr-xr-x    1 root     root             8 Jan  1  1980 state.77021

First of all you must determine disk device with vmkfs tool:

~ # vmkfstools -P /altbootbank
vfat-0.04 file system spanning 1 partitions.
File system label (if any):
Mode: private
Capacity 261853184 (63929 file blocks * 4096), 78376960 (19135 blocks) avail
UUID: 0xxxxxe
Partitions spanned (on "disks"):
        naa.60a980xxxxx2:5
Is Native Snapshot Capable: NO

In that case disk device is "naa.60a980xxxxx2:5".

Than you must run this commands to delete 4 broken filesystem entries (I copied dosfsck output only for 1st command):

~ # dosfsck -ad /STATE~1.198 /dev/disks/naa.60a980xxxxx2:5
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
Dropping STATE~1.198
Orphaned long file name part "state.19851"
  Auto-deleting.
Reclaimed 2 unused clusters (8192 bytes) in 1 chain.
Performing changes.
/dev/disks/naa.60a980xxxxx2:5: 6 files, 10/63929 clusters


~ # dosfsck -ad /STATE~1.297 /dev/disks/naa.60a980xxxxx2:5
~ # dosfsck -ad /STATE~1.721 /dev/disks/naa.60a980xxxxx2:5
~ # dosfsck -ad /STATE~1.770 /dev/disks/naa.60a980xxxxx2:5

After that you will have "FSCK" files in altbootbank instead of broken entries:

~ # ls -l /altbootbank/
-rwx------    1 root     root          8192 Jan  1  1980 FSCK0000.REC
-rwx------    1 root     root          8192 Jan  1  1980 FSCK0001.REC
-rwx------    1 root     root          8192 Jan  1  1980 FSCK0002.REC
-rwx------    1 root     root          8192 Jan  1  1980 FSCK0003.REC
-rwx------    1 root     root          1323 Aug  2 10:02 boot.cfg
-rwx------    1 root     root           197 Aug  2 10:02 imgdb.tgz

You now can safely delete them:

~ # rm /altbootbank/FSCK000*