toxadi
Contributor
Contributor

Bad file descriptor during update

Hello all,

I am trying to patch an ESXi 5.0.0.0-474610 but when I run

esxcli software vib install -d /vmfs/volumes/...

I get

[InstallationError]
Failed to clear bootbank content /altbootbank: [Errno 9] Bad file descriptor: '/altbootbank/state.167901'
Please refer to the log file for more details.

When I go to /altbootbank I see several state.XXXXXX folders and their contents, but the specific state.167901 is having problems. I cannot cd into it, I cannot delete it even with rm -Rf.


~ # cd /altbootbank/ /vmfs/volumes/295d3aaa-e07d5aaa-6aaa-2a6e97710aaa # ls -l

some other files... drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.167901 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.173542 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.1905743 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.2006991 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.2044903 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.2075069 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3439559 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3814560 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3869077 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.3999265 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4036776 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4059918 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4137343 drwxr-xr-x    1 root     root                  8 Jan  1  1970 state.4998361 -rwx------    1 root     root              23225 Aug  1 17:43 state.tgz some other files...

/vmfs/volumes/295d38eb-e07d5e9d-6f8e-2a6e97710fdd # cd state.167901/ sh: getcwd: No such file or directory (unknown) #

and

# rmdir state.167901 rmdir: 'state.167901': Bad file descriptor # rm -Rf state.167901 rm: cannot remove 'state.167901': Bad file descriptor

dosfsck finds no errors in that partition, so I am pretty stuck.

Ideas anyone? 🙂

4 Replies
sysxperts
Enthusiast
Enthusiast

I had similar problem and unfortunately found no workaround other than to rebuild the server.

Paul Valentino - VCP, EMCCA - @sysxperts @vcommunitytrust - Help the vCommunity one certification at a time! http://www.vcommunitytrust.org/ http://igg.me/p/212476?a=1091980
0 Kudos
toxadi
Contributor
Contributor

I opened a support ticket to DELL and this is exactly what they told me to do:-(

There must be something wrong in the SD boot card

0 Kudos
Ajsharma
VMware Employee
VMware Employee

This issue could be due to file system errors on the altbootbank partition. Can you VMware KB -  VMware KB: Remediating an ESXi 5.x host with Update Manager fails with the error: There was an error...

0 Kudos
nE0sIghT
Contributor
Contributor

To delete broken filesystem entries you must use dosfsck tool with "-d" key and short (8.3 format) filename.

For an example let's assume you have 4 broken "state" entries in altbootbank:

~ # ls -l /altbootbank/

-rwx------    1 root     root          1323 Aug  2 10:02 boot.cfg

-rwx------    1 root     root           197 Aug  2 10:02 imgdb.tgz

drwxr-xr-x    1 root     root             8 Jan  1  1980 state.19851

drwxr-xr-x    1 root     root             8 Jan  1  1980 state.29714

drwxr-xr-x    1 root     root             8 Jan  1  1980 state.72154

drwxr-xr-x    1 root     root             8 Jan  1  1980 state.77021

First of all you must determine disk device with vmkfs tool:

~ # vmkfstools -P /altbootbank

vfat-0.04 file system spanning 1 partitions.

File system label (if any):

Mode: private

Capacity 261853184 (63929 file blocks * 4096), 78376960 (19135 blocks) avail

UUID: 0xxxxxe

Partitions spanned (on "disks"):

        naa.60a980xxxxx2:5

Is Native Snapshot Capable: NO

In that case disk device is "naa.60a980xxxxx2:5".

Than you must run this commands to delete 4 broken filesystem entries (I copied dosfsck output only for 1st command):

~ # dosfsck -ad /STATE~1.198 /dev/disks/naa.60a980xxxxx2:5

dosfsck 2.11, 12 Mar 2005, FAT32, LFN

Dropping STATE~1.198

Orphaned long file name part "state.19851"

  Auto-deleting.

Reclaimed 2 unused clusters (8192 bytes) in 1 chain.

Performing changes.

/dev/disks/naa.60a980xxxxx2:5: 6 files, 10/63929 clusters

~ # dosfsck -ad /STATE~1.297 /dev/disks/naa.60a980xxxxx2:5

~ # dosfsck -ad /STATE~1.721 /dev/disks/naa.60a980xxxxx2:5

~ # dosfsck -ad /STATE~1.770 /dev/disks/naa.60a980xxxxx2:5

After that you will have "FSCK" files in altbootbank instead of broken entries:

~ # ls -l /altbootbank/

-rwx------    1 root     root          8192 Jan  1  1980 FSCK0000.REC

-rwx------    1 root     root          8192 Jan  1  1980 FSCK0001.REC

-rwx------    1 root     root          8192 Jan  1  1980 FSCK0002.REC

-rwx------    1 root     root          8192 Jan  1  1980 FSCK0003.REC

-rwx------    1 root     root          1323 Aug  2 10:02 boot.cfg

-rwx------    1 root     root           197 Aug  2 10:02 imgdb.tgz

You now can safely delete them:

~ # rm /altbootbank/FSCK000*

0 Kudos