VSphere Datastore doesn't show the correct free space

[fteijeiro]

I have 3 Vmware Sphere ESX 4.0.0 build-164009.

Well, one of them has a vmfs datastore with 820 Gb capacity. Actually, I have deleted all VMs on this host, but VI shows me that I have occupied 342.37Gb, as you can see "DataStoreESX2.png".

When I click on "Browse DataStore" I can only see ESX Console and I am sure it doesn't look like to be the problem. (see DataStoreBrowse.png).

I have checked via SSH connection with some commands following the KB http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100356..., these are the results:

[root@esx2 ~]# vdf -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             4.9G  1.8G  2.8G  40% /
/dev/cciss/c0d0p1     1.1G   75M  952M   8% /boot
/dev/sda2             2.0G  216M  1.7G  12% /var/log
/vmfs/devices         3.2T     0  3.2T   0% /vmfs/devices
/vmfs/volumes/4ae5b521-79834e81-42f3-00237d21deb6
                      819G  342G  476G  41% /vmfs/volumes/ESX2

I've tried to find where these files are, but it doesn't show me anything, only these files...

[root@esx2 ~]# find /vmfs/volumes/4ae5b521-79834e81-42f3-00237d21deb6/ -size +10M -exec du -h {} \;
61M     /vmfs/volumes/4ae5b521-79834e81-42f3-00237d21deb6/.fdc.sf
244M    /vmfs/volumes/4ae5b521-79834e81-42f3-00237d21deb6/.pbc.sf
249M    /vmfs/volumes/4ae5b521-79834e81-42f3-00237d21deb6/.sbc.sf
7.8G    /vmfs/volumes/4ae5b521-79834e81-42f3-00237d21deb6/esxconsole-4ae5d035-e725-d2dd-c93b-00237d21deb6/esxconsole-flat.vmdk

Also I have checked log vmkernel and I have some errors about datastore.

NOTE: The files ending with vrb extensions are because I'm using Veeam Backup software and I had only 20 Gb of free space datastore.

Sep 18 12:07:30 esx2 vmkernel: 4:17:26:03.790 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x4100060a6ac0) to NMP device "mpx.vmhba0:C0:T0:L0" failed on physical path "vmhba0:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

Sep 18 12:07:30 esx2 vmkernel: 4:17:26:03.790 cpu1:4097)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

Sep 18 12:07:31 esx2 vmkernel: 4:17:26:04.759 cpu3:4105)FSS: 3647: No FS driver claimed device '4ae5b4dd-ae7d1b88-1293-00237d21deb6': Not supported

Sep 19 03:31:32 esx2 vmkernel: 5:08:50:05.713 cpu2:4108)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-18T030041.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 28801e44 51c7 0 0 0 0 0: No space left on device

Sep 19 03:31:32 esx2 vmkernel: 5:08:50:05.831 cpu2:4108)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-18T030041.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 28801e44 51c7 0 0 0 0 0: No space left on device

Sep 19 03:32:05 esx2 vmkernel: 5:08:50:37.967 cpu2:4109)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-18T011532.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 22c01e44 51b1 0 0 0 0 0: No space left on device

Sep 19 03:32:05 esx2 vmkernel: 5:08:50:38.022 cpu3:4109)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-18T011532.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 22c01e44 51b1 0 0 0 0 0: No space left on device

Sep 20 03:57:22 esx2 vmkernel: 6:09:15:55.133 cpu3:4108)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-19T030027.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 9c01e44 5209 0 0 0 0 0: No space left on device

Sep 20 03:57:22 esx2 vmkernel: 6:09:15:55.281 cpu3:4108)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-19T030027.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 9c01e44 5209 0 0 0 0 0: No space left on device

Sep 20 03:57:22 esx2 vmkernel: 6:09:15:55.304 cpu3:4110)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-19T011522.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 3c01e44 51f3 0 0 0 0 0: No space left on device

Sep 20 03:57:22 esx2 vmkernel: 6:09:15:55.359 cpu3:4110)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-19T011522.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 3c01e44 51f3 0 0 0 0 0: No space left on device

Sep 21 03:39:20 esx2 vmkernel: 7:08:57:53.080 cpu3:4110)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-20T011518.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 16401e44 5235 0 0 0 0 0: No space left on device

Sep 21 03:39:20 esx2 vmkernel: 7:08:57:53.195 cpu3:4110)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-20T011518.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 16401e44 5235 0 0 0 0 0: No space left on device

Sep 21 03:39:20 esx2 vmkernel: 7:08:57:53.306 cpu2:4109)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-20T030030.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 1dc01e44 524b 0 0 0 0 0: No space left on device

Sep 21 03:39:20 esx2 vmkernel: 7:08:57:53.372 cpu2:4109)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-20T030030.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 1dc01e44 524b 0 0 0 0 0: No space left on device

Sep 24 04:02:13 esx2 vmkernel: 10:09:20:46.412 cpu3:4109)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-21T011519.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 2ac01e44 5277 0 0 0 0 0: No space left on device

Sep 24 04:02:13 esx2 vmkernel: 10:09:20:46.550 cpu3:4109)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-21T011519.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 2ac01e44 5277 0 0 0 0 0: No space left on device

Sep 24 04:02:13 esx2 vmkernel: 10:09:20:46.646 cpu2:4108)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-21T030023.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 30c01e44 528d 0 0 0 0 0: No space left on device

Sep 24 04:02:13 esx2 vmkernel: 10:09:20:46.801 cpu2:4108)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-21T030023.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 30c01e44 528d 0 0 0 0 0: No space left on device

Sep 25 04:19:33 esx2 vmkernel: 11:09:38:06.483 cpu2:4110)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-24T030022.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 12c01e44 52cf 0 0 0 0 0: No space left on device

Sep 25 04:19:33 esx2 vmkernel: 11:09:38:06.506 cpu3:4108)BC: 2940: Failed to flush 128 buffers of size 8192 each for object '2012-09-24T011534.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d c801e44 52b9 0 0 0 0 0: No space left on device

Sep 25 04:19:33 esx2 vmkernel: 11:09:38:06.603 cpu3:4110)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-24T030022.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d 12c01e44 52cf 0 0 0 0 0: No space left on device

Sep 25 04:19:33 esx2 vmkernel: 11:09:38:06.668 cpu2:4108)BC: 2940: Failed to flush 2 buffers of size 8192 each for object '2012-09-24T011534.vrb' f530 28 3 4ae5b521 79834e81 230042f3 b6de217d c801e44 52b9 0 0 0 0 0: No space left on device

I'm not sure, but it looks like a hardware failure... How can I check vmfs volume? How can I check hardware problems? My physical server is a HP Proliant DL380 G5 out of garantee.

Thanks in advance for your help