Hi,
We are running 5 VMWare ESX 3.5 Hosts connected to at Hitachi SAN via fibre.
Our VMWare Hosts all have 7 248Gb Luns presented to them which have been joined together using extents to form a 1.7TB VMFS
The VI Client says of this 1.7TB we have 430Gb free so we have used 1.3TB. However this is wrong I have totalled up the size of every file on the VMFS (by browsing datastore) and we have used 800Gb.
So where could our missing 500Gb be?
Any ideas \ suggestions \ recommendations etc?
Thanks
Phil
hmmm strange.. looks like your datastore has a serious problem.
can you use the other disks as seperate datastores or is that also not possible.
what kind of error do you get when you try to add the two disks back to the datastore ?
are there strange errors in the virtualcenter log files ?
i almost should say you have to recreate your datastore.
are you utilizing any RDMs?
No at the moment but we are planning to in future, why do you ask?
What does it say when you type the following command on the service console ?
vdf -h
do you get the same results ?
you can got to the esx host with ssh (putty)
logon as root.. (or first as another user and then su - for root access)
thean type: vdf -h
your result should look like..
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p2 6.0G 199M 5.5G 4% /
/dev/cciss/c0d0p1 247M 27M 209M 12% /boot
/dev/cciss/c0d0p7 2.0G 33M 1.9G 2% /home
/dev/cciss/c0d0p9 2.0G 226M 1.7G 12% /opt
none 250M 0 250M 0% /dev/shm
/dev/cciss/c0d0p8 2.0G 33M 1.9G 2% /tmp
/dev/cciss/c0d0p10 2.0G 1018M 896M 54% /usr
/dev/cciss/c0d0p6 2.0G 615M 1.3G 33% /var
/vmfs/devices 169G 0 169G 0% /vmfs/devices
/vmfs/volumes/48341f38-2b47d258-9433-0013216bc092
49G 517M 48G 1% /vmfs/volumes/Local-LUN
/vmfs/volumes/48776efd-8d1df95c-e70f-0013216bc093
25G 18G 7.1G 72% /vmfs/volumes/LUN1
Sorry.
Yes I do get the same result
/vmfs/volumes/48087ab4-f12cae2a-4684-0010181cae7b
1.7T 1.3T 436G 74% /vmfs/volumes/SANVmfs
did you take into account that the following also uses storage on the datastore:
the virtual machine swap file
snapshotfiles
maybe you have a virtual machine with a huge snapshot (delta) file..
maybe you have forgotten to cleanup some virtual disks or vm's from that datastore by selecting remove from inventory instead of delete from disk.
I used the VI Client to browse the datastore and went from folder to folder putting all of the file sizes into a spreadsheet.
I have just done this again for the second time and came out with pretty much the smae result
Very Strange, I am not sure how VMWare deals with extents for example can a vmdk be split across two extents \ luns or if there is 30Gb left in a extent and you create a 40Gb VMDK will that get put on a new extent leaving 30Gb that vmware does not want to use etc?
Is there anyway in vmware / linux to break down space usage to files etc
Just used the linux command du while SSH'ed onto one of the hosts to display the folder sizes in the VMFS volume here is what came back (I have changed the server names)
It totals up to about 820Gb not the 1.3TB the VI client says we have used
11G Server1
26G Server2
26G Server3
1.1M Server4
62G Server5
61G Server6
61G Server7
12G Server8
33G Server9
53G Server10
23G Server11
43G Server12
38G Server13
25G Server14
45G Server15
43G Server16
28G Server17
19G Server18
513M Server19
43G Server20
11G Server21
28G Server22
11G Server23
55G Server24
14G Server25
11G Server26
16G Server27
22G Server28
And there are these hidden files in the root
8.8M .fbb.sf
60M .fdc.sf
244M .pbc.sf
248M .sbc.sf
4.0M .vh.sf
hmmm starnge..
looks like you missing two disks of 248GB (extents)
do you something strange when you look at the properties of the datastore a failed disk perhaps ?
do you see some warnings or errors in your esx server log files related to datastore ? (/var/log/)
You may be onto something is this normal;
When I go to add a RDM to a virtual server in the list of available luns I can use it list 2 of my 248Gb extents???
I am going to take a look though the logs in a min
hmmm strange.. looks like your datastore has a serious problem.
can you use the other disks as seperate datastores or is that also not possible.
what kind of error do you get when you try to add the two disks back to the datastore ?
are there strange errors in the virtualcenter log files ?
i almost should say you have to recreate your datastore.
When adding extents, the only fool-proof way to make sure the space is visible to all hosts correctly, you have to reboot the host. I use extents as well, and I've noticed this issue, and the only other way I've noticed to do this is completely manual method using fdisk and vmkfstools. It is not the easiest thing to do. If you are using the LUNs when you add the extent to it, you will have to reboot the host to be safe. Otherwise, the added extents don't always get added correctly.
-KjB
I have gone for the option of re-creating the datastore but I am still having problems.
I have moved all of my virtual machines to another temp datastore, deleted the datastore that is casuing me problems then created a new one and added in the extents back in.
Everying looked good at this point but on different hosts if I do a Rescan in Storage Adapters I will somethings loose between 1 and 5 extents and somethings when I do a rescan they will come back or only a couple of them will come back.
I have checked all of my Hosts can see all of the LUNS on all of the Paths, all have the same LUN numbers etc and I cannot see LUNS going missing when I do a Rescan only my extents
When the extents go missing I cannont use those LUNS to create another datastore or re-add them in as an extent, VMWare knows there are in use but they go missing somewhere
Any ideas? What Logs could I check etc? Are extents even the way to go, should I just create one big 1.7Tb Lun ?
I found some similar threads all without a clear answer
Extents not adding up...
http://communities.vmware.com/message/833622#833622
ESX 3.0.2 - shown formatted capacity of datastore (in VC and on host) differs from real LUN capacity
http://communities.vmware.com/message/814990#814990
SAN storage: problem adding extents
http://communities.vmware.com/message/606485#606485
Large Extents - incorrect free space
http://communities.vmware.com/message/598751#598751
Hi PhilEddies,
i know it's a workaround and don't know if it's suitable to your environment.
can you just forget the extents and create 7 luns/datastores of 248GB out of it ?
using extents brings always extra risks, when loosing one disk can result in loosing all data on all 7 disks.