VMware Cloud Community
PhilEddies
Contributor
Contributor
Jump to solution

VMFS Extents and Missing Space

Hi,

We are running 5 VMWare ESX 3.5 Hosts connected to at Hitachi SAN via fibre.

Our VMWare Hosts all have 7 248Gb Luns presented to them which have been joined together using extents to form a 1.7TB VMFS

The VI Client says of this 1.7TB we have 430Gb free so we have used 1.3TB. However this is wrong I have totalled up the size of every file on the VMFS (by browsing datastore) and we have used 800Gb.

So where could our missing 500Gb be?

Any ideas \ suggestions \ recommendations etc?

Thanks

Phil

Reply
0 Kudos
1 Solution

Accepted Solutions
dmaster
VMware Employee
VMware Employee
Jump to solution

hmmm strange.. looks like your datastore has a serious problem.

can you use the other disks as seperate datastores or is that also not possible.

what kind of error do you get when you try to add the two disks back to the datastore ?

are there strange errors in the virtualcenter log files ?

i almost should say you have to recreate your datastore.

View solution in original post

Reply
0 Kudos
22 Replies
weinstein5
Immortal
Immortal
Jump to solution

are you utilizing any RDMs?

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

No at the moment but we are planning to in future, why do you ask?

Reply
0 Kudos
dmaster
VMware Employee
VMware Employee
Jump to solution

What does it say when you type the following command on the service console ?

vdf -h

do you get the same results ?

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

I assume you mean by using SSH onto one of the hosts, if so this is the result

$ sudo vdf -h

Broken pipe

Reply
0 Kudos
dmaster
VMware Employee
VMware Employee
Jump to solution

you can got to the esx host with ssh (putty)

logon as root.. (or first as another user and then su - for root access)

thean type: vdf -h

your result should look like..

# vdf -h

Filesystem Size Used Avail Use% Mounted on

/dev/cciss/c0d0p2 6.0G 199M 5.5G 4% /

/dev/cciss/c0d0p1 247M 27M 209M 12% /boot

/dev/cciss/c0d0p7 2.0G 33M 1.9G 2% /home

/dev/cciss/c0d0p9 2.0G 226M 1.7G 12% /opt

none 250M 0 250M 0% /dev/shm

/dev/cciss/c0d0p8 2.0G 33M 1.9G 2% /tmp

/dev/cciss/c0d0p10 2.0G 1018M 896M 54% /usr

/dev/cciss/c0d0p6 2.0G 615M 1.3G 33% /var

/vmfs/devices 169G 0 169G 0% /vmfs/devices

/vmfs/volumes/48341f38-2b47d258-9433-0013216bc092

49G 517M 48G 1% /vmfs/volumes/Local-LUN

/vmfs/volumes/48776efd-8d1df95c-e70f-0013216bc093

25G 18G 7.1G 72% /vmfs/volumes/LUN1

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

Sorry.

Yes I do get the same result

/vmfs/volumes/48087ab4-f12cae2a-4684-0010181cae7b

1.7T 1.3T 436G 74% /vmfs/volumes/SANVmfs

Reply
0 Kudos
dmaster
VMware Employee
VMware Employee
Jump to solution

did you take into account that the following also uses storage on the datastore:

the virtual machine swap file

snapshotfiles

maybe you have a virtual machine with a huge snapshot (delta) file..

maybe you have forgotten to cleanup some virtual disks or vm's from that datastore by selecting remove from inventory instead of delete from disk.

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

I used the VI Client to browse the datastore and went from folder to folder putting all of the file sizes into a spreadsheet.

I have just done this again for the second time and came out with pretty much the smae result

Very Strange, I am not sure how VMWare deals with extents for example can a vmdk be split across two extents \ luns or if there is 30Gb left in a extent and you create a 40Gb VMDK will that get put on a new extent leaving 30Gb that vmware does not want to use etc?

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

Is there anyway in vmware / linux to break down space usage to files etc

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

Just used the linux command du while SSH'ed onto one of the hosts to display the folder sizes in the VMFS volume here is what came back (I have changed the server names)

It totals up to about 820Gb not the 1.3TB the VI client says we have used

# du -hs *

11G Server1

26G Server2

26G Server3

1.1M Server4

62G Server5

61G Server6

61G Server7

12G Server8

33G Server9

53G Server10

23G Server11

43G Server12

38G Server13

25G Server14

45G Server15

43G Server16

28G Server17

19G Server18

513M Server19

43G Server20

11G Server21

28G Server22

11G Server23

55G Server24

14G Server25

11G Server26

16G Server27

22G Server28

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

And there are these hidden files in the root

8.8M .fbb.sf

60M .fdc.sf

244M .pbc.sf

248M .sbc.sf

4.0M .vh.sf

Reply
0 Kudos
dmaster
VMware Employee
VMware Employee
Jump to solution

hmmm starnge..

looks like you missing two disks of 248GB (extents)

do you something strange when you look at the properties of the datastore a failed disk perhaps ?

do you see some warnings or errors in your esx server log files related to datastore ? (/var/log/)

PhilEddies
Contributor
Contributor
Jump to solution

You may be onto something is this normal;

When I go to add a RDM to a virtual server in the list of available luns I can use it list 2 of my 248Gb extents???

I am going to take a look though the logs in a min

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

I am missing 2 extents, check the screenshot attached (there are 5) and the maths is wrong?

When I try to add the two missing extents back they will no go, no errors

Reply
0 Kudos
dmaster
VMware Employee
VMware Employee
Jump to solution

hmmm strange.. looks like your datastore has a serious problem.

can you use the other disks as seperate datastores or is that also not possible.

what kind of error do you get when you try to add the two disks back to the datastore ?

are there strange errors in the virtualcenter log files ?

i almost should say you have to recreate your datastore.

Reply
0 Kudos
kjb007
Immortal
Immortal
Jump to solution

When adding extents, the only fool-proof way to make sure the space is visible to all hosts correctly, you have to reboot the host. I use extents as well, and I've noticed this issue, and the only other way I've noticed to do this is completely manual method using fdisk and vmkfstools. It is not the easiest thing to do. If you are using the LUNs when you add the extent to it, you will have to reboot the host to be safe. Otherwise, the added extents don't always get added correctly.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

I have gone for the option of re-creating the datastore but I am still having problems.

I have moved all of my virtual machines to another temp datastore, deleted the datastore that is casuing me problems then created a new one and added in the extents back in.

Everying looked good at this point but on different hosts if I do a Rescan in Storage Adapters I will somethings loose between 1 and 5 extents and somethings when I do a rescan they will come back or only a couple of them will come back.

I have checked all of my Hosts can see all of the LUNS on all of the Paths, all have the same LUN numbers etc and I cannot see LUNS going missing when I do a Rescan only my extents

When the extents go missing I cannont use those LUNS to create another datastore or re-add them in as an extent, VMWare knows there are in use but they go missing somewhere

Any ideas? What Logs could I check etc? Are extents even the way to go, should I just create one big 1.7Tb Lun ?

Reply
0 Kudos
PhilEddies
Contributor
Contributor
Jump to solution

I found some similar threads all without a clear answer

Extents not adding up...

http://communities.vmware.com/message/833622#833622

ESX 3.0.2 - shown formatted capacity of datastore (in VC and on host) differs from real LUN capacity

http://communities.vmware.com/message/814990#814990

SAN storage: problem adding extents

http://communities.vmware.com/message/606485#606485

Large Extents - incorrect free space

http://communities.vmware.com/message/598751#598751

Reply
0 Kudos
dmaster
VMware Employee
VMware Employee
Jump to solution

Hi PhilEddies,

i know it's a workaround and don't know if it's suitable to your environment.

can you just forget the extents and create 7 luns/datastores of 248GB out of it ?

using extents brings always extra risks, when loosing one disk can result in loosing all data on all 7 disks.

Reply
0 Kudos