mishagreen
Contributor
Contributor

How to map LBA of undelying LUN to individual files in VMFS5

Hello,

I'm looking for a way to map LBA of a LUN that holds VMFS5 file system to file in the file system. In other words, given an LBA how can I find out what file (if any) sits on top of that LBA?

Is it possible? I'd appreciate if somebody could tell the technique or point to appropriate reading where I can learn it.

Thanks a lot,

Michael

0 Kudos
8 Replies
CedricAnto
VMware Employee
VMware Employee

AFAIK., I believe you can go as far as using hexdump to look at raw hex data associated to LBAs & perhaps take a back up using dd. But only vmfs driver can enumerate the contents/translate those into readable file.

There are certain vmkfstools options to understand disk layout/geometry

There is also a utility called VOMA from vSphere 5.1, if incase you are looking to check integrity of a VMFS volume

If I may, Can I understand the requirement , I may be able to provide more insight or directives.

Cedric http://in.linkedin.com/in/cedricrajendran/ http://virtualknightz.com/
0 Kudos
mishagreen
Contributor
Contributor

Hi CedricAnto,

Thank you for your comment. I work for an IT solutions provider. Our customer has an issue with a few LBAs in one of their VMFS5 LUNs. The data there is probably lost. We know that because the storage array logged an event that suggests that this is what might have happened.

We cannot ask the customer to offline the LUN to run integrity checks at the moment, and even if we could, it doesn't guarantee that the integrity check will identify the issue, let alone fix it.

Given what we know now, the only AP we can give the customer is to recover the whole volume from tape. The idea that we had is that if we can identify what file sits on top of those LBAs that opens up new possibilities:

  1. Tell the customer to restore only that file from backup (much more targeted, less impact-full AP)
  2. If we're really lucky, we may find out that the space doesn't currently hold any file data (used to have in the past) and then the whole issue can be dismissed.

Hope this makes it more clear,

Thanks a lot!

0 Kudos
CedricAnto
VMware Employee
VMware Employee

Hi ,

It is important to understand if the damage is on the metadata of the LUN or the actual data(where VM resides). Metadata spans about less than  first 1.5 GB of the vmfs volume.

You would not be able to selectively replace a file into a LBA region(even if you manage to trace it I not sure if this is possible at all), the volume will be corrupted nonetheless.

If its metadata or actual data, avoid powering off VM's

Safe Options are

1-Use converter to migrate V2V the VM's -- I say this since in the past, Storage VMotion of VM from a corrupted datastore can lead to VM crash and subsequently may fail to power on.

2-Re-create VMFS volumes on affected lun

I know this is the hard way. But certainly would not recommend replacing individual files from tape, knowing that there is some damage to the filesystem.

Hope this helps and all the best.

Cedric http://in.linkedin.com/in/cedricrajendran/ http://virtualknightz.com/
0 Kudos
mishagreen
Contributor
Contributor

Hi,

The damage is 8 LBAs (4k) in megabyte 497043 (485GB away from the start of the volume), so it must be the data zone of the file system, correct? We know exactly what the LBAs are and we've verified that they most likely contain invalid data in them. Can we find what the file sits on top of those LBAs? What kind of tools either VMware or 3rd party that can provide insight here?

Again, I think it makes sense to try this approach first before we try more painful, labor intensive things, because knowing what file owns those LBAs can open up new possibilities: the file might be an old and not used, or it might be an empty space (as far as VMFS is concerned). And even if we see that VMDK of a production VM sits there doing V2V will have to be applied to a single VM, not to all VMs that have leg in the datastore.

Thanks a lot!

0 Kudos
CedricAnto
VMware Employee
VMware Employee

Yes, that certainly is in the data region also it looks like you hardly have 1 MB of corruption- that is less than one block(if contiguous) of VMFS volume. So I can understand the pain 🙂 that you do not prefer full restore/recreation

But I doubt if it is possible to trace the file associated. If you are not specific about free utilities , try the vendor listed here

VMware KB: Data recovery services for data not recoverable by VMware Technical Support

Cedric http://in.linkedin.com/in/cedricrajendran/ http://virtualknightz.com/
0 Kudos
CedricAnto
VMware Employee
VMware Employee

Michael,

As indicated earlier this is not the best approach, sharing nonetheless since you have requested

Do note that VMware does not support/perform data recovery services and I doubt if there are any utilities to provide file to LBA mapping.

As reverse engineering , you can try these for vmdk's alone. see the below example,

~ # vmkfstools -t0 /vmfs/volumes/Openfiler_3/

Test/Test.vmdk
Mapping for file /vmfs/volumes/Openfiler_3/TestDBS/Test.vmdk (1073741824 bytes in size):
[           0:  1073741824] --> [VMFS Z- LVID:52d63c04-fcdad496-be74-000c29409d5c/52d63c03-c120e0bb-6715-000c29409d5c/1:(  2746220544 -->   3819962368)]

Logical LBA range is for file is - [ 0:  1073741824]

Physical Byte offsets - 2746220544 -->   3819962368

Divide the physical byte offset by 512 and you would have the physical HBA range. Which may or may not include the LBA's you know as impacted.

Pretty much needle in a haystack!!!

Cedric http://in.linkedin.com/in/cedricrajendran/ http://virtualknightz.com/
mishagreen
Contributor
Contributor

Hi Cedric,

Thanks a again. This utility looks interesting. A have couple of questions you might be able to answer:

  1. Can VMDKs (or any file for that matter) be segmented within the VMFS5 file system or are always written contiguously?
    1. If segmented will the utility print all segments of the file or just first/last one?
  2. Is there any impact expected from running this utility on single file, dozens of files (by batch shell script), thousands of files?

Much appreciated!

0 Kudos
CedricAnto
VMware Employee
VMware Employee

Hi Michael,

1- During provisioning the host attempts to keep files contiguous, but it truly depends on the how fragmented the volumes are. As you can imagine a SVMotion of a vmdk will punch a hole in the middle of a volume and host has re-use the space. So answer is "it depends", it may or may not be contiguous.

vmkfstools will display multiple ranges if its non-contiguous.

2-As long as the vmdk files are not locked(VM powered on) , this should work.

Cedric http://in.linkedin.com/in/cedricrajendran/ http://virtualknightz.com/