VMware Cloud Community
Kumargtk
Contributor
Contributor

How to find Orphaned VMDK files

How can we find out Orphaned VMDK files in all the Data-stores.

119 Replies
LucD
Leadership
Leadership

Have you deleted some of your previous replies?
That makes it harder to try and find a solution.

Can you run the attached script, and send me the result (PM will do if there is confidential info in there).


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
formz
Enthusiast
Enthusiast

Coming back to this I'm getting the error again. All hosts are connected and seemingly fine.

Exception calling "SearchDatastoreSubFolders" with "2" argument(s): "An error occurred while communicating with the remote host."

At C:\Users\mhenze_sa\Documents\Scripts\Delete Orphan VMDKs.ps1:76 char:9

+         $searchResult = $dsBrowser.SearchDatastoreSubFolders($rootPat ...

+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : VimException

Would it be a Write-Host Get-VMHost within the loop that does the search but outside of the if esleif statements?

IE:

foreach($folder in ($searchResult | where{$_.FolderPath -notmatch “\] \.snapshot/“})){

        Write-host Get-VMHost

          if($fldList.ContainsKey($folder.FolderPath.TrimEnd('/'))){ 

            foreach ($file in $folder.File){ 

Reply
0 Kudos
LucD
Leadership
Leadership

As a matter of fact you can't 'see' the ESXi node in this version of the script.

But I suspect you might be using a datastore that is not mapped to all the ESXi nodes (to which it normally should be).

Do you get that same error for all shared datastores?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
formz
Enthusiast
Enthusiast

A host not being mapped was my first thought as well so I checked and they are all mapped.

The environment is organized like this:

Cluster: Dev (29 hosts)

Datastore Cluster: DevStorage

Datastores inside the cluster: Dev001, 002, 003, 004, 005, 006, 007, 008

As just a test, I ran it against 001 rather than the entire DSC. That's where I got that error. I ran it against just 002 and it worked as expected and deleted the orphaned vmdks. Both datastores are mapped to the same 29 hosts.

Reply
0 Kudos
LucD
Leadership
Leadership

Strange, but I have seen reports of that error before, but was never able to get to the reason, or reproduce the error.

Can you run the following for 2 datastores, the one that produces the error and another one that works ok.

$dsName = '<datastore_name>'

$ds = Get-Datastore -Name $dsName

$ds | Select Name,State,DatastoreBrowserPath,FileSystemVersion,

    @{N='Host#';E={$_.ExtensionData.Host.Count}},

    @{N='Host';E={(Get-View -Id $_.ExtensionData.Host.Key -property Name).Name -join '|'}},

    @{N='Browser';E={$_.ExtensionData.Browser}}

Also, can you check the vpxd log to see if there is any further info available at the moment you get the error?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
formz
Enthusiast
Enthusiast

Broken Datastore (I redacted some info)

Name                 : dev001

State                : Available

DatastoreBrowserPath : vmstores:\vcentername.local@443\DevCluster\dev001

FileSystemVersion    :

Host#                : 29

Host                 : host listing

Browser              : HostDatastoreBrowser-datastoreBrowser-datastore-358028

Working Datastore:

Name                 : dev002

State                : Available

DatastoreBrowserPath : vmstores:\vcentername.local@443\DevCluster\dev002

FileSystemVersion    :

Host#                : 29

Host                 : same host listing

Browser              : HostDatastoreBrowser-datastoreBrowser-datastore-358029

Here is the vpdx.log output. I've bolded what I think to be the problem, ascvms034 is a host. But all the other datastores that work are also mapped to that host.

2018-02-27T10:04:48.115-06:00 warning vpxd[10592] [Originator@6876 sub=InvtVmDb opID=HB-host-82797@451276-30907f95] [VpxdInvtVm::SaveGuestToDb] Skipping update of NIC settings for unknown device; key=-1, VM id=349211

2018-02-27T10:04:48.525-06:00 warning vpxd[21364] [Originator@6876 sub=VpxProfiler opID=737b587c] ClientAdapterBase::InvokeOnSoap: (ascvms034.internal, vpxapi.VpxaService.searchDatastore) [SoapRpcTime] took 1800001 ms

2018-02-27T10:04:48.525-06:00 error vpxd[21364] [Originator@6876 sub=Vmomi opID=737b587c] [VpxdClientAdapter] Got vmacore exception: Operation timed out

2018-02-27T10:04:48.531-06:00 error vpxd[21364] [Originator@6876 sub=Vmomi opID=737b587c] [VpxdClientAdapter] Backtrace:

-->

--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-4541947, tag: vpxd

--> backtrace[00] vmacore.dll[0x001BEDFA]

--> backtrace[01] vmacore.dll[0x0005CB7F]

--> backtrace[02] vmacore.dll[0x0005DC5E]

--> backtrace[03] vmacore.dll[0x0001346B]

--> backtrace[04] vmacore.dll[0x00075966]

--> backtrace[05] vmacore.dll[0x000861B3]

--> backtrace[06] vmacore.dll[0x00089F32]

--> backtrace[07] vmomi.dll[0x0005DA9F]

--> backtrace[08] vmomi.dll[0x0005E179]

--> backtrace[09] vpxd.exe[0x0104F2BF]

--> backtrace[10] vpxd.exe[0x00425DA2]

--> backtrace[11] vpxd.exe[0x004288A7]

--> backtrace[12] vpxd.exe[0x0041B862]

--> backtrace[13] vpxd.exe[0x004279FA]

--> backtrace[14] vmomi.dll[0x000815CE]

--> backtrace[15] vpxapi-types.dll[0x0008D24B]

--> backtrace[16] vpxd.exe[0x004063B9]

--> backtrace[17] vpxd.exe[0x00405F49]

--> backtrace[18] types.dll[0x005F310B]

--> backtrace[19] vpxd.exe[0x000762AC]

--> backtrace[20] vmomi.dll[0x0004B0AC]

--> backtrace[21] vpxd.exe[0x000AF962]

--> backtrace[22] vpxd.exe[0x00086C8B]

--> backtrace[23] vpxd.exe[0x0008A784]

--> backtrace[24] vpxd.exe[0x0009AF88]

--> backtrace[25] vpxd.exe[0x000A1D23]

--> backtrace[26] vmacore.dll[0x00153D6E]

--> backtrace[27] vmacore.dll[0x001575AF]

--> backtrace[28] vmacore.dll[0x00158B01]

--> backtrace[29] vmacore.dll[0x0015A835]

--> backtrace[30] vmacore.dll[0x00065F1B]

--> backtrace[31] vmacore.dll[0x00155C90]

--> backtrace[32] vmacore.dll[0x001D2B9B]

--> backtrace[33] MSVCR90.dll[0x00002FDF]

--> backtrace[34] MSVCR90.dll[0x00003080]

--> backtrace[35] kernel32.dll[0x000159CD]

--> backtrace[36] ntdll.dll[0x0002A561]

--> [backtrace end]

-->

2018-02-27T10:04:48.535-06:00 info vpxd[21364] [Originator@6876 sub=vpxLro opID=737b587c] [VpxLRO] -- FINISH session[5275ca08-8730-a648-d890-a30f4b132ee0]52ac7eee-b72d-d3fd-d7df-63e7e6561cec

2018-02-27T10:04:48.535-06:00 info vpxd[21364] [Originator@6876 sub=Default opID=737b587c] [VpxLRO] -- ERROR session[5275ca08-8730-a648-d890-a30f4b132ee0]52ac7eee-b72d-d3fd-d7df-63e7e6561cec -- datastoreBrowser-datastore-358028 -- vim.host.DatastoreBrowser.searchSubFolders: vmodl.fault.HostCommunication:

--> Result:

--> (vmodl.fault.HostCommunication) {

-->    faultCause = (vmodl.MethodFault) null,

-->    msg = ""

--> }

--> Args:

-->

--> Arg datastorePath:

--> "[dev001]"

--> Arg searchSpec:

--> (vim.host.DatastoreBrowser.SearchSpec) {

-->    query = (vim.host.DatastoreBrowser.Query) [

-->       (vim.host.DatastoreBrowser.VmDiskQuery) {

-->          filter = (vim.host.DatastoreBrowser.VmDiskQuery.Filter) null,

-->          details = (vim.host.DatastoreBrowser.VmDiskQuery.Details) {

-->             diskType = true,

-->             capacityKb = true,

-->             hardwareVersion = false,

-->             controllerType = <unset>,

-->             diskExtents = true,

-->             thin = true

-->          }

-->       }

-->    ],

-->    details = (vim.host.DatastoreBrowser.FileInfo.Details) {

-->       fileType = true,

-->       fileSize = true,

-->       modification = false,

-->       fileOwner = false

-->    },

-->    searchCaseInsensitive = <unset>,

-->    sortFoldersFirst = true,

--> }

2018-02-27T10:04:48.648-06:00 info vpxd[225564] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-725482-7766c6c] Not persisting vFlash resource runtime info. If you have hosts configured with vFlash resource, please set vpxd config option: vpxd.PersistVFlashRuntimeInfo to true.

Reply
0 Kudos
LucD
Leadership
Leadership

Ok, seems there is a timeout on that first datastore, but I can see no obvious reason why that would happen on that specific datastore.

Does this datastore show high latency on the Performance tab?

Or are there that many folders & files on that datastore?

What I forgot to ask, is this FC, NFS or iSCSI storage?

Also strange the FileSystemVersion on either of the datastores. Could that be due to the PowerCLI version?

I'm using 6.5.4.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
formz
Enthusiast
Enthusiast

I just updated my PowerCLI. I was on 5 something. When I re-ran the previous commands for the filesystemversion I got 3.0.

They usage of the datastores, how busy they are, but I'm going to try and run it at an off time and see what happens.

I'll let you know how it goes.

Reply
0 Kudos
omar13489
Contributor
Contributor

Hi LucD,

Yes i deleted some replies because it contains sensitive data

Can you check the Output after the test.ps1 script execution

Reply
0 Kudos
LucD
Leadership
Leadership

The problem you are seeing definitely comes from the FolderPath.

To check further, can you run the attached script?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
omar13489
Contributor
Contributor

attached the output

can we modify this line :

$key = "$($folder.FolderPath)$(if($folder.FolderPath[-1] -eq ']'){' '})$($file.Path)"

To (Adding the '/' 😞

$key = "$($folder.FolderPath)/$(if($folder.FolderPath[-1] -eq ']'){' '})$($file.Path)"

Reply
0 Kudos
LucD
Leadership
Leadership

Ok, the issue seems to come from the fact that the FolderPath is not having a / at the end like I see in other environments.

I have no clue what could be causing this.

In the attached version I have adapted the code to intercept that.

Can you give this version a try?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
omar13489
Contributor
Contributor

Hi Lucd,

I think the issue had been solved after manipulating '/' inside FolderPath

The script results shows it find only 40 Orphaned Files

-Message

03/01/2018 13:58:54 Get-VmwOrphan 40 Found orphaned files on Local_DataStore_51.19!

I checked all the founded files and i find that the results are wonderful except that there is a folder called /scratch/log

here is an example :

Name         : sdrsinjector.log

Folder       : [Local_DataStore_51.19] scratch/log/sdrsinjector.log

Size         : 144942

CapacityKB   :

Modification : 2/28/2018 10:24:37 PM

Owner        : root

Thin         :

Extents      :

DiskType     :

HWVersion    :

Is it a System Log File ?

My Second Question :

Can you modify the script to include also the File Deletion Option ?

Reply
0 Kudos
LucD
Leadership
Leadership

Looks like someone configured the scratchspace of an ESXi node on that datastore.

The log can be useful, so I wouldn't delete it.

See also KB1033696

On the 2nd question I would suggest you open a new thread for that.

Point to the script in this thread and ask for the delete option.

This thread is becoming way too long :smileylaugh:


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
formz
Enthusiast
Enthusiast

Seems that no matter when I run the script it fails on the same couple data stores repeatedly. I'm at a loss at this point.

Reply
0 Kudos
LucD
Leadership
Leadership

Since the script is using public API you could consider opening a SR.

But I'm not sure you can open a SR without some kind of Developer Support (or whatever they call it).

You could run the VOMA health check on the datastore, and see if anything is reported.

See KB2036767


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
formz
Enthusiast
Enthusiast

These are all NFS datastores so VOMA isn't an option. Smiley Sad

Reply
0 Kudos
LucD
Leadership
Leadership

I see.
Are there any dropped packets on that link to the NFS export?

Is that a multi-NIC connection?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
socbizkaia
Contributor
Contributor

Hello LucD,

We are running your script to find orphaned files in our LUNs and it works great! We were able to discover a lot of vmdk files and we verified that all of them were orphaned before deleting them, so there wasn't any "fake" orphaned vmdk. However, I have some questions...

- Is there any special reason why so many orphaned vmdks are generated (problems with vMotion, snaphosts...) ?

- Do you use any Powershell command to pipe the output of your script to delete the files from Powershell? We performed manually "vmkfstools -U file.vmdk" to remove orphaned vmdk files. But we need to do it manually, SSH to the ESXi host and execute the command. How would you do it in Powershell? Any command better than remove-item?

- Your script doesn't provide output for NFS Datastores... Is there any alternative in order to analyze NFS datastores for orphaned files?

Thank you in advance and thank you for providing this useful script!

Christian

Reply
0 Kudos
LucD
Leadership
Leadership

Hi Christian,

On your questions:

  • There are many possible causes for orphaned vmdk files: vMotion gone wrong, unregistered VMs, removed harddisk from a VM with the delete option,.... There is definitely no single reason for their existance. Each case would have to be investigated individually. But if you find the same cause, you might want to review some scripts and/or procedures
  • if you are using the script from Orphaned Files Revisited, you can include NFS datastore by replacing line 67 with

if((($ds.Type -eq “VMFS”) -or ($ds.Type -eq “NFS”)) -and $ds.ExtensionData.Summary.MultipleHostAccess -and $ds.State -eq “Available”) {

I hope that helps.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference