How can we find out Orphaned VMDK files in all the Data-stores.
Have you deleted some of your previous replies?
That makes it harder to try and find a solution.
Can you run the attached script, and send me the result (PM will do if there is confidential info in there).
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Coming back to this I'm getting the error again. All hosts are connected and seemingly fine.
Exception calling "SearchDatastoreSubFolders" with "2" argument(s): "An error occurred while communicating with the remote host."
At C:\Users\mhenze_sa\Documents\Scripts\Delete Orphan VMDKs.ps1:76 char:9
+ $searchResult = $dsBrowser.SearchDatastoreSubFolders($rootPat ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : VimException
Would it be a Write-Host Get-VMHost within the loop that does the search but outside of the if esleif statements?
IE:
foreach($folder in ($searchResult | where{$_.FolderPath -notmatch “\] \.snapshot/“})){
Write-host Get-VMHost
if($fldList.ContainsKey($folder.FolderPath.TrimEnd('/'))){
foreach ($file in $folder.File){
As a matter of fact you can't 'see' the ESXi node in this version of the script.
But I suspect you might be using a datastore that is not mapped to all the ESXi nodes (to which it normally should be).
Do you get that same error for all shared datastores?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
A host not being mapped was my first thought as well so I checked and they are all mapped.
The environment is organized like this:
Cluster: Dev (29 hosts)
Datastore Cluster: DevStorage
Datastores inside the cluster: Dev001, 002, 003, 004, 005, 006, 007, 008
As just a test, I ran it against 001 rather than the entire DSC. That's where I got that error. I ran it against just 002 and it worked as expected and deleted the orphaned vmdks. Both datastores are mapped to the same 29 hosts.
Strange, but I have seen reports of that error before, but was never able to get to the reason, or reproduce the error.
Can you run the following for 2 datastores, the one that produces the error and another one that works ok.
$dsName = '<datastore_name>'
$ds = Get-Datastore -Name $dsName
$ds | Select Name,State,DatastoreBrowserPath,FileSystemVersion,
@{N='Host#';E={$_.ExtensionData.Host.Count}},
@{N='Host';E={(Get-View -Id $_.ExtensionData.Host.Key -property Name).Name -join '|'}},
@{N='Browser';E={$_.ExtensionData.Browser}}
Also, can you check the vpxd log to see if there is any further info available at the moment you get the error?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Broken Datastore (I redacted some info)
Name : dev001
State : Available
DatastoreBrowserPath : vmstores:\vcentername.local@443\DevCluster\dev001
FileSystemVersion :
Host# : 29
Host : host listing
Browser : HostDatastoreBrowser-datastoreBrowser-datastore-358028
Working Datastore:
Name : dev002
State : Available
DatastoreBrowserPath : vmstores:\vcentername.local@443\DevCluster\dev002
FileSystemVersion :
Host# : 29
Host : same host listing
Browser : HostDatastoreBrowser-datastoreBrowser-datastore-358029
Here is the vpdx.log output. I've bolded what I think to be the problem, ascvms034 is a host. But all the other datastores that work are also mapped to that host.
2018-02-27T10:04:48.115-06:00 warning vpxd[10592] [Originator@6876 sub=InvtVmDb opID=HB-host-82797@451276-30907f95] [VpxdInvtVm::SaveGuestToDb] Skipping update of NIC settings for unknown device; key=-1, VM id=349211
2018-02-27T10:04:48.525-06:00 warning vpxd[21364] [Originator@6876 sub=VpxProfiler opID=737b587c] ClientAdapterBase::InvokeOnSoap: (ascvms034.internal, vpxapi.VpxaService.searchDatastore) [SoapRpcTime] took 1800001 ms
2018-02-27T10:04:48.525-06:00 error vpxd[21364] [Originator@6876 sub=Vmomi opID=737b587c] [VpxdClientAdapter] Got vmacore exception: Operation timed out
2018-02-27T10:04:48.531-06:00 error vpxd[21364] [Originator@6876 sub=Vmomi opID=737b587c] [VpxdClientAdapter] Backtrace:
-->
--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-4541947, tag: vpxd
--> backtrace[00] vmacore.dll[0x001BEDFA]
--> backtrace[01] vmacore.dll[0x0005CB7F]
--> backtrace[02] vmacore.dll[0x0005DC5E]
--> backtrace[03] vmacore.dll[0x0001346B]
--> backtrace[04] vmacore.dll[0x00075966]
--> backtrace[05] vmacore.dll[0x000861B3]
--> backtrace[06] vmacore.dll[0x00089F32]
--> backtrace[07] vmomi.dll[0x0005DA9F]
--> backtrace[08] vmomi.dll[0x0005E179]
--> backtrace[09] vpxd.exe[0x0104F2BF]
--> backtrace[10] vpxd.exe[0x00425DA2]
--> backtrace[11] vpxd.exe[0x004288A7]
--> backtrace[12] vpxd.exe[0x0041B862]
--> backtrace[13] vpxd.exe[0x004279FA]
--> backtrace[14] vmomi.dll[0x000815CE]
--> backtrace[15] vpxapi-types.dll[0x0008D24B]
--> backtrace[16] vpxd.exe[0x004063B9]
--> backtrace[17] vpxd.exe[0x00405F49]
--> backtrace[18] types.dll[0x005F310B]
--> backtrace[19] vpxd.exe[0x000762AC]
--> backtrace[20] vmomi.dll[0x0004B0AC]
--> backtrace[21] vpxd.exe[0x000AF962]
--> backtrace[22] vpxd.exe[0x00086C8B]
--> backtrace[23] vpxd.exe[0x0008A784]
--> backtrace[24] vpxd.exe[0x0009AF88]
--> backtrace[25] vpxd.exe[0x000A1D23]
--> backtrace[26] vmacore.dll[0x00153D6E]
--> backtrace[27] vmacore.dll[0x001575AF]
--> backtrace[28] vmacore.dll[0x00158B01]
--> backtrace[29] vmacore.dll[0x0015A835]
--> backtrace[30] vmacore.dll[0x00065F1B]
--> backtrace[31] vmacore.dll[0x00155C90]
--> backtrace[32] vmacore.dll[0x001D2B9B]
--> backtrace[33] MSVCR90.dll[0x00002FDF]
--> backtrace[34] MSVCR90.dll[0x00003080]
--> backtrace[35] kernel32.dll[0x000159CD]
--> backtrace[36] ntdll.dll[0x0002A561]
--> [backtrace end]
-->
2018-02-27T10:04:48.535-06:00 info vpxd[21364] [Originator@6876 sub=vpxLro opID=737b587c] [VpxLRO] -- FINISH session[5275ca08-8730-a648-d890-a30f4b132ee0]52ac7eee-b72d-d3fd-d7df-63e7e6561cec
2018-02-27T10:04:48.535-06:00 info vpxd[21364] [Originator@6876 sub=Default opID=737b587c] [VpxLRO] -- ERROR session[5275ca08-8730-a648-d890-a30f4b132ee0]52ac7eee-b72d-d3fd-d7df-63e7e6561cec -- datastoreBrowser-datastore-358028 -- vim.host.DatastoreBrowser.searchSubFolders: vmodl.fault.HostCommunication:
--> Result:
--> (vmodl.fault.HostCommunication) {
--> faultCause = (vmodl.MethodFault) null,
--> msg = ""
--> }
--> Args:
-->
--> Arg datastorePath:
--> "[dev001]"
--> Arg searchSpec:
--> (vim.host.DatastoreBrowser.SearchSpec) {
--> query = (vim.host.DatastoreBrowser.Query) [
--> (vim.host.DatastoreBrowser.VmDiskQuery) {
--> filter = (vim.host.DatastoreBrowser.VmDiskQuery.Filter) null,
--> details = (vim.host.DatastoreBrowser.VmDiskQuery.Details) {
--> diskType = true,
--> capacityKb = true,
--> hardwareVersion = false,
--> controllerType = <unset>,
--> diskExtents = true,
--> thin = true
--> }
--> }
--> ],
--> details = (vim.host.DatastoreBrowser.FileInfo.Details) {
--> fileType = true,
--> fileSize = true,
--> modification = false,
--> fileOwner = false
--> },
--> searchCaseInsensitive = <unset>,
--> sortFoldersFirst = true,
--> }
2018-02-27T10:04:48.648-06:00 info vpxd[225564] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-725482-7766c6c] Not persisting vFlash resource runtime info. If you have hosts configured with vFlash resource, please set vpxd config option: vpxd.PersistVFlashRuntimeInfo to true.
Ok, seems there is a timeout on that first datastore, but I can see no obvious reason why that would happen on that specific datastore.
Does this datastore show high latency on the Performance tab?
Or are there that many folders & files on that datastore?
What I forgot to ask, is this FC, NFS or iSCSI storage?
Also strange the FileSystemVersion on either of the datastores. Could that be due to the PowerCLI version?
I'm using 6.5.4.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I just updated my PowerCLI. I was on 5 something. When I re-ran the previous commands for the filesystemversion I got 3.0.
They usage of the datastores, how busy they are, but I'm going to try and run it at an off time and see what happens.
I'll let you know how it goes.
The problem you are seeing definitely comes from the FolderPath.
To check further, can you run the attached script?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Ok, the issue seems to come from the fact that the FolderPath is not having a / at the end like I see in other environments.
I have no clue what could be causing this.
In the attached version I have adapted the code to intercept that.
Can you give this version a try?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi Lucd,
I think the issue had been solved after manipulating '/' inside FolderPath
The script results shows it find only 40 Orphaned Files
-Message
03/01/2018 13:58:54 Get-VmwOrphan 40 Found orphaned files on Local_DataStore_51.19!
I checked all the founded files and i find that the results are wonderful except that there is a folder called /scratch/log
here is an example :
Name : sdrsinjector.log
Folder : [Local_DataStore_51.19] scratch/log/sdrsinjector.log
Size : 144942
CapacityKB :
Modification : 2/28/2018 10:24:37 PM
Owner : root
Thin :
Extents :
DiskType :
HWVersion :
Is it a System Log File ?
My Second Question :
Can you modify the script to include also the File Deletion Option ?
Looks like someone configured the scratchspace of an ESXi node on that datastore.
The log can be useful, so I wouldn't delete it.
See also KB1033696
On the 2nd question I would suggest you open a new thread for that.
Point to the script in this thread and ask for the delete option.
This thread is becoming way too long :smileylaugh:
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Seems that no matter when I run the script it fails on the same couple data stores repeatedly. I'm at a loss at this point.
Since the script is using public API you could consider opening a SR.
But I'm not sure you can open a SR without some kind of Developer Support (or whatever they call it).
You could run the VOMA health check on the datastore, and see if anything is reported.
See KB2036767
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
These are all NFS datastores so VOMA isn't an option.
I see.
Are there any dropped packets on that link to the NFS export?
Is that a multi-NIC connection?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hello LucD,
We are running your script to find orphaned files in our LUNs and it works great! We were able to discover a lot of vmdk files and we verified that all of them were orphaned before deleting them, so there wasn't any "fake" orphaned vmdk. However, I have some questions...
- Is there any special reason why so many orphaned vmdks are generated (problems with vMotion, snaphosts...) ?
- Do you use any Powershell command to pipe the output of your script to delete the files from Powershell? We performed manually "vmkfstools -U file.vmdk" to remove orphaned vmdk files. But we need to do it manually, SSH to the ESXi host and execute the command. How would you do it in Powershell? Any command better than remove-item?
- Your script doesn't provide output for NFS Datastores... Is there any alternative in order to analyze NFS datastores for orphaned files?
Thank you in advance and thank you for providing this useful script!
Christian
Hi Christian,
On your questions:
if((($ds.Type -eq “VMFS”) -or ($ds.Type -eq “NFS”)) -and $ds.ExtensionData.Summary.MultipleHostAccess -and $ds.State -eq “Available”) {
I hope that helps.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference