Datastore search does not return all files

[Peter1230]

vSphere Server 4.1.0, SDK 4.1.0-254719

I'm running a Datastore search using SearchDatastoreSubFolders with a SeachSpec using a VmConfigFileQuery. For a given Datastore containing 9 virtual machines, only 1 .vmx is returned by that search. All 9 machines have all their files in their own folders, and are using the same config file version.

The relevant code sections:

my $filequery = FileQueryFlags->new(
    fileOwner    => 1,
    fileSize     => 1,
    fileType     => 1,
    modification => 1
);
my $vmxquery = VmConfigFileQuery->new(
    details => VmConfigFileQueryFlags->new(
        configVersion=>1,
    )
);
my $searchspec = HostDatastoreBrowserSearchSpec->new(
    details      => $filequery,
    query        => [ '*.vmx*' ],
);
my $results = $browser->SearchDatastoreSubFolders(
    datastorePath => '[datastore]',
    searchSpec    => $searchspec
);

Any pointers on what I could do different are much appreciated.

[stumpr]

A couple of things to note, the SearchDatastoreSubFolders is a task call, so you'll need to query the info.result property.  Also, I think you have some of your arguments to the call jumbled (query should be matchPattern in your example).

I attached a quick script.  Feel free to adapt it and expand on it, I didn't put too much effort into error handling or efficiency.

Sample run:

$ perl ~/Documents/Code/VMware/ListVmxFilesInDatastore.pl --username=administrator password=* --server=172.16.254.10 --dsname=DS01

[DS01] Win7/Win7.vmx (configVersion=8, size=2422B)

[DS01] VIBase/VIBase.vmx (configVersion=8, size=2660B)

[DS01] testoldhdw/testoldhdw.vmx (configVersion=8, size=970B)

[DS01] vlab-01/vlab-01.vmx (configVersion=8, size=2590B)

I also think the search is redundant with the string match and the VmConfigFileQuery, it will return the same files.  You could combine the two if you were looking to find a specific VMX file (e.g., use a string match of 'vlab*' to only match vlab-01.vmx).

[Peter1230]

SearchDatastoreSubFolders_Task is the Task variant (asynchronous call). But I'm calling SearchDatastoreSubFolders (the synchronous version).

Aside from that, I've tried your script, and it also returns only 1 config file, instead of the expected 9. What's interesting, the vSphere client shows 9/9 VMs powered on on that datastore, yet a search in the Datastore Browser also returns only 1 VM... And on other Datastores I see a similar behaviour, that machines are "missing", even though running.

[lamw]

The reason the script from Rueben is not returning all file is due to the use of the "VMConfigFileQuery" which specifies the type of file to search for, in this case it has a flag set for VM's configuration fle (e.g. .vmx) - http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.host.DatastoreBrowser.Se...

If you want to show ALL files under a given datastore, just remove that restriction. Here is another script that does pretty much the same thing as Rueben's, but probably with less error checking   - http://communities.vmware.com/docs/DOC-10788

[stumpr]

William, I'm pretty sure I have BadlyWrittenPerlScripts(tm) trademarked

I modified my script to use the async version, same results on my end.

I'm not sure why you're only getting one result then, what permissions are you using to run the script?  You can use the script William linked to list out all files or modify one of the scripts to remove the matchPattern and query parameters.

I just modified my script and ran it, I can see all the files on my datastore, including the vmx files.

Post up some code and try to run everything as a full admin. And to just be clear, are all the VMX files on the same Datastore?  Do the VMs you're expecting to see actually reside on multiple datastores?

[Peter1230]

Did a few more test, and concluded that it's not the error of the script, but a problem with the way vSphere handles multiple ESX hosts and datastores.

Apparently, when querying the VirtualCenter server for a certain datastore, it only queries the virtual machines on the first ESX it finds, and completely ignores the rest. Same if you query one ESX directly, it returns only those machines for a datastore that are running on this host.

So for my goal to query all config files on a given datastore, I'll probably have to get a list of connected hosts, and then query each of them for the files they see on that DS....

[stumpr]

I'm not sure I follow.

In my environment my virtual machines are running on different ESX hosts.  Even if you run the script against a host directly, you should get the same results, the contents of the Datastore.  It shouldn't have anything to do with the virtual machines running on a particular host or their power state. 

Could you explain what you're seeing in more detail?

[Peter1230]

Setup is this: multiple datastores connected to multiple ESX hosts, with each host able to see and use all datastores. Apparently, each host can only report on the files it has locked himself, and not those in use by other hosts (no surprise there really). But it seems that the VirtualCenter also only reports the details received from the first ESX host it queried, and ignores the others (check datastore for connected hosts, get first host in list, query that host for the information, exit)

[stumpr]

It's not like the browser is tryign to read file blocks, so the locks really shouldn't matter.

Are all your hosts in the same cluster?

Your assumption of how it goes through a single host from the list of hosts using the datastore is what I believe happens as well, I'll try to find out if that's 100% valid.  But I can't seem to reproduce the behavior you're seeing.  I can see all files on a datastore from any host (regardless of powers state and locks of the vms).

[Peter1230]

Sorry for the late reply, Life happened.

Attached is the complete script. When run against the VC it only returns the files for the specified DS as seen from 1 of the hosts, instead of all attached. Maybe it's just a coding error (quite probable since it's my first script using the vSphere Web API).

[stumpr]

That's as expected behavior.  However, you shouldn't see different views of the datastores from different hosts.  I think that's where I'm confused.  I'm not sure how you're getting different results from each of the hosts connected to the same shared Datastore.

Conceptually, the Datastore doesn't belong to the Host, it belongs to the Datacenter (you'll see it's parent object is a Datacenter).  I would probably check them as separate entities from the Hosts.

Also, in your code you can combine the FileQuery types and make your script more efficient -

my $FileQueryFlags = FileQueryFlags->new(

   fileOwner          => 1,

   fileSize          => 1,

   fileType          => 1,

   modification     => 1

);

my $VmConfigFileQuery = VmConfigFileQuery->new(

   details => VmConfigFileQueryFlags->new( configVersion => 1 )

);

my $VmDiskFileQueryFlags = VmDiskFileQueryFlags->new(

   capacityKb      => 1,

   controllerType  => 1,

   diskExtents     => 1,

   diskType        => 1,

   hardwareVersion => 1,

   thin            => 1,

);

my $VmDiskFileQuery = VmDiskFileQuery->new( details => $VmDiskFileQueryFlags );

my $VmSnapshotFileQuery = VmSnapshotFileQuery->new();

my $searchspec = HostDatastoreBrowserSearchSpec->new(

   details               => $FileQueryFlags,

   searchCaseInsensitive => 1,

   query => [ $VmConfigFileQuery, $VmDiskFileQuery, $VmSnapshotFileQuery ],

);

Though to get the VMXF file you'll probably have to regex filtering since I don't think the VMXF file falls into one of the query types.

[MR-Z]

Great stuff! I was not looking at the same issue, but this line clarified my confusion about how to use the "HostDatastoreBrowserSearchSpec":

query => [ $VmConfigFileQuery, $VmDiskFileQuery, $VmSnapshotFileQuery ],

I've been banging my head against the wall trying to figure out how to extract the disk backing info, thinking that the 'FileQueryFlags' itself is sufficient.

[stumpr]

The file queries are a nice way to simplify the search vs regex.  However, I just saw what I think is a bug for one customer where the VmConfigFileQuery type wasn't returning all VMX files.  Might be something to keep an eye on in case you have to fall back on regex (*.vmx)