NSX-T 3.0.1 Policy UI not working due to error "Failed to fetch Inventory details" - must use Manager UI

[Bill_Oyler]

Hello,

After recently upgrading from NSX-T 3.0 to 3.0.1, I can no longer use the "Policy" UI mode for most panels.  I receive the following error:

Failed to fetch Inventory details. Please contact the administrator. Error: Index out of sync, please resync via 'start search resync policy'.

When logging in via CLI and issuing the command "start search resync policy", nothing happens.  Additionally, I tried rebooting all NSX-T Manager nodes and NSX-T Edge nodes, and the problem persists. 

Various UI elements are broken in the "Policy" (default) view, such as viewing Segments, Tier-0 Gateways, and Tier-1 Gateways.  Error messages in the Network Overview panel looks like this:

Failed to get "NETWORKING" report - Index out of sync, please resync via start search resync policy. [].

Failed to get "IP ADDRESS MANAGEMENT" report - Index out of sync, please resync via start search resync policy. [].

Failed to get "NETWORK SERVICES" report - Index out of sync, please resync via start search resync policy. [].

Note that the Policy UI worked great in v3.0.0.  It broke after upgrading to v3.0.1.

Also note that previous upgrades to this system include 2.4.1 -> 2.5.0, 2.5.0 -> 2.5.1, and 2.5.1 -> 3.0.0, all of which were successful and quite quick.  The 3.0.0 -> 3.0.1 upgrade was bumpy, as I needed to manually update VIBs on all ESXi hosts to the 3.0.1 bits.  After doing that, the NSX-T Manager upgrade from 3.0.0 to 3.0.1 went well.

Any ideas for troubleshooting the broken Policy UI view?

Thanks,

Bill

[nand007 · Accepted solution]

Hi Bill,

What is the log message you are seeing in the policy.log? Is ti similar to the following:

exec-1 AuditingServiceImpl - - [nsx@6876 audit="true" comp="nsx-manager" level="INFO" reqId="ac44f659a5-5611-46ae-9ace-ae80e161b703c" subcomp="policy"] UserName="admin", ModuleName="NsxOverview", Operation="ReadNetworkingOverview", Operation status="failure", New value=[{"filter":"tier1_per_tier0s"}]

INFO http-nio-127.0.0.1-6440-exec-1 NsxBaseRestController - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="policy"] Error in API /policy/api/v1/ui-controller/networking-overview?filter=tier1_per_tier0s caused by exception com.vmware.nsx.management.common.exceptions.InvalidArgumentException: {"moduleName":"Policy","errorCode":513019,"errorMessage":"Error retrieving top 10 Tier-0 Gateways based on Tier-1 gateway connections. Index out of sync, please resync via 'start search resync policy'.."}

Can you try the following workaround: Set search.shoudBlockApiOnIndexingFailures to false in /etc/search/conf/policy.properties

[Lalegre]

I would suggest to open a ticket with VMware Support in this case because there were a lot of people having the same issue as you an nobody being able to solve it. To be honest I did not face this issue but a lot of thing break after I did the same upgrade as you and as I have a LAB I chose to simply re-install it.

[Bill_Oyler]

Thanks - that is indeed the solution that worked for me, after opening an SR with VMware Support.  Specifically, this is what worked for me:

1. Access the three NSX-T managers with the root user.  (Use VM Console to log in as root.)
   2. Navigate to and edit the file: /etc/search/conf/policy.properties
   3. Replace: "search.shoudBlockApiOnIndexingFailures=true" with "search.shoudBlockApiOnIndexingFailures=false".

VMware Support confirmed this was an issue and should be fixed in NSX-T 3.0.2.  But, the work-around worked for me, and 3.0.1 is working just fine.

Bill