topic Re: NSX-T REPO SYNC FAILED in VMware NSX Discussions

NSX-T REPO SYNC FAILED

ggovek — Tue, 11 Oct 2022 07:23:54 GMT

Hi,

I have 3 NSX-T Managers and I have a repo sync problem on two of them. How to resolve this problem?

I try to reboot all three appliances and this is not help. Also I try with Resolve button.

Re: NSX-T REPO SYNC FAILED

virtualgeoff — Wed, 16 Nov 2022 06:10:23 GMT

I have this problem too.

Did you find a fix for this issue?

Re: NSX-T REPO SYNC FAILED

jonedved — Fri, 02 Jun 2023 16:45:21 GMT

Hi,

I have the same issue. Running 4.1.0.0 by the way.

Searching KB, internet - no hits whatsoever.

Does anyone know how to troubleshoot this issue? Which logs and services to check?

I had a look through bunch of them - but I can't get this to work. One strange thing I noticed - it seems NSX database has a fqdn saved in the IPv4 field of NSX Manager nodes under get nodes verbose:

UUID : xx
Label : Manager
IPv4 Addresses : ['nsx-manager3.lab.local']
IPv6 Addresses : N/A
Hostname/FQDN : nsx-manager3.lab.local
Display Name : nsx-manager3

UUID : xx
Label : Manager
IPv4 Addresses : ['nsx-manager2.lab.local']
IPv6 Addresses : N/A
Hostname/FQDN : nsx-manager2.lab.local
Display Name : nsx-manager2

UUID : xx
Label : Manager
IPv4 Addresses : ['nsx-manager1.lab.local']
IPv6 Addresses : N/A
Hostname/FQDN : nsx-manager1.lab.local
Display Name : nsx-manager1.lab.local

There is a reference to invalid IP in proton logs:

2023-06-02T13:51:36.516Z ERROR http-nio-127.0.0.1-7440-exec-3 IPv4Address 4407 - [nsx@6876 comp="nsx-manager" errorCode="MP260" level="ERROR" reqId="d385ced2-52e5-4be0-808e-049e646a00df" subcomp="manager" username="uproton"] Invalid ip string 'nsx-manager1.lab.local'. Error parsing 'nsx-manager1'.

Also - there is a reference to missing IP:

2023-06-02T15:09:44.833Z WARN http-nio-127.0.0.1-7440-exec-1 FacadeInterceptorHelperImpl 4452 - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="df7a2453-7ffd-45c1-addd-6b4644d994c0" subcomp="manager" username="UC"] API result did not pass validationField level validation errors: {required property results[3].controller_role.control_cluster_listen_addr.ip_address is missing}, {required property results[4].controller_role.control_cluster_listen_addr.ip_address is missing}

Re: NSX-T REPO SYNC FAILED

avaharagov — Fri, 09 Jun 2023 09:55:48 GMT

Good afternoon, encountered the same problem on NSX 4.1.0, could you please tell me if you were able to solve this problem?

Re: NSX-T REPO SYNC FAILED

mynameisalex — Tue, 13 Jun 2023 09:52:49 GMT

I have the same problem. Is there any solution to the problem?

Re: NSX-T REPO SYNC FAILED

jonedved — Tue, 13 Jun 2023 09:59:19 GMT

I got it fixed - worked on it together with VMware support.

It seems there are potentially two problems:

1./ FQDN in IPv4Address field on NSX Manager "get node" output. If this happens - I suspect RepoSync process can't work.

2./ RepoSync failing for another reason - this reason can be discovered in nsxapi.log.

Re: NSX-T REPO SYNC FAILED

avaharagov — Tue, 13 Jun 2023 12:44:15 GMT

There is no FQDN in the IPv4Address field. There are errors in nsxapi.log that clearly show that the repository is going to synchronize files that are not there

Re: NSX-T REPO SYNC FAILED

jonedved — Tue, 13 Jun 2023 13:11:33 GMT

The question is why?

The correct way to fix it would be engaging support.

I have manually uploaded the bundle to my repository server. This however is not supported - while I can't see anything wrong with it, support will probably not like it.

Example - RepoSync is complaining about missing files for "4.1.0.0.0.21332672" version:

I downloaded the 4.1.0.0 MUB (upgrade bundle), then extracted all the files with 7z (gunzip should work). Then copied the entire extracted package to my repository node /repository/4.1.0.0.0.21332672/ with scp.

Last step was to change owner on the files I copied - in this example: chown -R uuc:grepodir 4.1.0.0.0.21332672\

After that, issue a manual repo-sync with API and it should make progress.

Let us know if that helps.

Re: NSX-T REPO SYNC FAILED

tayyabnono — Mon, 25 Sep 2023 17:53:26 GMT

@jonedved

I'm facing the REPO_SYNC issue, and I have tried all the mentioned methods and now I don't know how to solve it currently I'm running version 4.1.0.2.0.21761695. I have ALB running on Manager.

Re: NSX-T REPO SYNC FAILED

vPauleAKQ — Wed, 18 Oct 2023 05:35:56 GMT

@tayyabnono Did you by any chance upgrade from NSX-T 3.x?

If so, we did the same, ran into massive problems, in the end had to replace all 3 manager nodes with fresh ones and ran into the REPO_SYNC problem.

Check /var/log/proton/nsxapi.log for FAILED repo sync messages. We got something like "Unable to connect to File /repository/4.1.1.0.0.22224312/Manager/vmware-mount/libvixMntapi.so.1 on source xxx.xxx.xxx.xxx".

The IP was the one of the appliance which was unable to sync the repo.

We then ran the nsxcli command "get service install-upgrade" and the output showed, that this service was running on the node with the missing files/folders.

I connected to another manager node where there were no REPO_SYNC problems, checked if the files/folders were there, which they were and ran the nsxcli command "set repository-ip" which changed the repository to that node.

After a minute everything from that repository was synced to the node that had the error and the error was gone.

Re: NSX-T REPO SYNC FAILED

tayyabnono — Sun, 29 Oct 2023 06:01:25 GMT

Good to hear that your issue got resolved. I upgraded from 3.x to 4.x and I tried all those things but sync was not working seems that some files were missing from the upgrade on one of the managers however, I extracted .mib and copy pasted manually on all three repos and issue got resolved.

Re: NSX-T REPO SYNC FAILED

bustmove-ls — Fri, 10 Nov 2023 09:54:00 GMT

Did you tried to delete other 2 nodes and reAdd new appliance?