VMware Networking Community
ggovek
Enthusiast
Enthusiast

NSX-T REPO SYNC FAILED

Hi,

I have 3 NSX-T Managers and I have a  repo sync problem on two of them. How to resolve this problem?

ggovek_0-1665472935122.png

 

I try to reboot all three appliances and this is not help. Also I try with Resolve button.

Labels (1)
13 Replies
virtualgeoff
Contributor
Contributor

I have this problem too.

Did you find a fix for this issue?

Reply
0 Kudos
jonedved
Enthusiast
Enthusiast

Hi,

I have the same issue. Running 4.1.0.0 by the way.

Searching KB, internet - no hits whatsoever. 

Does anyone know how to troubleshoot this issue? Which logs and services to check? 

 

I had a look through bunch of them - but I can't get this to work. One strange thing I noticed - it seems NSX database has a fqdn saved in the IPv4 field of NSX Manager nodes under get nodes verbose:

UUID : xx
Label : Manager
IPv4 Addresses : ['nsx-manager3.lab.local']
IPv6 Addresses : N/A
Hostname/FQDN : nsx-manager3.lab.local
Display Name : nsx-manager3

UUID : xx
Label : Manager
IPv4 Addresses : ['nsx-manager2.lab.local']
IPv6 Addresses : N/A
Hostname/FQDN : nsx-manager2.lab.local
Display Name : nsx-manager2

UUID : xx
Label : Manager
IPv4 Addresses : ['nsx-manager1.lab.local']
IPv6 Addresses : N/A
Hostname/FQDN : nsx-manager1.lab.local
Display Name : nsx-manager1.lab.local

 

There is a reference to invalid IP in proton logs:

2023-06-02T13:51:36.516Z ERROR http-nio-127.0.0.1-7440-exec-3 IPv4Address 4407 - [nsx@6876 comp="nsx-manager" errorCode="MP260" level="ERROR" reqId="d385ced2-52e5-4be0-808e-049e646a00df" subcomp="manager" username="uproton"] Invalid ip string 'nsx-manager1.lab.local'. Error parsing 'nsx-manager1'.

Also - there is a reference to missing IP: 

2023-06-02T15:09:44.833Z WARN http-nio-127.0.0.1-7440-exec-1 FacadeInterceptorHelperImpl 4452 - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="df7a2453-7ffd-45c1-addd-6b4644d994c0" subcomp="manager" username="UC"] API result did not pass validationField level validation errors: {required property results[3].controller_role.control_cluster_listen_addr.ip_address is missing}, {required property results[4].controller_role.control_cluster_listen_addr.ip_address is missing}

 

Reply
0 Kudos
avaharagov
Contributor
Contributor

Good afternoon, encountered the same problem on NSX 4.1.0, could you please tell me if you were able to solve this problem?

Reply
0 Kudos
mynameisalex
Contributor
Contributor

I have the same problem. Is there any solution to the problem?

Reply
0 Kudos
jonedved
Enthusiast
Enthusiast

I got it fixed - worked on it together with VMware support.

It seems there are potentially two problems:

1./ FQDN in IPv4Address field on NSX Manager "get node" output. If this happens - I suspect RepoSync process can't work.

2./ RepoSync failing for another reason - this reason can be discovered in nsxapi.log.

Login to NSX Manager CLI as admin user - then check that you have FQDN in IPv4Address field using "get node" command?

Reply
0 Kudos
avaharagov
Contributor
Contributor

There is no FQDN in the IPv4Address field. There are errors in nsxapi.log that clearly show that the repository is going to synchronize files that are not there

Reply
0 Kudos
jonedved
Enthusiast
Enthusiast

The question is why? 

The correct way to fix it would be engaging support. 

I have manually uploaded the bundle to my repository server. This however is not supported - while I can't see anything wrong with it, support will probably not like it. 

Example - RepoSync is complaining about missing files for "4.1.0.0.0.21332672" version:

I downloaded the 4.1.0.0 MUB (upgrade bundle), then extracted all the files with 7z (gunzip should work). Then copied the entire extracted package to my repository node /repository/4.1.0.0.0.21332672/ with scp.

Last step was to change owner on the files I copied - in this example: chown -R uuc:grepodir 4.1.0.0.0.21332672\

After that, issue a manual repo-sync with API and it should make progress. 

Let us know if that helps.

Reply
0 Kudos
tayyabnono
Contributor
Contributor

@jonedved 

I'm facing the REPO_SYNC issue, and I have tried all the mentioned methods and now I don't know how to solve it currently I'm running version  4.1.0.2.0.21761695. I have ALB running on Manager. 

 
 
Reply
0 Kudos
vPauleAKQ
Enthusiast
Enthusiast

@tayyabnono Did you by any chance upgrade from NSX-T 3.x?

If so, we did the same, ran into massive problems, in the end had to replace all 3 manager nodes with fresh ones and ran into the REPO_SYNC problem.

Check /var/log/proton/nsxapi.log for FAILED repo sync messages. We got something like "Unable to connect to File /repository/4.1.1.0.0.22224312/Manager/vmware-mount/libvixMntapi.so.1 on source xxx.xxx.xxx.xxx".

The IP was the one of the appliance which was unable to sync the repo.

We then ran the nsxcli command "get service install-upgrade" and the output showed, that this service was running on the node with the missing files/folders.

I connected to another manager node where there were no REPO_SYNC problems, checked if the files/folders were there, which they were and ran the nsxcli command "set repository-ip" which changed the repository to that node.

After a minute everything from that repository was synced to the node that had the error and the error was gone.

tayyabnono
Contributor
Contributor

Good to hear that your issue got resolved. I upgraded from 3.x to 4.x and I tried all those things but sync was not working seems that some files were missing from the upgrade on one of the managers however, I extracted .mib and copy pasted manually on all three repos and issue got resolved. 

Reply
0 Kudos
bustmove-ls
Contributor
Contributor

Did you tried to delete other 2 nodes and reAdd new appliance?

Reply
0 Kudos
liviumanole
Contributor
Contributor

Hello,

I am facing this message when i am trying to get de logs 

"-bash: /var/log/proton/nsxapi.log: Permission denied"

ANyone has any ideea what this permission is? Who is supposed to give it to me keeping in mind that I deployed the whole solution?

 

Thank you,

Liviu

Reply
0 Kudos
eculbertsoniv
Contributor
Contributor


@vPauleAKQ wrote:

@tayyabnono Did you by any chance upgrade from NSX-T 3.x?

If so, we did the same, ran into massive problems, in the end had to replace all 3 manager nodes with fresh ones and ran into the REPO_SYNC problem.

Check /var/log/proton/nsxapi.log for FAILED repo sync messages. We got something like "Unable to connect to File /repository/4.1.1.0.0.22224312/Manager/vmware-mount/libvixMntapi.so.1 on source xxx.xxx.xxx.xxx".

The IP was the one of the appliance which was unable to sync the repo.

We then ran the nsxcli command "get service install-upgrade" and the output showed, that this service was running on the node with the missing files/folders.

I connected to another manager node where there were no REPO_SYNC problems, checked if the files/folders were there, which they were and ran the nsxcli command "set repository-ip" which changed the repository to that node.

After a minute everything from that repository was synced to the node that had the error and the error was gone.


This worked!!  Simplest solution is the best!