Timeout in 5480 port prevents upgrading to v6.7

[bocabits]

Hello; 

we are having a pretty hard time upgrading our last 6.5 vcenter. We have an issue with the 5480 port. When logging to the appliance management or trying to update the server we always get a timeout error.

There is no firewall problem as far as I know and regarding vapi service: 

# service-control --status
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmcam vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-psc-client vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
vmware-updatemgr vmware-vcha

# systemctl status vmware-vapi-endpoint
● vmware-vapi-endpoint.service - LSB: VMware vAPI Endpoint
Loaded: loaded (/etc/rc.d/init.d/vmware-vapi-endpoint; bad; vendor preset: enabled)
Active: inactive (dead)
Docs: man:systemd-sysv-generator(8)
# systemctl start vmware-vapi-endpoint
Job for vmware-vapi-endpoint.service failed because the control process exited with error code. See "systemctl status vmware-vapi-endpoint.service" and "journalctl -xe" for details.

# systemctl status vmware-vapi-endpoint
● vmware-vapi-endpoint.service - LSB: VMware vAPI Endpoint
Loaded: loaded (/etc/rc.d/init.d/vmware-vapi-endpoint; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2023-03-24 10:17:03 UTC; 1min 48s ago
Docs: man:systemd-sysv-generator(8)
Process: 39522 ExecStart=/etc/rc.d/init.d/vmware-vapi-endpoint start (code=exited, status=1/FAILURE)

Mar 24 10:17:00 sxdvvc702.igrupobbva su[39595]: pam_unix(su:session): session opened for user vapiEndpoint by (uid=0)
Mar 24 10:17:00 sxdvvc702.igrupobbva vmware-vapi-endpoint[39522]: Starting vAPI Endpoint...
Mar 24 10:17:00 sxdvvc702.igrupobbva vapi-endpoint[39671]: Unable to write to the configured log file: ${vapi_log_dir}/wrapper.log (No...ctory)
Falling back to the default file in the current working directory: wrapper.log
Mar 24 10:17:00 sxdvvc702.igrupobbva vapi-endpoint[39671]: Unable to write to the default log file: wrapper.log (Permission denied)
Disabling log file.
Mar 24 10:17:03 sxdvvc702.igrupobbva vmware-vapi-endpoint[39522]: Waiting for vAPI Endpoint......
Mar 24 10:17:03 sxdvvc702.igrupobbva vmware-vapi-endpoint[39522]: WARNING: vAPI Endpoint may have failed to start.
Mar 24 10:17:03 sxdvvc702.igrupobbva systemd[1]: vmware-vapi-endpoint.service: Control process exited, code=exited status=1
Mar 24 10:17:03 sxdvvc702.igrupobbva systemd[1]: Failed to start LSB: VMware vAPI Endpoint.
Mar 24 10:17:03 sxdvvc702.igrupobbva systemd[1]: vmware-vapi-endpoint.service: Unit entered failed state.
Mar 24 10:17:03 sxdvvc702.igrupobbva systemd[1]: vmware-vapi-endpoint.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.

Any idea would be greatly appreciated. Thanks a lot!!

[bocabits]

I think the port is OK

# curl -v telnet://23.4.130.57:5480
* Rebuilt URL to: telnet://23.4.130.57:5480/
* Trying 23.4.130.57...
* TCP_NODELAY set
* Connected to 23.4.130.57 (23.4.130.57) port 5480 (#0)

[mannharry]

Are you able to login to 5480 vami port , if no share the screenshot of the error.

try this 

service-control --stop applmgmt
service-control --start applmgmt  

Regards

Harry 

[bocabits]

Hello, thanks for your help.

Do you mean this picture for the vami login?

About your tip

# service-control --status applmgmt
Running:
applmgmt
# service-control --stop applmgmt
Perform stop operation. vmon_profile=None, svc_names=['applmgmt'], include_coreossvcs=False, include_leafossvcs=False
Successfully stopped service applmgmt
# service-control --start applmgmt
Perform start operation. vmon_profile=None, svc_names=['applmgmt'], include_coreossvcs=False, include_leafossvcs=False
2023-03-24T16:38:28.890Z Service applmgmt state STOPPED
Successfully started service applmgmt
# service-control --status applmgmt
Running:
applmgmt

We still get the time out at FQDN:5480.

Thank you!

[mannharry]

Kindly login via SSH and provide the output of this 

cat /etc/applmgmt/appliance/software_update_state.conf

[bocabits]

It seems I don't have that file

# cat /etc/applmgmt/appliance/software_update_state.conf
cat: /etc/applmgmt/appliance/software_update_state.conf: No such file or directory

[mannharry]

Can we go to
cd /var/log/applmgmt
check vami.log , and see for any backtrace or error

[bocabits]

This file is also missed

# cd /var/log/applmgmt
-bash: cd: /var/log/applmgmt: No such file or directory

[mannharry]

 cd /var/log/vmware/applmgt is the location
check vami.log  and StatsMonitor-xxx.log

• Also check the below articles 

https://kb.vmware.com/s/article/67179?lang=en_US

https://kb.vmware.com/s/article/55640

[bocabits]

No, we don't have that file.

# cd /var/log/vmware/applmgt
-bash: cd: /var/log/vmware/applmgt: No such file or directory

I'll take a look to the KBs.

Thank you!

[mannharry]

my bad mate

below is the location

/var/log/vmware/applmgmt

[bocabits]

Hello, I've been checking the KBs

KB 67179 has failed.

KB 55640 as it needs a reboot of the server I have not been able to perform it. 

The log files

firewall-reload.log | grep vami

2022-11-17T09:35:24.321 [919]DEBUG:firewall-reload:Loading service 'vami.conf' firewall rules.
2022-11-17T09:35:24.321 [919]DEBUG:firewall-reload:Service 'vami.conf' firewall rules enabled=True.
2022-11-17T09:35:24.321 [919]DEBUG:firewall-reload:Adding inbound firewall rule for service 'vami.conf' {u'portoffset': 0, u'protocol': u'tcp', u'name': u'vamiport', u'direction': u'inbound', u'porttype': u'dst', u'port': u'5480'}.

vami.log

vami.log - Pastebin.com

StatsMonitor-xxx.log,  the majority of the file is this kind of lines. No vami or 5480 appearance.

2023-03-25T13:22:25.802Z error StatsMonitor[7F7374BED700] [Originator@6876 sub=LinuxStorageStatsProvider(93988582393168)] statvfs(/storage/invsvc) failed with errno 2
2023-03-25T13:32:25.842Z error StatsMonitor[7F7374BED700] [Originator@6876 sub=LinuxStorageStatsProvider(93988582393168)] statvfs(/storage/invsvc) failed with errno 2
2023-03-25T13:42:25.879Z error StatsMonitor[7F7374BED700] [Originator@6876 sub=LinuxStorageStatsProvider(93988582393168)] statvfs(/storage/invsvc) failed with errno 2
2023-03-25T13:52:25.917Z error StatsMonitor[7F7374BED700] [Originator@6876 sub=LinuxStorageStatsProvider(93988582393168)] statvfs(/storage/invsvc) failed with errno 2
2023-03-25T14:02:25.957Z error StatsMonitor[7F7374BED700] [Originator@6876 sub=LinuxStorageStatsProvider(93988582393168)] statvfs(/storage/invsvc) failed with errno 2
2023-03-25T14:12:26.026Z error StatsMonitor[7F7374BED700] [Originator@6876 sub=LinuxStorageStatsProvider(93988582393168)] statvfs(/storage/invsvc) failed with errno 2

Thanks! Have a nice weekend!

[mannharry]

Do provide 

• cat /etc/vmware/statsmonitor/statsMonitor.xml
• df -h

[bocabits]

Hello, this is the outcome of those commands.

# cat /etc/vmware/statsmonitor/statsMonitor.xml

Thanks again.

[mannharry]

• /dev/mapper/invsvc_vg-invsvc is not mounted.
• The df -h returns the partition list 
• To workaround this issue 
• Take a cold snapshot of VCs and PSCs and try below 

Workaround :

• remove the /storage/invsvc entries from the Statsmonitor xml file (/etc/vmware/statsmonitor/statsMonitor.xml).
• 
• Statsmonitor is looking for the storage/invsvc to populate the xml file.
• /dev/mapper/invsvc_vg-invsvc is not mounted.
• Statsmonitor couldn't find the filesystem and thus  prompted for the errors.
• Restart all the services

[bocabits]

Hello,

I have followed your post.

We get this message after starting the services:

Service-control failed. Error Failed to start vmon services.vmon-cli RC=2, stderr=Failed to start updatemgr services. Error: Service crashed while starting

root@sxdvvc702 [ /etc/vmware/statsmonitor ]# service-control --status
Running:
applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-updatemgr vmware-vcha

[ ~ ]# cd /etc/vmware/statsmonitor/[ /etc/vmware/statsmonitor ]# ls -slatota - Pastebin.com

We still get the timeout error.

[bocabits]

Sorry I have marked an answer a Solution. I made a mistake.

[mannharry]

Are you still seeing errors in the below file?

StatsMonitor-xxx.log

[mannharry]

Can you restart the appliances once as well , shutdown and Power on.

[bocabits]

There is a new Stats-Monitor.log

2023-03-27T12:31:42.671Z Section for VMware VirtualCenter, pid=52036, version=6. - Pastebin.com

I've restarted manually all services. 

Running:
applmgmt lwsmd pschealth vmafdd vmcad vmcam vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-psc-client vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
vmware-updatemgr vmware-vcha

I cannot restart the server right now, I have to ask for permission.

Thank you!