VMware Cloud Community
reall123
Contributor
Contributor

Envoy Sidecar Proxy repeatedly stops

Hello

After several logins in vCenter https://FQDN/ui service Envoy Sidecar Proxy stops. After this vCenter UI not available. Manualy reboot service (5480) solves the problem.

VMware vCenter Server 7.0.2.00200 17958471

How to solve this problem?

Reply
0 Kudos
9 Replies
StefanGM
Contributor
Contributor

Hello,

Same issue on the same vCenter version. I'm also interested in a solution.

Reply
0 Kudos
ifescgit
Contributor
Contributor

Having same problem here in this vcenter version. Someone get a start point to solve this issue?

I found some entries on the envoy log at /var/log/vmware/envoy directory.

 

2021-06-30T17:24:55.718Z critical envoy[26446] [Originator@6876 sub=backtrace] Caught Segmentation fault, suspect faulting address 0x0
2021-06-30T17:24:55.718Z critical envoy[26446] [Originator@6876 sub=backtrace] Backtrace (use tools/stack_decode.py to get line numbers):
2021-06-30T17:24:55.718Z critical envoy[26446] [Originator@6876 sub=backtrace] Envoy version: b900a6e3b5edcf7319a6000e0000bf86b80dc066/1.16.2/Clean/RELEASE/OpenSSL-v1.0.2
2021-06-30T17:24:55.719Z critical envoy[26446] [Originator@6876 sub=backtrace] #0: [0x50610a0009ba5]

 

 

Reply
0 Kudos
shruthiM21
Contributor
Contributor

Any solution found?

I see the same issue on VC 7.0.2

 

2021-07-06T18:00:58.337Z critical envoy[25710] [Originator@6876 sub=backtrace] #0: [0x5563902b2ba5]
2021-07-06T18:00:58.338Z critical envoy[25710] [Originator@6876 sub=backtrace] #1: [0x5563902a8768]
2021-07-06T18:00:58.338Z critical envoy[25710] [Originator@6876 sub=backtrace] #2: [0x5563903d74c6]
2021-07-06T18:00:58.338Z critical envoy[25710] [Originator@6876 sub=backtrace] #3: [0x5563903d9478]
2021-07-06T18:00:58.338Z critical envoy[25710] [Originator@6876 sub=backtrace] #4: [0x5563903c69b3]
2021-07-06T18:00:58.338Z critical envoy[25710] [Originator@6876 sub=backtrace] #5: [0x556390899512]
2021-07-06T18:00:58.338Z critical envoy[25710] [Originator@6876 sub=backtrace] #6: [0x556390899c9f]

Reply
0 Kudos
ifescgit
Contributor
Contributor

Hi everyone. Nothing til now…

I’m trying to change configs in the /etc/vmware-envoy/ directory without success.

Also I setup the parameter SERVICE_LOG_LEVEL in the config.cfg file to DEBUG but nothing new was logged.

I read in other forums that the “Segmentation fault, suspect faulting address 0x0” error may be a problem with Ulimit settings but I didn’t try to change this on vcenter server yet.

Reply
0 Kudos
shruthiM21
Contributor
Contributor

Has anyone reached out to VMware already on this? Do we know if there is any PR raised

Reply
0 Kudos
reall12
Contributor
Contributor

I installed version 7.0.1-17491101 a long time ago. Works correctly. No time to wait for vmware to respond.

Reply
0 Kudos
shruthiM21
Contributor
Contributor

We are already on 7.0.2.

Reply
0 Kudos
ifescgit
Contributor
Contributor

Unfortunately our support has expired. if Vmware resolve this issue would you please share with us? Thanks!

Reply
0 Kudos
palonso
VMware Employee
VMware Employee

Hello,

If vSphere Client is not being accessible and vmware-envoy service is stops even if you restart it

...and if you find entries similar to the following in the envoy-??.log files ( at /var/log/vmware/envoy)

2021-07-28T14:40:05.922Z critical envoy[24193] [Originator@6876 sub=backtrace] Caught Segmentation fault, suspect faulting address 0x0
2021-07-28T14:40:05.922Z critical envoy[24193] [Originator@6876 sub=backtrace] Backtrace (use tools/stack_decode.py to get line numbers):
2021-07-28T14:40:05.922Z critical envoy[24193] [Originator@6876 sub=backtrace] Envoy version: b970a6e3b5edcf7319a6270e0600bf86b87dc466/1.16.2/Clean/RELEASE/OpenSSL-v1.0.2
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #0: [0x55b567867ba5]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #1: [0x55b56785d768]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #2: [0x55b56798c4c6]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #3: [0x55b56798e478]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #4: [0x55b56797b9b3]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #5: [0x55b567e4e512]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #6: [0x55b567e4ec9f]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #7: [0x55b56796ac11]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #8: [0x55b567f24d45]
2021-07-28T14:40:05.923Z critical envoy[24193] [Originator@6876 sub=backtrace] #9: [0x7fa14cf33f87]

 

It is showing a known issue with the envoy service. It will be fixed in future releases of vSphere 7

 

As a workaround you can try to do the following (using the envoy.gz attached)

For known issues/articles which do not have a resolution, add workaround information in this section.
NOTE: take snapshots in offline state for all the nodes in linked mode before proceeding. In the case that you need to revert, you will need to revert to snapshots for ALL the nodes

1. Download the attached envoy.zip file and upload it on vCenter on /tmp directory

2. SSH affected vCenter, login as root, type shell 

3. Run the command below to unzip the uploaded file:
unzip envoy.zip

4. Stop vmware-envoy as below
service-control --stop envoy

5. Take a copy of the existing envoy
mv /usr/lib/vmware-envoy/envoy /usr/lib/vmware-envoy/envoy.bak

6. Replace with the downloaded (and extracted) file
cp /tmp/envoy /usr/lib/vmware-envoy/envoy

7. Assign permissions as below:
chmod 755 /usr/lib/vmware-envoy/envoy

8. Run the command below:
/usr/sbin/setcap 'cap_net_bind_service=+ep' /usr/lib/vmware-envoy/envoy

9. Start vmware-envoy as below:
service-control --start envoy
 
Hope this helps
 
Best regards
Pablo
Reply
0 Kudos