This is usually a symptom of SAML not working properly between the Connection Servers and Workspace ONE Access.

Here are a few things to try ( in order):

1. Check that time sync is ok between the WS1 Access connector and Connection Servers. If the time difference is too large then SAML will fail.
2. Check the SAML Authenticatior from the Horizon Console. Edit one of the Connection Servers and then Edit the SAML 2.0 Authenticator. When you get a Failed when you click OK, then the Connection Server cannot communicate with the WS1 Access Connector.
3. Check in the WS1 Access console to see the status of the virtual apps collection

Thanks.

1. time is same on both server
2. added the same meta data url from virtual app tenant. Try to edit ,no error , it showing enabled
3. no issue observed .its syncing if i add new pools\entitlement

How to check the logs.

Check the Connection Server logs in C:\ProgramData\VMware\VDM\logs.

Look a the latest debug log and search for messages related to SAML.

On the WS1 Connector - C:\VMware\VMwareIdentityManager\Connector\opt\vmware\horizon\workspace\logs. Check connector.log

Has this ever worked or did this just stop working?

1)

please see the attched screen shot with saml error.

2)

no folder fiind.but see one in program files\workspace one access\virtual apps\logs -no error logs observed.

3)it is new implementation.

That would look like the SAML Authenticator is not defined properly on the Horizon Connection Server(s).

Looks like the error described in https://kb.vmware.com/s/article/2053290 and this page https://vdr.one/vmware-identity-manager-and-horizon-7-saml-expects-credentials-from-another-server/

When entering the SAML 2.0 Authenticator the only fields that need to be filled in are the Label and the Metadata URL. The Metadata URL should follow the format https://my.domain.com/SAAS/API/1.0/GET/metadata/idp.xml

Replace my.domain.com with the FQDN you set for WS1 Access.

Same issue here with no idea what will resolve this.

So far we have Rebooted UAG, rebooted connection server, checked time sync between UAG and connection server without any difference in behavior. 

No changes in our environment to cause this.

Copied meta data url directly from saas workspace console only.  Its also matches syntax shared by you. Tested on-prem version in the same infrastructure works fine. Seems some saas  certificate missing.While i add saas meta data url to connection server console i need to manually trust the certifcate .Do we need to open any outbound port from connection server?

That sounds like a certificate mis-match.  Specifically if in your on-prem version you might have installed an initial self-signed cert which is good enough for POC, but when you move to production, I'm guessing that original self-signed cert was not replaced with a proper cert - that's why you have to manually accept that cert - but it won't work on the SaaS side since services are looking for a valid cert from a CA.  It's a more common problem when migrating a POC into a prod environment because 'it works' so we often forget to make that compliance check. 

We had same issues in our environment, the steps you mentioned fixed the issue. Thanks.