mjha
Hot Shot
Hot Shot

Unable to Create layer 2 Extension in on-prem using vCloud Extender

I am trying to create a layer 2 extension in my on-prem VC and it is failing with error "MtaException: MTA-02"

I have configured L2 appliance using tenant-cx UI and added 2 IP in the pool

In Cloud side I have deployed an ESG and converted it into advance networking services. Added a routed network as sub-interface.

Basically followed all steps mentioned in blog: https://kiwicloud.ninja/2017/11/vcloud-director-extender-part-5-stretch-networking-l2vpn/#comment-17...

On checking mobility.log in my on-prem cx appliance, I am seeing below errors

2018-10-03 11:21:18.940 [CloudExt-WorkflowPool-Thread-4] DEBUG c.v.h.m.c.w.d.WorkflowStretchNetwork.handleError(92)-  - Error in WorkflowStretch : NetworkExtension [id=null, name=Cloud-New-Extn, sourceNetwork=NetworkIdentifier [id=dvportgroup-38, name=Mgmt-NW, type=PORT_GROUP], sourceEgressOptimization=[], destinationNetwork=VcdNetworkIdentifier [siteId=c01f97c2-3beb-4222-aac4-9a6636ffb307, siteName=vStellar Private Cloud, vdcId=d9e4ad92-9636-46b0-9a19-7e3bb8e82791, vdcName=Tenant1-VDC, id=406104db-64e0-47d5-a34a-c386c53c4776, name=Tenat1-Mgmt, type=ORG_VDC], destinationEgressOptimization=[], status=null, taskId=null, failedSubTask=null]

com.vmware.hybridcloud.mobility.cloudxt.adapter.mta.MtaException: MTA-02

        at com.vmware.hybridcloud.mobility.cloudxt.adapter.mta.MtaVcdAdapterProxyImpl.getL2vpnConfig(MtaVcdAdapterProxyImpl.java:90)

        at com.vmware.hybridcloud.mobility.cloudxt.adapter.VcdAdapter.getL2vpnConfig(VcdAdapter.java:127)

        at com.vmware.hybridcloud.mobility.cloudxt.adapter.VcdAdapter.populateEdge(VcdAdapter.java:382)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.activity.stretch.GenerateAndSaveL2vpnConfigurationActivity.getOrgVcdEdge(GenerateAndSaveL2vpnConfigurationActivity.java:96)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.activity.stretch.GenerateAndSaveL2vpnConfigurationActivity.execute(GenerateAndSaveL2vpnConfigurationActivity.java:138)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.activity.BaseActivity.executeActivity(BaseActivity.java:107)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.process.SequentialProcessingEngine.execActivities(SequentialProcessingEngine.java:76)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.process.SequentialProcessingEngine.execActivities(SequentialProcessingEngine.java:43)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.WorkflowManagerImpl.executeWorkflow(WorkflowManagerImpl.java:70)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.process.WorkflowRunnable.run(WorkflowRunnable.java:51)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

        at com.vmware.hybridcloud.mobility.cloudxt.workflow.process.WorkflowThread.run(WorkflowThread.java:20)

Caused by: com.vmware.hybridcloud.mobility.mta.exceptions.MTAdapterClientException: Exception occurred while fetching L2VpnConfig for edge with urn: cf08e281-912d-454d-8d0e-88da6d54bbd3

        at com.vmware.hybridcloud.mobility.mta.utils.MTAdapterUtils.handleAndThrowException(MTAdapterUtils.java:78)

        at com.vmware.hybridcloud.mobility.mta.vcd.VcdNetworkServiceClient.getL2VpnConfig(VcdNetworkServiceClient.java:309)

        at com.vmware.hybridcloud.mobility.mta.vcd.AdapterVcdClient.getL2VpnConfig(AdapterVcdClient.java:690)

        at com.vmware.hybridcloud.mobility.cloudxt.adapter.mta.MtaVcdAdapterProxyImpl.getL2vpnConfig(MtaVcdAdapterProxyImpl.java:83)

        ... 13 common frames omitted

Caused by: javax.ws.rs.ForbiddenException: This operation is denied.

        at com.vmware.hybridcloud.mobility.mta.vcd.networking.NsxClient.translateException(NsxClient.java:227)

        at com.vmware.hybridcloud.mobility.mta.vcd.networking.NsxClient.getResource(NsxClient.java:132)

        at com.vmware.hybridcloud.mobility.mta.vcd.networking.NsxClient.getResource(NsxClient.java:106)

        at com.vmware.hybridcloud.mobility.mta.vcd.VcdNetworkServiceClient.getL2VpnConfig(VcdNetworkServiceClient.java:305)

        ... 15 common frames omitted

Caused by: org.springframework.web.client.HttpClientErrorException: 403 Forbidden

        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:63)

        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)

        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)

        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)

        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531)

        at com.vmware.hybridcloud.mobility.mta.vcd.networking.NsxClient.getResource(NsxClient.java:129)

        ... 17 common frames omitted

No help available on google regarding this topic.

Looking for guidance on how to troubleshoot this issue.

Please consider marking this answer "correct" or "helpful" if you think your query have been answered correctly. Manish Jha | Operations Support Engineer | vCloud Air Operations vExpert 2015-17 | vExpert-NSX | vExpert-Cloud | VCAP6-DCV | VCP6-DCV | RHCE-7 Website : http://vstellar.com
0 Kudos
7 Replies
mjha
Hot Shot
Hot Shot

config-1.PNGconfig-2.PNGconfig-3.PNGerror-1.PNG

Here are some screenshot of config and the error i faced

Please consider marking this answer "correct" or "helpful" if you think your query have been answered correctly. Manish Jha | Operations Support Engineer | vCloud Air Operations vExpert 2015-17 | vExpert-NSX | vExpert-Cloud | VCAP6-DCV | VCP6-DCV | RHCE-7 Website : http://vstellar.com
0 Kudos
paluszekd
VMware Employee
VMware Employee

Manish, reading through the error state, I wonder if this is a permissions issue. Did you review all necessary permissions and add them via the REST API for the org admin? Do you see the standalone edge deploy or does it fail before it's even deployed?

mjha
Hot Shot
Hot Shot

@paluszekd looks like its permission issue. I checked the Org Admin role and found following rights are missing (No option to add from vCD UI)

<RightReference href="{url}/right/105191de-9e29-3495-a917-05fcb5ec1ad0" name="Organization vDC Gateway: View L2 VPN" type="application/vnd.vmware.admin.right+xml"/>

<RightReference href="{url}/right/eeb2b2a0-33a1-36d4-a121-6547ad992d59" name="Organization vDC Gateway: Configure L2 VPN" type="application/vnd.vmware.admin.right+xml"/>

<RightReference href="{url}/right/66b32e08-1eeb-37ac-9266-ffbd19b39dd8" name="Right: View" type="application/vnd.vmware.admin.right+xml"/>

<RightReference href="{url}/right/60be4106-1f9f-325c-8ff4-8bf2c6d9bc0a" name="Organization Network: Create or Delete"

type="application/vnd.vmware.admin.right+xml"/>

I am trying to use your script documented at https://www.paluszek.com/wp/2018/05/03/vcloud-director-extender-1-1-add-permissions-script-for-organ...

When I am trying to execute it via powercli or powershell I am getting error "Not connected to this vCloud endpoint, use 'Connect-CIServer' before running this script."

Here is what I am doing

PowerCLI E:\> Connect-CIServer mgmt-vcd-a.alex.local

Name                           User                           Org

----                           ----                           ---

mgmt-vcd-a.alex.local          admin                          System

PowerCLI E:\Scripts> .\vcdextender-perms.ps1

Not connected to this vCloud endpoint, use 'Connect-CIServer' before running this script.

PowerCLI E:\Scripts>

As you can see I connected to VCD in first step. Not sure why I am getting this error.

I am exploring API method to do so.

Will update this thread when I have some progress in this.

Please consider marking this answer "correct" or "helpful" if you think your query have been answered correctly. Manish Jha | Operations Support Engineer | vCloud Air Operations vExpert 2015-17 | vExpert-NSX | vExpert-Cloud | VCAP6-DCV | VCP6-DCV | RHCE-7 Website : http://vstellar.com
0 Kudos
mjha
Hot Shot
Hot Shot

Also forgot to mention that I dont see any standalone edge deploying in on-prem when configuring the layer 2 extension. It fails immediately

Please consider marking this answer "correct" or "helpful" if you think your query have been answered correctly. Manish Jha | Operations Support Engineer | vCloud Air Operations vExpert 2015-17 | vExpert-NSX | vExpert-Cloud | VCAP6-DCV | VCP6-DCV | RHCE-7 Website : http://vstellar.com
0 Kudos
paluszekd
VMware Employee
VMware Employee

What version of PowerCLI are you running? Very odd, I have not seen that before either.

0 Kudos
mjha
Hot Shot
Hot Shot

PowerCLI 6.5

Please consider marking this answer "correct" or "helpful" if you think your query have been answered correctly. Manish Jha | Operations Support Engineer | vCloud Air Operations vExpert 2015-17 | vExpert-NSX | vExpert-Cloud | VCAP6-DCV | VCP6-DCV | RHCE-7 Website : http://vstellar.com
0 Kudos
mjha
Hot Shot
Hot Shot

Tried adding rights to org vdc gateway and it failed. Here is what I tried to do

API Call

curl -sik -H "Accept:application/*+xml;version=30.0" -H "Content-Type:application/vnd.vmware.admin.right+xml" -H "x-vcloud-authorization:9f5c2e6e368c4e058ba333898022b2e1" -X PUT https://mgmt-vcd-a.alex.local/api/admin/org/18400ba5-f469-4508-9905-a88a2d9c8b83/right/f72af304-97b0... -d @vcdrights.xml

where contents of vcdrights.xml is as below

<OrgRights xmlns="http://www.vmware.com/vcloud/v1.5">

<RightReference href="https://mgmt-vcd-a.alex.local/api/admin/org/18400ba5-f469-4508-9905-a88a2d9c8b83/right/f72af304-97b0..." name="Organization vDC Gateway: View L2 VPN" type="application/vnd.vmware.admin.right+xml">

</OrgRights>

Output: HTTP/1.1 405 Method Not Allowed

What am I missing here?

Please consider marking this answer "correct" or "helpful" if you think your query have been answered correctly. Manish Jha | Operations Support Engineer | vCloud Air Operations vExpert 2015-17 | vExpert-NSX | vExpert-Cloud | VCAP6-DCV | VCP6-DCV | RHCE-7 Website : http://vstellar.com
0 Kudos