VMware Cloud Community
marv0
Contributor
Contributor

vcsa 6.5 trying to automate the backup - authentication failure

Hello Smiley Happy

I tried to make the Backup-VCSAToFile work in my development environment.

vSphere 6.5 – Automate VCSA Backup » Brian Graf's Virtualization Blog

I'm stuck here:

A server error occurred: 'com.vmware.vapi.std.errors.unauthorized': Unable to authorize user (Server error id:

'vapi.security.authorization.invalid'). Check $Error[0].Exception.ServerError for more details.

In Zeile:1 Zeichen:1

What did I do?

Windows 7 VM, installed VMware Powershell 6.5

Started ISE

loaded modules

Import-Module VMware.VimAutomation.Core

Import-Module VMware.VimAutomation.Vds

Import-Module VMware.VimAutomation.Cloud

Import-Module VMware.VimAutomation.PCloud

Import-Module VMware.VimAutomation.Cis.Core

Import-Module VMware.VimAutomation.Storage

Import-Module VMware.VimAutomation.HorizonView

Import-Module VMware.VimAutomation.HA

Import-Module VMware.VimAutomation.vROps

Import-Module VMware.VumAutomation

Import-Module VMware.DeployAutomation

Import-Module VMware.ImageBuilder

Import-Module VMware.VimAutomation.License

loaded the script itself as module (necessary?)

connected to vcsa with connect-viserver

Used given example in the script with my parameters.

Then I get a pop up

pastedImage_4.png

(Connection to CisServer)

I don't know what that is.:smileyconfused:

If I use my credentials for the vCenter connection - AD authentication, I get that error

Backup-VCSAToFile : A server error occurred: 'com.vmware.vapi.std.errors.unauthorized': Unable to authorize user (Server error id:

'vapi.security.authorization.invalid'). Check $Error[0].Exception.ServerError for more details.

In Zeile:1 Zeichen:1

Any hints?

33 Replies
LucD
Leadership
Leadership

The Connect-CisServer cmdlet connects to the API Service.

Can you try to connect with your SSO admin account (default is administrator@vsphere.local)?

Update:

You can find the internals of authenticating to the vSphere SDK Automation server in the vSphere Automation SDK for .NET Programming Guide

In PowerCLI this process is done for you behind the scenes by using the Connect-CisServer cmdlet.

From the Programming Guide:

"You connect to the vSphere Automation Endpoint by using a user name and password known to the

vCenter Single Sign-On service. The vSphere Automation uses your credentials to authenticate with the

vCenter Single Sign-On Service and obtain a SAML token."


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

marv0
Contributor
Contributor

Thank you,

looks like it works now. Tried it with the ssouser before, but did not use the @ssodomain.

The import of the script as module seems to be necessary, too. I know, you know that Smiley Wink. Just leaving the info here for the next part time scripter.

Have a nice day!

Martin

Reply
0 Kudos
MatthewPinkston
Contributor
Contributor

We're having a similar issue; however I have tried connecting to the CisServer before hand going through a piece at a time, and just providing the administrator@ssodomain creds when prompted. Neither work for me.

When I try and piece through the process one step at a time, when I run:

$BackupJob = $BackupAPI.create($CreateSpec)

I get:

Server session is not established.

At line:1 char:1

+ $BackupJob = $BackupAPI.create($CreateSpec)

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : OperationStopped: (:) [], CisException

    + FullyQualifiedErrorId : VMware.VimAutomation.Cis.Core.Types.V1.CisException

When I run the script and just provide the SSO creds I get many errors starting with this one (they all say Unable to authorize user):

A server error occurred: 'com.vmware.vapi.std.errors.unauthorized': Unable to authorize user (Server error id:

'vapi.security.authorization.invalid'). Check $Error[0].Exception.ServerError for more details.

At C:\scripts\Backup-VCSA.psm1:81 char:35

+                $BackupAPI.get("$($BackupJob.ID)") | select id, progr ...

+                                  ~~~~~~~~~~~~~

    + CategoryInfo          : OperationStopped: (:) [], CisServerException

    + FullyQualifiedErrorId : VMware.VimAutomation.Cis.Core.Types.V1.CisServerException



C:\scripts> $Error[0].Exception.ServerError | fl *

Help     : @{Documentation=The {@name Unauthorized} {@term error} indicates that the user is not authorized to perform

           the {@term operation}. <p> API requests may include a security context containing user credentials. For

           example, the user credentials could be a SAML token, a user name and password, or the session identifier

           for a previously established session. Invoking the {@term operation} may require that the user identified

           by those credentials has particular privileges on the {@term operation} or on one or more  resource

           identifiers passed to the {@term operation}. <p> Examples: <ul> <li>The {@term operation} requires that the

           user have one or more privileges on the {@term operation}, but the user identified by the credentials in

           the security context does not have the required privileges. </li> <li>The {@term operation} requires that

           the user have one or more privileges on a resource identifier passed to the {@term operation}, but the user

           identified by the credentials in the security context does not have the required privileges. </li> </ul>

           <p> <p> Counterexamples: <ul> <li>The SAML token in the request's security context has expired.  A {@link

           Unauthenticated} {@term error} would be used instead. </li> <li>The user name and password in the request's

           security context are invalid.  The {@link Unauthenticated} {@term error} would be used instead. </li>

           <li>The session identifier in the request's security context identifies a session that has expired.  The

           {@link Unauthenticated} {@term error} would be used instead. </li> </ul> <p> For security reasons, the

           {@link Error#data} {@term field} in this {@term error} is {@term unset}, and the {@link Error#messages}

           {@term field} in this {@term error} does not disclose why the user is not authorized to perform the {@term

           operation}.  For example the messages would not disclose which privilege the user did not have or which

           resource identifier the user did not have the required privilege to access.  The API documentation should

           indicate what privileges are required.; messages=; data=}

data     :

messages : {@{Help=; args=System.Collections.Generic.List`1[System.String]; default_message=Unable to authorize user;

           id=vapi.security.authorization.invalid}}

Connecting to the ViServer and CisServer completes without any errors. Any ideas what's going on here?

-Matt P.

LucD
Leadership
Leadership

Not really.

Can you find some more info in the vpxd logs?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
thedafa
Enthusiast
Enthusiast

Hi Matt.

Did you ever get anywhere with this error? I have the same issue in my env Smiley Sad

Reply
0 Kudos
MatthewPinkston
Contributor
Contributor

Sorry, I've been unable to stick with this ticket as more pressing matters came up.

Total speculation so far, but my gut reaction is this is probably a certificate issue. I have no evidence to support this theory, but with VMware, it's usually a good place to start.

I looked at it a little bit more today and delved a little deeper into the API. I used a Chrome extension called Postman to test the API.

For those new to this, I'll try and provide enough detail so you can reproduce:

Establish Session:

POST: https://vc/rest/com/vmware/cis/session

Authorization: Basic Auth

Username: administrator@vsphere.local

Password: <your password>

Check "Save helper data to request" (not sure if this is necessary)

Click Send

This will generate a SAML token and spit it back in a cookie -- it's not obvious in Postman, but the cookie is "vmware-api-session-id" and you can see the value in a json the POST returns.

Once you establish this authentication you can use the API documented here:

Online Documentation - vSphere Automation SDK for REST 6.5 - VMware {code}

When I use the "vcenter" or "cis" API it works fine. When I try anything under the "appliance" API it fails:

So these work:

GET: https://vc/rest/vcenter/vm

GET: https://vc/rest/vcenter/vm/vm-100

POST: https://vc/rest/com/vmware/cis/session?~action=get

These do not, and get that vapi.security.authorization.invalid error:

POST: https://vc/rest/appliance/recovery/backup/job

HEADER: Content-Type: application/json

HEADER: Accept: application/json

BODY: raw: JSON (application/json):

{ "piece":

      {

          "location_type":"FTP",

          "comment":"Automatic backup",

          "parts":["seat"],

          "location":"ftp://backup/vcsa-test",

          "location_user":"backup",

          "location_password":"yourpassword"

      }

}

GET: https://vc/rest/appliance/monitoring

The full error response in JSON is:

{

  "type": "com.vmware.vapi.std.errors.unauthorized",

  "value": {

    "messages": [

      {

        "args": [],

        "default_message": "Unable to authorize user",

        "id": "vapi.security.authorization.invalid"

      }

    ]

  }

}

I did not see anything in the vpxd logs. When monitoring them, they didn't even appear to have a response to my attempts.

I did however stumble across this in the /var/log/vmware/applmgmt/vapi.log and vami.log. The list is the attempted GET for monitoring, and the create is the attempted POST for the backup:

2017-04-04T16:08:11.094 [3000]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.monitoring, operation_id: list

2017-04-04T16:08:11.094 [3000]ERROR:vmware.appliance.extensions.authorization.authorization_sso:FindAllParentGroups Failed {[Errno 2] No such file or directory}

2017-04-04T16:08:11.094 [3000]ERROR:vmware.appliance.extensions.authorization.authorization_sso:FindAllParentGroups Failed {[Errno 2] No such file or directory}

2017-04-04T16:08:11.094 [3000]INFO:twisted:"127.0.0.1" - - [04/Apr/2017:16:08:10 +0000] "POST /api HTTP/1.1" 200 332 "-" "vAPI http client"

2017-04-04T16:10:50.094 [3000]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job, operation_id: create

2017-04-04T16:10:50.094 [3000]ERROR:vmware.appliance.extensions.authorization.authorization_sso:FindAllParentGroups Failed {[Errno 2] No such file or directory}

2017-04-04T16:10:50.094 [3000]ERROR:vmware.appliance.extensions.authorization.authorization_sso:FindAllParentGroups Failed {[Errno 2] No such file or directory}

2017-04-04T16:10:50.094 [3000]INFO:twisted:"127.0.0.1" - - [04/Apr/2017:16:10:49 +0000] "POST /api HTTP/1.1" 200 332 "-" "vAPI http client

No idea if this is related, as the timing didn't seem to work out. But I did find these entries in /var/log/vmware/vapi/endpoint interesting:

2017-04-04T15:32:00.500Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild cis-api-connections-builder

2017-04-04T15:32:00.633Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:16666/cls/

2017-04-04T15:32:00.633Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.cis.data.provider with protocol vapi.json.http at http://localhost:16666/cls/

2017-04-04T15:32:00.633Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Unable to find metadata endpoint in service Service with localization key cis.content-library.ServiceDescription and id 5e812b2e-01b8-49c7-9184-9e9782d8e86e.

2017-04-04T15:32:00.633Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:16666/cls/

2017-04-04T15:32:00.633Z | INFO  | state-manager1            | ApiConnectionsStateBuilder     | Cannot resolve protocol priorities between the following services. Will use the first one.

        First: 5e812b2e-01b8-49c7-9184-9e9782d8e86e\com.vmware.cis.cls.vapi at http://vc:80/cls/

        Second: 5e812b2e-01b8-49c7-9184-9e9782d8e86e\com.vmware.cis.cls.vapi at http://localhost:16666/cls/

2017-04-04T15:32:00.633Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:16666/cls/

2017-04-04T15:32:00.633Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.cdc.provider with protocol vapi.json.http at http://localhost:16666/cls/

2017-04-04T15:32:00.633Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Unable to find metadata endpoint in service Service with localization key cis.content-library.ServiceDescription and id 5e812b2e-01b8-49c7-9184-9e9782d8e86e.

2017-04-04T15:32:00.633Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:10080/invsvc/vapi

2017-04-04T15:32:00.633Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:8900/vmonapi

2017-04-04T15:32:00.633Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:9090/ds/vapi

2017-04-04T15:32:00.634Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Cannot find metadata source files/URLs in VAPI endpoint Service Endpoint of type com.vmware.vapi.endpoint with protocol vapi.json.http at http://vc:80/site/api

2017-04-04T15:32:00.634Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Unable to find metadata endpoint in service Service with localization key cis.vapi.endpoint.serviceDescriptionResourceKey and id e0cc58e8-7ce4-48f9-9426-61648da55b2d.

2017-04-04T15:32:00.634Z | INFO  | state-manager1            | ApiConnectionsCisUtil          | Unsupported source (metadata) type in metadata source entry cis.common.ep.localurl : http://localhost:12346/site/api

2017-04-04T15:32:00.634Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Cannot find metadata source definitions in VAPI endpoint Service Endpoint of type com.vmware.vapi.endpoint with protocol vapi.json.http at http://localhost:12346/site/api

2017-04-04T15:32:00.634Z | WARN  | state-manager1            | ApiConnectionsCisUtil          | Unable to find metadata endpoint in service Service with localization key cis.vapi.endpoint.serviceDescriptionResourceKey and id e0cc58e8-7ce4-48f9-9426-61648da55b2d.

2017-04-04T15:32:00.634Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vim-adapter-settings-builder

2017-04-04T15:32:00.709Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild vapi-vcenter-servlet-builder

2017-04-04T15:32:00.710Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild api-interfaces-builder

2017-04-04T15:32:00.727Z | INFO  | state-manager1            | DefaultStateManager            | Invoking rebuild metadata-sync-builder

MatthewPinkston
Contributor
Contributor

I think I found the file that is generating the errors in applmgmt/vapi.log. The file is:

/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authorization/authorization_sso.py

    def getGroups(self, user):

        """

        Return groups containing (directly or via nested groups) user.

        @type  user: str

        @param user: A user name.

        @rtype: str[]

        @return: Array of groups (empty array if user does not exist).

        """

        dom = self._systemDomain

        userId = Sso.PrincipalId()

        userId.name,userId.domain = utils.decomposePrincipal(user, dom)

        try:

            grpIds = self.groupcheck.groupCheckService.FindAllParentGroups(userId)

        except Sso.fault.InvalidPrincipalFault, e:

            # Ignore invalid users.

            logger.error("FindAllParentGroups Failed {%s}" % str(e))

            return []

        except vmodl.fault.InvalidArgument, e:

            # Ignore empty user names.

            logger.error("FindAllParentGroups Failed {%s}" % str(e))

            return []

        except Exception as e:

            logger.error("FindAllParentGroups Failed {%s}" % str(e))

            return []

        return [utils.generatePrincipal(g.name, g.domain, dom) for g in grpIds]

Haven't found the "FindAllParentGroups" method yet, but I could try throwing in some more logging to see what userId looks like when it's called.

Reply
0 Kudos
MatthewPinkston
Contributor
Contributor

Looks like restarting the appliance has resolved the issue at least for the time being.

I was investigating the status of various services and was seeing some odd messages such as:

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

and this was showing up when I ran 'service vmware-vapi-endpoint start':

Apr 04 17:38:36 vc.elsys.gtri.org vapi-endpoint[45161]: Unable to write to the configured log file: ${vapi_log_dir}/wrapper.log (No such file or directory)

                                                          Falling back to the default file in the current working directory: wrapper.log

Apr 04 17:38:36 vc.elsys.gtri.org vapi-endpoint[45161]: Unable to write to the default log file: wrapper.log (Permission denied)

                                                          Disabling log file.

I'm super confused on the service management now, as there is both 'service-control' and 'service' and they don't seem to jive.

How to stop, start, or restart vCenter Server 6.x services (2109881) | VMware KB

How to stop, start, or restart vCenter Server Appliance services (2054085) | VMware KB

root@vc [ ~ ]# service vmware-vapi-endpoint status

● vmware-vapi-endpoint.service - LSB: VMware vAPI Endpoint

   Loaded: loaded (/etc/rc.d/init.d/vmware-vapi-endpoint; bad; vendor preset: enabled)

   Active: inactive (dead)

     Docs: man:systemd-sysv-generator(8)

root@vc [ ~ ]# service-control --status

Running:

applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

Stopped:

vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha

Anyway, still not sure what caused it, but rebooting got it working again.

Reply
0 Kudos
multihawk
Contributor
Contributor

I'm having same issue but a reboot didn't change the status comes back as inactive (dead)

I stopped/started the service and it changed to "active  (running)" but I also get the following error:

  • vapi-endpoint[8457]: Unable to write to the configured log file: ${vapi_log_dir}/wrapper.log (No such file or directory)
    • Falling back to the default file in the current working directory: wrapper.log
  • Unable to write to the default log file: wrapper.log (Permission denied)
    • Disabling log file.

I was successfully able to run the backup script after restarting the service, though.

Thanks for your work tracking this down.

Anyone know why the service is in that state?

Reply
0 Kudos
DDinu
Enthusiast
Enthusiast

Am getting the same error, Restarted the service and rebooted the appliance no luck. Anything that could help fix this issue ?

GET: https://vc/rest/appliance/monitoring

The full error response in JSON is:

{

  "type": "com.vmware.vapi.std.errors.unauthorized",

  "value": {

    "messages": [

      {

        "args": [],

        "default_message": "Unable to authorize user",

        "id": "vapi.security.authorization.invalid"

      }

    ]

  }

}

Reply
0 Kudos
LucD
Leadership
Leadership

Did you check if there is sufficient free space on /storage/log?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
DDinu
Enthusiast
Enthusiast

Yes, have around 50GB of free space.

Reply
0 Kudos
LucD
Leadership
Leadership

Thanks, was just a guess, I saw that once when CIS was behaving strangely.

I would suggest to open a SR in any case, since the other methods in this thread don't seem to work for you.

And yes, PowerCLI is supported, see PowerCLI Support Breakdown


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
DDinu
Enthusiast
Enthusiast

Sure, Will open an SR.

Reply
0 Kudos
multihawk
Contributor
Contributor

I had this problem on another system as well. I had to manually start the service after rebooting the VM.  Didn't matter how many times I started/stopped it I couldn't logon until I rebooted it and then manually started the service.

Reply
0 Kudos
MarcelSwartjes
Contributor
Contributor

Same problem here, but it started right after updating to u1d, before that (u1c) everything was working fine. Have this problem on 3 from 4 VCSA 6.5.0 servers, all 3 migrated from windows, the only one without a problem is the one installed from scratch and then updated to u1d.
I opened an SR today.

Marcel
Reply
0 Kudos
stuckmcx
Contributor
Contributor

Hello all,

are there any Updates on you SRs?

I have the same problem on 1 of 3 VCSA.

Regards

Reply
0 Kudos
MarcelSwartjes
Contributor
Contributor

There is no progress on this SR yet.

I noticed that it does work calling the same API from the vCenter command console with the same credentials.
So when you start a ssh session to the vcsa appliance and run the following command, you get a backup:

Command> com.vmware.appliance.recovery.backup.job.create --backupPassword --locationType SCP --locationPassword --parts common --location fqdnofmybackupserver/vcsabackup/mybackup --locationUser vcsabackup

Enter backupPassword:

Reenter backupPassword:

Enter locationPassword:

Response:

   Messages:

     1:

         Backup job started.

   State: INPROGRESS

   Starttime: 2018-01-07T19:18:58.069Z

   Progress: 0

   Endtime: ''

   Id: 20180107-191858-7312210

When using the same API from Powershell with the same credentials it fails.

Marcel
Reply
0 Kudos
johnnymack
Contributor
Contributor

I had the same issue: VCSA 6.5

i was using administrator@vsphere.local to run both the connect to the viserver and CISserver.

A server error occurred: 'com.vmware.vapi.std.errors.unauthorized': Unable to authorize user (Server error id: 'vapi.security.authorization.invalid'). Check

$Error[0].Exception.ServerError for more details.

At D:\Scripts\vCsenter6-5backup\backupFunctions.ps1:75 char:13

+             throw $_.Exception.Message

+             ~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : OperationStopped: (A server error ...r more details.:String) [], RuntimeException

    + FullyQualifiedErrorId : A server error occurred: 'com.vmware.vapi.std.errors.unauthorized': Unable to authorize user (Server error id: 'vapi.security.

   authorization.invalid'). Check $Error[0].Exception.ServerError for more details.

in order to resolve the issue i logged on to the appliance via SSH, launched bash shell and restarted the service here and it started to work again!

service-control --stop applmgmt

service-control --start applmgmt

VMware Knowledge Base

Service Name

Description
applmgmtVMware Appliance Management Service