Weird error on simple cmdlet

[fladjova]

Hello,

We updated our vcenter to 8u1c (previously in 7.0.3g) and using pwsh 7.3.6 and got this message on a simple task :

PS /Desktop/ > get-vm testvm | set-vm -Notes "test1" -Confirm:$false
Set-VM: 25/08/2023 12:09:15 Set-VM One or more errors occurred. (com.vmware.vapi.std.errors.internal_server_error {'messages': [com.vmware.vapi.std.localizable_message {'id': vapi.bindings.method.impl.unexpected, 'default_message': Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.InternalServerError, 'args': [com.vmware.vapi.std.errors.InternalServerError], 'params': , 'localized':}, com.vmware.vapi.std.localizable_message {'id': com.vmware.vapi.endpoint.badGateway, 'default_message': The endpoint, while acting as a gateway, received an invalid response from an upstream server it accessed while attempting to fulfill the request., 'args': [], 'params': , 'localized':}], 'data': , 'error_type': INTERNAL_SERVER_ERROR})

i check vcenter services and they are in healthy state. Any idea about this?

[LucD]

Did you already check the vpxd log if it has more information?

I'm using the same versions and it seems to be working ok for me.
I assume you are using the latest PowerCLI version.

[fladjova]

Set-vm is working as it changes the Notes field. In vxpd.log i can see info about the task but no error regarding the task. so i don't know why it throw this error, but that's not a real error as set-vm is working.

[LucD]

Do you perhaps have multiple connections open?
Check with $global:defaultVIServers and $global:defaultCisServers

[fladjova]

After exiting et reloading terminal,  no more error when i use only cmdlet.

But it is still present when i use it on a script (and then also when i use again only cmdlet until i close and reload terminal) i will have a better look on this script, to find why it threw this error even if the job is done

Thanks as always for your help LucD have a nice day

[plan-b]

I have the same errors when using New-VM in a script while executing the command in a prompt just works fine. Any ideas?

[robinmalik]

We've hit this too with Start-VM as part of our "New VM" deployment process.

Running vCenter 8.0 U2 (8.0.2.00000 (Build: 22385739) from 21 SEP 2023 with the latest versions of PowerCLI related modules at the time of writing.

We patched ESXi to 8.0.2, 22380479 a few days ago, and the VM deployments we've tried since then have all failed at the point of power on with:

PSMessageDetails :
Exception : VMware.VimAutomation.Sdk.Types.V1.ErrorHandling.VimException.ViError: 08/11/2023 16:49:02 Start-VM One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> vmware.vapi.std.errors.InternalServerError: com.vmware.vapi.std.errors.internal_server_error {'messages': [com.vmware.vapi.std.localizable_message {'id': vapi.bindings.method.impl.unexpected, 'default_message': Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.InternalServerError, 'args': [com.vmware.vapi.std.errors.InternalServerError], 'params': , 'localized':}, com.vmware.vapi.std.localizable_message {'id': com.vmware.vapi.endpoint.badGateway, 'default_message': The endpoint, while acting as a gateway, received an invalid response from an upstream server it accessed while attempting to fulfill the request., 'args': [], 'params': , 'localized':}], 'data': , 'error_type': INTERNAL_SERVER_ERROR}
at vmware.vapi.bindings.Stub.<InvokeMethodAsync>d__8`1.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at vmware.cis.TasksStub.List(FilterSpec filterSpec, GetSpec resultSpec, Action`1 _progress, InvocationConfig _invocationConfig)
at vmware.cis.TasksStub.List(FilterSpec filterSpec, GetSpec resultSpec)
at VMware.VimAutomation.ViCore.Impl.V1.Service.CisTaskServiceImpl.<GetTasks>d__11.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at VMware.VimAutomation.ViCore.Impl.V1.Task.CisTaskServiceProvider.GetNotNullTasks(VimClient client, CisTaskService service, IEnumerable`1 ids)
at VMware.VimAutomation.ViCore.Impl.V1.Task.CisTaskServiceProvider.<GetTasks>d__11.MoveNext()
at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
at VMware.VimAutomation.ViCore.Impl.V1.Task.CisTaskServiceProvider.GetTaskList(IEnumerable`1 uidList, IEnumerable`1 connectionUidList)
at VMware.VimAutomation.Sdk.Impl.V1.Task.CoreTaskServiceImpl.GetTaskList(IEnumerable`1 uidList, IEnumerable`1 connectionUidList)
at VMware.VimAutomation.ViCore.Impl.V1.Service.TaskServiceImpl.<GetTaskCore>d__8.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at VMware.VimAutomation.ViCore.Impl.V1.Service.TaskServiceImpl.GetTaskById(String uid, CmdletTaskInfoProvider cmdletTaskInfoProvider)
at VMware.VimAutomation.ViCore.Impl.V1.Service.VmServiceImpl.StartVM(VirtualMachineInterop vm, Boolean isRequestedAsync)
--- End of inner exception stack trace ---
at VMware.VimAutomation.ViCore.Impl.V1.Service.VmServiceImpl.StartVM(VirtualMachineInterop vm, Boolean isRequestedAsync)
at VMware.VimAutomation.ViCore.Util10Ps.BaseCmdlet.GenericServerCallHelper.PerformOperationOnObjects[T](List`1 objects, OperationHandler`1 operation, Boolean runAsynch)
TargetObject :
CategoryInfo : NotSpecified: (:) [Start-VM], ViError
FullyQualifiedErrorId : Client20_VMServiceImpl_StartVM_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.StartVM
ErrorDetails :
InvocationInfo : System.Management.Automation.InvocationInfo
ScriptStackTrace : at <ScriptBlock>, F:\test\WebAutomatedManagement\Service\powershell\virtual-server-platform\new-vm.ps1: line 1207
at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

Having just done some testing from the command line (subtly different to how we drive deployments) I can replicate this error. It also triggers the same "One or more errors occurred" for Stop-VM.

It could be PowerShell 5.1 related, as when I attempt the same Start/Stop commands against the same VM from PowerShell 7.3.9 it doesn't error. Just seeking someone locally for confirmation of the same experience.

[LucD]

Did you open an SR?

[robinmalik]

Not yet, it was only brought to my attention this morning. Doing a little more checking/confirming and then I'll raise one tomorrow.

Edit: Sadly, it was not that simple. I'm not able to replicate it working with PowerShell 7 now, erroring at the same point in deployment (so there must be more to it). I also forgot to say, it does start the VM, despite the error

[LucD]

Great!
When you do GSS might try to tell you that you need a Developer Support contract for PowerCLI.
That is not correct, point them to PowerCLI Support Breakdown (para 3 & 4)