Previously working VM - 'A general system error occurred: vix error codes = (1, 0)' is getting displayed now

[sachingaikwad]

Invoke-VMScript has stopped working from past couple days. The same VM has been working from past few months. But now when I run any Invoke-VMScript command, Invoke-VMScript: 6/24/2020 1:21:39 PM  Invoke-VMScript  A general system error occurred: vix error codes = (1, 0). Error is getting displayed

Sample command:

$Command_text="dir C:\users\"

Invoke-VMScript -VM $TARGET_VM -ScriptText $Command_text -GuestUser $vm_user -GuestPassword '&rWn.

Based on the previous posts, I tried following things:

• Made sure that correct password is used
• Passed the password in single quote as password contains special char

Here is the vmware.log excerpt:

2020-06-24T13:51:29.004Z| vcpu-1| I125: Guest: DXUM_devapi: 2020-06-24T09:51:29.0092| Thread ID: 4524 |Application requested vm3ddevapi version 2

2020-06-24T13:51:29.022Z| vcpu-0| I125: TOOLS autoupgrade protocol version 2

2020-06-24T13:51:29.022Z| vcpu-0| I125: Vix: [mainDispatch.c:4156]: VMAutomationReportPowerStateChange: Reporting power state change (opcode=2, err=0).

2020-06-24T13:51:29.022Z| vcpu-0| I125: TOOLS Received tools.set.version rpc call, version = 10304, setting type to 1 from guest OS

2020-06-24T13:51:29.022Z| vcpu-0| I125: Tools_SetVersionAndType did nothing; new tools version (10304) and type (1) match old Tools version and type

2020-06-24T13:53:50.874Z| vmx| I125: VigorTransportProcessClientPayload: opID=f0c9f8f-2e-0c76 seq=29344: Receiving GuestOps.CreateTemporaryFile request.

2020-06-24T13:53:51.023Z| vcpu-1| I125: VigorTransport_ServerSendResponse opID=f0c9f8f-2e-0c76 seq=29344: Completed GuestOps request with messages.

Please suggest.

[LucD]

I don't think that VMware Tools from a snapshot cause any issues.
And if they did, the reinstall should have fixed that issue.

That logins don't work is understandable and is probably due to the AD secure channel being outdated/expired.

This connection is updated/refreshed automatically on an interval.

You should be able to reset the secure channel between the station and the AD DCs with the netdom command.

One step I would try at this point, would be to start ProcessMonitor from Sysinternals in the Guest OS.

Then do the Invoke-VMScript(Plus) call and check if the monitor shows any obvious errors.

[sachingaikwad]

LucD​ ProcessMonitor is showing lot of processes, could you please let us know which Process and Operation should I be looking for. For now, I filtered  vmtoolsd.exe and there are lot of "Name not found' errors. Please suggest.

[LucD]

If you use the Verbose switch on the Invoke-VMScriptPlus call, you should see the name of the folder and files the function is trying to create.

Then search in the ProcessMonitor output if that path (or parts of it) appear.

[sachingaikwad]

I turned on the Verbose switch, 'Created temp folder in guest OS' message is displayed but the folder name is not displayed. Below is the screenshot for the same:

[LucD]

Ok, the Verbose message should have listed the name of the directory (in $tempFolder).

Write-Verbose "Created temp folder in guest OS $tempFolder"

But nothing is shown, meaning that $tempFolder is empty.

So now we are sure that the issue is with the creation of the temporary folder.

You should have a look in the ProcessMonitor output for a CreateFile operation that failed

[sachingaikwad]

Below is the screenshot of ProcessMonitor. Filtered with 'CreateFile operation' and 'vmtoolsd.exe Process'. There are no failure events though

Filtered with 'CreateFile operation' and 'Result is not Success':

[LucD]

One more thing to investigate, on the VCSA can you open an SSH session?

Then go to folder /var/log/vmware/vpxd

Now change the VCSA logging level to verbose

Get-AdvancedSetting -Entity $global:DefaultVIServer -Name 'config.log.level' | Set-AdvancedSetting -Value 'verbose' -Confirm:$false

Run your script that calls Invoke-VMScriptPlus script

Switch the logging level back to info

Get-AdvancedSetting -Entity $global:DefaultVIServer -Name 'config.log.level' | Set-AdvancedSetting -Value 'info' -Confirm:$false

Then open, on the VCSA in your SSH session, the vpxd.log file.

Do a search for the last occurrence of [createTemporaryDirectory]

Are there any error messages?

[sachingaikwad]

I don't have the necessary access to SSH into VCSA. Will see if I can get some help from Infra team on Monday.

[sachingaikwad]

LucD​ getting this error when executing the Get-AdvancedSetting -Entity $global:DefaultVIServer -Name 'config.log.level'. Please suggest.

[LucD]

Are you connected to the vCenter?

What is in $global:defaultVIServer?

[sachingaikwad]

Looks like there is some problem with the Domain which the VM was connected to. Tried creating a new user in AD and used the new user to Invoke-VM script but the new user also was getting the same error. Invoke-VM script was working fine with local admin. So switched to different AD, and now Invoke-VM script started working fine on the same VM.

There must have been some change to DC, but not sure what that is (tried comparing both the domains with gpresult but there was lot of differences so didn't proceed with the analysis).

We were still not able to find the root cause for this issue, but I'm not blocked anymore.

Thank you LucD​ for your help.

[LucD · Accepted solution]

You're welcome.

Btw, when I experience an AD authentication issue, I most of the time start with checking the secure channel (see the Test-ComputerSecureChannel cmdlet).

[sachingaikwad]

Thank you @LucD for the tip, will use it incase if I come across this issue again.

[sachingaikwad]

LucD faced this issue again, so tried 'Test-ComputerSecureChannel' command which you gave last time and it returned False.

Then tried 'Test-ComputerSecureChannel -repair' and after that it started working!

Thanks a ton, this time I'm glad this issue got resolved in few minutes instead of few days

Posting the solution here so that it might be helpful for someone else.

[sachingaikwad]

LucD​, looks like the scripts are failing again with the same issue 'A general system error'. But if I do 'Test-ComputerSecureChannel -repair' then it starts working. But to execute 'Test-ComputerSecureChannel -repair' I have to login to the VM. Please suggest, if there is any VMWare PowerCLI command which can be used to do the same task (this would avoid me to manually login to VMs). 

[LucD]

Is that with a domain account or a local account.
With a local account you shouldn't get the secure channel error, so you could try running the repair via Invoke-VMScript(Plus) and a local account.

[sachingaikwad]

My scripts has to run with domain account.

This time I already logged into VM and ran the repair command. Next time when I get the same error, I'll try the below command with local admin first before logging into VM and execute repair command.

$Command_text="Test-ComputerSecureChannel -repair"

Invoke-VMScript -VM $TARGET_VM -ScriptText $Command_text -GuestUser "local_admin" -GuestPassword "admin_password"

Thank you.