VMware Cloud Community
vin01
Expert
Expert

invoke-vm hangs while executing on vms resides vcenter 6.5

I am using this script to get information inside the guest os. Its works as expected when executed on multiple vcenters but the only problem observed is it stops invoke at some stage  while executing on vms resides on vcenter 6.5.

Succeeded on below versions.

14 vcenters  with version 6.0 of vm count appox 2000 on each- Able to get the csv report.

But only single vcenter version 6.5 with 1500 vms appox shows script execution in progress but not output . I even verified whether it is stopping on specific vm but not.

Even tried to give 50 vms as a set but its not giving the output. If i try 10 set of vms able to get the output.

Can someone please help me how to fix for this specific version.

Write-Host "Enter the required Information when Prompted" -ForegroundColor Yellow

$password1= Get-Credential -Message "Enter One of the 3 Default password for Administrator User" -UserName "Administrator"

$password2= Get-Credential -Message "Enter One of the 3 Default password for Administrator User" -UserName "Administrator"

$password3= Get-Credential -Message "Enter One of the 3 Default password for Administrator User" -UserName "Administrator"

$password4= Get-Credential -Message "Enter password for Admin User" -UserName "Admin"

$vcenter = Read-Host "Enter vCenter Name Ex:Sez00vvm**.sweng.ncr.com"

$vcenterCred= Get-Credential -Message "Enter Credentials to Connect vCenter"

Write-Host "Connecting to vCenter $vcenter"

Connect-VIServer $vcenter -Credential $vcenterCred

#$resourcepool=Read-Host "Enter Resource Pool Name like this 'rp1'. If you want to enter multiple resource pools give input like this 'rp1','rp2'"

$csvfilepath= Read-Host -Prompt "Enter csv files with vmnames"

$csvfile= Import-Csv -Path $csvfilepath

Write-Host "All required information received output file will be saved in this location $filename" -ForegroundColor Yellow

$script = @'

if([IntPtr]::Size -eq 8){

    $text = Get-ItemProperty -Path "HKLM:\SOFTWARE\WOW6432Node\Network Associates\ePolicy Orchestrator\Agent\" -ErrorAction SilentlyContinue |

        %{"$($_.AgentGUID) and ePOServerList is $($_.ePOServerList)"}

}

elseif([IntPtr]::Size -eq 4){

    $text = Get-ItemProperty -Path "HKLM:\SOFTWARE\Network Associates\ePolicy Orchestrator\Agent\" -ErrorAction SilentlyContinue |

        %{"$($_.AgentGUID) and ePOServerList is $($_.ePOServerList)"}

}

if(-not $text){

    $text = Get-Service -Name "McAfee Framework Service" -ErrorAction SilentlyContinue | %{"$($_.Name) is $($_.Status)"}

    if(-not $text){

        $text = Get-Service -Name "Windows Time" | Select -First 1 |

            %{"$($_.Name) is $($_.Status)"}

    }

}

$cbstatusinfo= "CarbonBlack Not Installed"

$cbout = Get-Service -Name "CarbonBlack" -ErrorAction SilentlyContinue

if($cbout){

$cbstatusinfo = "$($cbout.name -join ',') installed"

}

$wuserv= Get-Service -Name "wuauserv" | Select -first 1 | %{"$($_.Name) is $($_.Status)"}

$hotfix = "KB4012598", "KB4012598", "KB4012598", "KB4012598", "KB4012212", "KB4012215", "KB4012212", "KB4012215", "KB4012213", "KB4012216", "KB4012214", "KB4012217", "KB4012213", "KB4012216", "KB4012606", "KB4013198", "KB4013429", "KB4013429"

$hotfixinfo = "No hotfix found!"

$fixes = Get-WmiObject -Class "win32_quickfixengineering" |

    where{$hotfix -contains $_.HotFixID} |

    Select -ExpandProperty HotFixID

if($fixes){

    $hotfixinfo = "$($fixes -join ',') installed"

}

$windowsUpdateObject= New-Object -ComObject Microsoft.Update.AutoUpdate

$wup= $windowsUpdateObject.Results

$windowsupdateresult= "LastSearchSuccessDate is $($wup.LastSearchSuccessDate) and LastInstallationSuccessDate $($wup.LastInstallationSuccessDate)"

$outputarray=@()

$outputarray+=$hotfixinfo

$outputarray+=$cbstatusinfo

$outputarray+=$text

$outputarray+=$wuserv

$outputarray+=$windowsupdateresult

$outputarray -join ';'

'@

$obj = foreach ($csvvmnames in $csvfile){

foreach($vm in Get-VM $csvvmnames.Name  |  Where-Object {$_.PowerState -eq "Poweredon" -and $_.ExtensionData.Config.GuestFullName -like "*Microsoft*"}) {

  $out1 = ""

  $out2 = ""

  $out3 = ""

  $out4 = ""

  $found = $false

   try{ 

    $out1 = Invoke-VMScript -VM $vm.name -GuestCredential $password1  -ScriptText $script -ScriptType Powershell -ErrorAction Stop | Select -ExpandProperty ScriptOutput

    $found = $true

  } 

  catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidGuestLogin]{ 

    $out1 = "Invalid logon" 

  } 

  catch{ 

    $out1 = "any other output" 

  } 

  if(!$found) {

    try{ 

      $out2 = Invoke-VMScript -VM $vm.name -GuestCredential $password2 -ScriptText $script -ScriptType Powershell -ErrorAction Stop | Select -ExpandProperty ScriptOutput

      $found = $true

    } 

    catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidGuestLogin]{ 

      $out2 = "Invalid logon" 

    } 

    catch{ 

      $out2 = "any other output" 

    } 

  }

  if(!$found){

    try{ 

      $out3 = Invoke-VMScript -VM $vm.name -GuestCredential $password3  -ScriptText $script -ScriptType Powershell -ErrorAction Stop | Select -ExpandProperty ScriptOutput

      $found = $true

    } 

    catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidGuestLogin]{ 

      $out3 = "Invalid logon" 

    } 

    catch{ 

      $out3 = "any other output" 

    }

  }

  if(!$found){

    try{ 

      $out4 = Invoke-VMScript -VM $vm.name -GuestCredential $password4  -ScriptText $script -ScriptType Powershell -ErrorAction Stop | Select -ExpandProperty ScriptOutput

      $found = $true

    } 

    catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidGuestLogin]{ 

      $out4 = "Invalid logon" 

    } 

    catch{ 

      $out4 = "any other output" 

    }

  }

New-Object PSObject -Property (

    [ordered]@{

        Name = $vm.Name

        OS   = $vm.Guest.OSFullName

        IP   = ($vm.Guest.IPAddress | Where {($_.Split(".")).length -eq 4}) -join (",")

        RP   = $vm.ResourcePool.Name

        Out1 = $out1

        Out2 = $out2

        Out3 = $out3

        Out4 = $out4

  })

}

}

$CurrentDate = Get-Date -Format 'MM-dd-yyyy_hh-mm-ss'

$filename = "C:\Scanreport-on-$($global:DefaultVIServer.Name)at$($CurrentDate).csv"

$obj |Export-Csv -Path $filename -NoTypeInformation -NoClobber -UseCulture

Send-MailMessage -From "wtrlabsvmssecurityscanreport@ncr.com" -To vk185112@ncr.com,Ian.Lovegrove@ncr.com,Miguel.Caringal@ncr.com -Subject "waterloo lab machines security scan report from $($global:DefaultVIServer.Name)" ` -Body "The attachment contains Scanreport for waterloo lab machines" ` -Attachments $filename -SmtpServer 'internalmail.sweng.ncr.com'

#

pastedImage_4.png

Regards Vineeth.K
Reply
0 Kudos
17 Replies
LucD
Leadership
Leadership

Isn't that warning message giving an indication of what could possibly be causing this?

I would suggest to test with a simpler script via Invoke-VMScript.

For example, only send a Get-Process to the VMs, and check if that comes back.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vin01
Expert
Expert

Isn't that warning message giving an indication of what could possibly be causing this?

I can see the same warnings on other vms when I executed on vcenter 6 version. But able to get the output.

I would suggest to test with a simpler script via Invoke-VMScript.

I have tested with simple one line output (Get-Service wuauserv) but same response.

Regards Vineeth.K
Reply
0 Kudos
LucD
Leadership
Leadership

Did you also try with a Bat type of script, that just contains dir for example?

There might be a PowerShell issue on these VMs.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vin01
Expert
Expert

Did you also try with a Bat type of script, that just contains dir for example?

There might be a PowerShell issue on these VMs.

I never tried. I will try now.

will be possible to add time out. (Some thing like If the invoke-vm executed on a VM and it will wait for some N seconds and move to next vm.)

Regards Vineeth.K
Reply
0 Kudos
LucD
Leadership
Leadership

Are you using the RunAsync switch with the Invoke-VMScript cmdlet?

The code you included at the beginning of this thread doesn't show that.

That also means that you have to capture the output of the script via the Task object that is returned on each invocation of Invoke-VMScript.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vin01
Expert
Expert

I didn't used RunAsync switch with the Invoke-VMScript cmdlet.

Should I use like this?

$task=Invoke-VMScript -VM vmname -GuestUser 'administrator' -GuestPassword '' -ScriptText $script -ScriptType Powershell -ErrorAction Stop -RunAsync

while($task.State -ne 'Success'){

sleep 5

$result=$task.Result |Select-Object -ExpandProperty Scriptoutput

}

$result

Regards Vineeth.K
Reply
0 Kudos
LucD
Leadership
Leadership

If you didn't use RunAsync, then I don't get why you have all these "Initializing" message on the screen.

The cmdlet should wait till the scriopt comes back.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vin01
Expert
Expert

Yeah. That is the same script I am using. There is no change. It will keep on showing  "Initializing" but  no results.I just executed on 20 VMs.

Even in other versions of vcenter it will show as "Initializing" on multiple VM's at once and I get the results as expected.

Regards Vineeth.K
Reply
0 Kudos
LucD
Leadership
Leadership

So it comes back, and then you start the next one.

But no output?
What did the Bat script do?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vin01
Expert
Expert

So it comes back, and then you start the next one.

But no output?

When I start the .ps1 with in few minutes it will start invoke-vm script by showing % completed on multiple vms. After  some hrs it will just show as 'vmcount activities pending' but no output. powershell windows just hangs. I need to cancel the process manually.

Like below.

pastedImage_0.png

I even tried passing script as a job. ($allvmsfrmVRA=Start-Job -FilePath 'C:\My Data\Working\McAfeeCBwannaycryinstallstatuscheckAllvmandrespoolsinVRA.ps1') but the job never completes. I see this behaviour only on one particular vcenter.

What did the Bat script do?

Its in progress. I tried on the same set of vms. Once completes I will let you know. This is status of invoke bash script.

pastedImage_1.png

Regards Vineeth.K
Reply
0 Kudos
vin01
Expert
Expert

What did the Bat script do?

The bat script executed successfully.

Regards Vineeth.K
Reply
0 Kudos
LucD
Leadership
Leadership

So it looks to be a PowerShell thing.

Did you try to logon, with the account you provided to Invoke-VMScript, to one of the target VMs, and start PowerShell?

Are you able to execute cmdlets in that session?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vin01
Expert
Expert

Yeah it’s only issue with powershell.

Did you try to logon, with the account you provided to Invoke-VMScript, to one of the target VMs, and start PowerShell?

Are you able to execute cmdlets in that session?

Yes I can able to login and execute the scrips in the  vms.

adding to it  i even noticed if I execute the complete script with a single problematic vm  I can able to get output in csv file.

Regards Vineeth.K
Reply
0 Kudos
vin01
Expert
Expert

Hi LucD,

Any thoughts on this issue?

I somehow managed to get the report by executing a set of 20vms on each powershell window.

Even while executing a set of 20vm some times the script is hanged at that point  I need to execute multiple times.

Regards Vineeth.K
Reply
0 Kudos
LucD
Leadership
Leadership

I would suggest to add a Start-Transcript and Stop-Trancript in your script.

Perhaps the output in the transcript makes us any wiser.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vin01
Expert
Expert

I have added Start-Transcript and Stop-Transcript  in the script and executed on problematic vcenter . The output is attached Can you please check and suggest.

The script was started 4 days ago on 900 vms and it written the Transcript output for some vms after that its not even writing the output.

Similar way I executed on 2000vms from different vcenter and it  completed successfully in 48hrs of time period. 

Regards Vineeth.K
Reply
0 Kudos
LucD
Leadership
Leadership

There seem to be a couple of things going on.

Some VMs return this "A general system error occurred: vix error codes = (3016, 0)"

That could indicate the VM guest OS is waiting for a reboot.

Perhaps due to Windows patching that was not yet completed (reboot needed).

Then there are the ones with "A general system error occurred: vix error codes = (1, 0)"

For those I would suggest to have a look at the vmware.log of these VM.

There should be more info on the failure of the call in there.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos