VMware Cloud Community
sjesse
Leadership
Leadership

Get fiber channel path io stats

We are migrating fiber switches and I'm trying to find a way to verify io is going over the paths we are moving. Using this

Get-VMHost $host | Get-ScsiLun | Get-ScsiLunPath

I only see active or standby, but comparing this against the gui there isn't an Active(io) state, or a way to tell which one has active io.   I made this based  and it works, but I'm hoping to improve it to tie the paths back to actual datastores if possible, but I can't see a way yet. Is it possible and is this the best way to test this?

$esxcli= Get-EsxCli -VMHost $esx -V2

$paths=$esxcli.storage.core.path.stats.get.Invoke()

foreach($path in $paths)

{

    $measure1 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$path.RuntimeName})

    sleep 5

    $measure2 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$path.RuntimeName})

    $pathSuccessCommands=$measure2.SuccessfulCommands - $measure1.SuccessfulCommands

    Write-Host $path.RuntimeName +" " + $pathSuccessCommands

}

0 Kudos
11 Replies
LucD
Leadership
Leadership

You could do something like this

$esxcli= Get-EsxCli -VMHost $esx -V2

$dsTab = @{}

foreach($ds in $esxcli.storage.vmfs.extent.list.Invoke()){

   $esxcli.storage.core.path.list.Invoke() |

   where{$_.Device -eq $ds.DeviceName} | %{

   $dsTab.Add($_.RuntimeName,$ds.VolumeName)

   }

}


$paths=$esxcli.storage.core.path.stats.get.Invoke()

foreach($path in $paths)

{

   $measure1 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$path.RuntimeName})

  sleep 5

   $measure2 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$path.RuntimeName})


   $pathSuccessCommands=$measure2.SuccessfulCommands - $measure1.SuccessfulCommands


   Write-Host "$($path.RuntimeName) $($dsTab[$path.RuntimeName]) $($pathSuccessCommands)"

}


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
sjesse
Leadership
Leadership

Thanks, I have something similar, I have to run this on 16 hosts is there a better way that doing start-job to run them in parrall. I'm running it on 16 core cpu and its taking 20 minutes just for a 2 second check. I'm not really familar with runspaces in powershell, is it possible to that with checking the paths, that was the next thing I was looking at.

0 Kudos
LucD
Leadership
Leadership

Runspaces and PowerCLI don't work together that weel, go for background jobs with Start-Job.

There are a few peculiarities to keep in mind.

Have a look at Re: Poweroff and Poweron 1000 VMs in Parallel

Let me know if you can go from there, otherwise I can try to come up with a script.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
sjesse
Leadership
Leadership

Looks pretty close to what I'm doing, I'm just looking to incorporate your device part. I had a way to tie it back to the datastore, but running that command per path is expensive. I'm going to try and pass the session id, didn't see that as an option before. Was getting wierd output from the csv if I didn't unpack the psobjects which is the reason for the bottom section. It "works" but is slower than I'd like, I need to run it 5 times while we do the moves.

$jobs=@()

Connect-VIServer serverUser $username -Password $password

$report=@()

foreach($esx in Get-VMHost)

{

    Write-Host "Starting" $esx.name " Path Check"

            $jobs+=Start-Job -Name $esx.Name -ScriptBlock {

            Connect-VIServer server -User $args[1] -Password $args[2]

            $esxcli= Get-EsxCli -VMHost $Args[0] -V2

            $paths=$esxcli.storage.core.path.stats.get.Invoke()

            $esxpaths=@()

            foreach($path in $paths)

            {

               Write-Host "Checking " $path.RuntimeName

            $measure1 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$path.RuntimeName})

            sleep 2

            $measure2 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$path.RuntimeName})

            $pathSuccessCommands=$measure2.SuccessfulCommands - $measure1.SuccessfulCommands

            $reads=$measure2.BlocksRead - $measure1.BlocksRead

            $writes=$measure2.BlocksWritten - $measure1.BlocksWritten

            $naaId=$path.UID.Substring($path.UID.LastIndexOf('-')+1,$($path.UID.Length - ($path.UID.LastIndexOf('-')+1)))

            #$datastore=Get-Datastore | Where-Object { $_.extensiondata.info.vmfs.extent.diskname -like $naaId}

            $pathspeed=New-Object PSObject

            $pathspeed | Add-Member -MemberType NoteProperty -Name "ESX Host" -Value $Args[0]

            $pathspeed | Add-Member -MemberType NoteProperty -Name "Path Runtime Name" -Value $path.RuntimeName

            $pathspeed | Add-Member -MemberType NoteProperty -Name "Path IO" -Value $pathSuccessCommands

            $pathspeed | Add-Member -MemberType NoteProperty -Name "naa" -Value $naaId

            $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksRead" -Value $reads

            $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksWritten" -Value $writes

            # $pathspeed | Add-Member -MemberType NoteProperty -Name "Datastore" -Value $datastore.Name

            $esxpaths+=$pathspeed

            }

            return $esxpaths

            }   -ArgumentList $esx.Name,$username,$password

}

Write-Host "Waiting For Jobs to Complate"

Wait-Job -Job $jobs

foreach($job in Get-Job){

   

    if($job.state -eq "Completed")

    {

        Write-Host $job.Name "Completed"

        $esxPaths=Receive-Job -Id $job.Id

        foreach($path in $esxPaths)

        {

            $pathspeed=New-Object PSObject

            $pathspeed | Add-Member -MemberType NoteProperty -Name "ESX Host" -Value $path.'ESX Host'

            $pathspeed | Add-Member -MemberType NoteProperty -Name "Path Runtime Name" -Value $path.'Path Runtime Name'

            $pathspeed | Add-Member -MemberType NoteProperty -Name "Path IO" -Value $path.'Path IO'

            $pathspeed | Add-Member -MemberType NoteProperty -Name "naa" -Value $path.naa

            $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksRead" -Value $path.BlocksRead

            $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksWritten" -Value $path.BlocksWritten

          #  $pathspeed | Add-Member -MemberType NoteProperty -Name "Datastore" -Value $path.Datastore

            $report+=$pathspeed

        }

        Remove-Job -id $job.id -Force

    }

}

Disconnect-VIServer

$report | Export-Csv C:\Users\sjesse_admin\Desktop\sdc_path_check.csv

0 Kudos
LucD
Leadership
Leadership

Not too sure why you are connecting to each ESXi node.

There is no need to that, the Get-EsxCli cmdlet works while only connected to the vCenter.
And that way you can reuse the existing connection with the SessionId.

You can also create the hash table with the mapping between the DeviceName and the DatastoreName only once.

And pass that as a parameter to the Start-Job cmdlet.

Unless you are looking for local datastores as well.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
sjesse
Leadership
Leadership

"There is no need to that, the Get-EsxCli cmdlet works while only connected to the vCenter."

Were you referring to the Connect-VIServer for each host, that was because I didn't know about the ability to pass the session. If not I'm not sure what you mean, because I need to tell it to look at that host right?

Would it be a bad idea to try and do a job per path? I want to see if its possible instead  of doing one at a time say try 5 at a time. I'm just looking to reduce as much time as possible, it takes about 20 minutes waiting 2 seconds between checks, and I'd like to increase that increasing the time since some times I'm not seeing any traffic.

0 Kudos
LucD
Leadership
Leadership

Indeed, you can reuse your open vCenter session.

I also meant that you know seems to do the Get-EsxCli while only connected to a specific ESXi node, not the vCenter.

My remark tried to explain that for Get-EsxCli you don't need to be connected to the ESXi node, the vCenter will do.

You can do as many background jobs as you want, there is a practical limit on how many background jobs can run in your PowerShell session and on your station.

Mostly determined by the available memory.

If you do a job per path, you would have to do the Get-EsxCli cmdlet again.

I don't think you can pass that to a background job.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
sjesse
Leadership
Leadership

Is it a bad idea to nest start-job commands? I realize this is probably basic powershell, but I can't seem to catch the problem. I don't think all of the jobs are being cleaned up properly, as I run out of memory and need to logout to get back on. A wierd side affect also seems to be locking by account, which is confusing me since I'm only logging in once.

$jobs=@()

$session=Connect-VIServer  $vcserver -User $username -Password $password

$report=@()

$script={

            param(

                [string]$vcServer,

                [string]$vcId,

                [string]$esxName

                )

           

            Connect-VIServer $vcServer -Session $vcId

            $esxcli= Get-EsxCli -VMHost $esxName -V2

            $paths=$esxcli.storage.core.path.stats.get.Invoke()

       

            $esxpaths=@()

            $script={

                    param(

                        [string]$vcServer,

                        [string]$vcId,

                        [string]$esxName,

                        [string]$pathRuntimeName,

                        [string]$naaId

                    )

                    Connect-VIServer $vcServer -Session $vcId

                    $esxcli= Get-EsxCli -VMHost $esxName -V2

                    $measure1 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$pathRuntimeName})

                    sleep 2

                    $measure2 = $esxcli.storage.core.path.stats.get.Invoke(@{path=$pathRuntimeName})

                    #Getting Successful

                    $pathSuccessCommands=$measure2.SuccessfulCommands - $measure1.SuccessfulCommands

                    $reads=$measure2.BlocksRead - $measure1.BlocksRead

                    $writes=$measure2.BlocksWritten - $measure1.BlocksWritten

                    $pathspeed=New-Object PSObject

                    $pathspeed | Add-Member -MemberType NoteProperty -Name "ESX Host" -Value $esxName

                    $pathspeed | Add-Member -MemberType NoteProperty -Name "Path Runtime Name" -Value $pathRuntimeName

                    $pathspeed | Add-Member -MemberType NoteProperty -Name "Path IO" -Value $pathSuccessCommands

                    $pathspeed | Add-Member -MemberType NoteProperty -Name "naa" -Value $naaId

                    $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksRead" -Value $reads

                    $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksWritten" -Value $writes

                    return $pathspeed

               }

               $jobs=@()

               $maxJobs=1

              

                      

               foreach($path in $paths)

               {

                   Write-Host $jobs.Count

                   Write-Host $path.RuntimeName

                   $naaId=$path.UID.Substring($path.UID.LastIndexOf('-')+1,$($path.UID.Length - ($path.UID.LastIndexOf('-')+1)))

                   $jobs+=(Start-Job -Name $path.RuntimeName -Verbose -ScriptBlock $script -ArgumentList $vcserver,$session.SessionId,$esxName,$path.RuntimeName,$naaId)

                   if($jobs.Count -ge $maxJobs)

                   {

                        Write-Host "Processing Jobs"

                        foreach($job in $jobs)

                        {

                            Wait-Job $jobs

                            Write-Host "Checking for Complated Jobs"

                            if($job.state -eq "Completed")

                            {

                                $return=Receive-Job $job.Id

                                $return                     

                                $esxpaths+=$return

                                Remove-Job $job.Id -Force

                            }

                        }

                        $jobs=@()

                   }

                  

               

               }

               Wait-Job $jobs

               foreach($job in $jobs)

                        {

                            if($job.state -eq "Completed")

                            {    

                                Wait-Job $jobs

                                $esxpaths+=Receive-Job $job.Id

                                Remove-Job $job.Id -Force

                             }

                        }              

               return $esxpaths

           }

foreach($esx in Get-VMHost)

{

    Write-Host "Starting" $esx.name " Path Check"

    $jobs+=Start-Job -verbose -Name $esx.Name -ScriptBlock $script  -ArgumentList $vcserver,$session.SessionId,$esx.Name

}

Write-Host "Waiting For Jobs to Complate"

Wait-Job -Job $jobs

foreach($job in $jobs){

   

    if($job.state -eq "Completed")

    {

        Write-Host $job.Name "Completed"

        $esxPaths=Receive-Job -Id $job.Id

        foreach($path in $esxPaths)

        {

            $pathspeed=New-Object PSObject

            $pathspeed | Add-Member -MemberType NoteProperty -Name "ESX Host" -Value $path.'ESX Host'

            $pathspeed | Add-Member -MemberType NoteProperty -Name "Path Runtime Name" -Value $path.'Path Runtime Name'

            $pathspeed | Add-Member -MemberType NoteProperty -Name "Path IO" -Value $path.'Path IO'

            $pathspeed | Add-Member -MemberType NoteProperty -Name "naa" -Value $path.naa

            $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksRead" -Value $path.BlocksRead

            $pathspeed | Add-Member -MemberType NoteProperty -Name "BlocksWritten" -Value $path.BlocksWritten

          #  $pathspeed | Add-Member -MemberType NoteProperty -Name "Datastore" -Value $path.Datastore

            $report+=$pathspeed

        }

        Remove-Job -id $job.id -Force

    }

}

Disconnect-VIServer

$report | Export-Csv C:\Users\sjesse_admin\Desktop\sdc_path_check.csv

0 Kudos
LucD
Leadership
Leadership

Normally PowerShell is quite good at cleaing up after itself.

When you do a Receive-Job (without the -Keep switch) or a Remove-Job, the memory from that job is cleaned up.

You can always try to insert the .Net garbage collection call

[System.GC]::Collect()

But since PowerShell v3 that shouldn't be needed.

Which account locks up?

The one with which you connect to the vCenter?


You could check if the $vcId is the same in each background job.
Perhaps there might be an issue there with nested jobs (never did that or needed it).

I'm not sure why you are using nested background jobs btw.

You start a background job for each ESXi node, and then in there you start a background job path by path.

Assume you have 3 ESXi nodes, each having 4 paths.

In you script, that would have 3 (ESXi nodes) + 3 x 1 (1 path/ESXi node) = 6 jobs running.


You could also run through all the ESXi nodes one by one, then for each path on a node you start a background job (until you hit the maximum)

You might also want to have a look at PSThreadJob.

This allows you to throttle background jobs, and these jobs use less memory (thread vs session).

But I haven't really validated if this causes any issues with PowerCLI (as RunSpaces do).


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
sjesse
Leadership
Leadership

I'm not sure why you are using nested background jobs btw.

You start a background job for each ESXi node, and then in there you start a background job path by path.

Assume you have 3 ESXi nodes, each having 4 paths.

In you script, that would have 3 (ESXi nodes) + 3 x 1 (1 path/ESXi node) = 6 jobs running.

Thats what I mean by nesting the first set of jobs are for each host. Then inside that job currently I'm limiting it to 1 at a time, but I'm hoping to get to 2 or 3.

I'm  not locking an account, I'm running out of memory, and I'm seeing way more powershell processes that I expect. I most not be removing them correctly, I think I might just start from the beginning based on what I've learned, its been a trial and error experience and I might be missing something.

0 Kudos
sjesse
Leadership
Leadership

Sorry I mistyped. It is locking by account, but its also creating way too many proccesses. On my commute home i think I have  a better way now, I'll share later if it works better

0 Kudos