Hi LucD and Co. I'm about to take on an interesting pet project surrounding automation of vSphere "things" based on custom attribute values found on VMs and I'm posting an open question for ideas as to how to go about doing this. To be very clear, I'm not asking for people to hand over code to me--only a methodology for those who have worked with custom attributes in PowerCLI before. I know it's not difficult, per se, but since there are always multiple ways to skin a cat I'm wondering what the shortest, cleanest, most efficient path may be here.
Background
VMware PKS deploys K8s clusters as virtual machines. These clusters can be of various sizes and contain one or more masters and one or more workers. Each node is represented by a single VM. Every VM belonging to the same cluster has a custom attribute written into a field called "deployment". An example value of this "deployment" field is "service-instance_9c854520-8e43-4566-845e-111e6a1e8425". Additionally, a second field is present called "job" which specifies the role type for each VM, values of which are either "master" or "worker". All VMs in the same deployment are also connected to the same NSX-T logical switch which appears in vSphere's inventory as a custom switch type. For example, all VMs in the same deployment that have "service-instance_9c854520-8e43-4566-845e-111e6a1e8425" written into that custom attribute would exist on a logical switch named "pks-9c854520-8e43-4566-845e-111e6a1e8425". There are other logical switches created that also begin with this name but end with different suffixes.
Problem Statement
While this system is all well and good, there are some problems created and therefore gaps I wish to close:
Goals
Based on the above problems/gaps, I wish to
Now that you have all the information, how would you tackle this from a high-level perspective? I think the emphasis needs to be on speed and as much reduced impact on the vCenter API as possible because these PKS VMs could exist in a sea of thousands, and recursing through the entire vCenter inventory would probably not go well. Interested in ideas, commentary, and questions.
Message was edited by: Chip Zoller Added note about these NSX-T logical switches are not returned by PowerCLI available cmdlets in modules VMware.VimAutomation.Core or .Vds.
Ok, there seems to be an "issue" when using Group-Object with the InputObject parameter.
And there was a logic error in my original skeleton.
This should work better.
$caJob = 'job'
$rootFolderName = 'Project'
# Folder where all 'deployment' folders shall go
$folder = Get-Folder -Name $rootFolderName
Get-VM -Name vm-* |
Group-Object -Property {$_.CustomFields[$caDeployment]} |
where {$_.Name} |
ForEach-Object -Process {
$deployment = $_.Name
# Group the VMs in a deployment on job type
$_.Group | Group-Object -Property {$_.CustomFields[$caJob]} |
ForEach-Object -Process {
# Handling the masters
if ($_.Name -eq 'master') {
$_.Group | Select Name, @{N = 'Cluster'; E = {Get-Cluster -VM $_ }} |
Group-Object -Property Cluster |
ForEach-Object -Process {
# More than 1 master
if ($_.Group.Count -gt 1) {
$sRule = @{
Cluster = $_.Name
Name = "$($deployment) Anti-affinity rule"
VM = $_.Group.Name
KeepTogether = $false
Confirm = $false
}
# Create the anti-affinity rule
# Not taking into account when there are more masters than ESXi nodes
New-DrsRule @sRule
}
}
}
}
# Create the VM folder for this deployment and move all VMs in there
$targetFolder = New-Folder -Name "Deployment $($deployment)" -Location $folder -Confirm:$false
$_.Group | Move-VM -InventoryLocation $targetFolder -Confirm:$false | Out-Null
}
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Without actually testing the code against the requirements, I think Group-Object might be your best friend here.
It will let PowerShell do the required grouping for you.
Since skeleton code is easier to read, compared to a lot of text (at least for me), I would start testing and validating with below code.
I added some comments, documenting the main steps.
PS: why CustomFields and not Tags?
$caJob = 'job'
$rootFolderName = 'Project'
# Folder where all 'deployment' folders shall go
$folder = Get-Folder -Name $rootFolderName
Get-VM -Name vm-* |
Group-Object -Property {$_.CustomFields[$caDeployment]} |
where {$_.Name} |
ForEach-Object -Process {
$deployment = $_.Name
# Group the VMs in a deployment on job type
Group-Object -InputObject $_.Group -Property {$_.CustomFields[$caJob]} |
ForEach-Object -Process {
# Handling the masters
if ($_.Name -eq 'master') {
$_.Group | Select Name, @{N = 'Cluster'; E = {Get-Cluster -VM $_ }} |
Group-Object -Property Cluster |
ForEach-Object -Process {
# More than 1 master
if ($_.Group.Count -gt 1) {
$sRule = @{
Cluster = $_.Name
Name = "$($deployment) Anti-affinity rule"
VM = $_.Group.Name
KeepTogether = $false
Confirm = $false
}
# Create the anti-affinity rule
# Not taking into account when there are more masters than ESXi nodes
New-DrsRule @sRule
}
}
}
# Create the VM folder for this deployment and move all VMs in there
$targetFolder = New-Folder -Name "Deployment $($deployment)" -Location $folder -Confirm:$false
$_.Group | Move-VM -InventoryLocation $targetFolder -Confirm:$false
}
}
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Thanks very much, Luc! Although I didn't intend for you to spend your Saturday helping me, I thank you for that code
Let me play with it for a while and see what I can come up with. I think you're spot on that Group-Object is the way to go here and eliminates a lot of manual matching effort.
PS: why CustomFields and not Tags?
This is just how PKS works, and there's no way to either add Tags in addition to CAs or use one in place of the other. CAs wouldn't be my choice, but that's what they did...
I just happened to be stuck on some other code, saw your thread passing by, and since I find tackling questions easier with code than long winded sentences... :smileygrin:
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
After some tweaking it seems to work great, except the anti-affinity rule(s) doesn't get created. Honestly haven't looked into that much though. Now I just need to make it more idempotent so it can be scheduled and only perform those actions if not done (which should be easy with some -ErrorAction SilentlyContinue statements.
You might want to display the content of $_.Group just before the If line, where the number of masters is checked.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Seems like posh isn't liking double grouping or nested group structures of some sort because it's returning no objects there. I'll have to fiddle with this since I've not used Group-Objects before.
Time to use the VSC debugger (it has breakpoints) ![]()
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I see what it is. The second grouping (Group-Object -InputObject $_.Group -Property {$_.CustomFields[$caJob]}) is returning null in the Name property, so the code basically stops executing there since the if statement is going to be false. What's also funny is the formatting in tabular form. If I check this snippet in ISE, the "Group" name have VMs formatting without comma separation (like not an array?) versus if I just execute the first Group-Object statement.
Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'} | ForEach-Object -Process {Group-Object -InputObject $_.Group -Property {$_.CustomFields['job']}}
Can you show some sample out from
Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'}
and
ForEach-Object -Process {
Group-Object -InputObject $_.Group -Property {$_.CustomFields['job']}
}
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'} | fl
Name : service-instance_9c854520-8e43-4566-845e-111e6a1e8425
Count : 6
Group : {vm-785dd36f-d984-4700-9b8d-54381758d141, vm-46e24f81-d4ad-4722-9cc3-1c70b631663d, vm-14876478-6e6e-4a00-933f-7bafecbef991, vm-bcc0550b-5203-48f0-a674-d59543ddb5e0...}
Values : {service-instance_9c854520-8e43-4566-845e-111e6a1e8425}
and
Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'} |
ForEach-Object -Process {
Group-Object -InputObject $_.Group -Property {$_.CustomFields['job']} | fl
}
Name :
Count : 1
Group : {vm-785dd36f-d984-4700-9b8d-54381758d141 vm-46e24f81-d4ad-4722-9cc3-1c70b631663d vm-14876478-6e6e-4a00-933f-7bafecbef991 vm-bcc0550b-5203-48f0-a674-d59543ddb5e0 vm-063992a7-c9f3-4cee-9287-27e4a4ac15b1 vm-73d25407-4e22-4e70-a2af-825b33ede569}
Values : {$null}
Sorry I didn't provide this earlier. Was being hurried to dinner
Boarding a flight in a few hours so may get time to play around some more with the second grouping to figure out why it's not picking that Name up.
Ok, there seems to be an "issue" when using Group-Object with the InputObject parameter.
And there was a logic error in my original skeleton.
This should work better.
$caJob = 'job'
$rootFolderName = 'Project'
# Folder where all 'deployment' folders shall go
$folder = Get-Folder -Name $rootFolderName
Get-VM -Name vm-* |
Group-Object -Property {$_.CustomFields[$caDeployment]} |
where {$_.Name} |
ForEach-Object -Process {
$deployment = $_.Name
# Group the VMs in a deployment on job type
$_.Group | Group-Object -Property {$_.CustomFields[$caJob]} |
ForEach-Object -Process {
# Handling the masters
if ($_.Name -eq 'master') {
$_.Group | Select Name, @{N = 'Cluster'; E = {Get-Cluster -VM $_ }} |
Group-Object -Property Cluster |
ForEach-Object -Process {
# More than 1 master
if ($_.Group.Count -gt 1) {
$sRule = @{
Cluster = $_.Name
Name = "$($deployment) Anti-affinity rule"
VM = $_.Group.Name
KeepTogether = $false
Confirm = $false
}
# Create the anti-affinity rule
# Not taking into account when there are more masters than ESXi nodes
New-DrsRule @sRule
}
}
}
}
# Create the VM folder for this deployment and move all VMs in there
$targetFolder = New-Folder -Name "Deployment $($deployment)" -Location $folder -Confirm:$false
$_.Group | Move-VM -InventoryLocation $targetFolder -Confirm:$false | Out-Null
}
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Nice, Luc, that's it exactly. Need to go back and compare your code to mine so I understand this for future reference. Cheers to you for the help, Luc! Now on to the next task!
