I suspect that should be and -and instead of an -or.
I also changed the NotLike to NotMatch, avoids the asterisk, and the text can appear anywhere in the string
Try like this
New-VIProperty -Name ToolsStatus -ObjectType VirtualMachine -ValueFromExtensionProperty 'Guest.ToolsStatus' -Force
New-VIProperty -Name GuestOS -ObjectType VirtualMachine -ValueFromExtensionProperty 'Config.GuestFullName' -Force
Get-VM | Where-Object {$_.GuestOS -NotMatch "Microsoft Windows Server" -and $_.GuestOS -NotMatch "Linux"} |
Select Name,PowerState,ToolsStatus, GuestOS