First you need to remove the port group from VM which showing connected. Once you remove you are able to delete the port group.
