Inactive as in shut off or inactive as in destroyed? If they're shut off you probably want enforcement dvfilters installed as soon as they boot up, so it shouldn't be a bad thing.

Presumably you can do that by removing the tag from those VMs.