I've done this sort of thing quite a bit in my environment. I have a runner I use for ssh commands. I keep config elements which are just an array of hostnames. I might have a config element for all the hosts I know are setup to run a certain cli (like our ipam management cli for instance). When I kick off the runner I specify the runtype of "vQIP" and the runner grabs the contents of that config element, selects a random number based on the number of items in the array, then tries to run the command on that host. If the command fails I slice out the array element and generate another random number on the remaining items and just cycle through the array. I also have a retries parameter both for iterations and a timeout I can pass in.