Why are you worrying about swapping ? Swapping is bad when the active applications are hitting the swap. If a background application was swaped out because it hasn't touched its memory in ages, there is nothing wrong about that. I for one would rather have my backgound, idle dæmons be swaped out than having them be killed because of the balloon driver. Of course, the correct solution is to get enough RAM to run everything in main memory.
Are your applications slow because they have to touch the swap? From the small screenshots you provided, it doesn't seem to be the case. Maybe there was a time during which memory was under pressure, ballooning couldn't free enough memory, so the host decided to swap out some idle memory. (Of course, when memory pressure is relaxed, the host is not going to swap back in the idle, untouched-in-ages memory it previously swapped out to disk.)
Also, note that ballooning is only available if your guests have the tools correctly installed. IIRC, for Linux guests this means the vmmemctl driver is loaded --check with lsmod|grep vm. I don't know for Windows guests.
