What is your guest OS? If Windows, often Explorer "pings" the CD drive periodically to see if a disc has been inserted.

Just disconnect the virtual CD drive from your guest if you're not using it. When you need to use it, reconnect it.