On esx3.0 server is it possible to allow only traffic from my own subnet to host machine via esxcfg-firewall and would that would effect the network traffic on guests?
type man iptables on the ESX console. You would have to alter the ESX server firewall rules using IPtables which does the firewalling for the Linux kernel.