ESXi is not able to pass through cards like modems, fax cards, USB devices to VMs. Usually if you need to use a special device in a VM you can use a USB / serial over IP device to attach the device to the VM. You might be able to use the PCI card with VMware Server if the modem is presented as a COM device.