It is not that simple. I think that I already answered it in some other thread, but I'm lazy to look for it, so I'll retype it here.
There are three (or maybe more) problems with generic support for PCI cards:
(1) It is not easy to say whether host OS uses card or not. ESX makes it simpler, but we do not want to run ESX, yes?
(2) PCI cards can do busmastering. When you program hardware, you cannot distinguish programming of busmastering transfers from normal programming of card. So either you'll maintain 1:1 mapping between physical guest memory and host's physical memory (which means that you can run only one guest OS, and you'll have to use special host OS. So for hosted products busmastering is impossible, for ESX maybe, but with very huge costs.
(3) After we eliminated busmastering, there are interrupts. As only guest OS has an idea how to acknowledge (and deassert) interrupts generated by hardware, it means that we cannot share generic PCI's interrupt line with any other device which is used by host - as we only know how to disable IRQ on PIC/APIC/IOAPIC. Maybe PCI-X's generic IRQ enable on card can lift this requirement, but I'm afraid that touching this bit could change hardware behavior.
(4) After we eliminated cards with busmastering and interrupts - who's still in contest? I'm afraid that nobody. And from those devices which are still in contest we have to eliminate devices which can lockup PCI bus while programmed - for example (from my exprience) when you setup Matrox cards, there are time windows during which framebuffer access locks up card hard (not that all except old MillenniumI were eliminated by no-busmastering condition above, but you could try to run them without busmastering). So you do not want to run untrusted VMs, as VM can now crash your host.
Due to conditions above I do not think that generic PCI cards access is possible without BIG support from hardware. Maybe chipsets with IOMMU could shield PCI card's busmastering from damaging your system, but until IOMMU is generally available I'm afraid that there is no chance that anybody can implement usable framework.
If we talk about it, I think that instead of direct PCI access you should ask for API which would allow you to write your own virtual devices - that way, with device datasheet, you should be able to write simple pass-through filter which will know which accesses are for busmastering/interrupts and which are not, and then you can map/translate what needs to be translated, and pass through safe operations.