Its not possible to boot without the SVGA adapter.
What NVIDIA GPU are you passing through and how do you intend to use it?
You could try to switch primary display in windows.
// Linjo
Best regards,
Linjo
Please follow me on twitter: @viewgeek
If you find this information useful, please award points for "correct" or "helpful".