I recently setup a 6.7 vSphere host in my home. It lays host to my Untangle web filter, an IoT server, a Minecraft server (for my kids), and a few testing rigs.
I have 2 onboard NIC's and a PCIe dual LAN giving me 4 physical ports in total. Pretty much what you have with your ESXi box.
Now, install the VMWare ESXi via DVD or USB and fill out the gritty bits. I set the IP of my management device "static", and to the range that will be on my home LAN (10.0.0.2/24). While it was finishing, grabbed my laptop and assigned a static IP of 10.0.0.10/24, and then also patched the server LAN0 to my internet modem, LAN1, LAN2 and LAN3 to my unmanaged gigabit switch. My specific needs were: Internet (modem) - UntangleVM - Home Network, so I technically only need 2 network interfaces.
When it was all finished and rebooted, logged into the management page (10.0.0.2) from my laptop, and went straight to the Networking area.
Click the vSwitch tab, and "Add a standard switch", then add your LAN0 NIC to that switch. This with be your Gateway, or external switch.
Then go to the "Port group" tab and "Add port group" called "Internet" or "External", and add it to the "Gateway" switch that is in the list.
Now when you create your firewall VM (pfSense, CleaOS, Untangle, Smoothwall, IPCop etc), you now have two virtual network devices to attach to the system for your LAN and EXT interfaces. I then installed the Untangle VM and setup DHCP serving on the LAN interface, and tested it with another PC connected to the Gigabit switch. All good.
I'm not sure if this is the way to do it officially, I am a self taught guy and this works for me quite well.
I prefer to split your uplinks between two virtual switches:
- First one for LAN connectivity (LANvSW)
- Second one for WAN connectivity (WANvSW)
For high availability and redundancy, you can configure the above virtual switches with 2 uplinks each one.
Or one uplink for WANvSW and three for LANvSW (depend on your config).
Note: uplinks of WANvSW should connect to WAN network and the same for LANvSW uplinks.
Then and for WANvSW, create a port group with settings needed and connect to it the first leg of pfsense machine.
For LANvSW, create the needed port groups and connect them to VMs (like second leg of pfsense)
Please consider marking this answer "correct" or "helpful" if you think your question have been answered correctly.
Is there a specific reason you want iSCSI?
We use NFS both in production ( on High End Storage Array ) and in test on a FreeNAS.