esxi 6 weird problems with internet connections on vms

[wbf0007]

Hi, my school got a server with esxi 6 firmware and has me installing vms on it to set up a network for our computer students. I have installed several vms, and have them connected through a network using pfsense.

The network looks like this:

The problem I am having with the network is that all of the computers can only partially access the web. By that I mean you can connect to websites using ftp or https, but you can't access websites using http. You also can't update (both windows and linux are unable to update). Windows can't even activate (although for some reason 10 and server 12 did, so I guess they use a different protocol for activating). I can ping any site, and each vm is setup with a static ip and the dns is the ip of pfsense (i can use automatic ip as well, as pfsense has a pool to use too, but the same problems occur whether I'm using static or dynamic). I have nmapped both the vms and the server (what little I could for the server since it is technically on the school network and we can't use mapping software on there, so all I could do was check for open ports, and it says that the server has ports 80 and 443 open, so technically I should be able to connect with http, but for some reason I can't).

I'm technically working part time through a third party, so I can't ask the school for help (as I have tried to ask before but they said we couldn't connect to the internet period).

Vmnic0 has internet access with 1 ip, and pfsense is using default settings for nat along with masking the mac so that we can access the internet to update the vms (which atm isn't happening as updates aren't working). Vmnic1 is connected to an access point rather than an ip address, as after updating the vms we will disconnect them from the world and let them be in their own little playground. Right now I just need to be able to connect and update them/download whatever programs we need for them.

So my question is how do I fix it so that I can update the vms as well as be able to access http (port 80).

EDIT:

another thing that me and the teacher i'm working under tried was ssh into pfsense, and use curl in the shell. https displayed information, but http gave this error:

curl http://www.google.com

curl: (56) Recv failure: Connection reset by peer

Also got a (55) error from it too

EDIT 2:

I have also attached a tcpdump that I did by ssh into the server, and running tcp dump and then telling it to only look for http requests (did it trying to connect to gamespot as that site doesn't seem to have https, so it didn't auto connect to the site using https) :

# tcpdump-uw -i vmk0 -s 1514 port 80  was the command I used.

[continuum]

That sounds like a problem with the pfSense firewall rules.
Are you sure it does not block port 80 ?

[UmeshAhuja]

Hi,

Its definitely look like firewall issue.

1) Cross verify the firewall rules whether http is allowed or blocked.

2) Also cross check the VMs , Windows firewall should not be in running mode.

3) Also cross check whether http is able to listen port 80 or is something else configured at routing / network level.

[wbf0007]

I have a rule in pfsense to allow all traffic for lan to wan and vice versa. I've tried adding in a rule that specifically said port 80 as well, but that didn't change anything.

[wbf0007]

I've tried turning off windows firewall in the windows vms, but that didn't change anything, the problem still persisted. I have a rule in pfsense to allow all traffic from wan to lan and vice versa, and even tried one specifically targeting port 80 and no luck there either. The tcpdump was done using esxi's command line, so technically it was on the outside of pfsense and the vms (i think).

I've seen some posts online about how people have had trouble with tcp traffic with pfsense and esxi/hyper-v, and the problem was with the broadcom nic and the tcp overloading, but I can't figure out how to turn that off. I've tried using the help topics on here that I've seen, and it doesn't turn it off, it gives the error that the function is not available, and there isn't an option in bios to turn it off, so unless there is a switch or something on the nic itself I am not sure how to turn that off. I did turn it off on the pfsense side, but it's the esxi side I can't.

I've done nmap within the vms, and I can get the vms to listen on port 80, and pfsense does show traffic happening, although I can't tell if port 80 is going both ways or not.

[wbf0007]

After some more testing, we have come to find that ftp passive mode works, but active mode doesn't, and ssh works on the lan, but when trying to connect anywhere on the external network, it doesn't. And while https works, any http port doesn't (80, 81, 8000, 8080).

[wbf0007]

Is there a default firewall setting with esxi that would somehow block http or parts of ftp for the vms? I haven't really looked at the firewall for esxi about that (like maybe something that was in the hardening guide that could have potentially blocked the http ports?

[shumy]

I'm having the same problem with similar configuration, did  you reach a solution?

[shumy]

I will hijack this thread, since my problem is exactly the same!

I have also post in the pfSense forum: https://forum.pfsense.org/index.php?topic=124088.0 because I don't really know the problem origin!

Can ESXi interfere with pfSense connections because it's in the same IP interface?