11 Replies Latest reply on Apr 26, 2016 6:40 AM by raiford

    No DNS in NAT (vmnet-natd broken)

    Manouchehri Lurker

      As of glibc 2.22, vmnet-natd is unable to look up external results with the default settings.

       

      Temporary workaround

       

      Open /etc/vmware/vmnet8/dhcpd/dhcpd.conf and change:

       

      option domain-name-servers 172.20.142.2;

       

      to

       

      option domain-name-servers 8.8.8.8;

       

      Bug Reports

       

       

      Debugging

       

      The issue seems to be with vmnet-natd not using res_init.c properly.

       

      https://sourceware.org/git/gitweb.cgi?p=glibc.git;a=history;f=resolv/res_init.c;hb=5e7fdabd7df1fc6c56d104e61390bf5a6b526c38

       

      My guess is that these are the changes that broke it.

       

      https://sourceware.org/git/gitweb.cgi?p=glibc.git;a=blobdiff;f=resolv/res_init.c;h=02590fb1fdbb15c9ff20295d7a96d2d426752…

       

      There's roughly only a dozen calls where res_init.c is being used, so it shouldn't be too difficult to debug. The only issue is that vmnet-natd is stripped; if someone could send me the symbols or a non-stripped binary, that would be great.

       

      ltrace -f -p 1337 # vmmnet-natd PID here

       

      2020 strncpy(0x7ffff83451e0, "ctftime.org", 64) = 0x7ffff83451e0

      2020 malloc(16) = 0x2644cb0

      2020 malloc(168) = 0x2644380

      2020 pipe(0x7ffff83451d0) = 0

      2020 fcntl(6, 3, 0x429280, 0x7ff3f92482e7) = 0

      2020 fcntl(6, 4, 2048, 0x7ff3f97221a2) = 0

      2020 malloc(65884) = 0x2654640

      2020 strncpy(0x26443e8, "ctftime.org", 64) = 0x26443e8

      2020 pthread_create(0x7ffff83451c8, 0, 0x428ba0, 0x2644cb0) = 0

      10054 mempcpy(0x7ff3f2aeebb0, 0x26449c6, 8, 7) = 0x7ff3f2aeebb8

      2020 pthread_detach(0x7ff3f2af0700, 0x7ff3f2aeffb0, 0, 0x800000 <unfinished ...>

      10054 mempcpy(0x7ff3f2aeebb8, 0x26449ce, 4, 3 <unfinished ...>

      2020 <... pthread_detach resumed> ) = 0

      10054 <... mempcpy resumed> ) = 0x7ff3f2aeebbc

      2020 poll(0x26121b0, 6, 1000, 0x80412 <unfinished ...>

      10054 __res_ninit(0x7ff3f2aefca0, 103, 11, 0 <unfinished ...>

      2020 <... poll resumed> ) = 1

      2020 gettimeofday(0x7ffff834d390, 0) = 0

      2020 read(3, "", 32768) = 71

      2020 malloc(199) = 0x2644430

      2020 memcpy(0x26444b0, "\0PV\3604\330\0\f)\335\0049\b\0E\0\09]\241\0\0\200\021\327:\300\250B\205\300\250"..., 71) = 0x26444b0

      2020 mempcpy(0x7ffff8344b80, 0x26444e6, 8, 7 <unfinished ...>

      10054 <... __res_ninit resumed> ) = 0

      2020 <... mempcpy resumed> ) = 0x7ffff8344b88

      10054 __res_iclose(0x7ff3f2aefca0, 0, 2, 0 <unfinished ...>

      2020 mempcpy(0x7ffff8344b88, 0x26444ee, 4, 3 <unfinished ...>

      10054 <... __res_iclose resumed> ) = 0

      10054 __errno_location() = 0x7ff3f2af0680

      10054 __res_nclose(0x7ff3f2aefca0, 0, 0, 0) = 0xffffffff

      10054 __errno_location() = 0x7ff3f2af0680

      10054 memcpy(0x265473c, "o\0\0\0E\0\09]\240\0\0\200\021\327;\300\250B\205\300\250B\002\344j\05\0%\210\237"..., 61) = 0x265473c

      10054 write(10, "\001", 1) = 1

      10054 close(10) = 0

      10054 free(0x2644cb0) = <void>

      10054 +++ exited (status 0) +++

      2020 <... mempcpy resumed> ) = 0x7ffff8344b8c

        • 1. Re: No DNS in NAT (vmnet-natd broken)
          raiford Enthusiast

          I am experiencing this same problem.  I have found many other reports of this online, but no solution yet.  Using an external DNS works, but for those of us who move between networks, this is not a good option.  Using a Google DNS server will prevent me from accessing local servers.  Hardcoding my local DNS server at work will not work at home, etc.

           

          I have not found any reference to VMware acknowledging that the issue exists.

           

          EDIT:

           

          I'm running Ubuntu 16.04 LTS and VMware Workstation Pro 12.1.0 build-3272444.  When the new 16.04 LTS is officially released at the end of this month this issue will likely affect a lot more people.  Hopefully VMware can fix it beforehand.

          • 2. Re: No DNS in NAT (vmnet-natd broken)
            Manouchehri Lurker

            Sooner or later this is going to hit all Workstation users on Linux. I think it would be wise to get on top of this before it becomes a widespread critical issue.

             

            I'd be quite happy to assist VMware in troubleshooting more, but I'm not interested in paying for a support license to give free work. If any developers are reading this, shoot me an email and I'll send my number (Google'ing my name will give contact info).

            • 3. Re: No DNS in NAT (vmnet-natd broken)
              www23456 Novice

              Are there any entries in /etc/resolv.conf? I tested the xenial.iso u1604 hsot with WS12.1.0, VM is linux, and there is 127.0.1.1 added automatically by

              network manager: netstat -an |grep 127; 127.0.1.1:53 LISTEN

              • 4. Re: No DNS in NAT (vmnet-natd broken)
                Manouchehri Lurker

                127.0.1.1:53 inside the guest is correct. I'm using that in other systems and it's fine.

                 

                From the ltrace I posted, it appears that the vmnet driver on the host might be the issue. Changing domain-name-servers on the host only fixes the bug for me, which implies it's the host.

                 

                Have you tried using my suggested temporary fix?

                 

                I'm tempted to try to use VirtualBox's NAT server with VMware, not sure how well that's going to work though.

                • 5. Re: No DNS in NAT (vmnet-natd broken)
                  www23456 Novice

                  actually the ubuntu1604 beta2 host nat working fine with linux VM.  On my host, the 127.0.1.1 was added

                  automatically by network manager and there is 127.0.1.1:53 tcp in 'listen'state. Basically  the nat will

                  forward the dns query to 127.0.1.1:53 and get response from the local 127.0.1.1 resolver and forward back

                  to VM.

                  on my host: # ldd --version showing glibc 2.23,

                  Are you using window VM?  what is your glibc version?

                  • 6. Re: No DNS in NAT (vmnet-natd broken)
                    Manouchehri Lurker

                    All of us are using Linux as our host (Windows does not use glibc). What is your /etc/resolv.conf and /etc/vmware/vmnet8/dhcpd/dhcpd.conf on your host machine?

                     

                    Both Linux & Windows guests are broken for most of us. With Linux guests, attempting a ping will eventually result in VMware returning the correct IP (but by appending .localdomain). Using dig or nslookup without .localdomain will always fail.

                    • 7. Re: No DNS in NAT (vmnet-natd broken)
                      raiford Enthusiast

                      I was hoping VMWare would acknowledge this issue but it seems they either don't recognize it or have decided it is not important.  I went ahead and bit the bullet and logged an official support issue for this.  I have referenced this thread since it has the best description of the error.  Hopefully others who have additional information will also post here rather in other threads so we can keep the information gathered in one place.

                      • 8. Re: No DNS in NAT (vmnet-natd broken)
                        raiford Enthusiast

                        FYI, my ticket has been archived awaiting the next maintenance release.  I've been told "The information is already there with Engineering team and they are looking into the issue.", so hopefully we will see this resolved sooner than later.

                        • 9. Re: No DNS in NAT (vmnet-natd broken)
                          dariusd Virtuoso
                          VMware EmployeesUser Moderators

                          Workstation 12.1.1 has just been released, and the problem you've encountered seems to be listed as a resolved issue.  Try installing Workstation 12.1.1 and let us know if the problem is resolved for you.

                           

                          VMware Workstation 12 Pro Version 12.1.1 Release Notes

                           

                          Download VMware Workstation Pro

                           

                          Cheers,

                          --

                          Darius

                          • 10. Re: No DNS in NAT (vmnet-natd broken)
                            Tanker Bob Novice

                            Thanks! Version 12.1.1 fixed the NAT problem for me in a Windows 10 guest on Ubuntu 16.04 64-bit host. Nice work!

                            • 11. Re: No DNS in NAT (vmnet-natd broken)
                              raiford Enthusiast

                              I can also confirm this update resolved my problem.  Thanks!