esxi 6.5 domain join with smb 2.0?

[robertrosit]

host: esxi 6.5.0 vmkernel release build 5969303
domain controller: server 2016 standard with latest cumulative update
no firewall in place.

when adding the esxi host to the domain with SMB 1.0 protocol (default setting), there are no issues. it works fine, tested via webgui and via command line.

unfortunately in our environment we want to get rid of SMB 1.0 completely and uninstall it from the domain controllers. so we followed this (ESXi 6 hangs when joining Active Directory Domain ) post to modify likewise to use smb 2.0

after this change the domain join via webgui "hangs" and does not complete. then the entire webgui becomes unresponsive and from this moment on, we have to reboot the esxi host.

we followed various troubleshooting guides, like this one  (ESXi and Likewise – troubleshooting guide – part 2 – Virtual Village )
for example, we disabled ipv6 on the domain controller like suggested, we disabled the windows firewall on the DC, we disabled the esx firewall.... did not help. dns config, hosts file, etc.. should all be fine and good, as domain join with SMB1.0 works.

to get better debugging info we then tried a manual join with this procedure:

/usr/lib/vmware/likewise/bin/lwsm restart lwio
/etc/init.d/lwsmd stop
/etc/init.d/lwsmd start
esxcli network firewall unload
/usr/lib/vmware/likewise/bin/lwsm set-log file /var/log/likewise.log
/usr/lib/vmware/likewise/bin/lwsm set-log-level debug
/usr/lib/vmware/likewise/bin/domainjoin-cli join domain.local domainadmin@domain.local somepassword

the command prints two messages:

     Joining to AD Domain:   domain.local
     With Computer DNS Name: HV001.domain.local

and then just hangs.

after a failed join attempt like this we have to  ps | grep lwsmd  and kill -9 *pid* - otherwise, we can't interact with lwio/lsass anymore.

the verbose logging gives the following information:

20170814141140:DEBUG:lwio:IoCreateFile():ioapi.c:218: LEAVE: -> 0x00000103 (EE = 0)
20170814141140:DEBUG:lwio:IopIpcCreateFile():ioipc.c:438: LEAVE_IF: -> 0x00000103 (STATUS_PENDING) (EE = 0)
20170814141140:DEBUG:lwio:RdrResolveToDomain():driver.c:889: Error at ../lwio/server/rdr/driver.c:889 [status: STATUS_NOT_FOUND = 0xC0000225 (-1073741275)]
20170814141140:DEBUG:lwio:RdrSocketTaskConnect():socket.c:1019: Error at ../lwio/server/rdr/socket.c:1019 [status: STATUS_PENDING = 0x00000103 (259)]
20170814141140:DEBUG:lwio:RdrSocketTask():socket.c:1246: Error at ../lwio/server/rdr/socket.c:1246 [status: STATUS_PENDING = 0x00000103 (259)]
20170814141140:DEBUG:lwio:RdrSocketRead():socket.c:1773: Error at ../lwio/server/rdr/socket.c:1773 [status: STATUS_PENDING = 0x00000103 (259)]
20170814141140:DEBUG:lwio:RdrSocketReceivePacket():socket.c:701: Error at ../lwio/server/rdr/socket.c:701 [status: STATUS_PENDING = 0x00000103 (259)]
20170814141140:DEBUG:lwio:RdrSocketDispatchPacket2():socket.c:1423: Error at ../lwio/server/rdr/socket.c:1423 [status: STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3 (-1073741629)]
20170814141140:DEBUG:lwio:RdrSocketTaskTransceive():socket.c:1134: Error at ../lwio/server/rdr/socket.c:1134 [status: STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3 (-1073741629)]
20170814141140:DEBUG:lwio:RdrSocketTask():socket.c:1251: Error at ../lwio/server/rdr/socket.c:1251 [status: STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3 (-1073741629)]
20170814141223:VERBOSE:lsass:LsaSrvIpcCheckPermissions():ipc_state.c:79: Permission granted for (uid = 0, gid = 0, pid = 72438) to open LsaIpcServer
20170814141223:VERBOSE:lsass-ipc:lwmsg_peer_log_accept():peer-task.c:271: (session:04df4955d842942b-f5af40d405e6b03c) Accepted association 0xb1016b8
20170814141223:VERBOSE:lwreg:RegDbOpenKey():sqldb.c:1068: Registry::sqldb.c RegDbOpenKey() finished
20170814141223:DEBUG:lwreg:RegDbGetKeyValue_inlock():sqldb_p.c:1227: Error at ../lwreg/server/providers/sqlite/sqldb_p.c:1227 [status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)]
20170814141223:DEBUG:lwreg:RegDbGetValueAttributes_inlock():sqldb_schema.c:846: Error at ../lwreg/server/providers/sqlite/sqldb_schema.c:846 [status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)]

STATUS_INVALID_NETWORK_RESPONSE gives me no additional clue of what's going wrong except that this may simply be a bug in likewise or esxi.

is there any way to get domain join working with SMB 2.0 ?

[msripada]

Hello Robertrosit,

Likewise is common in vCSA and ESXi host so you can try the steps mentioned in the below VMware community

Unable to join VCSA 6 u1 back to domain. Error messages are not found anywhere online.

Note : Please try it in lab first and check if that works and do not try directly on a production ESXi

Thanks,

MS

[robertrosit]

thank you for trying to help, but the post you linked just describes what i have discovered so far -> SMB1.0 (srv.sys) needs to be enabled for domain join to work.

how to make domain join work with SMB2.0 instead?

[Seniore]

Hi,

please have a look at the following entry:

Re: VMWare ESXi 6.0 Domain Integration - SMB1 Disabled on AD side through GPO, ESXi Domain Join Fail...

Login to the ESXi via SSH and execute:

"

Check Values:

    /usr/lib/vmware/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]'

Change SMB2 to be Enabled:

    /usr/lib/vmware/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]' SMB2Enabled 1

Restart lwio:

    /usr/lib/vmware/likewise/bin/lwsm restart lwio

"

[robertrosit]

we have done this (so we followed this (ESXi 6 hangs when joining Active Directory Domain ) post to modify likewise to use smb 2.0)

it results in the error i have described in the initial posting.

[msripada]

As per this KB VMware ESXi 6.0, Patch ESXi600-201706401-BG: Updates esx-base, vsan, and vsanhealth VIBs (2149955) |... 

vsphere 6.0 latest release has the smb2 enabled.. I guess the same time released update for 6.5 or the latest for 6.5 should have the same smb2 updated as well. can you update to 6.5 update 1 and check once.

Thanks,

MS

[msripada]

Its released on June 6, 2017

Windows 2012 domain controller supports SMBv2, whereas Likewise stack on ESXi supports only SMBv1.

With this release, the likewise stack on ESXi is enabled to support SMBv2.

The only release after June 6 2017 for esxi 6.5 is 6.5 update 1 but I do not see any info from the release notes but worth a try to check

VMware ESXi 6.0, Patch ESXi600-201706401-BG: Updates esx-base, vsan, and vsanhealth VIBs (2149955) |...

Thanks,

MS

[msripada]

I got it confirmed that the 6.5 Update 1 has the SMB2 enabled by default.

VMware ESXi 6.5 Update 1 Release Notes

Security Issues

• Update to the libcurl libraryThe ESXi userworld libcurl library is updated to version 7.53.1.
• Update to the NTP packageThe ESXi NTP package is updated to version 4.2.8p10.
• Update to the OpenSSH versionThe OpenSSH version is updated to version 7.5p1.
• The likewise stack on ESXi is not enabled to support SMBv2The Windows 2012 domain controller supports SMBv2, whereas likewise stack on ESXi supports only SMBv1. With this release, the likewise stack on ESXi is enabled to support SMBv2. This issue is resolved in this release.

[robertrosit]

thank you for your response.

however our esxi host is a fresh install from ISO 6.5.0 U1, done in the second week of august 2017.

The webgui says:

Version info: 6.5.0 Update 1 (Build 5969303)

this is the same version as mentioned on VMware ESXi 6.5 Update 1 Release Notes

ESXi 6.5 Update 1 | 27 JULY 2017 | ISO Build 5969303

so we're 100% sure we have the latest version in place.

still, lwregshell shows SMB2Enabled with value 0.

ook, lets try again. enable SMB2:

aand it fails with ERROR_GEN_FAILURE [code 0x0000001f]

i have then disabled the esxi firewall, and now i am back to where i started. the command just hangs, no error message, but also no domain join:

[robertrosit]

for completeness, here's the logfile:

20170829084549:INFO:netlogon:LWNetSrvGetDCTime():dcinfo.c:442: Determining the current time for domain 'DOMAIN.LOCAL'

20170829084549:INFO:netlogon:LWNetSrvGetDCName():dcinfo.c:97: Looking for a DC in domain 'DOMAIN.LOCAL', site '<null>' with flags 10

20170829084550:DEBUG:lsass:LsaSetSMBCreds():lsakrb5smb.c:174: Switching default credentials path for new access token

20170829084550:DEBUG:LwKrb5SetThreadDefaultCachePath():lwkrb5.c:410: Switched gss krb5 credentials path from FILE:/tmp/krb5cc_0 to FILE:/tmp/tktp1kjHc

20170829084550:INFO:netlogon:LWNetSrvGetDCName():dcinfo.c:97: Looking for a DC in domain 'DOMAIN.LOCAL', site '<null>' with flags 1001

20170829084550:DEBUG:netlogon:LWNetGetPreferredDcList():lwnet-plugin.c:184: Error at ../netlogon/server/api/lwnet-plugin.c:184 [code: 2453]

20170829084550:DEBUG:netlogon:LWNetSrvGetDCNameDiscoverInternal():lwnet.c:887: Error at ../netlogon/server/api/lwnet.c:887 [code: 2453]

20170829084550:INFO:netlogon:LWNetFilterFromBlackList():lwnet.c:725: Filtering list of 1 servers with list of 0 black listed servers

20170829084550:DEBUG:lwio:RdrCreateContext():driver.c:475: Created op context 0x854e828 for IRP 0x854e790

20170829084550:DEBUG:lwio:RdrCreateContext():driver.c:479: Created op context 0x854e918

20170829084550:DEBUG:lwio:RdrCreateContext():driver.c:479: Created op context 0x854ea50

20170829084550:DEBUG:lwio:RdrTreeConnect():connect.c:1106: Tree connect context 0x854ea50 will continue 0x854e918

20170829084550:DEBUG:lwio:RdrTransceiveNegotiate():connect.c:899: Error at ../lwio/server/rdr/connect.c:899 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:RdrTreeConnect():connect.c:1151: Error at ../lwio/server/rdr/connect.c:1151 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:RdrDfsConnectAttempt():dfs.c:559: Error at ../lwio/server/rdr/dfs.c:559 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:RdrDfsConnect():dfs.c:751: Error at ../lwio/server/rdr/dfs.c:751 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:IoCreateFile():ioapi.c:218: LEAVE: -> 0x00000103 (EE = 0)

20170829084550:DEBUG:lwio:IopIpcCreateFile():ioipc.c:438: LEAVE_IF: -> 0x00000103 (STATUS_PENDING) (EE = 0)

20170829084550:DEBUG:lwio:RdrResolveToDomain():driver.c:889: Error at ../lwio/server/rdr/driver.c:889 [status: STATUS_NOT_FOUND = 0xC0000225 (-1073741275)]

20170829084550:DEBUG:lwio:RdrSocketTaskConnect():socket.c:1019: Error at ../lwio/server/rdr/socket.c:1019 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:RdrSocketTask():socket.c:1246: Error at ../lwio/server/rdr/socket.c:1246 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:RdrSocketRead():socket.c:1773: Error at ../lwio/server/rdr/socket.c:1773 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:RdrSocketReceivePacket():socket.c:701: Error at ../lwio/server/rdr/socket.c:701 [status: STATUS_PENDING = 0x00000103 (259)]

20170829084550:DEBUG:lwio:RdrSocketDispatchPacket2():socket.c:1423: Error at ../lwio/server/rdr/socket.c:1423 [status: STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3 (-1073741629)]

20170829084550:DEBUG:lwio:RdrSocketTaskTransceive():socket.c:1134: Error at ../lwio/server/rdr/socket.c:1134 [status: STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3 (-1073741629)]

20170829084550:DEBUG:lwio:RdrSocketTask():socket.c:1251: Error at ../lwio/server/rdr/socket.c:1251 [status: STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3 (-1073741629)]

[msripada]

Hello Robert,

Thank you for the log snippet.

SMB2 is already enabled on the ESXi host as per the screenshot (may be you might have corrected it if it is not)

Ensure that the SMB1 is disabled on the AD.

Ping the domain from ESXi host and check if the ping is responding from specific DC or is it load balanced to multiple domain controllers, possibly, there is a chance that the request might be reaching to a DC (which might be unreachable)

Ensure DNS is configured for AD before joining and then try to join.

Thanks,

MS

[robertrosit]

There is only one domain controller, the logfile confirms this:

20170829084550:INFO:netlogon:LWNetFilterFromBlackList():lwnet.c:725: Filtering list of 1 servers with list of 0 black listed servers

i can lookup and ping the domain correctly from esxi.

what do you mean with "Ensure DNS is configured for AD before joining and then try to join" ? in esxi configuration, i set the DNS server to be the ip of the DC (which is the DNS server), and set a hostname in FQDN format (hv001.domain.local). anything else to do in this regard?

both esxi and domain controller are fresh installs with most recent patch levels (this is a lab environment).

i started monitoring the connection between esxi and DC, i can see ms-ds-smbv2, kerberos, ldap, ntp and dns packets. all looks good.

still, the domain join command just hangs, no domain join happens, and the likewise logfile mentions 20170829084550:DEBUG:lwio:RdrSocketTask():socket.c:1251: Error at ../lwio/server/rdr/socket.c:1251 [status: STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3 (-1073741629)] in the end.

[jfene72]

I have, give and take, your same lab setup the only difference being that I'm running AD on Windows 2012 Server; just 1 DC. I gave this a go and disabled SMB 1.0 on the DC and enabled SMB 2.0 on an ESXi 6.5 U1 host. I then joined ESXi using the domainjoin-cli command and found no issues. Disjoined and rejoined it again just to test it twice.

I'm running everything on the same subnet. Firewall is enabled on ESXi but it is disabled on the Windows DC and there's no additional firewalling in between. ESXi is using the DC's DNS server for name resolution.

I know this is not a solution but maybe it will help you narrow down the issue.

[robertrosit]

hy jfene72

thanks for trying to help (i already found some of your info on the net, i recognize your domain name)

out of options i went the far way and installed a fresh 2012R2 server, promoted it to a new domain ("test.local"), and put it in the same ip subnet as the esxi server. so the settings should be similar.

unfortunately i get the exact same error in esxi log (STATUS_INVALID_NETWORK_RESPONSE = 0xC00000C3)

so i went a step further, disabled SMB2 on esxi side, and enabled SMB1 on DC side - just to make sure that at least this works fine. and it does.

domain join to test.local on server2012r2 works flawless with smb1.

then i disabled SMB1 on the DC with powershell, and tried again.

and domain join still worked.....

... untill i rebooted the DC.

could you be so kind and test this again after rebooting your dc? it seems the powershell command will not have any effect before that.

[jfene72]

So, I rebooted the DC and verified that SMB 1.0 was still disabled on the DC and that SMB 2.0 was still enabled on ESXi. Both were. I then tried rejoining ESXi and this time it failed with the following:

Error: ERROR_GEN_FAILURE [code 0x0000001f]

Under syslog, you'll find multiple entries like this one:

2017-08-30T14:59:46Z lwsmd: [lsass] Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2111170

It took something like 20 seconds before throwing the error.

So I re-enabled SMB 1.0 and disable SMB 2.0 on the DC using Powershell. I did the same on ESXi using lwregshell. The ESXi host joined the domain just fine after doing this. Why? Your guess is as good as mine!

I'll try and dig deeper tomorrow if I find some time to spare.

Hope this helps.

[Esprimo1]

same issues here.

ESXI 6.5U1 build 5969303

SMB2 enabled in ESXI

SMB1 disabled in Windows AD. (Two AD Servers, one 2008r2, one 2012)

nslookup domain.local

gives both server back.

ntp.conf has both servers with DNS included.

/usr/lib/vmware/likewise/bin/lwsm restart lwio
/etc/init.d/lwsmd stop
/etc/init.d/lwsmd start
esxcli network firewall unload
/usr/lib/vmware/likewise/bin/lwsm set-log file /var/log/likewise.log
/usr/lib/vmware/likewise/bin/lwsm set-log-level debug
/usr/lib/vmware/likewise/bin/domainjoin-cli join domain.local administrator somepassword

the command prints two messages:

     Joining to AD Domain:   domain.local
     With Computer DNS Name: xxx.domain.local

and then just hangs.

bye the way, the VCSA 6.5u1a is joining. so AD admin can log into VCSA.

and the other stuff can join as well.

any new hints?

[JudgementDay]

Hi.

I also have the requirement to disable SMBv1 and experiencing the same symptoms when trying to join ESXi 6.5 U1

With SMBv1 disabled on the Windows side (2012 R2 for me) and SMBv2 enabled for ESXi, the ESXi Join domain will fail with the same errors you detail.

The ESXi likewise service appears to crash then the host becomes unresponsive. I have to hard boot the host to recover!

Did you get any further with your issue? Any response from VMware support?

Thanks.

[08Martin80]

Thats what happened here:

We have 3 HP DL380 Servers. One is Gen8 and two are Gen7.

Database is hosting on EMC VNXe

(Yes i know HP Gen7 Servers are not supported with ESXi 6.5)

One GEN7 is a cold Backup Server and used for testing, the other Gen7 Servers is only for VCSA 6.5 and some other stuff.

Produktion Server is Gen8

After switching off SMB1 in Windows AD Server we decided to Upgrade ESXi 5.5 to newest ESXi 6.5U1.

I used VCSA Updatemanager to upgrade the testing Server. After resolving the PSOP Issues with HP-SMX-Provider, it was running fine.

but I can't join the Domain. Some issues as above.

The next was the VCSA 6.5 Host. I decided to install it with a modified HPE ISO on USB Stick (without HP-SMX-Provider).

It was easy and works, but without Domain-Join.

Last one: The Main System. Upgrade with full HPE ISO. It works and after Reboot it was joined by its own in the AD Domain.

So perhaps its only an issue with old not supported Hardware?

I tried to figure out if there are some wrong drivers, some VIB differences in Gen7 and Gen8 Servers, but all i can see is nearly the same.

[heman013]

I can only state Robertrosit is completely right with what he writes. We have the same situation and no solution for now.

I upgraded to the latest build available, but still no solution.

vmware -vl
VMware ESXi 6.5.0 build-6765664 VMware ESXi 6.5.0 Update 1

SMB 1 is disabled on our domain controllers, which is rather I good thing, I would say.

/var/log/syslog

2017-11-16T10:20:59Z lwsmd: [lsass] Joining domain domainname 2017-11-16T10:20:59Z lwsmd: [netlogon] Looking for a DC in domain 'domainname', site '<null>' with flags 10 2017-11-16T10:20:59Z lwsmd: [netlogon] Filtering list of 27 servers with list of 0 black listed servers 2017-11-16T10:20:59Z lwsmd: [lsass] Affinitized to DC 'domcontrollername.domainname' for join request to domain 'domainname'' 2017-11-16T10:20:59Z lwsmd: [netlogon] Determining the current time for domain 'domainname'' 2017-11-16T10:20:59Z lwsmd: [netlogon] Looking for a DC in domain 'domainname'', site '<null>' with flags 10 2017-11-16T10:20:59Z lwsmd: [netlogon] Looking for a DC in domain 'domainname'', site '<null>' with flags 1001 2017-11-16T10:20:59Z lwsmd: [netlogon] Filtering list of 27 servers with list of 0 black listed servers 2017-11-16T10:21:27Z lwsmd: [lsass] Failed to run provider specific request (request code = 8, provider = 'lsa-activedirectory-provider') -> error = 31, symbol = ERROR_GEN_FAILURE, client pid = 67767 2017-11-16T10:21:39Z lwsmd: [lsass] Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 68331 2017-11-16T10:22:39Z lwsmd: [lsass] Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 68337

/usr/lib/vmware/likewise/bin/lwsm set-log-level verbose

20171116102420:VERBOSE:netlogon: Getting address for 'domcontrollername' 20171116102420:INFO:netlogon: Filtering list of 27 servers with list of 0 black listed servers 20171116102420:VERBOSE:netlogon: Getting address for 'domcontrollername' 20171116102420:VERBOSE:netlogon: Getting address for 'domcontrollername' 20171116102439:VERBOSE:lsass: Permission granted for (uid = 0, gid = 0, pid = 68644) to open LsaIpcServer 20171116102439:VERBOSE:lsass-ipc: (session:10128abdd4e34055-ea3550bd19beacc8) Accepted association 0xac05148 20171116102439:ERROR:lsass: Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 68644 20171116102439:VERBOSE:lsass-ipc: (assoc:0xac05148) Dropping: Connection closed by peer 20171116102443:ERROR:lsass: Failed to run provider specific request (request code = 8, provider = 'lsa-activedirectory-provider') -> error = 31, symbol = ERROR_GEN_FAILURE, client pid = 67769 20171116102443:VERBOSE:lsass-ipc: (assoc:0xac04eb8) Dropping: Connection closed by peer 20171116102501:VERBOSE:lsass: Permission granted for (uid = 0, gid = 0, pid = 68648) to open LsaIpcServer 20171116102501:VERBOSE:lsass-ipc: (session:49f43137bc9ef1b4-6b96affaf10ccbf6) Accepted association 0xac04eb8 20171116102501:VERBOSE:lsass-ipc: (assoc:0xac04eb8) Dropping: Connection closed by peer

[seagull123]

Same issue here!

Running ESXi 6.5.0 update 1 with smb2 enabled on the hosts and smb1 enabled on the windows win2016 domain controllers. If I turn of smb2 on the esxi hosts the AD joining operation will work fine.