VMware Cloud Community
g3uiss
Enthusiast
Enthusiast

Vcentrr server 5.5u3c failing after July ms Patches

my Server 2012 patched this week, since then  I was getting random dropouts of connection to the host, health service monitoring and host status of the vCentre Server. These also coincided with a TCPIP warning in the Server2012 event logs.

Nothing else had had changed so I removed rollip patch 2018-07 and it seems to be stable.

Any suggestions other than dont patch the server?

Reply
0 Kudos
17 Replies
diegodco31
Leadership
Leadership

Windows Patch is always one problem.

What specific windows patch?

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
g3uiss
Enthusiast
Enthusiast

July Monthly roll up security.

Reply
0 Kudos
g3uiss
Enthusiast
Enthusiast

Multiple issues with this patch reported.

So so I suspect by next months roll up it might be sorted.

Reply
0 Kudos
daphnissov
Immortal
Immortal

Other suggestion would be to migrate to the vCSA so you never have to deal with this crap again.

willc2004
Contributor
Contributor

I removed KB433815 and I'm still seeing the same problem.  Is there another patch I need to remove?

Reply
0 Kudos
g3uiss
Enthusiast
Enthusiast

KB4338380 was what I need to remove. I’m using Server 2012R1.

The patch you refer has not enough numbers ?

Reply
0 Kudos
willc2004
Contributor
Contributor

Sorry - I removed KB4338815 which didn't seem to help.  I can't find any reference to KB4338380 to compare it to my 2012R2 server.

Reply
0 Kudos
g3uiss
Enthusiast
Enthusiast

Yes that's the same patch but for Server 2012R2. As mine is a R1 its a different number, you wont be offered it. I gave the wrong patch number I removed KB4338830

Do you see any TCPIP warnings in the Server 2012 Event log, I think that's a clue ?

Reply
0 Kudos
willc2004
Contributor
Contributor

Sorry for the delay.

I'm getting Event ID 4227 randomly on the server: "TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint."  

I removed the patch in question and made sure it was hidden on server so it doesn't reinstall.  The alerts still came in for a few days and stopped for a few days - they are now coming in again.  I noticed my windows firewall was turned on which I thought I disabled so I just disabled it again.

Reply
0 Kudos
willc2004
Contributor
Contributor

Restarting all esx hosts seems to have solved my problem.  I'm not sure if the problem was the hosts were up and running for 470 straight days or that our DNS servers were updated a week or two before we started having this problem.  The new dns servers were added to the hosts thru vcenter but nothing was restarted.

Reply
0 Kudos
cwatarai
Contributor
Contributor

I'm seeing this same issue with our VCenter 5.5 running on Windows 2008 R2 ever since the July 2018 patches from Microsoft got installed.  The following updates were loaded on our server via WSUS...

KB4338818, KB4338823, KB4340004, KB4340556

KB4338818 is specified as the "Security Monthly Quality Rollup for Windows Server 2008 R2" and KB4338823 is "Security Only Quality Update for Windows Server 2008 R2".  I'm wondering whether KB4338823 is a "subset" of KB4338818.  If so, am I uninstalling both of these KBs to fix this issue?  Hopefully this problem gets resolved with the August 2018 MS updates but has VMware confirmed that this issue indeed exists for the July patches?  I'm almost positive this is the problem since we patched on the morning of 7/17/18 after which this problem started.

Reply
0 Kudos
huckfinn
Contributor
Contributor

We have vCenter 6.0 and it went crazy after July updates install. We ended up uninstalling every updates from July. Once the server reboots after install of the updates make sure you check the installed updates again since Windows seem to install other update(s) automatically when some specific updates are uninstalled. See my issue in the link below

KB4338815 KB4284815 KB4339093 KB4338824 vCenter Server 6.0 U2 Host Network Connection Failure

Reply
0 Kudos
cwatarai
Contributor
Contributor

Also, problem seems to come and go as though something in the environment is causing more frequent host disconnects, service failures, etc.  Didn't see this issue for a couple days but it's back again today.  Anyone open up a case with VMware Support yet to find out why July 2018's updates are doing this?

Reply
0 Kudos
msripada
Virtuoso
Virtuoso

This is Microsoft security patch issue due to port. Exhaustion and vcenter has nothing to do.. This is impacted and Microsoft SQL itself is impacted as well..

Microsoft has released in their blog with fixes

.

Thanks.,

MS

Reply
0 Kudos
bondsourajit
Contributor
Contributor

For Port exhaustion you can modify the registry

  • Open registry editor
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
  • Right click and select “new”
  • REG_DWORD, Decimal,  value=30
  • Rename it “TcpTimeWaitDelay”

Run this command in command prompt(admin):

netsh int ipv4 set dynamicport tcp start=20000 num=45500

There are some hotfixes released at last week of July.

https://support.microsoft.com/en-us/help/4345913/access-denied-errors-after-installing-july-2018-sec...

Reply
0 Kudos
horvathszabolcs
Contributor
Contributor

Just a quick update: we're experiencing issues since we applied July patches on 2012 R2 environment.

The following patches were installed:

Security Update  KB4338613

Security Update  KB4338824

Security Update  KB4339093

Luckily, we have several environments with different patching strategies, so we're pretty sure to know what went wrong.

Environment without these patches installed are ALL working fine.

Environment with these patches installed are ALL working with issues.

Symptoms:

- VSPP cannot logon to the vcenter with localuser@vsphere.local and raises [["07:05:12","vCenter Server","vcenter.acme.corp","com.vmware.vim25.InvalidLoginFaultMsg: Cannot complete login due to an incorrect user name or password."]] errors.

- vCenter loses connection with ALL hosts in the cluster (not at the same time) "Host is not responding error" and comes back within a few seconds

vpxd.log are full with messages like:

2018-08-14T02:26:38.951+02:00 [07336 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:000000000f5e7ec0, TCP:1.2.3.4:443>; cnx: (null), error: class Vmacore::SystemException(Only one usage of each socket address (protocol/network address/port) is normally permitted)

We removed all three patches from one vcenter instance. Since then, it's working fine.

Now we're trying to pinpoint which patch cause the problem. Yesterday we removed KB4338613, but the problem came back in a few hours.

Today morning we removed KB4338824.

Too soon to tell, but in the last 3 hours, we had no issues with VSPP or vCenter-ESXi connection.

I'll keep this thread updated, if anyone interested.

As far as I know, no statement from vmware (yet).

Reply
0 Kudos
horvathszabolcs
Contributor
Contributor

Okay, systems are running for more than 24 hours without any issues after removing the following patch:

2018-07 Security Only Quality Update for Windows Server 2012 R2 for x64-based Systems (KB4338824)

Reply
0 Kudos