Stopping SSH on ESXi 5.1 produces error

[Slingsh0t]

Getting this error message when trying to stop the SSH service on newly upgraded ESXi 5 hosts: 

Even though the error appears SSH is actually stopped on the host.  vCenter never seems to be properly aware of the hosts service status.

- This is happening on BOTH ESXi hosts in the environment.

- The hosts have been rebooted, also removed and re-added to vCenter.

- The service has been stopped/started from the DCUI, powerCLI and vSphere client, none of these seems to remove this error message or stop vCenter reporting the SSH service as running.

After additional testing this looks to be caused by the attempt to disable the SSH firewall rule when the SSH service is stopped.  When manually trying to disable the SSH rule on the firewall we also get the above error.  When disabling any other rule in the firewall policy, no error is produced.  It only seems to be related to SSH and the firewall.

Has anyone else seen this behaviour and have an idea of what is causing it and/or a fix?

[crmk]

Unfortunately I don't have a fix, but I have noticed the same behavoir on our ESXi 5.1 hosts.

[Munky711]

When you enable SSH for the first time the firewall rule "SSH Server" is enabled. With previous ESXi servers, when you disabled SSH the firewall rule was also disabled. Now with ESXi 5.1, this firewall rule has been made a "required" rule. So when you enble it, you no longer can disable it... Don't know why they changed this.

How do you fix this? SSH into the server and log on as root. Go to /etc/vmware/firewall and edit the "service.xml" file. Change the required setting for the "sshServer" to false.

  <service id='0000'>

    <id>sshServer</id>

    <rule id='0000'>

      <direction>inbound</direction>

      <protocol>tcp</protocol>

      <porttype>dst</porttype>

      <port>22</port>

    </rule>

    <enabled>true</enabled>

    <required>false</required>

  </service>

Save the file. Next enter the command: esxcli network firewall refresh. Exit the SSH session. Now you can disable SSH and the firewall rule will be disabled also.

Good luck!

[gaspipe]

Getting the same error and this looks like the solution, but how did you save service.xml? After editing it with vi and trying to save (:wq!), it says "operation not permitted"..

[Munky711]

Had the same problem. In the end I used "WinSCP". Logged in with root account, navigated to the location of the service.xml and downloaded the file to my computer. Edited the file + save and uploaded it again with overwrite. That did it.

[gaspipe]

Yes, that did it. Would give you the points if I could.

I should mention though that this change (in service.xml) is not persistent after reboots.

[Munky711]

gaspipe wrote:

Yes, that did it. Would give you the points if I could.

I should mention though that this change (in service.xml) is not persistent after reboots.

Hadn't thought of that. That's to bad. Hopefully we'll get an official VMware fix in the near future...

[admin]

I would not recommend to follow these instructions as this change was made intentionally. This change avoids an issue where the Firewall would block the port 22 and kill all exiting sessions, in case a SSH session times out.

This is a cosmetic issue and VMware is aware of it and working on a solution.

Thanks.

Tobias Sutor

VMware

[GregAndo]

The steps I used to achieve the above after logging in via SSH were as follows:

cd /etc/vmware/firewall

cp service.xml service.xml.bak

cp service.xml service.xml-mod

chmod 644 service.xml-mod

vi service.xml-mod

[Make config modifications as specified above]

cp service.xml-mod service.xml

esxcli network firewall refresh

rm service.xml-mod

[Stop SSH through the GUI]

As specified above, this may not be a recommended solution, however I like to see host warnings so that I can manage my infrastructure better, and I want to see if SSH has been left on (or enabled unexpectedly) and correct the issue accordingly.  This holds greater priority to me than making a firewall less overzealus.

If what was said above is correct (and I haven't tested it but intend to later today - will try to remember to supply my results) - a solution could be to restart the host immediately after after making these changes and stopping the SSH service (and before your maintenance window ends...surely you are in maintenance mode, right?).  Presumably, the configuration might then be reverted but because SSH is already disabled, there would be no warning...

I wonder if the hotfix for this is out...can anyone from VMware provide us a linky??

[dcorrigan1]

Yes, this is an annoying bug. I have also tried the last solution. modifying the services.xml and stopping SSH service, but I still get the same "cannot change the host configuration" error afterwards. The host still says SSH is running, sshd does stop however.

[GregAndo]

Did you follow all the steps?  Especially:

esxcli network firewall refresh

(after making all the changes).

If not, are you running 5.1?

[cffit]

So, I recently upgraded to 5.1 all this time later... and I have this issue too.  It's almost five months since people first started having the issue and there is still no official fix from VMware?  This is pretty sad.  I was told by tech support that this would be fixed in the first update for 5.1 so I guess I'll have to wait.  I hope they fix the buggy snapshot alerts and all the other issues that came along with 5.1.  I wish I'd stayed at 5.0.

[dcorrigan1]

Ya.. I wish I would have stayed at 5.0 myself. Too many bugs with vCenter and vSphere 5.1.

I simply do not run SSH on any of my 5.x hosts now though, so no problems.

[CarlAU]

Hello All,

I recently upgraded to 5.1 and had the same error. As you all know disabling alerts is not ideal so I logged a support call for this.

VMware sent me to this article: http://kb.vmware.com/kb/2037544

I worked through this fix on all hosts and it has rectified the problem. I can enable SSH, receive the alert that SSH is on, turn off SSH with no error and have the alert clear.

VMware inform me that 5.1 Update 1 will be out soon which will include a proper fix for this problem.

Regards,

Carl

[cffit]

I believe that "fix" they provide is wiped out after the host reboots and you'd have to go back in and do this after each reboot.

[CarlAU]

Hi Cffit,

I just restarted a host and tested this.

The fix does persist through a restart.

Regards,

Carl

[Jairon]

Does anyone know if 5.1 Update 1 resolves this issue? The release notes for update 1 mention:

"ESXi hosts might retain older version of the /etc/vmware/service/service.xml file after upgrade
When you modify the etc/vmware/service/service.xml, which has a sticky bit set, and then perform an ESX/ESXi upgrade from 4.0 to 5.1, the old service.xml files causes compatibility issues. This happens because the old service.xml file is retained by the ESXi host even after the upgrade.

This issue is resolved in this release."

[CarlAU]

Hi Jairon,

The VMware support person told me that 5.1 Update 1 would resolve this issue.

I have not applied it yet but I assume it should.

[GeoPerkins]

Very old thread. This is still a problem in ESXi 6.5 U2 !!!!

Created a new thread, similar topic here: Unable to disable firewall rule for SSH Server - message is Failed cannot change the host configurat...