Hello all
We have encountered an odd problem with syslog after upgrading vCenter 6.5u1g to vCenter 6.7u1.
Before the upgrade, vCSA was configured to forward syslog to a remote central syslog collector (UDP on port 514). After we performed the upgrade from 6.5u1 to 6.7u1 the administrator of the remote central syslog system highlighted that vCSA was now sending mal-formed syslog messages.
We are still seeing some correctly formed syslog messages, but it appears that vCSA is also sending mal-formed syslog messages with a reverse format date instead of the vCSA hostname. We performed some tests and monitored using tcpdump the messages that vCSA was sending.
This is the TCPDUMP taken when checking the syslog configuration in vCSA VMAI console (you can click to send a test message in the GUI so it is easy track). We clicked it three times:
root@vcsa01 [ ~ ]# tcpdump dst x.x.x.x -ttttnvS | grep -i 'vcenter'
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
Msg: 1 2019-01-18T12:07:38.358450+00:00 vcsa01 - - - This is a diagnostic syslog test message from vCenter Server.\0x0a
Msg: 1 2019-01-18T12:07:42.712657+00:00 vcsa01 - - - This is a diagnostic syslog test message from vCenter Server.\0x0a
Msg: 1 2019-01-18T12:07:43.494963+00:00 vcsa01 - - - This is a diagnostic syslog test message from vCenter Server.\0x0a
380 packets captured
383 packets received by filter
0 packets dropped by kernel
The syslog message we are seeing that is mal-formed has the 'logged out' string inside it. Again here is the TCPDUMP taken:
root@vcsa01[ ~ ]# tcpdump dst x.x.x.x -ttttnvS | grep -i 'logged out'
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
Msg: 1 2019-01-18T12:09:30.271720+00:00 vcsa01 vpxd 5552 - - Event [7561] [1-1] [2019-01-18T12:09:30.270543Z] [vim.event.UserLogoutSessionEvent] [info] [VSPHERE.LOCAL\vpxd-extension-d4ad4a50-d3eb-11e5-b37a-000c29a0ab87] [] [7561] [User VSPHERE.LOCAL\vpxd-extension-d4ad4a50-d3eb-11e5-b37a-000c29a0ab87@127.0.0.1 logged out (login time: Friday, 18 January, 2019 12:09:30 PM, number of API invocations: 2, user agent: VMware vim-java 1.0)]\0x0a
Msg: 1 2019-01-18T12:09:43.774170+00:00 2019-01-18 12 - - - 09:43,775 - UserId : e671ee02-d6de-4431-bb8e-735a63623545, UserName : maintenanceAdmin, AuthSource : LOCAL, Session : e671ee02-d6de-4431-bb8e-735a63623545::cf71e003-c944-4f54-bee7-e23d43c51f7f, UserAction : LOGOUT, - User logged out successfully\0x0a
Msg: 1 2019-01-18T12:09:49.925903+00:00 2019-01-18 12 - - - 09:49,927 - UserId : e671ee02-d6de-4431-bb8e-735a63623545, UserName : maintenanceAdmin, AuthSource : LOCAL, Session : e671ee02-d6de-4431-bb8e-735a63623545::40a221f5-ab4c-46c1-a04b-00df80c8dd38, UserAction : LOGOUT, - User logged out successfully\0x0a
Msg: 1 2019-01-18T12:09:49.975762+00:00 2019-01-18 12 - - - 09:49,977 - UserId : e671ee02-d6de-4431-bb8e-735a63623545, UserName : maintenanceAdmin, AuthSource : LOCAL, Session : e671ee02-d6de-4431-bb8e-735a63623545::c42feb72-551b-49db-bcf3-640f54236498, UserAction : LOGOUT, - User logged out successfully\0x0a
373 packets captured
373 packets received by filter
0 packets dropped by kernel
In this capture, you can see the first message is correctly formed. Field one has the date/time stamp, field two has the hostname.
The next three messages are mal-formed. Field one has the date/time stamp, but field two has date reversed, instead of the hostname.
This is causing us great headaches as we have several vCSA servers that will be upgraded and sending to a remote syslog server. The issue being that the syslog server will not know the source of these messages.
As an aside to this, we also see vROPs 6.6.1 sending similar and then mal-formed syslog messages when sending to a remote syslog server.
Has anyone see this behavior before and any ideas of how to resolve?
Regards
Tony
