13 Replies Latest reply on Jul 1, 2020 1:38 PM by MiMenl

    Logging DEM centrilized

    pieterheijms Novice

      Hi,

       

      We soon start with implementing Horizon DEM in our hospital. We want to centralize all "WARN," "ERROR," "and "FATAL" entries from the FlexEngine.log, FlexEngine-async.log and FlexEngine-DefaultApplications.log.

       

      Our plan is to make script which runs every x time, that checks all log directories in the profile dir. All "WARN," "ERROR," "and "FATAL" entries are written in a database, that we can check.

      In our environment there are working 4000 users (2000 concurrent), a lot of users don't call our helpdesk if logontimes are longer then expected. So we want to set up a proactive logging system, where we can see what problems there are and what needs to be fixed.

       

      Does anybody created such a script / system in the past? Otherwise we are going to make it by our own.

      Anybody has idea's about this 'system', how it should work?

       

      If anybody is interested in this logging system, please let us know, we are happy to share it, when it is finished.

       

      Pieter

        • 1. Re: Logging DEM centrilized
          MiMenl Novice

          Hi Pieter,

           

          The log files can be stored centralized on a network share.

          The log files will be stored on a per user bases

           

          I like the idea you have and might take a look at it cause it might be very handy.

           

          Just a wild idea here :

           

          It seems the logfiles are all called the same for each users so when indexing with tools like log inside or splunk you would need to create a custom script that for example appends the username somewhere when indexing the files, this part needs some thinking cause you would need to create some sort of flow that doesn't cause duplicate events.

          Then when it comes to indexing it would require some regex to substract the information in such a way it could be used.

          If I  look at the log files they are all build in a similar way :

           

          [Timestamp] [Message Type] [Message]

           

          2020-06-30 17:35:29.278 [INFO ] Exported file information successfully

          2020-06-30 17:35:29.319 [INFO ] Completed DirectFlex export (150 ms) [<<IFP#4aaaa44b-1148c24]

          2020-06-30 17:35:31.146 [INFO ] Performing DirectFlex export for config file '\\domain\uem\config\general\Applications\App.INI' [IFP#1a07110c-1148c24>>]

           

          The Timestamp, message type fields should be easy the last field depending on what you want to achieve gets more complicated since it contains a full message were we only want some info from for example the time in ms an action took, because this field contains a lot of different data this gets hard to filter. (maybe we should ask for a csv, Json, XML, or JSON formatted log with some fixed field names) something like :

           

          [TimeStamp],[username],[Message Type],[Duration(ms)],[Message] when doing csv (but not the case right now)

           

          once you are able to get the data into Log inside or Splunk properly it should be easy to search based on the indexed fields.

           

          this would be my approach since Log Inside and Splunk can do some great things when it comes to going trough events fast. it is however always possible to load the data to SQL or something but then you'll need to do a lot more programming.

           

          this is not a solution but just an idea how in my opinion something like you asked could be achieved.

           

          Mark

          • 2. Re: Logging DEM centrilized
            pieterheijms Novice

            Hi Mark,

             

            Thank you for your suggestions.

            Next week, we will discuss, how we are going to make the 'system'. When we are further, I'll post it in this thread.

             

            Pieter

            • 3. Re: Logging DEM centrilized
              pieterheijms Novice

              The Timestamp, message type fields should be easy the last field depending on what you want to achieve gets more complicated since it contains a full message were we only want some info from for example the time in ms an action took, because this field contains a lot of different data this gets hard to filter. (maybe we should ask for a csv, Json, XML, or JSON formatted log with some fixed field names) something like :

               

              [TimeStamp],[username],[Message Type],[Duration(ms)],[Message] when doing csv (but not the case right now)

              Maybe can DEMdev tell if the log file can be formatted in a csv or other format. And if there something on the road map for DEM, about logs / formatting

              • 4. Re: Logging DEM centrilized
                sjesse Master
                User ModeratorsvExpert

                You can use the elk stack(ELK Stack: Elasticsearch, Logstash, Kibana | Elastic ) which is free if you want, but is a bit of work to setup ,or you can pay and I'll think they will help set it up.You can have logstash(the l in elk) read the file, and you configure a filter to read the input like Mark memonitoned.Part of the process is writing a filter

                 

                Grok filter plugin | Logstash Reference [7.8] | Elastic

                 

                which should be pretty easy with the DEM log structure. I haven't tried it but I've been playing around with that in my LAB and may do it in the future.

                2 people found this helpful
                • 5. Re: Logging DEM centrilized
                  DEMdev Master
                  VMware Employees

                  Starting with Pieter's question: "human readable" is the only log file "flavor" we currently support, and there's nothing on the roadmap to add other formats. We generally recommend tools like vRealize Log Insight or Splunk (as already referenced by Mark) or indeed anything else that can consume log files and present some aggregated view.

                   

                  To be honest, I'm not familiar with those tools, so I don't know whether they can easily provide insight into how long a certain action took. That would require some intelligent parsing that goes beyond a line-by-line-split-everything-up-into-fields kind of approach.

                  However, I assume that it should be pretty straightforward to trigger on WARN-or-worse lines and report on those.

                   

                  On this topic, I just want to point out that we also (optionally) can log certain things to the event log. You'd still need some other tool to forward/retrieve/aggregate that information, but it's a bit more structured than our more verbose log file.

                  (And I just made a note for myself to add duration information to the respective "action finished" messages in a future version of the product .)

                   

                  Just thinking out loud (without any guarantees ): would it be useful to (optionally, based on configuration) log a warning message (to the log file and/or event log) if certain actions (path-based import at logon, path-based export at logoff, maybe more?) took longer than some configurable time?

                  1 person found this helpful
                  • 6. Re: Logging DEM centrilized
                    pieterheijms Novice

                    Thank you Demdev and Sjesse.

                     

                    We are going to look at the tools Sjesse mentioned.

                    I know that we can use the Windos log, but the problem is, the log is gone when a user logs off. We use a VDI environment. So I prefer a log system who scans the profile directory.

                     

                    When we created a method, I'll post it here, so other people can also use it.

                    • 7. Re: Logging DEM centrilized
                      DEMdev Master
                      VMware Employees

                      Hi Pieter,

                      I know that we can use the Windos log, but the problem is, the log is gone when a user logs off.

                      Are you familiar with Windows Event Forwarding? I've never used it, but that's a Windows feature that can forward event log information to a central server.

                      • 8. Re: Logging DEM centrilized
                        sjesse Master
                        vExpertUser Moderators

                        You can use nxlog to forward the windows log to a syslog server as well which most log servers support. With ELK you can also install Winlogbeat: Analyze Windows Event Logs | Elastic  to capture the windows logs. If you look at a solution Liquidware Labs Stratusphere they collect event logs as well which can be a solution if you do what DEMdev is suggesting.

                        • 9. Re: Logging DEM centrilized
                          pieterheijms Novice

                          Oja, I Windows forwarding, maybe also a solution..

                          • 10. Re: Logging DEM centrilized
                            pieterheijms Novice

                            Thanks again Sjesse.

                            We are going to look at all the suggestions.

                            • 11. Re: Logging DEM centrilized
                              MiMenl Novice

                              This sounds helpful, although it would be nice nice if the human readable log file could contain 2 more fields separated by a space or something before the whole message is presented.

                               

                               

                              if we look at the following log entries

                               

                               

                              2020-06-30 17:35:29.319 [INFO ] Completed DirectFlex export (150 ms) [<<IFP#4aaaa44b-1148c24]

                              2020-06-30 17:35:31.146 [INFO ] Performing DirectFlex export for config file '\\domain\uem\config\general\Applications\App.INI' [IFP#1a07110c-1148c24>>]

                              2020-06-30 16:55:35.124 [FATAL] Error backing up '\\domain\uem\users\m.platte\archives\Windows Settings\Windows 10 Start-Tile Screen.zip' to '\\domain\uem\users\m.platte\Backup\Windows Settings\Windows 10 Start-Tile Screen 2020-06-30 999999.zip' (2)

                              2020-06-30 16:55:35.575 [WARN ] Needed 4 retries to rename profile archive from '\\domian\uem\users\m.platte\archives\Windows Settings\Windows 10 Start-Tile Screen.tmp' to '\\domain\uem\users\m.platte\archives\Windows Settings\Windows 10 Start-Tile Screen.zip'

                              2020-05-01 12:25:19.590 [ERROR] Async UEM actions or UEM Refresh in progress

                               

                               

                              filtering out fields could be done like this (my regex sucks) :

                              so this might not work as intended I couldn't test it properly

                               

                              \d{4}-\d{2}-\d{2} \d{1,2}:\d{1,2}:\d{1,2}.\d{1,4} for date and time stamp

                              \[[A-Z]{4,8}.\] for extracting Label (ERROR, FATAL, INFO , WARN)

                              \(\d*\sms\) to extract the time when available

                              (\[\<\<.*\]|\[IFP.*\>\>\]) To get out the message correlation

                              (?<=\]).* to capture the whole message

                               

                               

                              Based on this you should be able to extract fields from the human readable log

                               

                               

                              By getting the time and correlation out of the message like this :

                               

                               

                              2020-06-30 17:35:29.319 [INFO ] [150 ms] [<<IFP#4aaaa44b-1148c24] Completed DirectFlex export

                              2020-06-30 17:35:31.146 [INFO ] [0 ms] [IFP#1a07110c-1148c24>>] Performing DirectFlex export for config file '\\domain\uem\config\general\Applications\App.INI'

                               

                               

                              It would become way easier to filter due to the fields always being in teh same order  when there is no correlation it might be handy to just add [ ] without data.

                               

                               

                              2020-05-01 13:12:14.775 [INFO ] [0ms] [ ] Importing profile archive 'Profit Communication Center - AS.zip' (\\domain\uem\users\m.platte\archives\Applications\app - AS.zip)

                               

                               

                              According to event forwarding I think Elastic en splunk have forwarder apps that can forward event logs for you.

                              In this case i would setup a windows log collector let your machines forward logs to your log collector (new entries)

                              Install a universal forwarder on your log collector (splunk/elastic (Win log beat as sjesse mentioned /log insight agent) and let it forward teh events to your indexer.

                               

                              seems working on my regex actually let me miss 2 posts

                               

                              Hope this helps.

                               

                               

                              Mark

                              • 12. Re: Logging DEM centrilized
                                DEMdev Master
                                VMware Employees

                                Hi Mark,

                                it would be nice nice if the human readable log file could contain 2 more fields separated by a space or something before the whole message is presented.

                                You mean inserting the correlation ID as an additional field into every log line, and also inserting the duration from each [INFO ] Done (221 ms) [<<IFP#f9b502b0-T5] line into a separate field (left empty for non-Done lines)? That could very easily be achieved by a pre-processing step before forwarding/inserting log file info (in)to your analysis solution.

                                • 13. Re: Logging DEM centrilized
                                  MiMenl Novice

                                  I Think i'll need to do some testing soon, this idea might be something that would really benefit us too.

                                  since I still need to get some experience with Log Insight this might be a good use case