Implementing a centralized logging system has a critical role in IT security. Logs from all critical infrastructure elements need to be collected and forwarded securely and reliably to a central log server. This means that many different types of log sources, formats and platforms need to be handled.
In the open source world the most common way to tackle this problem is to use syslog-ng or rsyslog for the central log collector. These can handle syslog natively, so basically this covers all sources which can send in syslog format (unix/linux, most network devices and appliances). In order to collect from Windows based systems, a special log collector tool needs to be installed which can read EventLog then convert it to syslog and forward it over the network to the central log server. Apart from closed source and proprietary solutions, the most popular open source tools for this task are Snare Agent and Eventlog-to-syslog. While this setup can get the job done, there are a couple problems with this approach which nxlog does not suffer from.
- Different agent or log collector for each platform and log source
- You need to deploy and manage different tools on each platform just to be able to collect logs. Their configuration method, syntax and configuration parameters are different for each. There is no uniform way to install, configure, run and manage these tools. While this problem is not significant while they can get the job done, this can still cause quite a lot of problems down the road.
- No security and reliability
- These EventLog forwarder tools utilize the old syslog protocol where UDP is used as the network transport. Because UDP is an unreliable protocol, messages can be lost even under normal circumstances. Some support TCP to solve the unreliable nature of UDP, but this does not guarantee secure transmission. TLS/SSL is hardly available among these tools which would enable forwarding the logs over the network securely and reliably. Another big problem is the lack of proper flow control and buffering. Even if TCP is used which ensures that messages will not be lost over the network, without proper flow control and buffering there is a chance that these tools will lose and drop log messages locally.
- Discarding meta-data and structure
- We think that converting Windows EventLog to syslog is a fundamentally flawed idea and is a hack around an incompatibility problem which should be solved the right way. Though the newer syslog standard (as defined in RFC 5424) would partly solve this issue by allowing to send structured meta-data along with the message, these EventLog forwarders only support the old syslog protocol (RFC 3164) which even lacks a proper date format as there is no year in the date.
Windows EventLog allows multi-line messages, so this text is a lot more readable and nicely formatted by spaces, tabs and line-breaks as can be seen in Event Viewer. Because syslog is a single-line message, this formatting must be stripped from the EventLog message. The most common is the format used by the Snare Agent and has become widespread in other tools also. This stores the EventID, User name, Domain and other fields in the message part of the syslog line in a TAB delimited format. Then the central logserver must be instructed to parse the log in order to store the fields in a database.
When additional meta-data is required (such as IP addresses, port numbers, email addresses etc.) special patterns need to be created which can extract these fields from the log message.
The newer EventLog format can store structured meta-data along with the message. When you look at logs in Event Viewer you can see such fields as in the following example:
<Data Name="TargetDomainName">NT AUTHORITY</Data>
<Data Name="WorkstationName" />
Why would you want to write patterns and parser rules to extract this data from the message when it is already available at the source in a structured format?
We think that converting to syslog is a flawed idea because it discards valuable meta-data. nxlog is capable of reading these fields and can forward these remotely, or act upon (i.e. alert, correlate), thus sparing you time and resources. There is no need to waste your time writing patterns to extract usernames, IP addresses and similar meta-data. Be smart and do something useful instead.
The right answer is nxlog. Both on the server and the client side.