NXLOG is a universal log collector and forwarder supporting different platforms (BSD, Unix, Linux, Windows, Android), log sources and protocols (Syslog, Windows EventLog, Graylog2 GELF, XML, JSON, CSV and more). Learn more...
The latest release of NXLog brings several bug fixes and enhancements such as better Snare compatibility and various regular expression modifiers.
The full changelog is listed below:
The rename_field() procedure was removing the field if the source and destination were the same.
The regexp and regexp replacement operators can now be used as statements, i.e. Exec $Message =~ s/aaa/bbb/;
Regular expressions now support the /m modifier to do multiline matching.
Regular expressions now support the /i modifier to do caseless matching.
Regular expressions now support the /s modifier to make the '.' match newline characters.
Fixed a regression introduced with the ActiveFiles directive in im_file when more than one truncation
did not get noticed. (ticket #40@sf) Credits go to 'savionat'.
Implemented missing parser support for IPv4 literals.
Added a host_ip() function to return the IP address associated with the hostname.
Using exec_async() could have exhausted the memory if it was called at a very high rate.
om_udp would stop sending messages in some cases after logging "apr_socket_send failed;Connection refused",
e.g. when graylog2 was not accepting udp packets. It should properly resume now.
The to_syslog_snare() formatter should now produce better snare compliant output.
Replace space, ']' and '"' with underscore in IETF syslog structured data field names.
Context cleaning would result in a segfault in pm_evcorr's thresholded rule if there was no triggering.
im_tcp and im_ssl on windows is not limited to 500 connections anymore.
Non-wildcarded File contents would get lost with ReadFromLast FALSE when the file did not exist
but did appear with unread data.
im_file does not emit "input file does not exist" warnings at every PollInterval.
The file_name() function caused assertion failures in some cases on shutdown.
A regression caused a crash with im_file when the File did not exist.
A typo in the code was causing a memory leak with rename_field().
The new release, 2.7.1189 brings a WTMP parser module and a dozen other fixes and enhancements. The following is an excerpt from the changelog:
The LICENSE has changed.
Added a new extension module to parse binary wtmp files on Linux.
Fixed a regression causing a crash after the 'failed to determine FQDN hostname' error message.
The to_syslog_*() procedures can now use $raw_event if $Message is unset to make it easier to convert to syslog.
Added a fix to im_msvistalog to handle the "EvtNext failed with error 13: The data is invalid." error better.
The im_file module now emits the last event when using with the xm_multiline extension.
Fixed the issue with more than 20 fiels and xm_multiline reported in ticket #33.
Json parse errors in raw_event could cause a double free resulting in a crash or undefined behavior.
It is now possible to use multiple instances of xm_perl.
Disallow using a single processor module instance in multiple routes.
The file_chown() procedure in xm_fileop works with user/group names in addtion to uid/gid values.
CloseWhenIdle directive for im_file.
File removal in some circumstances caused im_file to emit "input file does not exist" messages on windows.
In same rare cases im_file would give a panic on windows with "im_file got EAGAIN for read".
The regexp replacement operator s/// was leaking memory.
In some circumstances excess CPU was used when im_file watched several files.
Added some more performance optimizations to im_file to handle a large number of wildcarded files
so that it should consume less resources than before. It also comes with a new DirCheckInterval and
an ActiveFiles directive which can help in some cases when monitoring wildcarded files.
Added a RenameCheck directive to im_file which should help detecting renamed/rotated files.
The deb installer got stuck after trying to (re)start the daemon.
The detailed changes are as follows:
Service control manager could not properly shut down the service on windows2003 and possibly other windows versions. This could have resulted in unsaved positions and duplicated log collection on system restart.
Fixed a panic in nx_config_cache_write() during shutdown.
Fixed an assertion failure when pm_pattern was trying to set an invalid datetime field.
The installer adds 'eventlog' as a dependency for the nxlog service.
Some error conditions (e.g. The interface is unknown, Access is denied, Invalid parameter, etc) should be handled better now by im_mseventlog.
The reroute() and add_to_route() procedures were leaking memory.
Some xm_fileop procedures did not work properly on windows, e.g. file_remove() with wildcards.
im_exec was consuming excess CPU on windows in some cases.
Fixed a memory leak in pm_buffer.
Added xm_kvp to the windows wix build script so that it is now included in the msi package.
Added an UndefValue directive to the xm_csv module to make parsing W3C logs containing the dash "-" more painless.
The parsedate() function now returns an undefined datetime type instead of aborting execution with an error.
The drop() procedure now aborts further evaluation of statements so that an else branch is no longer required when dropping events conditionally.
Suppress repeating "failed to open directory" error messages in im_file.
Fixed bad filenames in logged config locations caused by the include directive.
Experimental MacOSX port.
Version 2.5.1089 has been released. There is a new extension module xm_kvp which makes it a lot easier to parse key-value pairs in log messages produced by many log sources.
This release includes several other enhancements and bug fixes, please consult the changelog for the details.
Version 2.4.1054 has been released. This version now compiles and works on IBM AIX and should be fine on Solaris as well. This release contains numerous stabilization and bug fixes, consult the ChangeLog file for the detailed list of changes.
Version 2.3.1027 has been released today. This version brings a new processor module named pm_evcorr which provides event correlation functionality in addition to the already available nxlog language features (variables and statistical counters). This module was greatly inspired by the Perl based sec.pl simple event correlation tool.
In addition to the above the following fixes and enhancements are available in this relese:
This release contains several smaller bug fixes and enhacements. The most notable feature addition is the SockBufSize option for the udp input module. Also added a section to the reference manual about parsing syslog from Cisco devices.
See the ChangeLog included in the sources for the detailed changes.
Version 2.1.956 has been released today. It comes with a shiny new om_http module which allows sending logs to HTTP services such as loggly, elasticsearch etc.
The pm_buffer module has been rewritten to use chunked file storage. This release contains several other fixes and improvements, see the included changelog for more information.
Version 2.0.926 of nxlog community edition has been released.
The most notable feature addition in this release is the xm_perl module. While the nxlog config language is already a powerful framework, it is not a full-featured programming language. The xm_perl module makes it possible to execute perl code and process event data using the perl language via a built-in perl interpreter. Now you can use thousands of available perl modules from CPAN or some other existing perl code directly from nxlog without the need to pipe data to external perl scripts.
There have been several other bug fixes and improvements, see the Changelog in the package.
Version 1.4.803 has been released today.
The im_file module has been enhanced so that it deals better with thousands of files and consumes less CPU. It will automatically retry files which gave a read error earlier instead of stopping completely. The im_msvistalog module now pulls all application logs by default in addition to the system logs. There were several other enhancements and bug fixes, the changelog is available in the source tarball.
Version 1.4.764 has been released.
size(), replace() and substr() functions are now available for string manipulation. Also added buffer_size() and buffer_count() functions to the pm_buffer module. This allows conditional buffering, see the Explicit drop section in the reference manual and this mailing list post about how this can be used.
Some other issues and bugs were fixed, details are in the ChangeLog.
Version 1.4.729 has been released.
Nxlog could already handle multi-line messages using module variables. This release adds a new extension module (xm_multiline) which makes it easier to process log messages spanning more than one line. It can handle java exception traces, DICOM logs and other multi-line log messages with a lot simpler configuration.
nxlog v1.4.712 has been released.
The most notable addition in this release is a new extension module xm_fileop which adds support for file operations. This allows more sophisticated log rotation which can be used also for nxlog's own logs.
See the full ChangeLog in the sources for the list of all enhancements and fixes included in this release.
I've added GELF output support to nxlog in version 1.4.624.
Now with this enhancement it is possible to collect logs from several platforms and sources and to forward it to graylog2 in GELF format so that the structured data is preserved and is available for search and analytics. The most notable source is probably Windows EventLog (from XP to W7) as you were requesting this, but I should note that there is a lot more that you can do with it.
See the docs for the details and usage.
Here is an example of a Windows EventLog message in graylog2.
XML and JSON is now supported as of version 1.4.615. nxlog can parse and generate both of these formats with the addition of two new extension modules: xm_json and xm_xml.
nxlog is the first open source logging tool to support both of these formats for parsing and generation. This paves the way for structured logging over standard formats. Now it is possible to convert logs between even more formats with the addition of these two.
The documentation has been greatly enhanced. Some possible memory leaks and race conditions were fixed. The code can now be compiled with older APR 1.2. Group memberships are honored on Unix/Linux, and a regression in the im_file module has been fixed when using wildcards.
The code can now be compiled for Android. SNARE Syslog format support has been added for output. The im_mseventlog module can now produce output in UTF-8 and its error handling was enhanced to be more fault tolerant against the EventLog subsystem's failures. The im_mseventlog and im_msvistalog modules now set the AccountType and Category fields. A ReadFromLast configuration directive was added for the im_mseventlog and im_file modules
This release fixes a database reconnection issue in om_dbi which was affecting PostgreSQL. Syslog conversion now strips newlines from the message. A new module, im_msvistalog, was added to support reading all messages from Windows EventLog on Windows2008, Vista, and later.