- Open Source
- The source code of nxlog community edition is available under an open source license (GPL/LGPL). Anyone is free to use and modify it under the terms of the license.
- Supports many different operating systems such as Linux (Debian, Redhat, Ubuntu), BSD, HP-UX, IBM AIX, Solaris, Android and also Microsoft Windows (no, it does not require cygwin) so you don't need a different collector/agent for that other platform.
- Modular architecture
- Dynamically loadable modules (=plugins) are available to provide different features and functionality similarly as the Apache HTTP server does it. It will only load modules which are needed by the log processing configuration. Log format parsers, transmission protocol handlers, database modules and nxlog language extensions are such modules. A module is only loaded if it is necessary resulting in a small memory footprint. Modules have a common API, developers can easily write new modules and extend the functionality of nxlog.
- Support for different message formats
- The nxlog community edition supports many different log formats such as Syslog (both BSD and IETF syslog as defined by RFC 3164 and RFC 5424), CSV, JSON, XML, GELF, Windows EventLog, etc. It's easy to create a parser rule for custom application logs or a write module to handle binary formats.
nxlog is not only syslog daemon but can fully function as one. Log messages need not to be flattened out and squeezed into the syslog format or similar single line messages if this is not required.
A special nxlog message format can preserve the parsed fields of log messages and transfer these across the network or store in files. This alleviates the need to parse the messages again at the receiver side and avoids loosing any information which was available at the source.
- Client-server mode
- nxlog can act both as a client and/or a server. It can collect logs from local files and the operating system then forward it to to a remote server. It can accept connections and receive logs over the network then write these to a database or files or forward it further. It all depends how it is configured.
- Log message sources and destinations
- In addition to reading from and writing to log files, nxlog supports different protocols on the network and transport layer such as TCP, UDP, TLS/SSL and Unix Domain Socket. It can both read and write from such sources and can convert between then, read input from an UDP socket and send out over TCP for example.
Many database servers are supported (PostgreSQL, MySQL, Oracle, MsSQL, SqlLite, Sybase, etc) so that log messages or data extracted from log messages can be stored in or read from a database.
- Secure operation, secured messages
- On unix systems nxlog can be run as a normal user by dropping its root privileges even if some sources require special privileges. To secure data and event logs, nxlog provides a TLS/SSL transport so that messages cannot be intercepted and/or altered during transmission.
- Written in good old C, nxlog is blazingly fast while consuming only a fraction of the resources. On average it only requires a few megabytes of memory and can be an ideal choice for embedded systems as well. It does not depend on any runtime environment such as Java, .NET, Ruby or Python and you will notice that it is quite slim compared to other log collector tools written in higher level languages.
- Scalable, multi-threaded and high-performance architecture
- Reading input, writing output and log processing (parsing, pattern matching, etc) are all handled in parallel. For example when single threaded syslog daemons block trying to write output to a file or database, UDP input can be be lost. The multi-threaded architecture of nxlog not only avoids this problem but enables to fully utilize today's multi-core and multi-processor systems for maximum throughput. The massively multi-threaded architecture of nxlog enables it to process log messages from thousands of simultaneous network connections above a hundred thousand event per second (EPS) rate.
- Message buffering and prioritization
- Log messages can be buffered in memory or on disk in order to avoid loosing messages. Together with the nxlog language it is also possible to do conditional buffering based on different parameters (time or system load for example). Not all log sources are always equally important. Some systems send critical logs which should be processed at a higher priority than others. nxlog supports assigning priorities to log routes, this ensures that higher priority log messages are dealt with (read, processed and written/sent out) first, only then are the messages with lower priorities handled.
- No message loss
- nxlog will not drop log messages, it will throttle back on the input side wherever possible. Though nxlog can be explicitly instructed to drop log messages depending on certain conditions in order to avoid a possible resource exhaustion. UDP syslog is a typical case where a message can be lost due to the nature of the UDP protocol. If the kernel buffer becomes full because it is not read, the operating system will drop any further received UDP messages. If a log processing system is busy processing logs, reading from TCP and UDP and writing to database or disk, the kernel UDP buffer can fill quickly. Utilizing the above mentioned parallel processing, buffering and I/O prioritization features it is possible to greatly reduce losing UDP syslog messages. Of course using TCP can help avoiding message loss, unfortunately there are many archaic devices which only support UDP syslog.
- Simple configuration
- nxlog uses Apache style configuration file syntax. This format is in use by many other popular system daemons and tools as it is easy to read and/or generate by both humans and scripts.
- Built-in config language
- A built-in configuration language enables administrators to create correlation rules, parse, format or rewrite messages, execute some action or convert to another message format. Using this language it is possible to do virtually anything without the need to forward messages to an external script.
This built-in nxlog language is very similar in syntax to Perl, which is highly popular in the log processing world. In addition to the normal operations it supports polymorphic functions and procedures, and regular expressions with captured substrings. It should be fairly trivial to write and understand, unlike some macro based configuration languages found in other solutions.
- Scheduled tasks and built-in log rotation
- nxlog has a built-in scheduler similar to cron, but with more advanced capabilities to specify the timings. Using this feature, administrators can automate tasks such as log rotation or system statistics within nxlog without having to use an external scheduler. Log files can be rotated by size, time and various conditions, there is no need for external log rotation tools. Log rotation can also be scheduled in order to guarantee timely file rotation. The file input reader module supports external log-rotation scripts, it can detect when an input file was moved/renamed and will reopen its input. Similarly, the file output writer module can also monitor when the file being written to is rotated and will reopen its original output. This way it is possible to keep using external log rotation tools without the need to migrate to the built-in log rotation.
- Event classification and pattern matching
- In addition to the features provided by the above mentioned built-in nxlog language, using additional modules nxlog is capable to solve all tasks related to log message processing such as message classification, event correlation, pattern matching, message filtering, rewrite, conditional alerting etc.
- Offline processing mode for post processing, conversion or transfer
- Sometimes messages need to be processed in an offline fashion, convert log files to another format, filter out messages or load files into a database for log analysis purposes. nxlog can also work in an offline mode when it processes log messages until there is no more input and then exits, thus it is possible to do batch processing tasks with it also.
- Internationalization support
- nxlog supports explicit character set conversion from one character set to another. In addition it can also detect the character set of a string and convert it to a specific character set. Using charset autodetection, nxlog is capable of normalizing log messages which can contain strings in mixed character sets even without knowing the exact encoding of the source log message.