File (im_file)
This module reads data from one or more files. You can use wildcards to match multiple files, allowing the module to process them concurrently and automatically open new files as they appear. When you enable recursion, the module also detects and reads files in newly created subdirectories.
The module populates the $raw_event core field with data read from the file.
You can then process this field further, for example, by parsing it into structured data or transforming it into another format, such as JSON or XML.
See Parsing and converting log records below and Parse common event formats in the NXLog Platform User Guide for examples.
Filepath definition
The module treats filenames as case-insensitive on Windows and case-sensitive on Unix/Linux systems.
When you specify a relative filename, note that NXLog Agent sets its working directory to / unless you define a different directory with the global SpoolDir directive.
On Windows, the directory separator is the backslash (\).
For compatibility, you can use the forward slash (/) as a directory separator, but only when the filename does not include wildcards.
If you specify a filename with wildcards, you must use the backslash (\) as the directory separator.
Wildcards
You can use wildcards in filenames and directories. Wildcards are not regular expressions but patterns commonly used by Unix shells to expand filenames (also known as "globbing").
The supported wildcards are:
- ?
-
Matches a single character only.
- *
-
Matches zero or more characters.
- \*
-
Matches the asterisk (
*) character. - \?
-
Matches the question mark (
?) character. - […]
-
Matches a single character using a character class. The class description consists of individual characters and character ranges separated by a hyphen (
-). If the class description starts with^or!, the class matches any character that is not listed. You can prefix any character with a backslash (\), which the parser ignores. This allows you to include special characters such as].-,^, and!.
Escape characters
By default, the escape sequence is the backslash character (\).
This character is also the directory separator on Windows.
Because of this, the module doesn’t support escaping wildcard characters on Windows.
See the EscapeGlobPatterns directive for more information.
The module evaluates string literals differently depending on the quotation type:
-
Single quoted strings are interpreted as-is without escaping, for example
'C:\t???\*.log'staysC:\t???\*.log. -
Escape sequences in double-quoted strings are processed, for example
"C:\\t???\*.log"becomesC:\t???\*.log.
In both cases, the evaluated string is the same and is separated into parts with different glob patterns at different levels.
On this example, the parts are C:, t???, and *.log.
NXLog Agent matches the parts at the relevant directory levels to find all matching files.
Configuration
The im_file module accepts the following directives in addition to the common module directives.
Required directives
The following directive is required for the module to start.
Set this directive to specify the input file(s). You can use this directive multiple times in a single im_file module instance, to read from multiple sources. Specify the value as a string expression. If the expression is not a constant string (contains functions, field names, or operators), it’s evaluated whenever the module checks for new content as defined by PollInterval and DirCheckInterval. See Filepath definition for more details on using this directive.
|
Optional directives
Set this directive to specify the maximum number of concurrent files to be actively monitored. This directive is only relevant when using File directives with wildcards. If there are concurrent modifications to more files than the value of this directive, then the remaining modifications will only get noticed after the DirCheckInterval. Typically there are only a few active log sources appending data to log files, so the default value should be sufficient in most cases. The default is 10. |
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is |
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is
|
|||||||||||||||||||||||||||||||||||||
Set this directive to specify the interval, in seconds, between each time the module will check monitored directories for modifications to files.
You can use fractional seconds, for example setting If the PollInterval directive is set the default of this directive is twice the value of the
|
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is |
|||||||||||||||||||||||||||||||||||||
Set this directive to specify a file, or files (using wildcards), to be ignored by the module. You can use this directive multiple times in a single im_file module instance, to ignore multiple files. See Filepath definition for more details on using this directive. |
|||||||||||||||||||||||||||||||||||||
Set this directive to If both this directive and the Recursive directive are set to The default is |
|||||||||||||||||||||||||||||||||||||
Set this block directive to specify one or more statements to execute when a file has been fully read (on end-of-file). Only one OnEOF block can be specified per im_file module instance. The following directives are allowed inside this block directive:
|
|||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||
Set this directive to specify how often, in seconds, the module checks for new files and new log entries in the files.
You can use fractional seconds, for example setting The default is 1 second. |
|||||||||||||||||||||||||||||||||||||
This boolean directive instructs the module on where to start reading events from the log source when NXLog Agent starts. When When The default is The following matrix shows the outcome of this directive in conjunction with the SavePos directive:
If the NoCache directive is |
|||||||||||||||||||||||||||||||||||||
Set this directive to specify the reading order of the elements in a directory. The accepted values are none, CtimeOldestFirst, CtimeNewestFirst (Ctime is file creating time), MtimeOldestFirst, MtimeNewestFirst (Mtime is file modification time), NameAsc and NameDesc (sort done according to ASCII codes of name characters). The default is none. |
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is
|
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is
|
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is If the NoCache directive is |
Examples
The following is a basic configuration that collects logs from files whose names start with file in a specific directory.
<Input messages>
Module im_file
File '/path/to/log/file*'
FollowSymlinks TRUE (1)
</Input>| 1 | The FollowSymlinks directive is set to TRUE, so the module follows symbolic links to files. |
This configuration reads logs from a file and parses the records using a regular expression that looks for a timestamp, severity, and message. It converts records that match the regular expression to JSON and discards those that do not.
<Extension json>
Module xm_json
</Extension>
<Input messages>
Module im_file
File '/path/to/log/file'
<Exec>
if $raw_event =~ /(?x)^(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\+\d\d:\d\d),
(.+),(.+)$/
{
$EventTime = parsedate($1); (1)
$Severity = $2;
$Message = $3;
}
else
{
drop();
}
to_json(); (2)
</Exec>
</Input>| 1 | The parsedate() function converts the timestamp to datetime. |
| 2 | The to_json() procedure converts the fields to JSON and writes the result to the $raw_event core field. |
The following input sample matches the regular expression used by the configuration above.
2026-01-05T14:03:40+01:00,INFO,The service started successfully{
"EventReceivedTime": "2026-01-05T14:04:24.244343+01:00",
"SourceModuleName": "messages",
"SourceModuleType": "im_file",
"Hostname": "SERVER-01",
"EventTime": "2026-01-05T14:03:40.000000+01:00",
"Severity": "INFO",
"Message": "The service started successfully"
}