Parse logs in W3C Extended Log File Format
The W3C Extended Log File Format is a text-based log format most commonly used for Microsoft IIS access logs. It is a customizable log format, allowing control over the recorded data. See the Extended Log File Format W3C draft for further details.
Below, we provide examples of collecting and parsing Microsft IIS access logs in the W3C log format, such as the following.
#Software: Microsoft Internet Information Services 10.0
#Version: 1.0
#Date: 2023-11-08 16:20:15
#Fields: date time c-ip cs-username s-ip s-port cs-method cs-uri-stem cs-uri-query sc-status cs(User-Agent) sc-substatus
2023-11-08 16:24:21 61.135.169.37 - 174.120.30.2 80 GET /index.htm - 200 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/119.0.0.0+Safari/537.36 0NXLog Agent’s dedicated xm_w3c log parser can automatically process W3C logs.
This configuration reads Microsoft IIS logs with the im_file input module. It sets the InputType directive of im_file to the xm_w3c instance name, which does not require further settings to process such logs.
<Extension w3c_parser>
Module xm_w3c
</Extension>
<Input iis_logs>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC*\ex*.log'
InputType w3c_parser
</Input>When the NXLog Agent configuration above processes an IIS access log event, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$date |
2023-11-08 |
$time |
16:24:21 |
$c-ip |
61.135.169.37 |
$cs-username |
null |
$s-ip |
174.120.30.2 |
$s-port |
80 |
$cs-method |
GET |
$cs-uri-stem |
/index.htm |
$cs-uri-query |
null |
$sc-status |
200 |
$cs(User-Agent) |
Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)AppleWebKit/537.36(KHTML,+like+Gecko)+Chrome/119.0.0.0+Safari/537.36 |
$sc-substatus |
0 |
$EventTime |
2023-11-08T17:24:21.000000+01:00 |
Alternatively, you can use the xm_csv module to parse logs in the W3C Extended Log Format. However, you must define the fields, data types, delimiter, and other settings according to your log format.
The example below processes Microsoft IIS access logs with the default fields. If you have customized the log fields, you must modify the xm_csv fields and data types accordingly.
This configuration reads Microsoft IIS logs with the im_file input module and parses records into structured data using the parse_csv() procedure of xm_csv.
It then creates the $EventTime field by joining the parsed $date and $time fields and converts it to datetime with the parsedate() function.
The configuration ignores lines starting with a hash mark (#).
<Extension w3c_parser>
Module xm_csv
Fields $date, $time, $c-ip, $cs-username, $s-ip, $s-port, \
$cs-method, $cs-uri-stem, $cs-uri-query, $sc-status, \
$cs-user-agent, $sc-substatus
FieldTypes string, string, string, string, string, integer, \
string, string, string, integer, \
string, integer
Delimiter ' '
UndefValue -
</Extension>
<Input iis_logs>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC*\ex*.log'
<Exec>
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date + " " + $time);
}
</Exec>
</Input>When the NXLog Agent configuration above processes an IIS access log event, it adds the following fields to the log record in addition to the core fields.
| Field | Value |
|---|---|
$date |
2023-11-08 |
$time |
16:24:21 |
$c-ip |
61.135.169.37 |
$cs-username |
null |
$s-ip |
174.120.30.2 |
$s-port |
80 |
$cs-method |
GET |
$cs-uri-stem |
/index.htm |
$cs-uri-query |
null |
$sc-status |
200 |
$cs-user-agent |
Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)AppleWebKit/537.36(KHTML,+like+Gecko)+Chrome/119.0.0.0+Safari/537.36 |
$sc-substatus |
0 |
$EventTime |
2023-11-08T17:24:21.000000+01:00 |