Browser history logs
This guide explains how to collect browser history logs using NXLog Agent.
Most popular browsers keep a log of the browsing history in an SQLite database. Information in this database includes the URL that was accessed, the title of the page, the time when the page was visited, and the number of times it was accessed. This data can be collected and processed with NXLog Agent using the im_odbc module.
Browser history database location and format
The browsing history database is located in the user’s profile folder and the path depends on the browser and operating system. This guide includes details of the databases used by Google Chrome, Mozilla Firefox, and Microsoft Edge browsers.
Google Chrome history location and details
Chrome history is stored in an SQLite database, the filename is History
and can be found in the following locations:
- Microsoft Windows Vista, 7, 8, 10, 11
-
C:\Users\<username>\AppData\Local\Google\Chrome\User Data\Default
- Linux/Unix
-
/home/<username>/.config/google-chrome/Default
- macOS
-
/Users/<username>/Library/Application Support/Google/Chrome/Default
The tables containing the Chrome browsing history in the History database are named urls and visits. Data from these tables can be joined together to retrieve the URL, page title, and time it was accessed.
Column | Type | Description |
---|---|---|
|
|
Primary Key - a unique ID for the record |
|
|
The URL that was accessed |
|
|
The title of the website |
|
|
The number of times the URL was accessed |
|
|
The number of times the user got to this website by typing the URL in the address bar |
|
|
Timestamp in nanoseconds when the website was last visited |
Column | Type | Description |
---|---|---|
|
|
A unique ID for the record |
|
|
An ID corresponding to a record in the urls table |
|
|
Timestamp in nanoseconds when the website was visited |
id | url | title | visit_time |
---|---|---|---|
|
High Performance Log Collection Solutions |
|
Mozilla Firefox history location and details
Mozilla Firefox history is stored in an SQLite database, the file name is places.sql
and can be found in the following locations:
- Microsoft Windows Vista, 7, 8, 10, 11
-
C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile folder>
- Linux/Unix
-
/home/<username>/.mozilla/firefox/<profile folder>
- macOS
-
/Users/<username>/Library/Application Support/Firefox/Profiles/<profile folder>
The tables containing the Firefox browsing history in the places database are named moz_places and moz_historyvisits. Data from these tables can be joined together to retrieve the URL, page title, and time it was accessed.
Column | Type | Description |
---|---|---|
|
|
Primary Key - a unique ID for the record |
|
|
The URL that was accessed |
|
|
The title of the website |
|
|
The number of times the URL was accessed |
|
|
The number of times the user got to this website by typing the URL in the address bar |
|
|
Timestamp in microseconds when the website was last visited |
Column | Type | Description |
---|---|---|
|
|
Primary Key - a unique ID for the record |
|
|
An ID corresponding to a record in the moz_places table |
|
|
Timestamp in microseconds when the website was visited |
id | url | title | visit_date |
---|---|---|---|
|
High Performance Log Collection Solutions |
|
Microsoft Edge (v79+) history location and details
Microsoft Edge history is stored in an SQLite database, the database file name is History
and can be found in the following location:
- Microsoft Windows Vista, 7, 8, 10, 11
-
C:\Users\<username>\AppData\Local\Microsoft\Edge\User Data\Default
The tables containing the Microsoft Edge browsing history in the History database are named urls and visits. Data from these tables can be joined together to retrieve the URL, page title, and the time it was accessed.
Column | Type | Description |
---|---|---|
|
|
Primary Key - a unique ID for the record |
|
|
The URL that was accessed |
|
|
The title of the website |
|
|
The number of times the URL was accessed |
|
|
The number of times the user got to this website by typing the URL in the address bar |
|
|
Timestamp in nanoseconds when the website was last visited |
Column | Type | Description |
---|---|---|
|
|
Primary Key - a unique ID for the record |
|
|
An ID corresponding to a record in the urls table |
|
|
Timestamp in nanoseconds when the website was visited |
id | url | title | visit_time |
---|---|---|---|
|
High Performance Log Collection Solutions |
|
Collecting browser history logs
This example configuration uses the xm_exec module to periodically run a batch script that copies the History database to a new location. This is done to avoid cases when the database is locked by Google Chrome. The copied database is then processed by the im_odbc module. Since the database does not contain fields that identify the user, the event record is enriched by using the hostname() function to add the name of the machine to each record.
<Extension json>
Module xm_json
</Extension>
<Extension exec>
Module xm_exec
<Schedule>
Every 30 min
<Exec>
odbc->module_stop();
sleep(5000000);
exec("C:\scripts\copy_chrome_db.cmd");
odbc->module_start();
</Exec>
</Schedule>
</Extension>
<Input odbc>
Module im_odbc
PollInterval 1200
ConnectionString DRIVER=SQLite3 ODBC Driver; \
Database=C:\logs\History_Chrome;Version=3;
SQL SELECT visits.id AS id, \
DATETIME(ROUND(visits.visit_time / 1000000-11644473600), \
'unixepoch', 'localtime') AS EventTime, \
urls.url AS URL, \
urls.title AS Title \
FROM visits \
INNER JOIN urls ON visits.url = urls.id \
WHERE visits.id > ?
Exec $Hostname = hostname();
Exec to_json();
</Input>
Google Chrome saves the visit_time as a timestamp in nanoseconds.
In this example, the SQL DATETIME function is used to convert it to local time.
|
@echo off
copy "%LOCALAPPDATA%\Google\Chrome\User Data\Default\History" C:\logs\History_Chrome /Y >nul
The cmd file needs to be saved in the path specified in the Exec block of the schedule in the configuration file.
The user running NXLog Agent must have permission to execute the file.
|
The same configuration may be used for Microsoft Edge by changing the location of the original browser history database. |
{
"id": 100,
"EventTime": "2022-10-26 14:12:18",
"URL": "https://nxlog.co",
"Title": "High Performance Log Collection Solutions",
"Hostname": "PC1",
"EventReceivedTime": "2022-10-26T14:48:10.360819+03:00",
"SourceModuleName": "odbc",
"SourceModuleType": "im_odbc"
}
This example configuration uses the xm_exec module to periodically copy the places.sqlite
database to a new location.
This is done to avoid cases when the database is locked by Mozilla Firefox.
The copied database is then processed by the im_odbc module.
Since the database does not contain fields that identify the user, the event record is enriched by using the hostname() function to add the name of the machine to each record.
define FDBPATH /home/<user>/.mozilla/firefox/<profile_folder>/places.sqlite
define FDBCOPY /var/log/browser/places.sqlite
<Extension json>
Module xm_json
</Extension>
<Extension exec>
Module xm_exec
<Schedule>
Every 30 min
<Exec>
odbc->module_stop();
sleep(5000000);
exec("/bin/sh", "-c", "cp %FDBPATH% %FDBCOPY%");
odbc->module_start();
</Exec>
</Schedule>
</Extension>
<Input odbc>
Module im_odbc
PollInterval 1200
ConnectionString DRIVER=SQLite3;Database=%FDBCOPY%;
SQL SELECT moz_historyvisits.id AS id, \
DATETIME(ROUND(moz_historyvisits.visit_date / 1000000), \
'unixepoch', 'localtime') AS EventTime, \
moz_places.url AS URL, \
moz_places.title AS Title \
FROM moz_historyvisits \
INNER JOIN moz_places ON \
moz_historyvisits.place_id = moz_places.id \
WHERE moz_historyvisits.id > ?
Exec $Hostname = hostname();
Exec to_json();
</Input>
Mozilla Firefox saves the visit_date as a timestamp in microseconds.
In this example, the SQL DATETIME function is used to convert it to local time.
|
{
"id": 100,
"EventTime": "2022-10-26 18:45:12",
"URL": "https://nxlog.co",
"Title": "High Performance Log Collection Solutions",
"Hostname": "PC1",
"EventReceivedTime": "2022-10-26T18:56:57.272942+03:00",
"SourceModuleName": "odbc",
"SourceModuleType": "im_odbc"
}