Browser history logs

This guide explains how to collect browser history logs using NXLog.

Most popular browsers keep a log of the browsing history in an SQLite database. Information in this database includes the URL that was accessed, the title of the page, the time when the page was visited, and the number of times it was accessed. This data can be collected and processed with NXLog using the im_odbc module.

Browser history database location and format

The browsing history database is located in the user’s profile folder and the path depends on the browser and operating system. This guide includes details of the databases used by Google Chrome, Mozilla Firefox and Microsoft Edge browsers.

Google Chrome history location and details

Chrome history is stored in an SQLite database, the filename is History and can be found in the following locations:

Microsoft Windows Vista, 7, 8, 10

C:\Users\<username>\AppData\Local\Google\Chrome\User Data\Default

Linux/Unix

/home/<username>/.config/google-chrome/Default

macOS

/Users/<username>/Library/Application Support/Google/Chrome/Default

The tables containing the Chrome browsing history in the History database are named urls and visits. Data from these tables can be joined together to retrieve the URL, page title, and time it was accessed.

Table 1. Chrome history - urls table structure
Column Type Description

id

INTEGER

Primary Key - a unique ID for the record

url

LONGVARCHAR

The URL that was accessed

title

LONGVARCHAR

The title of the website

visit_count

INTEGER

The number of times the URL was accessed

typed_count

INTEGER

The number of times the user got to this website by typing the URL in the address bar

last_visit_time

INTEGER

Timestamp in nanoseconds when the website was last visited

Table 2. Chrome history - visits table structure
Column Type Description

id

INTEGER

A unique ID for the record

url

INTEGER

An ID corresponding to a record in the urls table

visit_time

INTEGER

Timestamp in nanoseconds when the website was visited

Table 3. Example data from joined visits and urls tables
id url title visit_time

100

https://www.nxlog.co

High Performance Log Collection Solutions

13250071947070670

Mozilla Firefox history location and details

Mozilla Firefox history is stored in an SQLite database, the file name is places.sql and can be found in the following locations:

Microsoft Windows Vista, 7, 8, 10

C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile folder>

Linux/Unix

/home/<username>/.mozilla/firefox/<profile folder>

macOS

/Users/<username>/Library/Application Support/Firefox/Profiles/<profile folder>

The tables containing the Firefox browsing history in the places database are named moz_places and moz_historyvisits. Data from these tables can be joined together to retrieve the URL, page title, and time it was accessed.

Table 4. Firefox history - moz_places table structure
Column Type Description

id

INTEGER

Primary Key - a unique ID for the record

url

LONGVARCHAR

The URL that was accessed

title

LONGVARCHAR

The title of the website

visit_count

INTEGER

The number of times the URL was accessed

typed

INTEGER

The number of times the user got to this website by typing the URL in the address bar

last_visit_date

INTEGER

Timestamp in microseconds when the website was last visited

Table 5. Firefox history - moz_historyvisits table structure
Column Type Description

id

INTEGER

Primary Key - a unique ID for the record

place_id

INTEGER

An ID corresponding to a record in the moz_places table

visit_date

INTEGER

Timestamp in microseconds when the website was visited

Table 6. Example data from joined moz_historyvisits and moz_places tables
id url title visit_date

100

https://www.nxlog.co

High Performance Log Collection Solutions

1604932243415000

Microsoft Edge (v79+) history location and details

Microsoft Edge history is stored in an SQLite database, the database file name is History and can be found in the following location:

Microsoft Windows Vista, 7, 8, 10

C:\Users\<username>\AppData\Local\Microsoft\Edge\User Data\Default

The tables containing the Microsoft Edge browsing history in the History database are named urls and visits. Data from these tables can be joined together to retrieve the URL, page title, and the time it was accessed.

Table 7. Microsoft Edge history - urls table structure
Column Type Description

id

INTEGER

Primary Key - a unique ID for the record

url

LONGVARCHAR

The URL that was accessed

title

LONGVARCHAR

The title of the website

visit_count

INTEGER

The number of times the URL was accessed

typed_count

INTEGER

The number of times the user got to this website by typing the URL in the address bar

last_visit_time

INTEGER

Timestamp in nanoseconds when the website was last visited

Table 8. Microsoft Edge history - visits table structure
Column Type Description

id

INTEGER

Primary Key - a unique ID for the record

url

INTEGER

An ID corresponding to a record in the urls table

visit_time

INTEGER

Timestamp in nanoseconds when the website was visited

Table 9. Example data from joined visits and urls tables
id url title visit_time

100

https://www.nxlog.co

High Performance Log Collection Solutions

13250071947070670

Collecting browser history logs

Example 1. Collecting Google Chrome browsing history logs on Windows

This example configuration uses the xm_exec module to periodically run a batch script which copies the History database to a new location. This is done to avoid cases when the the database is locked by Google Chrome. The copied database is then processed by the im_odbc module. Since the database does not contain fields that identify the user, the event record is enriched by using the hostname() function to add the name of the machine to each record.

nxlog.conf
<Extension json>
    Module              xm_json
</Extension>

<Extension exec>
    Module              xm_exec
    <Schedule>
        Every           30 min
        <Exec>
            odbc->module_stop();
            sleep(5000000);
            exec("C:\scripts\copy_chrome_db.cmd");
            odbc->module_start();
        </Exec>
    </Schedule>
</Extension>

<Input odbc>
    Module              im_odbc
    PollInterval        1200
    ConnectionString    DRIVER=SQLite3 ODBC Driver; \
                        Database=C:\logs\History_Chrome;Version=3;
    SQL                 SELECT visits.id AS id, \
                        DATETIME(ROUND(visits.visit_time / 1000000-11644473600), \
                        'unixepoch', 'localtime') AS EventTime, \
                        urls.url AS URL, \
                        urls.title AS Title \
                        FROM visits \
                        INNER JOIN urls ON visits.url = urls.id \
                        WHERE visits.id > ?
    Exec                $Hostname = hostname();
    Exec                to_json();
</Input>
Google Chrome saves the visit_time as a timestamp in nanoseconds. In this example the SQL DATETIME function is used to convert it to local time.
copy_chrome_db.cmd
@echo off
copy "%LOCALAPPDATA%\Google\Chrome\User Data\Default\History" C:\logs\History_Chrome /Y >nul
The cmd file needs to be saved in the path specified in the Exec block of the schedule in the configuration file. The user running NXLog needs to have permission to execute the file.
The same configuration may be used for Microsoft Edge by changing the location of the original browser history database.
Output sample
{
  "id": 100,
  "EventTime": "2020-11-17 08:32:27",
  "URL": "https://nxlog.co",
  "Title": "High Performance Log Collection Solutions",
  "Hostname": "PC1",
  "EventReceivedTime": "2020-11-17T09:30:23.811555+01:00",
  "SourceModuleName": "odbc",
  "SourceModuleType": "im_odbc"
}
Example 2. Collecting Mozilla Firefox browsing history logs on Linux Ubuntu

This example configuration uses the xm_exec module to periodically copy the places.sqlite database to a new location. This is done to avoid cases when the the database is locked by Mozilla Firefox. The copied database is then processed by the im_odbc module. Since the database does not contain fields that identify the user, the event record is enriched by using the hostname() function to add the name of the machine to each record.

nxlog.conf
define FDBPATH /home/<user>/.mozilla/firefox/<profile_folder>/places.sqlite
define FDBCOPY /var/log/browser/places.sqlite

<Extension json>
    Module              xm_json
</Extension>

<Extension exec>
    Module              xm_exec
    <Schedule>
        Every           30 min
        <Exec>
            odbc->module_stop();
            sleep(5000000);
            exec("/bin/sh", "-c", "cp %FDBPATH% %FDBCOPY%");
            odbc->module_start();
        </Exec>
    </Schedule>
</Extension>

<Input odbc>
    Module              im_odbc
    PollInterval        1200
    ConnectionString    DRIVER=SQLite3;Database=%FDBCOPY%;
    SQL                 SELECT moz_historyvisits.id AS id, \
                        DATETIME(ROUND(moz_historyvisits.visit_date / 1000000), \
                        'unixepoch', 'localtime') AS EventTime, \
                        moz_places.url AS URL, \
                        moz_places.title AS Title \
                        FROM moz_historyvisits \
                        INNER JOIN moz_places ON \
                        moz_historyvisits.place_id = moz_places.id \
                        WHERE moz_historyvisits.id > ?
    Exec                $Hostname = hostname();
    Exec                to_json();
</Input>
Mozilla Firefox saves the visit_date as a timestamp in microseconds. In this example the SQL DATETIME function is used to convert it to local time.
Output sample
{
  "id": 100,
  "EventTime": "2020-11-09 14:30:43",
  "URL": "https://nxlog.co",
  "Title": "High Performance Log Collection Solutions",
  "Hostname": "PC1",
  "EventReceivedTime": "2020-11-09T15:32:51.811555+01:00",
  "SourceModuleName": "odbc",
  "SourceModuleType": "im_odbc"
}
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog 5.2.6388 on Windows
NXLog 5.2.6561 on Linux
Google Chrome 89.0.4389.114
Mozilla Firefox 87.0
Microsoft Edge 89.0.774.68

Last revision: 24 February 2021