Usage
As a package
To use firepit in a project:
from firepit import get_storage
db = get_storage('observations.db', session_id)
db.cache('my_query_id', ['bundle1.json', 'bundle2.json'])
After caching your STIX bundles, your database will contain the data.
Passing a file path to get_storage will use sqlite3. Passing in a PostgreSQL connection URI (e.g. postgresql://…) will instead attempt to connect to the PostgreSQL instance specified.
As a tool
You can use the firepit command line tool to ingest and query your data.
To make things easier, you can set a pair of environment variables:
export FIREPITDB=my_dbname
export FIREPITID=my_session_id
FIREPITDB is your DB filename (sqlite3) or connection URI (PostgreSQL). FIREPITID is a “session” ID you can use to keep your data organized.
$ firepit --help
Usage: firepit [OPTIONS] COMMAND [ARGS]...
Columnar storage for STIX observations
Options:
--dbname TEXT Path/name of database
--session TEXT Session ID to data separation [default: test-id]
--help Show this message and exit.
Commands:
assign Perform an operation on a column and name the result
cache Cache STIX observation data in SQL
columns Get the columns names of a view/table
count Get the count of rows in a view/table
delete Delete STIX observation data in SQL
extract Create a view of a subset of cached data
filter Create a filtered view of a subset of cached data
get-appdata Get the app-specific data for a view
join Join two views
load Cache STIX observation data in SQL
lookup Retrieve a view
merge Merge 2 or more views into a new view
number-observed Retrieve the count of values of a column from a view
reassign Update/replace STIX observation data in SQL
remove Remove a view
rename Rename a view
schema Get the schema of a view/table
set-appdata Set the app-specific data for a view
sql Run a SQL statement on the database [DANGEROUS!]
summary Retrieve timeframe and count from a view
tables Get all view/table names
timestamped Retrieve the timestamped values of a column from a view
type Get the SCO type of a view/table
value-counts Retrieve the value counts of a column from a view
values Retrieve the values of a STIX object path from a view
viewdata Get view data for views [default is all views]
views Get all view names
$ firepit cache --help
Usage: firepit cache [OPTIONS] QUERY_ID FILENAMES...
Cache STIX observation data in SQL
Arguments:
QUERY_ID An identifier for this set of data [required]
FILENAMES... STIX bundle files of query results [required]
Options:
--help Show this message and exit.
splint
Firepit also includes a utility called splint. This tool supports multiple commands for manipulating STIX 2.0 bundles (2.1 is not yet supported), including randomzing IDs, shifting timestamps, and converting other log formats to STIX observations.
The convert command currently supports the JSON format at https://securitydatasets.com as well as Zeek conn.log and dns.log.
$ splint
Usage: splint [OPTIONS] COMMAND [ARGS]...
STIX processing and linting
Options:
--help Show this message and exit.
Commands:
convert Convert various log files to STIX 2.0 bundles
dedup-ids Replace duplicate IDs with random IDs
limit Truncate STIX bundle
randomize-ids Randomize STIX observation IDs in a bundle
timeshift Timeshift STIX observations in a bundle
upgrade Upgrade a STIX 2.0 bundle to 2.1