firepit package

Subpackages

Submodules

firepit.deref module

firepit.deref.auto_deref(store, view, ignore=None, paths=None)

Automatically resolve refs for backward compatibility.

If paths is specified, only follow/deref those specific paths/properties.

Use auto_deref_cached if you already have col_dict in memory.

firepit.deref.auto_deref_cached(view, cols, col_dict, ignore=None, paths=None)

Automatically resolve refs for backward compatibility.

If paths is specified, only follow/deref those specific paths/properties.

firepit.deref.unresolve(objects)

Do the opposite of auto_deref: split out reference objects

firepit.exceptions module

exception firepit.exceptions.DatabaseMismatch(dbversion, expected)

Bases: Exception

exception firepit.exceptions.DuplicateTable

Bases: Exception

exception firepit.exceptions.IncompatibleType

Bases: Exception

exception firepit.exceptions.InvalidAttr(msg)

Bases: Exception

exception firepit.exceptions.InvalidObject(msg)

Bases: Exception

exception firepit.exceptions.InvalidStixPath

Bases: Exception

exception firepit.exceptions.InvalidViewname

Bases: Exception

exception firepit.exceptions.SessionExists

Bases: Exception

exception firepit.exceptions.SessionNotFound

Bases: Exception

exception firepit.exceptions.StixPatternError(stix)

Bases: Exception

exception firepit.exceptions.UnexpectedError

Bases: Exception

exception firepit.exceptions.UnknownViewname

Bases: Exception

firepit.generator module

firepit.pgcommon module

Some common PostgreSQL stuff used by both pgstorage.py (the normal interface) and aio.asyncpgstorage.py (the async interface).

firepit.pgcommon.pg_shorten(key)

firepit.pgstorage module

firepit.props module

Utility functions for STIX properties

firepit.props.auto_agg(sco_type, prop, col_type)

Infer an aggregation function based on column name and type

firepit.props.auto_agg_tuple(sco_type, prop, col_type)

Infer an aggregation function based on column name and type

firepit.props.get_last(prop)

firepit.props.is_ref(name)

firepit.props.parse_path(path)

firepit.props.parse_prop(sco_type, prop)

firepit.props.path_metadata(path)

Get metadata for a STIX object path

firepit.props.primary_prop(sco_type)

Returns the “primary” property name for each SCO type

firepit.props.prop_metadata(sco_type, prop)

Get metadata for a STIX object property

firepit.props.ref_type(sco_type, part)

Get STIX SCO type for reference prop part

firepit.query module

Utilities for generating SQL while avoiding SQL injection vulns

class firepit.query.Aggregation(aggs)

Bases: object

Aggregate rows

render(_placeholder, _dialect=None)

class firepit.query.BinnedColumn(prop: str, n: int, unit: Optional[str] = None, table: Optional[str] = None, alias: Optional[str] = None)

Bases: Column

Bin (or “bucket”) column values, persumably for easier grouping

render(_placeholder, dialect=None)

class firepit.query.CoalescedColumn(names, alias)

Bases: object

First non-null column from a list - used after a JOIN

class firepit.query.Column(name, table=None, alias=None)

Bases: object

SQL Column name

endswith(s)

class firepit.query.Count

Bases: object

Count the rows in a result set

render(_placeholder, _dialect=None)

class firepit.query.CountUnique(cols=None)

Bases: object

Unique count of the rows in a result set

render(_placeholder, _dialect=None)

class firepit.query.Filter(preds, op=' AND ')

Bases: object

Alternative SQL WHERE clause

AND = ' AND '

OR = ' OR '

render(placeholder, _dialect=None)

set_table(table)

Specify table for ALL Predicates in Filter

class firepit.query.Group(cols)

Bases: object

SQL GROUP clause

render(_placeholder, _dialect=None)

exception firepit.query.InvalidAggregateFunction

Bases: Exception

exception firepit.query.InvalidComparisonOperator

Bases: Exception

exception firepit.query.InvalidJoinOperator

Bases: Exception

exception firepit.query.InvalidPredicateOperand

Bases: Exception

exception firepit.query.InvalidPredicateOperator

Bases: Exception

exception firepit.query.InvalidQuery

Bases: Exception

class firepit.query.Join(name, left_col=None, op=None, right_col=None, preds=None, how='INNER', alias=None, lhs=None)

Bases: object

Join 2 tables

render(placeholder, _dialect=None)

class firepit.query.Limit(num)

Bases: object

SQL row count

render(_placeholder, _dialect=None)

class firepit.query.Offset(num)

Bases: object

SQL row offset

render(_placeholder, _dialect=None)

class firepit.query.Order(cols)

Bases: object

SQL ORDER BY clause

ASC = 'ASC'

DESC = 'DESC'

render(_placeholder, _dialect=None)

class firepit.query.Predicate(lhs, op, rhs)

Bases: object

Row value predicate

render(placeholder, _dialect=None)

set_table(table)

Specify table for ALL columns in Predicate

class firepit.query.Projection(cols)

Bases: object

SQL SELECT (really projection - pick column subset) clause

render(placeholder, dialect=None)

class firepit.query.Query(arg=None)

Bases: object

SQL Query statement

SQL order of evaluations: FROM, including JOINs WHERE GROUP BY HAVING WINDOW functions SELECT (projection) DISTINCT UNION ORDER BY LIMIT and OFFSET

append(stage)

extend(stages)

render(placeholder, dialect=None)

class firepit.query.Table(name)

Bases: object

SQL Table selection

render(_placeholder, _dialect=None)

class firepit.query.Unique

Bases: object

Reduce the rows in a result set to unique tuples

render(placeholder, dialect=None)

firepit.schemas module

firepit.sqlitestorage module

class firepit.sqlitestorage.SQLiteStorage(dbname)

Bases: SqlStorage

columns(viewname)

Get the column names (properties) of viewname

delete()

Delete ALL data in this store

schema(viewname=None)

Get the schema (names and types) of table/view viewname or all tables if not specified

tables()

Get all table names

types(private=False)

Get all table names that correspond to SCO types

firepit.sqlitestorage.get_storage(path)

firepit.sqlitestorage.row_factory(cursor, row)

firepit.sqlstorage module

class firepit.sqlstorage.SqlStorage

Bases: object

assign(viewname, on, op=None, by=None, ascending=True, limit=None)

DEPRECATED: Perform (unary) operation op on on and store result as viewname

assign_query(viewname, query, sco_type=None)

Create a new view viewname defined by query

cache(query_id, bundles, batchsize=2000, **kwargs)

Cache the result of a query/dataset

Takes the observed-data SDOs from bundles and “flattens” them, splits out SCOs by type, and inserts into a database with 1 table per type.

Accepts some keyword args for runtime options, some of which may depend on what database type is in use (e.g. sqlite3, postgresql, …)

Args:

query_id (str): a unique identifier for this set of bundles

bundles (list): STIX bundles (either in-memory Python objects or filename paths)

batchsize (int): number of objects to insert in 1 batch (defaults to 2000)

close()

columns(viewname)

Get the column names (properties) of viewname

count(viewname)

Get the count of objects (rows) in viewname

delete()

Delete ALL data in this store

extract(viewname, sco_type, query_id, pattern)

Extract all sco_type object from the results of query_id and store as viewname

extract_observeddata_attribute(viewname, name_of_attribute, path=None, value=None, limit=None, run=True)

Get the observations of value in viewname.`path` Returns list of dicts like {‘name_of_attribute’: ‘…’, ‘{column}’: ‘…’} name_of_attribute can be a str or list of str (to get multiple attrs)

filter(viewname, sco_type, input_view, pattern)

Extract all sco_type object from input_view and store as viewname

finish(index=True)

Do any DB-specific post-caching/insertion activity, such as indexing

get_appdata(viewname)

Retrieve app-specific data for a viewname

get_view_data(viewnames=None)

Retrieve information about one or more viewnames

group(newname, viewname, by, aggs=None)

Create new view newname defined by grouping viewname by by

join(viewname, l_var, l_on, r_var, r_on)

Join vars l_var and r_var and store result as viewname

load(viewname, objects, sco_type=None, query_id=None, preserve_ids=True)

Import objects as type sco_type and store as viewname

lookup(viewname, cols='*', limit=None, offset=None, col_dict=None)

Get the value of viewname

merge(viewname, input_views)

number_observed(viewname, path, value=None)

Get the count of observations of value in viewname.`path` Returns integer count

path_joins(viewname, sco_type, column)

Determine if column has implicit Joins and return them if so

reassign(viewname, objects)

Replace objects (or insert them if they’re not there)

remove_view(viewname)

Remove view viewname

rename_view(oldname, newname)

Rename view oldname to newname

run_query(query)

schema(viewname=None)

Get the schema (names and types) of table/view viewname or all tables if not specified

set_appdata(viewname, data)

Attach app-specific data to a viewname

summary(viewname, path=None, value=None)

Get the first and last observed time and number observed for observations of viewname, optionally specifying path and value. Returns list of dicts like {‘first_observed’: ‘2021-10-…’, ‘last_observed’: ‘2021-10-…’, ‘number_observed’: N}

table_type(viewname)

Get the SCO type for table/view viewname

tables()

Get all table names

timestamped(viewname, path=None, value=None, timestamp='first_observed', limit=None, run=True)

Get the timestamped observations of value in viewname.`path` Returns list of dicts like {‘timestamp’: ‘2021-10-…’, ‘{column}’: ‘…’}

types(private=False)

Get all table names that correspond to SCO types

upsert(cursor, tablename, obj, query_id, schema)

upsert_many(cursor, tablename, objs, query_id, schema)

value_counts(viewname, path)

Get the count of observations of each value in viewname.`path` Returns list of dicts like {‘{column}’: ‘…’, ‘count’: 1}

values(path, viewname)

Get the values of STIX object path path (a column) from viewname

views()

Get all view names

firepit.sqlstorage.get_path_joins(viewname, sco_type, column)

Determine if column has implicit Joins and return them if so

firepit.sqlstorage.infer_type(key, value)

firepit.stix20 module

firepit.stix20.comp2sql(sco_type, prop, op, value, dialect)

firepit.stix20.get_grammar()

firepit.stix20.path2sql(sco_type, path)

firepit.stix20.stix2sql(pattern, sco_type, dialect='sqlite3')

firepit.stix20.summarize_pattern(pattern)

firepit.stix21 module

firepit.stix21.makeid(sco, obs=None)

firepit.timestamp module

firepit.timestamp.timefmt(t, prec=3)

Format Python datetime t in RFC 3339-format

firepit.timestamp.to_datetime(timestamp)

Convert RFC 3339-format timestamp to Python datetime

firepit.validate module

STIX and SQL identifier validators

firepit.validate.validate_name(name)

Make sure name is a valid (SQL) identifier

firepit.validate.validate_path(path)

Make sure path is a valid STIX object path or property name

firepit.woodchipper module

class firepit.woodchipper.FlatJsonMapper

Bases: Mapper

convert(event)

detect(event)

class firepit.woodchipper.IscHoneypotJsonMapper

Bases: Mapper

convert(event)

detect(event)

mapping = {'dest': 'network-traffic:dst_ref.value', 'dport': 'network-traffic:dst_port', 'proto': 'network-traffic:protocols', 'source': 'network-traffic:src_ref.value', 'sport': 'network-traffic:src_port', 'ts': ['first_observed', 'last_observed'], 'url': 'url:value', 'user_agent': "network-traffic:extensions.'http-request-ext'.request_header.'User-Agent'"}

class firepit.woodchipper.Mapper

Bases: object

convert(event)

detect(event)

class firepit.woodchipper.SdsMapper

Bases: Mapper

common_mapping = {'@timestamp': ['first_observed', 'last_observed'], 'Application': <function split_image>, 'Category': <function to_cat_list>, 'Channel': 'x-oca-event:module', 'EventID': <function to_action_code>, 'Hostname': 'x-oca-asset:hostname', 'Message': <function SdsMapper.<lambda>>, 'ProcessGuid': 'process:x_unique_id', 'ProcessId': 'process:pid', 'ProcessName': <function split_image>, 'SourceName': 'x-oca-event:provider', 'TimeCreated': ['first_observed', 'last_observed']}

convert(event)

detect(event)

static enhanced_action(message)

event_types = {'ConnectPipe': 18, 'CreateKey': 12, 'CreatePipe': 17, 'DeleteKey': 12, 'DeleteValue': 12, 'SetValue': 13}

class firepit.woodchipper.ZeekCsvMapper

Bases: Mapper

convert(event)

detect(event)

zeek_mapping = {'id.orig_h': 'network-traffic:src_ref.value', 'id.orig_p': 'network-traffic:src_port', 'id.resp_h': 'network-traffic:dst_ref.value', 'id.resp_p': 'network-traffic:dst_port', 'orig_ip_bytes': 'network-traffic:src_byte_count', 'orig_pkts': 'network-traffic:src_packets', 'proto': 'network-traffic:protocols', 'resp_ip_bytes': 'network-traffic:dst_byte_count', 'resp_pkts': 'network-traffic:dst_packets', 'ts': <function from_unix_time>}

class firepit.woodchipper.ZeekJsonMapper

Bases: Mapper

common_mapping = {'id_orig_h': 'network-traffic:src_ref.value', 'id_orig_p': 'network-traffic:src_port', 'id_resp_h': 'network-traffic:dst_ref.value', 'id_resp_p': 'network-traffic:dst_port', 'proto': 'network-traffic:protocols', 'ts': <function from_unix_time>}

convert(event)

detect(event)

static process_answers(answers)

zeek_mapping = {'conn': {'orig_ip_bytes': 'network-traffic:src_byte_count', 'orig_l2_addr': 'network-traffic:src_ref.resolves_to_refs[0].value', 'orig_pkts': 'network-traffic:src_packets', 'resp_ip_bytes': 'network-traffic:dst_byte_count', 'resp_l2_addr': 'network-traffic:dst_ref.resolves_to_refs[0].value', 'resp_pkts': 'network-traffic:dst_packets'}, 'dns': {'answers': <function ZeekJsonMapper.<lambda>>, 'query': 'domain-name:value'}}

firepit.woodchipper.convert(input_file, output_file=None)

firepit.woodchipper.convert_to_stix(input_file)

firepit.woodchipper.detect_filetype(input_file)

firepit.woodchipper.dict2observation(creator, row)

firepit.woodchipper.fixup_hashes(hashes: dict)

firepit.woodchipper.format_val(sco_type, prop, val)

firepit.woodchipper.from_unix_time(ts)

firepit.woodchipper.guess_ref_type(sco_type, prop, val)

Get data type for sco_type:prop reference

firepit.woodchipper.is_file_event(event_id)

firepit.woodchipper.merge_mappings(common, specific, key=None)

Merge common mapping into specific[key] mapping

firepit.woodchipper.process_event(event, mapping, event_id=None)

firepit.woodchipper.process_events(events, mappers, ident)

firepit.woodchipper.process_mapping(event, mapping)

firepit.woodchipper.read_csv(fp, mappers, ident)

firepit.woodchipper.read_json(fp, mappers, ident)

firepit.woodchipper.read_log(fp, mappers, ident)

firepit.woodchipper.recreate_dict(obj, prop, rest, val)

firepit.woodchipper.set_obs_prop(observable, path, val, scos, key)

firepit.woodchipper.split_file_hash(hash_string: str)

firepit.woodchipper.split_file_path(abs_name: str, prefix='file:')

firepit.woodchipper.split_hash(hash_string: str, prefix: str, tag: str = '')

firepit.woodchipper.split_image(abs_name: str, prefix='process:')

firepit.woodchipper.split_image_hash(hash_string: str)

firepit.woodchipper.split_image_loaded(abs_name: str)

firepit.woodchipper.split_loaded_hash(hash_string: str)

firepit.woodchipper.split_parent_image(abs_name: str)

firepit.woodchipper.split_reg_key_value(path: str)

firepit.woodchipper.to_action_code(event_id)

Convert windows event ID to x-oca-event action and code

firepit.woodchipper.to_cat_list(category)

firepit.woodchipper.to_payload_bin(value)

firepit.woodchipper.to_protocol(value)

Module contents

Top-level package for STIX Columnar Storage.

firepit.get_storage(url, session_id=None)

Get a storage object for firepit. url will determine the type; a file path means sqlite3. session_id is used in the case of postgresql to partition your data.