capa/scripts/testbed/README.md

Testbed

Goal of the testbed is to support the development of new capa rules. Scripts allow to test rules against a large sample set and to batch process samples, e.g. to freeze features or to generate other meta data used for testing.

The testbed contains malicious and benign files. Data sources are:

• Microsoft EXE and DLL files from C:\Windows\System32, C:\Windows\SysWOW64, etc.
• samples analyzed and annotated by FLARE analysts during malware analysis

Samples containing the keyword slow in their path indicate a longer test run time (>20 seconds) and can be ignored via the -f argument.

Running a rule against a large set of executable programs helps to quickly determine on which functions/samples a rule hits. This helps to identify:

• true positives: hits on expected functions
• false positives: hits on unexpected functions, for example
  • if a rule is to generic or
  • if a rule hits on a capability present in many (benign) samples

To provide additional context the testbed contains function names from the following data sources:

• benign files: function names from Microsoft's PDB information
• malicious files: function names provided by FLARE analysts and obtained from the LabelMaker 2000 (LM2k) annotations repository

For each test sample the testbed contains the following files:

• a .frz file storing the extracted capa features
  • capa's serialized features, via capa.features.freeze
• a .fnames file mapping function addresses to function names
  • JSON file that maps fvas to function names or
  • CSV file with entries idbmd5;md5;fva;fname
• (optional) the binary file with extension .exe_, .dll_, or .mal_

Scripts

run_rule_on_testbed.py

Run a capa rule file against the testbed (frozen features in a directory).

Meant to be run on directories that contain .frz and .fnames files.

Example usage:

run_rule_on_testbed.py <testbed dir>
run_rule_on_testbed.py samples

With the -s <image_path> argument, the script exports images of function graphs to the provided path. Converting the images requires graphviz. See https://graphviz.gitlab.io/about/; get Python interface via pip install graphviz.

Helper Scripts

freeze_features.py

Use freeze_features.py to freeze capa features of a file or of files in a directory.

Example usage:

freeze_features.py <testbed dir>
freeze_features.py samples

start_ida_dump_fnames.py

Start IDA Pro in autonomous mode to dump JSON file of function names {fva: fname}. Processes a single file or a directory.

This script uses _dump_fnames.py to dump the JSON file of functions names and is meant to be run on benign files with PDB information. IDA should apply function names from the PDB information automatically.

Example usage:

start_ida_dump_fnames.py <candidate files dir>
start_ida_dump_fnames.py samples\benign

start_ida_export_fimages.py

Start IDA Pro in autonomous mode to export images of function graphs. run_rule_on_testbed.py integrates the export mechanism (-s option)

This script uses _export_fimages.py to export DOT files of function graphs and then converts them to PNG images using graphviz.

Example usage:

start_ida_export_fimages.py <target file> <output dir> -f <function list>
start_ida_export_fimages.py test.exe imgs -f 0x401000,0x402F90