freeze: fix schema to support overlapping functions

also, move null extractor to its own namespace

.github/CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 First off, thanks for taking the time to contribute!
 
-The following is a set of guidelines for contributing to capa and its packages, which are hosted in the [FireEye Organization](https://github.com/fireeye) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
+The following is a set of guidelines for contributing to capa and its packages, which are hosted in the [Mandiant Organization](https://github.com/mandiant) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
 
 #### Table Of Contents
 
@@ -32,9 +32,9 @@ This project and everyone participating in it is governed by the [Capa Code of C
 ### Capa and its repositories
 
 We host the capa project as three Github repositories:
-  - [capa](https://github.com/fireeye/capa)
-  - [capa-rules](https://github.com/fireeye/capa-rules)
-  - [capa-testfiles](https://github.com/fireeye/capa-testfiles)
+  - [capa](https://github.com/mandiant/capa)
+  - [capa-rules](https://github.com/mandiant/capa-rules)
+  - [capa-testfiles](https://github.com/mandiant/capa-testfiles)
   
 The command line tools, logic engine, and other Python source code are found in the `capa` repository.
 This is the repository to fork when you want to enhance the features, performance, or user interface of capa.
@@ -54,7 +54,7 @@ These are files you'll need in order to run the linter (in `--thorough` mode) an
 ### Design Decisions
 
 When we make a significant decision in how we maintain the project and what we can or cannot support,
- we will document it in the [capa issues tracker](https://github.com/fireeye/capa/issues).
+ we will document it in the [capa issues tracker](https://github.com/mandiant/capa/issues).
 This is the best place review our discussions about what/how/why we do things in the project.
 If you have a question, check to see if it is documented there.
 If it is *not* documented there, or you can't find an answer, please open a issue.
@@ -78,7 +78,7 @@ Fill out [the required template](./ISSUE_TEMPLATE/bug_report.md),
 #### Before Submitting A Bug Report
 
 * **Determine [which repository the problem should be reported in](#capa-and-its-repositories)**.
-* **Perform a [cursory search](https://github.com/fireeye/capa/issues?q=is%3Aissue)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one.
+* **Perform a [cursory search](https://github.com/mandiant/capa/issues?q=is%3Aissue)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one.
 
 #### How Do I Submit A (Good) Bug Report?
 
@@ -101,7 +101,7 @@ Explain the problem and include additional details to help maintainers reproduce
 Provide more context by answering these questions:
 
 * **Did the problem start happening recently** (e.g. after updating to a new version of capa) or was this always a problem?
-* If the problem started happening recently, **can you reproduce the problem in an older version of capa?** What's the most recent version in which the problem doesn't happen? You can download older versions of capa from [the releases page](https://github.com/fireeye/capa/releases).
+* If the problem started happening recently, **can you reproduce the problem in an older version of capa?** What's the most recent version in which the problem doesn't happen? You can download older versions of capa from [the releases page](https://github.com/mandiant/capa/releases).
 * **Can you reliably reproduce the issue?** If not, provide details about how often the problem happens and under which conditions it normally happens.
 * If the problem is related to working with files (e.g. opening and editing files), **does the problem happen for all files and projects or only some?** Does the problem happen only when working with local or remote files (e.g. on network drives), with files of a specific type (e.g. only JavaScript or Python files), with large files or files with very long lines, or with files in a specific encoding? Is there anything else special about the files you are using?
 
@@ -119,7 +119,7 @@ Before creating enhancement suggestions, please check [this list](#before-submit
 #### Before Submitting An Enhancement Suggestion
 
 * **Determine [which repository the enhancement should be suggested in](#capa-and-its-repositories).**
-* **Perform a [cursory search](https://github.com/fireeye/capa/issues?q=is%3Aissue)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
+* **Perform a [cursory search](https://github.com/mandiant/capa/issues?q=is%3Aissue)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
 
 #### How Do I Submit A (Good) Enhancement Suggestion?
 
@@ -138,15 +138,15 @@ Enhancement suggestions are tracked as [GitHub issues](https://guides.github.com
 
 Unsure where to begin contributing to capa? You can start by looking through these `good-first-issue` and `rule-idea` issues:
 
-* [good-first-issue](https://github.com/fireeye/capa/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - issues which should only require a few lines of code, and a test or two.
-* [rule-idea](https://github.com/fireeye/capa-rules/issues?q=is%3Aissue+is%3Aopen+label%3A%22rule+idea%22) - issues that describe potential new rule ideas.
+* [good-first-issue](https://github.com/mandiant/capa/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - issues which should only require a few lines of code, and a test or two.
+* [rule-idea](https://github.com/mandiant/capa-rules/issues?q=is%3Aissue+is%3Aopen+label%3A%22rule+idea%22) - issues that describe potential new rule ideas.
 
 Both issue lists are sorted by total number of comments. While not perfect, number of comments is a reasonable proxy for impact a given change will have.
 
 #### Local development
 
 capa and all its resources can be developed locally. 
-For instructions on how to do this, see the "Method 3" section of the [installation guide](https://github.com/fireeye/capa/blob/master/doc/installation.md).
+For instructions on how to do this, see the "Method 3" section of the [installation guide](https://github.com/mandiant/capa/blob/master/doc/installation.md).
 
 ### Pull Requests
 
@@ -190,8 +190,8 @@ Our CI pipeline will reformat and enforce the Python styleguide.
 
 All (non-nursery) capa rules must:
 
-  1. pass the [linter](https://github.com/fireeye/capa/blob/master/scripts/lint.py), and
-  2. be formatted with [capafmt](https://github.com/fireeye/capa/blob/master/scripts/capafmt.py)
+  1. pass the [linter](https://github.com/mandiant/capa/blob/master/scripts/lint.py), and
+  2. be formatted with [capafmt](https://github.com/mandiant/capa/blob/master/scripts/capafmt.py)
   
 This ensures that all rules meet the same minimum level of quality and are structured in a consistent way.
 Our CI pipeline will reformat and enforce the capa rules styleguide.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -5,16 +5,16 @@ about: Create a report to help us improve
 ---
 <!--
 # Is your bug report related to capa rules (for example a false positive)?
-We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/fireeye/capa-rules/issues.
+We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/mandiant/capa-rules/issues.
 
 # Have you checked that your issue isn't already filed?
-Please search if there is a similar issue at https://github.com/fireeye/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
+Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
 # Have you read capa's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/fireeye/capa/blob/master/.github/CODE_OF_CONDUCT.md
+By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
 
 # Have you read capa's CONTRIBUTING guide?
-It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs
+It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs
 -->
 
 ### Description

.github/ISSUE_TEMPLATE/feature_request.md

@@ -5,16 +5,16 @@ about: Suggest an idea for capa
 ---
 <!--
 # Is your issue related to capa rules (for example an idea for a new rule)?
-We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/fireeye/capa-rules/issues.
+We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/mandiant/capa-rules/issues.
 
 # Have you checked that your issue isn't already filed?
-Please search if there is a similar issue at https://github.com/fireeye/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
+Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
 # Have you read capa's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/fireeye/capa/blob/master/.github/CODE_OF_CONDUCT.md
+By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
 
 # Have you read capa's CONTRIBUTING guide?
-It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements
+It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements
 -->
 
 ### Summary

.github/mypy/mypy.ini

@@ -74,3 +74,6 @@ ignore_missing_imports = True
 
 [mypy-elftools.*]
 ignore_missing_imports = True
+
+[mypy-dncil.*]
+ignore_missing_imports = True

.github/pull_request_template.md

@@ -3,7 +3,7 @@
 Thank you for contributing to capa! <3
 
 Please read capa's CONTRIBUTING guide if you haven't done so already.
-It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md
+It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md
 
 Please describe the changes in this pull request (PR). Include your motivation and context to help us review.
 

.github/pyinstaller/hooks/hook-smda.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 import PyInstaller.utils.hooks
 
 # ref: https://groups.google.com/g/pyinstaller/c/amWi0-66uZI/m/miPoKfWjBAAJ

.github/pyinstaller/hooks/hook-vivisect.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 
 from PyInstaller.utils.hooks import copy_metadata
 
@@ -45,8 +45,8 @@ hiddenimports = [
     "vivisect.analysis.crypto",
     "vivisect.analysis.crypto.constants",
     "vivisect.analysis.elf",
-    "vivisect.analysis.elf",
     "vivisect.analysis.elf.elfplt",
+    "vivisect.analysis.elf.elfplt_late",
     "vivisect.analysis.elf.libc_start_main",
     "vivisect.analysis.generic",
     "vivisect.analysis.generic",

.github/pyinstaller/pyinstaller.spec

@@ -1,5 +1,5 @@
 # -*- mode: python -*-
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 import os.path
 import subprocess
 
@@ -95,7 +95,7 @@ exe = EXE(pyz,
           a.datas,
           exclude_binaries=False,
           name='capa',
-          icon='logo.ico',
+          # TODO not working anymore for unknown reason icon='logo.ico',
           debug=False,
           strip=None,
           upx=True,

.github/workflows/build.yml

@@ -1,8 +1,8 @@
 name: build
 
 on:
-  push:
-    branches: [master]
+  pull_request:
+    branches: [ master ]
   release:
     types: [edited, published]
 
@@ -11,13 +11,15 @@ jobs:
     name: PyInstaller for ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
     strategy:
+      # set to false for debugging
+      fail-fast: true
       matrix:
         include:
           - os: ubuntu-18.04
             # use old linux so that the shared library versioning is more portable
             artifact_name: capa
             asset_name: linux
-          - os: windows-2019
+          - os: windows-2022
             artifact_name: capa.exe
             asset_name: windows
           - os: macos-10.15
@@ -35,14 +37,18 @@ jobs:
           python-version: 3.8
       - if: matrix.os == 'ubuntu-18.04'
         run: sudo apt-get install -y libyaml-dev
-      - name: Install PyInstaller
-        run: pip install 'pyinstaller==4.2'
-      - name: Install capa
-        run: pip install -e .
+      - name: Upgrade pip, setuptools
+        run: pip install --upgrade pip setuptools
+      - name: Install capa with build requirements
+        run: pip install -e .[build]
       - name: Build standalone executable
-        run: pyinstaller .github/pyinstaller/pyinstaller.spec
-      - name: Does it run?
+        run: pyinstaller --log-level DEBUG .github/pyinstaller/pyinstaller.spec
+      - name: Does it run (PE)?
         run: dist/capa "tests/data/Practical Malware Analysis Lab 01-01.dll_"
+      - name: Does it run (Shellcode)?
+        run: dist/capa "tests/data/499c2a85f6e8142c3f48d4251c9c7cd6.raw32"
+      - name: Does it run (ELF)?
+        run: dist/capa "tests/data/7351f8a40c5450557b24622417fc478d.elf_"
       - uses: actions/upload-artifact@v2
         with:
           name: ${{ matrix.asset_name }}
@@ -64,7 +70,7 @@ jobs:
           - os: ubuntu-20.04
             artifact_name: capa
             asset_name: linux
-          - os: windows-2016
+          - os: windows-2022
             artifact_name: capa.exe
             asset_name: windows
     steps:
@@ -73,7 +79,7 @@ jobs:
         with:
           name: ${{ matrix.asset_name }}
       - name: Set executable flag
-        if: matrix.os != 'windows-2016'
+        if: matrix.os != 'windows-2022'      
         run: chmod +x ${{ matrix.artifact_name }}
       - name: Run capa
         run: ./${{ matrix.artifact_name }} -h

.github/workflows/publish.yml

@@ -15,7 +15,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v2
         with:
-          python-version: '3.6'
+          python-version: '3.7'
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

.github/workflows/tag.yml

@@ -12,18 +12,18 @@ jobs:
     - name: Checkout capa-rules
       uses: actions/checkout@v2
       with:
-        repository: fireeye/capa-rules
+        repository: mandiant/capa-rules
         token: ${{ secrets.CAPA_TOKEN }}
     - name: Tag capa-rules
       run: |
         # user information is needed to create annotated tags (with a message)
-        git config user.email 'capa-dev@fireeye.com'
+        git config user.email 'capa-dev@mandiant.com'
         git config user.name 'Capa Bot'
         name=${{ github.event.release.tag_name }}
-        git tag $name -m "https://github.com/fireeye/capa/releases/$name"
+        git tag $name -m "https://github.com/mandiant/capa/releases/$name"
     - name: Push tag to capa-rules
       uses: ad-m/github-push-action@master
       with:
-        repository: fireeye/capa-rules
+        repository: mandiant/capa-rules
         github_token: ${{ secrets.CAPA_TOKEN }}
         tags: true

.github/workflows/tests.yml

@@ -2,9 +2,10 @@ name: CI
 
 on:
   push:
-    branches: [ master ]
+    branches: [ master, dotnet-main ]
   pull_request:
-    branches: [ master ]
+    # TODO cleanup these
+    branches: [ master, dotnet-main, feature-981 ]
 
 # save workspaces to speed up testing
 env:
@@ -30,13 +31,15 @@ jobs:
     - name: Set up Python 3.8
       uses: actions/setup-python@v2
       with:
-        python-version: 3.8
+        python-version: "3.8"
     - name: Install dependencies
       run: pip install -e .[dev]
     - name: Lint with isort
       run: isort --profile black --length-sort --line-width 120 -c .
     - name: Lint with black
       run: black -l 120 --check .
+    - name: Lint with pycodestyle
+      run: pycodestyle --show-source capa/ scripts/ tests/
     - name: Check types with mypy
       run: mypy --config-file .github/mypy/mypy.ini capa/ scripts/ tests/
 
@@ -46,11 +49,11 @@ jobs:
     - name: Checkout capa with submodules
       uses: actions/checkout@v2
       with:
-        submodules: true
+        submodules: recursive
     - name: Set up Python 3.8
       uses: actions/setup-python@v2
       with:
-        python-version: 3.8
+        python-version: "3.8"
     - name: Install capa
       run: pip install -e .
     - name: Run rule linter
@@ -65,18 +68,18 @@ jobs:
       matrix:
         os: [ubuntu-20.04, windows-2019, macos-10.15]
         # across all operating systems
-        python-version: [3.6, 3.9]
+        python-version: ["3.7", "3.10"]
         include:
           # on Ubuntu run these as well
           - os: ubuntu-20.04
-            python-version: 3.7
+            python-version: "3.8"
           - os: ubuntu-20.04
-            python-version: 3.8
+            python-version: "3.9"
     steps:
     - name: Checkout capa with submodules
       uses: actions/checkout@v2
       with:
-        submodules: true
+        submodules: recursive
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
       with:

.gitignore

@@ -114,3 +114,11 @@ venv.bak/
 isort-output.log
 black-output.log
 rule-linter-output.log
+.vscode
+scripts/perf/*.txt
+scripts/perf/*.svg
+scripts/perf/*.zip
+.direnv
+.envrc
+.DS_Store
+*/.DS_Store

CHANGELOG.md

@@ -4,21 +4,213 @@
 
 ### New Features
 
+ - add new scope "instruction" for matching mnemonics and operands #767 @williballenthin
+ - add new feature "operand[{0, 1, 2}].number" for matching instruction operand immediate values #767 @williballenthin
+ - add new feature "operand[{0, 1, 2}].offset" for matching instruction operand offsets #767 @williballenthin
+ - extract additional offset/number features in certain circumstances #320 @williballenthin
+ - add detection and basic feature extraction for dotnet #987 @mr-tz, @mike-hunhoff, @williballenthin
+ - add file string extraction for dotnet files #1012 @mike-hunhoff
+ - add file function-name extraction for dotnet files #1015 @mike-hunhoff
+ - add unmanaged call characteristic for dotnet files #1023 @mike-hunhoff
+ - add mixed mode characteristic feature extraction for dotnet files #1024 @mike-hunhoff
+ - render: support Addresses that aren't simple integers, like .NET token+offset #981 @williballenthin
+
 ### Breaking Changes
 
-### New Rules (0)
+  - instruction scope and operand feature are new and are not backwards compatible with older versions of capa
+  - Python 3.7 is now the minimum supported Python version #866 @williballenthin
+  - remove /x32 and /x64 flavors of number and operand features #932 @williballenthin
+  - the tool now accepts multiple paths to rules, and JSON doc updated accordingly @williballenthin
+  - extractors must use handles to identify functions/basic blocks/instructions #981 @williballenthin
 
+### New Rules (6)
+
+- data-manipulation/encryption/aes/manually-build-aes-constants huynh.t.nhan@gmail.com
+- nursery/get-process-image-filename michael.hunhoff@mandiant.com
+- compiler/v/compiled-with-v jakub.jozwiak@mandiant.com
+- compiler/zig/compiled-with-zig jakub.jozwiak@mandiant.com
+- anti-analysis/packer/huan/packed-with-huan jakub.jozwiak@mandiant.com
+- internal/limitation/file/internal-dotnet-file-limitation william.ballenthin@mandiant.com
 -
 
 ### Bug Fixes
+- improve handling _ prefix compile/link artifact #924 @mike-hunhoff
 
 ### capa explorer IDA Pro plugin
+- improve file format extraction #918 @mike-hunhoff
+- remove decorators added by IDA to ELF imports #919 @mike-hunhoff
 
 ### Development
 
 ### Raw diffs
-- [capa <release>...master](https://github.com/fireeye/capa/compare/v3.0.0...master)
-- [capa-rules <release>...master](https://github.com/fireeye/capa-rules/compare/v3.0.0...master)
+- [capa v3.2.0...master](https://github.com/mandiant/capa/compare/v3.2.0...master)
+- [capa-rules v3.2.0...master](https://github.com/mandiant/capa-rules/compare/v3.2.0...master)
+
+## v3.2.0 (2022-03-03)
+This release adds a new characteristic `characteristic: call $+5` enabling users to create more explicit rules. The linter now also validates ATT&CK and MBC categories. Additionally, many dependencies, including the vivisect backend, have been updated.
+
+One rule has been added and many more have been improved.
+
+Thanks for all the support, especially to @kn0wl3dge and first time contributor @uckelman-sf!
+
+### New Features
+
+- linter: validate ATT&CK/MBC categories and IDs #103 @kn0wl3dge
+- extractor: add characteristic "call $+5" feature #366 @kn0wl3dge
+
+### New Rules (1)
+
+- anti-analysis/obfuscation/obfuscated-with-advobfuscator jakub.jozwiak@mandiant.com
+
+### Bug Fixes
+
+- remove typing package as a requirement for Python 3.7+ compatibility #901 @uckelman-sf
+- elf: fix OS detection for Linux kernel modules #867 @williballenthin
+
+### Raw diffs
+- [capa v3.1.0...v3.2.0](https://github.com/mandiant/capa/compare/v3.1.0...v3.2.0)
+- [capa-rules v3.1.0...v3.2.0](https://github.com/mandiant/capa-rules/compare/v3.1.0...v3.2.0)
+
+## v3.1.0 (2022-01-10)
+This release improves the performance of capa while also adding 23 new rules and many code quality enhancements. We profiled capa's CPU usage and optimized the way that it matches rules, such as by short circuiting when appropriate. According to our testing, the matching phase is approximately 66% faster than v3.0.3! We also added support for Python 3.10, aarch64 builds, and additional MAEC metadata in the rule headers.
+  
+This release adds 23 new rules, including nine by Jakub Jozwiak of Mandiant. @ryantxu1 and @dzbeck updated the ATT&CK and MBC mappings for many rules. Thank you!
+  
+And as always, welcome first time contributors!
+
+  - @kn0wl3dge
+  - @jtothej
+  - @cl30
+  
+
+### New Features
+
+- engine: short circuit logic nodes for better performance #824 @williballenthin
+- engine: add optimizer the order faster nodes first #829 @williballenthin
+- engine: optimize rule evaluation by skipping rules that can't match #830 @williballenthin
+- support python 3.10 #816 @williballenthin
+- support aarch64 #683 @williballenthin
+- rules: support maec/malware-family meta #841 @mr-tz
+- engine: better type annotations/exhaustiveness checking #839 @cl30
+
+### Breaking Changes: None
+
+### New Rules (23)
+
+- nursery/delete-windows-backup-catalog michael.hunhoff@mandiant.com
+- nursery/disable-automatic-windows-recovery-features michael.hunhoff@mandiant.com
+- nursery/capture-webcam-video @johnk3r
+- nursery/create-registry-key-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/delete-registry-key-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/delete-registry-value-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/query-or-enumerate-registry-key-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/query-or-enumerate-registry-value-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/set-registry-value-via-stdregprov michael.hunhoff@mandiant.com
+- data-manipulation/compression/decompress-data-using-ucl jakub.jozwiak@mandiant.com
+- linking/static/wolfcrypt/linked-against-wolfcrypt jakub.jozwiak@mandiant.com
+- linking/static/wolfssl/linked-against-wolfssl jakub.jozwiak@mandiant.com
+- anti-analysis/packer/pespin/packed-with-pespin jakub.jozwiak@mandiant.com
+- load-code/shellcode/execute-shellcode-via-windows-fibers jakub.jozwiak@mandiant.com
+- load-code/shellcode/execute-shellcode-via-enumuilanguages jakub.jozwiak@mandiant.com
+- anti-analysis/packer/themida/packed-with-themida william.ballenthin@mandiant.com
+- load-code/shellcode/execute-shellcode-via-createthreadpoolwait jakub.jozwiak@mandiant.com
+- host-interaction/process/inject/inject-shellcode-using-a-file-mapping-object jakub.jozwiak@mandiant.com
+- load-code/shellcode/execute-shellcode-via-copyfile2 jakub.jozwiak@mandiant.com
+- malware-family/plugx/match-known-plugx-module still@teamt5.org
+
+### Rule Changes
+
+  - update ATT&CK mappings by @ryantxu1
+  - update ATT&CK and MBC mappings by @dzbeck
+  - aplib detection by @cdong1012
+  - golang runtime detection by @stevemk14eber
+
+### Bug Fixes
+
+- fix circular import error #825 @williballenthin
+- fix smda negative number extraction #430 @kn0wl3dge
+
+### capa explorer IDA Pro plugin
+
+- pin supported versions to >= 7.4 and < 8.0 #849 @mike-hunhoff
+
+### Development
+
+- add profiling infrastructure #828 @williballenthin
+- linter: detect shellcode extension #820 @mr-tz
+- show features script: add backend flag #430 @kn0wl3dge
+
+### Raw diffs
+- [capa v3.0.3...v3.1.0](https://github.com/mandiant/capa/compare/v3.0.3...v3.1.0)
+- [capa-rules v3.0.3...v3.1.0](https://github.com/mandiant/capa-rules/compare/v3.0.3...v3.1.0)
+
+
+## v3.0.3 (2021-10-27)
+
+This is primarily a rule maintenance release:
+  - eight new rules, including all relevant techniques from [ATT&CK v10](https://medium.com/mitre-attack/introducing-attack-v10-7743870b37e3), and
+  - two rules removed, due to the prevalence of false positives
+
+We've also tweaked the status codes returned by capa.exe to be more specific and added a bit more metadata to the JSON output format.
+ 
+As always, welcome first time contributors!
+  - still@teamt5.org
+  - zander.work@mandiant.com
+                                                                                                     
+
+### New Features
+
+- show in which function a BB match is #130 @williballenthin
+- main: exit with unique error codes when bailing #802 @williballenthin
+
+### New Rules (8)
+
+- nursery/resolve-function-by-fnv-1a-hash still@teamt5.org
+- data-manipulation/encryption/encrypt-data-using-memfrob-from-glibc zander.work@mandiant.com
+- collection/group-policy/discover-group-policy-via-gpresult william.ballenthin@mandiant.com
+- host-interaction/bootloader/manipulate-safe-mode-programs william.ballenthin@mandiant.com
+- nursery/enable-safe-mode-boot william.ballenthin@mandiant.com
+- persistence/iis/persist-via-iis-module william.ballenthin@mandiant.com
+- persistence/iis/persist-via-isapi-extension william.ballenthin@mandiant.com
+- targeting/language/identify-system-language-via-api william.ballenthin@mandiant.com
+
+## Removed rules (2)
+- load-code/pe/parse-pe-exports: too many false positives in unrelated structure accesses
+- anti-analysis/anti-vm/vm-detection/execute-anti-vm-instructions: too many false positives in junk code
+
+### Bug Fixes
+
+- update references from FireEye to Mandiant
+
+### Raw diffs
+- [capa v3.0.2...v3.0.3](https://github.com/fireeye/capa/compare/v3.0.2...v3.0.3)
+- [capa-rules v3.0.2...v3.0.3](https://github.com/fireeye/capa-rules/compare/v3.0.2...v3.0.3)
+  
+## v3.0.2 (2021-09-28)
+  
+This release fixes an issue with the standalone executables built with PyInstaller when running capa against ELF files.
+
+### Bug Fixes
+
+- fix bug in PyInstaller config preventing ELF analysis #795 @mr-tz
+
+### Raw diffs
+- [capa v3.0.1...v3.0.2](https://github.com/fireeye/capa/compare/v3.0.1...v3.0.2)
+- [capa-rules v3.0.1...v3.0.2](https://github.com/fireeye/capa-rules/compare/v3.0.1...v3.0.2)
+
+## v3.0.1 (2021-09-27)
+
+This version updates the version of vivisect used by capa. Users will experience fewer bugs and find improved analysis results.
+
+Thanks to the community for highlighting issues and analysis misses. Your feedback is crucial to further improve capa.
+
+### Bug Fixes
+
+- fix many underlying bugs in vivisect analysis and update to version v1.0.5 #786 @williballenthin
+
+### Raw diffs
+- [capa v3.0.0...v3.0.1](https://github.com/fireeye/capa/compare/v3.0.0...v3.0.1)
+- [capa-rules v3.0.0...v3.0.1](https://github.com/fireeye/capa-rules/compare/v3.0.0...v3.0.1)
 
 ## v3.0.0 (2021-09-15)
 
@@ -56,29 +248,29 @@ Also, welcome first time contributors:
 ### New Rules (80)
 
 - collection/webcam/capture-webcam-image @johnk3r
-- nursery/list-drag-and-drop-files michael.hunhoff@fireeye.com
-- nursery/monitor-clipboard-content michael.hunhoff@fireeye.com
-- nursery/monitor-local-ipv4-address-changes michael.hunhoff@fireeye.com
-- nursery/load-windows-common-language-runtime michael.hunhoff@fireeye.com
-- nursery/resize-volume-shadow-copy-storage michael.hunhoff@fireeye.com
-- nursery/add-user-account-group michael.hunhoff@fireeye.com
-- nursery/add-user-account-to-group michael.hunhoff@fireeye.com
-- nursery/add-user-account michael.hunhoff@fireeye.com
-- nursery/change-user-account-password michael.hunhoff@fireeye.com
-- nursery/delete-user-account-from-group michael.hunhoff@fireeye.com
-- nursery/delete-user-account-group michael.hunhoff@fireeye.com
-- nursery/delete-user-account michael.hunhoff@fireeye.com
-- nursery/list-domain-servers michael.hunhoff@fireeye.com
-- nursery/list-groups-for-user-account michael.hunhoff@fireeye.com
-- nursery/list-user-account-groups michael.hunhoff@fireeye.com
-- nursery/list-user-accounts-for-group michael.hunhoff@fireeye.com
-- nursery/list-user-accounts michael.hunhoff@fireeye.com
-- nursery/parse-url michael.hunhoff@fireeye.com
-- nursery/register-raw-input-devices michael.hunhoff@fireeye.com
-- anti-analysis/packer/gopacker/packed-with-gopacker jared.wilson@fireeye.com
+- nursery/list-drag-and-drop-files michael.hunhoff@mandiant.com
+- nursery/monitor-clipboard-content michael.hunhoff@mandiant.com
+- nursery/monitor-local-ipv4-address-changes michael.hunhoff@mandiant.com
+- nursery/load-windows-common-language-runtime michael.hunhoff@mandiant.com
+- nursery/resize-volume-shadow-copy-storage michael.hunhoff@mandiant.com
+- nursery/add-user-account-group michael.hunhoff@mandiant.com
+- nursery/add-user-account-to-group michael.hunhoff@mandiant.com
+- nursery/add-user-account michael.hunhoff@mandiant.com
+- nursery/change-user-account-password michael.hunhoff@mandiant.com
+- nursery/delete-user-account-from-group michael.hunhoff@mandiant.com
+- nursery/delete-user-account-group michael.hunhoff@mandiant.com
+- nursery/delete-user-account michael.hunhoff@mandiant.com
+- nursery/list-domain-servers michael.hunhoff@mandiant.com
+- nursery/list-groups-for-user-account michael.hunhoff@mandiant.com
+- nursery/list-user-account-groups michael.hunhoff@mandiant.com
+- nursery/list-user-accounts-for-group michael.hunhoff@mandiant.com
+- nursery/list-user-accounts michael.hunhoff@mandiant.com
+- nursery/parse-url michael.hunhoff@mandiant.com
+- nursery/register-raw-input-devices michael.hunhoff@mandiant.com
+- anti-analysis/packer/gopacker/packed-with-gopacker jared.wilson@mandiant.com
 - host-interaction/driver/create-device-object @mr-tz
 - host-interaction/process/create/execute-command @mr-tz
-- data-manipulation/encryption/create-new-key-via-cryptacquirecontext chuong.dong@fireeye.com
+- data-manipulation/encryption/create-new-key-via-cryptacquirecontext chuong.dong@mandiant.com
 - host-interaction/log/clfs/append-data-to-clfs-log-container blaine.stancill@mandiant.com
 - host-interaction/log/clfs/read-data-from-clfs-log-container blaine.stancill@mandiant.com
 - data-manipulation/encryption/hc-128/encrypt-data-using-hc-128-via-wolfssl blaine.stancill@mandiant.com
@@ -100,15 +292,15 @@ Also, welcome first time contributors:
 - persistence/persist-via-shell-profile-or-rc-file joakim@intezer.com
 - persistence/service/persist-via-rc-script joakim@intezer.com
 - collection/get-current-user-on-linux joakim@intezer.com
-- collection/network/get-mac-address-on-windows moritz.raabe@fireeye.com
-- host-interaction/file-system/read/read-file-on-linux moritz.raabe@fireeye.com joakim@intezer.com
-- host-interaction/file-system/read/read-file-on-windows moritz.raabe@fireeye.com
-- host-interaction/file-system/write/write-file-on-windows william.ballenthin@fireeye.com
-- host-interaction/os/info/get-system-information-on-windows moritz.raabe@fireeye.com joakim@intezer.com
-- host-interaction/process/create/create-process-on-windows moritz.raabe@fireeye.com
-- linking/runtime-linking/link-function-at-runtime-on-windows moritz.raabe@fireeye.com
+- collection/network/get-mac-address-on-windows moritz.raabe@mandiant.com
+- host-interaction/file-system/read/read-file-on-linux moritz.raabe@mandiant.com joakim@intezer.com
+- host-interaction/file-system/read/read-file-on-windows moritz.raabe@mandiant.com
+- host-interaction/file-system/write/write-file-on-windows william.ballenthin@mandiant.com
+- host-interaction/os/info/get-system-information-on-windows moritz.raabe@mandiant.com joakim@intezer.com
+- host-interaction/process/create/create-process-on-windows moritz.raabe@mandiant.com
+- linking/runtime-linking/link-function-at-runtime-on-windows moritz.raabe@mandiant.com
 - nursery/create-process-on-linux joakim@intezer.com
-- nursery/enumerate-files-on-linux william.ballenthin@fireeye.com
+- nursery/enumerate-files-on-linux william.ballenthin@mandiant.com
 - nursery/get-mac-address-on-linux joakim@intezer.com
 - nursery/get-system-information-on-linux joakim@intezer.com
 - nursery/link-function-at-runtime-on-linux joakim@intezer.com
@@ -140,8 +332,8 @@ Also, welcome first time contributors:
 ### Development
 
 ### Raw diffs
-- [capa v2.0.0...v3.0.0](https://github.com/fireeye/capa/compare/v2.0.0...v3.0.0)
-- [capa-rules v2.0.0...v3.0.0](https://github.com/fireeye/capa-rules/compare/v2.0.0...v3.0.0)
+- [capa v2.0.0...v3.0.0](https://github.com/mandiant/capa/compare/v2.0.0...v3.0.0)
+- [capa-rules v2.0.0...v3.0.0](https://github.com/mandiant/capa-rules/compare/v2.0.0...v3.0.0)
 
 
 ## v2.0.0 (2021-07-19)
@@ -160,7 +352,7 @@ A huge thanks to everyone who submitted issues, provided feedback, and contribut
 
 ### New Features
 
-- rules: update ATT&CK and MBC mappings https://github.com/fireeye/capa-rules/pull/317 @williballenthin
+- rules: update ATT&CK and MBC mappings https://github.com/mandiant/capa-rules/pull/317 @williballenthin
 - main: use FLIRT signatures to identify and ignore library code #446 @williballenthin
 - tests: update test cases and caching #545 @mr-tz
 - scripts: capa2yara.py convert capa rules to YARA rules #561 @ruppde
@@ -248,34 +440,34 @@ A huge thanks to everyone who submitted issues, provided feedback, and contribut
 - nursery/run-in-container @williballenthin
 - persistence/registry/appinitdlls/disable-appinit_dlls-code-signature-enforcement @williballenthin
 - collection/password-manager/steal-keepass-passwords-using-keefarce @Ana06
-- host-interaction/network/connectivity/check-internet-connectivity-via-wininet matthew.williams@fireeye.com michael.hunhoff@fireeye.com
+- host-interaction/network/connectivity/check-internet-connectivity-via-wininet matthew.williams@mandiant.com michael.hunhoff@mandiant.com
 - nursery/create-bits-job @mr-tz
 - nursery/execute-syscall-instruction @kulinacs @mr-tz
-- nursery/connect-to-wmi-namespace-via-wbemlocator michael.hunhoff@fireeye.com
+- nursery/connect-to-wmi-namespace-via-wbemlocator michael.hunhoff@mandiant.com
 - anti-analysis/obfuscation/obfuscated-with-callobfuscator johnk3r
 - executable/installer/inno-setup/packaged-as-an-inno-setup-installer awillia2@cisco.com
 - data-manipulation/hashing/djb2/hash-data-using-djb2 awillia2@cisco.com
-- data-manipulation/encoding/base64/decode-data-using-base64-via-dword-translation-table gilbert.elliot@fireeye.com
-- nursery/list-tcp-connections-and-listeners michael.hunhoff@fireeye.com
-- nursery/list-udp-connections-and-listeners michael.hunhoff@fireeye.com
-- nursery/log-keystrokes-via-raw-input-data michael.hunhoff@fireeye.com
-- nursery/register-http-server-url michael.hunhoff@fireeye.com
-- internal/limitation/file/internal-autoit-file-limitation.yml william.ballenthin@fireeye.com
-- internal/limitation/file/internal-dotnet-file-limitation.yml william.ballenthin@fireeye.com
-- internal/limitation/file/internal-installer-file-limitation.yml william.ballenthin@fireeye.com
-- internal/limitation/file/internal-packer-file-limitation.yml william.ballenthin@fireeye.com
+- data-manipulation/encoding/base64/decode-data-using-base64-via-dword-translation-table gilbert.elliot@mandiant.com
+- nursery/list-tcp-connections-and-listeners michael.hunhoff@mandiant.com
+- nursery/list-udp-connections-and-listeners michael.hunhoff@mandiant.com
+- nursery/log-keystrokes-via-raw-input-data michael.hunhoff@mandiant.com
+- nursery/register-http-server-url michael.hunhoff@mandiant.com
+- internal/limitation/file/internal-autoit-file-limitation.yml william.ballenthin@mandiant.com
+- internal/limitation/file/internal-dotnet-file-limitation.yml william.ballenthin@mandiant.com
+- internal/limitation/file/internal-installer-file-limitation.yml william.ballenthin@mandiant.com
+- internal/limitation/file/internal-packer-file-limitation.yml william.ballenthin@mandiant.com
 - host-interaction/network/domain/enumerate-domain-computers-via-ldap awillia2@cisco.com
 - host-interaction/network/domain/get-domain-controller-name awillia2@cisco.com
 - internal/limitation/file/internal-visual-basic-file-limitation @mr-tz
-- data-manipulation/hashing/md5/hash-data-with-md5 moritz.raabe@fireeye.com
+- data-manipulation/hashing/md5/hash-data-with-md5 moritz.raabe@mandiant.com
 - compiler/autohotkey/compiled-with-autohotkey awillia2@cisco.com
 - internal/limitation/file/internal-autohotkey-file-limitation @mr-tz
-- host-interaction/process/dump/create-process-memory-minidump michael.hunhoff@fireeye.com
-- nursery/get-storage-device-properties michael.hunhoff@fireeye.com
-- nursery/execute-shell-command-via-windows-remote-management michael.hunhoff@fireeye.com
-- nursery/get-token-privileges michael.hunhoff@fireeye.com
-- nursery/prompt-user-for-credentials michael.hunhoff@fireeye.com
-- nursery/spoof-parent-pid michael.hunhoff@fireeye.com
+- host-interaction/process/dump/create-process-memory-minidump michael.hunhoff@mandiant.com
+- nursery/get-storage-device-properties michael.hunhoff@mandiant.com
+- nursery/execute-shell-command-via-windows-remote-management michael.hunhoff@mandiant.com
+- nursery/get-token-privileges michael.hunhoff@mandiant.com
+- nursery/prompt-user-for-credentials michael.hunhoff@mandiant.com
+- nursery/spoof-parent-pid michael.hunhoff@mandiant.com
 
 ### Bug Fixes
 
@@ -295,8 +487,8 @@ A huge thanks to everyone who submitted issues, provided feedback, and contribut
 ### Development
 
 - ci: add capa release link to capa-rules tag #517 @Ana06
-- ci, changelog: update `New Rules` section in CHANGELOG automatically https://github.com/fireeye/capa-rules/pull/374 #549 #604 @Ana06
-- ci, changelog: support multiple author in sync GH https://github.com/fireeye/capa-rules/pull/378 @Ana06
+- ci, changelog: update `New Rules` section in CHANGELOG automatically https://github.com/mandiant/capa-rules/pull/374 #549 #604 @Ana06
+- ci, changelog: support multiple author in sync GH https://github.com/mandiant/capa-rules/pull/378 @Ana06
 - ci, lint: check statements for single child statements #563 @mr-tz
 - ci: reject PRs without CHANGELOG update to ensure CHANGELOG is kept up-to-date #584 @Ana06
 - ci: test that scripts run #660 @mr-tz
@@ -304,8 +496,8 @@ A huge thanks to everyone who submitted issues, provided feedback, and contribut
 ### Raw diffs
 
 <!-- The diff uses v1.6.1 because master doesn't include v1.6.2 and v1.6.3 -->
-- [capa v1.6.1...v2.0.0](https://github.com/fireeye/capa/compare/v1.6.1...v2.0.0)
-- [capa-rules v1.6.1...v2.0.0](https://github.com/fireeye/capa-rules/compare/v1.6.1...v2.0.0)
+- [capa v1.6.1...v2.0.0](https://github.com/mandiant/capa/compare/v1.6.1...v2.0.0)
+- [capa-rules v1.6.1...v2.0.0](https://github.com/mandiant/capa-rules/compare/v1.6.1...v2.0.0)
 
 
 ## v1.6.3 (2021-04-29)
@@ -318,7 +510,7 @@ This release adds IDA 7.6 support to capa.
 
 ### Raw diffs
 
-  - [capa v1.6.2...v1.6.3](https://github.com/fireeye/capa/compare/v1.6.2...v1.6.3)
+  - [capa v1.6.2...v1.6.3](https://github.com/mandiant/capa/compare/v1.6.2...v1.6.3)
 
 
 ## v1.6.2 (2021-04-13)
@@ -331,7 +523,7 @@ This release backports a fix to capa 1.6: The Windows binary was built with Pyth
 
 ### Raw diffs
 
-  - [capa v1.6.1...v1.6.2](https://github.com/fireeye/capa/compare/v1.6.1...v1.6.2)
+  - [capa v1.6.1...v1.6.2](https://github.com/mandiant/capa/compare/v1.6.1...v1.6.2)
 
 
 ## v1.6.1 (2021-04-07)
@@ -399,8 +591,8 @@ This release includes several bug fixes, such as a vivisect issue that prevented
 
 ### Raw diffs
 
-  - [capa v1.6.0...v1.6.1](https://github.com/fireeye/capa/compare/v1.6.0...v1.6.1)
-  - [capa-rules v1.6.0...v1.6.1](https://github.com/fireeye/capa-rules/compare/v1.6.0...v1.6.1)
+  - [capa v1.6.0...v1.6.1](https://github.com/mandiant/capa/compare/v1.6.0...v1.6.1)
+  - [capa-rules v1.6.0...v1.6.1](https://github.com/mandiant/capa-rules/compare/v1.6.0...v1.6.1)
 
 
 ## v1.6.0 (2021-03-09)
@@ -409,7 +601,7 @@ This release adds the capa explorer rule generator plugin for IDA Pro, vivisect
 
 ### Rule Generator IDA Plugin
 
-The capa explorer IDA plugin now helps you quickly build new capa rules using features extracted directly from your IDA database. Without leaving the plugin interface you can use the features extracted by capa explorer to develop and test new rules and save your work directly to your capa rules directory. To get started select the new `Rule Generator` tab, navigate to a function in the IDA `Disassembly` view, and click `Analyze`. For more information check out the capa explorer [readme](https://github.com/fireeye/capa/blob/master/capa/ida/plugin/README.md).
+The capa explorer IDA plugin now helps you quickly build new capa rules using features extracted directly from your IDA database. Without leaving the plugin interface you can use the features extracted by capa explorer to develop and test new rules and save your work directly to your capa rules directory. To get started select the new `Rule Generator` tab, navigate to a function in the IDA `Disassembly` view, and click `Analyze`. For more information check out the capa explorer [readme](https://github.com/mandiant/capa/blob/master/capa/ida/plugin/README.md).
 
 ![](doc/img/rulegen_expanded.png)
 
@@ -471,8 +663,8 @@ If you have workflows that rely on the Python 2 version and need future maintena
 
 ### Raw diffs
 
-  - [capa v1.5.1...v1.6.0](https://github.com/fireeye/capa/compare/v1.5.1...v1.6.0)
-  - [capa-rules v1.5.1...v1.6.0](https://github.com/fireeye/capa-rules/compare/v1.5.1...v1.6.0)
+  - [capa v1.5.1...v1.6.0](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
+  - [capa-rules v1.5.1...v1.6.0](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
 
 
 ## v1.5.1 (2021-02-09)
@@ -485,8 +677,8 @@ This release fixes the version number that we forgot to update for v1.5.0 (there
 
 ### Raw diffs
 
-  - [capa v1.5.0...v1.5.1](https://github.com/fireeye/capa/compare/v1.5.1...v1.6.0)
-  - [capa-rules v1.5.0...v1.5.1](https://github.com/fireeye/capa-rules/compare/v1.5.1...v1.6.0)
+  - [capa v1.5.0...v1.5.1](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
+  - [capa-rules v1.5.0...v1.5.1](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
 
 
 ## v1.5.0 (2021-02-05)
@@ -501,7 +693,7 @@ This release brings support for running capa under Python 3 via [SMDA](https://g
   
 @dzbeck also added [Malware Behavior Catalog](https://github.com/MBCProject/mbc-markdown) (MBC) and ATT&CK mappings for many rules.
 
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
 
 
 ### New Features
@@ -584,8 +776,8 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
   
 ### Raw diffs
 
-  - [capa v1.4.1...v1.5.0](https://github.com/fireeye/capa/compare/v1.4.1...v1.5.0)
-  - [capa-rules v1.4.0...v1.5.0](https://github.com/fireeye/capa-rules/compare/v1.4.0...v1.5.0)
+  - [capa v1.4.1...v1.5.0](https://github.com/mandiant/capa/compare/v1.4.1...v1.5.0)
+  - [capa-rules v1.4.0...v1.5.0](https://github.com/mandiant/capa-rules/compare/v1.4.0...v1.5.0)
 
 ## v1.4.1 (2020-10-23)
 
@@ -597,8 +789,8 @@ This release fixes an issue building capa on our CI server, which prevented us f
   
 ### Raw diffs
 
-  - [capa v1.4.0...v1.4.1](https://github.com/fireeye/capa/compare/v1.4.0...v1.4.1)
-  - [capa-rules v1.4.0...v1.4.1](https://github.com/fireeye/capa-rules/compare/v1.4.0...v1.4.1)  
+  - [capa v1.4.0...v1.4.1](https://github.com/mandiant/capa/compare/v1.4.0...v1.4.1)
+  - [capa-rules v1.4.0...v1.4.1](https://github.com/mandiant/capa-rules/compare/v1.4.0...v1.4.1)  
 
 ## v1.4.0 (2020-10-23)
 
@@ -609,7 +801,7 @@ This capa release includes changes to the rule parsing, enhanced feature extract
 
 @dzbeck added [Malware Behavior Catalog](https://github.com/MBCProject/mbc-markdown) (MBC) and ATT&CK mappings for 86 rules.
 
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
 
 ### New features
 
@@ -712,8 +904,8 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
 
 ### Raw diffs
 
-  - [capa v1.3.0...v1.4.0](https://github.com/fireeye/capa/compare/v1.3.0...v1.4.0)
-  - [capa-rules v1.3.0...v1.4.0](https://github.com/fireeye/capa-rules/compare/v1.3.0...v1.4.0)
+  - [capa v1.3.0...v1.4.0](https://github.com/mandiant/capa/compare/v1.3.0...v1.4.0)
+  - [capa-rules v1.3.0...v1.4.0](https://github.com/mandiant/capa-rules/compare/v1.3.0...v1.4.0)
 
 ## v1.3.0 (2020-09-14)
 
@@ -727,7 +919,7 @@ This release brings newly updated mappings to the [Malware Behavior Catalog vers
   - @weslambert
   - @stevemk14ebr 
   
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
 
 ### Key changes to IDA Plugin
 
@@ -737,9 +929,9 @@ The IDA Pro integration is now distributed as a real plugin, instead of a script
   - updates distributed PyPI/`pip install --upgrade` without touching your `%IDADIR%`
   - generally doing thing the "right way"
 
-How to get this new version? Its easy: download [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory and update your capa installation (incidentally, this is a good opportunity to migrate to `pip install flare-capa` instead of git checkouts). Now you should see the plugin listed in the `Edit > Plugins > FLARE capa explorer` menu in IDA. 
+How to get this new version? Its easy: download [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory and update your capa installation (incidentally, this is a good opportunity to migrate to `pip install flare-capa` instead of git checkouts). Now you should see the plugin listed in the `Edit > Plugins > FLARE capa explorer` menu in IDA. 
 
-Please refer to the plugin [readme](https://github.com/fireeye/capa/blob/master/capa/ida/plugin/README.md) for additional information on installing and using the IDA Pro plugin.
+Please refer to the plugin [readme](https://github.com/mandiant/capa/blob/master/capa/ida/plugin/README.md) for additional information on installing and using the IDA Pro plugin.
 
 Please open an issue in this repository if you notice anything weird.
  
@@ -784,8 +976,8 @@ Please open an issue in this repository if you notice anything weird.
 
 ### Raw diffs
 
-  - [capa v1.2.0...v1.3.0](https://github.com/fireeye/capa/compare/v1.2.0...v1.3.0)
-  - [capa-rules v1.2.0...v1.3.0](https://github.com/fireeye/capa-rules/compare/v1.2.0...v1.3.0)
+  - [capa v1.2.0...v1.3.0](https://github.com/mandiant/capa/compare/v1.2.0...v1.3.0)
+  - [capa-rules v1.2.0...v1.3.0](https://github.com/mandiant/capa-rules/compare/v1.2.0...v1.3.0)
 
 ## v1.2.0 (2020-08-31)
 
@@ -801,9 +993,9 @@ We received contributions from ten reverse engineers, including five new ones:
   - @edeca
   - @winniepe 
   
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/).
-Report issues on our [issue tracker](https://github.com/fireeye/capa/issues)
-and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/).
+Report issues on our [issue tracker](https://github.com/mandiant/capa/issues)
+and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
  
 ### New features
 
@@ -882,8 +1074,8 @@ and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/)
 
 ### Raw diffs
 
-  - [capa v1.1.0...v1.2.0](https://github.com/fireeye/capa/compare/v1.1.0...v1.2.0)
-  - [capa-rules v1.1.0...v1.2.0](https://github.com/fireeye/capa-rules/compare/v1.1.0...v1.2.0)
+  - [capa v1.1.0...v1.2.0](https://github.com/mandiant/capa/compare/v1.1.0...v1.2.0)
+  - [capa-rules v1.1.0...v1.2.0](https://github.com/mandiant/capa-rules/compare/v1.1.0...v1.2.0)
 
 ## v1.1.0 (2020-08-05)
 
@@ -896,7 +1088,7 @@ We received contributions from eight reverse engineers, including four new ones:
   - @bitsofbinary
   - @threathive
   
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
   
 ### New features
 
@@ -969,5 +1161,5 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
 
 ### Raw diffs
 
-  - [capa v1.0.0...v1.1.0](https://github.com/fireeye/capa/compare/v1.0.0...v1.1.0)
-  - [capa-rules v1.0.0...v1.1.0](https://github.com/fireeye/capa-rules/compare/v1.0.0...v1.1.0)
+  - [capa v1.0.0...v1.1.0](https://github.com/mandiant/capa/compare/v1.0.0...v1.1.0)
+  - [capa-rules v1.0.0...v1.1.0](https://github.com/mandiant/capa-rules/compare/v1.0.0...v1.1.0)

LICENSE.txt

@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright (C) 2020 FireEye, Inc.
+   Copyright (C) 2020 Mandiant, Inc.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,10 +1,10 @@
-![capa](https://github.com/fireeye/capa/blob/master/.github/logo.png)
+![capa](https://github.com/mandiant/capa/blob/master/.github/logo.png)
 
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/flare-capa)](https://pypi.org/project/flare-capa)
-[![Last release](https://img.shields.io/github/v/release/fireeye/capa)](https://github.com/fireeye/capa/releases)
-[![Number of rules](https://img.shields.io/badge/rules-633-blue.svg)](https://github.com/fireeye/capa-rules)
-[![CI status](https://github.com/fireeye/capa/workflows/CI/badge.svg)](https://github.com/fireeye/capa/actions?query=workflow%3ACI+event%3Apush+branch%3Amaster)
-[![Downloads](https://img.shields.io/github/downloads/fireeye/capa/total)](https://github.com/fireeye/capa/releases)
+[![Last release](https://img.shields.io/github/v/release/mandiant/capa)](https://github.com/mandiant/capa/releases)
+[![Number of rules](https://img.shields.io/badge/rules-665-blue.svg)](https://github.com/mandiant/capa-rules)
+[![CI status](https://github.com/mandiant/capa/workflows/CI/badge.svg)](https://github.com/mandiant/capa/actions?query=workflow%3ACI+event%3Apush+branch%3Amaster)
+[![Downloads](https://img.shields.io/github/downloads/mandiant/capa/total)](https://github.com/mandiant/capa/releases)
 [![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE.txt)
 
 capa detects capabilities in executable files.
@@ -66,11 +66,11 @@ $ capa.exe suspicious.exe
 
 # download and usage
 
-Download stable releases of the standalone capa binaries [here](https://github.com/fireeye/capa/releases). You can run the standalone binaries without installation. capa is a command line tool that should be run from the terminal.
+Download stable releases of the standalone capa binaries [here](https://github.com/mandiant/capa/releases). You can run the standalone binaries without installation. capa is a command line tool that should be run from the terminal.
 
-To use capa as a library or integrate with another tool, see [doc/installation.md](https://github.com/fireeye/capa/blob/master/doc/installation.md) for further setup instructions.
+To use capa as a library or integrate with another tool, see [doc/installation.md](https://github.com/mandiant/capa/blob/master/doc/installation.md) for further setup instructions.
 
-For more information about how to use capa, see [doc/usage.md](https://github.com/fireeye/capa/blob/master/doc/usage.md).
+For more information about how to use capa, see [doc/usage.md](https://github.com/mandiant/capa/blob/master/doc/usage.md).
 
 # example
 
@@ -91,7 +91,7 @@ $ capa.exe suspicious.exe -vv
 ...
 execute shell command and capture output
 namespace   c2/shell
-author      matthew.williams@fireeye.com
+author      matthew.williams@mandiant.com
 scope       function
 att&ck      Execution::Command and Scripting Interpreter::Windows Command Shell [T1059.003]
 references  https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/ns-processthreadsapi-startupinfoa
@@ -127,7 +127,7 @@ rule:
   meta:
     name: hash data with CRC32
     namespace: data-manipulation/checksum/crc32
-    author: moritz.raabe@fireeye.com
+    author: moritz.raabe@mandiant.com
     scope: function
     examples:
       - 2D3EDC218A90F03089CC01715A9F047F:0x403CBD
@@ -142,24 +142,24 @@ rule:
       - api: RtlComputeCrc32
 ```
 
-The [github.com/fireeye/capa-rules](https://github.com/fireeye/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
+The [github.com/mandiant/capa-rules](https://github.com/mandiant/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
 Please learn to write rules and contribute new entries as you find interesting techniques in malware.
 
-If you use IDA Pro, then you can use the [capa explorer](https://github.com/fireeye/capa/tree/master/capa/ida/plugin) plugin.
+If you use IDA Pro, then you can use the [capa explorer](https://github.com/mandiant/capa/tree/master/capa/ida/plugin) plugin.
 capa explorer helps you identify interesting areas of a program and build new capa rules using features extracted directly from your IDA Pro database.
 
-![capa + IDA Pro integration](https://github.com/fireeye/capa/blob/master/doc/img/explorer_expanded.png)
+![capa + IDA Pro integration](https://github.com/mandiant/capa/blob/master/doc/img/explorer_expanded.png)
 
 # further information
 ## capa
-- [Installation](https://github.com/fireeye/capa/blob/master/doc/installation.md)
-- [Usage](https://github.com/fireeye/capa/blob/master/doc/usage.md)
-- [Limitations](https://github.com/fireeye/capa/blob/master/doc/limitations.md)
-- [Contributing Guide](https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md)
+- [Installation](https://github.com/mandiant/capa/blob/master/doc/installation.md)
+- [Usage](https://github.com/mandiant/capa/blob/master/doc/usage.md)
+- [Limitations](https://github.com/mandiant/capa/blob/master/doc/limitations.md)
+- [Contributing Guide](https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md)
 
 ## capa rules
-- [capa-rules repository](https://github.com/fireeye/capa-rules)
-- [capa-rules rule format](https://github.com/fireeye/capa-rules/blob/master/doc/format.md)
+- [capa-rules repository](https://github.com/mandiant/capa-rules)
+- [capa-rules rule format](https://github.com/mandiant/capa-rules/blob/master/doc/format.md)
 
 ## capa testfiles
-The [capa-testfiles repository](https://github.com/fireeye/capa-testfiles) contains the data we use to test capa's code and rules
+The [capa-testfiles repository](https://github.com/mandiant/capa-testfiles) contains the data we use to test capa's code and rules

capa/engine.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -8,11 +8,17 @@
 
 import copy
 import collections
-from typing import Set, Dict, List, Tuple, Union, Mapping, Iterable
+from typing import TYPE_CHECKING, Set, Dict, List, Tuple, Mapping, Iterable
 
-import capa.rules
+import capa.perf
 import capa.features.common
-from capa.features.common import Feature
+from capa.features.common import Result, Feature
+from capa.features.address import Address
+
+if TYPE_CHECKING:
+    # circular import, otherwise
+    import capa.rules
+
 
 # a collection of features and the locations at which they are found.
 #
@@ -21,7 +27,7 @@ from capa.features.common import Feature
 # to collect the locations of a feature, do: `features[Number(0x10)]`
 #
 # aliased here so that the type can be documented and xref'd.
-FeatureSet = Dict[Feature, Set[int]]
+FeatureSet = Dict[Feature, Set[Address]]
 
 
 class Statement:
@@ -45,15 +51,12 @@ class Statement:
     def __repr__(self):
         return str(self)
 
-    def evaluate(self, features: FeatureSet) -> "Result":
+    def evaluate(self, features: FeatureSet, short_circuit=True) -> Result:
         """
         classes that inherit `Statement` must implement `evaluate`
 
         args:
-          ctx (defaultdict[Feature, set[VA]])
-
-        returns:
-          Result
+            short_circuit (bool): if true, then statements like and/or/some may short circuit.
         """
         raise NotImplementedError()
 
@@ -77,70 +80,70 @@ class Statement:
                     children[i] = new
 
 
-class Result:
-    """
-    represents the results of an evaluation of statements against features.
-
-    instances of this class should behave like a bool,
-    e.g. `assert Result(True, ...) == True`
-
-    instances track additional metadata about evaluation results.
-    they contain references to the statement node (e.g. an And statement),
-     as well as the children Result instances.
-
-    we need this so that we can render the tree of expressions and their results.
-    """
-
-    def __init__(self, success: bool, statement: Union[Statement, Feature], children: List["Result"], locations=None):
-        """
-        args:
-          success (bool)
-          statement (capa.engine.Statement or capa.features.Feature)
-          children (list[Result])
-          locations (iterable[VA])
-        """
-        super(Result, self).__init__()
-        self.success = success
-        self.statement = statement
-        self.children = children
-        self.locations = locations if locations is not None else ()
-
-    def __eq__(self, other):
-        if isinstance(other, bool):
-            return self.success == other
-        return False
-
-    def __bool__(self):
-        return self.success
-
-    def __nonzero__(self):
-        return self.success
-
-
 class And(Statement):
-    """match if all of the children evaluate to True."""
+    """
+    match if all of the children evaluate to True.
+
+    the order of evaluation is dictated by the property
+    `And.children` (type: List[Statement|Feature]).
+    a query optimizer may safely manipulate the order of these children.
+    """
 
     def __init__(self, children, description=None):
         super(And, self).__init__(description=description)
         self.children = children
 
-    def evaluate(self, ctx):
-        results = [child.evaluate(ctx) for child in self.children]
-        success = all(results)
-        return Result(success, self, results)
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.and"] += 1
+
+        if short_circuit:
+            results = []
+            for child in self.children:
+                result = child.evaluate(ctx, short_circuit=short_circuit)
+                results.append(result)
+                if not result:
+                    # short circuit
+                    return Result(False, self, results)
+
+            return Result(True, self, results)
+        else:
+            results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
+            success = all(results)
+            return Result(success, self, results)
 
 
 class Or(Statement):
-    """match if any of the children evaluate to True."""
+    """
+    match if any of the children evaluate to True.
+
+    the order of evaluation is dictated by the property
+    `Or.children` (type: List[Statement|Feature]).
+    a query optimizer may safely manipulate the order of these children.
+    """
 
     def __init__(self, children, description=None):
         super(Or, self).__init__(description=description)
         self.children = children
 
-    def evaluate(self, ctx):
-        results = [child.evaluate(ctx) for child in self.children]
-        success = any(results)
-        return Result(success, self, results)
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.or"] += 1
+
+        if short_circuit:
+            results = []
+            for child in self.children:
+                result = child.evaluate(ctx, short_circuit=short_circuit)
+                results.append(result)
+                if result:
+                    # short circuit as soon as we hit one match
+                    return Result(True, self, results)
+
+            return Result(False, self, results)
+        else:
+            results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
+            success = any(results)
+            return Result(success, self, results)
 
 
 class Not(Statement):
@@ -150,28 +153,55 @@ class Not(Statement):
         super(Not, self).__init__(description=description)
         self.child = child
 
-    def evaluate(self, ctx):
-        results = [self.child.evaluate(ctx)]
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.not"] += 1
+
+        results = [self.child.evaluate(ctx, short_circuit=short_circuit)]
         success = not results[0]
         return Result(success, self, results)
 
 
 class Some(Statement):
-    """match if at least N of the children evaluate to True."""
+    """
+    match if at least N of the children evaluate to True.
+
+    the order of evaluation is dictated by the property
+    `Some.children` (type: List[Statement|Feature]).
+    a query optimizer may safely manipulate the order of these children.
+    """
 
     def __init__(self, count, children, description=None):
         super(Some, self).__init__(description=description)
         self.count = count
         self.children = children
 
-    def evaluate(self, ctx):
-        results = [child.evaluate(ctx) for child in self.children]
-        # note that here we cast the child result as a bool
-        # because we've overridden `__bool__` above.
-        #
-        # we can't use `if child is True` because the instance is not True.
-        success = sum([1 for child in results if bool(child) is True]) >= self.count
-        return Result(success, self, results)
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.some"] += 1
+
+        if short_circuit:
+            results = []
+            satisfied_children_count = 0
+            for child in self.children:
+                result = child.evaluate(ctx, short_circuit=short_circuit)
+                results.append(result)
+                if result:
+                    satisfied_children_count += 1
+
+                if satisfied_children_count >= self.count:
+                    # short circuit as soon as we hit the threshold
+                    return Result(True, self, results)
+
+            return Result(False, self, results)
+        else:
+            results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
+            # note that here we cast the child result as a bool
+            # because we've overridden `__bool__` above.
+            #
+            # we can't use `if child is True` because the instance is not True.
+            success = sum([1 for child in results if bool(child) is True]) >= self.count
+            return Result(success, self, results)
 
 
 class Range(Statement):
@@ -183,7 +213,10 @@ class Range(Statement):
         self.min = min if min is not None else 0
         self.max = max if max is not None else (1 << 64 - 1)
 
-    def evaluate(self, ctx):
+    def evaluate(self, ctx, **kwargs):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.range"] += 1
+
         count = len(ctx.get(self.child, []))
         if self.min == 0 and count == 0:
             return Result(True, self, [])
@@ -203,12 +236,12 @@ class Subscope(Statement):
     the engine should preprocess rules to extract subscope statements into their own rules.
     """
 
-    def __init__(self, scope, child):
-        super(Subscope, self).__init__()
+    def __init__(self, scope, child, description=None):
+        super(Subscope, self).__init__(description=description)
         self.scope = scope
         self.child = child
 
-    def evaluate(self, ctx):
+    def evaluate(self, ctx, **kwargs):
         raise ValueError("cannot evaluate a subscope directly!")
 
 
@@ -225,10 +258,10 @@ class Subscope(Statement):
 #         inspect(match_details)
 #
 # aliased here so that the type can be documented and xref'd.
-MatchResults = Mapping[str, List[Tuple[int, Result]]]
+MatchResults = Mapping[str, List[Tuple[Address, Result]]]
 
 
-def index_rule_matches(features: FeatureSet, rule: "capa.rules.Rule", locations: Iterable[int]):
+def index_rule_matches(features: FeatureSet, rule: "capa.rules.Rule", locations: Iterable[Address]):
     """
     record into the given featureset that the given rule matched at the given locations.
 
@@ -245,17 +278,22 @@ def index_rule_matches(features: FeatureSet, rule: "capa.rules.Rule", locations:
             namespace, _, _ = namespace.rpartition("/")
 
 
-def match(rules: List["capa.rules.Rule"], features: FeatureSet, va: int) -> Tuple[FeatureSet, MatchResults]:
+def match(rules: List["capa.rules.Rule"], features: FeatureSet, addr: Address) -> Tuple[FeatureSet, MatchResults]:
     """
-    Args:
-      rules (List[capa.rules.Rule]): these must already be ordered topologically by dependency.
-      features (Mapping[capa.features.Feature, int]):
-      va (int): location of the features
+    match the given rules against the given features,
+    returning an updated set of features and the matches.
 
-    Returns:
-      Tuple[FeatureSet, MatchResults]: two-tuple with entries:
-        - set of features used for matching (which may be a superset of the given `features` argument, due to rule match features), and
-        - mapping from rule name to [(location of match, result object)]
+    the updated features are just like the input,
+    but extended to include the match features (e.g. names of rules that matched).
+    the given feature set is not modified; an updated copy is returned.
+
+    the given list of rules must be ordered topologically by dependency,
+    or else `match` statements will not be handled correctly.
+
+    this routine should be fairly optimized, but is not guaranteed to be the fastest matcher possible.
+    it has a particularly convenient signature: (rules, features) -> matches
+    other strategies can be imagined that match differently; implement these elsewhere.
+    specifically, this routine does "top down" matching of the given rules against the feature set.
     """
     results = collections.defaultdict(list)  # type: MatchResults
 
@@ -266,12 +304,22 @@ def match(rules: List["capa.rules.Rule"], features: FeatureSet, va: int) -> Tupl
     features = collections.defaultdict(set, copy.copy(features))
 
     for rule in rules:
-        res = rule.evaluate(features)
+        res = rule.evaluate(features, short_circuit=True)
         if res:
-            results[rule.name].append((va, res))
+            # we first matched the rule with short circuiting enabled.
+            # this is much faster than without short circuiting.
+            # however, we want to collect all results thoroughly,
+            # so once we've found a match quickly,
+            # go back and capture results without short circuiting.
+            res = rule.evaluate(features, short_circuit=False)
+
+            # sanity check
+            assert bool(res) is True
+
+            results[rule.name].append((addr, res))
             # we need to update the current `features`
             # because subsequent iterations of this loop may use newly added features,
             # such as rule or namespace matches.
-            index_rule_matches(features, rule, [va])
+            index_rule_matches(features, rule, [addr])
 
     return (features, results)

capa/exceptions.py

@@ -0,0 +1,14 @@
+class UnsupportedRuntimeError(RuntimeError):
+    pass
+
+
+class UnsupportedFormatError(ValueError):
+    pass
+
+
+class UnsupportedArchError(ValueError):
+    pass
+
+
+class UnsupportedOSError(ValueError):
+    pass

capa/features/address.py

@@ -0,0 +1,96 @@
+import abc
+
+from dncil.clr.token import Token
+
+
+class Address(abc.ABC):
+    @abc.abstractmethod
+    def __lt__(self, other):
+        # implement < so that addresses can be sorted from low to high
+        ...
+
+    @abc.abstractmethod
+    def __hash__(self):
+        # implement hash so that addresses can be used in sets and dicts
+        ...
+
+    @abc.abstractmethod
+    def __str__(self):
+        # implement str so the address can be rendered in capa output
+        ...
+
+
+class AbsoluteVirtualAddress(int, Address):
+    """an absolute memory address"""
+
+    def __new__(cls, v):
+        assert v >= 0
+        return int.__new__(cls, v)
+
+    def __str__(self):
+        return str(self)
+
+
+class RelativeVirtualAddress(int, Address):
+    """a memory address relative to a base address"""
+
+    pass
+
+
+class FileOffsetAddress(int, Address):
+    """an address relative to the start of a file"""
+
+    def __new__(cls, v):
+        assert v >= 0
+        return int.__new__(cls, v)
+
+    def __str__(self):
+        return str(self)
+
+
+class DNTokenAddress(Address):
+    """a .NET token"""
+
+    def __init__(self, token: Token):
+        self.token = token
+
+    def __lt__(self, other):
+        return self.token.value < other.token.value
+
+    def __hash__(self):
+        return hash(self.token.value)
+
+    def __str__(self):
+        return str(self.token)
+
+
+class DNTokenOffsetAddress(Address):
+    """an offset into an object specified by a .NET token"""
+
+    def __init__(self, token: Token, offset: int):
+        assert offset >= 0
+        self.token = token
+        self.offset = offset
+
+    def __lt__(self, other):
+        return (self.token.value, self.offset) < (other.token.value, other.offset)
+
+    def __hash__(self):
+        return hash((self.token.value, self.offset))
+
+    def __str__(self):
+        return f"{self.token:s}+{hex(self.offset)}"
+
+
+class _NoAddress(Address):
+    def __lt__(self, other):
+        return False
+
+    def __hash__(self):
+        return hash(0)
+
+    def __str__(self):
+        return "no address"
+
+
+NO_ADDRESS = _NoAddress()

capa/features/basicblock.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -20,7 +20,7 @@ class BasicBlock(Feature):
         return ""
 
     def freeze_serialize(self):
-        return (self.__class__.__name__, [])
+        return (self.__class__.__name__, ())
 
     @classmethod
     def freeze_deserialize(cls, args):

capa/features/common.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -7,14 +7,20 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 import re
+import abc
 import codecs
 import logging
 import collections
-from typing import Set, Dict, Union
+from typing import TYPE_CHECKING, Set, Dict, List, Union, Optional, Sequence
 
-import capa.engine
+if TYPE_CHECKING:
+    # circular import, otherwise
+    import capa.engine
+
+import capa.perf
 import capa.features
 import capa.features.extractors.elf
+from capa.features.address import Address
 
 logger = logging.getLogger(__name__)
 MAX_BYTES_FEATURE_SIZE = 0x100
@@ -46,34 +52,66 @@ def escape_string(s: str) -> str:
     return s
 
 
-class Feature:
-    def __init__(self, value: Union[str, int, bytes], bitness=None, description=None):
+class Result:
+    """
+    represents the results of an evaluation of statements against features.
+
+    instances of this class should behave like a bool,
+    e.g. `assert Result(True, ...) == True`
+
+    instances track additional metadata about evaluation results.
+    they contain references to the statement node (e.g. an And statement),
+     as well as the children Result instances.
+
+    we need this so that we can render the tree of expressions and their results.
+    """
+
+    def __init__(
+        self,
+        success: bool,
+        statement: Union["capa.engine.Statement", "Feature"],
+        children: List["Result"],
+        locations: Optional[Set[Address]] = None,
+    ):
+        super(Result, self).__init__()
+        self.success = success
+        self.statement = statement
+        self.children = children
+        self.locations = locations if locations is not None else set()
+
+    def __eq__(self, other):
+        if isinstance(other, bool):
+            return self.success == other
+        return False
+
+    def __bool__(self):
+        return self.success
+
+    def __nonzero__(self):
+        return self.success
+
+
+class Feature(abc.ABC):
+    def __init__(self, value: Union[str, int, float, bytes], description=None):
         """
         Args:
           value (any): the value of the feature, such as the number or string.
-          bitness (str): one of the VALID_BITNESS values, or None.
-            When None, then the feature applies to any bitness.
-            Modifies the feature name from `feature` to `feature/bitness`, like `offset/x32`.
           description (str): a human-readable description that explains the feature value.
         """
         super(Feature, self).__init__()
-
-        if bitness is not None:
-            if bitness not in VALID_BITNESS:
-                raise ValueError("bitness '%s' must be one of %s" % (bitness, VALID_BITNESS))
-            self.name = self.__class__.__name__.lower() + "/" + bitness
-        else:
-            self.name = self.__class__.__name__.lower()
+        self.name = self.__class__.__name__.lower()
 
         self.value = value
-        self.bitness = bitness
         self.description = description
 
     def __hash__(self):
-        return hash((self.name, self.value, self.bitness))
+        return hash((self.name, self.value))
+
+    def __lt__(self, other):
+        return self.freeze_serialize() < other.freeze_serialize()
 
     def __eq__(self, other):
-        return self.name == other.name and self.value == other.value and self.bitness == other.bitness
+        return self.name == other.name and self.value == other.value
 
     def get_value_str(self) -> str:
         """
@@ -96,14 +134,13 @@ class Feature:
     def __repr__(self):
         return str(self)
 
-    def evaluate(self, ctx: Dict["Feature", Set[int]]) -> "capa.engine.Result":
-        return capa.engine.Result(self in ctx, self, [], locations=ctx.get(self, []))
+    def evaluate(self, ctx: Dict["Feature", Set[Address]], **kwargs) -> Result:
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature." + self.name] += 1
+        return Result(self in ctx, self, [], locations=ctx.get(self, set()))
 
     def freeze_serialize(self):
-        if self.bitness is not None:
-            return (self.__class__.__name__, [self.value, {"bitness": self.bitness}])
-        else:
-            return (self.__class__.__name__, [self.value])
+        return (self.__class__.__name__, (self.value,))
 
     @classmethod
     def freeze_deserialize(cls, args):
@@ -116,6 +153,8 @@ class Feature:
             kwargs = args[-1]
             args = args[:-1]
             return cls(*args, **kwargs)
+        else:
+            return cls(*args)
 
 
 class MatchedRule(Feature):
@@ -126,7 +165,6 @@ class MatchedRule(Feature):
 
 class Characteristic(Feature):
     def __init__(self, value: str, description=None):
-
         super(Characteristic, self).__init__(value, description=description)
 
 
@@ -140,7 +178,10 @@ class Substring(String):
         super(Substring, self).__init__(value, description=description)
         self.value = value
 
-    def evaluate(self, ctx):
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.substring"] += 1
+
         # mapping from string value to list of locations.
         # will unique the locations later on.
         matches = collections.defaultdict(list)
@@ -155,6 +196,10 @@ class Substring(String):
 
             if self.value in feature.value:
                 matches[feature.value].extend(locations)
+                if short_circuit:
+                    # we found one matching string, thats sufficient to match.
+                    # don't collect other matching strings in this mode.
+                    break
 
         if matches:
             # finalize: defaultdict -> dict
@@ -170,9 +215,9 @@ class Substring(String):
             # unlike other features, we cannot return put a reference to `self` directly in a `Result`.
             # this is because `self` may match on many strings, so we can't stuff the matched value into it.
             # instead, return a new instance that has a reference to both the substring and the matched values.
-            return capa.engine.Result(True, _MatchedSubstring(self, matches), [], locations=locations)
+            return Result(True, _MatchedSubstring(self, matches), [], locations=locations)
         else:
-            return capa.engine.Result(False, _MatchedSubstring(self, None), [])
+            return Result(False, _MatchedSubstring(self, None), [])
 
     def __str__(self):
         return "substring(%s)" % self.value
@@ -225,7 +270,10 @@ class Regex(String):
                 "invalid regular expression: %s it should use Python syntax, try it at https://pythex.org" % value
             )
 
-    def evaluate(self, ctx):
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.regex"] += 1
+
         # mapping from string value to list of locations.
         # will unique the locations later on.
         matches = collections.defaultdict(list)
@@ -244,6 +292,10 @@ class Regex(String):
             # so that they don't have to prefix/suffix their terms like: /.*foo.*/.
             if self.re.search(feature.value):
                 matches[feature.value].extend(locations)
+                if short_circuit:
+                    # we found one matching string, thats sufficient to match.
+                    # don't collect other matching strings in this mode.
+                    break
 
         if matches:
             # finalize: defaultdict -> dict
@@ -260,9 +312,9 @@ class Regex(String):
             # this is because `self` may match on many strings, so we can't stuff the matched value into it.
             # instead, return a new instance that has a reference to both the regex and the matched values.
             # see #262.
-            return capa.engine.Result(True, _MatchedRegex(self, matches), [], locations=locations)
+            return Result(True, _MatchedRegex(self, matches), [], locations=locations)
         else:
-            return capa.engine.Result(False, _MatchedRegex(self, None), [])
+            return Result(False, _MatchedRegex(self, None), [])
 
     def __str__(self):
         return "regex(string =~ %s)" % self.value
@@ -308,38 +360,36 @@ class Bytes(Feature):
         super(Bytes, self).__init__(value, description=description)
         self.value = value
 
-    def evaluate(self, ctx):
+    def evaluate(self, ctx, **kwargs):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.bytes"] += 1
+
         for feature, locations in ctx.items():
             if not isinstance(feature, (Bytes,)):
                 continue
 
             if feature.value.startswith(self.value):
-                return capa.engine.Result(True, self, [], locations=locations)
+                return Result(True, self, [], locations=locations)
 
-        return capa.engine.Result(False, self, [])
+        return Result(False, self, [])
 
     def get_value_str(self):
         return hex_string(bytes_to_str(self.value))
 
     def freeze_serialize(self):
-        return (self.__class__.__name__, [bytes_to_str(self.value).upper()])
+        return (self.__class__.__name__, (bytes_to_str(self.value).upper(),))
 
     @classmethod
     def freeze_deserialize(cls, args):
         return cls(*[codecs.decode(x, "hex") for x in args])
 
 
-# identifiers for supported bitness names that tweak a feature
-# for example, offset/x32
-BITNESS_X32 = "x32"
-BITNESS_X64 = "x64"
-VALID_BITNESS = (BITNESS_X32, BITNESS_X64)
-
-
 # other candidates here: https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#machine-types
 ARCH_I386 = "i386"
 ARCH_AMD64 = "amd64"
-VALID_ARCH = (ARCH_I386, ARCH_AMD64)
+# dotnet
+ARCH_ANY = "any"
+VALID_ARCH = (ARCH_I386, ARCH_AMD64, ARCH_ANY)
 
 
 class Arch(Feature):
@@ -351,8 +401,10 @@ class Arch(Feature):
 OS_WINDOWS = "windows"
 OS_LINUX = "linux"
 OS_MACOS = "macos"
+# dotnet
+OS_ANY = "any"
 VALID_OS = {os.value for os in capa.features.extractors.elf.OS}
-VALID_OS.update({OS_WINDOWS, OS_LINUX, OS_MACOS})
+VALID_OS.update({OS_WINDOWS, OS_LINUX, OS_MACOS, OS_ANY})
 
 
 class OS(Feature):
@@ -363,7 +415,14 @@ class OS(Feature):
 
 FORMAT_PE = "pe"
 FORMAT_ELF = "elf"
-VALID_FORMAT = (FORMAT_PE, FORMAT_ELF)
+FORMAT_DOTNET = "dotnet"
+VALID_FORMAT = (FORMAT_PE, FORMAT_ELF, FORMAT_DOTNET)
+# internal only, not to be used in rules
+FORMAT_AUTO = "auto"
+FORMAT_SC32 = "sc32"
+FORMAT_SC64 = "sc64"
+FORMAT_FREEZE = "freeze"
+FORMAT_UNKNOWN = "unknown"
 
 
 class Format(Feature):

capa/features/extractors/base_extractor.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -7,23 +7,59 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 import abc
-from typing import Tuple, Iterator, SupportsInt
+import dataclasses
+from typing import Any, Dict, Tuple, Iterator
+from dataclasses import dataclass
 
 from capa.features.common import Feature
+from capa.features.address import Address, AbsoluteVirtualAddress
 
 # feature extractors may reference functions, BBs, insns by opaque handle values.
-# the only requirement of these handles are that they support `__int__`,
-# so that they can be rendered as addresses.
+# you can use the `.address` property to get and render the address of the feature.
 #
 # these handles are only consumed by routines on
 # the feature extractor from which they were created.
-#
-# int(FunctionHandle) -> function start address
-# int(BBHandle) -> BasicBlock start address
-# int(InsnHandle) -> instruction address
-FunctionHandle = SupportsInt
-BBHandle = SupportsInt
-InsnHandle = SupportsInt
+
+
+@dataclass
+class FunctionHandle:
+    """reference to a function recognized by a feature extractor.
+
+    Attributes:
+        address: the address of the function.
+        inner: extractor-specific data.
+        ctx: a context object for the extractor.
+    """
+
+    address: Address
+    inner: Any
+    ctx: Dict[str, Any] = dataclasses.field(default_factory=dict)
+
+
+@dataclass
+class BBHandle:
+    """reference to a basic block recognized by a feature extractor.
+
+    Attributes:
+        address: the address of the basic block start address.
+        inner: extractor-specific data.
+    """
+
+    address: Address
+    inner: Any
+
+
+@dataclass
+class InsnHandle:
+    """reference to a instruction recognized by a feature extractor.
+
+    Attributes:
+        address: the address of the instruction address.
+        inner: extractor-specific data.
+    """
+
+    address: Address
+    inner: Any
 
 
 class FeatureExtractor:
@@ -53,14 +89,14 @@ class FeatureExtractor:
         super(FeatureExtractor, self).__init__()
 
     @abc.abstractmethod
-    def get_base_address(self) -> int:
+    def get_base_address(self) -> AbsoluteVirtualAddress:
         """
         fetch the preferred load address at which the sample was analyzed.
         """
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_global_features(self) -> Iterator[Tuple[Feature, int]]:
+    def extract_global_features(self) -> Iterator[Tuple[Feature, Address]]:
         """
         extract features found at every scope ("global").
 
@@ -71,12 +107,12 @@ class FeatureExtractor:
                 print('0x%x: %s', va, feature)
 
         yields:
-          Tuple[Feature, int]: feature and its location
+          Tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_file_features(self) -> Iterator[Tuple[Feature, int]]:
+    def extract_file_features(self) -> Iterator[Tuple[Feature, Address]]:
         """
         extract file-scope features.
 
@@ -87,7 +123,7 @@ class FeatureExtractor:
                 print('0x%x: %s', va, feature)
 
         yields:
-          Tuple[Feature, int]: feature and its location
+          Tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
@@ -99,32 +135,33 @@ class FeatureExtractor:
         """
         raise NotImplementedError()
 
-    def is_library_function(self, va: int) -> bool:
+    def is_library_function(self, addr: Address) -> bool:
         """
         is the given address a library function?
         the backend may implement its own function matching algorithm, or none at all.
-        we accept a VA here, rather than function object, to handle addresses identified in instructions.
+        we accept an address here, rather than function object,
+         to handle addresses identified in instructions.
 
         this information is used to:
           - filter out matches in library functions (by default), and
           - recognize when to fetch symbol names for called (non-API) functions
 
         args:
-          va (int): the virtual address of a function.
+          addr (Address): the address of a function.
 
         returns:
           bool: True if the given address is the start of a library function.
         """
         return False
 
-    def get_function_name(self, va: int) -> str:
+    def get_function_name(self, addr: Address) -> str:
         """
         fetch any recognized name for the given address.
         this is only guaranteed to return a value when the given function is a recognized library function.
         we accept a VA here, rather than function object, to handle addresses identified in instructions.
 
         args:
-          va (int): the virtual address of a function.
+          addr (Address): the address of a function.
 
         returns:
           str: the function name
@@ -132,10 +169,10 @@ class FeatureExtractor:
         raises:
           KeyError: when the given function does not have a name.
         """
-        raise KeyError(va)
+        raise KeyError(addr)
 
     @abc.abstractmethod
-    def extract_function_features(self, f: FunctionHandle) -> Iterator[Tuple[Feature, int]]:
+    def extract_function_features(self, f: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
         """
         extract function-scope features.
         the arguments are opaque values previously provided by `.get_functions()`, etc.
@@ -144,14 +181,14 @@ class FeatureExtractor:
 
             extractor = VivisectFeatureExtractor(vw, path)
             for function in extractor.get_functions():
-                for feature, va in extractor.extract_function_features(function):
-                    print('0x%x: %s', va, feature)
+                for feature, address in extractor.extract_function_features(function):
+                    print('0x%x: %s', address, feature)
 
         args:
           f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
 
         yields:
-          Tuple[Feature, int]: feature and its location
+          Tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
@@ -164,7 +201,7 @@ class FeatureExtractor:
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_basic_block_features(self, f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, int]]:
+    def extract_basic_block_features(self, f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
         """
         extract basic block-scope features.
         the arguments are opaque values previously provided by `.get_functions()`, etc.
@@ -174,15 +211,15 @@ class FeatureExtractor:
             extractor = VivisectFeatureExtractor(vw, path)
             for function in extractor.get_functions():
                 for bb in extractor.get_basic_blocks(function):
-                    for feature, va in extractor.extract_basic_block_features(function, bb):
-                        print('0x%x: %s', va, feature)
+                    for feature, address in extractor.extract_basic_block_features(function, bb):
+                        print('0x%x: %s', address, feature)
 
         args:
           f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
           bb [BBHandle]: an opaque value previously fetched from `.get_basic_blocks()`.
 
         yields:
-          Tuple[Feature, int]: feature and its location
+          Tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
@@ -195,7 +232,9 @@ class FeatureExtractor:
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_insn_features(self, f: FunctionHandle, bb: BBHandle, insn: InsnHandle) -> Iterator[Tuple[Feature, int]]:
+    def extract_insn_features(
+        self, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
+    ) -> Iterator[Tuple[Feature, Address]]:
         """
         extract instruction-scope features.
         the arguments are opaque values previously provided by `.get_functions()`, etc.
@@ -206,8 +245,8 @@ class FeatureExtractor:
             for function in extractor.get_functions():
                 for bb in extractor.get_basic_blocks(function):
                     for insn in extractor.get_instructions(function, bb):
-                        for feature, va in extractor.extract_insn_features(function, bb, insn):
-                            print('0x%x: %s', va, feature)
+                        for feature, address in extractor.extract_insn_features(function, bb, insn):
+                            print('0x%x: %s', address, feature)
 
         args:
           f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
@@ -215,123 +254,6 @@ class FeatureExtractor:
           insn [InsnHandle]: an opaque value previously fetched from `.get_instructions()`.
 
         yields:
-          Tuple[Feature, int]: feature and its location
+          Tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
-
-
-class NullFeatureExtractor(FeatureExtractor):
-    """
-    An extractor that extracts some user-provided features.
-    The structure of the single parameter is demonstrated in the example below.
-
-    This is useful for testing, as we can provide expected values and see if matching works.
-    Also, this is how we represent features deserialized from a freeze file.
-
-    example::
-
-        extractor = NullFeatureExtractor({
-            'base address: 0x401000,
-            'global features': [
-                (0x0, capa.features.Arch('i386')),
-                (0x0, capa.features.OS('linux')),
-            ],
-            'file features': [
-                (0x402345, capa.features.Characteristic('embedded pe')),
-            ],
-            'functions': {
-                0x401000: {
-                    'features': [
-                        (0x401000, capa.features.Characteristic('nzxor')),
-                    ],
-                    'basic blocks': {
-                        0x401000: {
-                            'features': [
-                                (0x401000, capa.features.Characteristic('tight-loop')),
-                            ],
-                            'instructions': {
-                                0x401000: {
-                                    'features': [
-                                        (0x401000, capa.features.Characteristic('nzxor')),
-                                    ],
-                                },
-                                0x401002: ...
-                            }
-                        },
-                        0x401005: ...
-                    }
-                },
-                0x40200: ...
-            }
-        )
-    """
-
-    def __init__(self, features):
-        super(NullFeatureExtractor, self).__init__()
-        self.features = features
-
-    def get_base_address(self):
-        return self.features["base address"]
-
-    def extract_global_features(self):
-        for p in self.features.get("global features", []):
-            va, feature = p
-            yield feature, va
-
-    def extract_file_features(self):
-        for p in self.features.get("file features", []):
-            va, feature = p
-            yield feature, va
-
-    def get_functions(self):
-        for va in sorted(self.features["functions"].keys()):
-            yield va
-
-    def extract_function_features(self, f):
-        for p in self.features.get("functions", {}).get(f, {}).get("features", []):  # noqa: E127 line over-indented
-            va, feature = p
-            yield feature, va
-
-    def get_basic_blocks(self, f):
-        for va in sorted(
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .keys()
-        ):
-            yield va
-
-    def extract_basic_block_features(self, f, bb):
-        for p in (
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .get(bb, {})
-            .get("features", [])
-        ):
-            va, feature = p
-            yield feature, va
-
-    def get_instructions(self, f, bb):
-        for va in sorted(
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .get(bb, {})
-            .get("instructions", {})
-            .keys()
-        ):
-            yield va
-
-    def extract_insn_features(self, f, bb, insn):
-        for p in (
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .get(bb, {})
-            .get("instructions", {})
-            .get(insn, {})
-            .get("features", [])
-        ):
-            va, feature = p
-            yield feature, va

capa/features/extractors/common.py

@@ -2,33 +2,38 @@ import io
 import logging
 import binascii
 import contextlib
+from typing import Tuple, Iterator
 
 import pefile
 
 import capa.features
 import capa.features.extractors.elf
 import capa.features.extractors.pefile
-from capa.features.common import OS, FORMAT_PE, FORMAT_ELF, OS_WINDOWS, Arch, Format, String
+from capa.features.common import OS, FORMAT_PE, FORMAT_ELF, OS_WINDOWS, FORMAT_FREEZE, Arch, Format, String, Feature
+from capa.features.freeze import is_freeze
+from capa.features.address import NO_ADDRESS, Address, FileOffsetAddress
 
 logger = logging.getLogger(__name__)
 
 
-def extract_file_strings(buf, **kwargs):
+def extract_file_strings(buf, **kwargs) -> Iterator[Tuple[String, Address]]:
     """
     extract ASCII and UTF-16 LE strings from file
     """
     for s in capa.features.extractors.strings.extract_ascii_strings(buf):
-        yield String(s.s), s.offset
+        yield String(s.s), FileOffsetAddress(s.offset)
 
     for s in capa.features.extractors.strings.extract_unicode_strings(buf):
-        yield String(s.s), s.offset
+        yield String(s.s), FileOffsetAddress(s.offset)
 
 
-def extract_format(buf):
+def extract_format(buf) -> Iterator[Tuple[Feature, Address]]:
     if buf.startswith(b"MZ"):
-        yield Format(FORMAT_PE), 0x0
+        yield Format(FORMAT_PE), NO_ADDRESS
     elif buf.startswith(b"\x7fELF"):
-        yield Format(FORMAT_ELF), 0x0
+        yield Format(FORMAT_ELF), NO_ADDRESS
+    elif is_freeze(buf):
+        yield Format(FORMAT_FREEZE), NO_ADDRESS
     else:
         # we likely end up here:
         #  1. handling a file format (e.g. macho)
@@ -38,7 +43,7 @@ def extract_format(buf):
         return
 
 
-def extract_arch(buf):
+def extract_arch(buf) -> Iterator[Tuple[Feature, Address]]:
     if buf.startswith(b"MZ"):
         yield from capa.features.extractors.pefile.extract_file_arch(pe=pefile.PE(data=buf))
 
@@ -50,7 +55,7 @@ def extract_arch(buf):
             logger.debug("unsupported arch: %s", arch)
             return
 
-        yield Arch(arch), 0x0
+        yield Arch(arch), NO_ADDRESS
 
     else:
         # we likely end up here:
@@ -67,9 +72,9 @@ def extract_arch(buf):
         return
 
 
-def extract_os(buf):
+def extract_os(buf) -> Iterator[Tuple[Feature, Address]]:
     if buf.startswith(b"MZ"):
-        yield OS(OS_WINDOWS), 0x0
+        yield OS(OS_WINDOWS), NO_ADDRESS
     elif buf.startswith(b"\x7fELF"):
         with contextlib.closing(io.BytesIO(buf)) as f:
             os = capa.features.extractors.elf.detect_elf_os(f)
@@ -78,7 +83,7 @@ def extract_os(buf):
             logger.debug("unsupported os: %s", os)
             return
 
-        yield OS(os), 0x0
+        yield OS(os), NO_ADDRESS
 
     else:
         # we likely end up here:

capa/features/extractors/dnfile/extractor.py

@@ -0,0 +1,71 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, List, Tuple, Iterator
+
+from capa.features.address import Address, DNTokenAddress, DNTokenOffsetAddress, AbsoluteVirtualAddress
+
+if TYPE_CHECKING:
+    from capa.features.common import Feature
+
+import dnfile
+from dncil.clr.token import Token
+
+import capa.features.extractors
+import capa.features.extractors.dnfile.file
+import capa.features.extractors.dnfile.insn
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
+from capa.features.extractors.dnfile.helpers import get_dotnet_managed_method_bodies
+
+
+class DnfileFeatureExtractor(FeatureExtractor):
+    def __init__(self, path: str):
+        super(DnfileFeatureExtractor, self).__init__()
+        self.pe: dnfile.dnPE = dnfile.dnPE(path)
+
+        # pre-compute these because we'll yield them at *every* scope.
+        self.global_features: List[Tuple[Feature, Address]] = []
+        self.global_features.extend(capa.features.extractors.dotnetfile.extract_file_os(pe=self.pe))
+        self.global_features.extend(capa.features.extractors.dotnetfile.extract_file_arch(pe=self.pe))
+
+    def get_base_address(self):
+        return AbsoluteVirtualAddress(0x0)
+
+    def extract_global_features(self):
+        yield from self.global_features
+
+    def extract_file_features(self):
+        yield from capa.features.extractors.dnfile.file.extract_features(self.pe)
+
+    def get_functions(self) -> Iterator[FunctionHandle]:
+        for token, f in get_dotnet_managed_method_bodies(self.pe):
+            yield FunctionHandle(address=DNTokenAddress(Token(token)), inner=f, ctx={"pe": self.pe})
+
+    def extract_function_features(self, f):
+        # TODO
+        yield from []
+
+    def get_basic_blocks(self, f) -> Iterator[BBHandle]:
+        # each dotnet method is considered 1 basic block
+        yield BBHandle(
+            address=f.address,
+            inner=f.inner,
+        )
+
+    def extract_basic_block_features(self, fh, bbh):
+        # we don't support basic block features
+        yield from []
+
+    def get_instructions(self, fh, bbh):
+        for insn in bbh.inner.instructions:
+            yield InsnHandle(address=DNTokenOffsetAddress(bbh.address.token, insn.offset - fh.inner.offset), inner=insn)
+
+    def extract_insn_features(self, fh, bbh, ih) -> Iterator[Tuple[Feature, Address]]:
+        yield from capa.features.extractors.dnfile.insn.extract_features(fh, bbh, ih)

capa/features/extractors/dnfile/file.py

@@ -0,0 +1,54 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Tuple, Iterator
+
+if TYPE_CHECKING:
+    import dnfile
+    from capa.features.common import Feature, Format, String, Characteristic
+    from capa.features.file import Import, FunctionName
+    from capa.features.address import Address
+
+import capa.features.extractors
+
+
+def extract_file_import_names(pe: dnfile.dnPE) -> Iterator[Tuple[Import, Address]]:
+    yield from capa.features.extractors.dotnetfile.extract_file_import_names(pe=pe)
+
+
+def extract_file_format(pe: dnfile.dnPE) -> Iterator[Tuple[Format, Address]]:
+    yield from capa.features.extractors.dotnetfile.extract_file_format(pe=pe)
+
+
+def extract_file_function_names(pe: dnfile.dnPE) -> Iterator[Tuple[FunctionName, Address]]:
+    yield from capa.features.extractors.dotnetfile.extract_file_function_names(pe=pe)
+
+
+def extract_file_strings(pe: dnfile.dnPE) -> Iterator[Tuple[String, Address]]:
+    yield from capa.features.extractors.dotnetfile.extract_file_strings(pe=pe)
+
+
+def extract_mixed_mode_characteristic_features(pe: dnfile.dnPE) -> Iterator[Tuple[Characteristic, Address]]:
+    yield from capa.features.extractors.dotnetfile.extract_mixed_mode_characteristic_features(pe=pe)
+
+
+def extract_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+    for file_handler in FILE_HANDLERS:
+        for (feature, address) in file_handler(pe):
+            yield feature, address
+
+
+FILE_HANDLERS = (
+    extract_file_import_names,
+    extract_file_function_names,
+    extract_file_strings,
+    extract_file_format,
+    extract_mixed_mode_characteristic_features,
+)

capa/features/extractors/dnfile/helpers.py

@@ -0,0 +1,213 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+from __future__ import annotations
+
+import logging
+from typing import Any, Tuple, Iterator, Optional
+
+import dnfile
+from dncil.cil.body import CilMethodBody
+from dncil.cil.error import MethodBodyFormatError
+from dncil.clr.token import Token, StringToken, InvalidToken
+from dncil.cil.body.reader import CilMethodBodyReaderBase
+
+logger = logging.getLogger(__name__)
+
+# key indexes to dotnet metadata tables
+DOTNET_META_TABLES_BY_INDEX = {table.value: table.name for table in dnfile.enums.MetadataTables}
+
+
+class DnfileMethodBodyReader(CilMethodBodyReaderBase):
+    def __init__(self, pe: dnfile.dnPE, row: dnfile.mdtable.MethodDefRow):
+        self.pe: dnfile.dnPE = pe
+        self.offset: int = self.pe.get_offset_from_rva(row.Rva)
+
+    def read(self, n: int) -> bytes:
+        data: bytes = self.pe.get_data(self.pe.get_rva_from_offset(self.offset), n)
+        self.offset += n
+        return data
+
+    def tell(self) -> int:
+        return self.offset
+
+    def seek(self, offset: int) -> int:
+        self.offset = offset
+        return self.offset
+
+
+def calculate_dotnet_token_value(table: int, rid: int) -> int:
+    return ((table & 0xFF) << Token.TABLE_SHIFT) | (rid & Token.RID_MASK)
+
+
+def resolve_dotnet_token(pe: dnfile.dnPE, token: Token) -> Any:
+    """map generic token to string or table row"""
+    if isinstance(token, StringToken):
+        user_string: Optional[str] = read_dotnet_user_string(pe, token)
+        if user_string is None:
+            return InvalidToken(token.value)
+        return user_string
+
+    table_name: str = DOTNET_META_TABLES_BY_INDEX.get(token.table, "")
+    if not table_name:
+        # table_index is not valid
+        return InvalidToken(token.value)
+
+    table: Any = getattr(pe.net.mdtables, table_name, None)
+    if table is None:
+        # table index is valid but table is not present
+        return InvalidToken(token.value)
+
+    try:
+        return table.rows[token.rid - 1]
+    except IndexError:
+        # table index is valid but row index is not valid
+        return InvalidToken(token.value)
+
+
+def read_dotnet_method_body(pe: dnfile.dnPE, row: dnfile.mdtable.MethodDefRow) -> Optional[CilMethodBody]:
+    """read dotnet method body"""
+    try:
+        return CilMethodBody(DnfileMethodBodyReader(pe, row))
+    except MethodBodyFormatError as e:
+        logger.warn("failed to parse managed method body @ 0x%08x (%s)" % (row.Rva, e))
+        return None
+
+
+def read_dotnet_user_string(pe: dnfile.dnPE, token: StringToken) -> Optional[str]:
+    """read user string from #US stream"""
+    try:
+        user_string: Optional[dnfile.stream.UserString] = pe.net.user_strings.get_us(token.rid)
+    except UnicodeDecodeError as e:
+        logger.warn("failed to decode #US stream index 0x%08x (%s)" % (token.rid, e))
+        return None
+    if user_string is None:
+        return None
+    return user_string.value
+
+
+def get_dotnet_managed_imports(pe: dnfile.dnPE) -> Iterator[Tuple[int, str]]:
+    """get managed imports from MemberRef table
+
+    see https://www.ntcore.com/files/dotnetformat.htm
+
+    10 - MemberRef Table
+        Each row represents an imported method
+            Class (index into the TypeRef, ModuleRef, MethodDef, TypeSpec or TypeDef tables)
+            Name (index into String heap)
+    01 - TypeRef Table
+        Each row represents an imported class, its namespace and the assembly which contains it
+            TypeName (index into String heap)
+            TypeNamespace (index into String heap)
+    """
+    if not is_dotnet_table_valid(pe, "MemberRef"):
+        return
+
+    for (rid, row) in enumerate(pe.net.mdtables.MemberRef):
+        if not isinstance(row.Class.row, (dnfile.mdtable.TypeRefRow,)):
+            continue
+
+        # like File::OpenRead
+        name = f"{row.Class.row.TypeName}::{row.Name}"
+
+        # ECMA II.22.38: TypeNamespace can be null or non-null
+        if row.Class.row.TypeNamespace:
+            # like System.IO.File::OpenRead
+            name = f"{row.Class.row.TypeNamespace}.{name}"
+
+        token: int = calculate_dotnet_token_value(pe.net.mdtables.MemberRef.number, rid + 1)
+
+        yield token, name
+
+
+def get_dotnet_unmanaged_imports(pe: dnfile.dnPE) -> Iterator[Tuple[int, str]]:
+    """get unmanaged imports from ImplMap table
+
+    see https://www.ntcore.com/files/dotnetformat.htm
+
+    28 - ImplMap Table
+        ImplMap table holds information about unmanaged methods that can be reached from managed code, using PInvoke dispatch
+            MemberForwarded (index into the Field or MethodDef table; more precisely, a MemberForwarded coded index)
+            ImportName (index into the String heap)
+            ImportScope (index into the ModuleRef table)
+    """
+    if not is_dotnet_table_valid(pe, "ImplMap"):
+        return
+
+    for row in pe.net.mdtables.ImplMap:
+        dll: str = row.ImportScope.row.Name
+        symbol: str = row.ImportName
+
+        # ECMA says "Each row of the ImplMap table associates a row in the MethodDef table (MemberForwarded) with the
+        # name of a routine (ImportName) in some unmanaged DLL (ImportScope)"; so we calculate and map the MemberForwarded
+        # MethodDef table token to help us later record native import method calls made from CIL
+        token: int = calculate_dotnet_token_value(row.MemberForwarded.table.number, row.MemberForwarded.row_index)
+
+        # like Kernel32.dll
+        if dll and "." in dll:
+            dll = dll.split(".")[0]
+
+        # like kernel32.CreateFileA
+        name: str = f"{dll}.{symbol}"
+
+        yield token, name
+
+
+def get_dotnet_managed_method_bodies(pe: dnfile.dnPE) -> Iterator[Tuple[int, CilMethodBody]]:
+    """get managed methods from MethodDef table"""
+    if not is_dotnet_table_valid(pe, "MethodDef"):
+        return
+
+    for (rid, row) in enumerate(pe.net.mdtables.MethodDef):
+        if not row.ImplFlags.miIL or any((row.Flags.mdAbstract, row.Flags.mdPinvokeImpl)):
+            # skip methods that do not have a method body
+            continue
+
+        body: Optional[CilMethodBody] = read_dotnet_method_body(pe, row)
+        if body is None:
+            continue
+
+        token: int = calculate_dotnet_token_value(dnfile.enums.MetadataTables.MethodDef.value, rid + 1)
+        yield token, body
+
+
+def is_dotnet_table_valid(pe: dnfile.dnPE, table_name: str) -> bool:
+    return bool(getattr(pe.net.mdtables, table_name, None))
+
+
+def get_dotnet_managed_method_names(pe: dnfile.dnPE) -> Iterator[Tuple[int, str]]:
+    """get managed method names from TypeDef table
+
+    see https://www.ntcore.com/files/dotnetformat.htm
+
+    02 - TypeDef Table
+        Each row represents a class in the current assembly.
+            TypeName (index into String heap)
+            TypeNamespace (index into String heap)
+            MethodList (index into MethodDef table; it marks the first of a continguous run of Methods owned by this Type)
+    """
+    if not is_dotnet_table_valid(pe, "TypeDef"):
+        return
+
+    for row in pe.net.mdtables.TypeDef:
+        for index in row.MethodList:
+            # like File::OpenRead
+            name = f"{row.TypeName}::{index.row.Name}"
+
+            # ECMA II.22.37: TypeNamespace can be null or non-null
+            if row.TypeNamespace:
+                # like System.IO.File::OpenRead
+                name = f"{row.TypeNamespace}.{name}"
+
+            token = calculate_dotnet_token_value(index.table.number, index.row_index)
+
+            yield token, name
+
+
+def is_dotnet_mixed_mode(pe: dnfile.dnPE) -> bool:
+    return not bool(pe.net.Flags.CLR_ILONLY)

capa/features/extractors/dnfile/insn.py

@@ -0,0 +1,149 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any, Dict, Tuple, Iterator, Optional
+
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
+
+if TYPE_CHECKING:
+    from dncil.cil.instruction import Instruction
+    from dncil.cil.body import CilMethodBody
+    from capa.features.common import Feature
+    from capa.features.address import Address
+
+import dnfile
+from dncil.clr.token import StringToken, InvalidToken
+from dncil.cil.opcode import OpCodes
+
+import capa.features.extractors.helpers
+from capa.features.insn import API, Number
+from capa.features.common import String, Characteristic
+from capa.features.extractors.dnfile.helpers import (
+    resolve_dotnet_token,
+    read_dotnet_user_string,
+    get_dotnet_managed_imports,
+    get_dotnet_unmanaged_imports,
+    get_dotnet_managed_method_names,
+)
+
+
+def get_managed_imports(ctx: Dict) -> Dict:
+    if "managed_imports_cache" not in ctx:
+        ctx["managed_imports_cache"] = {}
+        for (token, name) in get_dotnet_managed_imports(ctx["pe"]):
+            ctx["managed_imports_cache"][token] = name
+    return ctx["managed_imports_cache"]
+
+
+def get_unmanaged_imports(ctx: Dict) -> Dict:
+    if "unmanaged_imports_cache" not in ctx:
+        ctx["unmanaged_imports_cache"] = {}
+        for (token, name) in get_dotnet_unmanaged_imports(ctx["pe"]):
+            ctx["unmanaged_imports_cache"][token] = name
+    return ctx["unmanaged_imports_cache"]
+
+
+def get_methods(ctx: Dict) -> Dict:
+    if "methods_cache" not in ctx:
+        ctx["methods_cache"] = {}
+        for (token, name) in get_dotnet_managed_method_names(ctx["pe"]):
+            ctx["methods_cache"][token] = name
+    return ctx["methods_cache"]
+
+
+def get_callee_name(ctx: Dict, token: int) -> str:
+    """map dotnet token to method name"""
+    name: str = get_managed_imports(ctx).get(token, "")
+    if not name:
+        # we must check unmanaged imports before managed methods because we map forwarded managed methods
+        # to their unmanaged imports; we prefer a forwarded managed method be mapped to its unmanaged import for analysis
+        name = get_unmanaged_imports(ctx).get(token, "")
+        if not name:
+            name = get_methods(ctx).get(token, "")
+    return name
+
+
+def extract_insn_api_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """parse instruction API features"""
+    insn: Instruction = ih.inner
+
+    if insn.opcode not in (OpCodes.Call, OpCodes.Callvirt, OpCodes.Jmp, OpCodes.Calli):
+        return
+
+    name: str = get_callee_name(fh.ctx, insn.operand.value)
+    if not name:
+        return
+
+    if "::" in name:
+        # like System.IO.File::OpenRead
+        yield API(name), ih.address
+    else:
+        # like kernel32.CreateFileA
+        dll, _, symbol = name.rpartition(".")
+        for name_variant in capa.features.extractors.helpers.generate_symbols(dll, symbol):
+            yield API(name_variant), ih.address
+
+
+def extract_insn_number_features(fh, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """parse instruction number features"""
+    insn: Instruction = ih.inner
+
+    if insn.is_ldc():
+        yield Number(insn.get_ldc()), ih.address
+
+
+def extract_insn_string_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """parse instruction string features"""
+    f: CilMethodBody = fh.inner
+    insn: Instruction = ih.inner
+
+    if not insn.is_ldstr():
+        return
+
+    if not isinstance(insn.operand, StringToken):
+        return
+
+    user_string: Optional[str] = read_dotnet_user_string(fh.ctx["pe"], insn.operand)
+    if user_string is None:
+        return
+
+    yield String(user_string), ih.address
+
+
+def extract_unmanaged_call_characteristic_features(
+    fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Characteristic, Address]]:
+    insn: Instruction = ih.inner
+    if insn.opcode not in (OpCodes.Call, OpCodes.Callvirt, OpCodes.Jmp, OpCodes.Calli):
+        return
+
+    token: Any = resolve_dotnet_token(fh.ctx["pe"], insn.operand)
+    if isinstance(token, InvalidToken):
+        return
+    if not isinstance(token, dnfile.mdtable.MethodDefRow):
+        return
+
+    if any((token.Flags.mdPinvokeImpl, token.ImplFlags.miUnmanaged, token.ImplFlags.miNative)):
+        yield Characteristic("unmanaged call"), insn.offset
+
+
+def extract_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """extract instruction features"""
+    for inst_handler in INSTRUCTION_HANDLERS:
+        for (feature, addr) in inst_handler(fh, bbh, ih):
+            yield feature, addr
+
+
+INSTRUCTION_HANDLERS = (
+    extract_insn_api_features,
+    extract_insn_number_features,
+    extract_insn_string_features,
+    extract_unmanaged_call_characteristic_features,
+)

capa/features/extractors/dnfile_.py

@@ -0,0 +1,116 @@
+import logging
+from typing import Tuple, Iterator
+
+import dnfile
+import pefile
+
+from capa.features.common import OS, OS_ANY, ARCH_ANY, ARCH_I386, ARCH_AMD64, FORMAT_DOTNET, Arch, Format, Feature
+from capa.features.address import NO_ADDRESS, Address, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import FeatureExtractor
+
+logger = logging.getLogger(__name__)
+
+
+def extract_file_format(**kwargs) -> Iterator[Tuple[Feature, Address]]:
+    yield Format(FORMAT_DOTNET), NO_ADDRESS
+
+
+def extract_file_os(**kwargs) -> Iterator[Tuple[Feature, Address]]:
+    yield OS(OS_ANY), NO_ADDRESS
+
+
+def extract_file_arch(pe, **kwargs) -> Iterator[Tuple[Feature, Address]]:
+    # to distinguish in more detail, see https://stackoverflow.com/a/23614024/10548020
+    # .NET 4.5 added option: any CPU, 32-bit preferred
+    if pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE:
+        yield Arch(ARCH_I386), NO_ADDRESS
+    elif not pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE_PLUS:
+        yield Arch(ARCH_AMD64), NO_ADDRESS
+    else:
+        yield Arch(ARCH_ANY), NO_ADDRESS
+
+
+def extract_file_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+    for file_handler in FILE_HANDLERS:
+        for feature, address in file_handler(pe=pe):  # type: ignore
+            yield feature, address
+
+
+FILE_HANDLERS = (
+    # extract_file_export_names,
+    # extract_file_import_names,
+    # extract_file_section_names,
+    # extract_file_strings,
+    # extract_file_function_names,
+    extract_file_format,
+)
+
+
+def extract_global_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+    for handler in GLOBAL_HANDLERS:
+        for feature, addr in handler(pe=pe):  # type: ignore
+            yield feature, addr
+
+
+GLOBAL_HANDLERS = (
+    extract_file_os,
+    extract_file_arch,
+)
+
+
+class DnfileFeatureExtractor(FeatureExtractor):
+    def __init__(self, path: str):
+        super(DnfileFeatureExtractor, self).__init__()
+        self.path: str = path
+        self.pe: dnfile.dnPE = dnfile.dnPE(path)
+
+    def get_base_address(self) -> AbsoluteVirtualAddress:
+        return AbsoluteVirtualAddress(0x0)
+
+    def get_entry_point(self) -> int:
+        # self.pe.net.Flags.CLT_NATIVE_ENTRYPOINT
+        #  True: native EP: Token
+        #  False: managed EP: RVA
+        return self.pe.net.struct.EntryPointTokenOrRva
+
+    def extract_global_features(self):
+        yield from extract_global_features(self.pe)
+
+    def extract_file_features(self):
+        yield from extract_file_features(self.pe)
+
+    def is_dotnet_file(self) -> bool:
+        return bool(self.pe.net)
+
+    def is_mixed_mode(self) -> bool:
+        return not bool(self.pe.net.Flags.CLR_ILONLY)
+
+    def get_runtime_version(self) -> Tuple[int, int]:
+        return self.pe.net.struct.MajorRuntimeVersion, self.pe.net.struct.MinorRuntimeVersion
+
+    def get_meta_version_string(self) -> str:
+        return self.pe.net.metadata.struct.Version.rstrip(b"\x00").decode("utf-8")
+
+    def get_functions(self):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
+
+    def extract_function_features(self, f):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
+
+    def get_basic_blocks(self, f):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
+
+    def extract_basic_block_features(self, f, bb):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
+
+    def get_instructions(self, f, bb):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
+
+    def extract_insn_features(self, f, bb, insn):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
+
+    def is_library_function(self, va):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
+
+    def get_function_name(self, va):
+        raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")

capa/features/extractors/dotnetfile.py

@@ -0,0 +1,163 @@
+import logging
+from typing import Tuple, Iterator
+
+import dnfile
+import pefile
+from dncil.clr.token import Token
+
+import capa.features.extractors.helpers
+from capa.features.file import Import, FunctionName
+from capa.features.common import (
+    OS,
+    OS_ANY,
+    ARCH_ANY,
+    ARCH_I386,
+    ARCH_AMD64,
+    FORMAT_DOTNET,
+    Arch,
+    Format,
+    String,
+    Feature,
+    Characteristic,
+)
+from capa.features.address import NO_ADDRESS, Address, DNTokenAddress, DNTokenOffsetAddress, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import FeatureExtractor
+from capa.features.extractors.dnfile.helpers import (
+    is_dotnet_mixed_mode,
+    get_dotnet_managed_imports,
+    calculate_dotnet_token_value,
+    get_dotnet_unmanaged_imports,
+    get_dotnet_managed_method_names,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def extract_file_format(**kwargs) -> Iterator[Tuple[Format, Address]]:
+    yield Format(FORMAT_DOTNET), NO_ADDRESS
+
+
+def extract_file_import_names(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Import, Address]]:
+    for (token, name) in get_dotnet_managed_imports(pe):
+        # like System.IO.File::OpenRead
+        yield Import(name), DNTokenAddress(Token(token))
+
+    for (token, name) in get_dotnet_unmanaged_imports(pe):
+        # like kernel32.CreateFileA
+        dll, _, symbol = name.rpartition(".")
+        for name_variant in capa.features.extractors.helpers.generate_symbols(dll, symbol):
+            yield Import(name_variant), DNTokenAddress(Token(token))
+
+
+def extract_file_function_names(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[FunctionName, Address]]:
+    for (token, name) in get_dotnet_managed_method_names(pe):
+        yield FunctionName(name), DNTokenAddress(Token(token))
+
+
+def extract_file_os(**kwargs) -> Iterator[Tuple[OS, Address]]:
+    yield OS(OS_ANY), NO_ADDRESS
+
+
+def extract_file_arch(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Arch, Address]]:
+    # to distinguish in more detail, see https://stackoverflow.com/a/23614024/10548020
+    # .NET 4.5 added option: any CPU, 32-bit preferred
+    if pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE:
+        yield Arch(ARCH_I386), NO_ADDRESS
+    elif not pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE_PLUS:
+        yield Arch(ARCH_AMD64), NO_ADDRESS
+    else:
+        yield Arch(ARCH_ANY), NO_ADDRESS
+
+
+def extract_file_strings(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[String, Address]]:
+    yield from capa.features.extractors.common.extract_file_strings(pe.__data__)
+
+
+def extract_mixed_mode_characteristic_features(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Characteristic, Address]]:
+    if is_dotnet_mixed_mode(pe):
+        yield Characteristic("mixed mode"), NO_ADDRESS
+
+
+def extract_file_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+    for file_handler in FILE_HANDLERS:
+        for feature, addr in file_handler(pe=pe):  # type: ignore
+            yield feature, addr
+
+
+FILE_HANDLERS = (
+    extract_file_import_names,
+    extract_file_function_names,
+    extract_file_strings,
+    extract_file_format,
+    extract_mixed_mode_characteristic_features,
+)
+
+
+def extract_global_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+    for handler in GLOBAL_HANDLERS:
+        for feature, va in handler(pe=pe):  # type: ignore
+            yield feature, va
+
+
+GLOBAL_HANDLERS = (
+    extract_file_os,
+    extract_file_arch,
+)
+
+
+class DotnetFileFeatureExtractor(FeatureExtractor):
+    def __init__(self, path: str):
+        super(DotnetFileFeatureExtractor, self).__init__()
+        self.path: str = path
+        self.pe: dnfile.dnPE = dnfile.dnPE(path)
+
+    def get_base_address(self):
+        return AbsoluteVirtualAddress(0x0)
+
+    def get_entry_point(self) -> int:
+        # self.pe.net.Flags.CLT_NATIVE_ENTRYPOINT
+        #  True: native EP: Token
+        #  False: managed EP: RVA
+        return self.pe.net.struct.EntryPointTokenOrRva
+
+    def extract_global_features(self):
+        yield from extract_global_features(self.pe)
+
+    def extract_file_features(self):
+        yield from extract_file_features(self.pe)
+
+    def is_dotnet_file(self) -> bool:
+        return bool(self.pe.net)
+
+    def is_mixed_mode(self) -> bool:
+        return is_dotnet_mixed_mode(self.pe)
+
+    def get_runtime_version(self) -> Tuple[int, int]:
+        return self.pe.net.struct.MajorRuntimeVersion, self.pe.net.struct.MinorRuntimeVersion
+
+    def get_meta_version_string(self) -> str:
+        return self.pe.net.metadata.struct.Version.rstrip(b"\x00").decode("utf-8")
+
+    def get_functions(self):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
+
+    def extract_function_features(self, f):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
+
+    def get_basic_blocks(self, f):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
+
+    def extract_basic_block_features(self, f, bb):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
+
+    def get_instructions(self, f, bb):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
+
+    def extract_insn_features(self, f, bb, insn):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
+
+    def is_library_function(self, va):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
+
+    def get_function_name(self, va):
+        raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")

capa/features/extractors/elf.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -47,7 +47,10 @@ class OS(str, Enum):
     NACL = "nacl"
 
 
-def detect_elf_os(f: BinaryIO) -> str:
+def detect_elf_os(f) -> str:
+    """
+    f: type Union[BinaryIO, IDAIO]
+    """
     f.seek(0x0)
     file_header = f.read(0x40)
 
@@ -77,11 +80,13 @@ def detect_elf_os(f: BinaryIO) -> str:
         raise CorruptElfFile("not an ELF file: invalid ei_data: 0x%02x" % ei_data)
 
     if bitness == 32:
-        (e_phoff,) = struct.unpack_from(endian + "I", file_header, 0x1C)
+        (e_phoff, e_shoff) = struct.unpack_from(endian + "II", file_header, 0x1C)
         e_phentsize, e_phnum = struct.unpack_from(endian + "HH", file_header, 0x2A)
+        e_shentsize, e_shnum = struct.unpack_from(endian + "HH", file_header, 0x2E)
     elif bitness == 64:
-        (e_phoff,) = struct.unpack_from(endian + "Q", file_header, 0x20)
+        (e_phoff, e_shoff) = struct.unpack_from(endian + "QQ", file_header, 0x20)
         e_phentsize, e_phnum = struct.unpack_from(endian + "HH", file_header, 0x36)
+        e_shentsize, e_shnum = struct.unpack_from(endian + "HH", file_header, 0x3A)
     else:
         raise NotImplementedError()
 
@@ -236,6 +241,51 @@ def detect_elf_os(f: BinaryIO) -> str:
             # so we can get the debugging output of subsequent strategies
             ret = OS.LINUX if ret is None else ret
 
+    f.seek(e_shoff)
+    section_header_size = e_shnum * e_shentsize
+    section_headers = f.read(section_header_size)
+    if len(section_headers) != section_header_size:
+        logger.warning("failed to read section headers")
+        e_shnum = 0
+
+    # search for notes stored in sections that aren't visible in program headers.
+    # e.g. .note.Linux in Linux kernel modules.
+    for i in range(e_shnum):
+        offset = i * e_shentsize
+        shent = section_headers[offset : offset + e_shentsize]
+
+        if bitness == 32:
+            sh_name, sh_type, _, sh_addr, sh_offset, sh_size = struct.unpack_from(endian + "IIIIII", shent, 0x0)
+        elif bitness == 64:
+            sh_name, sh_type, _, sh_addr, sh_offset, sh_size = struct.unpack_from(endian + "IIQQQQ", shent, 0x0)
+        else:
+            raise NotImplementedError()
+
+        SHT_NOTE = 0x7
+        if sh_type != SHT_NOTE:
+            continue
+
+        logger.debug("sh_offset: 0x%02x sh_size: 0x%04x", sh_offset, sh_size)
+
+        f.seek(sh_offset)
+        note = f.read(sh_size)
+        if len(note) != sh_size:
+            logger.warning("failed to read note content")
+            continue
+
+        namesz, descsz, type_ = struct.unpack_from(endian + "III", note, 0x0)
+        name_offset = 0xC
+        desc_offset = name_offset + align(namesz, 0x4)
+
+        logger.debug("namesz: 0x%02x descsz: 0x%02x type: 0x%04x", namesz, descsz, type_)
+
+        name = note[name_offset : name_offset + namesz].partition(b"\x00")[0].decode("ascii")
+        logger.debug("name: %s", name)
+
+        if name == "Linux":
+            logger.debug("note owner: %s", "LINUX")
+            ret = OS.LINUX if not ret else ret
+
     return ret.value if ret is not None else "unknown"
 
 

capa/features/extractors/elffile.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -15,6 +15,7 @@ from elftools.elf.elffile import ELFFile, SymbolTableSection
 import capa.features.extractors.common
 from capa.features.file import Import, Section
 from capa.features.common import OS, FORMAT_ELF, Arch, Format, Feature
+from capa.features.address import NO_ADDRESS, FileOffsetAddress, AbsoluteVirtualAddress
 from capa.features.extractors.elf import Arch as ElfArch
 from capa.features.extractors.base_extractor import FeatureExtractor
 
@@ -39,15 +40,15 @@ def extract_file_import_names(elf, **kwargs):
             if symbol.name and symbol.entry.st_info.type == "STT_FUNC":
                 # TODO symbol address
                 # TODO symbol version info?
-                yield Import(symbol.name), 0x0
+                yield Import(symbol.name), FileOffsetAddress(0x0)
 
 
 def extract_file_section_names(elf, **kwargs):
     for section in elf.iter_sections():
         if section.name:
-            yield Section(section.name), section.header.sh_addr
+            yield Section(section.name), AbsoluteVirtualAddress(section.header.sh_addr)
         elif section.is_null():
-            yield Section("NULL"), section.header.sh_addr
+            yield Section("NULL"), AbsoluteVirtualAddress(section.header.sh_addr)
 
 
 def extract_file_strings(buf, **kwargs):
@@ -58,31 +59,31 @@ def extract_file_os(elf, buf, **kwargs):
     # our current approach does not always get an OS value, e.g. for packed samples
     # for file limitation purposes, we're more lax here
     try:
-        os = next(capa.features.extractors.common.extract_os(buf))
-        yield os
+        os_tuple = next(capa.features.extractors.common.extract_os(buf))
+        yield os_tuple
     except StopIteration:
-        yield OS("unknown"), 0x0
+        yield OS("unknown"), NO_ADDRESS
 
 
 def extract_file_format(**kwargs):
-    yield Format(FORMAT_ELF), 0x0
+    yield Format(FORMAT_ELF), NO_ADDRESS
 
 
 def extract_file_arch(elf, **kwargs):
     # TODO merge with capa.features.extractors.elf.detect_elf_arch()
     arch = elf.get_machine_arch()
     if arch == "x86":
-        yield Arch(ElfArch.I386), 0x0
+        yield Arch(ElfArch.I386), NO_ADDRESS
     elif arch == "x64":
-        yield Arch(ElfArch.AMD64), 0x0
+        yield Arch(ElfArch.AMD64), NO_ADDRESS
     else:
         logger.warning("unsupported architecture: %s", arch)
 
 
 def extract_file_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
     for file_handler in FILE_HANDLERS:
-        for feature, va in file_handler(elf=elf, buf=buf):  # type: ignore
-            yield feature, va
+        for feature, addr in file_handler(elf=elf, buf=buf):  # type: ignore
+            yield feature, addr
 
 
 FILE_HANDLERS = (
@@ -97,8 +98,8 @@ FILE_HANDLERS = (
 
 def extract_global_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
     for global_handler in GLOBAL_HANDLERS:
-        for feature, va in global_handler(elf=elf, buf=buf):  # type: ignore
-            yield feature, va
+        for feature, addr in global_handler(elf=elf, buf=buf):  # type: ignore
+            yield feature, addr
 
 
 GLOBAL_HANDLERS = (
@@ -118,21 +119,21 @@ class ElfFeatureExtractor(FeatureExtractor):
         # virtual address of the first segment with type LOAD
         for segment in self.elf.iter_segments():
             if segment.header.p_type == "PT_LOAD":
-                return segment.header.p_vaddr
+                return AbsoluteVirtualAddress(segment.header.p_vaddr)
 
     def extract_global_features(self):
         with open(self.path, "rb") as f:
             buf = f.read()
 
-        for feature, va in extract_global_features(self.elf, buf):
-            yield feature, va
+        for feature, addr in extract_global_features(self.elf, buf):
+            yield feature, addr
 
     def extract_file_features(self):
         with open(self.path, "rb") as f:
             buf = f.read()
 
-        for feature, va in extract_file_features(self.elf, buf):
-            yield feature, va
+        for feature, addr in extract_file_features(self.elf, buf):
+            yield feature, addr
 
     def get_functions(self):
         raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")

capa/features/extractors/helpers.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -29,8 +29,7 @@ def is_aw_function(symbol: str) -> bool:
     if symbol[-1] not in ("A", "W"):
         return False
 
-    # second to last character should be lowercase letter
-    return "a" <= symbol[-2] <= "z" or "0" <= symbol[-2] <= "9"
+    return True
 
 
 def is_ordinal(symbol: str) -> bool:
@@ -52,6 +51,9 @@ def generate_symbols(dll: str, symbol: str) -> Iterator[str]:
       - CreateFileA
       - CreateFile
     """
+    # normalize dll name
+    dll = dll.lower()
+
     # kernel32.CreateFileA
     yield "%s.%s" % (dll, symbol)
 

capa/features/extractors/ida/basicblock.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -8,22 +8,21 @@
 
 import string
 import struct
+from typing import Tuple, Iterator
 
 import idaapi
 
 import capa.features.extractors.ida.helpers
-from capa.features.common import Characteristic
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address
 from capa.features.basicblock import BasicBlock
 from capa.features.extractors.ida import helpers
 from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
+from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
 
 
-def get_printable_len(op):
-    """Return string length if all operand bytes are ascii or utf16-le printable
-
-    args:
-        op (IDA op_t)
-    """
+def get_printable_len(op: idaapi.op_t) -> int:
+    """Return string length if all operand bytes are ascii or utf16-le printable"""
     op_val = capa.features.extractors.ida.helpers.mask_op_val(op)
 
     if op.dtype == idaapi.dt_byte:
@@ -37,12 +36,12 @@ def get_printable_len(op):
     else:
         raise ValueError("Unhandled operand data type 0x%x." % op.dtype)
 
-    def is_printable_ascii(chars):
-        return all(c < 127 and chr(c) in string.printable for c in chars)
+    def is_printable_ascii(chars_: bytes):
+        return all(c < 127 and chr(c) in string.printable for c in chars_)
 
-    def is_printable_utf16le(chars):
-        if all(c == 0x00 for c in chars[1::2]):
-            return is_printable_ascii(chars[::2])
+    def is_printable_utf16le(chars_: bytes):
+        if all(c == 0x00 for c in chars_[1::2]):
+            return is_printable_ascii(chars_[::2])
 
     if is_printable_ascii(chars):
         return idaapi.get_dtype_size(op.dtype)
@@ -53,12 +52,8 @@ def get_printable_len(op):
     return 0
 
 
-def is_mov_imm_to_stack(insn):
-    """verify instruction moves immediate onto stack
-
-    args:
-        insn (IDA insn_t)
-    """
+def is_mov_imm_to_stack(insn: idaapi.insn_t) -> bool:
+    """verify instruction moves immediate onto stack"""
     if insn.Op2.type != idaapi.o_imm:
         return False
 
@@ -71,14 +66,10 @@ def is_mov_imm_to_stack(insn):
     return True
 
 
-def bb_contains_stackstring(f, bb):
+def bb_contains_stackstring(f: idaapi.func_t, bb: idaapi.BasicBlock) -> bool:
     """check basic block for stackstring indicators
 
     true if basic block contains enough moves of constant bytes to the stack
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
     """
     count = 0
     for insn in capa.features.extractors.ida.helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
@@ -89,39 +80,24 @@ def bb_contains_stackstring(f, bb):
     return False
 
 
-def extract_bb_stackstring(f, bb):
-    """extract stackstring indicators from basic block
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-    """
-    if bb_contains_stackstring(f, bb):
-        yield Characteristic("stack string"), bb.start_ea
+def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+    """extract stackstring indicators from basic block"""
+    if bb_contains_stackstring(fh.inner, bbh.inner):
+        yield Characteristic("stack string"), bbh.address
 
 
-def extract_bb_tight_loop(f, bb):
-    """extract tight loop indicators from a basic block
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-    """
-    if capa.features.extractors.ida.helpers.is_basic_block_tight_loop(bb):
-        yield Characteristic("tight loop"), bb.start_ea
+def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+    """extract tight loop indicators from a basic block"""
+    if capa.features.extractors.ida.helpers.is_basic_block_tight_loop(bbh.inner):
+        yield Characteristic("tight loop"), bbh.address
 
 
-def extract_features(f, bb):
-    """extract basic block features
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-    """
+def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+    """extract basic block features"""
     for bb_handler in BASIC_BLOCK_HANDLERS:
-        for (feature, ea) in bb_handler(f, bb):
-            yield feature, ea
-    yield BasicBlock(), bb.start_ea
+        for (feature, addr) in bb_handler(fh, bbh):
+            yield feature, addr
+    yield BasicBlock(), bbh.address
 
 
 BASIC_BLOCK_HANDLERS = (
@@ -132,9 +108,10 @@ BASIC_BLOCK_HANDLERS = (
 
 def main():
     features = []
-    for f in helpers.get_functions(skip_thunks=True, skip_libs=True):
+    for fhandle in helpers.get_functions(skip_thunks=True, skip_libs=True):
+        f: idaapi.func_t = fhandle.inner
         for bb in idaapi.FlowChart(f, flags=idaapi.FC_PREDS):
-            features.extend(list(extract_features(f, bb)))
+            features.extend(list(extract_features(fhandle, bb)))
 
     import pprint
 

capa/features/extractors/ida/extractor.py

@@ -1,10 +1,12 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import List, Tuple, Iterator
+
 import idaapi
 
 import capa.ida.helpers
@@ -14,57 +16,20 @@ import capa.features.extractors.ida.insn
 import capa.features.extractors.ida.global_
 import capa.features.extractors.ida.function
 import capa.features.extractors.ida.basicblock
-from capa.features.extractors.base_extractor import FeatureExtractor
-
-
-class FunctionHandle:
-    """this acts like an idaapi.func_t but with __int__()"""
-
-    def __init__(self, inner):
-        self._inner = inner
-
-    def __int__(self):
-        return self.start_ea
-
-    def __getattr__(self, name):
-        return getattr(self._inner, name)
-
-
-class BasicBlockHandle:
-    """this acts like an idaapi.BasicBlock but with __int__()"""
-
-    def __init__(self, inner):
-        self._inner = inner
-
-    def __int__(self):
-        return self.start_ea
-
-    def __getattr__(self, name):
-        return getattr(self._inner, name)
-
-
-class InstructionHandle:
-    """this acts like an idaapi.insn_t but with __int__()"""
-
-    def __init__(self, inner):
-        self._inner = inner
-
-    def __int__(self):
-        return self.ea
-
-    def __getattr__(self, name):
-        return getattr(self._inner, name)
+from capa.features.common import Feature
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
 
 
 class IdaFeatureExtractor(FeatureExtractor):
     def __init__(self):
         super(IdaFeatureExtractor, self).__init__()
-        self.global_features = []
+        self.global_features: List[Tuple[Feature, Address]] = []
         self.global_features.extend(capa.features.extractors.ida.global_.extract_os())
         self.global_features.extend(capa.features.extractors.ida.global_.extract_arch())
 
     def get_base_address(self):
-        return idaapi.get_imagebase()
+        return AbsoluteVirtualAddress(idaapi.get_imagebase())
 
     def extract_global_features(self):
         yield from self.global_features
@@ -72,41 +37,34 @@ class IdaFeatureExtractor(FeatureExtractor):
     def extract_file_features(self):
         yield from capa.features.extractors.ida.file.extract_features()
 
-    def get_functions(self):
+    def get_functions(self) -> Iterator[FunctionHandle]:
         import capa.features.extractors.ida.helpers as ida_helpers
 
-        # data structure shared across functions yielded here.
-        # useful for caching analysis relevant across a single workspace.
-        ctx = {}
-
         # ignore library functions and thunk functions as identified by IDA
-        for f in ida_helpers.get_functions(skip_thunks=True, skip_libs=True):
-            setattr(f, "ctx", ctx)
-            yield FunctionHandle(f)
+        yield from ida_helpers.get_functions(skip_thunks=True, skip_libs=True)
 
     @staticmethod
-    def get_function(ea):
+    def get_function(ea: int) -> FunctionHandle:
         f = idaapi.get_func(ea)
-        setattr(f, "ctx", {})
-        return FunctionHandle(f)
+        return FunctionHandle(address=AbsoluteVirtualAddress(f.start_ea), inner=f)
 
-    def extract_function_features(self, f):
-        yield from capa.features.extractors.ida.function.extract_features(f)
+    def extract_function_features(self, fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+        yield from capa.features.extractors.ida.function.extract_features(fh)
 
-    def get_basic_blocks(self, f):
+    def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
         import capa.features.extractors.ida.helpers as ida_helpers
 
-        for bb in ida_helpers.get_function_blocks(f):
-            yield BasicBlockHandle(bb)
+        for bb in ida_helpers.get_function_blocks(fh.inner):
+            yield BBHandle(address=AbsoluteVirtualAddress(bb.start_ea), inner=bb)
 
-    def extract_basic_block_features(self, f, bb):
-        yield from capa.features.extractors.ida.basicblock.extract_features(f, bb)
+    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+        yield from capa.features.extractors.ida.basicblock.extract_features(fh, bbh)
 
-    def get_instructions(self, f, bb):
+    def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:
         import capa.features.extractors.ida.helpers as ida_helpers
 
-        for insn in ida_helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
-            yield InstructionHandle(insn)
+        for insn in ida_helpers.get_instructions_in_range(bbh.inner.start_ea, bbh.inner.end_ea):
+            yield InsnHandle(address=AbsoluteVirtualAddress(insn.ea), inner=insn)
 
-    def extract_insn_features(self, f, bb, insn):
-        yield from capa.features.extractors.ida.insn.extract_features(f, bb, insn)
+    def extract_insn_features(self, fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle):
+        yield from capa.features.extractors.ida.insn.extract_features(fh, bbh, ih)

capa/features/extractors/ida/file.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -7,27 +7,26 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 import struct
+from typing import Tuple, Iterator
 
 import idc
 import idaapi
 import idautils
-import ida_loader
 
+import capa.features.extractors.common
 import capa.features.extractors.helpers
 import capa.features.extractors.strings
 import capa.features.extractors.ida.helpers
 from capa.features.file import Export, Import, Section, FunctionName
-from capa.features.common import OS, FORMAT_PE, FORMAT_ELF, OS_WINDOWS, Format, String, Characteristic
+from capa.features.common import FORMAT_PE, FORMAT_ELF, Format, Feature, Characteristic
+from capa.features.address import NO_ADDRESS, Address, FileOffsetAddress, AbsoluteVirtualAddress
 
 
-def check_segment_for_pe(seg):
+def check_segment_for_pe(seg: idaapi.segment_t) -> Iterator[Tuple[int, int]]:
     """check segment for embedded PE
 
     adapted for IDA from:
     https://github.com/vivisect/vivisect/blob/7be4037b1cecc4551b397f840405a1fc606f9b53/PE/carve.py#L19
-
-    args:
-        seg (IDA segment_t)
     """
     seg_max = seg.end_ea
     mz_xor = [
@@ -60,13 +59,13 @@ def check_segment_for_pe(seg):
             continue
 
         if idc.get_bytes(peoff, 2) == pex:
-            yield (off, i)
+            yield off, i
 
         for nextres in capa.features.extractors.ida.helpers.find_byte_sequence(off + 1, seg.end_ea, mzx):
             todo.append((nextres, mzx, pex, i))
 
 
-def extract_file_embedded_pe():
+def extract_file_embedded_pe() -> Iterator[Tuple[Feature, Address]]:
     """extract embedded PE features
 
     IDA must load resource sections for this to be complete
@@ -75,16 +74,16 @@ def extract_file_embedded_pe():
     """
     for seg in capa.features.extractors.ida.helpers.get_segments(skip_header_segments=True):
         for (ea, _) in check_segment_for_pe(seg):
-            yield Characteristic("embedded pe"), ea
+            yield Characteristic("embedded pe"), FileOffsetAddress(ea)
 
 
-def extract_file_export_names():
+def extract_file_export_names() -> Iterator[Tuple[Feature, Address]]:
     """extract function exports"""
     for (_, _, ea, name) in idautils.Entries():
-        yield Export(name), ea
+        yield Export(name), AbsoluteVirtualAddress(ea)
 
 
-def extract_file_import_names():
+def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
     """extract function imports
 
     1. imports by ordinal:
@@ -96,11 +95,12 @@ def extract_file_import_names():
      - importname
     """
     for (ea, info) in capa.features.extractors.ida.helpers.get_file_imports().items():
+        addr = AbsoluteVirtualAddress(ea)
         if info[1] and info[2]:
             # e.g. in mimikatz: ('cabinet', 'FCIAddFile', 11L)
             # extract by name here and by ordinal below
             for name in capa.features.extractors.helpers.generate_symbols(info[0], info[1]):
-                yield Import(name), ea
+                yield Import(name), addr
             dll = info[0]
             symbol = "#%d" % (info[2])
         elif info[1]:
@@ -113,10 +113,10 @@ def extract_file_import_names():
             continue
 
         for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
-            yield Import(name), ea
+            yield Import(name), addr
 
 
-def extract_file_section_names():
+def extract_file_section_names() -> Iterator[Tuple[Feature, Address]]:
     """extract section names
 
     IDA must load resource sections for this to be complete
@@ -124,10 +124,10 @@ def extract_file_section_names():
         - Check 'Load resource sections' when opening binary in IDA manually
     """
     for seg in capa.features.extractors.ida.helpers.get_segments(skip_header_segments=True):
-        yield Section(idaapi.get_segm_name(seg)), seg.start_ea
+        yield Section(idaapi.get_segm_name(seg)), AbsoluteVirtualAddress(seg.start_ea)
 
 
-def extract_file_strings():
+def extract_file_strings() -> Iterator[Tuple[Feature, Address]]:
     """extract ASCII and UTF-16 LE strings
 
     IDA must load resource sections for this to be complete
@@ -136,42 +136,45 @@ def extract_file_strings():
     """
     for seg in capa.features.extractors.ida.helpers.get_segments():
         seg_buff = capa.features.extractors.ida.helpers.get_segment_buffer(seg)
-
-        for s in capa.features.extractors.strings.extract_ascii_strings(seg_buff):
-            yield String(s.s), (seg.start_ea + s.offset)
-
-        for s in capa.features.extractors.strings.extract_unicode_strings(seg_buff):
-            yield String(s.s), (seg.start_ea + s.offset)
+        yield from capa.features.extractors.common.extract_file_strings(seg_buff)
 
 
-def extract_file_function_names():
+def extract_file_function_names() -> Iterator[Tuple[Feature, Address]]:
     """
     extract the names of statically-linked library functions.
     """
     for ea in idautils.Functions():
+        addr = AbsoluteVirtualAddress(ea)
         if idaapi.get_func(ea).flags & idaapi.FUNC_LIB:
             name = idaapi.get_name(ea)
-            yield FunctionName(name), ea
+            yield FunctionName(name), addr
+            if name.startswith("_"):
+                # some linkers may prefix linked routines with a `_` to avoid name collisions.
+                # extract features for both the mangled and un-mangled representations.
+                # e.g. `_fwrite` -> `fwrite`
+                # see: https://stackoverflow.com/a/2628384/87207
+                yield FunctionName(name[1:]), addr
 
 
-def extract_file_format():
-    format_name = ida_loader.get_file_type_name()
+def extract_file_format() -> Iterator[Tuple[Feature, Address]]:
+    file_info = idaapi.get_inf_structure()
 
-    if "PE" in format_name:
-        yield Format(FORMAT_PE), 0x0
-    elif "ELF64" in format_name:
-        yield Format(FORMAT_ELF), 0x0
-    elif "ELF32" in format_name:
-        yield Format(FORMAT_ELF), 0x0
+    if file_info.filetype == idaapi.f_PE:
+        yield Format(FORMAT_PE), NO_ADDRESS
+    elif file_info.filetype == idaapi.f_ELF:
+        yield Format(FORMAT_ELF), NO_ADDRESS
+    elif file_info.filetype == idaapi.f_BIN:
+        # no file type to return when processing a binary file, but we want to continue processing
+        return
     else:
-        raise NotImplementedError("file format: %s", format_name)
+        raise NotImplementedError("file format: %d" % file_info.filetype)
 
 
-def extract_features():
+def extract_features() -> Iterator[Tuple[Feature, Address]]:
     """extract file features"""
     for file_handler in FILE_HANDLERS:
-        for feature, va in file_handler():
-            yield feature, va
+        for feature, addr in file_handler():
+            yield feature, addr
 
 
 FILE_HANDLERS = (

capa/features/extractors/ida/function.py

@@ -1,35 +1,31 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import Tuple, Iterator
 
 import idaapi
 import idautils
 
 import capa.features.extractors.ida.helpers
-from capa.features.common import Characteristic
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
 from capa.features.extractors import loops
+from capa.features.extractors.base_extractor import FunctionHandle
 
 
-def extract_function_calls_to(f):
-    """extract callers to a function
-
-    args:
-        f (IDA func_t)
-    """
-    for ea in idautils.CodeRefsTo(f.start_ea, True):
-        yield Characteristic("calls to"), ea
+def extract_function_calls_to(fh: FunctionHandle):
+    """extract callers to a function"""
+    for ea in idautils.CodeRefsTo(fh.inner.start_ea, True):
+        yield Characteristic("calls to"), AbsoluteVirtualAddress(ea)
 
 
-def extract_function_loop(f):
-    """extract loop indicators from a function
-
-    args:
-        f (IDA func_t)
-    """
+def extract_function_loop(fh: FunctionHandle):
+    """extract loop indicators from a function"""
+    f: idaapi.func_t = fh.inner
     edges = []
 
     # construct control flow graph
@@ -41,25 +37,16 @@ def extract_function_loop(f):
         yield Characteristic("loop"), f.start_ea
 
 
-def extract_recursive_call(f):
-    """extract recursive function call
-
-    args:
-        f (IDA func_t)
-    """
-    if capa.features.extractors.ida.helpers.is_function_recursive(f):
-        yield Characteristic("recursive call"), f.start_ea
+def extract_recursive_call(fh: FunctionHandle):
+    """extract recursive function call"""
+    if capa.features.extractors.ida.helpers.is_function_recursive(fh.inner):
+        yield Characteristic("recursive call"), fh.address
 
 
-def extract_features(f):
-    """extract function features
-
-    arg:
-        f (IDA func_t)
-    """
+def extract_features(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
     for func_handler in FUNCTION_HANDLERS:
-        for (feature, ea) in func_handler(f):
-            yield feature, ea
+        for (feature, addr) in func_handler(fh):
+            yield feature, addr
 
 
 FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop, extract_recursive_call)
@@ -68,8 +55,8 @@ FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop, extract_r
 def main():
     """ """
     features = []
-    for f in capa.features.extractors.ida.get_functions(skip_thunks=True, skip_libs=True):
-        features.extend(list(extract_features(f)))
+    for fhandle in capa.features.extractors.ida.helpers.get_functions(skip_thunks=True, skip_libs=True):
+        features.extend(list(extract_features(fhandle)))
 
     import pprint
 

capa/features/extractors/ida/global_.py

@@ -1,27 +1,29 @@
 import logging
 import contextlib
+from typing import Tuple, Iterator
 
 import idaapi
 import ida_loader
 
 import capa.ida.helpers
 import capa.features.extractors.elf
-from capa.features.common import OS, ARCH_I386, ARCH_AMD64, OS_WINDOWS, Arch
+from capa.features.common import OS, ARCH_I386, ARCH_AMD64, OS_WINDOWS, Arch, Feature
+from capa.features.address import NO_ADDRESS, Address
 
 logger = logging.getLogger(__name__)
 
 
-def extract_os():
-    format_name = ida_loader.get_file_type_name()
+def extract_os() -> Iterator[Tuple[Feature, Address]]:
+    format_name: str = ida_loader.get_file_type_name()
 
     if "PE" in format_name:
-        yield OS(OS_WINDOWS), 0x0
+        yield OS(OS_WINDOWS), NO_ADDRESS
 
     elif "ELF" in format_name:
         with contextlib.closing(capa.ida.helpers.IDAIO()) as f:
             os = capa.features.extractors.elf.detect_elf_os(f)
 
-        yield OS(os), 0x0
+        yield OS(os), NO_ADDRESS
 
     else:
         # we likely end up here:
@@ -38,13 +40,13 @@ def extract_os():
         return
 
 
-def extract_arch():
-    info = idaapi.get_inf_structure()
-    if info.procName == "metapc" and info.is_64bit():
-        yield Arch(ARCH_AMD64), 0x0
-    elif info.procName == "metapc" and info.is_32bit():
-        yield Arch(ARCH_I386), 0x0
-    elif info.procName == "metapc":
+def extract_arch() -> Iterator[Tuple[Feature, Address]]:
+    info: idaapi.idainfo = idaapi.get_inf_structure()
+    if info.procname == "metapc" and info.is_64bit():
+        yield Arch(ARCH_AMD64), NO_ADDRESS
+    elif info.procname == "metapc" and info.is_32bit():
+        yield Arch(ARCH_I386), NO_ADDRESS
+    elif info.procname == "metapc":
         logger.debug("unsupported architecture: non-32-bit nor non-64-bit intel")
         return
     else:
@@ -52,5 +54,5 @@ def extract_arch():
         #  1. handling a new architecture (e.g. aarch64)
         #
         # for (1), this logic will need to be updated as the format is implemented.
-        logger.debug("unsupported architecture: %s", info.procName)
+        logger.debug("unsupported architecture: %s", info.procname)
         return

capa/features/extractors/ida/helpers.py

@@ -1,18 +1,22 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import Any, Dict, Tuple, Iterator
 
 import idc
 import idaapi
 import idautils
 import ida_bytes
 
+from capa.features.address import AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import FunctionHandle
 
-def find_byte_sequence(start, end, seq):
+
+def find_byte_sequence(start: int, end: int, seq: bytes) -> Iterator[int]:
     """yield all ea of a given byte sequence
 
     args:
@@ -20,32 +24,32 @@ def find_byte_sequence(start, end, seq):
         end: max virtual address
         seq: bytes to search e.g. b"\x01\x03"
     """
-    seq = " ".join(["%02x" % b for b in seq])
+    seqstr = " ".join(["%02x" % b for b in seq])
     while True:
-        ea = idaapi.find_binary(start, end, seq, 0, idaapi.SEARCH_DOWN)
+        # TODO find_binary: Deprecated. Please use ida_bytes.bin_search() instead.
+        ea = idaapi.find_binary(start, end, seqstr, 0, idaapi.SEARCH_DOWN)
         if ea == idaapi.BADADDR:
             break
         start = ea + 1
         yield ea
 
 
-def get_functions(start=None, end=None, skip_thunks=False, skip_libs=False):
+def get_functions(
+    start: int = None, end: int = None, skip_thunks: bool = False, skip_libs: bool = False
+) -> Iterator[FunctionHandle]:
     """get functions, range optional
 
     args:
         start: min virtual address
         end: max virtual address
-
-    ret:
-        yield func_t*
     """
     for ea in idautils.Functions(start=start, end=end):
         f = idaapi.get_func(ea)
         if not (skip_thunks and (f.flags & idaapi.FUNC_THUNK) or skip_libs and (f.flags & idaapi.FUNC_LIB)):
-            yield f
+            yield FunctionHandle(address=AbsoluteVirtualAddress(ea), inner=f)
 
 
-def get_segments(skip_header_segments=False):
+def get_segments(skip_header_segments=False) -> Iterator[idaapi.segment_t]:
     """get list of segments (sections) in the binary image
 
     args:
@@ -57,7 +61,7 @@ def get_segments(skip_header_segments=False):
             yield seg
 
 
-def get_segment_buffer(seg):
+def get_segment_buffer(seg: idaapi.segment_t) -> bytes:
     """return bytes stored in a given segment
 
     decrease buffer size until IDA is able to read bytes from the segment
@@ -75,7 +79,7 @@ def get_segment_buffer(seg):
     return buff if buff else b""
 
 
-def get_file_imports():
+def get_file_imports() -> Dict[int, Tuple[str, str, int]]:
     """get file imports"""
     imports = {}
 
@@ -85,10 +89,18 @@ def get_file_imports():
         if not library:
             continue
 
+        # IDA uses section names for the library of ELF imports, like ".dynsym"
+        library = library.lstrip(".")
+
         def inspect_import(ea, function, ordinal):
             if function and function.startswith("__imp_"):
-                # handle mangled names starting
+                # handle mangled PE imports
                 function = function[len("__imp_") :]
+
+            if function and "@@" in function:
+                # handle mangled ELF imports, like "fopen@@glibc_2.2.5"
+                function, _, _ = function.partition("@@")
+
             imports[ea] = (library.lower(), function, ordinal)
             return True
 
@@ -97,14 +109,12 @@ def get_file_imports():
     return imports
 
 
-def get_instructions_in_range(start, end):
+def get_instructions_in_range(start: int, end: int) -> Iterator[idaapi.insn_t]:
     """yield instructions in range
 
     args:
         start: virtual address (inclusive)
         end: virtual address (exclusive)
-    yield:
-        (insn_t*)
     """
     for head in idautils.Heads(start, end):
         insn = idautils.DecodeInstruction(head)
@@ -112,7 +122,7 @@ def get_instructions_in_range(start, end):
             yield insn
 
 
-def is_operand_equal(op1, op2):
+def is_operand_equal(op1: idaapi.op_t, op2: idaapi.op_t) -> bool:
     """compare two IDA op_t"""
     if op1.flags != op2.flags:
         return False
@@ -138,7 +148,7 @@ def is_operand_equal(op1, op2):
     return True
 
 
-def is_basic_block_equal(bb1, bb2):
+def is_basic_block_equal(bb1: idaapi.BasicBlock, bb2: idaapi.BasicBlock) -> bool:
     """compare two IDA BasicBlock"""
     if bb1.start_ea != bb2.start_ea:
         return False
@@ -152,12 +162,12 @@ def is_basic_block_equal(bb1, bb2):
     return True
 
 
-def basic_block_size(bb):
+def basic_block_size(bb: idaapi.BasicBlock) -> int:
     """calculate size of basic block"""
     return bb.end_ea - bb.start_ea
 
 
-def read_bytes_at(ea, count):
+def read_bytes_at(ea: int, count: int) -> bytes:
     """ """
     # check if byte has a value, see get_wide_byte doc
     if not idc.is_loaded(ea):
@@ -170,10 +180,10 @@ def read_bytes_at(ea, count):
         return idc.get_bytes(ea, count)
 
 
-def find_string_at(ea, min=4):
+def find_string_at(ea: int, min_: int = 4) -> str:
     """check if ASCII string exists at a given virtual address"""
     found = idaapi.get_strlit_contents(ea, -1, idaapi.STRTYPE_C)
-    if found and len(found) > min:
+    if found and len(found) > min_:
         try:
             found = found.decode("ascii")
             # hacky check for IDA bug; get_strlit_contents also reads Unicode as
@@ -187,7 +197,7 @@ def find_string_at(ea, min=4):
     return ""
 
 
-def get_op_phrase_info(op):
+def get_op_phrase_info(op: idaapi.op_t) -> Dict:
     """parse phrase features from operand
 
     Pretty much dup of sark's implementation:
@@ -224,23 +234,23 @@ def get_op_phrase_info(op):
     return {"base": base, "index": index, "scale": scale, "offset": offset}
 
 
-def is_op_write(insn, op):
+def is_op_write(insn: idaapi.insn_t, op: idaapi.op_t) -> bool:
     """Check if an operand is written to (destination operand)"""
     return idaapi.has_cf_chg(insn.get_canon_feature(), op.n)
 
 
-def is_op_read(insn, op):
+def is_op_read(insn: idaapi.insn_t, op: idaapi.op_t) -> bool:
     """Check if an operand is read from (source operand)"""
     return idaapi.has_cf_use(insn.get_canon_feature(), op.n)
 
 
-def is_op_offset(insn, op):
+def is_op_offset(insn: idaapi.insn_t, op: idaapi.op_t) -> bool:
     """Check is an operand has been marked as an offset (by auto-analysis or manually)"""
     flags = idaapi.get_flags(insn.ea)
     return ida_bytes.is_off(flags, op.n)
 
 
-def is_sp_modified(insn):
+def is_sp_modified(insn: idaapi.insn_t) -> bool:
     """determine if instruction modifies SP, ESP, RSP"""
     for op in get_insn_ops(insn, target_ops=(idaapi.o_reg,)):
         if op.reg == idautils.procregs.sp.reg and is_op_write(insn, op):
@@ -249,7 +259,7 @@ def is_sp_modified(insn):
     return False
 
 
-def is_bp_modified(insn):
+def is_bp_modified(insn: idaapi.insn_t) -> bool:
     """check if instruction modifies BP, EBP, RBP"""
     for op in get_insn_ops(insn, target_ops=(idaapi.o_reg,)):
         if op.reg == idautils.procregs.bp.reg and is_op_write(insn, op):
@@ -258,12 +268,12 @@ def is_bp_modified(insn):
     return False
 
 
-def is_frame_register(reg):
+def is_frame_register(reg: int) -> bool:
     """check if register is sp or bp"""
     return reg in (idautils.procregs.sp.reg, idautils.procregs.bp.reg)
 
 
-def get_insn_ops(insn, target_ops=()):
+def get_insn_ops(insn: idaapi.insn_t, target_ops: Tuple[Any] = None) -> idaapi.op_t:
     """yield op_t for instruction, filter on type if specified"""
     for op in insn.ops:
         if op.type == idaapi.o_void:
@@ -274,12 +284,12 @@ def get_insn_ops(insn, target_ops=()):
         yield op
 
 
-def is_op_stack_var(ea, index):
+def is_op_stack_var(ea: int, index: int) -> bool:
     """check if operand is a stack variable"""
     return idaapi.is_stkvar(idaapi.get_flags(ea), index)
 
 
-def mask_op_val(op):
+def mask_op_val(op: idaapi.op_t) -> int:
     """mask value by data type
 
     necessary due to a bug in AMD64
@@ -299,26 +309,18 @@ def mask_op_val(op):
     return masks.get(op.dtype, op.value) & op.value
 
 
-def is_function_recursive(f):
-    """check if function is recursive
-
-    args:
-        f (IDA func_t)
-    """
+def is_function_recursive(f: idaapi.func_t) -> bool:
+    """check if function is recursive"""
     for ref in idautils.CodeRefsTo(f.start_ea, True):
         if f.contains(ref):
             return True
     return False
 
 
-def is_basic_block_tight_loop(bb):
+def is_basic_block_tight_loop(bb: idaapi.BasicBlock) -> bool:
     """check basic block loops to self
 
     true if last instruction in basic block branches to basic block start
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
     """
     bb_end = idc.prev_head(bb.end_ea)
     if bb.start_ea < bb_end:
@@ -328,7 +330,7 @@ def is_basic_block_tight_loop(bb):
     return False
 
 
-def find_data_reference_from_insn(insn, max_depth=10):
+def find_data_reference_from_insn(insn: idaapi.insn_t, max_depth: int = 10) -> int:
     """search for data reference from instruction, return address of instruction if no reference exists"""
     depth = 0
     ea = insn.ea
@@ -358,19 +360,18 @@ def find_data_reference_from_insn(insn, max_depth=10):
     return ea
 
 
-def get_function_blocks(f):
-    """yield basic blocks contained in specified function
-
-    args:
-        f (IDA func_t)
-    yield:
-        block (IDA BasicBlock)
-    """
+def get_function_blocks(f: idaapi.func_t) -> Iterator[idaapi.BasicBlock]:
+    """yield basic blocks contained in specified function"""
     # leverage idaapi.FC_NOEXT flag to ignore useless external blocks referenced by the function
     for block in idaapi.FlowChart(f, flags=(idaapi.FC_PREDS | idaapi.FC_NOEXT)):
         yield block
 
 
-def is_basic_block_return(bb):
+def is_basic_block_return(bb: idaapi.BasicBlock) -> bool:
     """check if basic block is return block"""
     return bb.type == idaapi.fcb_ret
+
+
+def has_sib(oper: idaapi.op_t) -> bool:
+    # via: https://reverseengineering.stackexchange.com/a/14300
+    return oper.specflag1 == 1

capa/features/extractors/ida/insn.py

@@ -1,10 +1,11 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import Any, Dict, Tuple, Iterator
 
 import idc
 import idaapi
@@ -12,47 +13,23 @@ import idautils
 
 import capa.features.extractors.helpers
 import capa.features.extractors.ida.helpers
-from capa.features.insn import API, Number, Offset, Mnemonic
-from capa.features.common import (
-    BITNESS_X32,
-    BITNESS_X64,
-    MAX_BYTES_FEATURE_SIZE,
-    THUNK_CHAIN_DEPTH_DELTA,
-    Bytes,
-    String,
-    Characteristic,
-)
+from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
+from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
 # byte range within the first and returning basic blocks, this helps to reduce FP features
 SECURITY_COOKIE_BYTES_DELTA = 0x40
 
 
-def get_bitness(ctx):
-    """
-    fetch the BITNESS_* constant for the currently open workspace.
-
-    via Tamir Bahar/@tmr232
-    https://reverseengineering.stackexchange.com/a/11398/17194
-    """
-    if "bitness" not in ctx:
-        info = idaapi.get_inf_structure()
-        if info.is_64bit():
-            ctx["bitness"] = BITNESS_X64
-        elif info.is_32bit():
-            ctx["bitness"] = BITNESS_X32
-        else:
-            raise ValueError("unexpected bitness")
-    return ctx["bitness"]
-
-
-def get_imports(ctx):
+def get_imports(ctx: Dict[str, Any]) -> Dict[str, Any]:
     if "imports_cache" not in ctx:
         ctx["imports_cache"] = capa.features.extractors.ida.helpers.get_file_imports()
     return ctx["imports_cache"]
 
 
-def check_for_api_call(ctx, insn):
+def check_for_api_call(ctx: Dict[str, Any], insn: idaapi.insn_t) -> Iterator[str]:
     """check instruction for API call"""
     info = ()
     ref = insn.ea
@@ -81,24 +58,22 @@ def check_for_api_call(ctx, insn):
         yield "%s.%s" % (info[0], info[1])
 
 
-def extract_insn_api_features(f, bb, insn):
-    """parse instruction API features
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
+def extract_insn_api_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse instruction API features
 
     example:
-        call dword [0x00473038]
+       call dword [0x00473038]
     """
+    insn: idaapi.insn_t = ih.inner
+
     if not insn.get_canon_mnem() in ("call", "jmp"):
         return
 
-    for api in check_for_api_call(f.ctx, insn):
+    for api in check_for_api_call(fh.ctx, insn):
         dll, _, symbol = api.rpartition(".")
         for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
-            yield API(name), insn.ea
+            yield API(name), ih.address
 
     # extract IDA/FLIRT recognized API functions
     targets = tuple(idautils.CodeRefsFrom(insn.ea, False))
@@ -113,20 +88,25 @@ def extract_insn_api_features(f, bb, insn):
 
     if target_func.flags & idaapi.FUNC_LIB:
         name = idaapi.get_name(target_func.start_ea)
-        yield API(name), insn.ea
+        yield API(name), ih.address
+        if name.startswith("_"):
+            # some linkers may prefix linked routines with a `_` to avoid name collisions.
+            # extract features for both the mangled and un-mangled representations.
+            # e.g. `_fwrite` -> `fwrite`
+            # see: https://stackoverflow.com/a/2628384/87207
+            yield API(name[1:]), ih.address
 
 
-def extract_insn_number_features(f, bb, insn):
-    """parse instruction number features
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
-
+def extract_insn_number_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse instruction number features
     example:
         push    3136B0h         ; dwControlCode
     """
+    insn: idaapi.insn_t = ih.inner
+
     if idaapi.is_ret_insn(insn):
         # skip things like:
         #   .text:0042250E retn 8
@@ -137,7 +117,11 @@ def extract_insn_number_features(f, bb, insn):
         #   .text:00401145 add esp, 0Ch
         return
 
-    for op in capa.features.extractors.ida.helpers.get_insn_ops(insn, target_ops=(idaapi.o_imm, idaapi.o_mem)):
+    for i, op in enumerate(insn.ops):
+        if op.type == idaapi.o_void:
+            break
+        if op.type not in (idaapi.o_imm, idaapi.o_mem):
+            continue
         # skip things like:
         #   .text:00401100 shr eax, offset loc_C
         if capa.features.extractors.ida.helpers.is_op_offset(insn, op):
@@ -148,21 +132,27 @@ def extract_insn_number_features(f, bb, insn):
         else:
             const = op.addr
 
-        yield Number(const), insn.ea
-        yield Number(const, bitness=get_bitness(f.ctx)), insn.ea
+        yield Number(const), ih.address
+        yield OperandNumber(i, const), ih.address
+
+        if insn.itype == idaapi.NN_add and 0 < const < MAX_STRUCTURE_SIZE and op.type == idaapi.o_imm:
+            # for pattern like:
+            #
+            #     add eax, 0x10
+            #
+            # assume 0x10 is also an offset (imagine eax is a pointer).
+            yield Offset(const), ih.address
+            yield OperandOffset(i, const), ih.address
 
 
-def extract_insn_bytes_features(f, bb, insn):
-    """parse referenced byte sequences
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
-
+def extract_insn_bytes_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse referenced byte sequences
     example:
         push    offset iid_004118d4_IShellLinkA ; riid
     """
+    insn: idaapi.insn_t = ih.inner
+
     if idaapi.is_call_insn(insn):
         return
 
@@ -170,41 +160,46 @@ def extract_insn_bytes_features(f, bb, insn):
     if ref != insn.ea:
         extracted_bytes = capa.features.extractors.ida.helpers.read_bytes_at(ref, MAX_BYTES_FEATURE_SIZE)
         if extracted_bytes and not capa.features.extractors.helpers.all_zeros(extracted_bytes):
-            yield Bytes(extracted_bytes), insn.ea
+            yield Bytes(extracted_bytes), ih.address
 
 
-def extract_insn_string_features(f, bb, insn):
-    """parse instruction string features
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
+def extract_insn_string_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse instruction string features
 
     example:
         push offset aAcr     ; "ACR  > "
     """
+    insn: idaapi.insn_t = ih.inner
+
     ref = capa.features.extractors.ida.helpers.find_data_reference_from_insn(insn)
     if ref != insn.ea:
         found = capa.features.extractors.ida.helpers.find_string_at(ref)
         if found:
-            yield String(found), insn.ea
+            yield String(found), ih.address
 
 
-def extract_insn_offset_features(f, bb, insn):
-    """parse instruction structure offset features
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
+def extract_insn_offset_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse instruction structure offset features
 
     example:
         .text:0040112F cmp [esi+4], ebx
     """
-    for op in capa.features.extractors.ida.helpers.get_insn_ops(insn, target_ops=(idaapi.o_phrase, idaapi.o_displ)):
+    insn: idaapi.insn_t = ih.inner
+
+    for i, op in enumerate(insn.ops):
+        if op.type == idaapi.o_void:
+            break
+        if op.type not in (idaapi.o_phrase, idaapi.o_displ):
+            continue
         if capa.features.extractors.ida.helpers.is_op_stack_var(insn.ea, op.n):
             continue
+
         p_info = capa.features.extractors.ida.helpers.get_op_phrase_info(op)
         op_off = p_info.get("offset", 0)
         if idaapi.is_mapped(op_off):
@@ -217,12 +212,32 @@ def extract_insn_offset_features(f, bb, insn):
         # https://stackoverflow.com/questions/31853189/x86-64-assembly-why-displacement-not-64-bits
         op_off = capa.features.extractors.helpers.twos_complement(op_off, 32)
 
-        yield Offset(op_off), insn.ea
-        yield Offset(op_off, bitness=get_bitness(f.ctx)), insn.ea
+        yield Offset(op_off), ih.address
+        yield OperandOffset(i, op_off), ih.address
+
+        if (
+            insn.itype == idaapi.NN_lea
+            and i == 1
+            # o_displ is used for both:
+            #   [eax+1]
+            #   [eax+ebx+2]
+            and op.type == idaapi.o_displ
+            # but the SIB is only present for [eax+ebx+2]
+            # which we don't want
+            and not capa.features.extractors.ida.helpers.has_sib(op)
+        ):
+            # for pattern like:
+            #
+            #     lea eax, [ebx + 1]
+            #
+            # assume 1 is also an offset (imagine ebx is a zero register).
+            yield Number(op_off), ih.address
+            yield OperandNumber(i, op_off), ih.address
 
 
-def contains_stack_cookie_keywords(s):
-    """check if string contains stack cookie keywords
+def contains_stack_cookie_keywords(s: str) -> bool:
+    """
+    check if string contains stack cookie keywords
 
     Examples:
         xor     ecx, ebp ; StackCookie
@@ -236,7 +251,7 @@ def contains_stack_cookie_keywords(s):
     return any(keyword in s for keyword in ("stack", "security"))
 
 
-def bb_stack_cookie_registers(bb):
+def bb_stack_cookie_registers(bb: idaapi.BasicBlock) -> Iterator[int]:
     """scan basic block for stack cookie operations
 
     yield registers ids that may have been used for stack cookie operations
@@ -270,7 +285,7 @@ def bb_stack_cookie_registers(bb):
                     yield op.reg
 
 
-def is_nzxor_stack_cookie_delta(f, bb, insn):
+def is_nzxor_stack_cookie_delta(f: idaapi.func_t, bb: idaapi.BasicBlock, insn: idaapi.insn_t) -> bool:
     """check if nzxor exists within stack cookie delta"""
     # security cookie check should use SP or BP
     if not capa.features.extractors.ida.helpers.is_frame_register(insn.Op2.reg):
@@ -293,7 +308,7 @@ def is_nzxor_stack_cookie_delta(f, bb, insn):
     return False
 
 
-def is_nzxor_stack_cookie(f, bb, insn):
+def is_nzxor_stack_cookie(f: idaapi.func_t, bb: idaapi.BasicBlock, insn: idaapi.insn_t) -> bool:
     """check if nzxor is related to stack cookie"""
     if contains_stack_cookie_keywords(idaapi.get_cmt(insn.ea, False)):
         # Example:
@@ -310,37 +325,49 @@ def is_nzxor_stack_cookie(f, bb, insn):
     return False
 
 
-def extract_insn_nzxor_characteristic_features(f, bb, insn):
-    """parse instruction non-zeroing XOR instruction
-
-    ignore expected non-zeroing XORs, e.g. security cookies
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
+def extract_insn_nzxor_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """
+    parse instruction non-zeroing XOR instruction
+    ignore expected non-zeroing XORs, e.g. security cookies
+    """
+    insn: idaapi.insn_t = ih.inner
+
     if insn.itype not in (idaapi.NN_xor, idaapi.NN_xorpd, idaapi.NN_xorps, idaapi.NN_pxor):
         return
     if capa.features.extractors.ida.helpers.is_operand_equal(insn.Op1, insn.Op2):
         return
-    if is_nzxor_stack_cookie(f, bb, insn):
+    if is_nzxor_stack_cookie(fh.inner, bbh.inner, insn):
         return
-    yield Characteristic("nzxor"), insn.ea
+    yield Characteristic("nzxor"), ih.address
 
 
-def extract_insn_mnemonic_features(f, bb, insn):
-    """parse instruction mnemonic features
+def extract_insn_mnemonic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
+    """parse instruction mnemonic features"""
+    yield Mnemonic(idc.print_insn_mnem(ih.inner.ea)), ih.address
 
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
+
+def extract_insn_obfs_call_plus_5_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """
-    yield Mnemonic(idc.print_insn_mnem(insn.ea)), insn.ea
+    parse call $+5 instruction from the given instruction.
+    """
+    insn: idaapi.insn_t = ih.inner
+
+    if not idaapi.is_call_insn(insn):
+        return
+
+    if insn.ea + 5 == idc.get_operand_value(insn.ea, 0):
+        yield Characteristic("call $+5"), ih.address
 
 
-def extract_insn_peb_access_characteristic_features(f, bb, insn):
+def extract_insn_peb_access_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """parse instruction peb access
 
     fs:[0x30] on x86, gs:[0x60] on x64
@@ -348,6 +375,8 @@ def extract_insn_peb_access_characteristic_features(f, bb, insn):
     TODO:
         IDA should be able to do this..
     """
+    insn: idaapi.insn_t = ih.inner
+
     if insn.itype not in (idaapi.NN_push, idaapi.NN_mov):
         return
 
@@ -359,15 +388,19 @@ def extract_insn_peb_access_characteristic_features(f, bb, insn):
 
     if " fs:30h" in disasm or " gs:60h" in disasm:
         # TODO: replace above with proper IDA
-        yield Characteristic("peb access"), insn.ea
+        yield Characteristic("peb access"), ih.address
 
 
-def extract_insn_segment_access_features(f, bb, insn):
+def extract_insn_segment_access_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """parse instruction fs or gs access
 
     TODO:
         IDA should be able to do this...
     """
+    insn: idaapi.insn_t = ih.inner
+
     if all(map(lambda op: op.type != idaapi.o_mem, insn.ops)):
         # try to optimize for only memory references
         return
@@ -376,23 +409,21 @@ def extract_insn_segment_access_features(f, bb, insn):
 
     if " fs:" in disasm:
         # TODO: replace above with proper IDA
-        yield Characteristic("fs access"), insn.ea
+        yield Characteristic("fs access"), ih.address
 
     if " gs:" in disasm:
         # TODO: replace above with proper IDA
-        yield Characteristic("gs access"), insn.ea
+        yield Characteristic("gs access"), ih.address
 
 
-def extract_insn_cross_section_cflow(f, bb, insn):
-    """inspect the instruction for a CALL or JMP that crosses section boundaries
+def extract_insn_cross_section_cflow(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
+    """inspect the instruction for a CALL or JMP that crosses section boundaries"""
+    insn: idaapi.insn_t = ih.inner
 
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
-    """
     for ref in idautils.CodeRefsFrom(insn.ea, False):
-        if ref in get_imports(f.ctx).keys():
+        if ref in get_imports(fh.ctx).keys():
             # ignore API calls
             continue
         if not idaapi.getseg(ref):
@@ -400,50 +431,40 @@ def extract_insn_cross_section_cflow(f, bb, insn):
             continue
         if idaapi.getseg(ref) == idaapi.getseg(insn.ea):
             continue
-        yield Characteristic("cross section flow"), insn.ea
+        yield Characteristic("cross section flow"), ih.address
 
 
-def extract_function_calls_from(f, bb, insn):
+def extract_function_calls_from(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """extract functions calls from features
 
     most relevant at the function scope, however, its most efficient to extract at the instruction scope
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
     """
+    insn: idaapi.insn_t = ih.inner
+
     if idaapi.is_call_insn(insn):
         for ref in idautils.CodeRefsFrom(insn.ea, False):
-            yield Characteristic("calls from"), ref
+            yield Characteristic("calls from"), AbsoluteVirtualAddress(ref)
 
 
-def extract_function_indirect_call_characteristic_features(f, bb, insn):
+def extract_function_indirect_call_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """extract indirect function calls (e.g., call eax or call dword ptr [edx+4])
     does not include calls like => call ds:dword_ABD4974
 
     most relevant at the function or basic block scope;
     however, its most efficient to extract at the instruction scope
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
     """
+    insn: idaapi.insn_t = ih.inner
+
     if idaapi.is_call_insn(insn) and idc.get_operand_type(insn.ea, 0) in (idc.o_reg, idc.o_phrase, idc.o_displ):
-        yield Characteristic("indirect call"), insn.ea
+        yield Characteristic("indirect call"), ih.address
 
 
-def extract_features(f, bb, insn):
-    """extract instruction features
-
-    args:
-        f (IDA func_t)
-        bb (IDA BasicBlock)
-        insn (IDA insn_t)
-    """
+def extract_features(f: FunctionHandle, bbh: BBHandle, insn: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """extract instruction features"""
     for inst_handler in INSTRUCTION_HANDLERS:
-        for (feature, ea) in inst_handler(f, bb, insn):
+        for (feature, ea) in inst_handler(f, bbh, insn):
             yield feature, ea
 
 
@@ -455,6 +476,7 @@ INSTRUCTION_HANDLERS = (
     extract_insn_offset_features,
     extract_insn_nzxor_characteristic_features,
     extract_insn_mnemonic_features,
+    extract_insn_obfs_call_plus_5_characteristic_features,
     extract_insn_peb_access_characteristic_features,
     extract_insn_cross_section_cflow,
     extract_insn_segment_access_features,

capa/features/extractors/loops.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/features/extractors/null.py

@@ -0,0 +1,77 @@
+from typing import Dict, List, Tuple
+from dataclasses import dataclass
+
+from capa.features.common import Feature
+from capa.features.address import Address
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
+
+
+@dataclass
+class InstructionFeatures:
+    features: List[Tuple[Address, Feature]]
+
+
+@dataclass
+class BasicBlockFeatures:
+    features: List[Tuple[Address, Feature]]
+    instructions: Dict[Address, InstructionFeatures]
+
+
+@dataclass
+class FunctionFeatures:
+    features: List[Tuple[Address, Feature]]
+    basic_blocks: Dict[Address, BasicBlockFeatures]
+
+
+@dataclass
+class NullFeatureExtractor(FeatureExtractor):
+    """
+    An extractor that extracts some user-provided features.
+
+    This is useful for testing, as we can provide expected values and see if matching works.
+    """
+
+    base_address: Address
+    global_features: List[Feature]
+    file_features: List[Tuple[Address, Feature]]
+    functions: Dict[Address, FunctionFeatures]
+
+    def get_base_address(self):
+        return self.base_address
+
+    def extract_global_features(self):
+        for address, feature in self.global_features:
+            yield feature, address
+
+    def extract_file_features(self):
+        for address, feature in self.file_features:
+            yield feature, address
+
+    def get_functions(self):
+        for address in sorted(self.functions.keys()):
+            yield FunctionHandle(address, None)
+
+    def extract_function_features(self, f):
+        for address, feature in self.functions.get(f.address, {}).features:
+            yield feature, address
+
+    def get_basic_blocks(self, f):
+        for address in sorted(self.functions.get(f.address, {}).basic_blocks.keys()):
+            yield BBHandle(address, None)
+
+    def extract_basic_block_features(self, f, bb):
+        for address, feature in self.functions.get(f.address, {}).basic_blocks.get(bb.address, {}).features:
+            yield feature, address
+
+    def get_instructions(self, f, bb):
+        for address in sorted(self.functions.get(f.address, {}).basic_blocks.get(bb.address, {}).instructions.keys()):
+            yield InsnHandle(address, None)
+
+    def extract_insn_features(self, f, bb, insn):
+        for address, feature in (
+            self.functions.get(f.address, {})
+            .basic_blocks.get(bb.address, {})
+            .instructions.get(insn.address, {})
+            .features
+        ):
+            yield feature, address

capa/features/extractors/pefile.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -17,6 +17,7 @@ import capa.features.extractors.helpers
 import capa.features.extractors.strings
 from capa.features.file import Export, Import, Section
 from capa.features.common import OS, ARCH_I386, FORMAT_PE, ARCH_AMD64, OS_WINDOWS, Arch, Format, Characteristic
+from capa.features.address import NO_ADDRESS, FileOffsetAddress, AbsoluteVirtualAddress
 from capa.features.extractors.base_extractor import FeatureExtractor
 
 logger = logging.getLogger(__name__)
@@ -24,7 +25,7 @@ logger = logging.getLogger(__name__)
 
 def extract_file_embedded_pe(buf, **kwargs):
     for offset, _ in capa.features.extractors.helpers.carve_pe(buf, 1):
-        yield Characteristic("embedded pe"), offset
+        yield Characteristic("embedded pe"), FileOffsetAddress(offset)
 
 
 def extract_file_export_names(pe, **kwargs):
@@ -39,7 +40,7 @@ def extract_file_export_names(pe, **kwargs):
             except UnicodeDecodeError:
                 continue
             va = base_address + export.address
-            yield Export(name), va
+            yield Export(name), AbsoluteVirtualAddress(va)
 
 
 def extract_file_import_names(pe, **kwargs):
@@ -71,7 +72,7 @@ def extract_file_import_names(pe, **kwargs):
                         continue
 
                 for name in capa.features.extractors.helpers.generate_symbols(modname, impname):
-                    yield Import(name), imp.address
+                    yield Import(name), AbsoluteVirtualAddress(imp.address)
 
 
 def extract_file_section_names(pe, **kwargs):
@@ -83,7 +84,7 @@ def extract_file_section_names(pe, **kwargs):
         except UnicodeDecodeError:
             continue
 
-        yield Section(name), base_address + section.VirtualAddress
+        yield Section(name), AbsoluteVirtualAddress(base_address + section.VirtualAddress)
 
 
 def extract_file_strings(buf, **kwargs):
@@ -103,18 +104,18 @@ def extract_file_function_names(**kwargs):
 def extract_file_os(**kwargs):
     # assuming PE -> Windows
     # though i suppose they're also used by UEFI
-    yield OS(OS_WINDOWS), 0x0
+    yield OS(OS_WINDOWS), NO_ADDRESS
 
 
 def extract_file_format(**kwargs):
-    yield Format(FORMAT_PE), 0x0
+    yield Format(FORMAT_PE), NO_ADDRESS
 
 
 def extract_file_arch(pe, **kwargs):
     if pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_I386"]:
-        yield Arch(ARCH_I386), 0x0
+        yield Arch(ARCH_I386), NO_ADDRESS
     elif pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_AMD64"]:
-        yield Arch(ARCH_AMD64), 0x0
+        yield Arch(ARCH_AMD64), NO_ADDRESS
     else:
         logger.warning("unsupported architecture: %s", pefile.MACHINE_TYPE[pe.FILE_HEADER.Machine])
 
@@ -176,7 +177,7 @@ class PefileFeatureExtractor(FeatureExtractor):
         self.pe = pefile.PE(path)
 
     def get_base_address(self):
-        return self.pe.OPTIONAL_HEADER.ImageBase
+        return AbsoluteVirtualAddress(self.pe.OPTIONAL_HEADER.ImageBase)
 
     def extract_global_features(self):
         with open(self.path, "rb") as f:

capa/features/extractors/smda/basicblock.py

@@ -1,9 +1,12 @@
 import string
 import struct
+from typing import Tuple, Iterator
 
-from capa.features.common import Characteristic
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address
 from capa.features.basicblock import BasicBlock
 from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
+from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
 
 
 def _bb_has_tight_loop(f, bb):
@@ -13,10 +16,10 @@ def _bb_has_tight_loop(f, bb):
     return bb.offset in f.blockrefs[bb.offset] if bb.offset in f.blockrefs else False
 
 
-def extract_bb_tight_loop(f, bb):
+def extract_bb_tight_loop(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """check basic block for tight loop indicators"""
-    if _bb_has_tight_loop(f, bb):
-        yield Characteristic("tight loop"), bb.offset
+    if _bb_has_tight_loop(f.inner, bb.inner):
+        yield Characteristic("tight loop"), bb.address
 
 
 def _bb_has_stackstring(f, bb):
@@ -37,10 +40,10 @@ def get_operands(smda_ins):
     return [o.strip() for o in smda_ins.operands.split(",")]
 
 
-def extract_stackstring(f, bb):
+def extract_stackstring(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """check basic block for stackstring indicators"""
-    if _bb_has_stackstring(f, bb):
-        yield Characteristic("stack string"), bb.offset
+    if _bb_has_stackstring(f.inner, bb.inner):
+        yield Characteristic("stack string"), bb.address
 
 
 def is_mov_imm_to_stack(smda_ins):
@@ -107,21 +110,21 @@ def get_printable_len(instr):
     return 0
 
 
-def extract_features(f, bb):
+def extract_features(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     extract features from the given basic block.
 
     args:
-      f (smda.common.SmdaFunction): the function from which to extract features
-      bb (smda.common.SmdaBasicBlock): the basic block to process.
+      f: the function from which to extract features
+      bb: the basic block to process.
 
     yields:
-      Tuple[Feature, int]: the features and their location found in this basic block.
+      Tuple[Feature, Address]: the features and their location found in this basic block.
     """
-    yield BasicBlock(), bb.offset
+    yield BasicBlock(), bb.address
     for bb_handler in BASIC_BLOCK_HANDLERS:
-        for feature, va in bb_handler(f, bb):
-            yield feature, va
+        for feature, addr in bb_handler(f, bb):
+            yield feature, addr
 
 
 BASIC_BLOCK_HANDLERS = (

capa/features/extractors/smda/extractor.py

@@ -1,3 +1,5 @@
+from typing import List, Tuple
+
 from smda.common.SmdaReport import SmdaReport
 
 import capa.features.extractors.common
@@ -6,7 +8,9 @@ import capa.features.extractors.smda.insn
 import capa.features.extractors.smda.global_
 import capa.features.extractors.smda.function
 import capa.features.extractors.smda.basicblock
-from capa.features.extractors.base_extractor import FeatureExtractor
+from capa.features.common import Feature
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
 
 
 class SmdaFeatureExtractor(FeatureExtractor):
@@ -18,12 +22,12 @@ class SmdaFeatureExtractor(FeatureExtractor):
             self.buf = f.read()
 
         # pre-compute these because we'll yield them at *every* scope.
-        self.global_features = []
+        self.global_features: List[Tuple[Feature, Address]] = []
         self.global_features.extend(capa.features.extractors.common.extract_os(self.buf))
         self.global_features.extend(capa.features.extractors.smda.global_.extract_arch(self.smda_report))
 
     def get_base_address(self):
-        return self.smda_report.base_addr
+        return AbsoluteVirtualAddress(self.smda_report.base_addr)
 
     def extract_global_features(self):
         yield from self.global_features
@@ -33,21 +37,21 @@ class SmdaFeatureExtractor(FeatureExtractor):
 
     def get_functions(self):
         for function in self.smda_report.getFunctions():
-            yield function
+            yield FunctionHandle(address=AbsoluteVirtualAddress(function.offset), inner=function)
 
-    def extract_function_features(self, f):
-        yield from capa.features.extractors.smda.function.extract_features(f)
+    def extract_function_features(self, fh):
+        yield from capa.features.extractors.smda.function.extract_features(fh)
 
-    def get_basic_blocks(self, f):
-        for bb in f.getBlocks():
-            yield bb
+    def get_basic_blocks(self, fh):
+        for bb in fh.inner.getBlocks():
+            yield BBHandle(address=AbsoluteVirtualAddress(bb.offset), inner=bb)
 
-    def extract_basic_block_features(self, f, bb):
-        yield from capa.features.extractors.smda.basicblock.extract_features(f, bb)
+    def extract_basic_block_features(self, fh, bbh):
+        yield from capa.features.extractors.smda.basicblock.extract_features(fh, bbh)
 
-    def get_instructions(self, f, bb):
-        for smda_ins in bb.getInstructions():
-            yield smda_ins
+    def get_instructions(self, fh, bbh):
+        for smda_ins in bbh.inner.getInstructions():
+            yield InsnHandle(address=AbsoluteVirtualAddress(smda_ins.offset), inner=smda_ins)
 
-    def extract_insn_features(self, f, bb, insn):
-        yield from capa.features.extractors.smda.insn.extract_features(f, bb, insn)
+    def extract_insn_features(self, fh, bbh, ih):
+        yield from capa.features.extractors.smda.insn.extract_features(fh, bbh, ih)

capa/features/extractors/smda/file.py

@@ -6,11 +6,12 @@ import capa.features.extractors.helpers
 import capa.features.extractors.strings
 from capa.features.file import Export, Import, Section
 from capa.features.common import String, Characteristic
+from capa.features.address import FileOffsetAddress, AbsoluteVirtualAddress
 
 
 def extract_file_embedded_pe(buf, **kwargs):
     for offset, _ in capa.features.extractors.helpers.carve_pe(buf, 1):
-        yield Characteristic("embedded pe"), offset
+        yield Characteristic("embedded pe"), FileOffsetAddress(offset)
 
 
 def extract_file_export_names(buf, **kwargs):
@@ -18,7 +19,7 @@ def extract_file_export_names(buf, **kwargs):
 
     if lief_binary is not None:
         for function in lief_binary.exported_functions:
-            yield Export(function.name), function.address
+            yield Export(function.name), AbsoluteVirtualAddress(function.address)
 
 
 def extract_file_import_names(smda_report, buf):
@@ -33,10 +34,10 @@ def extract_file_import_names(smda_report, buf):
             va = func.iat_address + smda_report.base_addr
             if func.name:
                 for name in capa.features.extractors.helpers.generate_symbols(library_name, func.name):
-                    yield Import(name), va
+                    yield Import(name), AbsoluteVirtualAddress(va)
             elif func.is_ordinal:
                 for name in capa.features.extractors.helpers.generate_symbols(library_name, "#%s" % func.ordinal):
-                    yield Import(name), va
+                    yield Import(name), AbsoluteVirtualAddress(va)
 
 
 def extract_file_section_names(buf, **kwargs):
@@ -46,7 +47,7 @@ def extract_file_section_names(buf, **kwargs):
     if lief_binary and lief_binary.sections:
         base_address = lief_binary.optional_header.imagebase
         for section in lief_binary.sections:
-            yield Section(section.name), base_address + section.virtual_address
+            yield Section(section.name), AbsoluteVirtualAddress(base_address + section.virtual_address)
 
 
 def extract_file_strings(buf, **kwargs):
@@ -54,10 +55,10 @@ def extract_file_strings(buf, **kwargs):
     extract ASCII and UTF-16 LE strings from file
     """
     for s in capa.features.extractors.strings.extract_ascii_strings(buf):
-        yield String(s.s), s.offset
+        yield String(s.s), FileOffsetAddress(s.offset)
 
     for s in capa.features.extractors.strings.extract_unicode_strings(buf):
-        yield String(s.s), s.offset
+        yield String(s.s), FileOffsetAddress(s.offset)
 
 
 def extract_file_function_names(smda_report, **kwargs):
@@ -87,8 +88,8 @@ def extract_features(smda_report, buf):
     """
 
     for file_handler in FILE_HANDLERS:
-        for feature, va in file_handler(smda_report=smda_report, buf=buf):
-            yield feature, va
+        for feature, addr in file_handler(smda_report=smda_report, buf=buf):
+            yield feature, addr
 
 
 FILE_HANDLERS = (

capa/features/extractors/smda/function.py

@@ -1,38 +1,42 @@
-from capa.features.common import Characteristic
+from typing import Tuple, Iterator
+
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
 from capa.features.extractors import loops
+from capa.features.extractors.base_extractor import FunctionHandle
 
 
-def extract_function_calls_to(f):
-    for inref in f.inrefs:
-        yield Characteristic("calls to"), inref
+def extract_function_calls_to(f: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+    for inref in f.inner.inrefs:
+        yield Characteristic("calls to"), AbsoluteVirtualAddress(inref)
 
 
-def extract_function_loop(f):
+def extract_function_loop(f: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse if a function has a loop
     """
     edges = []
-    for bb_from, bb_tos in f.blockrefs.items():
+    for bb_from, bb_tos in f.inner.blockrefs.items():
         for bb_to in bb_tos:
             edges.append((bb_from, bb_to))
 
     if edges and loops.has_loop(edges):
-        yield Characteristic("loop"), f.offset
+        yield Characteristic("loop"), f.address
 
 
-def extract_features(f):
+def extract_features(f: FunctionHandle):
     """
     extract features from the given function.
 
     args:
-      f (smda.common.SmdaFunction): the function from which to extract features
+      f: the function from which to extract features
 
     yields:
-      Tuple[Feature, int]: the features and their location found in this function.
+      Tuple[Feature, Address]: the features and their location found in this function.
     """
     for func_handler in FUNCTION_HANDLERS:
-        for feature, va in func_handler(f):
-            yield feature, va
+        for feature, addr in func_handler(f):
+            yield feature, addr
 
 
 FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop)

capa/features/extractors/smda/global_.py

@@ -1,6 +1,7 @@
 import logging
 
 from capa.features.common import ARCH_I386, ARCH_AMD64, Arch
+from capa.features.address import NO_ADDRESS
 
 logger = logging.getLogger(__name__)
 
@@ -8,9 +9,9 @@ logger = logging.getLogger(__name__)
 def extract_arch(smda_report):
     if smda_report.architecture == "intel":
         if smda_report.bitness == 32:
-            yield Arch(ARCH_I386), 0x0
+            yield Arch(ARCH_I386), NO_ADDRESS
         elif smda_report.bitness == 64:
-            yield Arch(ARCH_AMD64), 0x0
+            yield Arch(ARCH_AMD64), NO_ADDRESS
     else:
         # we likely end up here:
         #  1. handling a new architecture (e.g. aarch64)

capa/features/extractors/smda/insn.py

@@ -1,20 +1,15 @@
 import re
 import string
 import struct
+from typing import Tuple, Iterator
 
-from smda.common.SmdaReport import SmdaReport
+import smda
 
 import capa.features.extractors.helpers
-from capa.features.insn import API, Number, Offset, Mnemonic
-from capa.features.common import (
-    BITNESS_X32,
-    BITNESS_X64,
-    MAX_BYTES_FEATURE_SIZE,
-    THUNK_CHAIN_DEPTH_DELTA,
-    Bytes,
-    String,
-    Characteristic,
-)
+from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
+from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
 # byte range within the first and returning basic blocks, this helps to reduce FP features
@@ -23,27 +18,20 @@ PATTERN_HEXNUM = re.compile(r"[+\-] (?P<num>0x[a-fA-F0-9]+)")
 PATTERN_SINGLENUM = re.compile(r"[+\-] (?P<num>[0-9])")
 
 
-def get_bitness(smda_report):
-    if smda_report.architecture == "intel":
-        if smda_report.bitness == 32:
-            return BITNESS_X32
-        elif smda_report.bitness == 64:
-            return BITNESS_X64
-    else:
-        raise NotImplementedError
-
-
-def extract_insn_api_features(f, bb, insn):
+def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse API features from the given instruction."""
-    if insn.offset in f.apirefs:
-        api_entry = f.apirefs[insn.offset]
+    f: smda.Function = fh.inner
+    insn: smda.Insn = ih.inner
+
+    if ih.address in f.apirefs:
+        api_entry = f.apirefs[ih.address]
         # reformat
         dll_name, api_name = api_entry.split("!")
         dll_name = dll_name.split(".")[0]
         dll_name = dll_name.lower()
         for name in capa.features.extractors.helpers.generate_symbols(dll_name, api_name):
-            yield API(name), insn.offset
-    elif insn.offset in f.outrefs:
+            yield API(name), ih.address
+    elif ih.address in f.outrefs:
         current_function = f
         current_instruction = insn
         for index in range(THUNK_CHAIN_DEPTH_DELTA):
@@ -62,7 +50,7 @@ def extract_insn_api_features(f, bb, insn):
                             dll_name = dll_name.split(".")[0]
                             dll_name = dll_name.lower()
                             for name in capa.features.extractors.helpers.generate_symbols(dll_name, api_name):
-                                yield API(name), insn.offset
+                                yield API(name), ih.address
                     elif referenced_function.num_instructions == 1 and referenced_function.num_outrefs == 1:
                         current_function = referenced_function
                         current_instruction = [i for i in referenced_function.getInstructions()][0]
@@ -70,11 +58,14 @@ def extract_insn_api_features(f, bb, insn):
                     return
 
 
-def extract_insn_number_features(f, bb, insn):
+def extract_insn_number_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse number features from the given instruction."""
     # example:
     #
     #     push    3136B0h         ; dwControlCode
+    f: smda.Function = fh.inner
+    insn: smda.Insn = ih.inner
+
     operands = [o.strip() for o in insn.operands.split(",")]
     if insn.mnemonic == "add" and operands[0] in ["esp", "rsp"]:
         # skip things like:
@@ -82,12 +73,25 @@ def extract_insn_number_features(f, bb, insn):
         #    .text:00401140                 call    sub_407E2B
         #    .text:00401145                 add     esp, 0Ch
         return
-    for operand in operands:
+    for i, operand in enumerate(operands):
         try:
-            yield Number(int(operand, 16)), insn.offset
-            yield Number(int(operand, 16), bitness=get_bitness(f.smda_report)), insn.offset
-        except:
+            # The result of bitwise operations is calculated as though carried out
+            # in two’s complement with an infinite number of sign bits
+            value = int(operand, 16) & ((1 << f.smda_report.bitness) - 1)
+        except ValueError:
             continue
+        else:
+            yield Number(value), ih.address
+            yield OperandNumber(i, value), ih.address
+
+            if insn.mnemonic == "add" and 0 < value < MAX_STRUCTURE_SIZE:
+                # for pattern like:
+                #
+                #     add eax, 0x10
+                #
+                # assume 0x10 is also an offset (imagine eax is a pointer).
+                yield Offset(value), ih.address
+                yield OperandOffset(i, value), ih.address
 
 
 def read_bytes(smda_report, va, num_bytes=None):
@@ -136,12 +140,15 @@ def derefs(smda_report, p):
         p = val
 
 
-def extract_insn_bytes_features(f, bb, insn):
+def extract_insn_bytes_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse byte sequence features from the given instruction.
     example:
         #     push    offset iid_004118d4_IShellLinkA ; riid
     """
+    f: smda.Function = fh.inner
+    insn: smda.Insn = ih.inner
+
     for data_ref in insn.getDataRefs():
         for v in derefs(f.smda_report, data_ref):
             bytes_read = read_bytes(f.smda_report, v)
@@ -150,7 +157,7 @@ def extract_insn_bytes_features(f, bb, insn):
             if capa.features.extractors.helpers.all_zeros(bytes_read):
                 continue
 
-            yield Bytes(bytes_read), insn.offset
+            yield Bytes(bytes_read), ih.address
 
 
 def detect_ascii_len(smda_report, offset):
@@ -194,30 +201,34 @@ def read_string(smda_report, offset):
         return read_bytes(smda_report, offset, ulen).decode("utf-16")
 
 
-def extract_insn_string_features(f, bb, insn):
+def extract_insn_string_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse string features from the given instruction."""
     # example:
     #
     #     push    offset aAcr     ; "ACR  > "
+    f: smda.Function = fh.inner
+    insn: smda.Insn = ih.inner
+
     for data_ref in insn.getDataRefs():
         for v in derefs(f.smda_report, data_ref):
             string_read = read_string(f.smda_report, v)
             if string_read:
-                yield String(string_read.rstrip("\x00")), insn.offset
+                yield String(string_read.rstrip("\x00")), ih.address
 
 
-def extract_insn_offset_features(f, bb, insn):
+def extract_insn_offset_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse structure offset features from the given instruction."""
     # examples:
     #
     #     mov eax, [esi + 4]
     #     mov eax, [esi + ecx + 16384]
+    insn: smda.Insn = ih.inner
+
     operands = [o.strip() for o in insn.operands.split(",")]
-    for operand in operands:
-        if not "ptr" in operand:
-            continue
+    for i, operand in enumerate(operands):
         if "esp" in operand or "ebp" in operand or "rbp" in operand:
             continue
+
         number = 0
         number_hex = re.search(PATTERN_HEXNUM, operand)
         number_int = re.search(PATTERN_SINGLENUM, operand)
@@ -227,8 +238,26 @@ def extract_insn_offset_features(f, bb, insn):
         elif number_int:
             number = int(number_int.group("num"))
             number = -1 * number if number_int.group().startswith("-") else number
-        yield Offset(number), insn.offset
-        yield Offset(number, bitness=get_bitness(f.smda_report)), insn.offset
+
+        if "ptr" not in operand:
+            if (
+                insn.mnemonic == "lea"
+                and i == 1
+                and (operand.count("+") + operand.count("-")) == 1
+                and operand.count("*") == 0
+            ):
+                # for pattern like:
+                #
+                #     lea eax, [ebx + 1]
+                #
+                # assume 1 is also an offset (imagine ebx is a zero register).
+                yield Number(number), ih.address
+                yield OperandNumber(i, number), ih.address
+
+            continue
+
+        yield Offset(number), ih.address
+        yield OperandOffset(i, number), ih.address
 
 
 def is_security_cookie(f, bb, insn):
@@ -252,11 +281,16 @@ def is_security_cookie(f, bb, insn):
     return False
 
 
-def extract_insn_nzxor_characteristic_features(f, bb, insn):
+def extract_insn_nzxor_characteristic_features(
+    fh: FunctionHandle, bh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """
     parse non-zeroing XOR instruction from the given instruction.
     ignore expected non-zeroing XORs, e.g. security cookies.
     """
+    f: smda.Function = fh.inner
+    bb: smda.BasicBlock = bh.inner
+    insn: smda.Insn = ih.inner
 
     if insn.mnemonic not in ("xor", "xorpd", "xorps", "pxor"):
         return
@@ -268,18 +302,35 @@ def extract_insn_nzxor_characteristic_features(f, bb, insn):
     if is_security_cookie(f, bb, insn):
         return
 
-    yield Characteristic("nzxor"), insn.offset
+    yield Characteristic("nzxor"), ih.address
 
 
-def extract_insn_mnemonic_features(f, bb, insn):
+def extract_insn_mnemonic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse mnemonic features from the given instruction."""
-    yield Mnemonic(insn.mnemonic), insn.offset
+    yield Mnemonic(ih.inner.mnemonic), ih.address
 
 
-def extract_insn_peb_access_characteristic_features(f, bb, insn):
+def extract_insn_obfs_call_plus_5_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse call $+5 instruction from the given instruction.
+    """
+    insn: smda.Insn = ih.inner
+
+    if insn.mnemonic != "call":
+        return
+
+    if not insn.operands.startswith("0x"):
+        return
+
+    if int(insn.operands, 16) == insn.offset + 5:
+        yield Characteristic("call $+5"), ih.address
+
+
+def extract_insn_peb_access_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse peb access from the given function. fs:[0x30] on x86, gs:[0x60] on x64
     """
+    insn: smda.Insn = ih.inner
 
     if insn.mnemonic not in ["push", "mov"]:
         return
@@ -287,65 +338,75 @@ def extract_insn_peb_access_characteristic_features(f, bb, insn):
     operands = [o.strip() for o in insn.operands.split(",")]
     for operand in operands:
         if "fs:" in operand and "0x30" in operand:
-            yield Characteristic("peb access"), insn.offset
+            yield Characteristic("peb access"), ih.address
         elif "gs:" in operand and "0x60" in operand:
-            yield Characteristic("peb access"), insn.offset
+            yield Characteristic("peb access"), ih.address
 
 
-def extract_insn_segment_access_features(f, bb, insn):
+def extract_insn_segment_access_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse the instruction for access to fs or gs"""
+    insn: smda.Insn = ih.inner
+
     operands = [o.strip() for o in insn.operands.split(",")]
     for operand in operands:
         if "fs:" in operand:
-            yield Characteristic("fs access"), insn.offset
+            yield Characteristic("fs access"), ih.address
         elif "gs:" in operand:
-            yield Characteristic("gs access"), insn.offset
+            yield Characteristic("gs access"), ih.address
 
 
-def extract_insn_cross_section_cflow(f, bb, insn):
+def extract_insn_cross_section_cflow(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     inspect the instruction for a CALL or JMP that crosses section boundaries.
     """
+    f: smda.Function = fh.inner
+    insn: smda.Insn = ih.inner
+
     if insn.mnemonic in ["call", "jmp"]:
-        if insn.offset in f.apirefs:
+        if ih.address in f.apirefs:
             return
 
         smda_report = insn.smda_function.smda_report
-        if insn.offset in f.outrefs:
-            for target in f.outrefs[insn.offset]:
-                if smda_report.getSection(insn.offset) != smda_report.getSection(target):
-                    yield Characteristic("cross section flow"), insn.offset
+        if ih.address in f.outrefs:
+            for target in f.outrefs[ih.address]:
+                if smda_report.getSection(ih.address) != smda_report.getSection(target):
+                    yield Characteristic("cross section flow"), ih.address
         elif insn.operands.startswith("0x"):
             target = int(insn.operands, 16)
-            if smda_report.getSection(insn.offset) != smda_report.getSection(target):
-                yield Characteristic("cross section flow"), insn.offset
+            if smda_report.getSection(ih.address) != smda_report.getSection(target):
+                yield Characteristic("cross section flow"), ih.address
 
 
 # this is a feature that's most relevant at the function scope,
 # however, its most efficient to extract at the instruction scope.
-def extract_function_calls_from(f, bb, insn):
+def extract_function_calls_from(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    f: smda.Function = fh.inner
+    insn: smda.Insn = ih.inner
+
     if insn.mnemonic != "call":
         return
 
-    if insn.offset in f.outrefs:
-        for outref in f.outrefs[insn.offset]:
-            yield Characteristic("calls from"), outref
+    if ih.address in f.outrefs:
+        for outref in f.outrefs[ih.address]:
+            yield Characteristic("calls from"), AbsoluteVirtualAddress(outref)
 
             if outref == f.offset:
                 # if we found a jump target and it's the function address
                 # mark as recursive
-                yield Characteristic("recursive call"), outref
-    if insn.offset in f.apirefs:
-        yield Characteristic("calls from"), insn.offset
+                yield Characteristic("recursive call"), AbsoluteVirtualAddress(outref)
+    if ih.address in f.apirefs:
+        yield Characteristic("calls from"), ih.address
 
 
 # this is a feature that's most relevant at the function or basic block scope,
 # however, its most efficient to extract at the instruction scope.
-def extract_function_indirect_call_characteristic_features(f, bb, insn):
+def extract_function_indirect_call_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     extract indirect function call characteristic (e.g., call eax or call dword ptr [edx+4])
     does not include calls like => call ds:dword_ABD4974
     """
+    insn: smda.Insn = ih.inner
+
     if insn.mnemonic != "call":
         return
     if insn.operands.startswith("0x"):
@@ -357,7 +418,7 @@ def extract_function_indirect_call_characteristic_features(f, bb, insn):
     # call edx
     # call dword ptr [eax+50h]
     # call qword ptr [rsp+78h]
-    yield Characteristic("indirect call"), insn.offset
+    yield Characteristic("indirect call"), ih.address
 
 
 def extract_features(f, bb, insn):
@@ -365,16 +426,16 @@ def extract_features(f, bb, insn):
     extract features from the given insn.
 
     args:
-      f (smda.common.SmdaFunction): the function to process.
-      bb (smda.common.SmdaBasicBlock): the basic block to process.
-      insn (smda.common.SmdaInstruction): the instruction to process.
+      f: the function to process.
+      bb: the basic block to process.
+      insn: the instruction to process.
 
     yields:
-      Tuple[Feature, int]: the features and their location found in this insn.
+      Tuple[Feature, Address]: the features and their location found in this insn.
     """
     for insn_handler in INSTRUCTION_HANDLERS:
-        for feature, va in insn_handler(f, bb, insn):
-            yield feature, va
+        for feature, addr in insn_handler(f, bb, insn):
+            yield feature, addr
 
 
 INSTRUCTION_HANDLERS = (
@@ -385,6 +446,7 @@ INSTRUCTION_HANDLERS = (
     extract_insn_offset_features,
     extract_insn_nzxor_characteristic_features,
     extract_insn_mnemonic_features,
+    extract_insn_obfs_call_plus_5_characteristic_features,
     extract_insn_peb_access_characteristic_features,
     extract_insn_cross_section_cflow,
     extract_insn_segment_access_features,

capa/features/extractors/strings.py

@@ -1,6 +1,6 @@
-# strings code from FLOSS, https://github.com/fireeye/flare-floss
+# strings code from FLOSS, https://github.com/mandiant/flare-floss
 #
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/features/extractors/viv/basicblock.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -8,27 +8,30 @@
 
 import string
 import struct
+from typing import Tuple, Iterator
 
 import envi
 import envi.archs.i386.disasm
 
-from capa.features.common import Characteristic
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
 from capa.features.basicblock import BasicBlock
 from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
+from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
 
 
-def interface_extract_basic_block_XXX(f, bb):
+def interface_extract_basic_block_XXX(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse features from the given basic block.
 
     args:
-      f (viv_utils.Function): the function to process.
-      bb (viv_utils.BasicBlock): the basic block to process.
+      f: the function to process.
+      bb: the basic block to process.
 
     yields:
-      (Feature, int): the feature and the address at which its found.
+      (Feature, Address): the feature and the address at which its found.
     """
-    yield NotImplementedError("feature"), NotImplementedError("virtual address")
+    ...
 
 
 def _bb_has_tight_loop(f, bb):
@@ -44,10 +47,10 @@ def _bb_has_tight_loop(f, bb):
     return False
 
 
-def extract_bb_tight_loop(f, bb):
+def extract_bb_tight_loop(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """check basic block for tight loop indicators"""
-    if _bb_has_tight_loop(f, bb):
-        yield Characteristic("tight loop"), bb.va
+    if _bb_has_tight_loop(f, bb.inner):
+        yield Characteristic("tight loop"), bb.address
 
 
 def _bb_has_stackstring(f, bb):
@@ -67,10 +70,10 @@ def _bb_has_stackstring(f, bb):
     return False
 
 
-def extract_stackstring(f, bb):
+def extract_stackstring(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """check basic block for stackstring indicators"""
-    if _bb_has_stackstring(f, bb):
-        yield Characteristic("stack string"), bb.va
+    if _bb_has_stackstring(f, bb.inner):
+        yield Characteristic("stack string"), bb.address
 
 
 def is_mov_imm_to_stack(instr: envi.archs.i386.disasm.i386Opcode) -> bool:
@@ -143,7 +146,7 @@ def is_printable_utf16le(chars: bytes) -> bool:
     return False
 
 
-def extract_features(f, bb):
+def extract_features(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     extract features from the given basic block.
 
@@ -154,10 +157,10 @@ def extract_features(f, bb):
     yields:
       Tuple[Feature, int]: the features and their location found in this basic block.
     """
-    yield BasicBlock(), bb.va
+    yield BasicBlock(), AbsoluteVirtualAddress(bb.inner.va)
     for bb_handler in BASIC_BLOCK_HANDLERS:
-        for feature, va in bb_handler(f, bb):
-            yield feature, va
+        for feature, addr in bb_handler(f, bb):
+            yield feature, addr
 
 
 BASIC_BLOCK_HANDLERS = (

capa/features/extractors/viv/extractor.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,6 +6,7 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 import logging
+from typing import List, Tuple, Iterator
 
 import viv_utils
 import viv_utils.flirt
@@ -16,12 +17,14 @@ import capa.features.extractors.viv.insn
 import capa.features.extractors.viv.global_
 import capa.features.extractors.viv.function
 import capa.features.extractors.viv.basicblock
-from capa.features.extractors.base_extractor import FeatureExtractor
+from capa.features.common import Feature
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
-class InstructionHandle:
+class VivInstructionHandle:
     """this acts like a vivisect.Opcode but with an __int__() method"""
 
     def __init__(self, inner):
@@ -43,13 +46,13 @@ class VivisectFeatureExtractor(FeatureExtractor):
             self.buf = f.read()
 
         # pre-compute these because we'll yield them at *every* scope.
-        self.global_features = []
+        self.global_features: List[Tuple[Feature, Address]] = []
         self.global_features.extend(capa.features.extractors.common.extract_os(self.buf))
         self.global_features.extend(capa.features.extractors.viv.global_.extract_arch(self.vw))
 
     def get_base_address(self):
         # assume there is only one file loaded into the vw
-        return list(self.vw.filemeta.values())[0]["imagebase"]
+        return AbsoluteVirtualAddress(list(self.vw.filemeta.values())[0]["imagebase"])
 
     def extract_global_features(self):
         yield from self.global_features
@@ -57,28 +60,33 @@ class VivisectFeatureExtractor(FeatureExtractor):
     def extract_file_features(self):
         yield from capa.features.extractors.viv.file.extract_features(self.vw, self.buf)
 
-    def get_functions(self):
+    def get_functions(self) -> Iterator[FunctionHandle]:
         for va in sorted(self.vw.getFunctions()):
-            yield viv_utils.Function(self.vw, va)
+            yield FunctionHandle(address=AbsoluteVirtualAddress(va), inner=viv_utils.Function(self.vw, va))
 
-    def extract_function_features(self, f):
-        yield from capa.features.extractors.viv.function.extract_features(f)
+    def extract_function_features(self, fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+        yield from capa.features.extractors.viv.function.extract_features(fh)
 
-    def get_basic_blocks(self, f):
-        return f.basic_blocks
+    def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
+        f: viv_utils.Function = fh.inner
+        for bb in f.basic_blocks:
+            yield BBHandle(address=AbsoluteVirtualAddress(bb.va), inner=bb)
 
-    def extract_basic_block_features(self, f, bb):
-        yield from capa.features.extractors.viv.basicblock.extract_features(f, bb)
+    def extract_basic_block_features(self, fh: FunctionHandle, bbh) -> Iterator[Tuple[Feature, Address]]:
+        yield from capa.features.extractors.viv.basicblock.extract_features(fh, bbh)
 
-    def get_instructions(self, f, bb):
+    def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:
+        bb: viv_utils.BasicBlock = bbh.inner
         for insn in bb.instructions:
-            yield InstructionHandle(insn)
+            yield InsnHandle(address=AbsoluteVirtualAddress(insn.va), inner=insn)
 
-    def extract_insn_features(self, f, bb, insn):
-        yield from capa.features.extractors.viv.insn.extract_features(f, bb, insn)
+    def extract_insn_features(
+        self, fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+    ) -> Iterator[Tuple[Feature, Address]]:
+        yield from capa.features.extractors.viv.insn.extract_features(fh, bbh, ih)
 
-    def is_library_function(self, va):
-        return viv_utils.flirt.is_library_function(self.vw, va)
+    def is_library_function(self, addr):
+        return viv_utils.flirt.is_library_function(self.vw, addr)
 
-    def get_function_name(self, va):
-        return viv_utils.get_function_name(self.vw, va)
+    def get_function_name(self, addr):
+        return viv_utils.get_function_name(self.vw, addr)

capa/features/extractors/viv/file.py

@@ -1,10 +1,11 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import Tuple, Iterator
 
 import PE.carve as pe_carve  # vivisect PE
 import viv_utils
@@ -15,20 +16,21 @@ import capa.features.extractors.common
 import capa.features.extractors.helpers
 import capa.features.extractors.strings
 from capa.features.file import Export, Import, Section, FunctionName
-from capa.features.common import String, Characteristic
+from capa.features.common import String, Feature, Characteristic
+from capa.features.address import Address, FileOffsetAddress, AbsoluteVirtualAddress
 
 
-def extract_file_embedded_pe(buf, **kwargs):
+def extract_file_embedded_pe(buf, **kwargs) -> Iterator[Tuple[Feature, Address]]:
     for offset, _ in pe_carve.carve(buf, 1):
-        yield Characteristic("embedded pe"), offset
+        yield Characteristic("embedded pe"), FileOffsetAddress(offset)
 
 
-def extract_file_export_names(vw, **kwargs):
+def extract_file_export_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
     for va, _, name, _ in vw.getExports():
-        yield Export(name), va
+        yield Export(name), AbsoluteVirtualAddress(va)
 
 
-def extract_file_import_names(vw, **kwargs):
+def extract_file_import_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
     """
     extract imported function names
     1. imports by ordinal:
@@ -44,8 +46,9 @@ def extract_file_import_names(vw, **kwargs):
             # replace ord prefix with #
             impname = "#%s" % impname[len("ord") :]
 
+        addr = AbsoluteVirtualAddress(va)
         for name in capa.features.extractors.helpers.generate_symbols(modname, impname):
-            yield Import(name), va
+            yield Import(name), addr
 
 
 def is_viv_ord_impname(impname: str) -> bool:
@@ -62,30 +65,37 @@ def is_viv_ord_impname(impname: str) -> bool:
         return True
 
 
-def extract_file_section_names(vw, **kwargs):
+def extract_file_section_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
     for va, _, segname, _ in vw.getSegments():
-        yield Section(segname), va
+        yield Section(segname), AbsoluteVirtualAddress(va)
 
 
-def extract_file_strings(buf, **kwargs):
+def extract_file_strings(buf, **kwargs) -> Iterator[Tuple[Feature, Address]]:
     yield from capa.features.extractors.common.extract_file_strings(buf)
 
 
-def extract_file_function_names(vw, **kwargs):
+def extract_file_function_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
     """
     extract the names of statically-linked library functions.
     """
     for va in sorted(vw.getFunctions()):
+        addr = AbsoluteVirtualAddress(va)
         if viv_utils.flirt.is_library_function(vw, va):
             name = viv_utils.get_function_name(vw, va)
-            yield FunctionName(name), va
+            yield FunctionName(name), addr
+            if name.startswith("_"):
+                # some linkers may prefix linked routines with a `_` to avoid name collisions.
+                # extract features for both the mangled and un-mangled representations.
+                # e.g. `_fwrite` -> `fwrite`
+                # see: https://stackoverflow.com/a/2628384/87207
+                yield FunctionName(name[1:]), addr
 
 
-def extract_file_format(buf, **kwargs):
+def extract_file_format(buf, **kwargs) -> Iterator[Tuple[Feature, Address]]:
     yield from capa.features.extractors.common.extract_format(buf)
 
 
-def extract_features(vw, buf: bytes):
+def extract_features(vw, buf: bytes) -> Iterator[Tuple[Feature, Address]]:
     """
     extract file features from given workspace
 
@@ -94,12 +104,12 @@ def extract_features(vw, buf: bytes):
       buf: the raw input file bytes
 
     yields:
-      Tuple[Feature, VA]: a feature and its location.
+      Tuple[Feature, Address]: a feature and its location.
     """
 
     for file_handler in FILE_HANDLERS:
-        for feature, va in file_handler(vw=vw, buf=buf):  # type: ignore
-            yield feature, va
+        for feature, addr in file_handler(vw=vw, buf=buf):  # type: ignore
+            yield feature, addr
 
 
 FILE_HANDLERS = (

capa/features/extractors/viv/function.py

@@ -1,40 +1,47 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import Tuple, Iterator
 
 import envi
+import viv_utils
 import vivisect.const
 
-from capa.features.common import Characteristic
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
 from capa.features.extractors import loops
+from capa.features.extractors.base_extractor import FunctionHandle
 
 
-def interface_extract_function_XXX(f):
+def interface_extract_function_XXX(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse features from the given function.
 
     args:
-      f (viv_utils.Function): the function to process.
+      f: the function to process.
 
     yields:
-      (Feature, int): the feature and the address at which its found.
+      (Feature, Address): the feature and the address at which its found.
     """
-    yield NotImplementedError("feature"), NotImplementedError("virtual address")
+    ...
 
 
-def extract_function_calls_to(f):
+def extract_function_calls_to(fhandle: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+    f: viv_utils.Function = fhandle.inner
     for src, _, _, _ in f.vw.getXrefsTo(f.va, rtype=vivisect.const.REF_CODE):
-        yield Characteristic("calls to"), src
+        yield Characteristic("calls to"), AbsoluteVirtualAddress(src)
 
 
-def extract_function_loop(f):
+def extract_function_loop(fhandle: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse if a function has a loop
     """
+    f: viv_utils.Function = fhandle.inner
+
     edges = []
 
     for bb in f.basic_blocks:
@@ -50,22 +57,22 @@ def extract_function_loop(f):
                     edges.append((bb.va, bva))
 
     if edges and loops.has_loop(edges):
-        yield Characteristic("loop"), f.va
+        yield Characteristic("loop"), fhandle.address
 
 
-def extract_features(f):
+def extract_features(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     extract features from the given function.
 
     args:
-      f (viv_utils.Function): the function from which to extract features
+      fh: the function handle from which to extract features
 
     yields:
       Tuple[Feature, int]: the features and their location found in this function.
     """
     for func_handler in FUNCTION_HANDLERS:
-        for feature, va in func_handler(f):
-            yield feature, va
+        for feature, addr in func_handler(fh):
+            yield feature, addr
 
 
 FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop)

capa/features/extractors/viv/global_.py

@@ -1,19 +1,21 @@
 import logging
+from typing import Tuple, Iterator
 
 import envi.archs.i386
 import envi.archs.amd64
 
-from capa.features.common import ARCH_I386, ARCH_AMD64, Arch
+from capa.features.common import ARCH_I386, ARCH_AMD64, Arch, Feature
+from capa.features.address import NO_ADDRESS, Address
 
 logger = logging.getLogger(__name__)
 
 
-def extract_arch(vw):
+def extract_arch(vw) -> Iterator[Tuple[Feature, Address]]:
     if isinstance(vw.arch, envi.archs.amd64.Amd64Module):
-        yield Arch(ARCH_AMD64), 0x0
+        yield Arch(ARCH_AMD64), NO_ADDRESS
 
     elif isinstance(vw.arch, envi.archs.i386.i386Module):
-        yield Arch(ARCH_I386), 0x0
+        yield Arch(ARCH_I386), NO_ADDRESS
 
     else:
         # we likely end up here:

capa/features/extractors/viv/helpers.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/features/extractors/viv/indirect_calls.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -16,7 +16,7 @@ import envi.archs.amd64.disasm
 from vivisect import VivWorkspace
 
 if TYPE_CHECKING:
-    from capa.features.extractors.viv.extractor import InstructionHandle
+    from capa.features.extractors.viv.extractor import VivInstructionHandle
 
 # pull out consts for lookup performance
 i386RegOper = envi.archs.i386.disasm.i386RegOper
@@ -135,7 +135,7 @@ def find_definition(vw: VivWorkspace, va: int, reg: int) -> Tuple[int, Union[int
     raise NotFoundError()
 
 
-def is_indirect_call(vw: VivWorkspace, va: int, insn: Optional["InstructionHandle"] = None) -> bool:
+def is_indirect_call(vw: VivWorkspace, va: int, insn: Optional["VivInstructionHandle"] = None) -> bool:
     if insn is None:
         insn = vw.parseOpcode(va)
 
@@ -143,7 +143,7 @@ def is_indirect_call(vw: VivWorkspace, va: int, insn: Optional["InstructionHandl
 
 
 def resolve_indirect_call(
-    vw: VivWorkspace, va: int, insn: Optional["InstructionHandle"] = None
+    vw: VivWorkspace, va: int, insn: Optional["VivInstructionHandle"] = None
 ) -> Tuple[int, Optional[int]]:
     """
     inspect the given indirect call instruction and attempt to resolve the target address.

capa/features/extractors/viv/insn.py

@@ -1,10 +1,12 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import List, Tuple, Callable, Iterator
+
 import envi
 import envi.exc
 import viv_utils
@@ -17,16 +19,10 @@ import envi.archs.amd64.disasm
 
 import capa.features.extractors.helpers
 import capa.features.extractors.viv.helpers
-from capa.features.insn import API, Number, Offset, Mnemonic
-from capa.features.common import (
-    BITNESS_X32,
-    BITNESS_X64,
-    MAX_BYTES_FEATURE_SIZE,
-    THUNK_CHAIN_DEPTH_DELTA,
-    Bytes,
-    String,
-    Characteristic,
-)
+from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
+from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
 from capa.features.extractors.viv.indirect_calls import NotFoundError, resolve_indirect_call
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
@@ -34,27 +30,21 @@ from capa.features.extractors.viv.indirect_calls import NotFoundError, resolve_i
 SECURITY_COOKIE_BYTES_DELTA = 0x40
 
 
-def get_bitness(vw):
-    bitness = vw.getMeta("Architecture")
-    if bitness == "i386":
-        return BITNESS_X32
-    elif bitness == "amd64":
-        return BITNESS_X64
-
-
-def interface_extract_instruction_XXX(f, bb, insn):
+def interface_extract_instruction_XXX(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """
     parse features from the given instruction.
 
     args:
-      f (viv_utils.Function): the function to process.
-      bb (viv_utils.BasicBlock): the basic block to process.
-      insn (vivisect...Instruction): the instruction to process.
+      fh: the function handle to process.
+      bbh: the basic block handle to process.
+      ih: the instruction handle to process.
 
     yields:
-      (Feature, int): the feature and the address at which its found.
+      (Feature, Address): the feature and the address at which its found.
     """
-    yield NotImplementedError("feature"), NotImplementedError("virtual address")
+    ...
 
 
 def get_imports(vw):
@@ -74,12 +64,15 @@ def get_imports(vw):
         return imports
 
 
-def extract_insn_api_features(f, bb, insn):
-    """parse API features from the given instruction."""
+def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse API features from the given instruction.
 
-    # example:
-    #
-    #    call dword [0x00473038]
+    example:
+       call dword [0x00473038]
+    """
+    insn: envi.Opcode = ih.inner
+    f: viv_utils.Function = fh.inner
     if insn.mnem not in ("call", "jmp"):
         return
 
@@ -96,7 +89,7 @@ def extract_insn_api_features(f, bb, insn):
         if target in imports:
             dll, symbol = imports[target]
             for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
-                yield API(name), insn.va
+                yield API(name), ih.address
 
     # call via thunk on x86,
     # see 9324d1a8ae37a36ae560c37448c9705a at 0x407985
@@ -118,14 +111,20 @@ def extract_insn_api_features(f, bb, insn):
 
         if viv_utils.flirt.is_library_function(f.vw, target):
             name = viv_utils.get_function_name(f.vw, target)
-            yield API(name), insn.va
+            yield API(name), ih.address
+            if name.startswith("_"):
+                # some linkers may prefix linked routines with a `_` to avoid name collisions.
+                # extract features for both the mangled and un-mangled representations.
+                # e.g. `_fwrite` -> `fwrite`
+                # see: https://stackoverflow.com/a/2628384/87207
+                yield API(name[1:]), ih.address
             return
 
         for _ in range(THUNK_CHAIN_DEPTH_DELTA):
             if target in imports:
                 dll, symbol = imports[target]
                 for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
-                    yield API(name), insn.va
+                    yield API(name), ih.address
 
             # if jump leads to an ENDBRANCH instruction, skip it
             if f.vw.getByteDef(target)[1].startswith(b"\xf3\x0f\x1e"):
@@ -145,7 +144,7 @@ def extract_insn_api_features(f, bb, insn):
         if target in imports:
             dll, symbol = imports[target]
             for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
-                yield API(name), insn.va
+                yield API(name), ih.address
 
     elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegOper):
         try:
@@ -162,38 +161,7 @@ def extract_insn_api_features(f, bb, insn):
         if target in imports:
             dll, symbol = imports[target]
             for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
-                yield API(name), insn.va
-
-
-def extract_insn_number_features(f, bb, insn):
-    """parse number features from the given instruction."""
-    # example:
-    #
-    #     push    3136B0h         ; dwControlCode
-    for oper in insn.opers:
-        # this is for both x32 and x64
-        if not isinstance(oper, (envi.archs.i386.disasm.i386ImmOper, envi.archs.i386.disasm.i386ImmMemOper)):
-            continue
-
-        if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
-            v = oper.getOperValue(oper)
-        else:
-            v = oper.getOperAddr(oper)
-
-        if f.vw.probeMemory(v, 1, envi.memory.MM_READ):
-            # this is a valid address
-            # assume its not also a constant.
-            continue
-
-        if insn.mnem == "add" and insn.opers[0].isReg() and insn.opers[0].reg == envi.archs.i386.regs.REG_ESP:
-            # skip things like:
-            #
-            #    .text:00401140                 call    sub_407E2B
-            #    .text:00401145                 add     esp, 0Ch
-            return
-
-        yield Number(v), insn.va
-        yield Number(v, bitness=get_bitness(f.vw)), insn.va
+                yield API(name), ih.address
 
 
 def derefs(vw, p):
@@ -266,12 +234,15 @@ def read_bytes(vw, va: int) -> bytes:
         raise
 
 
-def extract_insn_bytes_features(f, bb, insn):
+def extract_insn_bytes_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse byte sequence features from the given instruction.
     example:
         #     push    offset iid_004118d4_IShellLinkA ; riid
     """
+    insn: envi.Opcode = ih.inner
+    f: viv_utils.Function = fh.inner
+
     if insn.mnem == "call":
         return
 
@@ -300,7 +271,7 @@ def extract_insn_bytes_features(f, bb, insn):
             if capa.features.extractors.helpers.all_zeros(buf):
                 continue
 
-            yield Bytes(buf), insn.va
+            yield Bytes(buf), ih.address
 
 
 def read_string(vw, offset: int) -> str:
@@ -334,75 +305,6 @@ def read_string(vw, offset: int) -> str:
     raise ValueError("not a string", offset)
 
 
-def extract_insn_string_features(f, bb, insn):
-    """parse string features from the given instruction."""
-    # example:
-    #
-    #     push    offset aAcr     ; "ACR  > "
-
-    for oper in insn.opers:
-        if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
-            v = oper.getOperValue(oper)
-        elif isinstance(oper, envi.archs.i386.disasm.i386ImmMemOper):
-            # like 0x10056CB4 in `lea eax, dword [0x10056CB4]`
-            v = oper.imm
-        elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
-            # like 0x401000 in `mov eax, 0x401000[2 * ebx]`
-            v = oper.imm
-        elif isinstance(oper, envi.archs.amd64.disasm.Amd64RipRelOper):
-            v = oper.getOperAddr(insn)
-        else:
-            continue
-
-        for v in derefs(f.vw, v):
-            try:
-                s = read_string(f.vw, v)
-            except ValueError:
-                continue
-            else:
-                yield String(s.rstrip("\x00")), insn.va
-
-
-def extract_insn_offset_features(f, bb, insn):
-    """parse structure offset features from the given instruction."""
-    # example:
-    #
-    #     .text:0040112F    cmp     [esi+4], ebx
-    for oper in insn.opers:
-
-        # this is for both x32 and x64
-        # like [esi + 4]
-        #       reg   ^
-        #             disp
-        if isinstance(oper, envi.archs.i386.disasm.i386RegMemOper):
-            if oper.reg == envi.archs.i386.regs.REG_ESP:
-                continue
-
-            if oper.reg == envi.archs.i386.regs.REG_EBP:
-                continue
-
-            # TODO: do x64 support for real.
-            if oper.reg == envi.archs.amd64.regs.REG_RBP:
-                continue
-
-            # viv already decodes offsets as signed
-            v = oper.disp
-
-            yield Offset(v), insn.va
-            yield Offset(v, bitness=get_bitness(f.vw)), insn.va
-
-        # like: [esi + ecx + 16384]
-        #        reg   ^     ^
-        #              index ^
-        #                    disp
-        elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
-            # viv already decodes offsets as signed
-            v = oper.disp
-
-            yield Offset(v), insn.va
-            yield Offset(v, bitness=get_bitness(f.vw)), insn.va
-
-
 def is_security_cookie(f, bb, insn) -> bool:
     """
     check if an instruction is related to security cookie checks
@@ -431,11 +333,17 @@ def is_security_cookie(f, bb, insn) -> bool:
     return False
 
 
-def extract_insn_nzxor_characteristic_features(f, bb, insn):
+def extract_insn_nzxor_characteristic_features(
+    fh: FunctionHandle, bbhandle: BBHandle, ih: InsnHandle
+) -> Iterator[Tuple[Feature, Address]]:
     """
     parse non-zeroing XOR instruction from the given instruction.
     ignore expected non-zeroing XORs, e.g. security cookies.
     """
+    insn: envi.Opcode = ih.inner
+    bb: viv_utils.BasicBlock = bbhandle.inner
+    f: viv_utils.Function = fh.inner
+
     if insn.mnem not in ("xor", "xorpd", "xorps", "pxor"):
         return
 
@@ -445,19 +353,40 @@ def extract_insn_nzxor_characteristic_features(f, bb, insn):
     if is_security_cookie(f, bb, insn):
         return
 
-    yield Characteristic("nzxor"), insn.va
+    yield Characteristic("nzxor"), ih.address
 
 
-def extract_insn_mnemonic_features(f, bb, insn):
+def extract_insn_mnemonic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse mnemonic features from the given instruction."""
-    yield Mnemonic(insn.mnem), insn.va
+    yield Mnemonic(ih.inner.mnem), ih.address
 
 
-def extract_insn_peb_access_characteristic_features(f, bb, insn):
+def extract_insn_obfs_call_plus_5_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    """
+    parse call $+5 instruction from the given instruction.
+    """
+    insn: envi.Opcode = ih.inner
+
+    if insn.mnem != "call":
+        return
+
+    if isinstance(insn.opers[0], envi.archs.i386.disasm.i386PcRelOper):
+        if insn.va + 5 == insn.opers[0].getOperValue(insn):
+            yield Characteristic("call $+5"), ih.address
+
+    if isinstance(insn.opers[0], envi.archs.i386.disasm.i386ImmMemOper) or isinstance(
+        insn.opers[0], envi.archs.amd64.disasm.Amd64RipRelOper
+    ):
+        if insn.va + 5 == insn.opers[0].getOperAddr(insn):
+            yield Characteristic("call $+5"), ih.address
+
+
+def extract_insn_peb_access_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     parse peb access from the given function. fs:[0x30] on x86, gs:[0x60] on x64
     """
     # TODO handle where fs/gs are loaded into a register or onto the stack and used later
+    insn: envi.Opcode = ih.inner
 
     if insn.mnem not in ["push", "mov"]:
         return
@@ -476,7 +405,7 @@ def extract_insn_peb_access_characteristic_features(f, bb, insn):
             if (isinstance(oper, envi.archs.i386.disasm.i386RegMemOper) and oper.disp == 0x30) or (
                 isinstance(oper, envi.archs.i386.disasm.i386ImmMemOper) and oper.imm == 0x30
             ):
-                yield Characteristic("peb access"), insn.va
+                yield Characteristic("peb access"), ih.address
     elif "gs" in prefix:
         for oper in insn.opers:
             if (
@@ -484,20 +413,22 @@ def extract_insn_peb_access_characteristic_features(f, bb, insn):
                 or (isinstance(oper, envi.archs.amd64.disasm.i386SibOper) and oper.imm == 0x60)
                 or (isinstance(oper, envi.archs.amd64.disasm.i386ImmMemOper) and oper.imm == 0x60)
             ):
-                yield Characteristic("peb access"), insn.va
+                yield Characteristic("peb access"), ih.address
     else:
         pass
 
 
-def extract_insn_segment_access_features(f, bb, insn):
+def extract_insn_segment_access_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """parse the instruction for access to fs or gs"""
+    insn: envi.Opcode = ih.inner
+
     prefix = insn.getPrefixName()
 
     if prefix == "fs":
-        yield Characteristic("fs access"), insn.va
+        yield Characteristic("fs access"), ih.address
 
     if prefix == "gs":
-        yield Characteristic("gs access"), insn.va
+        yield Characteristic("gs access"), ih.address
 
 
 def get_section(vw, va: int):
@@ -508,10 +439,13 @@ def get_section(vw, va: int):
     raise KeyError(va)
 
 
-def extract_insn_cross_section_cflow(f, bb, insn):
+def extract_insn_cross_section_cflow(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     inspect the instruction for a CALL or JMP that crosses section boundaries.
     """
+    insn: envi.Opcode = ih.inner
+    f: viv_utils.Function = fh.inner
+
     for va, flags in insn.getBranches():
         if va is None:
             # va may be none for dynamic branches that haven't been resolved, such as `jmp eax`.
@@ -538,7 +472,7 @@ def extract_insn_cross_section_cflow(f, bb, insn):
                     continue
 
             if get_section(f.vw, insn.va) != get_section(f.vw, va):
-                yield Characteristic("cross section flow"), insn.va
+                yield Characteristic("cross section flow"), ih.address
 
         except KeyError:
             continue
@@ -546,7 +480,10 @@ def extract_insn_cross_section_cflow(f, bb, insn):
 
 # this is a feature that's most relevant at the function scope,
 # however, its most efficient to extract at the instruction scope.
-def extract_function_calls_from(f, bb, insn):
+def extract_function_calls_from(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    insn: envi.Opcode = ih.inner
+    f: viv_utils.Function = fh.inner
+
     if insn.mnem != "call":
         return
 
@@ -556,7 +493,7 @@ def extract_function_calls_from(f, bb, insn):
     if isinstance(insn.opers[0], envi.archs.i386.disasm.i386ImmMemOper):
         oper = insn.opers[0]
         target = oper.getOperAddr(insn)
-        yield Characteristic("calls from"), target
+        yield Characteristic("calls from"), AbsoluteVirtualAddress(target)
 
     # call via thunk on x86,
     # see 9324d1a8ae37a36ae560c37448c9705a at 0x407985
@@ -565,43 +502,190 @@ def extract_function_calls_from(f, bb, insn):
     # see Lab21-01.exe_:0x140001178
     elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386PcRelOper):
         target = insn.opers[0].getOperValue(insn)
-        yield Characteristic("calls from"), target
+        yield Characteristic("calls from"), AbsoluteVirtualAddress(target)
 
     # call via IAT, x64
     elif isinstance(insn.opers[0], envi.archs.amd64.disasm.Amd64RipRelOper):
         op = insn.opers[0]
         target = op.getOperAddr(insn)
-        yield Characteristic("calls from"), target
+        yield Characteristic("calls from"), AbsoluteVirtualAddress(target)
 
     if target and target == f.va:
         # if we found a jump target and it's the function address
         # mark as recursive
-        yield Characteristic("recursive call"), target
+        yield Characteristic("recursive call"), AbsoluteVirtualAddress(target)
 
 
 # this is a feature that's most relevant at the function or basic block scope,
 # however, its most efficient to extract at the instruction scope.
-def extract_function_indirect_call_characteristic_features(f, bb, insn):
+def extract_function_indirect_call_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
     """
     extract indirect function call characteristic (e.g., call eax or call dword ptr [edx+4])
     does not include calls like => call ds:dword_ABD4974
     """
+    insn: envi.Opcode = ih.inner
+
     if insn.mnem != "call":
         return
 
     # Checks below work for x86 and x64
     if isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegOper):
         # call edx
-        yield Characteristic("indirect call"), insn.va
+        yield Characteristic("indirect call"), ih.address
     elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegMemOper):
         # call dword ptr [eax+50h]
-        yield Characteristic("indirect call"), insn.va
+        yield Characteristic("indirect call"), ih.address
     elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386SibOper):
         # call qword ptr [rsp+78h]
-        yield Characteristic("indirect call"), insn.va
+        yield Characteristic("indirect call"), ih.address
 
 
-def extract_features(f, bb, insn):
+def extract_op_number_features(
+    fh: FunctionHandle, bb, ih: InsnHandle, i, oper: envi.Operand
+) -> Iterator[Tuple[Feature, Address]]:
+    """parse number features from the given operand.
+
+    example:
+        push    3136B0h         ; dwControlCode
+    """
+    insn: envi.Opcode = ih.inner
+    f: viv_utils.Function = fh.inner
+
+    # this is for both x32 and x64
+    if not isinstance(oper, (envi.archs.i386.disasm.i386ImmOper, envi.archs.i386.disasm.i386ImmMemOper)):
+        return
+
+    if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
+        v = oper.getOperValue(oper)
+    else:
+        v = oper.getOperAddr(oper)
+
+    if f.vw.probeMemory(v, 1, envi.memory.MM_READ):
+        # this is a valid address
+        # assume its not also a constant.
+        return
+
+    if insn.mnem == "add" and insn.opers[0].isReg() and insn.opers[0].reg == envi.archs.i386.regs.REG_ESP:
+        # skip things like:
+        #
+        #    .text:00401140                 call    sub_407E2B
+        #    .text:00401145                 add     esp, 0Ch
+        return
+
+    yield Number(v), ih.address
+    yield OperandNumber(i, v), ih.address
+
+    if insn.mnem == "add" and 0 < v < MAX_STRUCTURE_SIZE and isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
+        # for pattern like:
+        #
+        #     add eax, 0x10
+        #
+        # assume 0x10 is also an offset (imagine eax is a pointer).
+        yield Offset(v), ih.address
+        yield OperandOffset(i, v), ih.address
+
+
+def extract_op_offset_features(
+    fh: FunctionHandle, bb, ih: InsnHandle, i, oper: envi.Operand
+) -> Iterator[Tuple[Feature, Address]]:
+    """parse structure offset features from the given operand."""
+    # example:
+    #
+    #     .text:0040112F    cmp     [esi+4], ebx
+    insn: envi.Opcode = ih.inner
+    f: viv_utils.Function = fh.inner
+
+    # this is for both x32 and x64
+    # like [esi + 4]
+    #       reg   ^
+    #             disp
+    if isinstance(oper, envi.archs.i386.disasm.i386RegMemOper):
+        if oper.reg == envi.archs.i386.regs.REG_ESP:
+            return
+
+        if oper.reg == envi.archs.i386.regs.REG_EBP:
+            return
+
+        # TODO: do x64 support for real.
+        if oper.reg == envi.archs.amd64.regs.REG_RBP:
+            return
+
+        # viv already decodes offsets as signed
+        v = oper.disp
+
+        yield Offset(v), ih.address
+        yield OperandOffset(i, v), ih.address
+
+        if insn.mnem == "lea" and i == 1 and not f.vw.probeMemory(v, 1, envi.memory.MM_READ):
+            # for pattern like:
+            #
+            #     lea eax, [ebx + 1]
+            #
+            # assume 1 is also an offset (imagine ebx is a zero register).
+            yield Number(v), ih.address
+            yield OperandNumber(i, v), ih.address
+
+    # like: [esi + ecx + 16384]
+    #        reg   ^     ^
+    #              index ^
+    #                    disp
+    elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
+        # viv already decodes offsets as signed
+        v = oper.disp
+
+        yield Offset(v), ih.address
+        yield OperandOffset(i, v), ih.address
+
+
+def extract_op_string_features(
+    fh: FunctionHandle, bb, ih: InsnHandle, i, oper: envi.Operand
+) -> Iterator[Tuple[Feature, Address]]:
+    """parse string features from the given operand."""
+    # example:
+    #
+    #     push    offset aAcr     ; "ACR  > "
+    insn: envi.Opcode = ih.inner
+    f: viv_utils.Function = fh.inner
+
+    if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
+        v = oper.getOperValue(oper)
+    elif isinstance(oper, envi.archs.i386.disasm.i386ImmMemOper):
+        # like 0x10056CB4 in `lea eax, dword [0x10056CB4]`
+        v = oper.imm
+    elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
+        # like 0x401000 in `mov eax, 0x401000[2 * ebx]`
+        v = oper.imm
+    elif isinstance(oper, envi.archs.amd64.disasm.Amd64RipRelOper):
+        v = oper.getOperAddr(insn)
+    else:
+        return
+
+    for v in derefs(f.vw, v):
+        try:
+            s = read_string(f.vw, v)
+        except ValueError:
+            continue
+        else:
+            yield String(s.rstrip("\x00")), ih.address
+
+
+def extract_operand_features(f: FunctionHandle, bb, insn: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    for i, oper in enumerate(insn.inner.opers):
+        for op_handler in OPERAND_HANDLERS:
+            for feature, addr in op_handler(f, bb, insn, i, oper):
+                yield feature, addr
+
+
+OPERAND_HANDLERS: List[
+    Callable[[FunctionHandle, BBHandle, InsnHandle, int, envi.Operand], Iterator[Tuple[Feature, Address]]]
+] = [
+    extract_op_number_features,
+    extract_op_offset_features,
+    extract_op_string_features,
+]
+
+
+def extract_features(f, bb, insn) -> Iterator[Tuple[Feature, Address]]:
     """
     extract features from the given insn.
 
@@ -611,24 +695,23 @@ def extract_features(f, bb, insn):
       insn (vivisect...Instruction): the instruction to process.
 
     yields:
-      Tuple[Feature, int]: the features and their location found in this insn.
+      Tuple[Feature, Address]: the features and their location found in this insn.
     """
     for insn_handler in INSTRUCTION_HANDLERS:
-        for feature, va in insn_handler(f, bb, insn):
-            yield feature, va
+        for feature, addr in insn_handler(f, bb, insn):
+            yield feature, addr
 
 
-INSTRUCTION_HANDLERS = (
+INSTRUCTION_HANDLERS: List[Callable[[FunctionHandle, BBHandle, InsnHandle], Iterator[Tuple[Feature, Address]]]] = [
     extract_insn_api_features,
-    extract_insn_number_features,
-    extract_insn_string_features,
     extract_insn_bytes_features,
-    extract_insn_offset_features,
     extract_insn_nzxor_characteristic_features,
     extract_insn_mnemonic_features,
+    extract_insn_obfs_call_plus_5_characteristic_features,
     extract_insn_peb_access_characteristic_features,
     extract_insn_cross_section_cflow,
     extract_insn_segment_access_features,
     extract_function_calls_from,
     extract_function_indirect_call_characteristic_features,
-)
+    extract_operand_features,
+]

capa/features/file.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/features/freeze.py

@@ -1,51 +1,7 @@
 """
 capa freeze file format: `| capa0000 | + zlib(utf-8(json(...)))`
 
-json format:
-
-    {
-      'version': 1,
-      'base address': int(base address),
-      'functions': {
-        int(function va): {
-          'basic blocks': {
-            int(basic block va): {
-              'instructions': [instruction va, ...]
-            },
-            ...
-          },
-          ...
-        },
-        ...
-      },
-      'scopes': {
-        'global': [
-          (str(name), [any(arg), ...], int(va), ()),
-          ...
-        },
-        'file': [
-          (str(name), [any(arg), ...], int(va), ()),
-          ...
-        },
-        'function': [
-          (str(name), [any(arg), ...], int(va), (int(function va), )),
-          ...
-        ],
-        'basic block': [
-          (str(name), [any(arg), ...], int(va), (int(function va),
-                                                 int(basic block va))),
-          ...
-        ],
-        'instruction': [
-          (str(name), [any(arg), ...], int(va), (int(function va),
-                                                 int(basic block va),
-                                                 int(instruction va))),
-          ...
-        ],
-      }
-    }
-
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -53,191 +9,342 @@ Unless required by applicable law or agreed to in writing, software distributed
  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and limitations under the License.
 """
-import json
 import zlib
 import logging
+import collections
+from enum import Enum
+from typing import Any, Set, Dict, List, Type, Tuple
 
+from pydantic import Field, BaseModel
+
+import capa.helpers
 import capa.features.file
 import capa.features.insn
 import capa.features.common
+import capa.features.address
 import capa.features.basicblock
 import capa.features.extractors.base_extractor
-from capa.helpers import hex
+from capa.helpers import assert_never
 
 logger = logging.getLogger(__name__)
 
 
-def serialize_feature(feature):
-    return feature.freeze_serialize()
+class HashableModel(BaseModel):
+    class Config:
+        frozen = True
 
 
-KNOWN_FEATURES = {F.__name__: F for F in capa.features.common.Feature.__subclasses__()}
+class AddressType(str, Enum):
+    ABSOLUTE = "absolute"
+    RELATIVE = "relative"
+    FILE = "file"
+    DN_TOKEN = "dn token"
+    DN_TOKEN_OFFSET = "dn token offset"
+    NO_ADDRESS = "no address"
 
 
-def deserialize_feature(doc):
-    F = KNOWN_FEATURES[doc[0]]
-    return F.freeze_deserialize(doc[1])
+class Address(HashableModel):
+    type: AddressType
+    value: Any
+
+    @classmethod
+    def from_capa(cls, a: capa.features.address.Address) -> "Address":
+        if isinstance(a, capa.features.address.AbsoluteVirtualAddress):
+            return cls(type=AddressType.ABSOLUTE, value=int(a))
+
+        elif isinstance(a, capa.features.address.RelativeVirtualAddress):
+            return cls(type=AddressType.RELATIVE, value=int(a))
+
+        elif isinstance(a, capa.features.address.FileOffsetAddress):
+            return cls(type=AddressType.FILE, value=int(a))
+
+        elif isinstance(a, capa.features.address.DNTokenAddress):
+            # TODO: probably need serialization here
+            return cls(type=AddressType.DN_TOKEN, value=a.token)
+
+        elif isinstance(a, capa.features.address.DNTokenOffsetAddress):
+            # TODO: probably need serialization here
+            return cls(type=AddressType.DN_TOKEN_OFFSET, value=(a.token, a.offset))
+
+        elif a == capa.features.address.NO_ADDRESS or isinstance(a, capa.features.address._NoAddress):
+            return cls(type=AddressType.NO_ADDRESS, value=None)
+
+        elif isinstance(a, capa.features.address.Address) and not issubclass(type(a), capa.features.address.Address):
+            raise ValueError("don't use an Address instance directly")
+
+        elif isinstance(a, capa.features.address.Address):
+            raise ValueError("don't use an Address instance directly")
+
+        else:
+            assert_never(a)
+
+    def to_capa(self) -> capa.features.address.Address:
+        if self.type is AddressType.ABSOLUTE:
+            return capa.features.address.AbsoluteVirtualAddress(self.value)
+
+        elif self.type is AddressType.RELATIVE:
+            return capa.features.address.RelativeVirtualAddress(self.value)
+
+        elif self.type is AddressType.FILE:
+            return capa.features.address.FileOffsetAddress(self.value)
+
+        elif self.type is AddressType.DN_TOKEN:
+            return capa.features.address.DNTokenAddress(self.value)
+
+        elif self.type is AddressType.DN_TOKEN_OFFSET:
+            return capa.features.address.DNTokenOffsetAddress(*self.value)
+
+        elif self.type is AddressType.NO_ADDRESS:
+            return capa.features.address.NO_ADDRESS
+
+        else:
+            assert_never(self.type)
 
 
-def dumps(extractor):
+KNOWN_FEATURES: Dict[str, Type[capa.features.common.Feature]] = {
+    F.__name__: F for F in capa.features.common.Feature.__subclasses__()
+}
+KNOWN_FEATURES.update({F.__name__: F for F in capa.features.insn._Operand.__subclasses__()})  # type: ignore
+
+
+class Feature(HashableModel):
+    name: str
+    args: Tuple[Any, ...]
+
+    @classmethod
+    def from_capa(cls, f: capa.features.common.Feature) -> "Feature":
+        name, args = f.freeze_serialize()
+        return cls(name=name, args=tuple(args))
+
+    def to_capa(self) -> capa.features.common.Feature:
+        F = KNOWN_FEATURES[self.name]
+        return F.freeze_deserialize(self.args)
+
+
+class GlobalFeature(HashableModel):
+    feature: Feature
+
+
+class FileFeature(HashableModel):
+    address: Address
+    feature: Feature
+
+
+class FunctionFeature(HashableModel):
+    """
+    args:
+        function: the address of the function to which this feature belongs.
+        address: the address at which this feature is found.
+
+    function != address because, e.g., the feature may be found *within* the scope (function).
+    versus right at its starting address.
+    """
+
+    function: Address
+    address: Address
+    feature: Feature
+
+
+class BasicBlockFeature(HashableModel):
+    """
+    args:
+        basic_block: the address of the basic block to which this feature belongs.
+        address: the address at which this feature is found.
+
+    basic_block != address because, e.g., the feature may be found *within* the scope (basic block).
+    versus right at its starting address.
+    """
+
+    basic_block: Address
+    address: Address
+    feature: Feature
+
+
+class InstructionFeature(HashableModel):
+    """
+    args:
+        instruction: the address of the instruction to which this feature belongs.
+        address: the address at which this feature is found.
+
+    instruction != address because, e.g., the feature may be found *within* the scope (basic block),
+    versus right at its starting address.
+    """
+
+    instruction: Address
+    address: Address
+    feature: Feature
+
+
+class InstructionFeatures(BaseModel):
+    address: Address
+    features: List[InstructionFeature]
+
+
+class BasicBlockFeatures(BaseModel):
+    address: Address
+    features: List[BasicBlockFeature]
+    instructions: List[InstructionFeatures]
+
+
+class FunctionFeatures(BaseModel):
+    address: Address
+    features: List[FunctionFeature]
+    basic_blocks: List[BasicBlockFeatures] = Field(alias="basic block")
+
+    class Config:
+        allow_population_by_field_name = True
+
+
+class Features(BaseModel):
+    global_: List[GlobalFeature] = Field(alias="global")
+    file: List[FileFeature]
+    functions: List[FunctionFeatures]
+
+    class Config:
+        allow_population_by_field_name = True
+
+
+class Freeze(BaseModel):
+    version: int = 2
+    base_address: Address = Field(alias="base address")
+    features: Features
+
+    class Config:
+        allow_population_by_field_name = True
+
+
+def dumps(extractor: capa.features.extractors.base_extractor.FeatureExtractor) -> str:
     """
     serialize the given extractor to a string
-
-    args:
-      extractor: capa.features.extractors.base_extractor.FeatureExtractor:
-
-    returns:
-      str: the serialized features.
     """
-    ret = {
-        "version": 1,
-        "base address": extractor.get_base_address(),
-        "functions": {},
-        "scopes": {
-            "global": [],
-            "file": [],
-            "function": [],
-            "basic block": [],
-            "instruction": [],
-        },
-    }
-    for feature, va in extractor.extract_global_features():
-        ret["scopes"]["global"].append(serialize_feature(feature) + (hex(va), ()))
 
-    for feature, va in extractor.extract_file_features():
-        ret["scopes"]["file"].append(serialize_feature(feature) + (hex(va), ()))
+    global_features: List[GlobalFeature] = []
+    for feature, _ in extractor.extract_global_features():
+        global_features.append(
+            GlobalFeature(
+                feature=Feature.from_capa(feature),
+            )
+        )
 
+    file_features: List[FileFeature] = []
+    for feature, address in extractor.extract_file_features():
+        file_features.append(
+            FileFeature(
+                feature=Feature.from_capa(feature),
+                address=Address.from_capa(address),
+            )
+        )
+
+    function_features: List[FunctionFeatures] = []
     for f in extractor.get_functions():
-        ret["functions"][hex(f)] = {}
-
-        for feature, va in extractor.extract_function_features(f):
-            ret["scopes"]["function"].append(serialize_feature(feature) + (hex(va), (hex(f),)))
+        faddr = Address.from_capa(f.address)
+        ffeatures = [
+            FunctionFeature(
+                function=faddr,
+                address=Address.from_capa(addr),
+                feature=Feature.from_capa(feature),
+            )
+            for feature, addr in extractor.extract_function_features(f)
+        ]
 
+        basic_blocks = []
         for bb in extractor.get_basic_blocks(f):
-            ret["functions"][hex(f)][hex(bb)] = []
+            bbaddr = Address.from_capa(bb.address)
+            bbfeatures = [
+                BasicBlockFeature(
+                    basic_block=bbaddr,
+                    address=Address.from_capa(addr),
+                    feature=Feature.from_capa(feature),
+                )
+                for feature, addr in extractor.extract_basic_block_features(f, bb)
+            ]
 
-            for feature, va in extractor.extract_basic_block_features(f, bb):
-                ret["scopes"]["basic block"].append(
-                    serialize_feature(feature)
-                    + (
-                        hex(va),
-                        (
-                            hex(f),
-                            hex(bb),
-                        ),
+            instructions = []
+            for insn in extractor.get_instructions(f, bb):
+                iaddr = Address.from_capa(insn.address)
+                ifeatures = [
+                    InstructionFeature(
+                        instruction=iaddr,
+                        address=Address.from_capa(addr),
+                        feature=Feature.from_capa(feature),
+                    )
+                    for feature, addr in extractor.extract_insn_features(f, bb, insn)
+                ]
+                instructions.append(
+                    InstructionFeatures(
+                        address=iaddr,
+                        features=ifeatures,
                     )
                 )
 
-            for insnva, insn in sorted(
-                [(int(insn), insn) for insn in extractor.get_instructions(f, bb)], key=lambda p: p[0]
-            ):
-                ret["functions"][hex(f)][hex(bb)].append(hex(insnva))
+            basic_blocks.append(
+                BasicBlockFeatures(
+                    address=bbaddr,
+                    features=bbfeatures,
+                    instructions=instructions,
+                )
+            )
 
-                for feature, va in extractor.extract_insn_features(f, bb, insn):
-                    ret["scopes"]["instruction"].append(
-                        serialize_feature(feature)
-                        + (
-                            hex(va),
-                            (
-                                hex(f),
-                                hex(bb),
-                                hex(insnva),
-                            ),
-                        )
-                    )
-    return json.dumps(ret)
+        function_features.append(
+            FunctionFeatures(
+                address=faddr,
+                features=ffeatures,
+                basic_blocks=basic_blocks,
+            )
+        )
+
+    features = Features(
+        global_=global_features,
+        file=file_features,
+        functions=function_features,
+    )
+
+    freeze = Freeze(
+        version=2,
+        base_address=Address.from_capa(extractor.get_base_address()),
+        features=features,
+    )
+
+    return freeze.json()
 
 
-def loads(s):
+def loads(s: str) -> capa.features.extractors.base_extractor.FeatureExtractor:
     """deserialize a set of features (as a NullFeatureExtractor) from a string."""
-    doc = json.loads(s)
+    import capa.features.extractors.null as null
 
-    if doc.get("version") != 1:
-        raise ValueError("unsupported freeze format version: %d" % (doc.get("version")))
+    freeze = Freeze.parse_raw(s)
+    if freeze.version != 2:
+        raise ValueError("unsupported freeze format version: %d", freeze.version)
 
-    features = {
-        "base address": doc.get("base address"),
-        "global features": [],
-        "file features": [],
-        "functions": {},
-    }
-
-    for fva, function in doc.get("functions", {}).items():
-        fva = int(fva, 0x10)
-        features["functions"][fva] = {
-            "features": [],
-            "basic blocks": {},
-        }
-
-        for bbva, bb in function.items():
-            bbva = int(bbva, 0x10)
-            features["functions"][fva]["basic blocks"][bbva] = {
-                "features": [],
-                "instructions": {},
-            }
-
-            for insnva in bb:
-                insnva = int(insnva, 0x10)
-                features["functions"][fva]["basic blocks"][bbva]["instructions"][insnva] = {
-                    "features": [],
-                }
-
-    # in the following blocks, each entry looks like:
-    #
-    #     ('MatchedRule', ('foo', ), '0x401000', ('0x401000', ))
-    #      ^^^^^^^^^^^^^  ^^^^^^^^^  ^^^^^^^^^^  ^^^^^^^^^^^^^^
-    #      feature name   args       addr         func/bb/insn
-    for feature in doc.get("scopes", {}).get("global", []):
-        va, loc = feature[2:]
-        va = int(va, 0x10)
-        feature = deserialize_feature(feature[:2])
-        features["global features"].append((va, feature))
-
-    for feature in doc.get("scopes", {}).get("file", []):
-        va, loc = feature[2:]
-        va = int(va, 0x10)
-        feature = deserialize_feature(feature[:2])
-        features["file features"].append((va, feature))
-
-    for feature in doc.get("scopes", {}).get("function", []):
-        # fetch the pair like:
-        #
-        #     ('0x401000', ('0x401000', ))
-        #      ^^^^^^^^^^  ^^^^^^^^^^^^^^
-        #      addr         func/bb/insn
-        va, loc = feature[2:]
-        va = int(va, 0x10)
-        loc = [int(lo, 0x10) for lo in loc]
-
-        # decode the feature from the pair like:
-        #
-        #     ('MatchedRule', ('foo', ))
-        #      ^^^^^^^^^^^^^  ^^^^^^^^^
-        #      feature name   args
-        feature = deserialize_feature(feature[:2])
-        features["functions"][loc[0]]["features"].append((va, feature))
-
-    for feature in doc.get("scopes", {}).get("basic block", []):
-        va, loc = feature[2:]
-        va = int(va, 0x10)
-        loc = [int(lo, 0x10) for lo in loc]
-        feature = deserialize_feature(feature[:2])
-        features["functions"][loc[0]]["basic blocks"][loc[1]]["features"].append((va, feature))
-
-    for feature in doc.get("scopes", {}).get("instruction", []):
-        va, loc = feature[2:]
-        va = int(va, 0x10)
-        loc = [int(lo, 0x10) for lo in loc]
-        feature = deserialize_feature(feature[:2])
-        features["functions"][loc[0]]["basic blocks"][loc[1]]["instructions"][loc[2]]["features"].append((va, feature))
-
-    return capa.features.extractors.base_extractor.NullFeatureExtractor(features)
+    return null.NullFeatureExtractor(
+        base_address=freeze.base_address.to_capa(),
+        global_features=[f.feature.to_capa() for f in freeze.features.global_],
+        file_features=[(f.address.to_capa(), f.feature.to_capa()) for f in freeze.features.file],
+        functions={
+            f.address.to_capa(): null.FunctionFeatures(
+                features=[(fe.address.to_capa(), fe.feature.to_capa()) for fe in f.features],
+                basic_blocks={
+                    bb.address.to_capa(): null.BasicBlockFeatures(
+                        features=[(fe.address.to_capa(), fe.feature.to_capa()) for fe in bb.features],
+                        instructions={
+                            i.address.to_capa(): null.InstructionFeatures(
+                                features=[(fe.address.to_capa(), fe.feature.to_capa()) for fe in i.features]
+                            )
+                            for i in bb.instructions
+                        },
+                    )
+                    for bb in f.basic_blocks
+                },
+            )
+            for f in freeze.features.functions
+        },
+    )
 
 
 MAGIC = "capa0000".encode("ascii")
 
 
-def dump(extractor):
+def dump(extractor: capa.features.extractors.base_extractor.FeatureExtractor) -> bytes:
     """serialize the given extractor to a byte array."""
     return MAGIC + zlib.compress(dumps(extractor).encode("utf-8"))
 
@@ -246,7 +353,7 @@ def is_freeze(buf: bytes) -> bool:
     return buf[: len(MAGIC)] == MAGIC
 
 
-def load(buf):
+def load(buf: bytes) -> capa.features.extractors.base_extractor.FeatureExtractor:
     """deserialize a set of features (as a NullFeatureExtractor) from a byte array."""
     if not is_freeze(buf):
         raise ValueError("missing magic header")

capa/features/insn.py

@@ -1,10 +1,12 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+import abc
+from typing import Union
 
 import capa.render.utils
 from capa.features.common import Feature
@@ -12,25 +14,29 @@ from capa.features.common import Feature
 
 class API(Feature):
     def __init__(self, name: str, description=None):
-        # Downcase library name if given
-        if "." in name:
-            modname, _, impname = name.rpartition(".")
-            name = modname.lower() + "." + impname
-
         super(API, self).__init__(name, description=description)
 
 
 class Number(Feature):
-    def __init__(self, value: int, bitness=None, description=None):
-        super(Number, self).__init__(value, bitness=bitness, description=description)
+    def __init__(self, value: Union[int, float], description=None):
+        super(Number, self).__init__(value, description=description)
 
     def get_value_str(self):
-        return capa.render.utils.hex(self.value)
+        if isinstance(self.value, int):
+            return capa.render.utils.hex(self.value)
+        elif isinstance(self.value, float):
+            return str(self.value)
+        else:
+            raise ValueError("invalid value type")
+
+
+# max recognized structure size (and therefore, offset size)
+MAX_STRUCTURE_SIZE = 0x10000
 
 
 class Offset(Feature):
-    def __init__(self, value: int, bitness=None, description=None):
-        super(Offset, self).__init__(value, bitness=bitness, description=description)
+    def __init__(self, value: int, description=None):
+        super(Offset, self).__init__(value, description=description)
 
     def get_value_str(self):
         return capa.render.utils.hex(self.value)
@@ -39,3 +45,55 @@ class Offset(Feature):
 class Mnemonic(Feature):
     def __init__(self, value: str, description=None):
         super(Mnemonic, self).__init__(value, description=description)
+
+
+# max number of operands to consider for a given instrucion.
+# since we only support Intel and .NET, we can assume this is 3
+# which covers cases up to e.g. "vinserti128 ymm0,ymm0,ymm5,1"
+MAX_OPERAND_COUNT = 4
+MAX_OPERAND_INDEX = MAX_OPERAND_COUNT - 1
+
+
+class _Operand(Feature, abc.ABC):
+    # superclass: don't use directly
+    # subclasses should set self.name and provide the value string formatter
+    def __init__(self, index: int, value: int, description=None):
+        super(_Operand, self).__init__(value, description=description)
+        self.index = index
+
+    def __hash__(self):
+        return hash((self.name, self.value))
+
+    def __eq__(self, other):
+        return super().__eq__(other) and self.index == other.index
+
+    def freeze_serialize(self):
+        return (self.__class__.__name__, (self.index, self.value))
+
+
+class OperandNumber(_Operand):
+    # cached names so we don't do extra string formatting every ctor
+    NAMES = ["operand[%d].number" % i for i in range(MAX_OPERAND_COUNT)]
+
+    # operand[i].number: 0x12
+    def __init__(self, index: int, value: int, description=None):
+        super(OperandNumber, self).__init__(index, value, description=description)
+        self.name = self.NAMES[index]
+
+    def get_value_str(self) -> str:
+        assert isinstance(self.value, int)
+        return capa.render.utils.hex(self.value)
+
+
+class OperandOffset(_Operand):
+    # cached names so we don't do extra string formatting every ctor
+    NAMES = ["operand[%d].offset" % i for i in range(MAX_OPERAND_COUNT)]
+
+    # operand[i].offset: 0x12
+    def __init__(self, index: int, value: int, description=None):
+        super(OperandOffset, self).__init__(index, value, description=description)
+        self.name = self.NAMES[index]
+
+    def get_value_str(self) -> str:
+        assert isinstance(self.value, int)
+        return capa.render.utils.hex(self.value)

capa/helpers.py

@@ -1,12 +1,22 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
-
 import os
+import logging
+from typing import NoReturn
+
+from capa.exceptions import UnsupportedFormatError
+from capa.features.common import FORMAT_SC32, FORMAT_SC64, FORMAT_UNKNOWN
+
+EXTENSIONS_SHELLCODE_32 = ("sc32", "raw32")
+EXTENSIONS_SHELLCODE_64 = ("sc64", "raw64")
+
+
+logger = logging.getLogger("capa")
 
 _hex = hex
 
@@ -30,3 +40,79 @@ def is_runtime_ida():
         return False
     else:
         return True
+
+
+def assert_never(value: NoReturn) -> NoReturn:
+    assert False, f"Unhandled value: {value} ({type(value).__name__})"
+
+
+def get_format_from_extension(sample: str) -> str:
+    if sample.endswith(EXTENSIONS_SHELLCODE_32):
+        return FORMAT_SC32
+    elif sample.endswith(EXTENSIONS_SHELLCODE_64):
+        return FORMAT_SC64
+    return FORMAT_UNKNOWN
+
+
+def get_auto_format(path: str) -> str:
+    format_ = get_format(path)
+    if format_ == FORMAT_UNKNOWN:
+        format_ = get_format_from_extension(path)
+    if format_ == FORMAT_UNKNOWN:
+        raise UnsupportedFormatError()
+    return format_
+
+
+def get_format(sample: str) -> str:
+    # imported locally to avoid import cycle
+    from capa.features.extractors.common import extract_format
+
+    with open(sample, "rb") as f:
+        buf = f.read()
+
+    for feature, _ in extract_format(buf):
+        assert isinstance(feature.value, str)
+        return feature.value
+
+    return FORMAT_UNKNOWN
+
+
+def log_unsupported_format_error():
+    logger.error("-" * 80)
+    logger.error(" Input file does not appear to be a PE or ELF file.")
+    logger.error(" ")
+    logger.error(
+        " capa currently only supports analyzing PE and ELF files (or shellcode, when using --format sc32|sc64)."
+    )
+    logger.error(" If you don't know the input file type, you can try using the `file` utility to guess it.")
+    logger.error("-" * 80)
+
+
+def log_unsupported_os_error():
+    logger.error("-" * 80)
+    logger.error(" Input file does not appear to target a supported OS.")
+    logger.error(" ")
+    logger.error(
+        " capa currently only supports analyzing executables for some operating systems (including Windows and Linux)."
+    )
+    logger.error("-" * 80)
+
+
+def log_unsupported_arch_error():
+    logger.error("-" * 80)
+    logger.error(" Input file does not appear to target a supported architecture.")
+    logger.error(" ")
+    logger.error(" capa currently only supports analyzing x86 (32- and 64-bit).")
+    logger.error("-" * 80)
+
+
+def log_unsupported_runtime_error():
+    logger.error("-" * 80)
+    logger.error(" Unsupported runtime or Python interpreter.")
+    logger.error(" ")
+    logger.error(" capa supports running under Python 3.7 and higher.")
+    logger.error(" ")
+    logger.error(
+        " If you're seeing this message on the command line, please ensure you're running a supported Python version."
+    )
+    logger.error("-" * 80)

capa/ida/helpers.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -21,13 +21,6 @@ import capa.features.common
 
 logger = logging.getLogger("capa")
 
-# IDA version as returned by idaapi.get_kernel_version()
-SUPPORTED_IDA_VERSIONS = (
-    "7.4",
-    "7.5",
-    "7.6",
-)
-
 # file type as returned by idainfo.file_type
 SUPPORTED_FILE_TYPES = (
     idaapi.f_PE,
@@ -45,13 +38,11 @@ def inform_user_ida_ui(message):
 
 
 def is_supported_ida_version():
-    version = idaapi.get_kernel_version()
-    if version not in SUPPORTED_IDA_VERSIONS:
+    version = float(idaapi.get_kernel_version())
+    if version < 7.4 or version >= 8:
         warning_msg = "This plugin does not support your IDA Pro version"
         logger.warning(warning_msg)
-        logger.warning(
-            "Your IDA Pro version is: %s. Supported versions are: %s." % (version, ", ".join(SUPPORTED_IDA_VERSIONS))
-        )
+        logger.warning("Your IDA Pro version is: %s. Supported versions are: IDA >= 7.4 and IDA < 8.0." % version)
         return False
     return True
 
@@ -134,6 +125,12 @@ def collect_metadata():
             "format": idaapi.get_file_type_name(),
             "extractor": "ida",
             "base_address": idaapi.get_imagebase(),
+            "layout": {
+                # this is updated after capabilities have been collected.
+                # will look like:
+                #
+                # "functions": { 0x401000: { "matched_basic_blocks": [ 0x401000, 0x401005, ... ] }, ... }
+            },
         },
         "version": capa.version.__version__,
     }

capa/ida/plugin/README.md

@@ -34,18 +34,18 @@ For more information on the FLARE team's open-source framework, capa, check out
 
 ### Requirements
 
-capa explorer supports Python versions >= 3.6.x and the following IDA Pro versions:
+capa explorer supports Python versions >= 3.7.x and the following IDA Pro versions:
 
 * IDA 7.4
 * IDA 7.5
 * IDA 7.6 (caveat below)
+* IDA 7.7
 
-capa explorer is however limited to the Python versions supported by your IDA installation (which may not include all Python versions >= 3.6.x). Based on our testing the following matrix shows the Python versions supported
+capa explorer is however limited to the Python versions supported by your IDA installation (which may not include all Python versions >= 3.7.x). Based on our testing the following matrix shows the Python versions supported
 by each supported IDA version:
 
 | | IDA 7.4 | IDA 7.5 | IDA 7.6 |
 | --- | --- | --- | --- |
-| Python 3.6.x | Yes | Yes | Yes |
 | Python 3.7.x | Yes | Yes | Yes |
 | Python 3.8.x | Partial (see below) | Yes | Yes |
 | Python 3.9.x | No | Partial (see below) | Yes |
@@ -54,7 +54,7 @@ To use capa explorer with IDA 7.4 and Python 3.8.x you must follow the instructi
 
 To use capa explorer with IDA 7.5 and Python 3.9.x you must follow the instructions provided by hex-rays [here](https://hex-rays.com/blog/python-3-9-support-for-ida-7-5/).
 
-If you encounter issues with your specific setup, please open a new [Issue](https://github.com/fireeye/capa/issues).
+If you encounter issues with your specific setup, please open a new [Issue](https://github.com/mandiant/capa/issues).
 
 #### IDA 7.6 caveat: IDA 7.6sp1 or patch required
 
@@ -86,8 +86,8 @@ You can install capa explorer using the following steps:
     ```
     $ pip install flare-capa
     ```
-3. Download the [standard collection of capa rules](https://github.com/fireeye/capa-rules) (capa explorer needs capa rules to analyze a database)
-4. Copy [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory
+3. Download the [standard collection of capa rules](https://github.com/mandiant/capa-rules) (capa explorer needs capa rules to analyze a database)
+4. Copy [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory
 
 ### Usage
 
@@ -99,7 +99,7 @@ You can install capa explorer using the following steps:
 
 When running capa explorer for the first time you are prompted to select a file directory containing capa rules. The plugin conveniently
 remembers your selection for future runs; you can change this selection and other default settings by clicking `Settings`. We recommend 
-downloading and using the [standard collection of capa rules](https://github.com/fireeye/capa-rules) when getting started with the plugin.
+downloading and using the [standard collection of capa rules](https://github.com/mandiant/capa-rules) when getting started with the plugin.
 
 #### Tips for Program Analysis
 
@@ -125,15 +125,15 @@ downloading and using the [standard collection of capa rules](https://github.com
 ## Development
 
 capa explorer is packaged with capa so you will need to install capa locally for development. You can install capa locally by following the steps outlined in `Method 3: Inspecting the capa source code` of the [capa 
-installation guide](https://github.com/fireeye/capa/blob/master/doc/installation.md#method-3-inspecting-the-capa-source-code). Once installed, copy [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) 
+installation guide](https://github.com/mandiant/capa/blob/master/doc/installation.md#method-3-inspecting-the-capa-source-code). Once installed, copy [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) 
 to your plugins directory to install capa explorer in IDA.
 
 ### Components
 
 capa explorer consists of two main components:
 
-* An [feature extractor](https://github.com/fireeye/capa/tree/master/capa/features/extractors/ida) built on top of IDA's binary analysis engine
-  * This component uses IDAPython to extract [capa features](https://github.com/fireeye/capa-rules/blob/master/doc/format.md#extracted-features) from your IDBs such as strings, 
+* An [feature extractor](https://github.com/mandiant/capa/tree/master/capa/features/extractors/ida) built on top of IDA's binary analysis engine
+  * This component uses IDAPython to extract [capa features](https://github.com/mandiant/capa-rules/blob/master/doc/format.md#extracted-features) from your IDBs such as strings, 
 disassembly, and control flow; these extracted features are used by capa to find feature combinations that result in a rule match
-* An [interactive user interface](https://github.com/fireeye/capa/tree/master/capa/ida/plugin) for displaying and exploring capa rule matches
+* An [interactive user interface](https://github.com/mandiant/capa/tree/master/capa/ida/plugin) for displaying and exploring capa rule matches
   * This component integrates the feature extractor and capa, providing an interactive user interface to dissect rule matches found by capa using features extracted directly from your IDBs

capa/ida/plugin/__init__.py

@@ -1,11 +1,10 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
-
 import logging
 
 import idaapi
@@ -27,8 +26,8 @@ class CapaExplorerPlugin(idaapi.plugin_t):
     wanted_name = PLUGIN_NAME
     wanted_hotkey = "ALT-F5"
     comment = "IDA Pro plugin for the FLARE team's capa tool to identify capabilities in executable files."
-    website = "https://github.com/fireeye/capa"
-    help = "See https://github.com/fireeye/capa/blob/master/doc/usage.md"
+    website = "https://github.com/mandiant/capa"
+    help = "See https://github.com/mandiant/capa/blob/master/doc/usage.md"
     version = ""
     flags = 0
 

capa/ida/plugin/capa_explorer.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/ida/plugin/form.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -12,6 +12,7 @@ import json
 import logging
 import itertools
 import collections
+from typing import Set, Dict
 
 import idaapi
 import ida_kernwin
@@ -26,6 +27,8 @@ import capa.render.json
 import capa.features.common
 import capa.render.result_document
 import capa.features.extractors.ida.extractor
+from capa.engine import FeatureSet
+from capa.features.common import Feature
 from capa.ida.plugin.icon import QICON
 from capa.ida.plugin.view import (
     CapaExplorerQtreeView,
@@ -33,9 +36,11 @@ from capa.ida.plugin.view import (
     CapaExplorerRulgenPreview,
     CapaExplorerRulegenFeatures,
 )
+from capa.features.address import NO_ADDRESS, Address
 from capa.ida.plugin.hooks import CapaExplorerIdaHooks
 from capa.ida.plugin.model import CapaExplorerDataModel
 from capa.ida.plugin.proxy import CapaExplorerRangeProxyModel, CapaExplorerSearchProxyModel
+from capa.features.extractors.base_extractor import FunctionHandle
 
 logger = logging.getLogger(__name__)
 settings = ida_settings.IDASettings("capa")
@@ -66,32 +71,32 @@ def trim_function_name(f, max_length=25):
     return n
 
 
-def find_func_features(f, extractor):
+def find_func_features(fh: FunctionHandle, extractor):
     """ """
-    func_features = collections.defaultdict(set)
-    bb_features = collections.defaultdict(dict)
+    func_features: Dict[Feature, Set] = collections.defaultdict(set)
+    bb_features: Dict[Address, Dict] = collections.defaultdict(dict)
 
-    for (feature, ea) in extractor.extract_function_features(f):
-        func_features[feature].add(ea)
+    for (feature, addr) in extractor.extract_function_features(fh):
+        func_features[feature].add(addr)
 
-    for bb in extractor.get_basic_blocks(f):
+    for bbh in extractor.get_basic_blocks(fh):
         _bb_features = collections.defaultdict(set)
 
-        for (feature, ea) in extractor.extract_basic_block_features(f, bb):
-            _bb_features[feature].add(ea)
-            func_features[feature].add(ea)
+        for (feature, addr) in extractor.extract_basic_block_features(fh, bbh):
+            _bb_features[feature].add(addr)
+            func_features[feature].add(addr)
 
-        for insn in extractor.get_instructions(f, bb):
-            for (feature, ea) in extractor.extract_insn_features(f, bb, insn):
-                _bb_features[feature].add(ea)
-                func_features[feature].add(ea)
+        for insn in extractor.get_instructions(fh, bbh):
+            for (feature, addr) in extractor.extract_insn_features(fh, bbh, insn):
+                _bb_features[feature].add(addr)
+                func_features[feature].add(addr)
 
-        bb_features[int(bb)] = _bb_features
+        bb_features[bbh.address] = _bb_features
 
     return func_features, bb_features
 
 
-def find_func_matches(f, ruleset, func_features, bb_features):
+def find_func_matches(f: FunctionHandle, ruleset, func_features, bb_features):
     """ """
     func_matches = collections.defaultdict(list)
     bb_matches = collections.defaultdict(list)
@@ -108,7 +113,7 @@ def find_func_matches(f, ruleset, func_features, bb_features):
                 func_features[capa.features.common.MatchedRule(name)].add(ea)
 
     # find rule matches for function, function features include rule matches for basic blocks
-    _, matches = capa.engine.match(ruleset.function_rules, func_features, int(f))
+    _, matches = capa.engine.match(ruleset.function_rules, func_features, f.address)
     for (name, res) in matches.items():
         func_matches[name].extend(res)
 
@@ -117,19 +122,19 @@ def find_func_matches(f, ruleset, func_features, bb_features):
 
 def find_file_features(extractor):
     """ """
-    file_features = collections.defaultdict(set)
-    for (feature, ea) in extractor.extract_file_features():
-        if ea:
-            file_features[feature].add(ea)
+    file_features = collections.defaultdict(set)  # type: FeatureSet
+    for (feature, addr) in extractor.extract_file_features():
+        if addr:
+            file_features[feature].add(addr)
         else:
             if feature not in file_features:
                 file_features[feature] = set()
     return file_features
 
 
-def find_file_matches(ruleset, file_features):
+def find_file_matches(ruleset, file_features: FeatureSet):
     """ """
-    _, matches = capa.engine.match(ruleset.file_rules, file_features, 0x0)
+    _, matches = capa.engine.match(ruleset.file_rules, file_features, NO_ADDRESS)
     return matches
 
 
@@ -173,9 +178,9 @@ class CapaExplorerFeatureExtractor(capa.features.extractors.ida.extractor.IdaFea
         super(CapaExplorerFeatureExtractor, self).__init__()
         self.indicator = CapaExplorerProgressIndicator()
 
-    def extract_function_features(self, f):
-        self.indicator.update("function at 0x%X" % f.start_ea)
-        return super(CapaExplorerFeatureExtractor, self).extract_function_features(f)
+    def extract_function_features(self, fh: FunctionHandle):
+        self.indicator.update("function at 0x%X" % fh.inner.start_ea)
+        return super(CapaExplorerFeatureExtractor, self).extract_function_features(fh)
 
 
 class QLineEditClicked(QtWidgets.QLineEdit):
@@ -627,7 +632,7 @@ class CapaExplorerForm(idaapi.PluginForm):
                 path = self.ask_user_directory()
                 if not path:
                     logger.warning(
-                        "You must select a file directory containing capa rules before analysis can be run. The standard collection of capa rules can be downloaded from https://github.com/fireeye/capa-rules."
+                        "You must select a file directory containing capa rules before analysis can be run. The standard collection of capa rules can be downloaded from https://github.com/mandiant/capa-rules."
                     )
                     return False
                 settings.user[CAPA_SETTINGS_RULE_PATH] = path
@@ -694,7 +699,7 @@ class CapaExplorerForm(idaapi.PluginForm):
             )
             logger.error("Failed to load rules from %s (error: %s).", settings.user[CAPA_SETTINGS_RULE_PATH], e)
             logger.error(
-                "Make sure your file directory contains properly formatted capa rules. You can download the standard collection of capa rules from https://github.com/fireeye/capa-rules."
+                "Make sure your file directory contains properly formatted capa rules. You can download the standard collection of capa rules from https://github.com/mandiant/capa-rules."
             )
             settings.user[CAPA_SETTINGS_RULE_PATH] = ""
             return False
@@ -751,6 +756,7 @@ class CapaExplorerForm(idaapi.PluginForm):
                 meta = capa.ida.helpers.collect_metadata()
                 capabilities, counts = capa.main.find_capabilities(self.ruleset_cache, extractor, disable_progress=True)
                 meta["analysis"].update(counts)
+                meta["analysis"]["layout"] = capa.main.compute_layout(self.ruleset_cache, extractor, capabilities)
             except UserCancelledError:
                 logger.info("User cancelled analysis.")
                 return False
@@ -860,7 +866,7 @@ class CapaExplorerForm(idaapi.PluginForm):
             # must use extractor to get function, as capa analysis requires casted object
             extractor = CapaExplorerFeatureExtractor()
         except Exception as e:
-            logger.error("Failed to load IDA feature extractor (error: %s)" % e)
+            logger.error("Failed to load IDA feature extractor (error: %s)", e)
             return False
 
         if ida_kernwin.user_cancelled():
@@ -871,10 +877,10 @@ class CapaExplorerForm(idaapi.PluginForm):
         try:
             f = idaapi.get_func(idaapi.get_screen_ea())
             if f:
-                f = extractor.get_function(f.start_ea)
-                self.rulegen_current_function = f
+                fh: FunctionHandle = extractor.get_function(f.start_ea)
+                self.rulegen_current_function = fh
 
-                func_features, bb_features = find_func_features(f, extractor)
+                func_features, bb_features = find_func_features(fh, extractor)
                 self.rulegen_func_features_cache = collections.defaultdict(set, copy.copy(func_features))
                 self.rulegen_bb_features_cache = collections.defaultdict(dict, copy.copy(bb_features))
 
@@ -885,15 +891,15 @@ class CapaExplorerForm(idaapi.PluginForm):
 
                 try:
                     # add function and bb rule matches to function features, for display purposes
-                    func_matches, bb_matches = find_func_matches(f, self.ruleset_cache, func_features, bb_features)
-                    for (name, res) in itertools.chain(func_matches.items(), bb_matches.items()):
+                    func_matches, bb_matches = find_func_matches(fh, self.ruleset_cache, func_features, bb_features)
+                    for (name, addrs) in itertools.chain(func_matches.items(), bb_matches.items()):
                         rule = self.ruleset_cache[name]
                         if rule.meta.get("capa/subscope-rule"):
                             continue
-                        for (ea, _) in res:
-                            func_features[capa.features.common.MatchedRule(name)].add(ea)
+                        for (addr, _) in addrs:
+                            func_features[capa.features.common.MatchedRule(name)].add(addr)
                 except Exception as e:
-                    logger.error("Failed to match function/basic block rule scope (error: %s)" % e)
+                    logger.error("Failed to match function/basic block rule scope (error: %s)", e)
                     return False
             else:
                 func_features = {}
@@ -901,7 +907,7 @@ class CapaExplorerForm(idaapi.PluginForm):
             logger.info("User cancelled analysis.")
             return False
         except Exception as e:
-            logger.error("Failed to extract function features (error: %s)" % e)
+            logger.error("Failed to extract function features (error: %s)", e)
             return False
 
         if ida_kernwin.user_cancelled():
@@ -911,7 +917,7 @@ class CapaExplorerForm(idaapi.PluginForm):
 
         try:
             file_features = find_file_features(extractor)
-            self.rulegen_file_features_cache = collections.defaultdict(dict, copy.copy(file_features))
+            self.rulegen_file_features_cache = copy.copy(file_features)
 
             if ida_kernwin.user_cancelled():
                 logger.info("User cancelled analysis.")
@@ -920,17 +926,17 @@ class CapaExplorerForm(idaapi.PluginForm):
 
             try:
                 # add file matches to file features, for display purposes
-                for (name, res) in find_file_matches(self.ruleset_cache, file_features).items():
+                for (name, addrs) in find_file_matches(self.ruleset_cache, file_features).items():
                     rule = self.ruleset_cache[name]
                     if rule.meta.get("capa/subscope-rule"):
                         continue
-                    for (ea, _) in res:
-                        file_features[capa.features.common.MatchedRule(name)].add(ea)
+                    for (addr, _) in addrs:
+                        file_features[capa.features.common.MatchedRule(name)].add(addr)
             except Exception as e:
-                logger.error("Failed to match file scope rules (error: %s)" % e)
+                logger.error("Failed to match file scope rules (error: %s)", e)
                 return False
         except Exception as e:
-            logger.error("Failed to extract file features (error: %s)" % e)
+            logger.error("Failed to extract file features (error: %s)", e)
             return False
 
         if ida_kernwin.user_cancelled():
@@ -941,7 +947,7 @@ class CapaExplorerForm(idaapi.PluginForm):
         try:
             # load preview and feature tree
             self.view_rulegen_preview.load_preview_meta(
-                f.start_ea if f else None,
+                fh.address if fh else None,
                 settings.user.get(CAPA_SETTINGS_RULEGEN_AUTHOR, "<insert_author>"),
                 settings.user.get(CAPA_SETTINGS_RULEGEN_SCOPE, "function"),
             )
@@ -952,7 +958,7 @@ class CapaExplorerForm(idaapi.PluginForm):
                 "capa rules directory: %s (%d rules)" % (settings.user[CAPA_SETTINGS_RULE_PATH], len(self.rules_cache))
             )
         except Exception as e:
-            logger.error("Failed to render views (error: %s)" % e)
+            logger.error("Failed to render views (error: %s)", e, exc_info=True)
             return False
 
         return True

capa/ida/plugin/hooks.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/ida/plugin/item.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -216,6 +216,8 @@ class CapaExplorerFunctionItem(CapaExplorerDataItem):
         @param parent: parent node
         @param location: virtual address of function as seen by IDA
         """
+        # location can be an Address now, so need to get the VA
+        location = int(location)
         super(CapaExplorerFunctionItem, self).__init__(
             parent, [self.fmt % idaapi.get_name(location), location_to_hex(location), ""], can_check
         )

capa/ida/plugin/model.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,7 +6,8 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-from collections import deque, defaultdict
+from typing import List
+from collections import deque
 
 import idc
 import idaapi
@@ -545,6 +546,14 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
         @param location: address of feature
         @param display: text to display in plugin UI
         """
+
+        # convert to offset from locations: List[Address]
+        try:
+            location = int(location)
+        except TypeError:
+            # e.g. capa.features.address._NoAddress, global features
+            return
+
         # special handling for characteristic pending type
         if feature["type"] == "characteristic":
             if feature[feature["type"]] in ("embedded pe",):
@@ -581,10 +590,6 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
             "mnemonic",
             "number",
             "offset",
-            "number/x32",
-            "number/x64",
-            "offset/x32",
-            "offset/x64",
         ):
             # display instruction preview
             return CapaExplorerInstructionViewItem(parent, display, location)

capa/ida/plugin/proxy.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/ida/plugin/view.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -18,6 +18,7 @@ import capa.ida.helpers
 import capa.features.common
 import capa.features.basicblock
 from capa.ida.plugin.item import CapaExplorerFunctionItem
+from capa.features.address import NO_ADDRESS
 from capa.ida.plugin.model import CapaExplorerDataModel
 
 MAX_SECTION_SIZE = 750
@@ -1010,6 +1011,8 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
         self.parent_items = {}
 
         def format_address(e):
+            if e == NO_ADDRESS:
+                return ""
             return "%X" % e if e else ""
 
         def format_feature(feature):
@@ -1020,7 +1023,7 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
                 value = '"%s"' % capa.features.common.escape_string(value)
             return "%s(%s)" % (name, value)
 
-        for (feature, eas) in sorted(features.items(), key=lambda k: sorted(k[1])):
+        for (feature, addrs) in sorted(features.items(), key=lambda k: sorted(k[1])):
             if isinstance(feature, capa.features.basicblock.BasicBlock):
                 # filter basic blocks for now, we may want to add these back in some time
                 # in the future
@@ -1032,7 +1035,7 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
 
             # level 1
             if feature not in self.parent_items:
-                if len(eas) > 1:
+                if len(addrs) > 1:
                     self.parent_items[feature] = self.new_parent_node(
                         self.parent_items[type(feature)], (format_feature(feature),), feature=feature
                     )
@@ -1042,18 +1045,18 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
                     )
 
             # level n > 1
-            if len(eas) > 1:
-                for ea in sorted(eas):
+            if len(addrs) > 1:
+                for addr in sorted(addrs):
                     self.new_leaf_node(
-                        self.parent_items[feature], (format_feature(feature), format_address(ea)), feature=feature
+                        self.parent_items[feature], (format_feature(feature), format_address(addr)), feature=feature
                     )
             else:
-                if eas:
-                    ea = eas.pop()
+                if addrs:
+                    addr = addrs.pop()
                 else:
                     # some features may not have an address e.g. "format"
-                    ea = ""
-                for (i, v) in enumerate((format_feature(feature), format_address(ea))):
+                    addr = ""
+                for (i, v) in enumerate((format_feature(feature), format_address(addr))):
                     self.parent_items[feature].setText(i, v)
                 self.parent_items[feature].setData(0, 0x100, feature)
 

capa/main.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -10,7 +10,6 @@ See the License for the specific language governing permissions and limitations
 """
 import os
 import sys
-import gzip
 import time
 import hashlib
 import logging
@@ -29,6 +28,7 @@ import colorama
 from pefile import PEFormatError
 from elftools.common.exceptions import ELFError
 
+import capa.perf
 import capa.rules
 import capa.engine
 import capa.version
@@ -41,19 +41,48 @@ import capa.render.vverbose
 import capa.features.extractors
 import capa.features.extractors.common
 import capa.features.extractors.pefile
+import capa.features.extractors.dnfile_
 import capa.features.extractors.elffile
-from capa.rules import Rule, RuleSet
+import capa.features.extractors.dotnetfile
+from capa.rules import Rule, Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
-from capa.helpers import get_file_taste
-from capa.features.extractors.base_extractor import FunctionHandle, FeatureExtractor
+from capa.helpers import (
+    get_format,
+    get_file_taste,
+    get_auto_format,
+    log_unsupported_os_error,
+    log_unsupported_arch_error,
+    log_unsupported_format_error,
+)
+from capa.exceptions import UnsupportedOSError, UnsupportedArchError, UnsupportedFormatError, UnsupportedRuntimeError
+from capa.features.common import (
+    FORMAT_PE,
+    FORMAT_ELF,
+    FORMAT_AUTO,
+    FORMAT_SC32,
+    FORMAT_SC64,
+    FORMAT_DOTNET,
+    FORMAT_FREEZE,
+)
+from capa.features.address import NO_ADDRESS
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
 
 RULES_PATH_DEFAULT_STRING = "(embedded rules)"
 SIGNATURES_PATH_DEFAULT_STRING = "(embedded signatures)"
 BACKEND_VIV = "vivisect"
 BACKEND_SMDA = "smda"
-EXTENSIONS_SHELLCODE_32 = ("sc32", "raw32")
-EXTENSIONS_SHELLCODE_64 = ("sc64", "raw64")
+BACKEND_DOTNET = "dotnet"
 
+E_MISSING_RULES = -10
+E_MISSING_FILE = -11
+E_INVALID_RULE = -12
+E_CORRUPT_FILE = -13
+E_FILE_LIMITATION = -14
+E_INVALID_SIG = -15
+E_INVALID_FILE_TYPE = -16
+E_INVALID_FILE_ARCH = -17
+E_INVALID_FILE_OS = -18
+E_UNSUPPORTED_IDA_VERSION = -19
 
 logger = logging.getLogger("capa")
 
@@ -75,45 +104,109 @@ def set_vivisect_log_level(level):
     logging.getLogger("envi.codeflow").setLevel(level)
 
 
-def find_function_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, f: FunctionHandle):
-    # contains features from:
-    #  - insns
-    #  - function
+def find_instruction_capabilities(
+    ruleset: RuleSet, extractor: FeatureExtractor, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
+) -> Tuple[FeatureSet, MatchResults]:
+    """
+    find matches for the given rules for the given instruction.
+
+    returns: tuple containing (features for instruction, match results for instruction)
+    """
+    # all features found for the instruction.
+    features = collections.defaultdict(set)  # type: FeatureSet
+
+    for feature, addr in itertools.chain(
+        extractor.extract_insn_features(f, bb, insn), extractor.extract_global_features()
+    ):
+        features[feature].add(addr)
+
+    # matches found at this instruction.
+    _, matches = ruleset.match(Scope.INSTRUCTION, features, insn.address)
+
+    for rule_name, res in matches.items():
+        rule = ruleset[rule_name]
+        for addr, _ in res:
+            capa.engine.index_rule_matches(features, rule, [addr])
+
+    return features, matches
+
+
+def find_basic_block_capabilities(
+    ruleset: RuleSet, extractor: FeatureExtractor, f: FunctionHandle, bb: BBHandle
+) -> Tuple[FeatureSet, MatchResults, MatchResults]:
+    """
+    find matches for the given rules within the given basic block.
+
+    returns: tuple containing (features for basic block, match results for basic block, match results for instructions)
+    """
+    # all features found within this basic block,
+    # includes features found within instructions.
+    features = collections.defaultdict(set)  # type: FeatureSet
+
+    # matches found at the instruction scope.
+    # might be found at different instructions, thats ok.
+    insn_matches = collections.defaultdict(list)  # type: MatchResults
+
+    for insn in extractor.get_instructions(f, bb):
+        ifeatures, imatches = find_instruction_capabilities(ruleset, extractor, f, bb, insn)
+        for feature, vas in ifeatures.items():
+            features[feature].update(vas)
+
+        for rule_name, res in imatches.items():
+            insn_matches[rule_name].extend(res)
+
+    for feature, va in itertools.chain(
+        extractor.extract_basic_block_features(f, bb), extractor.extract_global_features()
+    ):
+        features[feature].add(va)
+
+    # matches found within this basic block.
+    _, matches = ruleset.match(Scope.BASIC_BLOCK, features, bb.address)
+
+    for rule_name, res in matches.items():
+        rule = ruleset[rule_name]
+        for va, _ in res:
+            capa.engine.index_rule_matches(features, rule, [va])
+
+    return features, matches, insn_matches
+
+
+def find_code_capabilities(
+    ruleset: RuleSet, extractor: FeatureExtractor, fh: FunctionHandle
+) -> Tuple[MatchResults, MatchResults, MatchResults, int]:
+    """
+    find matches for the given rules within the given function.
+
+    returns: tuple containing (match results for function, match results for basic blocks, match results for instructions, number of features)
+    """
+    # all features found within this function,
+    # includes features found within basic blocks (and instructions).
     function_features = collections.defaultdict(set)  # type: FeatureSet
+
+    # matches found at the basic block scope.
+    # might be found at different basic blocks, thats ok.
     bb_matches = collections.defaultdict(list)  # type: MatchResults
 
-    for feature, va in itertools.chain(extractor.extract_function_features(f), extractor.extract_global_features()):
+    # matches found at the instruction scope.
+    # might be found at different instructions, thats ok.
+    insn_matches = collections.defaultdict(list)  # type: MatchResults
+
+    for bb in extractor.get_basic_blocks(fh):
+        features, bmatches, imatches = find_basic_block_capabilities(ruleset, extractor, fh, bb)
+        for feature, vas in features.items():
+            function_features[feature].update(vas)
+
+        for rule_name, res in bmatches.items():
+            bb_matches[rule_name].extend(res)
+
+        for rule_name, res in imatches.items():
+            insn_matches[rule_name].extend(res)
+
+    for feature, va in itertools.chain(extractor.extract_function_features(fh), extractor.extract_global_features()):
         function_features[feature].add(va)
 
-    for bb in extractor.get_basic_blocks(f):
-        # contains features from:
-        #  - insns
-        #  - basic blocks
-        bb_features = collections.defaultdict(set)
-
-        for feature, va in itertools.chain(
-            extractor.extract_basic_block_features(f, bb), extractor.extract_global_features()
-        ):
-            bb_features[feature].add(va)
-            function_features[feature].add(va)
-
-        for insn in extractor.get_instructions(f, bb):
-            for feature, va in itertools.chain(
-                extractor.extract_insn_features(f, bb, insn), extractor.extract_global_features()
-            ):
-                bb_features[feature].add(va)
-                function_features[feature].add(va)
-
-        _, matches = capa.engine.match(ruleset.basic_block_rules, bb_features, int(bb))
-
-        for rule_name, res in matches.items():
-            bb_matches[rule_name].extend(res)
-            rule = ruleset[rule_name]
-            for va, _ in res:
-                capa.engine.index_rule_matches(function_features, rule, [va])
-
-    _, function_matches = capa.engine.match(ruleset.function_rules, function_features, int(f))
-    return function_matches, bb_matches, len(function_features)
+    _, function_matches = ruleset.match(Scope.FUNCTION, function_features, fh.address)
+    return function_matches, bb_matches, insn_matches, len(function_features)
 
 
 def find_file_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, function_features: FeatureSet):
@@ -133,13 +226,14 @@ def find_file_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, functi
 
     file_features.update(function_features)
 
-    _, matches = capa.engine.match(ruleset.file_rules, file_features, 0x0)
+    _, matches = ruleset.match(Scope.FILE, file_features, NO_ADDRESS)
     return matches, len(file_features)
 
 
 def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None) -> Tuple[MatchResults, Any]:
     all_function_matches = collections.defaultdict(list)  # type: MatchResults
     all_bb_matches = collections.defaultdict(list)  # type: MatchResults
+    all_insn_matches = collections.defaultdict(list)  # type: MatchResults
 
     meta = {
         "feature_counts": {
@@ -160,31 +254,33 @@ def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_pro
 
     pb = pbar(functions, desc="matching", unit=" functions", postfix="skipped 0 library functions")
     for f in pb:
-        function_address = int(f)
-
-        if extractor.is_library_function(function_address):
-            function_name = extractor.get_function_name(function_address)
-            logger.debug("skipping library function 0x%x (%s)", function_address, function_name)
-            meta["library_functions"][function_address] = function_name
+        if extractor.is_library_function(f.address):
+            function_name = extractor.get_function_name(f.address)
+            logger.debug("skipping library function 0x%x (%s)", f.address, function_name)
+            meta["library_functions"][f.address] = function_name
             n_libs = len(meta["library_functions"])
             percentage = 100 * (n_libs / n_funcs)
             if isinstance(pb, tqdm.tqdm):
                 pb.set_postfix_str("skipped %d library functions (%d%%)" % (n_libs, percentage))
             continue
 
-        function_matches, bb_matches, feature_count = find_function_capabilities(ruleset, extractor, f)
-        meta["feature_counts"]["functions"][function_address] = feature_count
-        logger.debug("analyzed function 0x%x and extracted %d features", function_address, feature_count)
+        function_matches, bb_matches, insn_matches, feature_count = find_code_capabilities(ruleset, extractor, f)
+        meta["feature_counts"]["functions"][f.address] = feature_count
+        logger.debug("analyzed function 0x%x and extracted %d features", f.address, feature_count)
 
         for rule_name, res in function_matches.items():
             all_function_matches[rule_name].extend(res)
         for rule_name, res in bb_matches.items():
             all_bb_matches[rule_name].extend(res)
+        for rule_name, res in insn_matches.items():
+            all_insn_matches[rule_name].extend(res)
 
-    # collection of features that captures the rule matches within function and BB scopes.
+    # collection of features that captures the rule matches within function, BB, and instruction scopes.
     # mapping from feature (matched rule) to set of addresses at which it matched.
     function_and_lower_features: FeatureSet = collections.defaultdict(set)
-    for rule_name, results in itertools.chain(all_function_matches.items(), all_bb_matches.items()):
+    for rule_name, results in itertools.chain(
+        all_function_matches.items(), all_bb_matches.items(), all_insn_matches.items()
+    ):
         locations = set(map(lambda p: p[0], results))
         rule = ruleset[rule_name]
         capa.engine.index_rule_matches(function_and_lower_features, rule, locations)
@@ -198,6 +294,7 @@ def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_pro
             # each rule exists in exactly one scope,
             # so there won't be any overlap among these following MatchResults,
             # and we can merge the dictionaries naively.
+            all_insn_matches.items(),
             all_bb_matches.items(),
             all_function_matches.items(),
             all_file_matches.items(),
@@ -207,6 +304,7 @@ def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_pro
     return matches, meta
 
 
+# TODO move all to helpers?
 def has_rule_with_namespace(rules, capabilities, rule_cat):
     for rule_name in capabilities.keys():
         if rules.rules[rule_name].meta.get("namespace", "").startswith(rule_cat):
@@ -254,17 +352,6 @@ def is_supported_format(sample: str) -> bool:
     return len(list(capa.features.extractors.common.extract_format(taste))) == 1
 
 
-def get_format(sample: str) -> str:
-    with open(sample, "rb") as f:
-        buf = f.read()
-
-    for feature, _ in capa.features.extractors.common.extract_format(buf):
-        assert isinstance(feature.value, str)
-        return feature.value
-
-    return "unknown"
-
-
 def is_supported_arch(sample: str) -> bool:
     with open(sample, "rb") as f:
         buf = f.read()
@@ -301,40 +388,6 @@ def get_os(sample: str) -> str:
     return "unknown"
 
 
-SHELLCODE_BASE = 0x690000
-
-
-def get_shellcode_vw(sample, arch="auto"):
-    """
-    Return shellcode workspace using explicit arch or via auto detect.
-    The workspace is *not* analyzed nor saved. Its up to the caller to do this.
-    Then, they can register FLIRT analyzers or decide not to write to disk.
-    """
-    import viv_utils
-
-    with open(sample, "rb") as f:
-        sample_bytes = f.read()
-
-    if arch == "auto":
-        # choose arch with most functions, idea by Jay G.
-        vw_cands = []
-        for arch in ["i386", "amd64"]:
-            vw_cands.append(
-                viv_utils.getShellcodeWorkspace(
-                    sample_bytes, arch, base=SHELLCODE_BASE, analyze=False, should_save=False
-                )
-            )
-        if not vw_cands:
-            raise ValueError("could not generate vivisect workspace")
-        vw = max(vw_cands, key=lambda vw: len(vw.getFunctions()))
-    else:
-        vw = viv_utils.getShellcodeWorkspace(sample_bytes, arch, base=SHELLCODE_BASE, analyze=False, should_save=False)
-
-    vw.setMeta("StorageName", "%s.viv" % sample)
-
-    return vw
-
-
 def get_meta_str(vw):
     """
     Return workspace meta information string
@@ -346,58 +399,6 @@ def get_meta_str(vw):
     return "%s, number of functions: %d" % (", ".join(meta), len(vw.getFunctions()))
 
 
-def load_flirt_signature(path):
-    # lazy import enables us to only require flirt here and not in IDA, for example
-    import flirt
-
-    if path.endswith(".sig"):
-        with open(path, "rb") as f:
-            with timing("flirt: parsing .sig: " + path):
-                sigs = flirt.parse_sig(f.read())
-
-    elif path.endswith(".pat"):
-        with open(path, "rb") as f:
-            with timing("flirt: parsing .pat: " + path):
-                sigs = flirt.parse_pat(f.read().decode("utf-8").replace("\r\n", "\n"))
-
-    elif path.endswith(".pat.gz"):
-        with gzip.open(path, "rb") as f:
-            with timing("flirt: parsing .pat.gz: " + path):
-                sigs = flirt.parse_pat(f.read().decode("utf-8").replace("\r\n", "\n"))
-
-    else:
-        raise ValueError("unexpect signature file extension: " + path)
-
-    return sigs
-
-
-def register_flirt_signature_analyzers(vw, sigpaths):
-    """
-    args:
-      vw (vivisect.VivWorkspace):
-      sigpaths (List[str]): file system paths of .sig/.pat files
-    """
-    # lazy import enables us to only require flirt here and not in IDA, for example
-    import flirt
-    import viv_utils.flirt
-
-    for sigpath in sigpaths:
-        try:
-            sigs = load_flirt_signature(sigpath)
-        except ValueError as e:
-            logger.warning("could not load %s: %s", sigpath, str(e))
-            continue
-
-        logger.debug("flirt: sig count: %d", len(sigs))
-
-        with timing("flirt: compiling sigs"):
-            matcher = flirt.compile(sigs)
-
-        analyzer = viv_utils.flirt.FlirtFunctionAnalyzer(matcher, sigpath)
-        logger.debug("registering viv function analyzer: %s", repr(analyzer))
-        viv_utils.flirt.addFlirtFunctionAnalyzer(vw, analyzer)
-
-
 def is_running_standalone() -> bool:
     """
     are we running from a PyInstaller'd executable?
@@ -439,27 +440,16 @@ def get_default_signatures() -> List[str]:
     return ret
 
 
-class UnsupportedFormatError(ValueError):
-    pass
-
-
-class UnsupportedArchError(ValueError):
-    pass
-
-
-class UnsupportedOSError(ValueError):
-    pass
-
-
-def get_workspace(path, format, sigpaths):
+def get_workspace(path, format_, sigpaths):
     """
     load the program at the given path into a vivisect workspace using the given format.
     also apply the given FLIRT signatures.
 
     supported formats:
       - pe
-      - sc32
-      - sc64
+      - elf
+      - shellcode 32-bit
+      - shellcode 64-bit
       - auto
 
     this creates and analyzes the workspace; however, it does *not* save the workspace.
@@ -470,23 +460,24 @@ def get_workspace(path, format, sigpaths):
     import viv_utils
 
     logger.debug("generating vivisect workspace for: %s", path)
-    if format == "auto":
+    # TODO should not be auto at this point, anymore
+    if format_ == FORMAT_AUTO:
         if not is_supported_format(path):
             raise UnsupportedFormatError()
 
         # don't analyze, so that we can add our Flirt function analyzer first.
         vw = viv_utils.getWorkspace(path, analyze=False, should_save=False)
-    elif format in {"pe", "elf"}:
+    elif format_ in {FORMAT_PE, FORMAT_ELF}:
         vw = viv_utils.getWorkspace(path, analyze=False, should_save=False)
-    elif format == "sc32":
+    elif format_ == FORMAT_SC32:
         # these are not analyzed nor saved.
-        vw = get_shellcode_vw(path, arch="i386")
-    elif format == "sc64":
-        vw = get_shellcode_vw(path, arch="amd64")
+        vw = viv_utils.getShellcodeWorkspaceFromFile(path, arch="i386", analyze=False)
+    elif format_ == FORMAT_SC64:
+        vw = viv_utils.getShellcodeWorkspaceFromFile(path, arch="amd64", analyze=False)
     else:
-        raise ValueError("unexpected format: " + format)
+        raise ValueError("unexpected format: " + format_)
 
-    register_flirt_signature_analyzers(vw, sigpaths)
+    viv_utils.flirt.register_flirt_signature_analyzers(vw, sigpaths)
 
     vw.analyze()
 
@@ -494,12 +485,9 @@ def get_workspace(path, format, sigpaths):
     return vw
 
 
-class UnsupportedRuntimeError(RuntimeError):
-    pass
-
-
+# TODO get_extractors -> List[FeatureExtractor]?
 def get_extractor(
-    path: str, format: str, backend: str, sigpaths: List[str], should_save_workspace=False, disable_progress=False
+    path: str, format_: str, backend: str, sigpaths: List[str], should_save_workspace=False, disable_progress=False
 ) -> FeatureExtractor:
     """
     raises:
@@ -507,7 +495,7 @@ def get_extractor(
       UnsupportedArchError
       UnsupportedOSError
     """
-    if format not in ("sc32", "sc64"):
+    if format_ not in (FORMAT_SC32, FORMAT_SC64):
         if not is_supported_format(path):
             raise UnsupportedFormatError()
 
@@ -517,6 +505,11 @@ def get_extractor(
         if not is_supported_os(path):
             raise UnsupportedOSError()
 
+    if format_ == FORMAT_DOTNET:
+        import capa.features.extractors.dnfile.extractor
+
+        return capa.features.extractors.dnfile.extractor.DnfileFeatureExtractor(path)
+
     if backend == "smda":
         from smda.SmdaConfig import SmdaConfig
         from smda.Disassembler import Disassembler
@@ -535,7 +528,7 @@ def get_extractor(
         import capa.features.extractors.viv.extractor
 
         with halo.Halo(text="analyzing program", spinner="simpleDots", stream=sys.stderr, enabled=not disable_progress):
-            vw = get_workspace(path, format, sigpaths)
+            vw = get_workspace(path, format_, sigpaths)
 
             if should_save_workspace:
                 logger.debug("saving workspace")
@@ -550,6 +543,22 @@ def get_extractor(
         return capa.features.extractors.viv.extractor.VivisectFeatureExtractor(vw, path)
 
 
+def get_file_extractors(sample: str, format_: str) -> List[FeatureExtractor]:
+    file_extractors: List[FeatureExtractor] = list()
+
+    if format_ == capa.features.extractors.common.FORMAT_PE:
+        file_extractors.append(capa.features.extractors.pefile.PefileFeatureExtractor(sample))
+
+        dnfile_extractor = capa.features.extractors.dnfile_.DnfileFeatureExtractor(sample)
+        if dnfile_extractor.is_dotnet_file():
+            file_extractors.append(dnfile_extractor)
+
+    elif format_ == capa.features.extractors.common.FORMAT_ELF:
+        file_extractors.append(capa.features.extractors.elffile.ElfFeatureExtractor(sample))
+
+    return file_extractors
+
+
 def is_nursery_rule_path(path: str) -> bool:
     """
     The nursery is a spot for rules that have not yet been fully polished.
@@ -563,32 +572,33 @@ def is_nursery_rule_path(path: str) -> bool:
     return "nursery" in path
 
 
-def get_rules(rule_path: str, disable_progress=False) -> List[Rule]:
-    if not os.path.exists(rule_path):
-        raise IOError("rule path %s does not exist or cannot be accessed" % rule_path)
+def get_rules(rule_paths: List[str], disable_progress=False) -> List[Rule]:
+    rule_file_paths = []
+    for rule_path in rule_paths:
+        if not os.path.exists(rule_path):
+            raise IOError("rule path %s does not exist or cannot be accessed" % rule_path)
 
-    rule_paths = []
-    if os.path.isfile(rule_path):
-        rule_paths.append(rule_path)
-    elif os.path.isdir(rule_path):
-        logger.debug("reading rules from directory %s", rule_path)
-        for root, dirs, files in os.walk(rule_path):
-            if ".github" in root:
-                # the .github directory contains CI config in capa-rules
-                # this includes some .yml files
-                # these are not rules
-                continue
-
-            for file in files:
-                if not file.endswith(".yml"):
-                    if not (file.startswith(".git") or file.endswith((".git", ".md", ".txt"))):
-                        # expect to see .git* files, readme.md, format.md, and maybe a .git directory
-                        # other things maybe are rules, but are mis-named.
-                        logger.warning("skipping non-.yml file: %s", file)
+        if os.path.isfile(rule_path):
+            rule_file_paths.append(rule_path)
+        elif os.path.isdir(rule_path):
+            logger.debug("reading rules from directory %s", rule_path)
+            for root, dirs, files in os.walk(rule_path):
+                if ".github" in root:
+                    # the .github directory contains CI config in capa-rules
+                    # this includes some .yml files
+                    # these are not rules
                     continue
 
-                rule_path = os.path.join(root, file)
-                rule_paths.append(rule_path)
+                for file in files:
+                    if not file.endswith(".yml"):
+                        if not (file.startswith(".git") or file.endswith((".git", ".md", ".txt"))):
+                            # expect to see .git* files, readme.md, format.md, and maybe a .git directory
+                            # other things maybe are rules, but are mis-named.
+                            logger.warning("skipping non-.yml file: %s", file)
+                        continue
+
+                    rule_path = os.path.join(root, file)
+                    rule_file_paths.append(rule_path)
 
     rules = []  # type: List[Rule]
 
@@ -598,14 +608,14 @@ def get_rules(rule_path: str, disable_progress=False) -> List[Rule]:
         # to disable progress completely
         pbar = lambda s, *args, **kwargs: s
 
-    for rule_path in pbar(list(rule_paths), desc="loading ", unit=" rules"):
+    for rule_file_path in pbar(list(rule_file_paths), desc="loading ", unit=" rules"):
         try:
-            rule = capa.rules.Rule.from_yaml_file(rule_path)
+            rule = capa.rules.Rule.from_yaml_file(rule_file_path)
         except capa.rules.InvalidRule:
             raise
         else:
-            rule.meta["capa/path"] = rule_path
-            if is_nursery_rule_path(rule_path):
+            rule.meta["capa/path"] = rule_file_path
+            if is_nursery_rule_path(rule_file_path):
                 rule.meta["capa/nursery"] = True
 
             rules.append(rule)
@@ -654,10 +664,10 @@ def collect_metadata(argv, sample_path, rules_path, extractor):
     sha1.update(buf)
     sha256.update(buf)
 
-    if rules_path != RULES_PATH_DEFAULT_STRING:
-        rules_path = os.path.abspath(os.path.normpath(rules_path))
+    if rules_path != [RULES_PATH_DEFAULT_STRING]:
+        rules_path = [os.path.abspath(os.path.normpath(r)) for r in rules_path]
 
-    format = get_format(sample_path)
+    format_ = get_format(sample_path)
     arch = get_arch(sample_path)
     os_ = get_os(sample_path)
 
@@ -672,16 +682,61 @@ def collect_metadata(argv, sample_path, rules_path, extractor):
             "path": os.path.normpath(sample_path),
         },
         "analysis": {
-            "format": format,
+            "format": format_,
             "arch": arch,
             "os": os_,
             "extractor": extractor.__class__.__name__,
             "rules": rules_path,
             "base_address": extractor.get_base_address(),
+            "layout": {
+                # this is updated after capabilities have been collected.
+                # will look like:
+                #
+                # "functions": { 0x401000: { "matched_basic_blocks": [ 0x401000, 0x401005, ... ] }, ... }
+            },
         },
     }
 
 
+def compute_layout(rules, extractor, capabilities):
+    """
+    compute a metadata structure that links basic blocks
+    to the functions in which they're found.
+
+    only collect the basic blocks at which some rule matched.
+    otherwise, we may pollute the json document with
+    a large amount of un-referenced data.
+    """
+    functions_by_bb = {}
+    bbs_by_function = {}
+    for f in extractor.get_functions():
+        bbs_by_function[f.address] = []
+        for bb in extractor.get_basic_blocks(f):
+            functions_by_bb[bb.address] = f.address
+            bbs_by_function[f.address].append(bb.address)
+
+    matched_bbs = set()
+    for rule_name, matches in capabilities.items():
+        rule = rules[rule_name]
+        if rule.meta.get("scope") == capa.rules.BASIC_BLOCK_SCOPE:
+            for (addr, match) in matches:
+                assert addr in functions_by_bb
+                matched_bbs.add(addr)
+
+    layout = {
+        "functions": {
+            f: {
+                "matched_basic_blocks": [bb for bb in bbs if bb in matched_bbs]
+                # this object is open to extension in the future,
+                # such as with the function name, etc.
+            }
+            for f, bbs in bbs_by_function.items()
+        }
+    }
+
+    return layout
+
+
 def install_common_args(parser, wanted=None):
     """
     register a common set of command line arguments for re-use by main & scripts.
@@ -742,19 +797,20 @@ def install_common_args(parser, wanted=None):
 
     if "format" in wanted:
         formats = [
-            ("auto", "(default) detect file type automatically"),
-            ("pe", "Windows PE file"),
-            ("elf", "Executable and Linkable Format"),
-            ("sc32", "32-bit shellcode"),
-            ("sc64", "64-bit shellcode"),
-            ("freeze", "features previously frozen by capa"),
+            (FORMAT_AUTO, "(default) detect file type automatically"),
+            (FORMAT_PE, "Windows PE file"),
+            (FORMAT_DOTNET, ".NET PE file"),
+            (FORMAT_ELF, "Executable and Linkable Format"),
+            (FORMAT_SC32, "32-bit shellcode"),
+            (FORMAT_SC64, "64-bit shellcode"),
+            (FORMAT_FREEZE, "features previously frozen by capa"),
         ]
         format_help = ", ".join(["%s: %s" % (f[0], f[1]) for f in formats])
         parser.add_argument(
             "-f",
             "--format",
             choices=[f[0] for f in formats],
-            default="auto",
+            default=FORMAT_AUTO,
             help="select sample format, %s" % format_help,
         )
 
@@ -773,7 +829,8 @@ def install_common_args(parser, wanted=None):
             "-r",
             "--rules",
             type=str,
-            default=RULES_PATH_DEFAULT_STRING,
+            default=[RULES_PATH_DEFAULT_STRING],
+            action="append",
             help="path to rule file or directory, use embedded rules by default",
         )
 
@@ -814,7 +871,7 @@ def handle_common_args(args):
     # disable vivisect-related logging, it's verbose and not relevant for capa users
     set_vivisect_log_level(logging.CRITICAL)
 
-    # Since Python 3.8 cp65001 is an alias to utf_8, but not for Pyhton < 3.8
+    # Since Python 3.8 cp65001 is an alias to utf_8, but not for Python < 3.8
     # TODO: remove this code when only supporting Python 3.8+
     # https://stackoverflow.com/a/3259271/87207
     import codecs
@@ -835,29 +892,38 @@ def handle_common_args(args):
         raise RuntimeError("unexpected --color value: " + args.color)
 
     if hasattr(args, "rules"):
-        if args.rules == RULES_PATH_DEFAULT_STRING:
+        rules_paths: List[str] = []
+
+        if args.rules == [RULES_PATH_DEFAULT_STRING]:
             logger.debug("-" * 80)
             logger.debug(" Using default embedded rules.")
             logger.debug(" To provide your own rules, use the form `capa.exe -r ./path/to/rules/  /path/to/mal.exe`.")
             logger.debug(" You can see the current default rule set here:")
-            logger.debug("     https://github.com/fireeye/capa-rules")
+            logger.debug("     https://github.com/mandiant/capa-rules")
             logger.debug("-" * 80)
 
-            rules_path = os.path.join(get_default_root(), "rules")
+            default_rule_path = os.path.join(get_default_root(), "rules")
 
-            if not os.path.exists(rules_path):
+            if not os.path.exists(default_rule_path):
                 # when a users installs capa via pip,
                 # this pulls down just the source code - not the default rules.
                 # i'm not sure the default rules should even be written to the library directory,
                 # so in this case, we require the user to use -r to specify the rule directory.
                 logger.error("default embedded rules not found! (maybe you installed capa as a library?)")
                 logger.error("provide your own rule set via the `-r` option.")
-                return -1
-        else:
-            rules_path = args.rules
-            logger.debug("using rules path: %s", rules_path)
+                return E_MISSING_RULES
 
-        args.rules = rules_path
+            rules_paths.append(default_rule_path)
+        else:
+            rules_paths = args.rules
+
+            if RULES_PATH_DEFAULT_STRING in rules_paths:
+                rules_paths.remove(RULES_PATH_DEFAULT_STRING)
+
+            for rule_path in rules_paths:
+                logger.debug("using rules path: %s", rule_path)
+
+        args.rules = rules_paths
 
     if hasattr(args, "signatures"):
         if args.signatures == SIGNATURES_PATH_DEFAULT_STRING:
@@ -877,8 +943,8 @@ def handle_common_args(args):
 
 
 def main(argv=None):
-    if sys.version_info < (3, 6):
-        raise UnsupportedRuntimeError("This version of capa can only be used with Python 3.6+")
+    if sys.version_info < (3, 7):
+        raise UnsupportedRuntimeError("This version of capa can only be used with Python 3.7+")
 
     if argv is None:
         argv = sys.argv[1:]
@@ -888,7 +954,7 @@ def main(argv=None):
         """
         By default, capa uses a default set of embedded rules.
         You can see the rule set here:
-          https://github.com/fireeye/capa-rules
+          https://github.com/mandiant/capa-rules
 
         To provide your own rule set, use the `-r` flag:
           capa  --rules /path/to/rules  suspicious.exe
@@ -918,19 +984,30 @@ def main(argv=None):
     install_common_args(parser, {"sample", "format", "backend", "signatures", "rules", "tag"})
     parser.add_argument("-j", "--json", action="store_true", help="emit JSON instead of text")
     args = parser.parse_args(args=argv)
-    handle_common_args(args)
+    ret = handle_common_args(args)
+    if ret is not None and ret != 0:
+        return ret
 
     try:
-        taste = get_file_taste(args.sample)
+        _ = get_file_taste(args.sample)
     except IOError as e:
         # per our research there's not a programmatic way to render the IOError with non-ASCII filename unless we
         # handle the IOError separately and reach into the args
         logger.error("%s", e.args[0])
-        return -1
+        return E_MISSING_FILE
+
+    format_ = args.format
+    if format_ == FORMAT_AUTO:
+        try:
+            format_ = get_auto_format(args.sample)
+        except UnsupportedFormatError:
+            log_unsupported_format_error()
+            return E_INVALID_FILE_TYPE
 
     try:
         rules = get_rules(args.rules, disable_progress=args.quiet)
         rules = capa.rules.RuleSet(rules)
+
         logger.debug(
             "successfully loaded %s rules",
             # during the load of the RuleSet, we extract subscope statements into their own rules
@@ -946,36 +1023,33 @@ def main(argv=None):
                 logger.debug(" %d. %s", i, r)
     except (IOError, capa.rules.InvalidRule, capa.rules.InvalidRuleSet) as e:
         logger.error("%s", str(e))
-        return -1
+        return E_INVALID_RULE
 
-    file_extractor = None
-    if args.format == "pe" or (args.format == "auto" and taste.startswith(b"MZ")):
-        # these pefile and elffile file feature extractors are pretty light weight: they don't do any code analysis.
-        # so we can fairly quickly determine if the given file has "pure" file-scope rules
-        # that indicate a limitation (like "file is packed based on section names")
-        # and avoid doing a full code analysis on difficult/impossible binaries.
-        try:
-            file_extractor = capa.features.extractors.pefile.PefileFeatureExtractor(args.sample)
-        except PEFormatError as e:
-            logger.error("Input file '%s' is not a valid PE file: %s", args.sample, str(e))
-            return -1
+    # file feature extractors are pretty lightweight: they don't do any code analysis.
+    # so we can fairly quickly determine if the given file has "pure" file-scope rules
+    # that indicate a limitation (like "file is packed based on section names")
+    # and avoid doing a full code analysis on difficult/impossible binaries.
+    #
+    # this pass can inspect multiple file extractors, e.g., dotnet and pe to identify
+    # various limitations
+    try:
+        file_extractors = get_file_extractors(args.sample, format_)
+    except PEFormatError as e:
+        logger.error("Input file '%s' is not a valid PE file: %s", args.sample, str(e))
+        return E_CORRUPT_FILE
+    except (ELFError, OverflowError) as e:
+        logger.error("Input file '%s' is not a valid ELF file: %s", args.sample, str(e))
+        return E_CORRUPT_FILE
 
-    elif args.format == "elf" or (args.format == "auto" and taste.startswith(b"\x7fELF")):
-        try:
-            file_extractor = capa.features.extractors.elffile.ElfFeatureExtractor(args.sample)
-        except (ELFError, OverflowError) as e:
-            logger.error("Input file '%s' is not a valid ELF file: %s", args.sample, str(e))
-            return -1
-
-    if file_extractor:
+    for file_extractor in file_extractors:
         try:
             pure_file_capabilities, _ = find_file_capabilities(rules, file_extractor, {})
         except PEFormatError as e:
             logger.error("Input file '%s' is not a valid PE file: %s", args.sample, str(e))
-            return -1
+            return E_CORRUPT_FILE
         except (ELFError, OverflowError) as e:
             logger.error("Input file '%s' is not a valid ELF file: %s", args.sample, str(e))
-            return -1
+            return E_CORRUPT_FILE
 
         # file limitations that rely on non-file scope won't be detected here.
         # nor on FunctionName features, because pefile doesn't support this.
@@ -984,72 +1058,52 @@ def main(argv=None):
             # do show the output in verbose mode, though.
             if not (args.verbose or args.vverbose or args.json):
                 logger.debug("file limitation short circuit, won't analyze fully.")
-                return -1
+                return E_FILE_LIMITATION
 
-    try:
-        if args.format == "pe" or (args.format == "auto" and taste.startswith(b"MZ")):
-            sig_paths = get_signatures(args.signatures)
-        else:
-            sig_paths = []
-            logger.debug("skipping library code matching: only have PE signatures")
-    except (IOError) as e:
-        logger.error("%s", str(e))
-        return -1
+        if isinstance(file_extractor, capa.features.extractors.dotnetfile.DotnetFileFeatureExtractor):
+            format_ = FORMAT_DOTNET
 
-    if (args.format == "freeze") or (args.format == "auto" and capa.features.freeze.is_freeze(taste)):
-        format = "freeze"
+    if format_ == FORMAT_FREEZE:
         with open(args.sample, "rb") as f:
             extractor = capa.features.freeze.load(f.read())
     else:
-        format = args.format
-        if format == "auto" and args.sample.endswith(EXTENSIONS_SHELLCODE_32):
-            format = "sc32"
-        elif format == "auto" and args.sample.endswith(EXTENSIONS_SHELLCODE_64):
-            format = "sc64"
+        try:
+            if format_ == FORMAT_PE:
+                sig_paths = get_signatures(args.signatures)
+            else:
+                sig_paths = []
+                logger.debug("skipping library code matching: only have native PE signatures")
+        except IOError as e:
+            logger.error("%s", str(e))
+            return E_INVALID_SIG
 
         should_save_workspace = os.environ.get("CAPA_SAVE_WORKSPACE") not in ("0", "no", "NO", "n", None)
 
         try:
             extractor = get_extractor(
-                args.sample, format, args.backend, sig_paths, should_save_workspace, disable_progress=args.quiet
+                args.sample, format_, args.backend, sig_paths, should_save_workspace, disable_progress=args.quiet
             )
         except UnsupportedFormatError:
-            logger.error("-" * 80)
-            logger.error(" Input file does not appear to be a PE or ELF file.")
-            logger.error(" ")
-            logger.error(
-                " capa currently only supports analyzing PE and ELF files (or shellcode, when using --format sc32|sc64)."
-            )
-            logger.error(" If you don't know the input file type, you can try using the `file` utility to guess it.")
-            logger.error("-" * 80)
-            return -1
+            log_unsupported_format_error()
+            return E_INVALID_FILE_TYPE
         except UnsupportedArchError:
-            logger.error("-" * 80)
-            logger.error(" Input file does not appear to target a supported architecture.")
-            logger.error(" ")
-            logger.error(" capa currently only supports analyzing x86 (32- and 64-bit).")
-            logger.error("-" * 80)
-            return -1
+            log_unsupported_arch_error()
+            return E_INVALID_FILE_ARCH
         except UnsupportedOSError:
-            logger.error("-" * 80)
-            logger.error(" Input file does not appear to target a supported OS.")
-            logger.error(" ")
-            logger.error(
-                " capa currently only supports analyzing executables for some operating systems (including Windows and Linux)."
-            )
-            logger.error("-" * 80)
-            return -1
+            log_unsupported_os_error()
+            return E_INVALID_FILE_OS
 
     meta = collect_metadata(argv, args.sample, args.rules, extractor)
 
     capabilities, counts = find_capabilities(rules, extractor, disable_progress=args.quiet)
     meta["analysis"].update(counts)
+    meta["analysis"]["layout"] = compute_layout(rules, extractor, capabilities)
 
     if has_file_limitation(rules, capabilities):
         # bail if capa encountered file limitation e.g. a packed binary
         # do show the output in verbose mode, though.
         if not (args.verbose or args.vverbose or args.json):
-            return -1
+            return E_FILE_LIMITATION
 
     if args.json:
         print(capa.render.json.render(meta, rules, capabilities))
@@ -1076,16 +1130,16 @@ def ida_main():
     logging.getLogger().setLevel(logging.INFO)
 
     if not capa.ida.helpers.is_supported_ida_version():
-        return -1
+        return E_UNSUPPORTED_IDA_VERSION
 
     if not capa.ida.helpers.is_supported_file_type():
-        return -1
+        return E_INVALID_FILE_TYPE
 
     logger.debug("-" * 80)
     logger.debug(" Using default embedded rules.")
     logger.debug(" ")
     logger.debug(" You can see the current default rule set here:")
-    logger.debug("     https://github.com/fireeye/capa-rules")
+    logger.debug("     https://github.com/mandiant/capa-rules")
     logger.debug("-" * 80)
 
     rules_path = os.path.join(get_default_root(), "rules")

capa/optimizer.py

@@ -0,0 +1,70 @@
+import logging
+
+import capa.engine as ceng
+import capa.features.common
+
+logger = logging.getLogger(__name__)
+
+
+def get_node_cost(node):
+    if isinstance(node, (capa.features.common.OS, capa.features.common.Arch, capa.features.common.Format)):
+        # we assume these are the most restrictive features:
+        # authors commonly use them at the start of rules to restrict the category of samples to inspect
+        return 0
+
+    # elif "everything else":
+    #   return 1
+    #
+    # this should be all hash-lookup features.
+    # see below.
+
+    elif isinstance(node, (capa.features.common.Substring, capa.features.common.Regex, capa.features.common.Bytes)):
+        # substring and regex features require a full scan of each string
+        # which we anticipate is more expensive then a hash lookup feature (e.g. mnemonic or count).
+        #
+        # TODO: compute the average cost of these feature relative to hash feature
+        # and adjust the factor accordingly.
+        return 2
+
+    elif isinstance(node, (ceng.Not, ceng.Range)):
+        # the cost of these nodes are defined by the complexity of their single child.
+        return 1 + get_node_cost(node.child)
+
+    elif isinstance(node, (ceng.And, ceng.Or, ceng.Some)):
+        # the cost of these nodes is the full cost of their children
+        # as this is the worst-case scenario.
+        return 1 + sum(map(get_node_cost, node.children))
+
+    else:
+        # this should be all hash-lookup features.
+        # we give this a arbitrary weight of 1.
+        # the only thing more "important" than this is checking OS/Arch/Format.
+        return 1
+
+
+def optimize_statement(statement):
+    # this routine operates in-place
+
+    if isinstance(statement, (ceng.And, ceng.Or, ceng.Some)):
+        # has .children
+        statement.children = sorted(statement.children, key=lambda n: get_node_cost(n))
+        return
+    elif isinstance(statement, (ceng.Not, ceng.Range)):
+        # has .child
+        optimize_statement(statement.child)
+        return
+    else:
+        # appears to be "simple"
+        return
+
+
+def optimize_rule(rule):
+    # this routine operates in-place
+    optimize_statement(rule.statement)
+
+
+def optimize_rules(rules):
+    logger.debug("optimizing %d rules", len(rules))
+    for rule in rules:
+        optimize_rule(rule)
+    return rules

capa/perf.py

@@ -0,0 +1,10 @@
+import collections
+from typing import Dict
+
+# this structure is unstable and may change before the next major release.
+counters: Dict[str, int] = collections.Counter()
+
+
+def reset():
+    global counters
+    counters = collections.Counter()

capa/render/default.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -64,7 +64,7 @@ def find_subrule_matches(doc):
                 matches.add(node["node"]["feature"]["match"])
 
     for rule in rutils.capability_rules(doc):
-        for node in rule["matches"].values():
+        for address, node in rule["matches"]:
             rec(node)
 
     return matches

capa/render/json.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -7,9 +7,9 @@
 # See the License for the specific language governing permissions and limitations under the License.
 import json
 
-import capa.render.result_document
 from capa.rules import RuleSet
 from capa.engine import MatchResults
+from capa.render.result_document import convert_capabilities_to_result_document
 
 
 class CapaJsonObjectEncoder(json.JSONEncoder):
@@ -27,7 +27,7 @@ class CapaJsonObjectEncoder(json.JSONEncoder):
 
 def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
     return json.dumps(
-        capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities),
+        convert_capabilities_to_result_document(meta, rules, capabilities),
         cls=CapaJsonObjectEncoder,
         sort_keys=True,
     )

capa/render/result_document.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,13 +6,70 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 import copy
+from typing import Any, List
 
-import capa.rules
 import capa.engine
 import capa.render.utils
 import capa.features.common
+import capa.features.address
 from capa.rules import RuleSet
 from capa.engine import MatchResults
+from capa.helpers import assert_never
+from capa.features.address import Address
+
+
+def serialize_address(a: Address) -> Any:
+    if isinstance(a, capa.features.address.AbsoluteVirtualAddress):
+        return ("absolute", int(a))
+
+    elif isinstance(a, capa.features.address.RelativeVirtualAddress):
+        return ("relative", int(a))
+
+    elif isinstance(a, capa.features.address.FileOffsetAddress):
+        return ("file", int(a))
+
+    elif isinstance(a, capa.features.address.DNTokenAddress):
+        return ("dn token", a.token)
+
+    elif isinstance(a, capa.features.address.DNTokenOffsetAddress):
+        return ("dn token offset", a.token, a.offset)
+
+    elif a == capa.features.address.NO_ADDRESS or isinstance(a, capa.features.address._NoAddress):
+        return ("no address",)
+
+    elif isinstance(a, capa.features.address.Address) and not issubclass(type(a), capa.features.address.Address):
+        raise ValueError("don't use an Address instance directly")
+
+    elif isinstance(a, capa.features.address.Address):
+        raise ValueError("don't use an Address instance directly")
+
+    else:
+        assert_never(a)
+
+
+def deserialize_address(doc: List[Any]) -> Address:
+    atype = doc[0]
+
+    if atype == "absolute":
+        return capa.features.address.AbsoluteVirtualAddress(doc[1])
+
+    elif atype == "relative":
+        return capa.features.address.RelativeVirtualAddress(doc[1])
+
+    elif atype == "file":
+        return capa.features.address.FileOffsetAddress(doc[1])
+
+    elif atype == "dn token":
+        return capa.features.address.DNTokenAddress(doc[1])
+
+    elif atype == "dn token offset":
+        return capa.features.address.DNTokenOffsetAddress(doc[1], doc[2])
+
+    elif atype == "no address":
+        return capa.features.address.NO_ADDRESS
+
+    else:
+        assert_never(atype)
 
 
 def convert_statement_to_result_document(statement):
@@ -75,7 +132,13 @@ def convert_feature_to_result_document(feature):
     if feature.description:
         result["description"] = feature.description
     if feature.name in ("regex", "substring"):
-        result["matches"] = feature.matches
+        if feature.matches:
+            # regex featur matches are a dict from the capture group to list of location addresses
+            result["matches"] = {k: list(map(serialize_address, vs)) for k, vs in feature.matches.items()}
+        else:
+            # there were no matches
+            pass
+
     return result
 
 
@@ -121,10 +184,10 @@ def convert_match_to_result_document(rules, capabilities, result):
     # so only add `locations` to feature nodes.
     if isinstance(result.statement, capa.features.common.Feature):
         if bool(result.success):
-            doc["locations"] = result.locations
+            doc["locations"] = list(map(serialize_address, result.locations))
     elif isinstance(result.statement, capa.engine.Range):
         if bool(result.success):
-            doc["locations"] = result.locations
+            doc["locations"] = list(map(serialize_address, result.locations))
 
     # if we have a `match` statement, then we're referencing another rule or namespace.
     # this could an external rule (written by a human), or
@@ -165,7 +228,7 @@ def convert_match_to_result_document(rules, capabilities, result):
                     },
                 }
 
-            for location in doc["locations"]:
+            for location in result.locations:
                 doc["children"].append(convert_match_to_result_document(rules, capabilities, rule_matches[location]))
         else:
             # this is a namespace that we're matching
@@ -197,7 +260,7 @@ def convert_match_to_result_document(rules, capabilities, result):
                     # this would be a breaking change and require updates to the renderers.
                     # in the meantime, the above might be sufficient.
                     rule_matches = {address: result for (address, result) in capabilities[rule.name]}
-                    for location in doc["locations"]:
+                    for location in result.locations:
                         # doc[locations] contains all matches for the given namespace.
                         # for example, the feature might be `match: anti-analysis/packer`
                         # which matches against "generic unpacker" and "UPX".
@@ -281,30 +344,27 @@ def convert_capabilities_to_result_document(meta, rules: RuleSet, capabilities:
      to render as text.
 
     see examples of substructures in above routines.
-
-    schema:
-
-    ```json
-    {
-      "meta": {...},
-      "rules: {
-        $rule-name: {
-          "meta": {...copied from rule.meta...},
-          "matches: {
-            $address: {...match details...},
-            ...
-          }
-        },
-        ...
-      }
-    }
-    ```
-
-    Args:
-      meta (Dict[str, Any]):
-      rules (RuleSet):
-      capabilities (Dict[str, List[Tuple[int, Result]]]):
     """
+    meta["analysis"]["base_address"] = serialize_address(meta["analysis"]["base_address"])
+
+    meta["analysis"]["feature_counts"]["functions"] = [
+        {"address": serialize_address(address), "count": count}
+        for address, count in meta["analysis"]["feature_counts"]["functions"].items()
+    ]
+
+    meta["analysis"]["library_functions"] = [
+        {"address": serialize_address(address), "name": name}
+        for address, name in meta["analysis"]["library_functions"].items()
+    ]
+
+    meta["analysis"]["layout"]["functions"] = [
+        {
+            "address": serialize_address(faddr),
+            "matched_basic_blocks": list({"address": serialize_address(bb)} for bb in f["matched_basic_blocks"]),
+        }
+        for faddr, f in meta["analysis"]["layout"]["functions"].items()
+    ]
+
     doc = {
         "meta": meta,
         "rules": {},
@@ -321,9 +381,10 @@ def convert_capabilities_to_result_document(meta, rules: RuleSet, capabilities:
         doc["rules"][rule_name] = {
             "meta": rule_meta,
             "source": rule.definition,
-            "matches": {
-                addr: convert_match_to_result_document(rules, capabilities, match) for (addr, match) in matches
-            },
+            "matches": [
+                [serialize_address(addr), convert_match_to_result_document(rules, capabilities, match)]
+                for (addr, match) in matches
+            ],
         }
 
     return doc

capa/render/utils.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -60,6 +60,8 @@ def capability_rules(doc):
             continue
         if rule["meta"].get("maec/analysis-conclusion-ov"):
             continue
+        if rule["meta"].get("maec/malware-family"):
+            continue
         if rule["meta"].get("maec/malware-category"):
             continue
         if rule["meta"].get("maec/malware-category-ov"):

capa/render/verbose.py

@@ -3,7 +3,7 @@ example::
 
     send data
     namespace    communication
-    author       william.ballenthin@fireeye.com
+    author       william.ballenthin@mandiant.com
     description  all known techniques for sending data to a potential C2 server
     scope        function
     examples     BFB9B5391A13D0AFD787E87AB90F14F5:0x13145D60
@@ -14,7 +14,7 @@ example::
                  0x10003415
                  0x10003797
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -23,12 +23,42 @@ Unless required by applicable law or agreed to in writing, software distributed
 See the License for the specific language governing permissions and limitations under the License.
 """
 import tabulate
+import dnfile.mdtable
 
 import capa.rules
 import capa.render.utils as rutils
 import capa.render.result_document
+import capa.render.result_document as rd
 from capa.rules import RuleSet
 from capa.engine import MatchResults
+from capa.features.address import (
+    NO_ADDRESS,
+    Address,
+    DNTokenAddress,
+    FileOffsetAddress,
+    DNTokenOffsetAddress,
+    AbsoluteVirtualAddress,
+    RelativeVirtualAddress,
+)
+
+
+def format_address(address: Address) -> str:
+    if isinstance(address, AbsoluteVirtualAddress):
+        return rutils.hex(int(address))
+    elif isinstance(address, RelativeVirtualAddress):
+        return f"base address+{rutils.hex(int(address))}"
+    elif isinstance(address, FileOffsetAddress):
+        return f"file+{rutils.hex(int(address))}"
+    elif isinstance(address, DNTokenAddress):
+        return str(address)
+    elif isinstance(address, DNTokenOffsetAddress):
+        name = dnfile.mdtable.ClrMetaDataTableFactory._table_number_map[address.token.table].name
+        rid = address.token.rid
+        return f"{name}[{rid}]+{rutils.hex(int(address.offset))}"
+    elif address == NO_ADDRESS:
+        return "global"
+    else:
+        raise ValueError("unexpected address type")
 
 
 def render_meta(ostream, doc):
@@ -61,16 +91,17 @@ def render_meta(ostream, doc):
         ("format", doc["meta"]["analysis"]["format"]),
         ("arch", doc["meta"]["analysis"]["arch"]),
         ("extractor", doc["meta"]["analysis"]["extractor"]),
-        ("base address", hex(doc["meta"]["analysis"]["base_address"])),
-        ("rules", doc["meta"]["analysis"]["rules"]),
+        ("base address", format_address(rd.deserialize_address(doc["meta"]["analysis"]["base_address"]))),
+        ("rules", "\n".join(doc["meta"]["analysis"]["rules"])),
         ("function count", len(doc["meta"]["analysis"]["feature_counts"]["functions"])),
         ("library function count", len(doc["meta"]["analysis"]["library_functions"])),
         (
             "total feature count",
             doc["meta"]["analysis"]["feature_counts"]["file"]
-            + sum(doc["meta"]["analysis"]["feature_counts"]["functions"].values()),
+            + sum(map(lambda f: f["count"], doc["meta"]["analysis"]["feature_counts"]["functions"])),
         ),
     ]
+
     ostream.writeln(tabulate.tabulate(rows, tablefmt="plain"))
 
 
@@ -107,8 +138,8 @@ def render_rules(ostream, doc):
             rows.append((key, v))
 
         if rule["meta"]["scope"] != capa.rules.FILE_SCOPE:
-            locations = doc["rules"][rule["meta"]["name"]]["matches"].keys()
-            rows.append(("matches", "\n".join(map(rutils.hex, locations))))
+            locations = list(map(lambda m: m[0], doc["rules"][rule["meta"]["name"]]["matches"]))
+            rows.append(("matches", "\n".join(map(lambda d: format_address(rd.deserialize_address(d)), locations))))
 
         ostream.writeln(tabulate.tabulate(rows, tablefmt="plain"))
         ostream.write("\n")
@@ -130,5 +161,5 @@ def render_verbose(doc):
 
 
 def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
-    doc = capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities)
+    doc = rd.convert_capabilities_to_result_document(meta, rules, capabilities)
     return render_verbose(doc)

capa/render/vverbose.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -12,34 +12,42 @@ import capa.rules
 import capa.render.utils as rutils
 import capa.render.verbose
 import capa.features.common
-import capa.render.result_document
+import capa.render.result_document as rd
 from capa.rules import RuleSet
 from capa.engine import MatchResults
 
 
 def render_locations(ostream, match):
+    import capa.render.verbose as v
+
     # its possible to have an empty locations array here,
     # such as when we're in MODE_FAILURE and showing the logic
     # under a `not` statement (which will have no matched locations).
     locations = list(sorted(match.get("locations", [])))
     if len(locations) == 1:
         ostream.write(" @ ")
-        ostream.write(rutils.hex(locations[0]))
+        ostream.write(v.format_address(rd.deserialize_address(locations[0])))
     elif len(locations) > 1:
         ostream.write(" @ ")
         if len(locations) > 4:
             # don't display too many locations, because it becomes very noisy.
             # probably only the first handful of locations will be useful for inspection.
-            ostream.write(", ".join(map(rutils.hex, locations[0:4])))
+            ostream.write(", ".join(map(lambda d: v.format_address(rd.deserialize_address(d)), locations[0:4])))
             ostream.write(", and %d more..." % (len(locations) - 4))
         else:
-            ostream.write(", ".join(map(rutils.hex, locations)))
+            ostream.write(", ".join(map(lambda d: v.format_address(rd.deserialize_address(d)), locations)))
 
 
 def render_statement(ostream, match, statement, indent=0):
     ostream.write("  " * indent)
     if statement["type"] in ("and", "or", "optional", "not", "subscope"):
-        ostream.write(statement["type"])
+        if statement["type"] == "subscope":
+            # emit `basic block:`
+            # rather than `subscope:`
+            ostream.write(statement["subscope"])
+        else:
+            # emit `and:`
+            ostream.write(statement["type"])
         ostream.write(":")
         if statement.get("description"):
             ostream.write(" = %s" % statement["description"])
@@ -193,18 +201,35 @@ def render_rules(ostream, doc):
         ## rules
         check for OutputDebugString error
         namespace  anti-analysis/anti-debugging/debugger-detection
-        author     michael.hunhoff@fireeye.com
+        author     michael.hunhoff@mandiant.com
         scope      function
         mbc        Anti-Behavioral Analysis::Detect Debugger::OutputDebugString
-        examples   Practical Malware Analysis Lab 16-02.exe_:0x401020
         function @ 0x10004706
           and:
             api: kernel32.SetLastError @ 0x100047C2
             api: kernel32.GetLastError @ 0x10004A87
             api: kernel32.OutputDebugString @ 0x10004767, 0x10004787, 0x10004816, 0x10004895
     """
+    functions_by_bb = {}
+    for finfo in doc["meta"]["analysis"]["layout"]["functions"]:
+        faddress = rd.deserialize_address(finfo["address"])
+
+        for bb in finfo["matched_basic_blocks"]:
+            bbaddress = rd.deserialize_address(bb["address"])
+            functions_by_bb[bbaddress] = faddress
+
     had_match = False
-    for rule in rutils.capability_rules(doc):
+
+    for (_, _, rule) in sorted(
+        map(lambda rule: (rule["meta"].get("namespace", ""), rule["meta"]["name"], rule), doc["rules"].values())
+    ):
+        # default scope hides things like lib rules, malware-category rules, etc.
+        # but in vverbose mode, we really want to show everything.
+        #
+        # still ignore subscope rules because they're stitched into the final document.
+        if rule["meta"].get("capa/subscope"):
+            continue
+
         count = len(rule["matches"])
         if count == 1:
             capability = rutils.bold(rule["meta"]["name"])
@@ -219,6 +244,13 @@ def render_rules(ostream, doc):
             if key == "name" or key not in rule["meta"]:
                 continue
 
+            if key == "examples":
+                # I can't think of a reason that an analyst would pivot to the concrete example
+                # directly from the capa output.
+                # the more likely flow is to review the rule and go from there.
+                # so, don't make the output messy by showing the examples.
+                continue
+
             v = rule["meta"][key]
             if not v:
                 continue
@@ -235,19 +267,27 @@ def render_rules(ostream, doc):
         ostream.writeln(tabulate.tabulate(rows, tablefmt="plain"))
 
         if rule["meta"]["scope"] == capa.rules.FILE_SCOPE:
-            matches = list(doc["rules"][rule["meta"]["name"]]["matches"].values())
+            matches = doc["rules"][rule["meta"]["name"]]["matches"]
             if len(matches) != 1:
                 # i think there should only ever be one match per file-scope rule,
                 # because we do the file-scope evaluation a single time.
                 # but i'm not 100% sure if this is/will always be true.
                 # so, lets be explicit about our assumptions and raise an exception if they fail.
                 raise RuntimeError("unexpected file scope match count: %d" % (len(matches)))
-            render_match(ostream, matches[0], indent=0)
+            first_address, first_match = matches[0]
+            render_match(ostream, first_match, indent=0)
         else:
-            for location, match in sorted(doc["rules"][rule["meta"]["name"]]["matches"].items()):
+            for location, match in sorted(doc["rules"][rule["meta"]["name"]]["matches"]):
+                location = rd.deserialize_address(location)
+
                 ostream.write(rule["meta"]["scope"])
                 ostream.write(" @ ")
-                ostream.writeln(rutils.hex(location))
+                ostream.write(capa.render.verbose.format_address(location))
+
+                if rule["meta"]["scope"] == capa.rules.BASIC_BLOCK_SCOPE:
+                    ostream.write(" in function " + capa.render.verbose.format_address(functions_by_bb[location]))
+
+                ostream.write("\n")
                 render_match(ostream, match, indent=1)
         ostream.write("\n")
 
@@ -268,5 +308,5 @@ def render_vverbose(doc):
 
 
 def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
-    doc = capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities)
+    doc = rd.convert_capabilities_to_result_document(meta, rules, capabilities)
     return render_vverbose(doc)

capa/rules.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -12,8 +12,10 @@ import uuid
 import codecs
 import logging
 import binascii
-import functools
 import collections
+from enum import Enum
+
+from capa.helpers import assert_never
 
 try:
     from functools import lru_cache
@@ -22,19 +24,22 @@ except ImportError:
     # https://github.com/python/mypy/issues/1153
     from backports.functools_lru_cache import lru_cache  # type: ignore
 
-from typing import Any, Dict, List, Union, Iterator
+from typing import Any, Set, Dict, List, Tuple, Union, Iterator
 
 import yaml
 import ruamel.yaml
 
+import capa.perf
 import capa.engine as ceng
 import capa.features
+import capa.optimizer
 import capa.features.file
 import capa.features.insn
 import capa.features.common
 import capa.features.basicblock
 from capa.engine import Statement, FeatureSet
 from capa.features.common import MAX_BYTES_FEATURE_SIZE, Feature
+from capa.features.address import Address
 
 logger = logging.getLogger(__name__)
 
@@ -46,6 +51,7 @@ META_KEYS = (
     "rule-category",
     "maec/analysis-conclusion",
     "maec/analysis-conclusion-ov",
+    "maec/malware-family",
     "maec/malware-category",
     "maec/malware-category-ov",
     "author",
@@ -64,12 +70,28 @@ META_KEYS = (
 HIDDEN_META_KEYS = ("capa/nursery", "capa/path")
 
 
-FILE_SCOPE = "file"
-FUNCTION_SCOPE = "function"
-BASIC_BLOCK_SCOPE = "basic block"
+class Scope(str, Enum):
+    FILE = "file"
+    FUNCTION = "function"
+    BASIC_BLOCK = "basic block"
+    INSTRUCTION = "instruction"
 
 
-SUPPORTED_FEATURES = {
+FILE_SCOPE = Scope.FILE.value
+FUNCTION_SCOPE = Scope.FUNCTION.value
+BASIC_BLOCK_SCOPE = Scope.BASIC_BLOCK.value
+INSTRUCTION_SCOPE = Scope.INSTRUCTION.value
+# used only to specify supported features per scope.
+# not used to validate rules.
+GLOBAL_SCOPE = "global"
+
+
+SUPPORTED_FEATURES: Dict[str, Set] = {
+    GLOBAL_SCOPE: {
+        # these will be added to other scopes, see below.
+        capa.features.common.OS,
+        capa.features.common.Arch,
+    },
     FILE_SCOPE: {
         capa.features.common.MatchedRule,
         capa.features.file.Export,
@@ -79,20 +101,24 @@ SUPPORTED_FEATURES = {
         capa.features.common.Characteristic("embedded pe"),
         capa.features.common.String,
         capa.features.common.Format,
-        capa.features.common.OS,
-        capa.features.common.Arch,
+        capa.features.common.Characteristic("mixed mode"),
     },
     FUNCTION_SCOPE: {
-        # plus basic block scope features, see below
+        capa.features.common.MatchedRule,
         capa.features.basicblock.BasicBlock,
         capa.features.common.Characteristic("calls from"),
         capa.features.common.Characteristic("calls to"),
         capa.features.common.Characteristic("loop"),
         capa.features.common.Characteristic("recursive call"),
-        capa.features.common.OS,
-        capa.features.common.Arch,
+        # plus basic block scope features, see below
     },
     BASIC_BLOCK_SCOPE: {
+        capa.features.common.MatchedRule,
+        capa.features.common.Characteristic("tight loop"),
+        capa.features.common.Characteristic("stack string"),
+        # plus instruction scope features, see below
+    },
+    INSTRUCTION_SCOPE: {
         capa.features.common.MatchedRule,
         capa.features.insn.API,
         capa.features.insn.Number,
@@ -100,19 +126,27 @@ SUPPORTED_FEATURES = {
         capa.features.common.Bytes,
         capa.features.insn.Offset,
         capa.features.insn.Mnemonic,
+        capa.features.insn.OperandNumber,
+        capa.features.insn.OperandOffset,
         capa.features.common.Characteristic("nzxor"),
         capa.features.common.Characteristic("peb access"),
         capa.features.common.Characteristic("fs access"),
         capa.features.common.Characteristic("gs access"),
-        capa.features.common.Characteristic("cross section flow"),
-        capa.features.common.Characteristic("tight loop"),
-        capa.features.common.Characteristic("stack string"),
         capa.features.common.Characteristic("indirect call"),
-        capa.features.common.OS,
-        capa.features.common.Arch,
+        capa.features.common.Characteristic("call $+5"),
+        capa.features.common.Characteristic("cross section flow"),
+        capa.features.common.Characteristic("unmanaged call"),
     },
 }
 
+# global scope features are available in all other scopes
+SUPPORTED_FEATURES[INSTRUCTION_SCOPE].update(SUPPORTED_FEATURES[GLOBAL_SCOPE])
+SUPPORTED_FEATURES[BASIC_BLOCK_SCOPE].update(SUPPORTED_FEATURES[GLOBAL_SCOPE])
+SUPPORTED_FEATURES[FUNCTION_SCOPE].update(SUPPORTED_FEATURES[GLOBAL_SCOPE])
+SUPPORTED_FEATURES[FILE_SCOPE].update(SUPPORTED_FEATURES[GLOBAL_SCOPE])
+
+# all instruction scope features are also basic block features
+SUPPORTED_FEATURES[BASIC_BLOCK_SCOPE].update(SUPPORTED_FEATURES[INSTRUCTION_SCOPE])
 # all basic block scope features are also function scope features
 SUPPORTED_FEATURES[FUNCTION_SCOPE].update(SUPPORTED_FEATURES[BASIC_BLOCK_SCOPE])
 
@@ -225,20 +259,15 @@ def parse_feature(key: str):
         return capa.features.common.Bytes
     elif key == "number":
         return capa.features.insn.Number
-    elif key.startswith("number/"):
-        bitness = key.partition("/")[2]
-        # the other handlers here return constructors for features,
-        # and we want to as well,
-        # however, we need to preconfigure one of the arguments (`bitness`).
-        # so, instead we return a partially-applied function that
-        #  provides `bitness` to the feature constructor.
-        # it forwards any other arguments provided to the closure along to the constructor.
-        return functools.partial(capa.features.insn.Number, bitness=bitness)
     elif key == "offset":
         return capa.features.insn.Offset
+    # TODO remove x32/x64 flavor keys once fixed master/rules
+    elif key.startswith("number/"):
+        logger.warning("x32/x64 flavor currently not supported and deprecated")
+        return capa.features.insn.Number
     elif key.startswith("offset/"):
-        bitness = key.partition("/")[2]
-        return functools.partial(capa.features.insn.Offset, bitness=bitness)
+        logger.warning("x32/x64 flavor currently not supported and deprecated")
+        return capa.features.insn.Offset
     elif key == "mnemonic":
         return capa.features.insn.Mnemonic
     elif key == "basic blocks":
@@ -328,7 +357,14 @@ def parse_description(s: Union[str, int, bytes], value_type: str, description=No
             # the string "10" that needs to become the number 10.
             if value_type == "bytes":
                 value = parse_bytes(value)
-            elif value_type in ("number", "offset") or value_type.startswith(("number/", "offset/")):
+            elif (
+                value_type in ("number", "offset")
+                or value_type.startswith(("number/", "offset/"))
+                or (
+                    value_type.startswith("operand[")
+                    and (value_type.endswith("].number") or value_type.endswith("].offset"))
+                )
+            ):
                 try:
                     value = parse_int(value)
                 except ValueError:
@@ -406,7 +442,7 @@ def build_statements(d, scope: str):
         if len(d[key]) != 1:
             raise InvalidRule("subscope must have exactly one child statement")
 
-        return ceng.Subscope(FUNCTION_SCOPE, build_statements(d[key][0], FUNCTION_SCOPE))
+        return ceng.Subscope(FUNCTION_SCOPE, build_statements(d[key][0], FUNCTION_SCOPE), description=description)
 
     elif key == "basic block":
         if scope != FUNCTION_SCOPE:
@@ -415,7 +451,30 @@ def build_statements(d, scope: str):
         if len(d[key]) != 1:
             raise InvalidRule("subscope must have exactly one child statement")
 
-        return ceng.Subscope(BASIC_BLOCK_SCOPE, build_statements(d[key][0], BASIC_BLOCK_SCOPE))
+        return ceng.Subscope(BASIC_BLOCK_SCOPE, build_statements(d[key][0], BASIC_BLOCK_SCOPE), description=description)
+
+    elif key == "instruction":
+        if scope not in (FUNCTION_SCOPE, BASIC_BLOCK_SCOPE):
+            raise InvalidRule("instruction subscope supported only for function and basic block scope")
+
+        if len(d[key]) == 1:
+            statements = build_statements(d[key][0], INSTRUCTION_SCOPE)
+        else:
+            # for instruction subscopes, we support a shorthand in which the top level AND is implied.
+            # the following are equivalent:
+            #
+            #     - instruction:
+            #       - and:
+            #         - arch: i386
+            #         - mnemonic: cmp
+            #
+            #     - instruction:
+            #       - arch: i386
+            #       - mnemonic: cmp
+            #
+            statements = ceng.And([build_statements(dd, INSTRUCTION_SCOPE) for dd in d[key]])
+
+        return ceng.Subscope(INSTRUCTION_SCOPE, statements, description=description)
 
     elif key.startswith("count(") and key.endswith(")"):
         # e.g.:
@@ -472,6 +531,37 @@ def build_statements(d, scope: str):
             raise InvalidRule("unexpected range: %s" % (count))
     elif key == "string" and not isinstance(d[key], str):
         raise InvalidRule("ambiguous string value %s, must be defined as explicit string" % d[key])
+
+    elif key.startswith("operand[") and key.endswith("].number"):
+        index = key[len("operand[") : -len("].number")]
+        try:
+            index = int(index)
+        except ValueError:
+            raise InvalidRule("operand index must be an integer")
+
+        value, description = parse_description(d[key], key, d.get("description"))
+        try:
+            feature = capa.features.insn.OperandNumber(index, value, description=description)
+        except ValueError as e:
+            raise InvalidRule(str(e))
+        ensure_feature_valid_for_scope(scope, feature)
+        return feature
+
+    elif key.startswith("operand[") and key.endswith("].offset"):
+        index = key[len("operand[") : -len("].offset")]
+        try:
+            index = int(index)
+        except ValueError:
+            raise InvalidRule("operand index must be an integer")
+
+        value, description = parse_description(d[key], key, d.get("description"))
+        try:
+            feature = capa.features.insn.OperandOffset(index, value, description=description)
+        except ValueError as e:
+            raise InvalidRule(str(e))
+        ensure_feature_valid_for_scope(scope, feature)
+        return feature
+
     elif (
         (key == "os" and d[key] not in capa.features.common.VALID_OS)
         or (key == "format" and d[key] not in capa.features.common.VALID_FORMAT)
@@ -619,8 +709,10 @@ class Rule:
         for new_rule in self._extract_subscope_rules_rec(self.statement):
             yield new_rule
 
-    def evaluate(self, features: FeatureSet):
-        return self.statement.evaluate(features)
+    def evaluate(self, features: FeatureSet, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.rule"] += 1
+        return self.statement.evaluate(features, short_circuit=short_circuit)
 
     @classmethod
     def from_dict(cls, d, definition):
@@ -958,12 +1050,27 @@ class RuleSet:
         if len(rules) == 0:
             raise InvalidRuleSet("no rules selected")
 
+        rules = capa.optimizer.optimize_rules(rules)
+
         self.file_rules = self._get_rules_for_scope(rules, FILE_SCOPE)
         self.function_rules = self._get_rules_for_scope(rules, FUNCTION_SCOPE)
         self.basic_block_rules = self._get_rules_for_scope(rules, BASIC_BLOCK_SCOPE)
+        self.instruction_rules = self._get_rules_for_scope(rules, INSTRUCTION_SCOPE)
         self.rules = {rule.name: rule for rule in rules}
         self.rules_by_namespace = index_rules_by_namespace(rules)
 
+        # unstable
+        (self._easy_file_rules_by_feature, self._hard_file_rules) = self._index_rules_by_feature(self.file_rules)
+        (self._easy_function_rules_by_feature, self._hard_function_rules) = self._index_rules_by_feature(
+            self.function_rules
+        )
+        (self._easy_basic_block_rules_by_feature, self._hard_basic_block_rules) = self._index_rules_by_feature(
+            self.basic_block_rules
+        )
+        (self._easy_instruction_rules_by_feature, self._hard_instruction_rules) = self._index_rules_by_feature(
+            self.instruction_rules
+        )
+
     def __len__(self):
         return len(self.rules)
 
@@ -973,6 +1080,156 @@ class RuleSet:
     def __contains__(self, rulename):
         return rulename in self.rules
 
+    @staticmethod
+    def _index_rules_by_feature(rules) -> Tuple[Dict[Feature, Set[str]], List[str]]:
+        """
+        split the given rules into two structures:
+          - "easy rules" are indexed by feature,
+            such that you can quickly find the rules that contain a given feature.
+          - "hard rules" are those that contain substring/regex/bytes features or match statements.
+            these continue to be ordered topologically.
+
+        a rule evaluator can use the "easy rule" index to restrict the
+        candidate rules that might match a given set of features.
+
+        at this time, a rule evaluator can't do anything special with
+        the "hard rules". it must still do a full top-down match of each
+        rule, in topological order.
+
+        this does not index global features, because these are not selective, and
+        won't be used as the sole feature used to match.
+        """
+
+        # we'll do a couple phases:
+        #
+        #  1. recursively visit all nodes in all rules,
+        #    a. indexing all features
+        #    b. recording the types of features found per rule
+        #  2. compute the easy and hard rule sets
+        #  3. remove hard rules from the rules-by-feature index
+        #  4. construct the topologically ordered list of hard rules
+        rules_with_easy_features: Set[str] = set()
+        rules_with_hard_features: Set[str] = set()
+        rules_by_feature: Dict[Feature, Set[str]] = collections.defaultdict(set)
+
+        def rec(rule_name: str, node: Union[Feature, Statement]):
+            """
+            walk through a rule's logic tree, indexing the easy and hard rules,
+            and the features referenced by easy rules.
+            """
+            if isinstance(
+                node,
+                (
+                    # these are the "hard features"
+                    # substring: scanning feature
+                    capa.features.common.Substring,
+                    # regex: scanning feature
+                    capa.features.common.Regex,
+                    # bytes: scanning feature
+                    capa.features.common.Bytes,
+                    # match: dependency on another rule,
+                    # which we have to evaluate first,
+                    # and is therefore tricky.
+                    capa.features.common.MatchedRule,
+                ),
+            ):
+                # hard feature: requires scan or match lookup
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, capa.features.common.Feature):
+                if capa.features.common.is_global_feature(node):
+                    # we don't want to index global features
+                    # because they're not very selective.
+                    #
+                    # they're global, so if they match at one location in a file,
+                    # they'll match at every location in a file.
+                    # so thats not helpful to decide how to downselect.
+                    #
+                    # and, a global rule will never be the sole selector in a rule.
+                    # TODO: probably want a lint for this.
+                    pass
+                else:
+                    # easy feature: hash lookup
+                    rules_with_easy_features.add(rule_name)
+                    rules_by_feature[node].add(rule_name)
+            elif isinstance(node, (ceng.Not)):
+                # `not:` statements are tricky to deal with.
+                #
+                # first, features found under a `not:` should not be indexed,
+                # because they're not wanted to be found.
+                # second, `not:` can be nested under another `not:`, or two, etc.
+                # third, `not:` at the root or directly under an `or:`
+                # means the rule will match against *anything* not specified there,
+                # which is a difficult set of things to compute and index.
+                #
+                # so, if a rule has a `not:` statement, its hard.
+                # as of writing, this is an uncommon statement, with only 6 instances in 740 rules.
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, (ceng.Some)) and node.count == 0:
+                # `optional:` and `0 or more:` are tricky to deal with.
+                #
+                # when a subtree is optional, it may match, but not matching
+                # doesn't have any impact either.
+                # now, our rule authors *should* not put this under `or:`
+                # and this is checked by the linter,
+                # but this could still happen (e.g. private rule set without linting)
+                # and would be hard to trace down.
+                #
+                # so better to be safe than sorry and consider this a hard case.
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, (ceng.Range)) and node.min == 0:
+                # `count(foo): 0 or more` are tricky to deal with.
+                # because the min is 0,
+                # this subtree *can* match just about any feature
+                # (except the given one)
+                # which is a difficult set of things to compute and index.
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, (ceng.Range)):
+                rec(rule_name, node.child)
+            elif isinstance(node, (ceng.And, ceng.Or, ceng.Some)):
+                for child in node.children:
+                    rec(rule_name, child)
+            elif isinstance(node, ceng.Statement):
+                # unhandled type of statement.
+                # this should only happen if a new subtype of `Statement`
+                # has since been added to capa.
+                #
+                # ideally, we'd like to use mypy for exhaustiveness checking
+                # for all the subtypes of `Statement`.
+                # but, as far as i can tell, mypy does not support this type
+                # of checking.
+                #
+                # in a way, this makes some intuitive sense:
+                # the set of subtypes of type A is unbounded,
+                # because any user might come along and create a new subtype B,
+                # so mypy can't reason about this set of types.
+                assert False, f"Unhandled value: {node} ({type(node).__name__})"
+            else:
+                # programming error
+                assert_never(node)
+
+        for rule in rules:
+            rule_name = rule.meta["name"]
+            root = rule.statement
+            rec(rule_name, root)
+
+        # if a rule has a hard feature,
+        # dont consider it easy, and therefore,
+        # don't index any of its features.
+        #
+        # otherwise, its an easy rule, and index its features
+        for rules_with_feature in rules_by_feature.values():
+            rules_with_feature.difference_update(rules_with_hard_features)
+        easy_rules_by_feature = rules_by_feature
+
+        # `rules` is already topologically ordered,
+        # so extract our hard set into the topological ordering.
+        hard_rules = []
+        for rule in rules:
+            if rule.meta["name"] in rules_with_hard_features:
+                hard_rules.append(rule.meta["name"])
+
+        return (easy_rules_by_feature, hard_rules)
+
     @staticmethod
     def _get_rules_for_scope(rules, scope):
         """
@@ -1035,3 +1292,69 @@ class RuleSet:
                     rules_filtered.update(set(capa.rules.get_rules_and_dependencies(rules, rule.name)))
                     break
         return RuleSet(list(rules_filtered))
+
+    def match(self, scope: Scope, features: FeatureSet, addr: Address) -> Tuple[FeatureSet, ceng.MatchResults]:
+        """
+        match rules from this ruleset at the given scope against the given features.
+
+        this routine should act just like `capa.engine.match`,
+        except that it may be more performant.
+        """
+        easy_rules_by_feature = {}
+        if scope is Scope.FILE:
+            easy_rules_by_feature = self._easy_file_rules_by_feature
+            hard_rule_names = self._hard_file_rules
+        elif scope is Scope.FUNCTION:
+            easy_rules_by_feature = self._easy_function_rules_by_feature
+            hard_rule_names = self._hard_function_rules
+        elif scope is Scope.BASIC_BLOCK:
+            easy_rules_by_feature = self._easy_basic_block_rules_by_feature
+            hard_rule_names = self._hard_basic_block_rules
+        elif scope is Scope.INSTRUCTION:
+            easy_rules_by_feature = self._easy_instruction_rules_by_feature
+            hard_rule_names = self._hard_instruction_rules
+        else:
+            assert_never(scope)
+
+        candidate_rule_names = set()
+        for feature in features:
+            easy_rule_names = easy_rules_by_feature.get(feature)
+            if easy_rule_names:
+                candidate_rule_names.update(easy_rule_names)
+
+        # first, match against the set of rules that have at least one
+        # feature shared with our feature set.
+        candidate_rules = [self.rules[name] for name in candidate_rule_names]
+        features2, easy_matches = ceng.match(candidate_rules, features, addr)
+
+        # note that we've stored the updated feature set in `features2`.
+        # this contains a superset of the features in `features`;
+        # it contains additional features for any easy rule matches.
+        # we'll pass this feature set to hard rule matching, since one
+        # of those rules might rely on an easy rule match.
+        #
+        # the updated feature set from hard matching will go into `features3`.
+        # this is a superset of `features2` is a superset of `features`.
+        # ultimately, this is what we'll return to the caller.
+        #
+        # in each case, we could have assigned the updated feature set back to `features`,
+        # but this is slightly more explicit how we're tracking the data.
+
+        # now, match against (topologically ordered) list of rules
+        # that we can't really make any guesses about.
+        # these are rules with hard features, like substring/regex/bytes and match statements.
+        hard_rules = [self.rules[name] for name in hard_rule_names]
+        features3, hard_matches = ceng.match(hard_rules, features2, addr)
+
+        # note that above, we probably are skipping matching a bunch of
+        # rules that definitely would never hit.
+        # specifically, "easy rules" that don't share any features with
+        # feature set.
+
+        # MatchResults doesn't technically have an .update() method
+        # but a dict does.
+        matches = {}  # type: ignore
+        matches.update(easy_matches)
+        matches.update(hard_matches)
+
+        return (features3, matches)

capa/version.py

@@ -1 +1 @@
-__version__ = "3.0.0"
+__version__ = "3.2.0"

doc/installation.md

@@ -1,8 +1,8 @@
 # Installation
-You can install capa in a few different ways. First, if you simply want to use capa, just download the [standalone binary](https://github.com/fireeye/capa/releases). If you want to use capa as a Python library, you can install the package directly from GitHub using `pip`. If you'd like to contribute patches or features to capa, you can work with a local copy of the source code.
+You can install capa in a few different ways. First, if you simply want to use capa, just download the [standalone binary](https://github.com/mandiant/capa/releases). If you want to use capa as a Python library, you can install the package directly from GitHub using `pip`. If you'd like to contribute patches or features to capa, you can work with a local copy of the source code.
 
 ## Method 1: Standalone installation
-If you simply want to use capa, use the standalone binaries we host on GitHub: https://github.com/fireeye/capa/releases. These binary executable files contain all the source code, Python interpreter, and associated resources needed to make capa run. This means you can run it without any installation! Just invoke the file using your terminal shell to see the help documentation.
+If you simply want to use capa, use the standalone binaries we host on GitHub: https://github.com/mandiant/capa/releases. These binary executable files contain all the source code, Python interpreter, and associated resources needed to make capa run. This means you can run it without any installation! Just invoke the file using your terminal shell to see the help documentation.
 
 We use PyInstaller to create these packages.
 
@@ -26,8 +26,8 @@ To install capa as a Python library use `pip` to fetch the `flare-capa` module.
 
 #### *Note*:
 This method is appropriate for integrating capa in an existing project.
-This technique doesn't pull the default rule set, so you should check it out separately from [capa-rules](https://github.com/fireeye/capa-rules/) and pass the directory to the entrypoint using `-r` or set the rules path in the IDA Pro plugin.
-This technique also doesn't set up the default library identification [signatures](https://github.com/fireeye/capa/tree/master/sigs). You can pass the signature directory using the `-s` argument.
+This technique doesn't pull the default rule set, so you should check it out separately from [capa-rules](https://github.com/mandiant/capa-rules/) and pass the directory to the entrypoint using `-r` or set the rules path in the IDA Pro plugin.
+This technique also doesn't set up the default library identification [signatures](https://github.com/mandiant/capa/tree/master/sigs). You can pass the signature directory using the `-s` argument.
 For example, to run capa with both a rule path and a signature path:
 
     capa   -r /path/to/capa-rules   -s /path/to/capa-sigs  suspicious.exe
@@ -44,16 +44,16 @@ If you'd like to review and modify the capa source code, you'll need to check it
 
 ### 1. Check out source code
 Next, clone the capa git repository.
-We use submodules to separate [code](https://github.com/fireeye/capa), [rules](https://github.com/fireeye/capa-rules), and [test data](https://github.com/fireeye/capa-testfiles).
+We use submodules to separate [code](https://github.com/mandiant/capa), [rules](https://github.com/mandiant/capa-rules), and [test data](https://github.com/mandiant/capa-testfiles).
 To clone everything use the `--recurse-submodules` option:
 - CAUTION: The capa testfiles repository contains many malware samples. If you pull down everything using this method, you may want to install to a directory that won't trigger your anti-virus software.
-- `$ git clone --recurse-submodules https://github.com/fireeye/capa.git /local/path/to/src` (HTTPS)
-- `$ git clone --recurse-submodules git@github.com:fireeye/capa.git /local/path/to/src` (SSH)
+- `$ git clone --recurse-submodules https://github.com/mandiant/capa.git /local/path/to/src` (HTTPS)
+- `$ git clone --recurse-submodules git@github.com:mandiant/capa.git /local/path/to/src` (SSH)
 
 To only get the source code and our provided rules (common), follow these steps:
 - clone repository
-  - `$ git clone https://github.com/fireeye/capa.git /local/path/to/src` (HTTPS)
-  - `$ git clone git@github.com:fireeye/capa.git /local/path/to/src` (SSH)
+  - `$ git clone https://github.com/mandiant/capa.git /local/path/to/src` (HTTPS)
+  - `$ git clone git@github.com:mandiant/capa.git /local/path/to/src` (SSH)
 - `$ cd /local/path/to/src`
 - `$ git submodule update --init rules`
 
@@ -87,7 +87,7 @@ We use the following tools to ensure consistent code style and formatting:
   - [black](https://github.com/psf/black) code formatter, with `-l 120`
   - [isort 5](https://pypi.org/project/isort/) code formatter, with `--profile black --length-sort --line-width 120`
   - [dos2unix](https://linux.die.net/man/1/dos2unix) for UNIX-style LF newlines
-  - [capafmt](https://github.com/fireeye/capa/blob/master/scripts/capafmt.py) rule formatter
+  - [capafmt](https://github.com/mandiant/capa/blob/master/scripts/capafmt.py) rule formatter
 
 To install these development dependencies, run:
 

doc/limitations.md

@@ -46,6 +46,6 @@ We need more practical use cases and test samples to justify the additional work
 
 
 # ATT&CK, MAEC, MBC, and other capability tagging
-capa uses namespaces to group capabilities (see https://github.com/fireeye/capa-rules/tree/master#namespace-organization).
+capa uses namespaces to group capabilities (see https://github.com/mandiant/capa-rules/tree/master#namespace-organization).
 
-The `rule.meta` field also supports `att&ck`, `mbc`, and `maec` fields to associate rules with the respective taxonomy (see https://github.com/fireeye/capa-rules/blob/master/doc/format.md#meta-block).
+The `rule.meta` field also supports `att&ck`, `mbc`, and `maec` fields to associate rules with the respective taxonomy (see https://github.com/mandiant/capa-rules/blob/master/doc/format.md#meta-block).

doc/release.md

@@ -1,13 +1,13 @@
 # Release checklist
 
-- [ ] Ensure all [milestoned issues/PRs](https://github.com/fireeye/capa/milestones) are addressed, or reassign to a new milestone.
-- [ ] Add the `dont merge` label to all PRs that are close to be ready to merge (or merge them if they are ready) in [capa](https://github.com/fireeye/capa/pulls) and [capa-rules](https://github.com/fireeye/capa-rules/pulls).
-- [ ] Ensure the [CI workflow succeeds in master](https://github.com/fireeye/capa/actions/workflows/tests.yml?query=branch%3Amaster).
+- [ ] Ensure all [milestoned issues/PRs](https://github.com/mandiant/capa/milestones) are addressed, or reassign to a new milestone.
+- [ ] Add the `dont merge` label to all PRs that are close to be ready to merge (or merge them if they are ready) in [capa](https://github.com/mandiant/capa/pulls) and [capa-rules](https://github.com/mandiant/capa-rules/pulls).
+- [ ] Ensure the [CI workflow succeeds in master](https://github.com/mandiant/capa/actions/workflows/tests.yml?query=branch%3Amaster).
 - [ ] Ensure that `python scripts/lint.py rules/ --thorough` succeeds (only `missing examples`  offenses are allowed in the nursery).
 - [ ] Review changes
-  - capa https://github.com/fireeye/capa/compare/\<last-release\>...master
-  - capa-rules https://github.com/fireeye/capa-rules/compare/\<last-release>\...master
-- [ ] Update [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md)
+  - capa https://github.com/mandiant/capa/compare/\<last-release\>...master
+  - capa-rules https://github.com/mandiant/capa-rules/compare/\<last-release>\...master
+- [ ] Update [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md)
   - Do not forget to add a nice introduction thanking contributors
   - Remember that we need a major release if we introduce breaking changes
   - Sections: see template below
@@ -31,13 +31,13 @@
     ### Development
 
     ### Raw diffs
-    - [capa <release>...master](https://github.com/fireeye/capa/compare/<release>...master)
-    - [capa-rules <release>...master](https://github.com/fireeye/capa-rules/compare/<release>...master)
+    - [capa <release>...master](https://github.com/mandiant/capa/compare/<release>...master)
+    - [capa-rules <release>...master](https://github.com/mandiant/capa-rules/compare/<release>...master)
     ```
-- [ ] Update [capa/version.py](https://github.com/fireeye/capa/blob/master/capa/version.py)
-- [ ] Create a PR with the updated [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md) and [capa/version.py](https://github.com/fireeye/capa/blob/master/capa/version.py). Copy this checklist in the PR description.
-- [ ] After PR review, merge the PR and [create the release in GH](https://github.com/fireeye/capa/releases/new) using text from the [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md).
-- [ ] Verify GH actions [upload artifacts](https://github.com/fireeye/capa/releases), [publish to PyPI](https://pypi.org/project/flare-capa) and [create a tag in capa rules](https://github.com/fireeye/capa-rules/tags) upon completion.
+- [ ] Update [capa/version.py](https://github.com/mandiant/capa/blob/master/capa/version.py)
+- [ ] Create a PR with the updated [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md) and [capa/version.py](https://github.com/mandiant/capa/blob/master/capa/version.py). Copy this checklist in the PR description.
+- [ ] After PR review, merge the PR and [create the release in GH](https://github.com/mandiant/capa/releases/new) using text from the [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md).
+- [ ] Verify GH actions [upload artifacts](https://github.com/mandiant/capa/releases), [publish to PyPI](https://pypi.org/project/flare-capa) and [create a tag in capa rules](https://github.com/mandiant/capa-rules/tags) upon completion.
 - [ ] [Spread the word](https://twitter.com)
 - [ ] Update internal service
 

scripts/bulk-process.py

@@ -47,7 +47,7 @@ usage:
                             parallelism factor
       --no-mp               disable subprocesses
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -129,6 +129,7 @@ def get_capa_results(args):
     meta = capa.main.collect_metadata("", path, "", extractor)
     capabilities, counts = capa.main.find_capabilities(rules, extractor, disable_progress=True)
     meta["analysis"].update(counts)
+    meta["analysis"]["layout"] = capa.main.compute_layout(rules, extractor, capabilities)
 
     return {
         "path": path,

scripts/capa2yara.py

@@ -21,7 +21,7 @@ optional arguments:
   -t TAG, --tag TAG     filter on rule meta field values
 
 
-Copyright (C) 2020, 2021 Arnim Rupp (@ruppde) and FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020, 2021 Arnim Rupp (@ruppde) and Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -43,7 +43,7 @@ import capa.rules
 import capa.engine
 import capa.features
 import capa.features.insn
-from capa.features.common import BITNESS_X32, BITNESS_X64, String
+from capa.features.common import String
 
 logger = logging.getLogger("capa2yara")
 
@@ -57,8 +57,8 @@ var_names = ["".join(letters) for letters in itertools.product(string.ascii_lowe
 unsupported = ["characteristic", "mnemonic", "offset", "subscope", "Range"]
 # TODO shorten this list, possible stuff:
 # - 2 or more strings: e.g.
-# -- https://github.com/fireeye/capa-rules/blob/master/collection/file-managers/gather-direct-ftp-information.yml
-# -- https://github.com/fireeye/capa-rules/blob/master/collection/browser/gather-firefox-profile-information.yml
+# -- https://github.com/mandiant/capa-rules/blob/master/collection/file-managers/gather-direct-ftp-information.yml
+# -- https://github.com/mandiant/capa-rules/blob/master/collection/browser/gather-firefox-profile-information.yml
 # - count(string    (1 rule: /executable/subfile/pe/contain-an-embedded-pe-file.yml)
 # - count(match( could be done by creating the referenced rule a 2nd time with the condition, that it hits x times (only 1 rule: ./anti-analysis/anti-disasm/contain-anti-disasm-techniques.yml)
 # - it would be technically possible to get the "basic blocks" working, but the rules contain mostly other non supported statements in there => not worth the effort.
@@ -104,7 +104,7 @@ def check_feature(statement, rulename):
 def get_rule_url(path):
     path = re.sub(r"\.\.\/", "", path)
     path = re.sub(r"capa-rules\/", "", path)
-    return "https://github.com/fireeye/capa-rules/blob/master/" + path
+    return "https://github.com/mandiant/capa-rules/blob/master/" + path
 
 
 def convert_capa_number_to_yara_bytes(number):
@@ -176,7 +176,7 @@ def convert_rule(rule, rulename, cround, depth):
         elif s_type == "api" or s_type == "import":
             # TODO: is it possible in YARA to make a difference between api & import?
 
-            # https://github.com/fireeye/capa-rules/blob/master/doc/format.md#api
+            # https://github.com/mandiant/capa-rules/blob/master/doc/format.md#api
             api = kid.value
             logger.info("doing api: " + repr(api))
 
@@ -208,7 +208,7 @@ def convert_rule(rule, rulename, cround, depth):
             yara_condition += '\tpe.exports("' + export + '") '
 
         elif s_type == "section":
-            # https://github.com/fireeye/capa-rules/blob/master/doc/format.md#section
+            # https://github.com/mandiant/capa-rules/blob/master/doc/format.md#section
             section = kid.value
             logger.info("doing section: " + repr(section))
 
@@ -220,7 +220,7 @@ def convert_rule(rule, rulename, cround, depth):
             )
 
         elif s_type == "match":
-            # https://github.com/fireeye/capa-rules/blob/master/doc/format.md#matching-prior-rule-matches-and-namespaces
+            # https://github.com/mandiant/capa-rules/blob/master/doc/format.md#matching-prior-rule-matches-and-namespaces
             match = kid.value
             logger.info("doing match: " + repr(match))
 
@@ -311,7 +311,7 @@ def convert_rule(rule, rulename, cround, depth):
 
         return yara_strings, yara_condition
 
-    ############################## end def do_statement
+    # end: def do_statement
 
     yara_strings_list = []
     yara_condition_list = []
@@ -390,7 +390,9 @@ def convert_rule(rule, rulename, cround, depth):
                 logger.info("kid coming: " + repr(kid.name))
                 # logger.info("grandchildren: " + repr(kid.children))
 
-                ##### here we go into RECURSION  ##################################################################################
+                #
+                # here we go into RECURSION
+                #
                 yara_strings_sub, yara_condition_sub, rule_comment_sub, incomplete_sub = convert_rule(
                     kid, rulename, cround, depth
                 )
@@ -496,9 +498,7 @@ def convert_rule(rule, rulename, cround, depth):
 
         yara_condition = "\n\t" + yara_condition_list[0]
 
-    logger.info(
-        f"################# end of convert_rule() #strings: {len(yara_strings_list)} #conditions: {len(yara_condition_list)}"
-    )
+    logger.info(f"# end of convert_rule() #strings: {len(yara_strings_list)} #conditions: {len(yara_condition_list)}")
     logger.info(f"strings: {yara_strings} conditions: {yara_condition}")
 
     return yara_strings, yara_condition, rule_comment, incomplete
@@ -617,7 +617,7 @@ def convert_rules(rules, namespaces, cround):
 
                             # examples in capa can contain the same hash several times with different offset, so check if it's already there:
                             # (keeping the offset might be interessting for some but breaks yara-ci for checking of the final rules
-                            if not value in seen_hashes:
+                            if value not in seen_hashes:
                                 yara_meta += "\t" + meta_name + ' = "' + value + '"\n'
                                 seen_hashes.append(value)
 
@@ -703,7 +703,7 @@ def main(argv=None):
     logging.getLogger("capa2yara").setLevel(level)
 
     try:
-        rules = capa.main.get_rules(args.rules, disable_progress=True)
+        rules = capa.main.get_rules([args.rules], disable_progress=True)
         namespaces = capa.rules.index_rules_by_namespace(list(rules))
         rules = capa.rules.RuleSet(rules)
         logger.info("successfully loaded %s rules (including subscope rules which will be ignored)", len(rules))
@@ -717,7 +717,7 @@ def main(argv=None):
         return -1
 
     output_yar(
-        "// Rules from FireEye's https://github.com/fireeye/capa-rules converted to YARA using https://github.com/fireeye/capa/blob/master/scripts/capa2yara.py by Arnim Rupp"
+        "// Rules from Mandiant's https://github.com/mandiant/capa-rules converted to YARA using https://github.com/mandiant/capa/blob/master/scripts/capa2yara.py by Arnim Rupp"
     )
     output_yar(
         "// Beware: These are less rules than capa (because not all fit into YARA, stats at EOF) and is less precise because e.g. capas function scopes are applied to the whole file"

scripts/capa_as_library.py

@@ -17,7 +17,7 @@ from capa.engine import *
 RULES_PATH = "/tmp/capa/rules/"
 
 # load rules from disk
-rules = capa.rules.RuleSet(capa.main.get_rules(RULES_PATH, disable_progress=True))
+rules = capa.rules.RuleSet(capa.main.get_rules([RULES_PATH], disable_progress=True))
 
 # == Render ddictionary helpers
 def render_meta(doc, ostream):
@@ -163,7 +163,6 @@ def render_dictionary(doc):
 
 # ==== render dictionary helpers
 def capa_details(file_path, output_format="dictionary"):
-
     # extract features and find capabilities
     extractor = capa.main.get_extractor(file_path, "auto", capa.main.BACKEND_VIV, [], False, disable_progress=True)
     capabilities, counts = capa.main.find_capabilities(rules, extractor, disable_progress=True)
@@ -171,6 +170,7 @@ def capa_details(file_path, output_format="dictionary"):
     # collect metadata (used only to make rendering more complete)
     meta = capa.main.collect_metadata("", file_path, RULES_PATH, extractor)
     meta["analysis"].update(counts)
+    meta["analysis"]["layout"] = capa.main.compute_layout(rules, extractor, capabilities)
 
     capa_output = False
     if output_format == "dictionary":

scripts/capafmt.py

@@ -6,7 +6,7 @@ Usage:
 
    $ python capafmt.py -i foo.yml
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt

scripts/ci.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

scripts/detect-elf-os.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python2
 """
-Copyright (C) 2021 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt

scripts/import-to-bn.py

@@ -20,7 +20,7 @@ Adapted for Binary Ninja by @psifertex
 This script will verify that the report matches the workspace.
 Check the log window for any errors, and/or the summary of changes.
 
-Derived from: https://github.com/fireeye/capa/blob/master/scripts/import-to-ida.py
+Derived from: https://github.com/mandiant/capa/blob/master/scripts/import-to-ida.py
 """
 import os
 import json

scripts/import-to-ida.py

@@ -20,7 +20,7 @@ and then select the existing capa report from the file system.
 This script will verify that the report matches the workspace.
 Check the output window for any errors, and/or the summary of changes.
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt

scripts/lint.py

@@ -5,7 +5,7 @@ Usage:
 
    $ python scripts/lint.py rules/
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -15,14 +15,15 @@ See the License for the specific language governing permissions and limitations
 """
 import gc
 import os
+import re
 import sys
+import json
 import time
 import string
 import difflib
 import hashlib
 import inspect
 import logging
-import os.path
 import pathlib
 import argparse
 import itertools
@@ -40,6 +41,7 @@ import tqdm.contrib.logging
 import capa.main
 import capa.rules
 import capa.engine
+import capa.helpers
 import capa.features.insn
 import capa.features.common
 from capa.rules import Rule, RuleSet
@@ -160,10 +162,10 @@ class MissingScope(Lint):
 
 class InvalidScope(Lint):
     name = "invalid scope"
-    recommendation = "Use only file, function, or basic block rule scopes"
+    recommendation = "Use only file, function, basic block, or instruction rule scopes"
 
     def check_rule(self, ctx: Context, rule: Rule):
-        return rule.meta.get("scope") not in ("file", "function", "basic block")
+        return rule.meta.get("scope") not in ("file", "function", "basic block", "instruction")
 
 
 class MissingAuthor(Lint):
@@ -221,6 +223,61 @@ class ExampleFileDNE(Lint):
         return not found
 
 
+class InvalidAttckOrMbcTechnique(Lint):
+    name = "att&ck/mbc entry is malformed or does not exist"
+    recommendation = """
+    The att&ck and mbc fields must respect the following format:
+    <Tactic/Objective>::<Technique/Behavior> [<ID>]
+    OR
+    <Tactic/Objective>::<Technique/Behavior>::<Subtechnique/Method> [<ID.SubID>]
+    """
+
+    def __init__(self):
+        super(InvalidAttckOrMbcTechnique, self).__init__()
+
+        try:
+            with open(f"{os.path.dirname(__file__)}/linter-data.json", "rb") as fd:
+                self.data = json.load(fd)
+            self.enabled_frameworks = self.data.keys()
+        except BaseException:
+            # If linter-data.json is not present, or if an error happen
+            # we log an error and lint nothing.
+            logger.warning(
+                "Could not load 'scripts/linter-data.json'. The att&ck and mbc information will not be linted."
+            )
+            self.enabled_frameworks = []
+
+        # This regex matches the format defined in the recommendation attribute
+        self.reg = re.compile(r"^([\w\s-]+)::(.+) \[([A-Za-z0-9.]+)\]$")
+
+    def _entry_check(self, framework, category, entry, eid):
+        if category not in self.data[framework].keys():
+            self.recommendation = f'Unknown category: "{category}"'
+            return True
+        if eid not in self.data[framework][category].keys():
+            self.recommendation = f"Unknown entry ID: {eid}"
+            return True
+        if self.data[framework][category][eid] != entry:
+            self.recommendation = (
+                f'{eid} should be associated to entry "{self.data[framework][category][eid]}" instead of "{entry}"'
+            )
+            return True
+        return False
+
+    def check_rule(self, ctx: Context, rule: Rule):
+        for framework in self.enabled_frameworks:
+            if framework in rule.meta.keys():
+                for r in rule.meta[framework]:
+                    m = self.reg.match(r)
+                    if m is None:
+                        return True
+
+                    args = m.group(1, 2, 3)
+                    if self._entry_check(framework, *args):
+                        return True
+        return False
+
+
 DEFAULT_SIGNATURES = capa.main.get_default_signatures()
 
 
@@ -230,9 +287,16 @@ def get_sample_capabilities(ctx: Context, path: Path) -> Set[str]:
         logger.debug("found cached results: %s: %d capabilities", nice_path, len(ctx.capabilities_by_sample[path]))
         return ctx.capabilities_by_sample[path]
 
+    if nice_path.endswith(capa.helpers.EXTENSIONS_SHELLCODE_32):
+        format_ = "sc32"
+    elif nice_path.endswith(capa.helpers.EXTENSIONS_SHELLCODE_64):
+        format_ = "sc64"
+    else:
+        format_ = "auto"
+
     logger.debug("analyzing sample: %s", nice_path)
     extractor = capa.main.get_extractor(
-        nice_path, "auto", capa.main.BACKEND_VIV, DEFAULT_SIGNATURES, False, disable_progress=True
+        nice_path, format_, capa.main.BACKEND_VIV, DEFAULT_SIGNATURES, False, disable_progress=True
     )
 
     capabilities, _ = capa.main.find_capabilities(ctx.rules, extractor, disable_progress=True)
@@ -332,6 +396,52 @@ class OrStatementWithAlwaysTrueChild(Lint):
         return self.violation
 
 
+class NotNotUnderAnd(Lint):
+    name = "rule contains a `not` statement that's not found under an `and` statement"
+    recommendation = "clarify the rule logic and ensure `not` is always found under `and`"
+    violation = False
+
+    def check_rule(self, ctx: Context, rule: Rule):
+        self.violation = False
+
+        def rec(statement):
+            if isinstance(statement, capa.engine.Statement):
+                if not isinstance(statement, capa.engine.And):
+                    for child in statement.get_children():
+                        if isinstance(child, capa.engine.Not):
+                            self.violation = True
+
+                for child in statement.get_children():
+                    rec(child)
+
+        rec(rule.statement)
+
+        return self.violation
+
+
+class OptionalNotUnderAnd(Lint):
+    name = "rule contains an `optional` or `0 or more` statement that's not found under an `and` statement"
+    recommendation = "clarify the rule logic and ensure `optional` and `0 or more` is always found under `and`"
+    violation = False
+
+    def check_rule(self, ctx: Context, rule: Rule):
+        self.violation = False
+
+        def rec(statement):
+            if isinstance(statement, capa.engine.Statement):
+                if not isinstance(statement, capa.engine.And):
+                    for child in statement.get_children():
+                        if isinstance(child, capa.engine.Some) and child.count == 0:
+                            self.violation = True
+
+                for child in statement.get_children():
+                    rec(child)
+
+        rec(rule.statement)
+
+        return self.violation
+
+
 class UnusualMetaField(Lint):
     name = "unusual meta field"
     recommendation = "Remove the meta field"
@@ -594,6 +704,7 @@ META_LINTS = (
     UnusualMetaField(),
     LibRuleNotInLibDirectory(),
     LibRuleHasNamespace(),
+    InvalidAttckOrMbcTechnique(),
 )
 
 
@@ -653,6 +764,8 @@ LOGIC_LINTS = (
     DoesntMatchExample(),
     StatementWithSingleChildStatement(),
     OrStatementWithAlwaysTrueChild(),
+    NotNotUnderAnd(),
+    OptionalNotUnderAnd(),
 )
 
 
@@ -850,7 +963,7 @@ def main(argv=None):
 
     parser = argparse.ArgumentParser(description="Lint capa rules.")
     capa.main.install_common_args(parser, wanted={"tag"})
-    parser.add_argument("rules", type=str, help="Path to rules")
+    parser.add_argument("rules", type=str, action="append", help="Path to rules")
     parser.add_argument("--samples", type=str, default=samples_path, help="Path to samples")
     parser.add_argument(
         "--thorough",

scripts/match-function-id.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-Copyright (C) 2021 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -105,7 +105,7 @@ def main(argv=None):
 
     analyzers = []
     for sigpath in args.signatures:
-        sigs = capa.main.load_flirt_signature(sigpath)
+        sigs = viv_utils.flirt.load_flirt_signature(sigpath)
 
         with capa.main.timing("flirt: compiling sigs"):
             matcher = flirt.compile(sigs)

scripts/profile-time.py

@@ -0,0 +1,150 @@
+"""
+Invoke capa multiple times and record profiling informations.
+Use the --number and --repeat options to change the number of iterations.
+By default, the script will emit a markdown table with a label pulled from git.
+
+Note: you can run this script against pre-generated .frz files to reduce the startup time.
+
+usage:
+
+    usage: profile-time.py [--number NUMBER] [--repeat REPEAT] [--label LABEL] sample
+
+    Profile capa performance
+
+    positional arguments:
+      sample                path to sample to analyze
+
+    optional arguments:
+      --number NUMBER       batch size of profile collection
+      --repeat REPEAT       batch count of profile collection
+      --label LABEL         description of the profile collection
+
+example:
+
+    $ python profile-time.py ./tests/data/kernel32.dll_.frz --number 1 --repeat 2
+
+    | label                                | count(evaluations)   | avg(time)   | min(time)   | max(time)   |
+    |--------------------------------------|----------------------|-------------|-------------|-------------|
+    | 18c30e4 main: remove perf debug msgs | 66,561,622           | 132.13s     | 125.14s     | 139.12s     |
+
+      ^^^ --label or git hash               
+"""
+import sys
+import timeit
+import logging
+import argparse
+import subprocess
+
+import tqdm
+import tabulate
+
+import capa.main
+import capa.perf
+import capa.rules
+import capa.engine
+import capa.helpers
+import capa.features
+import capa.features.common
+import capa.features.freeze
+
+logger = logging.getLogger("capa.profile")
+
+
+def main(argv=None):
+    if argv is None:
+        argv = sys.argv[1:]
+
+    label = subprocess.run(
+        "git show --pretty=oneline --abbrev-commit | head -n 1", shell=True, capture_output=True, text=True
+    ).stdout.strip()
+    is_dirty = (
+        subprocess.run(
+            "git status | grep 'modified: ' | grep -v 'rules' | grep -v 'tests/data'",
+            shell=True,
+            capture_output=True,
+            text=True,
+        ).stdout
+        != ""
+    )
+
+    if is_dirty:
+        label += " (dirty)"
+
+    parser = argparse.ArgumentParser(description="Profile capa performance")
+    capa.main.install_common_args(parser, wanted={"format", "sample", "signatures", "rules"})
+
+    parser.add_argument("--number", type=int, default=3, help="batch size of profile collection")
+    parser.add_argument("--repeat", type=int, default=30, help="batch count of profile collection")
+    parser.add_argument("--label", type=str, default=label, help="description of the profile collection")
+
+    args = parser.parse_args(args=argv)
+    capa.main.handle_common_args(args)
+
+    try:
+        taste = capa.helpers.get_file_taste(args.sample)
+    except IOError as e:
+        logger.error("%s", str(e))
+        return -1
+
+    try:
+        with capa.main.timing("load rules"):
+            rules = capa.rules.RuleSet(capa.main.get_rules(args.rules, disable_progress=True))
+    except (IOError) as e:
+        logger.error("%s", str(e))
+        return -1
+
+    try:
+        sig_paths = capa.main.get_signatures(args.signatures)
+    except (IOError) as e:
+        logger.error("%s", str(e))
+        return -1
+
+    if (args.format == "freeze") or (args.format == "auto" and capa.features.freeze.is_freeze(taste)):
+        with open(args.sample, "rb") as f:
+            extractor = capa.features.freeze.load(f.read())
+    else:
+        extractor = capa.main.get_extractor(
+            args.sample, args.format, capa.main.BACKEND_VIV, sig_paths, should_save_workspace=False
+        )
+
+    with tqdm.tqdm(total=args.number * args.repeat) as pbar:
+
+        def do_iteration():
+            capa.perf.reset()
+            capa.main.find_capabilities(rules, extractor, disable_progress=True)
+            pbar.update(1)
+
+        samples = timeit.repeat(do_iteration, number=args.number, repeat=args.repeat)
+
+    logger.debug("perf: find capabilities: min: %0.2fs" % (min(samples) / float(args.number)))
+    logger.debug("perf: find capabilities: avg: %0.2fs" % (sum(samples) / float(args.repeat) / float(args.number)))
+    logger.debug("perf: find capabilities: max: %0.2fs" % (max(samples) / float(args.number)))
+
+    for (counter, count) in capa.perf.counters.most_common():
+        logger.debug("perf: counter: {:}: {:,}".format(counter, count))
+
+    print(
+        tabulate.tabulate(
+            [
+                (
+                    args.label,
+                    "{:,}".format(capa.perf.counters["evaluate.feature"]),
+                    # python documentation indicates that min(samples) should be preferred,
+                    # so lets put that first.
+                    #
+                    # https://docs.python.org/3/library/timeit.html#timeit.Timer.repeat
+                    "%0.2fs" % (min(samples) / float(args.number)),
+                    "%0.2fs" % (sum(samples) / float(args.repeat) / float(args.number)),
+                    "%0.2fs" % (max(samples) / float(args.number)),
+                )
+            ],
+            headers=["label", "count(evaluations)", "min(time)", "avg(time)", "max(time)"],
+            tablefmt="github",
+        )
+    )
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/setup-hooks.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt