Merge pull request #863 from mandiant/v3.1.0

version: v3.1.0

.github/CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 First off, thanks for taking the time to contribute!
 
-The following is a set of guidelines for contributing to capa and its packages, which are hosted in the [FireEye Organization](https://github.com/fireeye) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
+The following is a set of guidelines for contributing to capa and its packages, which are hosted in the [Mandiant Organization](https://github.com/mandiant) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
 
 #### Table Of Contents
 
@@ -32,9 +32,9 @@ This project and everyone participating in it is governed by the [Capa Code of C
 ### Capa and its repositories
 
 We host the capa project as three Github repositories:
-  - [capa](https://github.com/fireeye/capa)
-  - [capa-rules](https://github.com/fireeye/capa-rules)
-  - [capa-testfiles](https://github.com/fireeye/capa-testfiles)
+  - [capa](https://github.com/mandiant/capa)
+  - [capa-rules](https://github.com/mandiant/capa-rules)
+  - [capa-testfiles](https://github.com/mandiant/capa-testfiles)
   
 The command line tools, logic engine, and other Python source code are found in the `capa` repository.
 This is the repository to fork when you want to enhance the features, performance, or user interface of capa.
@@ -54,7 +54,7 @@ These are files you'll need in order to run the linter (in `--thorough` mode) an
 ### Design Decisions
 
 When we make a significant decision in how we maintain the project and what we can or cannot support,
- we will document it in the [capa issues tracker](https://github.com/fireeye/capa/issues).
+ we will document it in the [capa issues tracker](https://github.com/mandiant/capa/issues).
 This is the best place review our discussions about what/how/why we do things in the project.
 If you have a question, check to see if it is documented there.
 If it is *not* documented there, or you can't find an answer, please open a issue.
@@ -78,7 +78,7 @@ Fill out [the required template](./ISSUE_TEMPLATE/bug_report.md),
 #### Before Submitting A Bug Report
 
 * **Determine [which repository the problem should be reported in](#capa-and-its-repositories)**.
-* **Perform a [cursory search](https://github.com/fireeye/capa/issues?q=is%3Aissue)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one.
+* **Perform a [cursory search](https://github.com/mandiant/capa/issues?q=is%3Aissue)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one.
 
 #### How Do I Submit A (Good) Bug Report?
 
@@ -101,7 +101,7 @@ Explain the problem and include additional details to help maintainers reproduce
 Provide more context by answering these questions:
 
 * **Did the problem start happening recently** (e.g. after updating to a new version of capa) or was this always a problem?
-* If the problem started happening recently, **can you reproduce the problem in an older version of capa?** What's the most recent version in which the problem doesn't happen? You can download older versions of capa from [the releases page](https://github.com/fireeye/capa/releases).
+* If the problem started happening recently, **can you reproduce the problem in an older version of capa?** What's the most recent version in which the problem doesn't happen? You can download older versions of capa from [the releases page](https://github.com/mandiant/capa/releases).
 * **Can you reliably reproduce the issue?** If not, provide details about how often the problem happens and under which conditions it normally happens.
 * If the problem is related to working with files (e.g. opening and editing files), **does the problem happen for all files and projects or only some?** Does the problem happen only when working with local or remote files (e.g. on network drives), with files of a specific type (e.g. only JavaScript or Python files), with large files or files with very long lines, or with files in a specific encoding? Is there anything else special about the files you are using?
 
@@ -119,7 +119,7 @@ Before creating enhancement suggestions, please check [this list](#before-submit
 #### Before Submitting An Enhancement Suggestion
 
 * **Determine [which repository the enhancement should be suggested in](#capa-and-its-repositories).**
-* **Perform a [cursory search](https://github.com/fireeye/capa/issues?q=is%3Aissue)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
+* **Perform a [cursory search](https://github.com/mandiant/capa/issues?q=is%3Aissue)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
 
 #### How Do I Submit A (Good) Enhancement Suggestion?
 
@@ -138,15 +138,15 @@ Enhancement suggestions are tracked as [GitHub issues](https://guides.github.com
 
 Unsure where to begin contributing to capa? You can start by looking through these `good-first-issue` and `rule-idea` issues:
 
-* [good-first-issue](https://github.com/fireeye/capa/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - issues which should only require a few lines of code, and a test or two.
-* [rule-idea](https://github.com/fireeye/capa-rules/issues?q=is%3Aissue+is%3Aopen+label%3A%22rule+idea%22) - issues that describe potential new rule ideas.
+* [good-first-issue](https://github.com/mandiant/capa/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - issues which should only require a few lines of code, and a test or two.
+* [rule-idea](https://github.com/mandiant/capa-rules/issues?q=is%3Aissue+is%3Aopen+label%3A%22rule+idea%22) - issues that describe potential new rule ideas.
 
 Both issue lists are sorted by total number of comments. While not perfect, number of comments is a reasonable proxy for impact a given change will have.
 
 #### Local development
 
 capa and all its resources can be developed locally. 
-For instructions on how to do this, see the "Method 3" section of the [installation guide](https://github.com/fireeye/capa/blob/master/doc/installation.md).
+For instructions on how to do this, see the "Method 3" section of the [installation guide](https://github.com/mandiant/capa/blob/master/doc/installation.md).
 
 ### Pull Requests
 
@@ -159,8 +159,8 @@ The process described here has several goals:
 
 Please follow these steps to have your contribution considered by the maintainers:
 
-1. Follow all instructions in [the template](PULL_REQUEST_TEMPLATE.md)
-2. Follow the [styleguides](#styleguides)
+1. Follow the [styleguides](#styleguides)
+2. Update the CHANGELOG and add tests and documentation. In case they are not needed, indicate it in [the PR template](pull_request_template.md).
 3. After you submit your pull request, verify that all [status checks](https://help.github.com/articles/about-status-checks/) are passing <details><summary>What if the status checks are failing? </summary>If a status check is failing, and you believe that the failure is unrelated to your change, please leave a comment on the pull request explaining why you believe the failure is unrelated. A maintainer will re-run the status check for you. If we conclude that the failure was a false positive, then we will open an issue to track that problem with our status check suite.</details>
 
 While the prerequisites above must be satisfied prior to having your pull request reviewed, the reviewer(s) may ask you to complete additional design work, tests, or other changes before your pull request can be ultimately accepted.
@@ -190,8 +190,8 @@ Our CI pipeline will reformat and enforce the Python styleguide.
 
 All (non-nursery) capa rules must:
 
-  1. pass the [linter](https://github.com/fireeye/capa/blob/master/scripts/lint.py), and
-  2. be formatted with [capafmt](https://github.com/fireeye/capa/blob/master/scripts/capafmt.py)
+  1. pass the [linter](https://github.com/mandiant/capa/blob/master/scripts/lint.py), and
+  2. be formatted with [capafmt](https://github.com/mandiant/capa/blob/master/scripts/capafmt.py)
   
 This ensures that all rules meet the same minimum level of quality and are structured in a consistent way.
 Our CI pipeline will reformat and enforce the capa rules styleguide.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -5,16 +5,16 @@ about: Create a report to help us improve
 ---
 <!--
 # Is your bug report related to capa rules (for example a false positive)?
-We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/fireeye/capa-rules/issues.
+We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/mandiant/capa-rules/issues.
 
 # Have you checked that your issue isn't already filed?
-Please search if there is a similar issue at https://github.com/fireeye/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
+Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
 # Have you read capa's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/fireeye/capa/blob/master/.github/CODE_OF_CONDUCT.md
+By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
 
 # Have you read capa's CONTRIBUTING guide?
-It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs
+It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs
 -->
 
 ### Description

.github/ISSUE_TEMPLATE/feature_request.md

@@ -5,16 +5,16 @@ about: Suggest an idea for capa
 ---
 <!--
 # Is your issue related to capa rules (for example an idea for a new rule)?
-We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/fireeye/capa-rules/issues.
+We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/mandiant/capa-rules/issues.
 
 # Have you checked that your issue isn't already filed?
-Please search if there is a similar issue at https://github.com/fireeye/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
+Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
 # Have you read capa's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/fireeye/capa/blob/master/.github/CODE_OF_CONDUCT.md
+By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
 
 # Have you read capa's CONTRIBUTING guide?
-It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements
+It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements
 -->
 
 ### Summary

.github/mypy/mypy.ini

@@ -0,0 +1,76 @@
+[mypy]
+
+[mypy-halo.*]
+ignore_missing_imports = True
+
+[mypy-tqdm.*]
+ignore_missing_imports = True
+
+[mypy-ruamel.*]
+ignore_missing_imports = True
+
+[mypy-networkx.*]
+ignore_missing_imports = True
+
+[mypy-pefile.*]
+ignore_missing_imports = True
+
+[mypy-viv_utils.*]
+ignore_missing_imports = True
+
+[mypy-flirt.*]
+ignore_missing_imports = True
+
+[mypy-smda.*]
+ignore_missing_imports = True
+
+[mypy-lief.*]
+ignore_missing_imports = True
+
+[mypy-idc.*]
+ignore_missing_imports = True
+
+[mypy-vivisect.*]
+ignore_missing_imports = True
+
+[mypy-envi.*]
+ignore_missing_imports = True
+
+[mypy-PE.*]
+ignore_missing_imports = True
+
+[mypy-idaapi.*]
+ignore_missing_imports = True
+
+[mypy-idautils.*]
+ignore_missing_imports = True
+
+[mypy-ida_bytes.*]
+ignore_missing_imports = True
+
+[mypy-ida_kernwin.*]
+ignore_missing_imports = True
+
+[mypy-ida_settings.*]
+ignore_missing_imports = True
+
+[mypy-ida_funcs.*]
+ignore_missing_imports = True
+
+[mypy-ida_loader.*]
+ignore_missing_imports = True
+
+[mypy-PyQt5.*]
+ignore_missing_imports = True
+
+[mypy-binaryninja.*]
+ignore_missing_imports = True
+
+[mypy-pytest.*]
+ignore_missing_imports = True
+
+[mypy-devtools.*]
+ignore_missing_imports = True
+
+[mypy-elftools.*]
+ignore_missing_imports = True

.github/pull_request_template.md

@@ -1,32 +1,22 @@
-<!--
-Thank you for contributing to capa! :heart:
 
-IMPORTANT NOTE
-It's most important that you submit your improvements. So even if you don't use this complete template we look forward to collaborating!
+<!--
+Thank you for contributing to capa! <3
 
 Please read capa's CONTRIBUTING guide if you haven't done so already.
-It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md
+It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md
 
-PR template based on https://embeddedartistry.com/blog/2017/08/04/a-github-pull-request-template-for-your-projects/
+Please describe the changes in this pull request (PR). Include your motivation and context to help us review.
+
+Please mention the issue your PR addresses (if any):
+closes #issue_number
 -->
 
-### Description
 
-<!-- Please describe the changes in this PR. Including your motivation and context helps us to review. -->
+### Checklist
 
-closes # (issue)
-
-### Type of change
-
-Please update the [CHANGELOG.md](/CHANGELOG.md)
-
-- [ ] Bug fix (non-breaking change which fixes an issue)
-- [ ] New feature (non-breaking change which adds functionality)
-- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
-- [ ] This change requires a documentation update
-  - [ ] I have made the corresponding changes to the documentation
-
-### Tests
-
-- [ ] I have added tests that prove my fix is effective or that my feature works
+<!-- CHANGELOG.md has a `master (unreleased)` section. Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning in the release notes to this file. -->
+- [ ] No CHANGELOG update needed
+<!-- Tests prove that your fix/work as expected and ensure it doesn't break on the feature. -->
 - [ ] No new tests needed
+<!-- Please help us keeping capa documentation up-to-date -->
+- [ ] No documentation update needed

.github/pyinstaller/hooks/hook-smda.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 import PyInstaller.utils.hooks
 
 # ref: https://groups.google.com/g/pyinstaller/c/amWi0-66uZI/m/miPoKfWjBAAJ

.github/pyinstaller/hooks/hook-vivisect.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 
 from PyInstaller.utils.hooks import copy_metadata
 
@@ -45,8 +45,8 @@ hiddenimports = [
     "vivisect.analysis.crypto",
     "vivisect.analysis.crypto.constants",
     "vivisect.analysis.elf",
-    "vivisect.analysis.elf",
     "vivisect.analysis.elf.elfplt",
+    "vivisect.analysis.elf.elfplt_late",
     "vivisect.analysis.elf.libc_start_main",
     "vivisect.analysis.generic",
     "vivisect.analysis.generic",

.github/pyinstaller/pyinstaller.spec

@@ -1,5 +1,5 @@
 # -*- mode: python -*-
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 import os.path
 import subprocess
 
@@ -33,6 +33,7 @@ a = Analysis(
         # this gets invoked from the directory of the spec file,
         # i.e. ./.github/pyinstaller
         ('../../rules', 'rules'),
+        ('../../sigs', 'sigs'),
 
         # capa.render.default uses tabulate that depends on wcwidth.
         # it seems wcwidth uses a json file `version.json`
@@ -108,5 +109,4 @@ exe = EXE(pyz,
 #               a.datas,
 #               strip=None,
 #               upx=True,
-#               name='capa-dat')
-
+#               name='capa-dat')

.github/workflows/build.yml

@@ -1,6 +1,8 @@
 name: build
 
 on:
+  push:
+    branches: [master]
   release:
     types: [edited, published]
 
@@ -11,7 +13,7 @@ jobs:
     strategy:
       matrix:
         include:
-          - os: ubuntu-16.04
+          - os: ubuntu-18.04
             # use old linux so that the shared library versioning is more portable
             artifact_name: capa
             asset_name: linux
@@ -26,11 +28,12 @@ jobs:
         uses: actions/checkout@v2
         with:
           submodules: true
-      - name: Set up Python 3.9
+      # using Python 3.8 to support running across multiple operating systems including Windows 7
+      - name: Set up Python 3.8
         uses: actions/setup-python@v2
         with:
-          python-version: 3.9
-      - if: matrix.os == 'ubuntu-16.04'
+          python-version: 3.8
+      - if: matrix.os == 'ubuntu-18.04'
         run: sudo apt-get install -y libyaml-dev
       - name: Install PyInstaller
         run: pip install 'pyinstaller==4.2'
@@ -38,17 +41,53 @@ jobs:
         run: pip install -e .
       - name: Build standalone executable
         run: pyinstaller .github/pyinstaller/pyinstaller.spec
-      - name: Does it run?
+      - name: Does it run (PE)?
         run: dist/capa "tests/data/Practical Malware Analysis Lab 01-01.dll_"
+      - name: Does it run (Shellcode)?
+        run: dist/capa "tests/data/499c2a85f6e8142c3f48d4251c9c7cd6.raw32"
+      - name: Does it run (ELF)?
+        run: dist/capa "tests/data/7351f8a40c5450557b24622417fc478d.elf_"
       - uses: actions/upload-artifact@v2
         with:
           name: ${{ matrix.asset_name }}
           path: dist/${{ matrix.artifact_name }}
 
-  zip:
-    name: zip ${{ matrix.asset_name }}
+  test_run:
+    # test that binaries run on push to master
+    if: github.event_name == 'push'
+    name: Test run on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    needs: [build]
+    strategy:
+      matrix:
+        include:
+          # OSs not already tested above
+          - os: ubuntu-18.04
+            artifact_name: capa
+            asset_name: linux
+          - os: ubuntu-20.04
+            artifact_name: capa
+            asset_name: linux
+          - os: windows-2016
+            artifact_name: capa.exe
+            asset_name: windows
+    steps:
+      - name: Download ${{ matrix.asset_name }}
+        uses: actions/download-artifact@v2
+        with:
+          name: ${{ matrix.asset_name }}
+      - name: Set executable flag
+        if: matrix.os != 'windows-2016'
+        run: chmod +x ${{ matrix.artifact_name }}
+      - name: Run capa
+        run: ./${{ matrix.artifact_name }} -h
+
+  zip_and_upload:
+    # upload zipped binaries to Release page
+    if: github.event_name == 'release'
+    name: zip and upload ${{ matrix.asset_name }}
     runs-on: ubuntu-20.04
-    needs: build
+    needs: [build]
     strategy:
       matrix:
         include:

.github/workflows/changelog.yml

@@ -0,0 +1,41 @@
+name: changelog
+
+on:
+  # We need pull_request_target instead of pull_request because a write
+  # repository token is needed to add a review to a PR. DO NOT BUILD
+  # OR RUN UNTRUSTED CODE FROM PRs IN THIS ACTION
+  pull_request_target:
+    types: [opened, edited, synchronize]
+
+jobs:
+  check_changelog:
+    # no need to check for dependency updates via dependabot
+    if: github.actor != 'dependabot[bot]' && github.actor != 'dependabot-preview[bot]'
+    runs-on: ubuntu-20.04
+    env:
+      NO_CHANGELOG: '[x] No CHANGELOG update needed'
+    steps:
+    - name: Get changed files
+      id: files
+      uses: Ana06/get-changed-files@v1.2
+    - name: check changelog updated
+      id: changelog_updated
+      env:
+        PR_BODY: ${{ github.event.pull_request.body }}
+        FILES: ${{ steps.files.outputs.modified }}
+      run: |
+        echo $FILES | grep -qF 'CHANGELOG.md' || echo $PR_BODY | grep -qiF "$NO_CHANGELOG"
+    - name: Reject pull request if no CHANGELOG update
+      if: ${{ always() && steps.changelog_updated.outcome == 'failure' }}
+      uses: Ana06/automatic-pull-request-review@v0.1.0
+      with:
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        event: REQUEST_CHANGES
+        body: "Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the `master (unreleased)` section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: `${{ env.NO_CHANGELOG }}`"
+        allow_duplicate: false
+    - name: Dismiss previous review if CHANGELOG update
+      uses: Ana06/automatic-pull-request-review@v0.1.0
+      with:
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        event: DISMISS
+        body: "CHANGELOG updated or no update needed, thanks! :smile:"

.github/workflows/publish.yml

@@ -15,7 +15,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v2
         with:
-          python-version: '2.7'
+          python-version: '3.6'
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
@@ -27,3 +27,4 @@ jobs:
         run: |
           python setup.py sdist bdist_wheel
           twine upload --skip-existing dist/*
+

.github/workflows/tag.yml

@@ -12,13 +12,18 @@ jobs:
     - name: Checkout capa-rules
       uses: actions/checkout@v2
       with:
-        repository: fireeye/capa-rules
+        repository: mandiant/capa-rules
         token: ${{ secrets.CAPA_TOKEN }}
     - name: Tag capa-rules
-      run: git tag ${{ github.event.release.tag_name }}
+      run: |
+        # user information is needed to create annotated tags (with a message)
+        git config user.email 'capa-dev@mandiant.com'
+        git config user.name 'Capa Bot'
+        name=${{ github.event.release.tag_name }}
+        git tag $name -m "https://github.com/mandiant/capa/releases/$name"
     - name: Push tag to capa-rules
       uses: ad-m/github-push-action@master
       with:
-        repository: fireeye/capa-rules
+        repository: mandiant/capa-rules
         github_token: ${{ secrets.CAPA_TOKEN }}
         tags: true

.github/workflows/tests.yml

@@ -6,7 +6,22 @@ on:
   pull_request:
     branches: [ master ]
 
+# save workspaces to speed up testing
+env:
+  CAPA_SAVE_WORKSPACE: "True"
+
 jobs:
+  changelog_format:
+    runs-on: ubuntu-20.04
+    steps:
+    - name: Checkout capa
+      uses: actions/checkout@v2
+    # The sync GH action in capa-rules relies on a single '- *$' in the CHANGELOG file
+    - name: Ensure CHANGELOG has '- *$'
+      run: |
+        number=$(grep '\- *$' CHANGELOG.md | wc -l)
+        if [ $number != 1 ]; then exit 1; fi
+
   code_style:
     runs-on: ubuntu-20.04
     steps:
@@ -15,26 +30,27 @@ jobs:
     - name: Set up Python 3.8
       uses: actions/setup-python@v2
       with:
-        python-version: 3.8
+        python-version: "3.8"
     - name: Install dependencies
-      run: pip install 'isort==5.*' black
+      run: pip install -e .[dev]
     - name: Lint with isort
       run: isort --profile black --length-sort --line-width 120 -c .
     - name: Lint with black
       run: black -l 120 --check .
+    - name: Check types with mypy
+      run: mypy --config-file .github/mypy/mypy.ini capa/ scripts/ tests/
 
   rule_linter:
     runs-on: ubuntu-20.04
     steps:
-    - name: Checkout capa with rules submodule
+    - name: Checkout capa with submodules
       uses: actions/checkout@v2
       with:
         submodules: true
     - name: Set up Python 3.8
       uses: actions/setup-python@v2
       with:
-        python-version: 3.8
-    # We don't need vivisect, so we can install capa using Python3
+        python-version: "3.8"
     - name: Install capa
       run: pip install -e .
     - name: Run rule linter
@@ -49,15 +65,15 @@ jobs:
       matrix:
         os: [ubuntu-20.04, windows-2019, macos-10.15]
         # across all operating systems
-        python-version: [3.6, 3.9]
+        python-version: ["3.6", "3.10"]
         include:
           # on Ubuntu run these as well
           - os: ubuntu-20.04
-            python-version: 2.7
+            python-version: "3.7"
           - os: ubuntu-20.04
-            python-version: 3.7
+            python-version: "3.8"
           - os: ubuntu-20.04
-            python-version: 3.8
+            python-version: "3.9"
     steps:
     - name: Checkout capa with submodules
       uses: actions/checkout@v2
@@ -70,10 +86,7 @@ jobs:
     - name: Install pyyaml
       if: matrix.os == 'ubuntu-20.04'
       run: sudo apt-get install -y libyaml-dev
-    - name: Install Microsoft Visual C++ 9.0
-      if: matrix.os == 'windows-2019' && matrix.python-version == '2.7'
-      run: choco install vcpython27
     - name: Install capa
       run: pip install -e .[dev]
     - name: Run tests
-      run: pytest tests/
+      run: pytest -v tests/

.gitignore

@@ -114,3 +114,7 @@ venv.bak/
 isort-output.log
 black-output.log
 rule-linter-output.log
+.vscode
+scripts/perf/*.txt
+scripts/perf/*.svg
+scripts/perf/*.zip

CHANGELOG.md

@@ -4,17 +4,475 @@
 
 ### New Features
 
-### New Rules
+### Breaking Changes
+
+### New Rules (0)
+
+-
 
 ### Bug Fixes
 
-### Changes
+### capa explorer IDA Pro plugin
 
 ### Development
 
 ### Raw diffs
-- [capa v1.6.1...master](https://github.com/fireeye/capa/compare/v1.6.1...master)
-- [capa-rules v1.6.1...master](https://github.com/fireeye/capa-rules/compare/v1.6.1...master)
+- [capa v3.1.0...master](https://github.com/mandiant/capa/compare/v3.1.0...master)
+- [capa-rules v3.1.0...master](https://github.com/mandiant/capa-rules/compare/v3.1.0...master)
+
+## v3.1.0 (2022-01-10)
+This release improves the performance of capa while also adding 23 new rules and many code quality enhancements. We profiled capa's CPU usage and optimized the way that it matches rules, such as by short circuiting when appropriate. According to our testing, the matching phase is approximately 66% faster than v3.0.3! We also added support for Python 3.10, aarch64 builds, and additional MAEC metadata in the rule headers.
+  
+This release adds 23 new rules, including nine by Jakub Jozwiak of Mandiant. @ryantxu1 and @dzbeck updated the ATT&CK and MBC mappings for many rules. Thank you!
+  
+And as always, welcome first time contributors!
+
+  - @kn0wl3dge
+  - @jtothej
+  - @cl30
+  
+
+### New Features
+
+- engine: short circuit logic nodes for better performance #824 @williballenthin
+- engine: add optimizer the order faster nodes first #829 @williballenthin
+- engine: optimize rule evaluation by skipping rules that can't match #830 @williballenthin
+- support python 3.10 #816 @williballenthin
+- support aarch64 #683 @williballenthin
+- rules: support maec/malware-family meta #841 @mr-tz
+- engine: better type annotations/exhaustiveness checking #839 @cl30
+
+### Breaking Changes: None
+
+### New Rules (23)
+
+- nursery/delete-windows-backup-catalog michael.hunhoff@mandiant.com
+- nursery/disable-automatic-windows-recovery-features michael.hunhoff@mandiant.com
+- nursery/capture-webcam-video @johnk3r
+- nursery/create-registry-key-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/delete-registry-key-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/delete-registry-value-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/query-or-enumerate-registry-key-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/query-or-enumerate-registry-value-via-stdregprov michael.hunhoff@mandiant.com
+- nursery/set-registry-value-via-stdregprov michael.hunhoff@mandiant.com
+- data-manipulation/compression/decompress-data-using-ucl jakub.jozwiak@mandiant.com
+- linking/static/wolfcrypt/linked-against-wolfcrypt jakub.jozwiak@mandiant.com
+- linking/static/wolfssl/linked-against-wolfssl jakub.jozwiak@mandiant.com
+- anti-analysis/packer/pespin/packed-with-pespin jakub.jozwiak@mandiant.com
+- load-code/shellcode/execute-shellcode-via-windows-fibers jakub.jozwiak@mandiant.com
+- load-code/shellcode/execute-shellcode-via-enumuilanguages jakub.jozwiak@mandiant.com
+- anti-analysis/packer/themida/packed-with-themida william.ballenthin@mandiant.com
+- load-code/shellcode/execute-shellcode-via-createthreadpoolwait jakub.jozwiak@mandiant.com
+- host-interaction/process/inject/inject-shellcode-using-a-file-mapping-object jakub.jozwiak@mandiant.com
+- load-code/shellcode/execute-shellcode-via-copyfile2 jakub.jozwiak@mandiant.com
+- malware-family/plugx/match-known-plugx-module still@teamt5.org
+
+### Rule Changes
+
+  - update ATT&CK mappings by @ryantxu1
+  - update ATT&CK and MBC mappings by @dzbeck
+  - aplib detection by @cdong1012
+  - golang runtime detection by @stevemk14eber
+
+### Bug Fixes
+
+- fix circular import error #825 @williballenthin
+- fix smda negative number extraction #430 @kn0wl3dge
+
+### capa explorer IDA Pro plugin
+
+- pin supported versions to >= 7.4 and < 8.0 #849 @mike-hunhoff
+
+### Development
+
+- add profiling infrastructure #828 @williballenthin
+- linter: detect shellcode extension #820 @mr-tz
+- show features script: add backend flag #430 @kn0wl3dge
+
+### Raw diffs
+- [capa v3.0.3...v3.1.0](https://github.com/mandiant/capa/compare/v3.0.3...v3.1.0)
+- [capa-rules v3.0.3...v3.1.0](https://github.com/mandiant/capa-rules/compare/v3.0.3...v3.1.0)
+
+
+## v3.0.3 (2021-10-27)
+
+This is primarily a rule maintenance release:
+  - eight new rules, including all relevant techniques from [ATT&CK v10](https://medium.com/mitre-attack/introducing-attack-v10-7743870b37e3), and
+  - two rules removed, due to the prevalence of false positives
+
+We've also tweaked the status codes returned by capa.exe to be more specific and added a bit more metadata to the JSON output format.
+ 
+As always, welcome first time contributors!
+  - still@teamt5.org
+  - zander.work@mandiant.com
+                                                                                                     
+
+### New Features
+
+- show in which function a BB match is #130 @williballenthin
+- main: exit with unique error codes when bailing #802 @williballenthin
+
+### New Rules (8)
+
+- nursery/resolve-function-by-fnv-1a-hash still@teamt5.org
+- data-manipulation/encryption/encrypt-data-using-memfrob-from-glibc zander.work@mandiant.com
+- collection/group-policy/discover-group-policy-via-gpresult william.ballenthin@mandiant.com
+- host-interaction/bootloader/manipulate-safe-mode-programs william.ballenthin@mandiant.com
+- nursery/enable-safe-mode-boot william.ballenthin@mandiant.com
+- persistence/iis/persist-via-iis-module william.ballenthin@mandiant.com
+- persistence/iis/persist-via-isapi-extension william.ballenthin@mandiant.com
+- targeting/language/identify-system-language-via-api william.ballenthin@mandiant.com
+
+## Removed rules (2)
+- load-code/pe/parse-pe-exports: too many false positives in unrelated structure accesses
+- anti-analysis/anti-vm/vm-detection/execute-anti-vm-instructions: too many false positives in junk code
+
+### Bug Fixes
+
+- update references from FireEye to Mandiant
+
+### Raw diffs
+- [capa v3.0.2...v3.0.3](https://github.com/fireeye/capa/compare/v3.0.2...v3.0.3)
+- [capa-rules v3.0.2...v3.0.3](https://github.com/fireeye/capa-rules/compare/v3.0.2...v3.0.3)
+  
+## v3.0.2 (2021-09-28)
+  
+This release fixes an issue with the standalone executables built with PyInstaller when running capa against ELF files.
+
+### Bug Fixes
+
+- fix bug in PyInstaller config preventing ELF analysis #795 @mr-tz
+
+### Raw diffs
+- [capa v3.0.1...v3.0.2](https://github.com/fireeye/capa/compare/v3.0.1...v3.0.2)
+- [capa-rules v3.0.1...v3.0.2](https://github.com/fireeye/capa-rules/compare/v3.0.1...v3.0.2)
+
+## v3.0.1 (2021-09-27)
+
+This version updates the version of vivisect used by capa. Users will experience fewer bugs and find improved analysis results.
+
+Thanks to the community for highlighting issues and analysis misses. Your feedback is crucial to further improve capa.
+
+### Bug Fixes
+
+- fix many underlying bugs in vivisect analysis and update to version v1.0.5 #786 @williballenthin
+
+### Raw diffs
+- [capa v3.0.0...v3.0.1](https://github.com/fireeye/capa/compare/v3.0.0...v3.0.1)
+- [capa-rules v3.0.0...v3.0.1](https://github.com/fireeye/capa-rules/compare/v3.0.0...v3.0.1)
+
+## v3.0.0 (2021-09-15)
+
+We are excited to announce version 3.0! :tada:
+
+capa 3.0:
+- adds support for ELF files targeting Linux thanks to [Intezer](https://www.intezer.com/)
+- adds new features to specify OS, CPU architecture, and file format
+- fixes a few bugs that may have led to false negatives (missed capabilities) in older versions
+- adds 80 new rules, including 36 describing techniques for Linux
+
+A huge thanks to everyone who submitted issues, provided feedback, and contributed code and rules.
+Special acknowledgement to @Adir-Shemesh and @TcM1911 of [Intezer](https://www.intezer.com/) for contributing the code to enable ELF support.
+Also, welcome first time contributors:
+  - @jaredscottwilson
+  - @cdong1012
+  - @jlepore-fe 
+
+### New Features
+
+- all: add support for ELF files #700 @Adir-Shemesh @TcM1911
+- rule format: add feature `format: ` for file format, like `format: pe` #723 @williballenthin
+- rule format: add feature `arch: ` for architecture, like `arch: amd64` #723 @williballenthin
+- rule format: add feature `os: ` for operating system, like `os: windows` #723 @williballenthin
+- rule format: add feature `substring: ` for verbatim strings with leading/trailing wildcards #737 @williballenthin
+- scripts: add `profile-memory.py` for profiling memory usage #736 @williballenthin
+- main: add light weight ELF file feature extractor to detect file limitations #770 @mr-tz
+
+### Breaking Changes
+
+- rules using `format`, `arch`, `os`, or `substring` features cannot be used by capa versions prior to v3
+- legacy term `arch` (i.e., "x32") is now called `bitness` @williballenthin
+- freeze format gains new section for "global" features #759 @williballenthin
+
+### New Rules (80)
+
+- collection/webcam/capture-webcam-image @johnk3r
+- nursery/list-drag-and-drop-files michael.hunhoff@mandiant.com
+- nursery/monitor-clipboard-content michael.hunhoff@mandiant.com
+- nursery/monitor-local-ipv4-address-changes michael.hunhoff@mandiant.com
+- nursery/load-windows-common-language-runtime michael.hunhoff@mandiant.com
+- nursery/resize-volume-shadow-copy-storage michael.hunhoff@mandiant.com
+- nursery/add-user-account-group michael.hunhoff@mandiant.com
+- nursery/add-user-account-to-group michael.hunhoff@mandiant.com
+- nursery/add-user-account michael.hunhoff@mandiant.com
+- nursery/change-user-account-password michael.hunhoff@mandiant.com
+- nursery/delete-user-account-from-group michael.hunhoff@mandiant.com
+- nursery/delete-user-account-group michael.hunhoff@mandiant.com
+- nursery/delete-user-account michael.hunhoff@mandiant.com
+- nursery/list-domain-servers michael.hunhoff@mandiant.com
+- nursery/list-groups-for-user-account michael.hunhoff@mandiant.com
+- nursery/list-user-account-groups michael.hunhoff@mandiant.com
+- nursery/list-user-accounts-for-group michael.hunhoff@mandiant.com
+- nursery/list-user-accounts michael.hunhoff@mandiant.com
+- nursery/parse-url michael.hunhoff@mandiant.com
+- nursery/register-raw-input-devices michael.hunhoff@mandiant.com
+- anti-analysis/packer/gopacker/packed-with-gopacker jared.wilson@mandiant.com
+- host-interaction/driver/create-device-object @mr-tz
+- host-interaction/process/create/execute-command @mr-tz
+- data-manipulation/encryption/create-new-key-via-cryptacquirecontext chuong.dong@mandiant.com
+- host-interaction/log/clfs/append-data-to-clfs-log-container blaine.stancill@mandiant.com
+- host-interaction/log/clfs/read-data-from-clfs-log-container blaine.stancill@mandiant.com
+- data-manipulation/encryption/hc-128/encrypt-data-using-hc-128-via-wolfssl blaine.stancill@mandiant.com
+- c2/shell/create-unix-reverse-shell joakim@intezer.com
+- c2/shell/execute-shell-command-received-from-socket joakim@intezer.com
+- collection/get-current-user joakim@intezer.com
+- host-interaction/file-system/change-file-permission joakim@intezer.com
+- host-interaction/hardware/memory/get-memory-information joakim@intezer.com
+- host-interaction/mutex/lock-file joakim@intezer.com
+- host-interaction/os/version/get-kernel-version joakim@intezer.com
+- host-interaction/os/version/get-linux-distribution joakim@intezer.com
+- host-interaction/process/terminate/terminate-process-via-kill joakim@intezer.com
+- lib/duplicate-stdin-and-stdout joakim@intezer.com
+- nursery/capture-network-configuration-via-ifconfig joakim@intezeer.com
+- nursery/collect-ssh-keys joakim@intezer.com
+- nursery/enumerate-processes-via-procfs joakim@intezer.com
+- nursery/interact-with-iptables joakim@intezer.com
+- persistence/persist-via-desktop-autostart joakim@intezer.com
+- persistence/persist-via-shell-profile-or-rc-file joakim@intezer.com
+- persistence/service/persist-via-rc-script joakim@intezer.com
+- collection/get-current-user-on-linux joakim@intezer.com
+- collection/network/get-mac-address-on-windows moritz.raabe@mandiant.com
+- host-interaction/file-system/read/read-file-on-linux moritz.raabe@mandiant.com joakim@intezer.com
+- host-interaction/file-system/read/read-file-on-windows moritz.raabe@mandiant.com
+- host-interaction/file-system/write/write-file-on-windows william.ballenthin@mandiant.com
+- host-interaction/os/info/get-system-information-on-windows moritz.raabe@mandiant.com joakim@intezer.com
+- host-interaction/process/create/create-process-on-windows moritz.raabe@mandiant.com
+- linking/runtime-linking/link-function-at-runtime-on-windows moritz.raabe@mandiant.com
+- nursery/create-process-on-linux joakim@intezer.com
+- nursery/enumerate-files-on-linux william.ballenthin@mandiant.com
+- nursery/get-mac-address-on-linux joakim@intezer.com
+- nursery/get-system-information-on-linux joakim@intezer.com
+- nursery/link-function-at-runtime-on-linux joakim@intezer.com
+- nursery/write-file-on-linux joakim@intezer.com
+- communication/socket/tcp/send/obtain-transmitpackets-callback-function-via-wsaioctl jonathan.lepore@mandiant.com
+- nursery/linked-against-cpp-http-library @mr-tz
+- nursery/linked-against-cpp-json-library @mr-tz
+
+### Bug Fixes
+
+- main: fix `KeyError: 0` when reporting results @williballehtin #703
+- main: fix potential false negatives due to namespaces across scopes @williballenthin #721
+- linter: suppress some warnings about imports from ntdll/ntoskrnl @williballenthin #743
+- linter: suppress some warnings about missing examples in the nursery @williballenthin #747
+
+### capa explorer IDA Pro plugin
+
+- explorer: add additional filter logic when displaying matches by function #686 @mike-hunhoff
+- explorer: remove duplicate check when saving file #687 @mike-hunhoff
+- explorer: update IDA extractor to use non-canon mnemonics #688 @mike-hunhoff
+- explorer: allow user to add specified number of bytes when adding a Bytes feature in the Rule Generator #689 @mike-hunhoff
+- explorer: enforce max column width Features and Editor panes #691 @mike-hunhoff
+- explorer: add option to limit features to currently selected disassembly address #692 @mike-hunhoff
+- explorer: update support documentation and runtime checks #741 @mike-hunhoff
+- explorer: small performance boost to rule generator search functionality #742 @mike-hunhoff
+- explorer: add support for arch, os, and format features #758 @mike-hunhoff
+- explorer: improve parsing algorithm for rule generator feature editor #768 @mike-hunhoff
+
+### Development
+
+### Raw diffs
+- [capa v2.0.0...v3.0.0](https://github.com/mandiant/capa/compare/v2.0.0...v3.0.0)
+- [capa-rules v2.0.0...v3.0.0](https://github.com/mandiant/capa-rules/compare/v2.0.0...v3.0.0)
+
+
+## v2.0.0 (2021-07-19)
+
+We are excited to announce version 2.0! :tada:
+capa 2.0:
+- enables anyone to contribute rules more easily
+- is the first Python 3 ONLY version
+- provides more concise and relevant result via identification of library functions using FLIRT
+  ![capa v2.0 results ignoring library code functions](doc/img/changelog/flirt-ignore.png)
+- includes many features and enhancements for the capa explorer IDA plugin
+- adds 93 new rules, including all new techniques introduced in MITRE ATT&CK v9
+
+A huge thanks to everyone who submitted issues, provided feedback, and contributed code and rules. Many colleagues across dozens of organizations have volunteered their experience to improve this tool! :heart:
+
+
+### New Features
+
+- rules: update ATT&CK and MBC mappings https://github.com/mandiant/capa-rules/pull/317 @williballenthin
+- main: use FLIRT signatures to identify and ignore library code #446 @williballenthin
+- tests: update test cases and caching #545 @mr-tz
+- scripts: capa2yara.py convert capa rules to YARA rules #561 @ruppde
+- rule: add file-scope feature (`function-name`) for recognized library functions #567 @williballenthin
+- main: auto detect shellcode based on file extension #516 @mr-tz
+- main: more detailed progress bar output when matching functions #562 @mr-tz
+- main: detect file limitations without doing code analysis for better performance #583 @williballenthin
+- show-features: don't show features from library functions #569 @williballenthin
+- linter: summarize results at the end #571 @williballenthin
+- linter: check for `or` with always true child statement, e.g. `optional`, colors #348 @mr-tz
+
+### Breaking Changes
+
+- py3: drop Python 2 support #480 @Ana06
+- meta: added `library_functions` field, `feature_counts.functions` does not include library functions any more #562 @mr-tz
+- json: results document now contains parsed ATT&CK and MBC fields instead of canonical representation #526 @mr-tz
+- json: record all matching strings for regex #159 @williballenthin
+- main: implement file limitations via rules not code #390 @williballenthin
+- json: correctly render negative offsets #619 @williballenthin
+- library: remove logic from `__init__.py` throughout #622 @williballenthin
+
+### New Rules (93)
+
+- anti-analysis/packer/amber/packed-with-amber @gormaniac
+- collection/file-managers/gather-3d-ftp-information @re-fox
+- collection/file-managers/gather-alftp-information @re-fox
+- collection/file-managers/gather-bitkinex-information @re-fox
+- collection/file-managers/gather-blazeftp-information @re-fox
+- collection/file-managers/gather-bulletproof-ftp-information @re-fox
+- collection/file-managers/gather-classicftp-information @re-fox
+- collection/file-managers/gather-coreftp-information @re-fox
+- collection/file-managers/gather-cuteftp-information @re-fox
+- collection/file-managers/gather-cyberduck-information @re-fox
+- collection/file-managers/gather-direct-ftp-information @re-fox
+- collection/file-managers/gather-directory-opus-information @re-fox
+- collection/file-managers/gather-expandrive-information @re-fox
+- collection/file-managers/gather-faststone-browser-information @re-fox
+- collection/file-managers/gather-fasttrack-ftp-information @re-fox
+- collection/file-managers/gather-ffftp-information @re-fox
+- collection/file-managers/gather-filezilla-information @re-fox
+- collection/file-managers/gather-flashfxp-information @re-fox
+- collection/file-managers/gather-fling-ftp-information @re-fox
+- collection/file-managers/gather-freshftp-information @re-fox
+- collection/file-managers/gather-frigate3-information @re-fox
+- collection/file-managers/gather-ftp-commander-information @re-fox
+- collection/file-managers/gather-ftp-explorer-information @re-fox
+- collection/file-managers/gather-ftp-voyager-information @re-fox
+- collection/file-managers/gather-ftpgetter-information @re-fox
+- collection/file-managers/gather-ftpinfo-information @re-fox
+- collection/file-managers/gather-ftpnow-information @re-fox
+- collection/file-managers/gather-ftprush-information @re-fox
+- collection/file-managers/gather-ftpshell-information @re-fox
+- collection/file-managers/gather-global-downloader-information @re-fox
+- collection/file-managers/gather-goftp-information @re-fox
+- collection/file-managers/gather-leapftp-information @re-fox
+- collection/file-managers/gather-netdrive-information @re-fox
+- collection/file-managers/gather-nexusfile-information @re-fox
+- collection/file-managers/gather-nova-ftp-information @re-fox
+- collection/file-managers/gather-robo-ftp-information @re-fox
+- collection/file-managers/gather-securefx-information @re-fox
+- collection/file-managers/gather-smart-ftp-information @re-fox
+- collection/file-managers/gather-softx-ftp-information @re-fox
+- collection/file-managers/gather-southriver-webdrive-information @re-fox
+- collection/file-managers/gather-staff-ftp-information @re-fox
+- collection/file-managers/gather-total-commander-information @re-fox
+- collection/file-managers/gather-turbo-ftp-information @re-fox
+- collection/file-managers/gather-ultrafxp-information @re-fox
+- collection/file-managers/gather-winscp-information @re-fox
+- collection/file-managers/gather-winzip-information @re-fox
+- collection/file-managers/gather-wise-ftp-information @re-fox
+- collection/file-managers/gather-ws-ftp-information @re-fox
+- collection/file-managers/gather-xftp-information @re-fox
+- data-manipulation/compression/decompress-data-using-aplib @r3c0nst @mr-tz
+- host-interaction/bootloader/disable-code-signing @williballenthin
+- host-interaction/bootloader/manipulate-boot-configuration @williballenthin
+- host-interaction/driver/disable-driver-code-integrity @williballenthin
+- host-interaction/file-system/bypass-mark-of-the-web @williballenthin
+- host-interaction/network/domain/get-domain-information @recvfrom
+- host-interaction/session/get-logon-sessions @recvfrom
+- linking/runtime-linking/resolve-function-by-fin8-fasthash @r3c0nst @mr-tz
+- nursery/build-docker-image @williballenthin
+- nursery/create-container @williballenthin
+- nursery/encrypt-data-using-fakem-cipher @mike-hunhoff
+- nursery/list-containers @williballenthin
+- nursery/run-in-container @williballenthin
+- persistence/registry/appinitdlls/disable-appinit_dlls-code-signature-enforcement @williballenthin
+- collection/password-manager/steal-keepass-passwords-using-keefarce @Ana06
+- host-interaction/network/connectivity/check-internet-connectivity-via-wininet matthew.williams@mandiant.com michael.hunhoff@mandiant.com
+- nursery/create-bits-job @mr-tz
+- nursery/execute-syscall-instruction @kulinacs @mr-tz
+- nursery/connect-to-wmi-namespace-via-wbemlocator michael.hunhoff@mandiant.com
+- anti-analysis/obfuscation/obfuscated-with-callobfuscator johnk3r
+- executable/installer/inno-setup/packaged-as-an-inno-setup-installer awillia2@cisco.com
+- data-manipulation/hashing/djb2/hash-data-using-djb2 awillia2@cisco.com
+- data-manipulation/encoding/base64/decode-data-using-base64-via-dword-translation-table gilbert.elliot@mandiant.com
+- nursery/list-tcp-connections-and-listeners michael.hunhoff@mandiant.com
+- nursery/list-udp-connections-and-listeners michael.hunhoff@mandiant.com
+- nursery/log-keystrokes-via-raw-input-data michael.hunhoff@mandiant.com
+- nursery/register-http-server-url michael.hunhoff@mandiant.com
+- internal/limitation/file/internal-autoit-file-limitation.yml william.ballenthin@mandiant.com
+- internal/limitation/file/internal-dotnet-file-limitation.yml william.ballenthin@mandiant.com
+- internal/limitation/file/internal-installer-file-limitation.yml william.ballenthin@mandiant.com
+- internal/limitation/file/internal-packer-file-limitation.yml william.ballenthin@mandiant.com
+- host-interaction/network/domain/enumerate-domain-computers-via-ldap awillia2@cisco.com
+- host-interaction/network/domain/get-domain-controller-name awillia2@cisco.com
+- internal/limitation/file/internal-visual-basic-file-limitation @mr-tz
+- data-manipulation/hashing/md5/hash-data-with-md5 moritz.raabe@mandiant.com
+- compiler/autohotkey/compiled-with-autohotkey awillia2@cisco.com
+- internal/limitation/file/internal-autohotkey-file-limitation @mr-tz
+- host-interaction/process/dump/create-process-memory-minidump michael.hunhoff@mandiant.com
+- nursery/get-storage-device-properties michael.hunhoff@mandiant.com
+- nursery/execute-shell-command-via-windows-remote-management michael.hunhoff@mandiant.com
+- nursery/get-token-privileges michael.hunhoff@mandiant.com
+- nursery/prompt-user-for-credentials michael.hunhoff@mandiant.com
+- nursery/spoof-parent-pid michael.hunhoff@mandiant.com
+
+### Bug Fixes
+
+- build: use Python 3.8 for PyInstaller to support consistently running across multiple operating systems including Windows 7 #505 @mr-tz
+- main: correctly match BB-scope matches at file scope #605 @williballenthin
+- main: do not process non-PE files even when --format explicitly provided #664 @mr-tz
+
+### capa explorer IDA Pro plugin
+- explorer: IDA 7.6 support #497 @williballenthin
+- explorer: explain how to install IDA 7.6 patch to enable the plugin #528 @williballenthin
+- explorer: document IDA 7.6sp1 as alternative to the patch #536 @Ana06
+- explorer: add support for function-name feature #618 @mike-hunhoff
+- explorer: circular import workaround #654 @mike-hunhoff
+- explorer: add argument to control whether to automatically analyze when running capa explorer #548 @Ana06
+- explorer: extract API features via function names recognized by IDA/FLIRT #661 @mr-tz
+
+### Development
+
+- ci: add capa release link to capa-rules tag #517 @Ana06
+- ci, changelog: update `New Rules` section in CHANGELOG automatically https://github.com/mandiant/capa-rules/pull/374 #549 #604 @Ana06
+- ci, changelog: support multiple author in sync GH https://github.com/mandiant/capa-rules/pull/378 @Ana06
+- ci, lint: check statements for single child statements #563 @mr-tz
+- ci: reject PRs without CHANGELOG update to ensure CHANGELOG is kept up-to-date #584 @Ana06
+- ci: test that scripts run #660 @mr-tz
+
+### Raw diffs
+
+<!-- The diff uses v1.6.1 because master doesn't include v1.6.2 and v1.6.3 -->
+- [capa v1.6.1...v2.0.0](https://github.com/mandiant/capa/compare/v1.6.1...v2.0.0)
+- [capa-rules v1.6.1...v2.0.0](https://github.com/mandiant/capa-rules/compare/v1.6.1...v2.0.0)
+
+
+## v1.6.3 (2021-04-29)
+
+This release adds IDA 7.6 support to capa.
+
+### Changes
+
+- IDA 7.6 support @williballenthin @Ana06
+
+### Raw diffs
+
+  - [capa v1.6.2...v1.6.3](https://github.com/mandiant/capa/compare/v1.6.2...v1.6.3)
+
+
+## v1.6.2 (2021-04-13)
+
+This release backports a fix to capa 1.6: The Windows binary was built with Python 3.9 which doesn't support Windows 7.
+
+### Bug Fixes
+
+- build: use Python 3.8 for PyInstaller to support consistently running across multiple operating systems including Windows 7 @mr-tz @Ana06
+
+### Raw diffs
+
+  - [capa v1.6.1...v1.6.2](https://github.com/mandiant/capa/compare/v1.6.1...v1.6.2)
 
 
 ## v1.6.1 (2021-04-07)
@@ -82,8 +540,8 @@ This release includes several bug fixes, such as a vivisect issue that prevented
 
 ### Raw diffs
 
-  - [capa v1.6.0...v1.6.1](https://github.com/fireeye/capa/compare/v1.6.0...v1.6.1)
-  - [capa-rules v1.6.0...v1.6.1](https://github.com/fireeye/capa-rules/compare/v1.6.0...v1.6.1)
+  - [capa v1.6.0...v1.6.1](https://github.com/mandiant/capa/compare/v1.6.0...v1.6.1)
+  - [capa-rules v1.6.0...v1.6.1](https://github.com/mandiant/capa-rules/compare/v1.6.0...v1.6.1)
 
 
 ## v1.6.0 (2021-03-09)
@@ -92,7 +550,7 @@ This release adds the capa explorer rule generator plugin for IDA Pro, vivisect
 
 ### Rule Generator IDA Plugin
 
-The capa explorer IDA plugin now helps you quickly build new capa rules using features extracted directly from your IDA database. Without leaving the plugin interface you can use the features extracted by capa explorer to develop and test new rules and save your work directly to your capa rules directory. To get started select the new `Rule Generator` tab, navigate to a function in the IDA `Disassembly` view, and click `Analyze`. For more information check out the capa explorer [readme](https://github.com/fireeye/capa/blob/master/capa/ida/plugin/README.md).
+The capa explorer IDA plugin now helps you quickly build new capa rules using features extracted directly from your IDA database. Without leaving the plugin interface you can use the features extracted by capa explorer to develop and test new rules and save your work directly to your capa rules directory. To get started select the new `Rule Generator` tab, navigate to a function in the IDA `Disassembly` view, and click `Analyze`. For more information check out the capa explorer [readme](https://github.com/mandiant/capa/blob/master/capa/ida/plugin/README.md).
 
 ![](doc/img/rulegen_expanded.png)
 
@@ -154,8 +612,8 @@ If you have workflows that rely on the Python 2 version and need future maintena
 
 ### Raw diffs
 
-  - [capa v1.5.1...v1.6.0](https://github.com/fireeye/capa/compare/v1.5.1...v1.6.0)
-  - [capa-rules v1.5.1...v1.6.0](https://github.com/fireeye/capa-rules/compare/v1.5.1...v1.6.0)
+  - [capa v1.5.1...v1.6.0](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
+  - [capa-rules v1.5.1...v1.6.0](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
 
 
 ## v1.5.1 (2021-02-09)
@@ -168,8 +626,8 @@ This release fixes the version number that we forgot to update for v1.5.0 (there
 
 ### Raw diffs
 
-  - [capa v1.5.0...v1.5.1](https://github.com/fireeye/capa/compare/v1.5.1...v1.6.0)
-  - [capa-rules v1.5.0...v1.5.1](https://github.com/fireeye/capa-rules/compare/v1.5.1...v1.6.0)
+  - [capa v1.5.0...v1.5.1](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
+  - [capa-rules v1.5.0...v1.5.1](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
 
 
 ## v1.5.0 (2021-02-05)
@@ -184,7 +642,7 @@ This release brings support for running capa under Python 3 via [SMDA](https://g
   
 @dzbeck also added [Malware Behavior Catalog](https://github.com/MBCProject/mbc-markdown) (MBC) and ATT&CK mappings for many rules.
 
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
 
 
 ### New Features
@@ -267,8 +725,8 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
   
 ### Raw diffs
 
-  - [capa v1.4.1...v1.5.0](https://github.com/fireeye/capa/compare/v1.4.1...v1.5.0)
-  - [capa-rules v1.4.0...v1.5.0](https://github.com/fireeye/capa-rules/compare/v1.4.0...v1.5.0)
+  - [capa v1.4.1...v1.5.0](https://github.com/mandiant/capa/compare/v1.4.1...v1.5.0)
+  - [capa-rules v1.4.0...v1.5.0](https://github.com/mandiant/capa-rules/compare/v1.4.0...v1.5.0)
 
 ## v1.4.1 (2020-10-23)
 
@@ -280,8 +738,8 @@ This release fixes an issue building capa on our CI server, which prevented us f
   
 ### Raw diffs
 
-  - [capa v1.4.0...v1.4.1](https://github.com/fireeye/capa/compare/v1.4.0...v1.4.1)
-  - [capa-rules v1.4.0...v1.4.1](https://github.com/fireeye/capa-rules/compare/v1.4.0...v1.4.1)  
+  - [capa v1.4.0...v1.4.1](https://github.com/mandiant/capa/compare/v1.4.0...v1.4.1)
+  - [capa-rules v1.4.0...v1.4.1](https://github.com/mandiant/capa-rules/compare/v1.4.0...v1.4.1)  
 
 ## v1.4.0 (2020-10-23)
 
@@ -292,7 +750,7 @@ This capa release includes changes to the rule parsing, enhanced feature extract
 
 @dzbeck added [Malware Behavior Catalog](https://github.com/MBCProject/mbc-markdown) (MBC) and ATT&CK mappings for 86 rules.
 
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
 
 ### New features
 
@@ -395,8 +853,8 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
 
 ### Raw diffs
 
-  - [capa v1.3.0...v1.4.0](https://github.com/fireeye/capa/compare/v1.3.0...v1.4.0)
-  - [capa-rules v1.3.0...v1.4.0](https://github.com/fireeye/capa-rules/compare/v1.3.0...v1.4.0)
+  - [capa v1.3.0...v1.4.0](https://github.com/mandiant/capa/compare/v1.3.0...v1.4.0)
+  - [capa-rules v1.3.0...v1.4.0](https://github.com/mandiant/capa-rules/compare/v1.3.0...v1.4.0)
 
 ## v1.3.0 (2020-09-14)
 
@@ -410,7 +868,7 @@ This release brings newly updated mappings to the [Malware Behavior Catalog vers
   - @weslambert
   - @stevemk14ebr 
   
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
 
 ### Key changes to IDA Plugin
 
@@ -420,9 +878,9 @@ The IDA Pro integration is now distributed as a real plugin, instead of a script
   - updates distributed PyPI/`pip install --upgrade` without touching your `%IDADIR%`
   - generally doing thing the "right way"
 
-How to get this new version? Its easy: download [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory and update your capa installation (incidentally, this is a good opportunity to migrate to `pip install flare-capa` instead of git checkouts). Now you should see the plugin listed in the `Edit > Plugins > FLARE capa explorer` menu in IDA. 
+How to get this new version? Its easy: download [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory and update your capa installation (incidentally, this is a good opportunity to migrate to `pip install flare-capa` instead of git checkouts). Now you should see the plugin listed in the `Edit > Plugins > FLARE capa explorer` menu in IDA. 
 
-Please refer to the plugin [readme](https://github.com/fireeye/capa/blob/master/capa/ida/plugin/README.md) for additional information on installing and using the IDA Pro plugin.
+Please refer to the plugin [readme](https://github.com/mandiant/capa/blob/master/capa/ida/plugin/README.md) for additional information on installing and using the IDA Pro plugin.
 
 Please open an issue in this repository if you notice anything weird.
  
@@ -467,8 +925,8 @@ Please open an issue in this repository if you notice anything weird.
 
 ### Raw diffs
 
-  - [capa v1.2.0...v1.3.0](https://github.com/fireeye/capa/compare/v1.2.0...v1.3.0)
-  - [capa-rules v1.2.0...v1.3.0](https://github.com/fireeye/capa-rules/compare/v1.2.0...v1.3.0)
+  - [capa v1.2.0...v1.3.0](https://github.com/mandiant/capa/compare/v1.2.0...v1.3.0)
+  - [capa-rules v1.2.0...v1.3.0](https://github.com/mandiant/capa-rules/compare/v1.2.0...v1.3.0)
 
 ## v1.2.0 (2020-08-31)
 
@@ -484,9 +942,9 @@ We received contributions from ten reverse engineers, including five new ones:
   - @edeca
   - @winniepe 
   
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/).
-Report issues on our [issue tracker](https://github.com/fireeye/capa/issues)
-and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/).
+Report issues on our [issue tracker](https://github.com/mandiant/capa/issues)
+and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
  
 ### New features
 
@@ -565,8 +1023,8 @@ and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/)
 
 ### Raw diffs
 
-  - [capa v1.1.0...v1.2.0](https://github.com/fireeye/capa/compare/v1.1.0...v1.2.0)
-  - [capa-rules v1.1.0...v1.2.0](https://github.com/fireeye/capa-rules/compare/v1.1.0...v1.2.0)
+  - [capa v1.1.0...v1.2.0](https://github.com/mandiant/capa/compare/v1.1.0...v1.2.0)
+  - [capa-rules v1.1.0...v1.2.0](https://github.com/mandiant/capa-rules/compare/v1.1.0...v1.2.0)
 
 ## v1.1.0 (2020-08-05)
 
@@ -579,7 +1037,7 @@ We received contributions from eight reverse engineers, including four new ones:
   - @bitsofbinary
   - @threathive
   
-Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
+Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
   
 ### New features
 
@@ -652,5 +1110,5 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
 
 ### Raw diffs
 
-  - [capa v1.0.0...v1.1.0](https://github.com/fireeye/capa/compare/v1.0.0...v1.1.0)
-  - [capa-rules v1.0.0...v1.1.0](https://github.com/fireeye/capa-rules/compare/v1.0.0...v1.1.0)
+  - [capa v1.0.0...v1.1.0](https://github.com/mandiant/capa/compare/v1.0.0...v1.1.0)
+  - [capa-rules v1.0.0...v1.1.0](https://github.com/mandiant/capa-rules/compare/v1.0.0...v1.1.0)

LICENSE.txt

@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright (C) 2020 FireEye, Inc.
+   Copyright (C) 2020 Mandiant, Inc.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,17 +1,20 @@
-![capa](.github/logo.png)
+![capa](https://github.com/mandiant/capa/blob/master/.github/logo.png)
 
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/flare-capa)](https://pypi.org/project/flare-capa)
-[![Last release](https://img.shields.io/github/v/release/fireeye/capa)](https://github.com/fireeye/capa/releases)
-[![Number of rules](https://img.shields.io/badge/rules-485-blue.svg)](https://github.com/fireeye/capa-rules)
-[![CI status](https://github.com/fireeye/capa/workflows/CI/badge.svg)](https://github.com/fireeye/capa/actions?query=workflow%3ACI+event%3Apush+branch%3Amaster)
-[![Downloads](https://img.shields.io/github/downloads/fireeye/capa/total)](https://github.com/fireeye/capa/releases)
+[![Last release](https://img.shields.io/github/v/release/mandiant/capa)](https://github.com/mandiant/capa/releases)
+[![Number of rules](https://img.shields.io/badge/rules-658-blue.svg)](https://github.com/mandiant/capa-rules)
+[![CI status](https://github.com/mandiant/capa/workflows/CI/badge.svg)](https://github.com/mandiant/capa/actions?query=workflow%3ACI+event%3Apush+branch%3Amaster)
+[![Downloads](https://img.shields.io/github/downloads/mandiant/capa/total)](https://github.com/mandiant/capa/releases)
 [![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE.txt)
 
 capa detects capabilities in executable files.
-You run it against a PE file or shellcode and it tells you what it thinks the program can do.
+You run it against a PE, ELF, or shellcode file and it tells you what it thinks the program can do.
 For example, it might suggest that the file is a backdoor, is capable of installing services, or relies on HTTP to communicate.
 
-Check out the overview in our first [capa blog post](https://www.fireeye.com/blog/threat-research/2020/07/capa-automatically-identify-malware-capabilities.html).
+Check out:
+- the overview in our first [capa blog post](https://www.fireeye.com/blog/threat-research/2020/07/capa-automatically-identify-malware-capabilities.html)
+- the major version 2.0 updates described in our [second blog post](https://www.fireeye.com/blog/threat-research/2021/07/capa-2-better-stronger-faster.html)
+- the major version 3.0 (ELF support) described in the [third blog post](https://www.fireeye.com/blog/threat-research/2021/09/elfant-in-the-room-capa-v3.html)
 
 ```
 $ capa.exe suspicious.exe
@@ -63,18 +66,11 @@ $ capa.exe suspicious.exe
 
 # download and usage
 
-Download stable releases of the standalone capa binaries [here](https://github.com/fireeye/capa/releases). You can run the standalone binaries without installation. capa is a command line tool that should be run from the terminal.
+Download stable releases of the standalone capa binaries [here](https://github.com/mandiant/capa/releases). You can run the standalone binaries without installation. capa is a command line tool that should be run from the terminal.
 
-<!--
-Alternatively, you can fetch a nightly build of a standalone binary from one of the following links. These are built using the latest development branch.
-- Windows 64bit: TODO
-- Linux: TODO
-- OSX: TODO
--->
+To use capa as a library or integrate with another tool, see [doc/installation.md](https://github.com/mandiant/capa/blob/master/doc/installation.md) for further setup instructions.
 
-To use capa as a library or integrate with another tool, see [doc/installation.md](doc/installation.md) for further setup instructions.
-
-For more information about how to use capa, see [doc/usage.md](doc/usage.md).
+For more information about how to use capa, see [doc/usage.md](https://github.com/mandiant/capa/blob/master/doc/usage.md).
 
 # example
 
@@ -91,11 +87,11 @@ This is useful for at least two reasons:
   - it shows where within the binary an experienced analyst might study with IDA Pro
 
 ```
-λ capa.exe suspicious.exe -vv
+$ capa.exe suspicious.exe -vv
 ...
 execute shell command and capture output
 namespace   c2/shell
-author      matthew.williams@fireeye.com
+author      matthew.williams@mandiant.com
 scope       function
 att&ck      Execution::Command and Scripting Interpreter::Windows Command Shell [T1059.003]
 references  https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/ns-processthreadsapi-startupinfoa
@@ -131,7 +127,7 @@ rule:
   meta:
     name: hash data with CRC32
     namespace: data-manipulation/checksum/crc32
-    author: moritz.raabe@fireeye.com
+    author: moritz.raabe@mandiant.com
     scope: function
     examples:
       - 2D3EDC218A90F03089CC01715A9F047F:0x403CBD
@@ -146,21 +142,24 @@ rule:
       - api: RtlComputeCrc32
 ```
 
-The [github.com/fireeye/capa-rules](https://github.com/fireeye/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
+The [github.com/mandiant/capa-rules](https://github.com/mandiant/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
 Please learn to write rules and contribute new entries as you find interesting techniques in malware.
 
-If you use IDA Pro, then you can use the [capa explorer](capa/ida/plugin/) plugin.
+If you use IDA Pro, then you can use the [capa explorer](https://github.com/mandiant/capa/tree/master/capa/ida/plugin) plugin.
 capa explorer helps you identify interesting areas of a program and build new capa rules using features extracted directly from your IDA Pro database.
 
-![capa + IDA Pro integration](doc/img/explorer_expanded.png)
+![capa + IDA Pro integration](https://github.com/mandiant/capa/blob/master/doc/img/explorer_expanded.png)
 
 # further information
 ## capa
-- [doc/installation](doc/installation.md)
-- [doc/usage](doc/usage.md)
-- [doc/limitations](doc/limitations.md)
-- [Contributing Guide](.github/CONTRIBUTING.md)
+- [Installation](https://github.com/mandiant/capa/blob/master/doc/installation.md)
+- [Usage](https://github.com/mandiant/capa/blob/master/doc/usage.md)
+- [Limitations](https://github.com/mandiant/capa/blob/master/doc/limitations.md)
+- [Contributing Guide](https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md)
 
 ## capa rules
-- [capa-rules repository](https://github.com/fireeye/capa-rules)
-- [capa-rules rule format](https://github.com/fireeye/capa-rules/blob/master/doc/format.md)
+- [capa-rules repository](https://github.com/mandiant/capa-rules)
+- [capa-rules rule format](https://github.com/mandiant/capa-rules/blob/master/doc/format.md)
+
+## capa testfiles
+The [capa-testfiles repository](https://github.com/mandiant/capa-testfiles) contains the data we use to test capa's code and rules

capa/engine.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -8,11 +8,28 @@
 
 import copy
 import collections
+from typing import TYPE_CHECKING, Set, Dict, List, Tuple, Mapping, Iterable
 
-import capa.features
+import capa.perf
+import capa.features.common
+from capa.features.common import Result, Feature
+
+if TYPE_CHECKING:
+    # circular import, otherwise
+    import capa.rules
 
 
-class Statement(object):
+# a collection of features and the locations at which they are found.
+#
+# used throughout matching as the context in which features are searched:
+# to check if a feature exists, do: `Number(0x10) in features`.
+# to collect the locations of a feature, do: `features[Number(0x10)]`
+#
+# aliased here so that the type can be documented and xref'd.
+FeatureSet = Dict[Feature, Set[int]]
+
+
+class Statement:
     """
     superclass for structural nodes, such as and/or/not.
     this exists to provide a default impl for `__str__` and `__repr__`,
@@ -33,15 +50,12 @@ class Statement(object):
     def __repr__(self):
         return str(self)
 
-    def evaluate(self, ctx):
+    def evaluate(self, features: FeatureSet, short_circuit=True) -> Result:
         """
         classes that inherit `Statement` must implement `evaluate`
 
         args:
-          ctx (defaultdict[Feature, set[VA]])
-
-        returns:
-          Result
+            short_circuit (bool): if true, then statements like and/or/some may short circuit.
         """
         raise NotImplementedError()
 
@@ -50,7 +64,7 @@ class Statement(object):
             yield self.child
 
         if hasattr(self, "children"):
-            for child in self.children:
+            for child in getattr(self, "children"):
                 yield child
 
     def replace_child(self, existing, new):
@@ -59,75 +73,76 @@ class Statement(object):
                 self.child = new
 
         if hasattr(self, "children"):
-            for i, child in enumerate(self.children):
+            children = getattr(self, "children")
+            for i, child in enumerate(children):
                 if child is existing:
-                    self.children[i] = new
-
-
-class Result(object):
-    """
-    represents the results of an evaluation of statements against features.
-
-    instances of this class should behave like a bool,
-    e.g. `assert Result(True, ...) == True`
-
-    instances track additional metadata about evaluation results.
-    they contain references to the statement node (e.g. an And statement),
-     as well as the children Result instances.
-
-    we need this so that we can render the tree of expressions and their results.
-    """
-
-    def __init__(self, success, statement, children, locations=None):
-        """
-        args:
-          success (bool)
-          statement (capa.engine.Statement or capa.features.Feature)
-          children (list[Result])
-          locations (iterable[VA])
-        """
-        super(Result, self).__init__()
-        self.success = success
-        self.statement = statement
-        self.children = children
-        self.locations = locations if locations is not None else ()
-
-    def __eq__(self, other):
-        if isinstance(other, bool):
-            return self.success == other
-        return False
-
-    def __bool__(self):
-        return self.success
-
-    def __nonzero__(self):
-        return self.success
+                    children[i] = new
 
 
 class And(Statement):
-    """match if all of the children evaluate to True."""
+    """
+    match if all of the children evaluate to True.
+
+    the order of evaluation is dictated by the property
+    `And.children` (type: List[Statement|Feature]).
+    a query optimizer may safely manipulate the order of these children.
+    """
 
     def __init__(self, children, description=None):
         super(And, self).__init__(description=description)
         self.children = children
 
-    def evaluate(self, ctx):
-        results = [child.evaluate(ctx) for child in self.children]
-        success = all(results)
-        return Result(success, self, results)
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.and"] += 1
+
+        if short_circuit:
+            results = []
+            for child in self.children:
+                result = child.evaluate(ctx, short_circuit=short_circuit)
+                results.append(result)
+                if not result:
+                    # short circuit
+                    return Result(False, self, results)
+
+            return Result(True, self, results)
+        else:
+            results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
+            success = all(results)
+            return Result(success, self, results)
 
 
 class Or(Statement):
-    """match if any of the children evaluate to True."""
+    """
+    match if any of the children evaluate to True.
+
+    the order of evaluation is dictated by the property
+    `Or.children` (type: List[Statement|Feature]).
+    a query optimizer may safely manipulate the order of these children.
+    """
 
     def __init__(self, children, description=None):
         super(Or, self).__init__(description=description)
         self.children = children
 
-    def evaluate(self, ctx):
-        results = [child.evaluate(ctx) for child in self.children]
-        success = any(results)
-        return Result(success, self, results)
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.or"] += 1
+
+        if short_circuit:
+            results = []
+            for child in self.children:
+                result = child.evaluate(ctx, short_circuit=short_circuit)
+                results.append(result)
+                if result:
+                    # short circuit as soon as we hit one match
+                    return Result(True, self, results)
+
+            return Result(False, self, results)
+        else:
+            results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
+            success = any(results)
+            return Result(success, self, results)
 
 
 class Not(Statement):
@@ -137,28 +152,55 @@ class Not(Statement):
         super(Not, self).__init__(description=description)
         self.child = child
 
-    def evaluate(self, ctx):
-        results = [self.child.evaluate(ctx)]
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.not"] += 1
+
+        results = [self.child.evaluate(ctx, short_circuit=short_circuit)]
         success = not results[0]
         return Result(success, self, results)
 
 
 class Some(Statement):
-    """match if at least N of the children evaluate to True."""
+    """
+    match if at least N of the children evaluate to True.
+
+    the order of evaluation is dictated by the property
+    `Some.children` (type: List[Statement|Feature]).
+    a query optimizer may safely manipulate the order of these children.
+    """
 
     def __init__(self, count, children, description=None):
         super(Some, self).__init__(description=description)
         self.count = count
         self.children = children
 
-    def evaluate(self, ctx):
-        results = [child.evaluate(ctx) for child in self.children]
-        # note that here we cast the child result as a bool
-        # because we've overridden `__bool__` above.
-        #
-        # we can't use `if child is True` because the instance is not True.
-        success = sum([1 for child in results if bool(child) is True]) >= self.count
-        return Result(success, self, results)
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.some"] += 1
+
+        if short_circuit:
+            results = []
+            satisfied_children_count = 0
+            for child in self.children:
+                result = child.evaluate(ctx, short_circuit=short_circuit)
+                results.append(result)
+                if result:
+                    satisfied_children_count += 1
+
+                if satisfied_children_count >= self.count:
+                    # short circuit as soon as we hit the threshold
+                    return Result(True, self, results)
+
+            return Result(False, self, results)
+        else:
+            results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
+            # note that here we cast the child result as a bool
+            # because we've overridden `__bool__` above.
+            #
+            # we can't use `if child is True` because the instance is not True.
+            success = sum([1 for child in results if bool(child) is True]) >= self.count
+            return Result(success, self, results)
 
 
 class Range(Statement):
@@ -170,7 +212,10 @@ class Range(Statement):
         self.min = min if min is not None else 0
         self.max = max if max is not None else (1 << 64 - 1)
 
-    def evaluate(self, ctx):
+    def evaluate(self, ctx, **kwargs):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.range"] += 1
+
         count = len(ctx.get(self.child, []))
         if self.min == 0 and count == 0:
             return Result(True, self, [])
@@ -195,54 +240,61 @@ class Subscope(Statement):
         self.scope = scope
         self.child = child
 
-    def evaluate(self, ctx):
+    def evaluate(self, ctx, **kwargs):
         raise ValueError("cannot evaluate a subscope directly!")
 
 
-def topologically_order_rules(rules):
+# mapping from rule name to list of: (location of match, result object)
+#
+# used throughout matching and rendering to collection the results
+#  of statement evaluation and their locations.
+#
+# to check if a rule matched, do: `"TCP client" in matches`.
+# to find where a rule matched, do: `map(first, matches["TCP client"])`
+# to see how a rule matched, do:
+#
+#     for address, match_details in matches["TCP client"]:
+#         inspect(match_details)
+#
+# aliased here so that the type can be documented and xref'd.
+MatchResults = Mapping[str, List[Tuple[int, Result]]]
+
+
+def index_rule_matches(features: FeatureSet, rule: "capa.rules.Rule", locations: Iterable[int]):
     """
-    order the given rules such that dependencies show up before dependents.
-    this means that as we match rules, we can add features for the matches, and these
-     will be matched by subsequent rules if they follow this order.
+    record into the given featureset that the given rule matched at the given locations.
 
-    assumes that the rule dependency graph is a DAG.
+    naively, this is just adding a MatchedRule feature;
+    however, we also want to record matches for the rule's namespaces.
+
+    updates `features` in-place. doesn't modify the remaining arguments.
     """
-    # we evaluate `rules` multiple times, so if its a generator, realize it into a list.
-    rules = list(rules)
-    namespaces = capa.rules.index_rules_by_namespace(rules)
-    rules = {rule.name: rule for rule in rules}
-    seen = set([])
-    ret = []
-
-    def rec(rule):
-        if rule.name in seen:
-            return
-
-        for dep in rule.get_dependencies(namespaces):
-            rec(rules[dep])
-
-        ret.append(rule)
-        seen.add(rule.name)
-
-    for rule in rules.values():
-        rec(rule)
-
-    return ret
+    features[capa.features.common.MatchedRule(rule.name)].update(locations)
+    namespace = rule.meta.get("namespace")
+    if namespace:
+        while namespace:
+            features[capa.features.common.MatchedRule(namespace)].update(locations)
+            namespace, _, _ = namespace.rpartition("/")
 
 
-def match(rules, features, va):
+def match(rules: List["capa.rules.Rule"], features: FeatureSet, va: int) -> Tuple[FeatureSet, MatchResults]:
     """
-    Args:
-      rules (List[capa.rules.Rule]): these must already be ordered topologically by dependency.
-      features (Mapping[capa.features.Feature, int]):
-      va (int): location of the features
+    match the given rules against the given features,
+    returning an updated set of features and the matches.
 
-    Returns:
-      Tuple[List[capa.features.Feature], Dict[str, Tuple[int, capa.engine.Result]]]: two-tuple with entries:
-        - list of features used for matching (which may be greater than argument, due to rule match features), and
-        - mapping from rule name to (location of match, result object)
+    the updated features are just like the input,
+    but extended to include the match features (e.g. names of rules that matched).
+    the given feature set is not modified; an updated copy is returned.
+
+    the given list of rules must be ordered topologically by dependency,
+    or else `match` statements will not be handled correctly.
+
+    this routine should be fairly optimized, but is not guaranteed to be the fastest matcher possible.
+    it has a particularly convenient signature: (rules, features) -> matches
+    other strategies can be imagined that match differently; implement these elsewhere.
+    specifically, this routine does "top down" matching of the given rules against the feature set.
     """
-    results = collections.defaultdict(list)
+    results = collections.defaultdict(list)  # type: MatchResults
 
     # copy features so that we can modify it
     # without affecting the caller (keep this function pure)
@@ -251,15 +303,22 @@ def match(rules, features, va):
     features = collections.defaultdict(set, copy.copy(features))
 
     for rule in rules:
-        res = rule.evaluate(features)
+        res = rule.evaluate(features, short_circuit=True)
         if res:
-            results[rule.name].append((va, res))
-            features[capa.features.MatchedRule(rule.name)].add(va)
+            # we first matched the rule with short circuiting enabled.
+            # this is much faster than without short circuiting.
+            # however, we want to collect all results thoroughly,
+            # so once we've found a match quickly,
+            # go back and capture results without short circuiting.
+            res = rule.evaluate(features, short_circuit=False)
 
-            namespace = rule.meta.get("namespace")
-            if namespace:
-                while namespace:
-                    features[capa.features.MatchedRule(namespace)].add(va)
-                    namespace, _, _ = namespace.rpartition("/")
+            # sanity check
+            assert bool(res) is True
+
+            results[rule.name].append((va, res))
+            # we need to update the current `features`
+            # because subsequent iterations of this loop may use newly added features,
+            # such as rule or namespace matches.
+            index_rule_matches(features, rule, [va])
 
     return (features, results)

capa/features/__init__.py

@@ -1,236 +0,0 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-
-import re
-import sys
-import codecs
-import logging
-
-import capa.engine
-
-logger = logging.getLogger(__name__)
-MAX_BYTES_FEATURE_SIZE = 0x100
-
-# thunks may be chained so we specify a delta to control the depth to which these chains are explored
-THUNK_CHAIN_DEPTH_DELTA = 5
-
-# identifiers for supported architectures names that tweak a feature
-# for example, offset/x32
-ARCH_X32 = "x32"
-ARCH_X64 = "x64"
-VALID_ARCH = (ARCH_X32, ARCH_X64)
-
-
-def bytes_to_str(b):
-    if sys.version_info[0] >= 3:
-        return str(codecs.encode(b, "hex").decode("utf-8"))
-    else:
-        return codecs.encode(b, "hex")
-
-
-def hex_string(h):
-    """ render hex string e.g. "0a40b1" as "0A 40 B1" """
-    return " ".join(h[i : i + 2] for i in range(0, len(h), 2)).upper()
-
-
-def escape_string(s):
-    """escape special characters"""
-    s = repr(s)
-    if not s.startswith(('"', "'")):
-        # u'hello\r\nworld' -> hello\\r\\nworld
-        s = s[2:-1]
-    else:
-        # 'hello\r\nworld' -> hello\\r\\nworld
-        s = s[1:-1]
-    s = s.replace("\\'", "'")  # repr() may escape "'" in some edge cases, remove
-    s = s.replace('"', '\\"')  # repr() does not escape '"', add
-    return s
-
-
-class Feature(object):
-    def __init__(self, value, arch=None, description=None):
-        """
-        Args:
-          value (any): the value of the feature, such as the number or string.
-          arch (str): one of the VALID_ARCH values, or None.
-            When None, then the feature applies to any architecture.
-            Modifies the feature name from `feature` to `feature/arch`, like `offset/x32`.
-          description (str): a human-readable description that explains the feature value.
-        """
-        super(Feature, self).__init__()
-
-        if arch is not None:
-            if arch not in VALID_ARCH:
-                raise ValueError("arch '%s' must be one of %s" % (arch, VALID_ARCH))
-            self.name = self.__class__.__name__.lower() + "/" + arch
-        else:
-            self.name = self.__class__.__name__.lower()
-
-        self.value = value
-        self.arch = arch
-        self.description = description
-
-    def __hash__(self):
-        return hash((self.name, self.value, self.arch))
-
-    def __eq__(self, other):
-        return self.name == other.name and self.value == other.value and self.arch == other.arch
-
-    def get_value_str(self):
-        """
-        render the value of this feature, for use by `__str__` and friends.
-        subclasses should override to customize the rendering.
-
-        Returns: any
-        """
-        return self.value
-
-    def __str__(self):
-        if self.value is not None:
-            if self.description:
-                return "%s(%s = %s)" % (self.name, self.get_value_str(), self.description)
-            else:
-                return "%s(%s)" % (self.name, self.get_value_str())
-        else:
-            return "%s" % self.name
-
-    def __repr__(self):
-        return str(self)
-
-    def evaluate(self, ctx):
-        return capa.engine.Result(self in ctx, self, [], locations=ctx.get(self, []))
-
-    def freeze_serialize(self):
-        if self.arch is not None:
-            return (self.__class__.__name__, [self.value, {"arch": self.arch}])
-        else:
-            return (self.__class__.__name__, [self.value])
-
-    @classmethod
-    def freeze_deserialize(cls, args):
-        # as you can see below in code,
-        # if the last argument is a dictionary,
-        # consider it to be kwargs passed to the feature constructor.
-        if len(args) == 1:
-            return cls(*args)
-        elif isinstance(args[-1], dict):
-            kwargs = args[-1]
-            args = args[:-1]
-            return cls(*args, **kwargs)
-
-
-class MatchedRule(Feature):
-    def __init__(self, value, description=None):
-        super(MatchedRule, self).__init__(value, description=description)
-        self.name = "match"
-
-
-class Characteristic(Feature):
-    def __init__(self, value, description=None):
-        super(Characteristic, self).__init__(value, description=description)
-
-
-class String(Feature):
-    def __init__(self, value, description=None):
-        super(String, self).__init__(value, description=description)
-
-
-class Regex(String):
-    def __init__(self, value, description=None):
-        super(Regex, self).__init__(value, description=description)
-        pat = self.value[len("/") : -len("/")]
-        flags = re.DOTALL
-        if value.endswith("/i"):
-            pat = self.value[len("/") : -len("/i")]
-            flags |= re.IGNORECASE
-        try:
-            self.re = re.compile(pat, flags)
-        except re.error:
-            if value.endswith("/i"):
-                value = value[: -len("i")]
-            raise ValueError(
-                "invalid regular expression: %s it should use Python syntax, try it at https://pythex.org" % value
-            )
-
-    def evaluate(self, ctx):
-        for feature, locations in ctx.items():
-            if not isinstance(feature, (capa.features.String,)):
-                continue
-
-            # `re.search` finds a match anywhere in the given string
-            # which implies leading and/or trailing whitespace.
-            # using this mode cleans is more convenient for rule authors,
-            # so that they don't have to prefix/suffix their terms like: /.*foo.*/.
-            if self.re.search(feature.value):
-                # unlike other features, we cannot return put a reference to `self` directly in a `Result`.
-                # this is because `self` may match on many strings, so we can't stuff the matched value into it.
-                # instead, return a new instance that has a reference to both the regex and the matched value.
-                # see #262.
-                return capa.engine.Result(True, _MatchedRegex(self, feature.value), [], locations=locations)
-
-        return capa.engine.Result(False, _MatchedRegex(self, None), [])
-
-    def __str__(self):
-        return "regex(string =~ %s)" % self.value
-
-
-class _MatchedRegex(Regex):
-    """
-    this represents a specific instance of a regular expression feature match.
-    treat it the same as a `Regex` except it has the `match` field that contains the complete string that matched.
-
-    note: this type should only ever be constructed by `Regex.evaluate()`. it is not part of the public API.
-    """
-
-    def __init__(self, regex, match):
-        """
-        args:
-          regex (Regex): the regex feature that matches
-          match (string|None): the matching string or None if it doesn't match
-        """
-        super(_MatchedRegex, self).__init__(regex.value, description=regex.description)
-        # we want this to collide with the name of `Regex` above,
-        # so that it works nicely with the renderers.
-        self.name = "regex"
-        # this may be None if the regex doesn't match
-        self.match = match
-
-    def __str__(self):
-        return 'regex(string =~ %s, matched = "%s")' % (self.value, self.match)
-
-
-class StringFactory(object):
-    def __new__(self, value, description=None):
-        if value.startswith("/") and (value.endswith("/") or value.endswith("/i")):
-            return Regex(value, description=description)
-        return String(value, description=description)
-
-
-class Bytes(Feature):
-    def __init__(self, value, description=None):
-        super(Bytes, self).__init__(value, description=description)
-
-    def evaluate(self, ctx):
-        for feature, locations in ctx.items():
-            if not isinstance(feature, (capa.features.Bytes,)):
-                continue
-
-            if feature.value.startswith(self.value):
-                return capa.engine.Result(True, self, [], locations=locations)
-
-        return capa.engine.Result(False, self, [])
-
-    def get_value_str(self):
-        return hex_string(bytes_to_str(self.value))
-
-    def freeze_serialize(self):
-        return (self.__class__.__name__, [bytes_to_str(self.value).upper()])
-
-    @classmethod
-    def freeze_deserialize(cls, args):
-        return cls(*[codecs.decode(x, "hex") for x in args])

capa/features/basicblock.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,7 +6,7 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-from capa.features import Feature
+from capa.features.common import Feature
 
 
 class BasicBlock(Feature):

capa/features/common.py

@@ -0,0 +1,449 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+import re
+import codecs
+import logging
+import collections
+from typing import TYPE_CHECKING, Set, Dict, List, Union
+
+if TYPE_CHECKING:
+    # circular import, otherwise
+    import capa.engine
+
+import capa.perf
+import capa.features
+import capa.features.extractors.elf
+
+logger = logging.getLogger(__name__)
+MAX_BYTES_FEATURE_SIZE = 0x100
+
+# thunks may be chained so we specify a delta to control the depth to which these chains are explored
+THUNK_CHAIN_DEPTH_DELTA = 5
+
+
+def bytes_to_str(b: bytes) -> str:
+    return str(codecs.encode(b, "hex").decode("utf-8"))
+
+
+def hex_string(h: str) -> str:
+    """render hex string e.g. "0a40b1" as "0A 40 B1" """
+    return " ".join(h[i : i + 2] for i in range(0, len(h), 2)).upper()
+
+
+def escape_string(s: str) -> str:
+    """escape special characters"""
+    s = repr(s)
+    if not s.startswith(('"', "'")):
+        # u'hello\r\nworld' -> hello\\r\\nworld
+        s = s[2:-1]
+    else:
+        # 'hello\r\nworld' -> hello\\r\\nworld
+        s = s[1:-1]
+    s = s.replace("\\'", "'")  # repr() may escape "'" in some edge cases, remove
+    s = s.replace('"', '\\"')  # repr() does not escape '"', add
+    return s
+
+
+class Result:
+    """
+    represents the results of an evaluation of statements against features.
+
+    instances of this class should behave like a bool,
+    e.g. `assert Result(True, ...) == True`
+
+    instances track additional metadata about evaluation results.
+    they contain references to the statement node (e.g. an And statement),
+     as well as the children Result instances.
+
+    we need this so that we can render the tree of expressions and their results.
+    """
+
+    def __init__(
+        self,
+        success: bool,
+        statement: Union["capa.engine.Statement", "Feature"],
+        children: List["Result"],
+        locations=None,
+    ):
+        """
+        args:
+          success (bool)
+          statement (capa.engine.Statement or capa.features.Feature)
+          children (list[Result])
+          locations (iterable[VA])
+        """
+        super(Result, self).__init__()
+        self.success = success
+        self.statement = statement
+        self.children = children
+        self.locations = locations if locations is not None else ()
+
+    def __eq__(self, other):
+        if isinstance(other, bool):
+            return self.success == other
+        return False
+
+    def __bool__(self):
+        return self.success
+
+    def __nonzero__(self):
+        return self.success
+
+
+class Feature:
+    def __init__(self, value: Union[str, int, bytes], bitness=None, description=None):
+        """
+        Args:
+          value (any): the value of the feature, such as the number or string.
+          bitness (str): one of the VALID_BITNESS values, or None.
+            When None, then the feature applies to any bitness.
+            Modifies the feature name from `feature` to `feature/bitness`, like `offset/x32`.
+          description (str): a human-readable description that explains the feature value.
+        """
+        super(Feature, self).__init__()
+
+        if bitness is not None:
+            if bitness not in VALID_BITNESS:
+                raise ValueError("bitness '%s' must be one of %s" % (bitness, VALID_BITNESS))
+            self.name = self.__class__.__name__.lower() + "/" + bitness
+        else:
+            self.name = self.__class__.__name__.lower()
+
+        self.value = value
+        self.bitness = bitness
+        self.description = description
+
+    def __hash__(self):
+        return hash((self.name, self.value, self.bitness))
+
+    def __eq__(self, other):
+        return self.name == other.name and self.value == other.value and self.bitness == other.bitness
+
+    def get_value_str(self) -> str:
+        """
+        render the value of this feature, for use by `__str__` and friends.
+        subclasses should override to customize the rendering.
+
+        Returns: any
+        """
+        return str(self.value)
+
+    def __str__(self):
+        if self.value is not None:
+            if self.description:
+                return "%s(%s = %s)" % (self.name, self.get_value_str(), self.description)
+            else:
+                return "%s(%s)" % (self.name, self.get_value_str())
+        else:
+            return "%s" % self.name
+
+    def __repr__(self):
+        return str(self)
+
+    def evaluate(self, ctx: Dict["Feature", Set[int]], **kwargs) -> Result:
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature." + self.name] += 1
+        return Result(self in ctx, self, [], locations=ctx.get(self, []))
+
+    def freeze_serialize(self):
+        if self.bitness is not None:
+            return (self.__class__.__name__, [self.value, {"bitness": self.bitness}])
+        else:
+            return (self.__class__.__name__, [self.value])
+
+    @classmethod
+    def freeze_deserialize(cls, args):
+        # as you can see below in code,
+        # if the last argument is a dictionary,
+        # consider it to be kwargs passed to the feature constructor.
+        if len(args) == 1:
+            return cls(*args)
+        elif isinstance(args[-1], dict):
+            kwargs = args[-1]
+            args = args[:-1]
+            return cls(*args, **kwargs)
+
+
+class MatchedRule(Feature):
+    def __init__(self, value: str, description=None):
+        super(MatchedRule, self).__init__(value, description=description)
+        self.name = "match"
+
+
+class Characteristic(Feature):
+    def __init__(self, value: str, description=None):
+
+        super(Characteristic, self).__init__(value, description=description)
+
+
+class String(Feature):
+    def __init__(self, value: str, description=None):
+        super(String, self).__init__(value, description=description)
+
+
+class Substring(String):
+    def __init__(self, value: str, description=None):
+        super(Substring, self).__init__(value, description=description)
+        self.value = value
+
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.substring"] += 1
+
+        # mapping from string value to list of locations.
+        # will unique the locations later on.
+        matches = collections.defaultdict(list)
+
+        for feature, locations in ctx.items():
+            if not isinstance(feature, (String,)):
+                continue
+
+            if not isinstance(feature.value, str):
+                # this is a programming error: String should only contain str
+                raise ValueError("unexpected feature value type")
+
+            if self.value in feature.value:
+                matches[feature.value].extend(locations)
+                if short_circuit:
+                    # we found one matching string, thats sufficient to match.
+                    # don't collect other matching strings in this mode.
+                    break
+
+        if matches:
+            # finalize: defaultdict -> dict
+            # which makes json serialization easier
+            matches = dict(matches)
+
+            # collect all locations
+            locations = set()
+            for s in matches.keys():
+                matches[s] = list(set(matches[s]))
+                locations.update(matches[s])
+
+            # unlike other features, we cannot return put a reference to `self` directly in a `Result`.
+            # this is because `self` may match on many strings, so we can't stuff the matched value into it.
+            # instead, return a new instance that has a reference to both the substring and the matched values.
+            return Result(True, _MatchedSubstring(self, matches), [], locations=locations)
+        else:
+            return Result(False, _MatchedSubstring(self, None), [])
+
+    def __str__(self):
+        return "substring(%s)" % self.value
+
+
+class _MatchedSubstring(Substring):
+    """
+    this represents specific match instances of a substring feature.
+    treat it the same as a `Substring` except it has the `matches` field that contains the complete strings that matched.
+
+    note: this type should only ever be constructed by `Substring.evaluate()`. it is not part of the public API.
+    """
+
+    def __init__(self, substring: Substring, matches):
+        """
+        args:
+          substring (Substring): the substring feature that matches.
+          match (Dict[string, List[int]]|None): mapping from matching string to its locations.
+        """
+        super(_MatchedSubstring, self).__init__(str(substring.value), description=substring.description)
+        # we want this to collide with the name of `Substring` above,
+        # so that it works nicely with the renderers.
+        self.name = "substring"
+        # this may be None if the substring doesn't match
+        self.matches = matches
+
+    def __str__(self):
+        return 'substring("%s", matches = %s)' % (
+            self.value,
+            ", ".join(map(lambda s: '"' + s + '"', (self.matches or {}).keys())),
+        )
+
+
+class Regex(String):
+    def __init__(self, value: str, description=None):
+        super(Regex, self).__init__(value, description=description)
+        self.value = value
+
+        pat = self.value[len("/") : -len("/")]
+        flags = re.DOTALL
+        if value.endswith("/i"):
+            pat = self.value[len("/") : -len("/i")]
+            flags |= re.IGNORECASE
+        try:
+            self.re = re.compile(pat, flags)
+        except re.error:
+            if value.endswith("/i"):
+                value = value[: -len("i")]
+            raise ValueError(
+                "invalid regular expression: %s it should use Python syntax, try it at https://pythex.org" % value
+            )
+
+    def evaluate(self, ctx, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.regex"] += 1
+
+        # mapping from string value to list of locations.
+        # will unique the locations later on.
+        matches = collections.defaultdict(list)
+
+        for feature, locations in ctx.items():
+            if not isinstance(feature, (String,)):
+                continue
+
+            if not isinstance(feature.value, str):
+                # this is a programming error: String should only contain str
+                raise ValueError("unexpected feature value type")
+
+            # `re.search` finds a match anywhere in the given string
+            # which implies leading and/or trailing whitespace.
+            # using this mode cleans is more convenient for rule authors,
+            # so that they don't have to prefix/suffix their terms like: /.*foo.*/.
+            if self.re.search(feature.value):
+                matches[feature.value].extend(locations)
+                if short_circuit:
+                    # we found one matching string, thats sufficient to match.
+                    # don't collect other matching strings in this mode.
+                    break
+
+        if matches:
+            # finalize: defaultdict -> dict
+            # which makes json serialization easier
+            matches = dict(matches)
+
+            # collect all locations
+            locations = set()
+            for s in matches.keys():
+                matches[s] = list(set(matches[s]))
+                locations.update(matches[s])
+
+            # unlike other features, we cannot return put a reference to `self` directly in a `Result`.
+            # this is because `self` may match on many strings, so we can't stuff the matched value into it.
+            # instead, return a new instance that has a reference to both the regex and the matched values.
+            # see #262.
+            return Result(True, _MatchedRegex(self, matches), [], locations=locations)
+        else:
+            return Result(False, _MatchedRegex(self, None), [])
+
+    def __str__(self):
+        return "regex(string =~ %s)" % self.value
+
+
+class _MatchedRegex(Regex):
+    """
+    this represents specific match instances of a regular expression feature.
+    treat it the same as a `Regex` except it has the `matches` field that contains the complete strings that matched.
+
+    note: this type should only ever be constructed by `Regex.evaluate()`. it is not part of the public API.
+    """
+
+    def __init__(self, regex: Regex, matches):
+        """
+        args:
+          regex (Regex): the regex feature that matches.
+          match (Dict[string, List[int]]|None): mapping from matching string to its locations.
+        """
+        super(_MatchedRegex, self).__init__(str(regex.value), description=regex.description)
+        # we want this to collide with the name of `Regex` above,
+        # so that it works nicely with the renderers.
+        self.name = "regex"
+        # this may be None if the regex doesn't match
+        self.matches = matches
+
+    def __str__(self):
+        return "regex(string =~ %s, matches = %s)" % (
+            self.value,
+            ", ".join(map(lambda s: '"' + s + '"', (self.matches or {}).keys())),
+        )
+
+
+class StringFactory:
+    def __new__(cls, value: str, description=None):
+        if value.startswith("/") and (value.endswith("/") or value.endswith("/i")):
+            return Regex(value, description=description)
+        return String(value, description=description)
+
+
+class Bytes(Feature):
+    def __init__(self, value: bytes, description=None):
+        super(Bytes, self).__init__(value, description=description)
+        self.value = value
+
+    def evaluate(self, ctx, **kwargs):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.bytes"] += 1
+
+        for feature, locations in ctx.items():
+            if not isinstance(feature, (Bytes,)):
+                continue
+
+            if feature.value.startswith(self.value):
+                return Result(True, self, [], locations=locations)
+
+        return Result(False, self, [])
+
+    def get_value_str(self):
+        return hex_string(bytes_to_str(self.value))
+
+    def freeze_serialize(self):
+        return (self.__class__.__name__, [bytes_to_str(self.value).upper()])
+
+    @classmethod
+    def freeze_deserialize(cls, args):
+        return cls(*[codecs.decode(x, "hex") for x in args])
+
+
+# identifiers for supported bitness names that tweak a feature
+# for example, offset/x32
+BITNESS_X32 = "x32"
+BITNESS_X64 = "x64"
+VALID_BITNESS = (BITNESS_X32, BITNESS_X64)
+
+
+# other candidates here: https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#machine-types
+ARCH_I386 = "i386"
+ARCH_AMD64 = "amd64"
+VALID_ARCH = (ARCH_I386, ARCH_AMD64)
+
+
+class Arch(Feature):
+    def __init__(self, value: str, description=None):
+        super(Arch, self).__init__(value, description=description)
+        self.name = "arch"
+
+
+OS_WINDOWS = "windows"
+OS_LINUX = "linux"
+OS_MACOS = "macos"
+VALID_OS = {os.value for os in capa.features.extractors.elf.OS}
+VALID_OS.update({OS_WINDOWS, OS_LINUX, OS_MACOS})
+
+
+class OS(Feature):
+    def __init__(self, value: str, description=None):
+        super(OS, self).__init__(value, description=description)
+        self.name = "os"
+
+
+FORMAT_PE = "pe"
+FORMAT_ELF = "elf"
+VALID_FORMAT = (FORMAT_PE, FORMAT_ELF)
+
+
+class Format(Feature):
+    def __init__(self, value: str, description=None):
+        super(Format, self).__init__(value, description=description)
+        self.name = "format"
+
+
+def is_global_feature(feature):
+    """
+    is this a feature that is extracted at every scope?
+    today, these are OS and arch features.
+    """
+    return isinstance(feature, (OS, Arch))

capa/features/extractors/__init__.py

@@ -1,286 +0,0 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-
-import abc
-
-
-class FeatureExtractor(object):
-    """
-    FeatureExtractor defines the interface for fetching features from a sample.
-
-    There may be multiple backends that support fetching features for capa.
-    For example, we use vivisect by default, but also want to support saving
-     and restoring features from a JSON file.
-    When we restore the features, we'd like to use exactly the same matching logic
-     to find matching rules.
-    Therefore, we can define a FeatureExtractor that provides features from the
-     serialized JSON file and do matching without a binary analysis pass.
-    Also, this provides a way to hook in an IDA backend.
-
-    This class is not instantiated directly; it is the base class for other implementations.
-    """
-
-    __metaclass__ = abc.ABCMeta
-
-    def __init__(self):
-        #
-        # note: a subclass should define ctor parameters for its own use.
-        #  for example, the Vivisect feature extract might require the vw and/or path.
-        # this base class doesn't know what to do with that info, though.
-        #
-        super(FeatureExtractor, self).__init__()
-
-    @abc.abstractmethod
-    def get_base_address(self):
-        """
-        fetch the preferred load address at which the sample was analyzed.
-
-        returns: int
-        """
-        raise NotImplemented
-
-    @abc.abstractmethod
-    def extract_file_features(self):
-        """
-        extract file-scope features.
-
-        example::
-
-            extractor = VivisectFeatureExtractor(vw, path)
-            for feature, va in extractor.get_file_features():
-                print('0x%x: %s', va, feature)
-
-        yields:
-          Tuple[capa.features.Feature, int]: feature and its location
-        """
-        raise NotImplemented
-
-    @abc.abstractmethod
-    def get_functions(self):
-        """
-        enumerate the functions and provide opaque values that will
-         subsequently be provided to `.extract_function_features()`, etc.
-
-        by "opaque value", we mean that this can be any object, as long as it
-         provides enough context to `.extract_function_features()`.
-
-        the opaque value should support casting to int (`__int__`) for the function start address.
-
-        yields:
-          any: the opaque function value.
-        """
-        raise NotImplemented
-
-    @abc.abstractmethod
-    def extract_function_features(self, f):
-        """
-        extract function-scope features.
-        the arguments are opaque values previously provided by `.get_functions()`, etc.
-
-        example::
-
-            extractor = VivisectFeatureExtractor(vw, path)
-            for function in extractor.get_functions():
-                for feature, va in extractor.extract_function_features(function):
-                    print('0x%x: %s', va, feature)
-
-        args:
-          f [any]: an opaque value previously fetched from `.get_functions()`.
-
-        yields:
-          Tuple[capa.features.Feature, int]: feature and its location
-        """
-        raise NotImplemented
-
-    @abc.abstractmethod
-    def get_basic_blocks(self, f):
-        """
-        enumerate the basic blocks in the given function and provide opaque values that will
-         subsequently be provided to `.extract_basic_block_features()`, etc.
-
-        by "opaque value", we mean that this can be any object, as long as it
-         provides enough context to `.extract_basic_block_features()`.
-
-        the opaque value should support casting to int (`__int__`) for the basic block start address.
-
-        yields:
-          any: the opaque basic block value.
-        """
-        raise NotImplemented
-
-    @abc.abstractmethod
-    def extract_basic_block_features(self, f, bb):
-        """
-        extract basic block-scope features.
-        the arguments are opaque values previously provided by `.get_functions()`, etc.
-
-        example::
-
-            extractor = VivisectFeatureExtractor(vw, path)
-            for function in extractor.get_functions():
-                for bb in extractor.get_basic_blocks(function):
-                    for feature, va in extractor.extract_basic_block_features(function, bb):
-                        print('0x%x: %s', va, feature)
-
-        args:
-          f [any]: an opaque value previously fetched from `.get_functions()`.
-          bb [any]: an opaque value previously fetched from `.get_basic_blocks()`.
-
-        yields:
-          Tuple[capa.features.Feature, int]: feature and its location
-        """
-        raise NotImplemented
-
-    @abc.abstractmethod
-    def get_instructions(self, f, bb):
-        """
-        enumerate the instructions in the given basic block and provide opaque values that will
-         subsequently be provided to `.extract_insn_features()`, etc.
-
-        by "opaque value", we mean that this can be any object, as long as it
-         provides enough context to `.extract_insn_features()`.
-
-        the opaque value should support casting to int (`__int__`) for the instruction address.
-
-        yields:
-          any: the opaque function value.
-        """
-        raise NotImplemented
-
-    @abc.abstractmethod
-    def extract_insn_features(self, f, bb, insn):
-        """
-        extract instruction-scope features.
-        the arguments are opaque values previously provided by `.get_functions()`, etc.
-
-        example::
-
-            extractor = VivisectFeatureExtractor(vw, path)
-            for function in extractor.get_functions():
-                for bb in extractor.get_basic_blocks(function):
-                    for insn in extractor.get_instructions(function, bb):
-                        for feature, va in extractor.extract_insn_features(function, bb, insn):
-                            print('0x%x: %s', va, feature)
-
-        args:
-          f [any]: an opaque value previously fetched from `.get_functions()`.
-          bb [any]: an opaque value previously fetched from `.get_basic_blocks()`.
-          insn [any]: an opaque value previously fetched from `.get_instructions()`.
-
-        yields:
-          Tuple[capa.features.Feature, int]: feature and its location
-        """
-        raise NotImplemented
-
-
-class NullFeatureExtractor(FeatureExtractor):
-    """
-    An extractor that extracts some user-provided features.
-    The structure of the single parameter is demonstrated in the example below.
-
-    This is useful for testing, as we can provide expected values and see if matching works.
-    Also, this is how we represent features deserialized from a freeze file.
-
-    example::
-
-        extractor = NullFeatureExtractor({
-            'base address: 0x401000,
-            'file features': [
-                (0x402345, capa.features.Characteristic('embedded pe')),
-            ],
-            'functions': {
-                0x401000: {
-                    'features': [
-                        (0x401000, capa.features.Characteristic('nzxor')),
-                    ],
-                    'basic blocks': {
-                        0x401000: {
-                            'features': [
-                                (0x401000, capa.features.Characteristic('tight-loop')),
-                            ],
-                            'instructions': {
-                                0x401000: {
-                                    'features': [
-                                        (0x401000, capa.features.Characteristic('nzxor')),
-                                    ],
-                                },
-                                0x401002: ...
-                            }
-                        },
-                        0x401005: ...
-                    }
-                },
-                0x40200: ...
-            }
-        )
-    """
-
-    def __init__(self, features):
-        super(NullFeatureExtractor, self).__init__()
-        self.features = features
-
-    def get_base_address(self):
-        return self.features["base address"]
-
-    def extract_file_features(self):
-        for p in self.features.get("file features", []):
-            va, feature = p
-            yield feature, va
-
-    def get_functions(self):
-        for va in sorted(self.features["functions"].keys()):
-            yield va
-
-    def extract_function_features(self, f):
-        for p in self.features.get("functions", {}).get(f, {}).get("features", []):  # noqa: E127 line over-indented
-            va, feature = p
-            yield feature, va
-
-    def get_basic_blocks(self, f):
-        for va in sorted(
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .keys()
-        ):
-            yield va
-
-    def extract_basic_block_features(self, f, bb):
-        for p in (
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .get(bb, {})
-            .get("features", [])
-        ):
-            va, feature = p
-            yield feature, va
-
-    def get_instructions(self, f, bb):
-        for va in sorted(
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .get(bb, {})
-            .get("instructions", {})
-            .keys()
-        ):
-            yield va
-
-    def extract_insn_features(self, f, bb, insn):
-        for p in (
-            self.features.get("functions", {})  # noqa: E127 line over-indented
-            .get(f, {})
-            .get("basic blocks", {})
-            .get(bb, {})
-            .get("instructions", {})
-            .get(insn, {})
-            .get("features", [])
-        ):
-            va, feature = p
-            yield feature, va

capa/features/extractors/base_extractor.py

@@ -0,0 +1,337 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+import abc
+from typing import Tuple, Iterator, SupportsInt
+
+from capa.features.common import Feature
+
+# feature extractors may reference functions, BBs, insns by opaque handle values.
+# the only requirement of these handles are that they support `__int__`,
+# so that they can be rendered as addresses.
+#
+# these handles are only consumed by routines on
+# the feature extractor from which they were created.
+#
+# int(FunctionHandle) -> function start address
+# int(BBHandle) -> BasicBlock start address
+# int(InsnHandle) -> instruction address
+FunctionHandle = SupportsInt
+BBHandle = SupportsInt
+InsnHandle = SupportsInt
+
+
+class FeatureExtractor:
+    """
+    FeatureExtractor defines the interface for fetching features from a sample.
+
+    There may be multiple backends that support fetching features for capa.
+    For example, we use vivisect by default, but also want to support saving
+     and restoring features from a JSON file.
+    When we restore the features, we'd like to use exactly the same matching logic
+     to find matching rules.
+    Therefore, we can define a FeatureExtractor that provides features from the
+     serialized JSON file and do matching without a binary analysis pass.
+    Also, this provides a way to hook in an IDA backend.
+
+    This class is not instantiated directly; it is the base class for other implementations.
+    """
+
+    __metaclass__ = abc.ABCMeta
+
+    def __init__(self):
+        #
+        # note: a subclass should define ctor parameters for its own use.
+        #  for example, the Vivisect feature extract might require the vw and/or path.
+        # this base class doesn't know what to do with that info, though.
+        #
+        super(FeatureExtractor, self).__init__()
+
+    @abc.abstractmethod
+    def get_base_address(self) -> int:
+        """
+        fetch the preferred load address at which the sample was analyzed.
+        """
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def extract_global_features(self) -> Iterator[Tuple[Feature, int]]:
+        """
+        extract features found at every scope ("global").
+
+        example::
+
+            extractor = VivisectFeatureExtractor(vw, path)
+            for feature, va in extractor.get_global_features():
+                print('0x%x: %s', va, feature)
+
+        yields:
+          Tuple[Feature, int]: feature and its location
+        """
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def extract_file_features(self) -> Iterator[Tuple[Feature, int]]:
+        """
+        extract file-scope features.
+
+        example::
+
+            extractor = VivisectFeatureExtractor(vw, path)
+            for feature, va in extractor.get_file_features():
+                print('0x%x: %s', va, feature)
+
+        yields:
+          Tuple[Feature, int]: feature and its location
+        """
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def get_functions(self) -> Iterator[FunctionHandle]:
+        """
+        enumerate the functions and provide opaque values that will
+         subsequently be provided to `.extract_function_features()`, etc.
+        """
+        raise NotImplementedError()
+
+    def is_library_function(self, va: int) -> bool:
+        """
+        is the given address a library function?
+        the backend may implement its own function matching algorithm, or none at all.
+        we accept a VA here, rather than function object, to handle addresses identified in instructions.
+
+        this information is used to:
+          - filter out matches in library functions (by default), and
+          - recognize when to fetch symbol names for called (non-API) functions
+
+        args:
+          va (int): the virtual address of a function.
+
+        returns:
+          bool: True if the given address is the start of a library function.
+        """
+        return False
+
+    def get_function_name(self, va: int) -> str:
+        """
+        fetch any recognized name for the given address.
+        this is only guaranteed to return a value when the given function is a recognized library function.
+        we accept a VA here, rather than function object, to handle addresses identified in instructions.
+
+        args:
+          va (int): the virtual address of a function.
+
+        returns:
+          str: the function name
+
+        raises:
+          KeyError: when the given function does not have a name.
+        """
+        raise KeyError(va)
+
+    @abc.abstractmethod
+    def extract_function_features(self, f: FunctionHandle) -> Iterator[Tuple[Feature, int]]:
+        """
+        extract function-scope features.
+        the arguments are opaque values previously provided by `.get_functions()`, etc.
+
+        example::
+
+            extractor = VivisectFeatureExtractor(vw, path)
+            for function in extractor.get_functions():
+                for feature, va in extractor.extract_function_features(function):
+                    print('0x%x: %s', va, feature)
+
+        args:
+          f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
+
+        yields:
+          Tuple[Feature, int]: feature and its location
+        """
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def get_basic_blocks(self, f: FunctionHandle) -> Iterator[BBHandle]:
+        """
+        enumerate the basic blocks in the given function and provide opaque values that will
+         subsequently be provided to `.extract_basic_block_features()`, etc.
+        """
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def extract_basic_block_features(self, f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, int]]:
+        """
+        extract basic block-scope features.
+        the arguments are opaque values previously provided by `.get_functions()`, etc.
+
+        example::
+
+            extractor = VivisectFeatureExtractor(vw, path)
+            for function in extractor.get_functions():
+                for bb in extractor.get_basic_blocks(function):
+                    for feature, va in extractor.extract_basic_block_features(function, bb):
+                        print('0x%x: %s', va, feature)
+
+        args:
+          f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
+          bb [BBHandle]: an opaque value previously fetched from `.get_basic_blocks()`.
+
+        yields:
+          Tuple[Feature, int]: feature and its location
+        """
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def get_instructions(self, f: FunctionHandle, bb: BBHandle) -> Iterator[InsnHandle]:
+        """
+        enumerate the instructions in the given basic block and provide opaque values that will
+         subsequently be provided to `.extract_insn_features()`, etc.
+        """
+        raise NotImplementedError()
+
+    @abc.abstractmethod
+    def extract_insn_features(self, f: FunctionHandle, bb: BBHandle, insn: InsnHandle) -> Iterator[Tuple[Feature, int]]:
+        """
+        extract instruction-scope features.
+        the arguments are opaque values previously provided by `.get_functions()`, etc.
+
+        example::
+
+            extractor = VivisectFeatureExtractor(vw, path)
+            for function in extractor.get_functions():
+                for bb in extractor.get_basic_blocks(function):
+                    for insn in extractor.get_instructions(function, bb):
+                        for feature, va in extractor.extract_insn_features(function, bb, insn):
+                            print('0x%x: %s', va, feature)
+
+        args:
+          f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
+          bb [BBHandle]: an opaque value previously fetched from `.get_basic_blocks()`.
+          insn [InsnHandle]: an opaque value previously fetched from `.get_instructions()`.
+
+        yields:
+          Tuple[Feature, int]: feature and its location
+        """
+        raise NotImplementedError()
+
+
+class NullFeatureExtractor(FeatureExtractor):
+    """
+    An extractor that extracts some user-provided features.
+    The structure of the single parameter is demonstrated in the example below.
+
+    This is useful for testing, as we can provide expected values and see if matching works.
+    Also, this is how we represent features deserialized from a freeze file.
+
+    example::
+
+        extractor = NullFeatureExtractor({
+            'base address: 0x401000,
+            'global features': [
+                (0x0, capa.features.Arch('i386')),
+                (0x0, capa.features.OS('linux')),
+            ],
+            'file features': [
+                (0x402345, capa.features.Characteristic('embedded pe')),
+            ],
+            'functions': {
+                0x401000: {
+                    'features': [
+                        (0x401000, capa.features.Characteristic('nzxor')),
+                    ],
+                    'basic blocks': {
+                        0x401000: {
+                            'features': [
+                                (0x401000, capa.features.Characteristic('tight-loop')),
+                            ],
+                            'instructions': {
+                                0x401000: {
+                                    'features': [
+                                        (0x401000, capa.features.Characteristic('nzxor')),
+                                    ],
+                                },
+                                0x401002: ...
+                            }
+                        },
+                        0x401005: ...
+                    }
+                },
+                0x40200: ...
+            }
+        )
+    """
+
+    def __init__(self, features):
+        super(NullFeatureExtractor, self).__init__()
+        self.features = features
+
+    def get_base_address(self):
+        return self.features["base address"]
+
+    def extract_global_features(self):
+        for p in self.features.get("global features", []):
+            va, feature = p
+            yield feature, va
+
+    def extract_file_features(self):
+        for p in self.features.get("file features", []):
+            va, feature = p
+            yield feature, va
+
+    def get_functions(self):
+        for va in sorted(self.features["functions"].keys()):
+            yield va
+
+    def extract_function_features(self, f):
+        for p in self.features.get("functions", {}).get(f, {}).get("features", []):  # noqa: E127 line over-indented
+            va, feature = p
+            yield feature, va
+
+    def get_basic_blocks(self, f):
+        for va in sorted(
+            self.features.get("functions", {})  # noqa: E127 line over-indented
+            .get(f, {})
+            .get("basic blocks", {})
+            .keys()
+        ):
+            yield va
+
+    def extract_basic_block_features(self, f, bb):
+        for p in (
+            self.features.get("functions", {})  # noqa: E127 line over-indented
+            .get(f, {})
+            .get("basic blocks", {})
+            .get(bb, {})
+            .get("features", [])
+        ):
+            va, feature = p
+            yield feature, va
+
+    def get_instructions(self, f, bb):
+        for va in sorted(
+            self.features.get("functions", {})  # noqa: E127 line over-indented
+            .get(f, {})
+            .get("basic blocks", {})
+            .get(bb, {})
+            .get("instructions", {})
+            .keys()
+        ):
+            yield va
+
+    def extract_insn_features(self, f, bb, insn):
+        for p in (
+            self.features.get("functions", {})  # noqa: E127 line over-indented
+            .get(f, {})
+            .get("basic blocks", {})
+            .get(bb, {})
+            .get("instructions", {})
+            .get(insn, {})
+            .get("features", [])
+        ):
+            va, feature = p
+            yield feature, va

capa/features/extractors/common.py

@@ -0,0 +1,95 @@
+import io
+import logging
+import binascii
+import contextlib
+
+import pefile
+
+import capa.features
+import capa.features.extractors.elf
+import capa.features.extractors.pefile
+from capa.features.common import OS, FORMAT_PE, FORMAT_ELF, OS_WINDOWS, Arch, Format, String
+
+logger = logging.getLogger(__name__)
+
+
+def extract_file_strings(buf, **kwargs):
+    """
+    extract ASCII and UTF-16 LE strings from file
+    """
+    for s in capa.features.extractors.strings.extract_ascii_strings(buf):
+        yield String(s.s), s.offset
+
+    for s in capa.features.extractors.strings.extract_unicode_strings(buf):
+        yield String(s.s), s.offset
+
+
+def extract_format(buf):
+    if buf.startswith(b"MZ"):
+        yield Format(FORMAT_PE), 0x0
+    elif buf.startswith(b"\x7fELF"):
+        yield Format(FORMAT_ELF), 0x0
+    else:
+        # we likely end up here:
+        #  1. handling a file format (e.g. macho)
+        #
+        # for (1), this logic will need to be updated as the format is implemented.
+        logger.debug("unsupported file format: %s", binascii.hexlify(buf[:4]).decode("ascii"))
+        return
+
+
+def extract_arch(buf):
+    if buf.startswith(b"MZ"):
+        yield from capa.features.extractors.pefile.extract_file_arch(pe=pefile.PE(data=buf))
+
+    elif buf.startswith(b"\x7fELF"):
+        with contextlib.closing(io.BytesIO(buf)) as f:
+            arch = capa.features.extractors.elf.detect_elf_arch(f)
+
+        if arch not in capa.features.common.VALID_ARCH:
+            logger.debug("unsupported arch: %s", arch)
+            return
+
+        yield Arch(arch), 0x0
+
+    else:
+        # we likely end up here:
+        #  1. handling shellcode, or
+        #  2. handling a new file format (e.g. macho)
+        #
+        # for (1) we can't do much - its shellcode and all bets are off.
+        # we could maybe accept a futher CLI argument to specify the arch,
+        # but i think this would be rarely used.
+        # rules that rely on arch conditions will fail to match on shellcode.
+        #
+        # for (2), this logic will need to be updated as the format is implemented.
+        logger.debug("unsupported file format: %s, will not guess Arch", binascii.hexlify(buf[:4]).decode("ascii"))
+        return
+
+
+def extract_os(buf):
+    if buf.startswith(b"MZ"):
+        yield OS(OS_WINDOWS), 0x0
+    elif buf.startswith(b"\x7fELF"):
+        with contextlib.closing(io.BytesIO(buf)) as f:
+            os = capa.features.extractors.elf.detect_elf_os(f)
+
+        if os not in capa.features.common.VALID_OS:
+            logger.debug("unsupported os: %s", os)
+            return
+
+        yield OS(os), 0x0
+
+    else:
+        # we likely end up here:
+        #  1. handling shellcode, or
+        #  2. handling a new file format (e.g. macho)
+        #
+        # for (1) we can't do much - its shellcode and all bets are off.
+        # we could maybe accept a futher CLI argument to specify the OS,
+        # but i think this would be rarely used.
+        # rules that rely on OS conditions will fail to match on shellcode.
+        #
+        # for (2), this logic will need to be updated as the format is implemented.
+        logger.debug("unsupported file format: %s, will not guess OS", binascii.hexlify(buf[:4]).decode("ascii"))
+        return

capa/features/extractors/elf.py

@@ -0,0 +1,276 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import struct
+import logging
+from enum import Enum
+from typing import BinaryIO
+
+logger = logging.getLogger(__name__)
+
+
+def align(v, alignment):
+    remainder = v % alignment
+    if remainder == 0:
+        return v
+    else:
+        return v + (alignment - remainder)
+
+
+class CorruptElfFile(ValueError):
+    pass
+
+
+class OS(str, Enum):
+    HPUX = "hpux"
+    NETBSD = "netbsd"
+    LINUX = "linux"
+    HURD = "hurd"
+    _86OPEN = "86open"
+    SOLARIS = "solaris"
+    AIX = "aix"
+    IRIX = "irix"
+    FREEBSD = "freebsd"
+    TRU64 = "tru64"
+    MODESTO = "modesto"
+    OPENBSD = "openbsd"
+    OPENVMS = "openvms"
+    NSK = "nsk"
+    AROS = "aros"
+    FENIXOS = "fenixos"
+    CLOUD = "cloud"
+    SYLLABLE = "syllable"
+    NACL = "nacl"
+
+
+def detect_elf_os(f: BinaryIO) -> str:
+    f.seek(0x0)
+    file_header = f.read(0x40)
+
+    # we'll set this to the detected OS
+    # prefer the first heuristics,
+    # but rather than short circuiting,
+    # we'll still parse out the remainder, for debugging.
+    ret = None
+
+    if not file_header.startswith(b"\x7fELF"):
+        raise CorruptElfFile("missing magic header")
+
+    ei_class, ei_data = struct.unpack_from("BB", file_header, 4)
+    logger.debug("ei_class: 0x%02x ei_data: 0x%02x", ei_class, ei_data)
+    if ei_class == 1:
+        bitness = 32
+    elif ei_class == 2:
+        bitness = 64
+    else:
+        raise CorruptElfFile("invalid ei_class: 0x%02x" % ei_class)
+
+    if ei_data == 1:
+        endian = "<"
+    elif ei_data == 2:
+        endian = ">"
+    else:
+        raise CorruptElfFile("not an ELF file: invalid ei_data: 0x%02x" % ei_data)
+
+    if bitness == 32:
+        (e_phoff,) = struct.unpack_from(endian + "I", file_header, 0x1C)
+        e_phentsize, e_phnum = struct.unpack_from(endian + "HH", file_header, 0x2A)
+    elif bitness == 64:
+        (e_phoff,) = struct.unpack_from(endian + "Q", file_header, 0x20)
+        e_phentsize, e_phnum = struct.unpack_from(endian + "HH", file_header, 0x36)
+    else:
+        raise NotImplementedError()
+
+    logger.debug("e_phoff: 0x%02x e_phentsize: 0x%02x e_phnum: %d", e_phoff, e_phentsize, e_phnum)
+
+    (ei_osabi,) = struct.unpack_from(endian + "B", file_header, 7)
+    OSABI = {
+        # via pyelftools: https://github.com/eliben/pyelftools/blob/0664de05ed2db3d39041e2d51d19622a8ef4fb0f/elftools/elf/enums.py#L35-L58
+        # some candidates are commented out because the are not useful values,
+        # at least when guessing OSes
+        # 0: "SYSV",  # too often used when OS is not SYSV
+        1: OS.HPUX,
+        2: OS.NETBSD,
+        3: OS.LINUX,
+        4: OS.HURD,
+        5: OS._86OPEN,
+        6: OS.SOLARIS,
+        7: OS.AIX,
+        8: OS.IRIX,
+        9: OS.FREEBSD,
+        10: OS.TRU64,
+        11: OS.MODESTO,
+        12: OS.OPENBSD,
+        13: OS.OPENVMS,
+        14: OS.NSK,
+        15: OS.AROS,
+        16: OS.FENIXOS,
+        17: OS.CLOUD,
+        # 53: "SORTFIX",      # i can't find any reference to this OS, i dont think it exists
+        # 64: "ARM_AEABI",    # not an OS
+        # 97: "ARM",          # not an OS
+        # 255: "STANDALONE",  # not an OS
+    }
+    logger.debug("ei_osabi: 0x%02x (%s)", ei_osabi, OSABI.get(ei_osabi, "unknown"))
+
+    # os_osabi == 0 is commonly set even when the OS is not SYSV.
+    # other values are unused or unknown.
+    if ei_osabi in OSABI and ei_osabi != 0x0:
+        # subsequent strategies may overwrite this value
+        ret = OSABI[ei_osabi]
+
+    f.seek(e_phoff)
+    program_header_size = e_phnum * e_phentsize
+    program_headers = f.read(program_header_size)
+    if len(program_headers) != program_header_size:
+        logger.warning("failed to read program headers")
+        e_phnum = 0
+
+    # search for PT_NOTE sections that specify an OS
+    # for example, on Linux there is a GNU section with minimum kernel version
+    for i in range(e_phnum):
+        offset = i * e_phentsize
+        phent = program_headers[offset : offset + e_phentsize]
+
+        PT_NOTE = 0x4
+
+        (p_type,) = struct.unpack_from(endian + "I", phent, 0x0)
+        logger.debug("p_type: 0x%04x", p_type)
+        if p_type != PT_NOTE:
+            continue
+
+        if bitness == 32:
+            p_offset, _, _, p_filesz = struct.unpack_from(endian + "IIII", phent, 0x4)
+        elif bitness == 64:
+            p_offset, _, _, p_filesz = struct.unpack_from(endian + "QQQQ", phent, 0x8)
+        else:
+            raise NotImplementedError()
+
+        logger.debug("p_offset: 0x%02x p_filesz: 0x%04x", p_offset, p_filesz)
+
+        f.seek(p_offset)
+        note = f.read(p_filesz)
+        if len(note) != p_filesz:
+            logger.warning("failed to read note content")
+            continue
+
+        namesz, descsz, type_ = struct.unpack_from(endian + "III", note, 0x0)
+        name_offset = 0xC
+        desc_offset = name_offset + align(namesz, 0x4)
+
+        logger.debug("namesz: 0x%02x descsz: 0x%02x type: 0x%04x", namesz, descsz, type_)
+
+        name = note[name_offset : name_offset + namesz].partition(b"\x00")[0].decode("ascii")
+        logger.debug("name: %s", name)
+
+        if type_ != 1:
+            continue
+
+        if name == "GNU":
+            if descsz < 16:
+                continue
+
+            desc = note[desc_offset : desc_offset + descsz]
+            abi_tag, kmajor, kminor, kpatch = struct.unpack_from(endian + "IIII", desc, 0x0)
+            # via readelf: https://github.com/bminor/binutils-gdb/blob/c0e94211e1ac05049a4ce7c192c9d14d1764eb3e/binutils/readelf.c#L19635-L19658
+            # and here: https://github.com/bminor/binutils-gdb/blob/34c54daa337da9fadf87d2706d6a590ae1f88f4d/include/elf/common.h#L933-L939
+            GNU_ABI_TAG = {
+                0: OS.LINUX,
+                1: OS.HURD,
+                2: OS.SOLARIS,
+                3: OS.FREEBSD,
+                4: OS.NETBSD,
+                5: OS.SYLLABLE,
+                6: OS.NACL,
+            }
+            logger.debug("GNU_ABI_TAG: 0x%02x", abi_tag)
+
+            if abi_tag in GNU_ABI_TAG:
+                # update only if not set
+                # so we can get the debugging output of subsequent strategies
+                ret = GNU_ABI_TAG[abi_tag] if not ret else ret
+                logger.debug("abi tag: %s earliest compatible kernel: %d.%d.%d", ret, kmajor, kminor, kpatch)
+        elif name == "OpenBSD":
+            logger.debug("note owner: %s", "OPENBSD")
+            ret = OS.OPENBSD if not ret else ret
+        elif name == "NetBSD":
+            logger.debug("note owner: %s", "NETBSD")
+            ret = OS.NETBSD if not ret else ret
+        elif name == "FreeBSD":
+            logger.debug("note owner: %s", "FREEBSD")
+            ret = OS.FREEBSD if not ret else ret
+
+    # search for recognizable dynamic linkers (interpreters)
+    # for example, on linux, we see file paths like: /lib64/ld-linux-x86-64.so.2
+    for i in range(e_phnum):
+        offset = i * e_phentsize
+        phent = program_headers[offset : offset + e_phentsize]
+
+        PT_INTERP = 0x3
+
+        (p_type,) = struct.unpack_from(endian + "I", phent, 0x0)
+        if p_type != PT_INTERP:
+            continue
+
+        if bitness == 32:
+            p_offset, _, _, p_filesz = struct.unpack_from(endian + "IIII", phent, 0x4)
+        elif bitness == 64:
+            p_offset, _, _, p_filesz = struct.unpack_from(endian + "QQQQ", phent, 0x8)
+        else:
+            raise NotImplementedError()
+
+        f.seek(p_offset)
+        interp = f.read(p_filesz)
+        if len(interp) != p_filesz:
+            logger.warning("failed to read interp content")
+            continue
+
+        linker = interp.partition(b"\x00")[0].decode("ascii")
+        logger.debug("linker: %s", linker)
+        if "ld-linux" in linker:
+            # update only if not set
+            # so we can get the debugging output of subsequent strategies
+            ret = OS.LINUX if ret is None else ret
+
+    return ret.value if ret is not None else "unknown"
+
+
+class Arch(str, Enum):
+    I386 = "i386"
+    AMD64 = "amd64"
+
+
+def detect_elf_arch(f: BinaryIO) -> str:
+    f.seek(0x0)
+    file_header = f.read(0x40)
+
+    if not file_header.startswith(b"\x7fELF"):
+        raise CorruptElfFile("missing magic header")
+
+    (ei_data,) = struct.unpack_from("B", file_header, 5)
+    logger.debug("ei_data: 0x%02x", ei_data)
+
+    if ei_data == 1:
+        endian = "<"
+    elif ei_data == 2:
+        endian = ">"
+    else:
+        raise CorruptElfFile("not an ELF file: invalid ei_data: 0x%02x" % ei_data)
+
+    (ei_machine,) = struct.unpack_from(endian + "H", file_header, 0x12)
+    logger.debug("ei_machine: 0x%02x", ei_machine)
+
+    EM_386 = 0x3
+    EM_X86_64 = 0x3E
+    if ei_machine == EM_386:
+        return Arch.I386
+    elif ei_machine == EM_X86_64:
+        return Arch.AMD64
+    else:
+        # not really unknown, but unsupport at the moment:
+        # https://github.com/eliben/pyelftools/blob/ab444d982d1849191e910299a985989857466620/elftools/elf/enums.py#L73
+        return "unknown"

capa/features/extractors/elffile.py

@@ -0,0 +1,159 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import io
+import logging
+import contextlib
+from typing import Tuple, Iterator
+
+from elftools.elf.elffile import ELFFile, SymbolTableSection
+
+import capa.features.extractors.common
+from capa.features.file import Import, Section
+from capa.features.common import OS, FORMAT_ELF, Arch, Format, Feature
+from capa.features.extractors.elf import Arch as ElfArch
+from capa.features.extractors.base_extractor import FeatureExtractor
+
+logger = logging.getLogger(__name__)
+
+
+def extract_file_import_names(elf, **kwargs):
+    # see https://github.com/eliben/pyelftools/blob/0664de05ed2db3d39041e2d51d19622a8ef4fb0f/scripts/readelf.py#L372
+    symbol_tables = [(idx, s) for idx, s in enumerate(elf.iter_sections()) if isinstance(s, SymbolTableSection)]
+
+    for section_index, section in symbol_tables:
+        if not isinstance(section, SymbolTableSection):
+            continue
+
+        if section["sh_entsize"] == 0:
+            logger.debug("Symbol table '%s' has a sh_entsize of zero!" % (section.name))
+            continue
+
+        logger.debug("Symbol table '%s' contains %s entries:" % (section.name, section.num_symbols()))
+
+        for nsym, symbol in enumerate(section.iter_symbols()):
+            if symbol.name and symbol.entry.st_info.type == "STT_FUNC":
+                # TODO symbol address
+                # TODO symbol version info?
+                yield Import(symbol.name), 0x0
+
+
+def extract_file_section_names(elf, **kwargs):
+    for section in elf.iter_sections():
+        if section.name:
+            yield Section(section.name), section.header.sh_addr
+        elif section.is_null():
+            yield Section("NULL"), section.header.sh_addr
+
+
+def extract_file_strings(buf, **kwargs):
+    yield from capa.features.extractors.common.extract_file_strings(buf)
+
+
+def extract_file_os(elf, buf, **kwargs):
+    # our current approach does not always get an OS value, e.g. for packed samples
+    # for file limitation purposes, we're more lax here
+    try:
+        os = next(capa.features.extractors.common.extract_os(buf))
+        yield os
+    except StopIteration:
+        yield OS("unknown"), 0x0
+
+
+def extract_file_format(**kwargs):
+    yield Format(FORMAT_ELF), 0x0
+
+
+def extract_file_arch(elf, **kwargs):
+    # TODO merge with capa.features.extractors.elf.detect_elf_arch()
+    arch = elf.get_machine_arch()
+    if arch == "x86":
+        yield Arch(ElfArch.I386), 0x0
+    elif arch == "x64":
+        yield Arch(ElfArch.AMD64), 0x0
+    else:
+        logger.warning("unsupported architecture: %s", arch)
+
+
+def extract_file_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
+    for file_handler in FILE_HANDLERS:
+        for feature, va in file_handler(elf=elf, buf=buf):  # type: ignore
+            yield feature, va
+
+
+FILE_HANDLERS = (
+    # TODO extract_file_export_names,
+    extract_file_import_names,
+    extract_file_section_names,
+    extract_file_strings,
+    # no library matching
+    extract_file_format,
+)
+
+
+def extract_global_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
+    for global_handler in GLOBAL_HANDLERS:
+        for feature, va in global_handler(elf=elf, buf=buf):  # type: ignore
+            yield feature, va
+
+
+GLOBAL_HANDLERS = (
+    extract_file_os,
+    extract_file_arch,
+)
+
+
+class ElfFeatureExtractor(FeatureExtractor):
+    def __init__(self, path: str):
+        super(ElfFeatureExtractor, self).__init__()
+        self.path = path
+        with open(self.path, "rb") as f:
+            self.elf = ELFFile(io.BytesIO(f.read()))
+
+    def get_base_address(self):
+        # virtual address of the first segment with type LOAD
+        for segment in self.elf.iter_segments():
+            if segment.header.p_type == "PT_LOAD":
+                return segment.header.p_vaddr
+
+    def extract_global_features(self):
+        with open(self.path, "rb") as f:
+            buf = f.read()
+
+        for feature, va in extract_global_features(self.elf, buf):
+            yield feature, va
+
+    def extract_file_features(self):
+        with open(self.path, "rb") as f:
+            buf = f.read()
+
+        for feature, va in extract_file_features(self.elf, buf):
+            yield feature, va
+
+    def get_functions(self):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
+
+    def extract_function_features(self, f):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
+
+    def get_basic_blocks(self, f):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
+
+    def extract_basic_block_features(self, f, bb):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
+
+    def get_instructions(self, f, bb):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
+
+    def extract_insn_features(self, f, bb, insn):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
+
+    def is_library_function(self, va):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
+
+    def get_function_name(self, va):
+        raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")

capa/features/extractors/helpers.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,23 +6,18 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-import sys
+import struct
 import builtins
-
-from capa.features.file import Import
-from capa.features.insn import API
+from typing import Tuple, Iterator
 
 MIN_STACKSTRING_LEN = 8
 
 
-def xor_static(data, i):
-    if sys.version_info >= (3, 0):
-        return bytes(c ^ i for c in data)
-    else:
-        return "".join(chr(ord(c) ^ i) for c in data)
+def xor_static(data: bytes, i: int) -> bytes:
+    return bytes(c ^ i for c in data)
 
 
-def is_aw_function(symbol):
+def is_aw_function(symbol: str) -> bool:
     """
     is the given function name an A/W function?
     these are variants of functions that, on Windows, accept either a narrow or wide string.
@@ -38,7 +33,7 @@ def is_aw_function(symbol):
     return "a" <= symbol[-2] <= "z" or "0" <= symbol[-2] <= "9"
 
 
-def is_ordinal(symbol):
+def is_ordinal(symbol: str) -> bool:
     """
     is the given symbol an ordinal that is prefixed by "#"?
     """
@@ -47,7 +42,7 @@ def is_ordinal(symbol):
     return False
 
 
-def generate_symbols(dll, symbol):
+def generate_symbols(dll: str, symbol: str) -> Iterator[str]:
     """
     for a given dll and symbol name, generate variants.
     we over-generate features to make matching easier.
@@ -73,11 +68,11 @@ def generate_symbols(dll, symbol):
             yield symbol[:-1]
 
 
-def all_zeros(bytez):
+def all_zeros(bytez: bytes) -> bool:
     return all(b == 0 for b in builtins.bytes(bytez))
 
 
-def twos_complement(val, bits):
+def twos_complement(val: int, bits: int) -> int:
     """
     compute the 2's complement of int value val
 
@@ -90,3 +85,49 @@ def twos_complement(val, bits):
     else:
         # return positive value as is
         return val
+
+
+def carve_pe(pbytes: bytes, offset: int = 0) -> Iterator[Tuple[int, int]]:
+    """
+    Generate (offset, key) tuples of embedded PEs
+
+    Based on the version from vivisect:
+      https://github.com/vivisect/vivisect/blob/7be4037b1cecc4551b397f840405a1fc606f9b53/PE/carve.py#L19
+    And its IDA adaptation:
+      capa/features/extractors/ida/file.py
+    """
+    mz_xor = [
+        (
+            xor_static(b"MZ", key),
+            xor_static(b"PE", key),
+            key,
+        )
+        for key in range(256)
+    ]
+
+    pblen = len(pbytes)
+    todo = [(pbytes.find(mzx, offset), mzx, pex, key) for mzx, pex, key in mz_xor]
+    todo = [(off, mzx, pex, key) for (off, mzx, pex, key) in todo if off != -1]
+
+    while len(todo):
+
+        off, mzx, pex, key = todo.pop()
+
+        # The MZ header has one field we will check
+        # e_lfanew is at 0x3c
+        e_lfanew = off + 0x3C
+        if pblen < (e_lfanew + 4):
+            continue
+
+        newoff = struct.unpack("<I", xor_static(pbytes[e_lfanew : e_lfanew + 4], key))[0]
+
+        nextres = pbytes.find(mzx, off + 1)
+        if nextres != -1:
+            todo.append((nextres, mzx, pex, key))
+
+        peoff = off + newoff
+        if pblen < (peoff + 2):
+            continue
+
+        if pbytes[peoff : peoff + 2] == pex:
+            yield (off, key)

capa/features/extractors/ida/__init__.py

@@ -1,93 +0,0 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-
-import sys
-import types
-
-import idaapi
-
-import capa.features.extractors.ida.file
-import capa.features.extractors.ida.insn
-import capa.features.extractors.ida.function
-import capa.features.extractors.ida.basicblock
-from capa.features.extractors import FeatureExtractor
-
-
-def get_ea(self):
-    """ """
-    if isinstance(self, (idaapi.BasicBlock, idaapi.func_t)):
-        return self.start_ea
-    if isinstance(self, idaapi.insn_t):
-        return self.ea
-    raise TypeError
-
-
-def add_ea_int_cast(o):
-    """
-    dynamically add a cast-to-int (`__int__`) method to the given object
-    that returns the value of the `.ea` property.
-    this bit of skullduggery lets use cast viv-utils objects as ints.
-    the correct way of doing this is to update viv-utils (or subclass the objects here).
-    """
-    if sys.version_info[0] >= 3:
-        setattr(o, "__int__", types.MethodType(get_ea, o))
-    else:
-        setattr(o, "__int__", types.MethodType(get_ea, o, type(o)))
-    return o
-
-
-class IdaFeatureExtractor(FeatureExtractor):
-    def __init__(self):
-        super(IdaFeatureExtractor, self).__init__()
-
-    def get_base_address(self):
-        return idaapi.get_imagebase()
-
-    def extract_file_features(self):
-        for (feature, ea) in capa.features.extractors.ida.file.extract_features():
-            yield feature, ea
-
-    def get_functions(self):
-        import capa.features.extractors.ida.helpers as ida_helpers
-
-        # data structure shared across functions yielded here.
-        # useful for caching analysis relevant across a single workspace.
-        ctx = {}
-
-        # ignore library functions and thunk functions as identified by IDA
-        for f in ida_helpers.get_functions(skip_thunks=True, skip_libs=True):
-            setattr(f, "ctx", ctx)
-            yield add_ea_int_cast(f)
-
-    @staticmethod
-    def get_function(ea):
-        f = idaapi.get_func(ea)
-        setattr(f, "ctx", {})
-        return add_ea_int_cast(f)
-
-    def extract_function_features(self, f):
-        for (feature, ea) in capa.features.extractors.ida.function.extract_features(f):
-            yield feature, ea
-
-    def get_basic_blocks(self, f):
-        for bb in capa.features.extractors.ida.helpers.get_function_blocks(f):
-            yield add_ea_int_cast(bb)
-
-    def extract_basic_block_features(self, f, bb):
-        for (feature, ea) in capa.features.extractors.ida.basicblock.extract_features(f, bb):
-            yield feature, ea
-
-    def get_instructions(self, f, bb):
-        import capa.features.extractors.ida.helpers as ida_helpers
-
-        for insn in ida_helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
-            yield add_ea_int_cast(insn)
-
-    def extract_insn_features(self, f, bb, insn):
-        for (feature, ea) in capa.features.extractors.ida.insn.extract_features(f, bb, insn):
-            yield feature, ea

capa/features/extractors/ida/basicblock.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,14 +6,13 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-import sys
 import string
 import struct
 
 import idaapi
 
 import capa.features.extractors.ida.helpers
-from capa.features import Characteristic
+from capa.features.common import Characteristic
 from capa.features.basicblock import BasicBlock
 from capa.features.extractors.ida import helpers
 from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
@@ -39,18 +38,11 @@ def get_printable_len(op):
         raise ValueError("Unhandled operand data type 0x%x." % op.dtype)
 
     def is_printable_ascii(chars):
-        if sys.version_info[0] >= 3:
-            return all(c < 127 and chr(c) in string.printable for c in chars)
-        else:
-            return all(ord(c) < 127 and c in string.printable for c in chars)
+        return all(c < 127 and chr(c) in string.printable for c in chars)
 
     def is_printable_utf16le(chars):
-        if sys.version_info[0] >= 3:
-            if all(c == 0x00 for c in chars[1::2]):
-                return is_printable_ascii(chars[::2])
-        else:
-            if all(c == "\x00" for c in chars[1::2]):
-                return is_printable_ascii(chars[::2])
+        if all(c == 0x00 for c in chars[1::2]):
+            return is_printable_ascii(chars[::2])
 
     if is_printable_ascii(chars):
         return idaapi.get_dtype_size(op.dtype)

capa/features/extractors/ida/extractor.py

@@ -0,0 +1,112 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import idaapi
+
+import capa.ida.helpers
+import capa.features.extractors.elf
+import capa.features.extractors.ida.file
+import capa.features.extractors.ida.insn
+import capa.features.extractors.ida.global_
+import capa.features.extractors.ida.function
+import capa.features.extractors.ida.basicblock
+from capa.features.extractors.base_extractor import FeatureExtractor
+
+
+class FunctionHandle:
+    """this acts like an idaapi.func_t but with __int__()"""
+
+    def __init__(self, inner):
+        self._inner = inner
+
+    def __int__(self):
+        return self.start_ea
+
+    def __getattr__(self, name):
+        return getattr(self._inner, name)
+
+
+class BasicBlockHandle:
+    """this acts like an idaapi.BasicBlock but with __int__()"""
+
+    def __init__(self, inner):
+        self._inner = inner
+
+    def __int__(self):
+        return self.start_ea
+
+    def __getattr__(self, name):
+        return getattr(self._inner, name)
+
+
+class InstructionHandle:
+    """this acts like an idaapi.insn_t but with __int__()"""
+
+    def __init__(self, inner):
+        self._inner = inner
+
+    def __int__(self):
+        return self.ea
+
+    def __getattr__(self, name):
+        return getattr(self._inner, name)
+
+
+class IdaFeatureExtractor(FeatureExtractor):
+    def __init__(self):
+        super(IdaFeatureExtractor, self).__init__()
+        self.global_features = []
+        self.global_features.extend(capa.features.extractors.ida.global_.extract_os())
+        self.global_features.extend(capa.features.extractors.ida.global_.extract_arch())
+
+    def get_base_address(self):
+        return idaapi.get_imagebase()
+
+    def extract_global_features(self):
+        yield from self.global_features
+
+    def extract_file_features(self):
+        yield from capa.features.extractors.ida.file.extract_features()
+
+    def get_functions(self):
+        import capa.features.extractors.ida.helpers as ida_helpers
+
+        # data structure shared across functions yielded here.
+        # useful for caching analysis relevant across a single workspace.
+        ctx = {}
+
+        # ignore library functions and thunk functions as identified by IDA
+        for f in ida_helpers.get_functions(skip_thunks=True, skip_libs=True):
+            setattr(f, "ctx", ctx)
+            yield FunctionHandle(f)
+
+    @staticmethod
+    def get_function(ea):
+        f = idaapi.get_func(ea)
+        setattr(f, "ctx", {})
+        return FunctionHandle(f)
+
+    def extract_function_features(self, f):
+        yield from capa.features.extractors.ida.function.extract_features(f)
+
+    def get_basic_blocks(self, f):
+        import capa.features.extractors.ida.helpers as ida_helpers
+
+        for bb in ida_helpers.get_function_blocks(f):
+            yield BasicBlockHandle(bb)
+
+    def extract_basic_block_features(self, f, bb):
+        yield from capa.features.extractors.ida.basicblock.extract_features(f, bb)
+
+    def get_instructions(self, f, bb):
+        import capa.features.extractors.ida.helpers as ida_helpers
+
+        for insn in ida_helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
+            yield InstructionHandle(insn)
+
+    def extract_insn_features(self, f, bb, insn):
+        yield from capa.features.extractors.ida.insn.extract_features(f, bb, insn)

capa/features/extractors/ida/file.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -11,12 +11,13 @@ import struct
 import idc
 import idaapi
 import idautils
+import ida_loader
 
 import capa.features.extractors.helpers
 import capa.features.extractors.strings
 import capa.features.extractors.ida.helpers
-from capa.features import String, Characteristic
-from capa.features.file import Export, Import, Section
+from capa.features.file import Export, Import, Section, FunctionName
+from capa.features.common import OS, FORMAT_PE, FORMAT_ELF, OS_WINDOWS, Format, String, Characteristic
 
 
 def check_segment_for_pe(seg):
@@ -78,7 +79,7 @@ def extract_file_embedded_pe():
 
 
 def extract_file_export_names():
-    """ extract function exports """
+    """extract function exports"""
     for (_, _, ea, name) in idautils.Entries():
         yield Export(name), ea
 
@@ -143,8 +144,31 @@ def extract_file_strings():
             yield String(s.s), (seg.start_ea + s.offset)
 
 
+def extract_file_function_names():
+    """
+    extract the names of statically-linked library functions.
+    """
+    for ea in idautils.Functions():
+        if idaapi.get_func(ea).flags & idaapi.FUNC_LIB:
+            name = idaapi.get_name(ea)
+            yield FunctionName(name), ea
+
+
+def extract_file_format():
+    format_name = ida_loader.get_file_type_name()
+
+    if "PE" in format_name:
+        yield Format(FORMAT_PE), 0x0
+    elif "ELF64" in format_name:
+        yield Format(FORMAT_ELF), 0x0
+    elif "ELF32" in format_name:
+        yield Format(FORMAT_ELF), 0x0
+    else:
+        raise NotImplementedError("file format: %s", format_name)
+
+
 def extract_features():
-    """ extract file features """
+    """extract file features"""
     for file_handler in FILE_HANDLERS:
         for feature, va in file_handler():
             yield feature, va
@@ -156,6 +180,8 @@ FILE_HANDLERS = (
     extract_file_strings,
     extract_file_section_names,
     extract_file_embedded_pe,
+    extract_file_function_names,
+    extract_file_format,
 )
 
 

capa/features/extractors/ida/function.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -10,7 +10,7 @@ import idaapi
 import idautils
 
 import capa.features.extractors.ida.helpers
-from capa.features import Characteristic
+from capa.features.common import Characteristic
 from capa.features.extractors import loops
 
 

capa/features/extractors/ida/global_.py

@@ -0,0 +1,56 @@
+import logging
+import contextlib
+
+import idaapi
+import ida_loader
+
+import capa.ida.helpers
+import capa.features.extractors.elf
+from capa.features.common import OS, ARCH_I386, ARCH_AMD64, OS_WINDOWS, Arch
+
+logger = logging.getLogger(__name__)
+
+
+def extract_os():
+    format_name = ida_loader.get_file_type_name()
+
+    if "PE" in format_name:
+        yield OS(OS_WINDOWS), 0x0
+
+    elif "ELF" in format_name:
+        with contextlib.closing(capa.ida.helpers.IDAIO()) as f:
+            os = capa.features.extractors.elf.detect_elf_os(f)
+
+        yield OS(os), 0x0
+
+    else:
+        # we likely end up here:
+        #  1. handling shellcode, or
+        #  2. handling a new file format (e.g. macho)
+        #
+        # for (1) we can't do much - its shellcode and all bets are off.
+        # we could maybe accept a futher CLI argument to specify the OS,
+        # but i think this would be rarely used.
+        # rules that rely on OS conditions will fail to match on shellcode.
+        #
+        # for (2), this logic will need to be updated as the format is implemented.
+        logger.debug("unsupported file format: %s, will not guess OS", format_name)
+        return
+
+
+def extract_arch():
+    info = idaapi.get_inf_structure()
+    if info.procname == "metapc" and info.is_64bit():
+        yield Arch(ARCH_AMD64), 0x0
+    elif info.procname == "metapc" and info.is_32bit():
+        yield Arch(ARCH_I386), 0x0
+    elif info.procname == "metapc":
+        logger.debug("unsupported architecture: non-32-bit nor non-64-bit intel")
+        return
+    else:
+        # we likely end up here:
+        #  1. handling a new architecture (e.g. aarch64)
+        #
+        # for (1), this logic will need to be updated as the format is implemented.
+        logger.debug("unsupported architecture: %s", info.procname)
+        return

capa/features/extractors/ida/helpers.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,9 +6,6 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-import sys
-import string
-
 import idc
 import idaapi
 import idautils
@@ -23,11 +20,7 @@ def find_byte_sequence(start, end, seq):
         end: max virtual address
         seq: bytes to search e.g. b"\x01\x03"
     """
-    if sys.version_info[0] >= 3:
-        seq = " ".join(["%02x" % b for b in seq])
-    else:
-        seq = " ".join(["%02x" % ord(b) for b in seq])
-
+    seq = " ".join(["%02x" % b for b in seq])
     while True:
         ea = idaapi.find_binary(start, end, seq, 0, idaapi.SEARCH_DOWN)
         if ea == idaapi.BADADDR:
@@ -83,7 +76,7 @@ def get_segment_buffer(seg):
 
 
 def get_file_imports():
-    """ get file imports """
+    """get file imports"""
     imports = {}
 
     for idx in range(idaapi.get_import_module_qty()):
@@ -120,7 +113,7 @@ def get_instructions_in_range(start, end):
 
 
 def is_operand_equal(op1, op2):
-    """ compare two IDA op_t """
+    """compare two IDA op_t"""
     if op1.flags != op2.flags:
         return False
 
@@ -146,7 +139,7 @@ def is_operand_equal(op1, op2):
 
 
 def is_basic_block_equal(bb1, bb2):
-    """ compare two IDA BasicBlock """
+    """compare two IDA BasicBlock"""
     if bb1.start_ea != bb2.start_ea:
         return False
 
@@ -160,7 +153,7 @@ def is_basic_block_equal(bb1, bb2):
 
 
 def basic_block_size(bb):
-    """ calculate size of basic block """
+    """calculate size of basic block"""
     return bb.end_ea - bb.start_ea
 
 
@@ -178,7 +171,7 @@ def read_bytes_at(ea, count):
 
 
 def find_string_at(ea, min=4):
-    """ check if ASCII string exists at a given virtual address """
+    """check if ASCII string exists at a given virtual address"""
     found = idaapi.get_strlit_contents(ea, -1, idaapi.STRTYPE_C)
     if found and len(found) > min:
         try:
@@ -232,23 +225,23 @@ def get_op_phrase_info(op):
 
 
 def is_op_write(insn, op):
-    """ Check if an operand is written to (destination operand) """
+    """Check if an operand is written to (destination operand)"""
     return idaapi.has_cf_chg(insn.get_canon_feature(), op.n)
 
 
 def is_op_read(insn, op):
-    """ Check if an operand is read from (source operand) """
+    """Check if an operand is read from (source operand)"""
     return idaapi.has_cf_use(insn.get_canon_feature(), op.n)
 
 
 def is_op_offset(insn, op):
-    """ Check is an operand has been marked as an offset (by auto-analysis or manually) """
+    """Check is an operand has been marked as an offset (by auto-analysis or manually)"""
     flags = idaapi.get_flags(insn.ea)
     return ida_bytes.is_off(flags, op.n)
 
 
 def is_sp_modified(insn):
-    """ determine if instruction modifies SP, ESP, RSP """
+    """determine if instruction modifies SP, ESP, RSP"""
     for op in get_insn_ops(insn, target_ops=(idaapi.o_reg,)):
         if op.reg == idautils.procregs.sp.reg and is_op_write(insn, op):
             # register is stack and written
@@ -257,7 +250,7 @@ def is_sp_modified(insn):
 
 
 def is_bp_modified(insn):
-    """ check if instruction modifies BP, EBP, RBP """
+    """check if instruction modifies BP, EBP, RBP"""
     for op in get_insn_ops(insn, target_ops=(idaapi.o_reg,)):
         if op.reg == idautils.procregs.bp.reg and is_op_write(insn, op):
             # register is base and written
@@ -266,12 +259,12 @@ def is_bp_modified(insn):
 
 
 def is_frame_register(reg):
-    """ check if register is sp or bp """
+    """check if register is sp or bp"""
     return reg in (idautils.procregs.sp.reg, idautils.procregs.bp.reg)
 
 
 def get_insn_ops(insn, target_ops=()):
-    """ yield op_t for instruction, filter on type if specified """
+    """yield op_t for instruction, filter on type if specified"""
     for op in insn.ops:
         if op.type == idaapi.o_void:
             # avoid looping all 6 ops if only subset exists
@@ -282,7 +275,7 @@ def get_insn_ops(insn, target_ops=()):
 
 
 def is_op_stack_var(ea, index):
-    """ check if operand is a stack variable """
+    """check if operand is a stack variable"""
     return idaapi.is_stkvar(idaapi.get_flags(ea), index)
 
 
@@ -336,7 +329,7 @@ def is_basic_block_tight_loop(bb):
 
 
 def find_data_reference_from_insn(insn, max_depth=10):
-    """ search for data reference from instruction, return address of instruction if no reference exists """
+    """search for data reference from instruction, return address of instruction if no reference exists"""
     depth = 0
     ea = insn.ea
 
@@ -379,5 +372,5 @@ def get_function_blocks(f):
 
 
 def is_basic_block_return(bb):
-    """ check if basic block is return block """
+    """check if basic block is return block"""
     return bb.type == idaapi.fcb_ret

capa/features/extractors/ida/insn.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -12,38 +12,38 @@ import idautils
 
 import capa.features.extractors.helpers
 import capa.features.extractors.ida.helpers
-from capa.features import (
-    ARCH_X32,
-    ARCH_X64,
+from capa.features.insn import API, Number, Offset, Mnemonic
+from capa.features.common import (
+    BITNESS_X32,
+    BITNESS_X64,
     MAX_BYTES_FEATURE_SIZE,
     THUNK_CHAIN_DEPTH_DELTA,
     Bytes,
     String,
     Characteristic,
 )
-from capa.features.insn import API, Number, Offset, Mnemonic
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
 # byte range within the first and returning basic blocks, this helps to reduce FP features
 SECURITY_COOKIE_BYTES_DELTA = 0x40
 
 
-def get_arch(ctx):
+def get_bitness(ctx):
     """
-    fetch the ARCH_* constant for the currently open workspace.
+    fetch the BITNESS_* constant for the currently open workspace.
 
     via Tamir Bahar/@tmr232
     https://reverseengineering.stackexchange.com/a/11398/17194
     """
-    if "arch" not in ctx:
+    if "bitness" not in ctx:
         info = idaapi.get_inf_structure()
         if info.is_64bit():
-            ctx["arch"] = ARCH_X64
+            ctx["bitness"] = BITNESS_X64
         elif info.is_32bit():
-            ctx["arch"] = ARCH_X32
+            ctx["bitness"] = BITNESS_X32
         else:
-            raise ValueError("unexpected architecture")
-    return ctx["arch"]
+            raise ValueError("unexpected bitness")
+    return ctx["bitness"]
 
 
 def get_imports(ctx):
@@ -53,10 +53,7 @@ def get_imports(ctx):
 
 
 def check_for_api_call(ctx, insn):
-    """ check instruction for API call """
-    if not insn.get_canon_mnem() in ("call", "jmp"):
-        return
-
+    """check instruction for API call"""
     info = ()
     ref = insn.ea
 
@@ -95,11 +92,29 @@ def extract_insn_api_features(f, bb, insn):
     example:
         call dword [0x00473038]
     """
+    if not insn.get_canon_mnem() in ("call", "jmp"):
+        return
+
     for api in check_for_api_call(f.ctx, insn):
         dll, _, symbol = api.rpartition(".")
         for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
             yield API(name), insn.ea
 
+    # extract IDA/FLIRT recognized API functions
+    targets = tuple(idautils.CodeRefsFrom(insn.ea, False))
+    if not targets:
+        return
+
+    target = targets[0]
+    target_func = idaapi.get_func(target)
+    if not target_func or target_func.start_ea != target:
+        # not a function (start)
+        return
+
+    if target_func.flags & idaapi.FUNC_LIB:
+        name = idaapi.get_name(target_func.start_ea)
+        yield API(name), insn.ea
+
 
 def extract_insn_number_features(f, bb, insn):
     """parse instruction number features
@@ -134,7 +149,7 @@ def extract_insn_number_features(f, bb, insn):
             const = op.addr
 
         yield Number(const), insn.ea
-        yield Number(const, arch=get_arch(f.ctx)), insn.ea
+        yield Number(const, bitness=get_bitness(f.ctx)), insn.ea
 
 
 def extract_insn_bytes_features(f, bb, insn):
@@ -203,7 +218,7 @@ def extract_insn_offset_features(f, bb, insn):
         op_off = capa.features.extractors.helpers.twos_complement(op_off, 32)
 
         yield Offset(op_off), insn.ea
-        yield Offset(op_off, arch=get_arch(f.ctx)), insn.ea
+        yield Offset(op_off, bitness=get_bitness(f.ctx)), insn.ea
 
 
 def contains_stack_cookie_keywords(s):
@@ -256,7 +271,7 @@ def bb_stack_cookie_registers(bb):
 
 
 def is_nzxor_stack_cookie_delta(f, bb, insn):
-    """ check if nzxor exists within stack cookie delta """
+    """check if nzxor exists within stack cookie delta"""
     # security cookie check should use SP or BP
     if not capa.features.extractors.ida.helpers.is_frame_register(insn.Op2.reg):
         return False
@@ -279,7 +294,7 @@ def is_nzxor_stack_cookie_delta(f, bb, insn):
 
 
 def is_nzxor_stack_cookie(f, bb, insn):
-    """ check if nzxor is related to stack cookie """
+    """check if nzxor is related to stack cookie"""
     if contains_stack_cookie_keywords(idaapi.get_cmt(insn.ea, False)):
         # Example:
         #   xor     ecx, ebp        ; StackCookie
@@ -322,7 +337,7 @@ def extract_insn_mnemonic_features(f, bb, insn):
         bb (IDA BasicBlock)
         insn (IDA insn_t)
     """
-    yield Mnemonic(insn.get_canon_mnem()), insn.ea
+    yield Mnemonic(idc.print_insn_mnem(insn.ea)), insn.ea
 
 
 def extract_insn_peb_access_characteristic_features(f, bb, insn):

capa/features/extractors/loops.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,7 +6,7 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-from networkx import nx
+import networkx
 from networkx.algorithms.components import strongly_connected_components
 
 
@@ -20,6 +20,6 @@ def has_loop(edges, threshold=2):
     returns:
         bool
     """
-    g = nx.DiGraph()
+    g = networkx.DiGraph()
     g.add_edges_from(edges)
     return any(len(comp) >= threshold for comp in strongly_connected_components(g))

capa/features/extractors/pefile.py

@@ -0,0 +1,215 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+import logging
+
+import pefile
+
+import capa.features.common
+import capa.features.extractors
+import capa.features.extractors.common
+import capa.features.extractors.helpers
+import capa.features.extractors.strings
+from capa.features.file import Export, Import, Section
+from capa.features.common import OS, ARCH_I386, FORMAT_PE, ARCH_AMD64, OS_WINDOWS, Arch, Format, Characteristic
+from capa.features.extractors.base_extractor import FeatureExtractor
+
+logger = logging.getLogger(__name__)
+
+
+def extract_file_embedded_pe(buf, **kwargs):
+    for offset, _ in capa.features.extractors.helpers.carve_pe(buf, 1):
+        yield Characteristic("embedded pe"), offset
+
+
+def extract_file_export_names(pe, **kwargs):
+    base_address = pe.OPTIONAL_HEADER.ImageBase
+
+    if hasattr(pe, "DIRECTORY_ENTRY_EXPORT"):
+        for export in pe.DIRECTORY_ENTRY_EXPORT.symbols:
+            if not export.name:
+                continue
+            try:
+                name = export.name.partition(b"\x00")[0].decode("ascii")
+            except UnicodeDecodeError:
+                continue
+            va = base_address + export.address
+            yield Export(name), va
+
+
+def extract_file_import_names(pe, **kwargs):
+    """
+    extract imported function names
+    1. imports by ordinal:
+     - modulename.#ordinal
+    2. imports by name, results in two features to support importname-only matching:
+     - modulename.importname
+     - importname
+    """
+    if hasattr(pe, "DIRECTORY_ENTRY_IMPORT"):
+        for dll in pe.DIRECTORY_ENTRY_IMPORT:
+            try:
+                modname = dll.dll.partition(b"\x00")[0].decode("ascii")
+            except UnicodeDecodeError:
+                continue
+
+            # strip extension
+            modname = modname.rpartition(".")[0].lower()
+
+            for imp in dll.imports:
+                if imp.import_by_ordinal:
+                    impname = "#%s" % imp.ordinal
+                else:
+                    try:
+                        impname = imp.name.partition(b"\x00")[0].decode("ascii")
+                    except UnicodeDecodeError:
+                        continue
+
+                for name in capa.features.extractors.helpers.generate_symbols(modname, impname):
+                    yield Import(name), imp.address
+
+
+def extract_file_section_names(pe, **kwargs):
+    base_address = pe.OPTIONAL_HEADER.ImageBase
+
+    for section in pe.sections:
+        try:
+            name = section.Name.partition(b"\x00")[0].decode("ascii")
+        except UnicodeDecodeError:
+            continue
+
+        yield Section(name), base_address + section.VirtualAddress
+
+
+def extract_file_strings(buf, **kwargs):
+    yield from capa.features.extractors.common.extract_file_strings(buf)
+
+
+def extract_file_function_names(**kwargs):
+    """
+    extract the names of statically-linked library functions.
+    """
+    if False:
+        # using a `yield` here to force this to be a generator, not function.
+        yield NotImplementedError("pefile doesn't have library matching")
+    return
+
+
+def extract_file_os(**kwargs):
+    # assuming PE -> Windows
+    # though i suppose they're also used by UEFI
+    yield OS(OS_WINDOWS), 0x0
+
+
+def extract_file_format(**kwargs):
+    yield Format(FORMAT_PE), 0x0
+
+
+def extract_file_arch(pe, **kwargs):
+    if pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_I386"]:
+        yield Arch(ARCH_I386), 0x0
+    elif pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_AMD64"]:
+        yield Arch(ARCH_AMD64), 0x0
+    else:
+        logger.warning("unsupported architecture: %s", pefile.MACHINE_TYPE[pe.FILE_HEADER.Machine])
+
+
+def extract_file_features(pe, buf):
+    """
+    extract file features from given workspace
+
+    args:
+      pe (pefile.PE): the parsed PE
+      buf: the raw sample bytes
+
+    yields:
+      Tuple[Feature, VA]: a feature and its location.
+    """
+
+    for file_handler in FILE_HANDLERS:
+        for feature, va in file_handler(pe=pe, buf=buf):
+            yield feature, va
+
+
+FILE_HANDLERS = (
+    extract_file_embedded_pe,
+    extract_file_export_names,
+    extract_file_import_names,
+    extract_file_section_names,
+    extract_file_strings,
+    extract_file_function_names,
+    extract_file_format,
+)
+
+
+def extract_global_features(pe, buf):
+    """
+    extract global features from given workspace
+
+    args:
+      pe (pefile.PE): the parsed PE
+      buf: the raw sample bytes
+
+    yields:
+      Tuple[Feature, VA]: a feature and its location.
+    """
+    for handler in GLOBAL_HANDLERS:
+        for feature, va in handler(pe=pe, buf=buf):
+            yield feature, va
+
+
+GLOBAL_HANDLERS = (
+    extract_file_os,
+    extract_file_arch,
+)
+
+
+class PefileFeatureExtractor(FeatureExtractor):
+    def __init__(self, path: str):
+        super(PefileFeatureExtractor, self).__init__()
+        self.path = path
+        self.pe = pefile.PE(path)
+
+    def get_base_address(self):
+        return self.pe.OPTIONAL_HEADER.ImageBase
+
+    def extract_global_features(self):
+        with open(self.path, "rb") as f:
+            buf = f.read()
+
+        yield from extract_global_features(self.pe, buf)
+
+    def extract_file_features(self):
+        with open(self.path, "rb") as f:
+            buf = f.read()
+
+        yield from extract_file_features(self.pe, buf)
+
+    def get_functions(self):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
+
+    def extract_function_features(self, f):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
+
+    def get_basic_blocks(self, f):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
+
+    def extract_basic_block_features(self, f, bb):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
+
+    def get_instructions(self, f, bb):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
+
+    def extract_insn_features(self, f, bb, insn):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
+
+    def is_library_function(self, va):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
+
+    def get_function_name(self, va):
+        raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")

capa/features/extractors/smda/__init__.py

@@ -1,52 +0,0 @@
-import sys
-import types
-
-from smda.common.SmdaReport import SmdaReport
-from smda.common.SmdaInstruction import SmdaInstruction
-
-import capa.features.extractors.smda.file
-import capa.features.extractors.smda.insn
-import capa.features.extractors.smda.function
-import capa.features.extractors.smda.basicblock
-from capa.main import UnsupportedRuntimeError
-from capa.features.extractors import FeatureExtractor
-
-
-class SmdaFeatureExtractor(FeatureExtractor):
-    def __init__(self, smda_report: SmdaReport, path):
-        super(SmdaFeatureExtractor, self).__init__()
-        if sys.version_info < (3, 0):
-            raise UnsupportedRuntimeError("SMDA should only be used with Python 3.")
-        self.smda_report = smda_report
-        self.path = path
-
-    def get_base_address(self):
-        return self.smda_report.base_addr
-
-    def extract_file_features(self):
-        for feature, va in capa.features.extractors.smda.file.extract_features(self.smda_report, self.path):
-            yield feature, va
-
-    def get_functions(self):
-        for function in self.smda_report.getFunctions():
-            yield function
-
-    def extract_function_features(self, f):
-        for feature, va in capa.features.extractors.smda.function.extract_features(f):
-            yield feature, va
-
-    def get_basic_blocks(self, f):
-        for bb in f.getBlocks():
-            yield bb
-
-    def extract_basic_block_features(self, f, bb):
-        for feature, va in capa.features.extractors.smda.basicblock.extract_features(f, bb):
-            yield feature, va
-
-    def get_instructions(self, f, bb):
-        for smda_ins in bb.getInstructions():
-            yield smda_ins
-
-    def extract_insn_features(self, f, bb, insn):
-        for feature, va in capa.features.extractors.smda.insn.extract_features(f, bb, insn):
-            yield feature, va

capa/features/extractors/smda/basicblock.py

@@ -1,8 +1,7 @@
-import sys
 import string
 import struct
 
-from capa.features import Characteristic
+from capa.features.common import Characteristic
 from capa.features.basicblock import BasicBlock
 from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
 
@@ -15,7 +14,7 @@ def _bb_has_tight_loop(f, bb):
 
 
 def extract_bb_tight_loop(f, bb):
-    """ check basic block for tight loop indicators """
+    """check basic block for tight loop indicators"""
     if _bb_has_tight_loop(f, bb):
         yield Characteristic("tight loop"), bb.offset
 
@@ -39,7 +38,7 @@ def get_operands(smda_ins):
 
 
 def extract_stackstring(f, bb):
-    """ check basic block for stackstring indicators """
+    """check basic block for stackstring indicators"""
     if _bb_has_stackstring(f, bb):
         yield Characteristic("stack string"), bb.offset
 
@@ -117,7 +116,7 @@ def extract_features(f, bb):
       bb (smda.common.SmdaBasicBlock): the basic block to process.
 
     yields:
-      Feature, set[VA]: the features and their location found in this basic block.
+      Tuple[Feature, int]: the features and their location found in this basic block.
     """
     yield BasicBlock(), bb.offset
     for bb_handler in BASIC_BLOCK_HANDLERS:

capa/features/extractors/smda/extractor.py

@@ -0,0 +1,53 @@
+from smda.common.SmdaReport import SmdaReport
+
+import capa.features.extractors.common
+import capa.features.extractors.smda.file
+import capa.features.extractors.smda.insn
+import capa.features.extractors.smda.global_
+import capa.features.extractors.smda.function
+import capa.features.extractors.smda.basicblock
+from capa.features.extractors.base_extractor import FeatureExtractor
+
+
+class SmdaFeatureExtractor(FeatureExtractor):
+    def __init__(self, smda_report: SmdaReport, path):
+        super(SmdaFeatureExtractor, self).__init__()
+        self.smda_report = smda_report
+        self.path = path
+        with open(self.path, "rb") as f:
+            self.buf = f.read()
+
+        # pre-compute these because we'll yield them at *every* scope.
+        self.global_features = []
+        self.global_features.extend(capa.features.extractors.common.extract_os(self.buf))
+        self.global_features.extend(capa.features.extractors.smda.global_.extract_arch(self.smda_report))
+
+    def get_base_address(self):
+        return self.smda_report.base_addr
+
+    def extract_global_features(self):
+        yield from self.global_features
+
+    def extract_file_features(self):
+        yield from capa.features.extractors.smda.file.extract_features(self.smda_report, self.buf)
+
+    def get_functions(self):
+        for function in self.smda_report.getFunctions():
+            yield function
+
+    def extract_function_features(self, f):
+        yield from capa.features.extractors.smda.function.extract_features(f)
+
+    def get_basic_blocks(self, f):
+        for bb in f.getBlocks():
+            yield bb
+
+    def extract_basic_block_features(self, f, bb):
+        yield from capa.features.extractors.smda.basicblock.extract_features(f, bb)
+
+    def get_instructions(self, f, bb):
+        for smda_ins in bb.getInstructions():
+            yield smda_ins
+
+    def extract_insn_features(self, f, bb, insn):
+        yield from capa.features.extractors.smda.insn.extract_features(f, bb, insn)

capa/features/extractors/smda/file.py

@@ -1,86 +1,37 @@
-import struct
-
 # if we have SMDA we definitely have lief
 import lief
 
+import capa.features.extractors.common
 import capa.features.extractors.helpers
 import capa.features.extractors.strings
-from capa.features import String, Characteristic
 from capa.features.file import Export, Import, Section
+from capa.features.common import String, Characteristic
 
 
-def carve(pbytes, offset=0):
-    """
-    Return a list of (offset, size, xor) tuples of embedded PEs
-
-    Based on the version from vivisect:
-    https://github.com/vivisect/vivisect/blob/7be4037b1cecc4551b397f840405a1fc606f9b53/PE/carve.py#L19
-    And its IDA adaptation:
-    capa/features/extractors/ida/file.py
-    """
-    mz_xor = [
-        (
-            capa.features.extractors.helpers.xor_static(b"MZ", i),
-            capa.features.extractors.helpers.xor_static(b"PE", i),
-            i,
-        )
-        for i in range(256)
-    ]
-
-    pblen = len(pbytes)
-    todo = [(pbytes.find(mzx, offset), mzx, pex, i) for mzx, pex, i in mz_xor]
-    todo = [(off, mzx, pex, i) for (off, mzx, pex, i) in todo if off != -1]
-
-    while len(todo):
-
-        off, mzx, pex, i = todo.pop()
-
-        # The MZ header has one field we will check
-        # e_lfanew is at 0x3c
-        e_lfanew = off + 0x3C
-        if pblen < (e_lfanew + 4):
-            continue
-
-        newoff = struct.unpack("<I", capa.features.extractors.helpers.xor_static(pbytes[e_lfanew : e_lfanew + 4], i))[0]
-
-        nextres = pbytes.find(mzx, off + 1)
-        if nextres != -1:
-            todo.append((nextres, mzx, pex, i))
-
-        peoff = off + newoff
-        if pblen < (peoff + 2):
-            continue
-
-        if pbytes[peoff : peoff + 2] == pex:
-            yield (off, i)
-
-
-def extract_file_embedded_pe(smda_report, file_path):
-    with open(file_path, "rb") as f:
-        fbytes = f.read()
-
-    for offset, i in carve(fbytes, 1):
+def extract_file_embedded_pe(buf, **kwargs):
+    for offset, _ in capa.features.extractors.helpers.carve_pe(buf, 1):
         yield Characteristic("embedded pe"), offset
 
 
-def extract_file_export_names(smda_report, file_path):
-    lief_binary = lief.parse(file_path)
+def extract_file_export_names(buf, **kwargs):
+    lief_binary = lief.parse(buf)
+
     if lief_binary is not None:
         for function in lief_binary.exported_functions:
             yield Export(function.name), function.address
 
 
-def extract_file_import_names(smda_report, file_path):
+def extract_file_import_names(smda_report, buf):
     # extract import table info via LIEF
-    lief_binary = lief.parse(file_path)
+    lief_binary = lief.parse(buf)
     if not isinstance(lief_binary, lief.PE.Binary):
         return
     for imported_library in lief_binary.imports:
         library_name = imported_library.name.lower()
         library_name = library_name[:-4] if library_name.endswith(".dll") else library_name
         for func in imported_library.entries:
+            va = func.iat_address + smda_report.base_addr
             if func.name:
-                va = func.iat_address + smda_report.base_addr
                 for name in capa.features.extractors.helpers.generate_symbols(library_name, func.name):
                     yield Import(name), va
             elif func.is_ordinal:
@@ -88,8 +39,8 @@ def extract_file_import_names(smda_report, file_path):
                     yield Import(name), va
 
 
-def extract_file_section_names(smda_report, file_path):
-    lief_binary = lief.parse(file_path)
+def extract_file_section_names(buf, **kwargs):
+    lief_binary = lief.parse(buf)
     if not isinstance(lief_binary, lief.PE.Binary):
         return
     if lief_binary and lief_binary.sections:
@@ -98,35 +49,45 @@ def extract_file_section_names(smda_report, file_path):
             yield Section(section.name), base_address + section.virtual_address
 
 
-def extract_file_strings(smda_report, file_path):
+def extract_file_strings(buf, **kwargs):
     """
     extract ASCII and UTF-16 LE strings from file
     """
-    with open(file_path, "rb") as f:
-        b = f.read()
-
-    for s in capa.features.extractors.strings.extract_ascii_strings(b):
+    for s in capa.features.extractors.strings.extract_ascii_strings(buf):
         yield String(s.s), s.offset
 
-    for s in capa.features.extractors.strings.extract_unicode_strings(b):
+    for s in capa.features.extractors.strings.extract_unicode_strings(buf):
         yield String(s.s), s.offset
 
 
-def extract_features(smda_report, file_path):
+def extract_file_function_names(smda_report, **kwargs):
+    """
+    extract the names of statically-linked library functions.
+    """
+    if False:
+        # using a `yield` here to force this to be a generator, not function.
+        yield NotImplementedError("SMDA doesn't have library matching")
+    return
+
+
+def extract_file_format(buf, **kwargs):
+    yield from capa.features.extractors.common.extract_format(buf)
+
+
+def extract_features(smda_report, buf):
     """
     extract file features from given workspace
 
     args:
       smda_report (smda.common.SmdaReport): a SmdaReport
-      file_path: path to the input file
+      buf: the raw bytes of the sample
 
     yields:
       Tuple[Feature, VA]: a feature and its location.
     """
 
     for file_handler in FILE_HANDLERS:
-        result = file_handler(smda_report, file_path)
-        for feature, va in file_handler(smda_report, file_path):
+        for feature, va in file_handler(smda_report=smda_report, buf=buf):
             yield feature, va
 
 
@@ -136,4 +97,6 @@ FILE_HANDLERS = (
     extract_file_import_names,
     extract_file_section_names,
     extract_file_strings,
+    extract_file_function_names,
+    extract_file_format,
 )

capa/features/extractors/smda/function.py

@@ -1,4 +1,4 @@
-from capa.features import Characteristic
+from capa.features.common import Characteristic
 from capa.features.extractors import loops
 
 
@@ -28,7 +28,7 @@ def extract_features(f):
       f (smda.common.SmdaFunction): the function from which to extract features
 
     yields:
-      Feature, set[VA]: the features and their location found in this function.
+      Tuple[Feature, int]: the features and their location found in this function.
     """
     for func_handler in FUNCTION_HANDLERS:
         for feature, va in func_handler(f):

capa/features/extractors/smda/global_.py

@@ -0,0 +1,20 @@
+import logging
+
+from capa.features.common import ARCH_I386, ARCH_AMD64, Arch
+
+logger = logging.getLogger(__name__)
+
+
+def extract_arch(smda_report):
+    if smda_report.architecture == "intel":
+        if smda_report.bitness == 32:
+            yield Arch(ARCH_I386), 0x0
+        elif smda_report.bitness == 64:
+            yield Arch(ARCH_AMD64), 0x0
+    else:
+        # we likely end up here:
+        #  1. handling a new architecture (e.g. aarch64)
+        #
+        # for (1), this logic will need to be updated as the format is implemented.
+        logger.debug("unsupported architecture: %s", smda_report.architecture)
+        return

capa/features/extractors/smda/insn.py

@@ -5,16 +5,16 @@ import struct
 from smda.common.SmdaReport import SmdaReport
 
 import capa.features.extractors.helpers
-from capa.features import (
-    ARCH_X32,
-    ARCH_X64,
+from capa.features.insn import API, Number, Offset, Mnemonic
+from capa.features.common import (
+    BITNESS_X32,
+    BITNESS_X64,
     MAX_BYTES_FEATURE_SIZE,
     THUNK_CHAIN_DEPTH_DELTA,
     Bytes,
     String,
     Characteristic,
 )
-from capa.features.insn import API, Number, Offset, Mnemonic
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
 # byte range within the first and returning basic blocks, this helps to reduce FP features
@@ -23,12 +23,12 @@ PATTERN_HEXNUM = re.compile(r"[+\-] (?P<num>0x[a-fA-F0-9]+)")
 PATTERN_SINGLENUM = re.compile(r"[+\-] (?P<num>[0-9])")
 
 
-def get_arch(smda_report):
+def get_bitness(smda_report):
     if smda_report.architecture == "intel":
         if smda_report.bitness == 32:
-            return ARCH_X32
+            return BITNESS_X32
         elif smda_report.bitness == 64:
-            return ARCH_X64
+            return BITNESS_X64
     else:
         raise NotImplementedError
 
@@ -84,8 +84,12 @@ def extract_insn_number_features(f, bb, insn):
         return
     for operand in operands:
         try:
-            yield Number(int(operand, 16)), insn.offset
-            yield Number(int(operand, 16), arch=get_arch(f.smda_report)), insn.offset
+            # The result of bitwise operations is calculated as though carried out
+            # in two’s complement with an infinite number of sign bits
+            value = int(operand, 16) & ((1 << f.smda_report.bitness) - 1)
+
+            yield Number(value), insn.offset
+            yield Number(value, bitness=get_bitness(f.smda_report)), insn.offset
         except:
             continue
 
@@ -97,7 +101,7 @@ def read_bytes(smda_report, va, num_bytes=None):
 
     rva = va - smda_report.base_addr
     if smda_report.buffer is None:
-        return
+        raise ValueError("buffer is empty")
     buffer_end = len(smda_report.buffer)
     max_bytes = num_bytes if num_bytes is not None else MAX_BYTES_FEATURE_SIZE
     if rva + max_bytes > buffer_end:
@@ -228,7 +232,7 @@ def extract_insn_offset_features(f, bb, insn):
             number = int(number_int.group("num"))
             number = -1 * number if number_int.group().startswith("-") else number
         yield Offset(number), insn.offset
-        yield Offset(number, arch=get_arch(f.smda_report)), insn.offset
+        yield Offset(number, bitness=get_bitness(f.smda_report)), insn.offset
 
 
 def is_security_cookie(f, bb, insn):
@@ -293,7 +297,7 @@ def extract_insn_peb_access_characteristic_features(f, bb, insn):
 
 
 def extract_insn_segment_access_features(f, bb, insn):
-    """ parse the instruction for access to fs or gs """
+    """parse the instruction for access to fs or gs"""
     operands = [o.strip() for o in insn.operands.split(",")]
     for operand in operands:
         if "fs:" in operand:
@@ -370,7 +374,7 @@ def extract_features(f, bb, insn):
       insn (smda.common.SmdaInstruction): the instruction to process.
 
     yields:
-      Feature, set[VA]: the features and their location found in this insn.
+      Tuple[Feature, int]: the features and their location found in this insn.
     """
     for insn_handler in INSTRUCTION_HANDLERS:
         for feature, va in insn_handler(f, bb, insn):

capa/features/extractors/strings.py

@@ -1,6 +1,6 @@
-# strings code from FLOSS, https://github.com/fireeye/flare-floss
+# strings code from FLOSS, https://github.com/mandiant/flare-floss
 #
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/features/extractors/viv/__init__.py

@@ -1,81 +0,0 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-
-import types
-
-import viv_utils
-
-import capa.features.extractors
-import capa.features.extractors.viv.file
-import capa.features.extractors.viv.insn
-import capa.features.extractors.viv.function
-import capa.features.extractors.viv.basicblock
-from capa.features.extractors import FeatureExtractor
-
-__all__ = ["file", "function", "basicblock", "insn"]
-
-
-def get_va(self):
-    try:
-        # vivisect type
-        return self.va
-    except AttributeError:
-        pass
-
-    raise TypeError()
-
-
-def add_va_int_cast(o):
-    """
-    dynamically add a cast-to-int (`__int__`) method to the given object
-    that returns the value of the `.va` property.
-
-    this bit of skullduggery lets use cast viv-utils objects as ints.
-    the correct way of doing this is to update viv-utils (or subclass the objects here).
-    """
-    setattr(o, "__int__", types.MethodType(get_va, o))
-    return o
-
-
-class VivisectFeatureExtractor(FeatureExtractor):
-    def __init__(self, vw, path):
-        super(VivisectFeatureExtractor, self).__init__()
-        self.vw = vw
-        self.path = path
-
-    def get_base_address(self):
-        # assume there is only one file loaded into the vw
-        return list(self.vw.filemeta.values())[0]["imagebase"]
-
-    def extract_file_features(self):
-        for feature, va in capa.features.extractors.viv.file.extract_features(self.vw, self.path):
-            yield feature, va
-
-    def get_functions(self):
-        for va in sorted(self.vw.getFunctions()):
-            yield add_va_int_cast(viv_utils.Function(self.vw, va))
-
-    def extract_function_features(self, f):
-        for feature, va in capa.features.extractors.viv.function.extract_features(f):
-            yield feature, va
-
-    def get_basic_blocks(self, f):
-        for bb in f.basic_blocks:
-            yield add_va_int_cast(bb)
-
-    def extract_basic_block_features(self, f, bb):
-        for feature, va in capa.features.extractors.viv.basicblock.extract_features(f, bb):
-            yield feature, va
-
-    def get_instructions(self, f, bb):
-        for insn in bb.instructions:
-            yield add_va_int_cast(insn)
-
-    def extract_insn_features(self, f, bb, insn):
-        for feature, va in capa.features.extractors.viv.insn.extract_features(f, bb, insn):
-            yield feature, va

capa/features/extractors/viv/basicblock.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -10,9 +10,9 @@ import string
 import struct
 
 import envi
-import vivisect.const
+import envi.archs.i386.disasm
 
-from capa.features import Characteristic
+from capa.features.common import Characteristic
 from capa.features.basicblock import BasicBlock
 from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
 
@@ -37,7 +37,7 @@ def _bb_has_tight_loop(f, bb):
     """
     if len(bb.instructions) > 0:
         for bva, bflags in bb.instructions[-1].getBranches():
-            if bflags & vivisect.envi.BR_COND:
+            if bflags & envi.BR_COND:
                 if bva == bb.va:
                     return True
 
@@ -45,7 +45,7 @@ def _bb_has_tight_loop(f, bb):
 
 
 def extract_bb_tight_loop(f, bb):
-    """ check basic block for tight loop indicators """
+    """check basic block for tight loop indicators"""
     if _bb_has_tight_loop(f, bb):
         yield Characteristic("tight loop"), bb.va
 
@@ -68,12 +68,12 @@ def _bb_has_stackstring(f, bb):
 
 
 def extract_stackstring(f, bb):
-    """ check basic block for stackstring indicators """
+    """check basic block for stackstring indicators"""
     if _bb_has_stackstring(f, bb):
         yield Characteristic("stack string"), bb.va
 
 
-def is_mov_imm_to_stack(instr):
+def is_mov_imm_to_stack(instr: envi.archs.i386.disasm.i386Opcode) -> bool:
     """
     Return if instruction moves immediate onto stack
     """
@@ -105,7 +105,7 @@ def is_mov_imm_to_stack(instr):
     return True
 
 
-def get_printable_len(oper):
+def get_printable_len(oper: envi.archs.i386.disasm.i386ImmOper) -> int:
     """
     Return string length if all operand bytes are ascii or utf16-le printable
     """
@@ -117,14 +117,18 @@ def get_printable_len(oper):
         chars = struct.pack("<I", oper.imm)
     elif oper.tsize == 8:
         chars = struct.pack("<Q", oper.imm)
+    else:
+        raise ValueError("unexpected oper.tsize: %d" % (oper.tsize))
+
     if is_printable_ascii(chars):
         return oper.tsize
-    if is_printable_utf16le(chars):
+    elif is_printable_utf16le(chars):
         return oper.tsize / 2
-    return 0
+    else:
+        return 0
 
 
-def is_printable_ascii(chars):
+def is_printable_ascii(chars: bytes) -> bool:
     try:
         chars_str = chars.decode("ascii")
     except UnicodeDecodeError:
@@ -133,9 +137,10 @@ def is_printable_ascii(chars):
         return all(c in string.printable for c in chars_str)
 
 
-def is_printable_utf16le(chars):
+def is_printable_utf16le(chars: bytes) -> bool:
     if all(c == b"\x00" for c in chars[1::2]):
         return is_printable_ascii(chars[::2])
+    return False
 
 
 def extract_features(f, bb):
@@ -147,7 +152,7 @@ def extract_features(f, bb):
       bb (viv_utils.BasicBlock): the basic block to process.
 
     yields:
-      Feature, set[VA]: the features and their location found in this basic block.
+      Tuple[Feature, int]: the features and their location found in this basic block.
     """
     yield BasicBlock(), bb.va
     for bb_handler in BASIC_BLOCK_HANDLERS:

capa/features/extractors/viv/extractor.py

@@ -0,0 +1,84 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import logging
+
+import viv_utils
+import viv_utils.flirt
+
+import capa.features.extractors.common
+import capa.features.extractors.viv.file
+import capa.features.extractors.viv.insn
+import capa.features.extractors.viv.global_
+import capa.features.extractors.viv.function
+import capa.features.extractors.viv.basicblock
+from capa.features.extractors.base_extractor import FeatureExtractor
+
+logger = logging.getLogger(__name__)
+
+
+class InstructionHandle:
+    """this acts like a vivisect.Opcode but with an __int__() method"""
+
+    def __init__(self, inner):
+        self._inner = inner
+
+    def __int__(self):
+        return self.va
+
+    def __getattr__(self, name):
+        return getattr(self._inner, name)
+
+
+class VivisectFeatureExtractor(FeatureExtractor):
+    def __init__(self, vw, path):
+        super(VivisectFeatureExtractor, self).__init__()
+        self.vw = vw
+        self.path = path
+        with open(self.path, "rb") as f:
+            self.buf = f.read()
+
+        # pre-compute these because we'll yield them at *every* scope.
+        self.global_features = []
+        self.global_features.extend(capa.features.extractors.common.extract_os(self.buf))
+        self.global_features.extend(capa.features.extractors.viv.global_.extract_arch(self.vw))
+
+    def get_base_address(self):
+        # assume there is only one file loaded into the vw
+        return list(self.vw.filemeta.values())[0]["imagebase"]
+
+    def extract_global_features(self):
+        yield from self.global_features
+
+    def extract_file_features(self):
+        yield from capa.features.extractors.viv.file.extract_features(self.vw, self.buf)
+
+    def get_functions(self):
+        for va in sorted(self.vw.getFunctions()):
+            yield viv_utils.Function(self.vw, va)
+
+    def extract_function_features(self, f):
+        yield from capa.features.extractors.viv.function.extract_features(f)
+
+    def get_basic_blocks(self, f):
+        return f.basic_blocks
+
+    def extract_basic_block_features(self, f, bb):
+        yield from capa.features.extractors.viv.basicblock.extract_features(f, bb)
+
+    def get_instructions(self, f, bb):
+        for insn in bb.instructions:
+            yield InstructionHandle(insn)
+
+    def extract_insn_features(self, f, bb, insn):
+        yield from capa.features.extractors.viv.insn.extract_features(f, bb, insn)
+
+    def is_library_function(self, va):
+        return viv_utils.flirt.is_library_function(self.vw, va)
+
+    def get_function_name(self, va):
+        return viv_utils.get_function_name(self.vw, va)

capa/features/extractors/viv/file.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -7,27 +7,28 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 import PE.carve as pe_carve  # vivisect PE
+import viv_utils
+import viv_utils.flirt
 
+import capa.features.insn
+import capa.features.extractors.common
 import capa.features.extractors.helpers
 import capa.features.extractors.strings
-from capa.features import String, Characteristic
-from capa.features.file import Export, Import, Section
+from capa.features.file import Export, Import, Section, FunctionName
+from capa.features.common import String, Characteristic
 
 
-def extract_file_embedded_pe(vw, file_path):
-    with open(file_path, "rb") as f:
-        fbytes = f.read()
-
-    for offset, i in pe_carve.carve(fbytes, 1):
+def extract_file_embedded_pe(buf, **kwargs):
+    for offset, _ in pe_carve.carve(buf, 1):
         yield Characteristic("embedded pe"), offset
 
 
-def extract_file_export_names(vw, file_path):
-    for va, etype, name, _ in vw.getExports():
+def extract_file_export_names(vw, **kwargs):
+    for va, _, name, _ in vw.getExports():
         yield Export(name), va
 
 
-def extract_file_import_names(vw, file_path):
+def extract_file_import_names(vw, **kwargs):
     """
     extract imported function names
     1. imports by ordinal:
@@ -38,7 +39,7 @@ def extract_file_import_names(vw, file_path):
     """
     for va, _, _, tinfo in vw.getImports():
         # vivisect source: tinfo = "%s.%s" % (libname, impname)
-        modname, impname = tinfo.split(".")
+        modname, impname = tinfo.split(".", 1)
         if is_viv_ord_impname(impname):
             # replace ord prefix with #
             impname = "#%s" % impname[len("ord") :]
@@ -47,7 +48,7 @@ def extract_file_import_names(vw, file_path):
             yield Import(name), va
 
 
-def is_viv_ord_impname(impname):
+def is_viv_ord_impname(impname: str) -> bool:
     """
     return if import name matches vivisect's ordinal naming scheme `'ord%d' % ord`
     """
@@ -61,39 +62,43 @@ def is_viv_ord_impname(impname):
         return True
 
 
-def extract_file_section_names(vw, file_path):
+def extract_file_section_names(vw, **kwargs):
     for va, _, segname, _ in vw.getSegments():
         yield Section(segname), va
 
 
-def extract_file_strings(vw, file_path):
+def extract_file_strings(buf, **kwargs):
+    yield from capa.features.extractors.common.extract_file_strings(buf)
+
+
+def extract_file_function_names(vw, **kwargs):
     """
-    extract ASCII and UTF-16 LE strings from file
+    extract the names of statically-linked library functions.
     """
-    with open(file_path, "rb") as f:
-        b = f.read()
-
-    for s in capa.features.extractors.strings.extract_ascii_strings(b):
-        yield String(s.s), s.offset
-
-    for s in capa.features.extractors.strings.extract_unicode_strings(b):
-        yield String(s.s), s.offset
+    for va in sorted(vw.getFunctions()):
+        if viv_utils.flirt.is_library_function(vw, va):
+            name = viv_utils.get_function_name(vw, va)
+            yield FunctionName(name), va
 
 
-def extract_features(vw, file_path):
+def extract_file_format(buf, **kwargs):
+    yield from capa.features.extractors.common.extract_format(buf)
+
+
+def extract_features(vw, buf: bytes):
     """
     extract file features from given workspace
 
     args:
       vw (vivisect.VivWorkspace): the vivisect workspace
-      file_path: path to the input file
+      buf: the raw input file bytes
 
     yields:
       Tuple[Feature, VA]: a feature and its location.
     """
 
     for file_handler in FILE_HANDLERS:
-        for feature, va in file_handler(vw, file_path):
+        for feature, va in file_handler(vw=vw, buf=buf):  # type: ignore
             yield feature, va
 
 
@@ -103,4 +108,6 @@ FILE_HANDLERS = (
     extract_file_import_names,
     extract_file_section_names,
     extract_file_strings,
+    extract_file_function_names,
+    extract_file_format,
 )

capa/features/extractors/viv/function.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,9 +6,10 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
+import envi
 import vivisect.const
 
-from capa.features import Characteristic
+from capa.features.common import Characteristic
 from capa.features.extractors import loops
 
 
@@ -41,9 +42,9 @@ def extract_function_loop(f):
             for bva, bflags in bb.instructions[-1].getBranches():
                 # vivisect does not set branch flags for non-conditional jmp so add explicit check
                 if (
-                    bflags & vivisect.envi.BR_COND
-                    or bflags & vivisect.envi.BR_FALL
-                    or bflags & vivisect.envi.BR_TABLE
+                    bflags & envi.BR_COND
+                    or bflags & envi.BR_FALL
+                    or bflags & envi.BR_TABLE
                     or bb.instructions[-1].mnem == "jmp"
                 ):
                     edges.append((bb.va, bva))
@@ -60,7 +61,7 @@ def extract_features(f):
       f (viv_utils.Function): the function from which to extract features
 
     yields:
-      Feature, set[VA]: the features and their location found in this function.
+      Tuple[Feature, int]: the features and their location found in this function.
     """
     for func_handler in FUNCTION_HANDLERS:
         for feature, va in func_handler(f):

capa/features/extractors/viv/global_.py

@@ -0,0 +1,24 @@
+import logging
+
+import envi.archs.i386
+import envi.archs.amd64
+
+from capa.features.common import ARCH_I386, ARCH_AMD64, Arch
+
+logger = logging.getLogger(__name__)
+
+
+def extract_arch(vw):
+    if isinstance(vw.arch, envi.archs.amd64.Amd64Module):
+        yield Arch(ARCH_AMD64), 0x0
+
+    elif isinstance(vw.arch, envi.archs.i386.i386Module):
+        yield Arch(ARCH_I386), 0x0
+
+    else:
+        # we likely end up here:
+        #  1. handling a new architecture (e.g. aarch64)
+        #
+        # for (1), this logic will need to be updated as the format is implemented.
+        logger.debug("unsupported architecture: %s", vw.arch.__class__.__name__)
+        return

capa/features/extractors/viv/helpers.py

@@ -1,14 +1,17 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
+from typing import Optional
+
+from vivisect import VivWorkspace
 from vivisect.const import XR_TO, REF_CODE
 
 
-def get_coderef_from(vw, va):
+def get_coderef_from(vw: VivWorkspace, va: int) -> Optional[int]:
     """
     return first code `tova` whose origin is the specified va
     return None if no code reference is found

capa/features/extractors/viv/indirect_calls.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -7,11 +7,16 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 import collections
+from typing import TYPE_CHECKING, Set, List, Deque, Tuple, Union, Optional
 
 import envi
 import vivisect.const
 import envi.archs.i386.disasm
 import envi.archs.amd64.disasm
+from vivisect import VivWorkspace
+
+if TYPE_CHECKING:
+    from capa.features.extractors.viv.extractor import InstructionHandle
 
 # pull out consts for lookup performance
 i386RegOper = envi.archs.i386.disasm.i386RegOper
@@ -26,7 +31,7 @@ FAR_BRANCH_MASK = envi.BR_PROC | envi.BR_DEREF | envi.BR_ARCH
 DESTRUCTIVE_MNEMONICS = ("mov", "lea", "pop", "xor")
 
 
-def get_previous_instructions(vw, va):
+def get_previous_instructions(vw: VivWorkspace, va: int) -> List[int]:
     """
     collect the instructions that flow to the given address, local to the current function.
 
@@ -43,12 +48,14 @@ def get_previous_instructions(vw, va):
     # ensure that it fallsthrough to this one.
     loc = vw.getPrevLocation(va, adjacent=True)
     if loc is not None:
-        # from vivisect.const:
-        # location: (L_VA, L_SIZE, L_LTYPE, L_TINFO)
-        (pva, _, ptype, pinfo) = vw.getPrevLocation(va, adjacent=True)
+        ploc = vw.getPrevLocation(va, adjacent=True)
+        if ploc is not None:
+            # from vivisect.const:
+            # location: (L_VA, L_SIZE, L_LTYPE, L_TINFO)
+            (pva, _, ptype, pinfo) = ploc
 
-        if ptype == LOC_OP and not (pinfo & IF_NOFALL):
-            ret.append(pva)
+            if ptype == LOC_OP and not (pinfo & IF_NOFALL):
+                ret.append(pva)
 
     # find any code refs, e.g. jmp, to this location.
     # ignore any calls.
@@ -67,7 +74,7 @@ class NotFoundError(Exception):
     pass
 
 
-def find_definition(vw, va, reg):
+def find_definition(vw: VivWorkspace, va: int, reg: int) -> Tuple[int, Union[int, None]]:
     """
     scan backwards from the given address looking for assignments to the given register.
     if a constant, return that value.
@@ -83,8 +90,8 @@ def find_definition(vw, va, reg):
     raises:
       NotFoundError: when the definition cannot be found.
     """
-    q = collections.deque()
-    seen = set([])
+    q = collections.deque()  # type: Deque[int]
+    seen = set([])  # type: Set[int]
 
     q.extend(get_previous_instructions(vw, va))
     while q:
@@ -128,14 +135,16 @@ def find_definition(vw, va, reg):
     raise NotFoundError()
 
 
-def is_indirect_call(vw, va, insn=None):
+def is_indirect_call(vw: VivWorkspace, va: int, insn: Optional["InstructionHandle"] = None) -> bool:
     if insn is None:
         insn = vw.parseOpcode(va)
 
     return insn.mnem in ("call", "jmp") and isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegOper)
 
 
-def resolve_indirect_call(vw, va, insn=None):
+def resolve_indirect_call(
+    vw: VivWorkspace, va: int, insn: Optional["InstructionHandle"] = None
+) -> Tuple[int, Optional[int]]:
     """
     inspect the given indirect call instruction and attempt to resolve the target address.
 

capa/features/extractors/viv/insn.py

@@ -1,26 +1,32 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
-
+import envi
+import envi.exc
+import viv_utils
 import envi.memory
+import viv_utils.flirt
+import envi.archs.i386.regs
+import envi.archs.amd64.regs
 import envi.archs.i386.disasm
+import envi.archs.amd64.disasm
 
 import capa.features.extractors.helpers
 import capa.features.extractors.viv.helpers
-from capa.features import (
-    ARCH_X32,
-    ARCH_X64,
+from capa.features.insn import API, Number, Offset, Mnemonic
+from capa.features.common import (
+    BITNESS_X32,
+    BITNESS_X64,
     MAX_BYTES_FEATURE_SIZE,
     THUNK_CHAIN_DEPTH_DELTA,
     Bytes,
     String,
     Characteristic,
 )
-from capa.features.insn import API, Number, Offset, Mnemonic
 from capa.features.extractors.viv.indirect_calls import NotFoundError, resolve_indirect_call
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
@@ -28,12 +34,12 @@ from capa.features.extractors.viv.indirect_calls import NotFoundError, resolve_i
 SECURITY_COOKIE_BYTES_DELTA = 0x40
 
 
-def get_arch(vw):
-    arch = vw.getMeta("Architecture")
-    if arch == "i386":
-        return ARCH_X32
-    elif arch == "amd64":
-        return ARCH_X64
+def get_bitness(vw):
+    bitness = vw.getMeta("Architecture")
+    if bitness == "i386":
+        return BITNESS_X32
+    elif bitness == "amd64":
+        return BITNESS_X64
 
 
 def interface_extract_instruction_XXX(f, bb, insn):
@@ -74,7 +80,6 @@ def extract_insn_api_features(f, bb, insn):
     # example:
     #
     #    call dword [0x00473038]
-
     if insn.mnem not in ("call", "jmp"):
         return
 
@@ -96,7 +101,7 @@ def extract_insn_api_features(f, bb, insn):
     # call via thunk on x86,
     # see 9324d1a8ae37a36ae560c37448c9705a at 0x407985
     #
-    # this is also how calls to internal functions may be decoded on x64.
+    # this is also how calls to internal functions may be decoded on x32 and x64.
     # see Lab21-01.exe_:0x140001178
     #
     # follow chained thunks, e.g. in 82bf6347acf15e5d883715dc289d8a2b at 0x14005E0FF in
@@ -111,12 +116,21 @@ def extract_insn_api_features(f, bb, insn):
         if not target:
             return
 
+        if viv_utils.flirt.is_library_function(f.vw, target):
+            name = viv_utils.get_function_name(f.vw, target)
+            yield API(name), insn.va
+            return
+
         for _ in range(THUNK_CHAIN_DEPTH_DELTA):
             if target in imports:
                 dll, symbol = imports[target]
                 for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
                     yield API(name), insn.va
 
+            # if jump leads to an ENDBRANCH instruction, skip it
+            if f.vw.getByteDef(target)[1].startswith(b"\xf3\x0f\x1e"):
+                target += 4
+
             target = capa.features.extractors.viv.helpers.get_coderef_from(f.vw, target)
             if not target:
                 return
@@ -171,7 +185,7 @@ def extract_insn_number_features(f, bb, insn):
             # assume its not also a constant.
             continue
 
-        if insn.mnem == "add" and insn.opers[0].isReg() and insn.opers[0].reg == envi.archs.i386.disasm.REG_ESP:
+        if insn.mnem == "add" and insn.opers[0].isReg() and insn.opers[0].reg == envi.archs.i386.regs.REG_ESP:
             # skip things like:
             #
             #    .text:00401140                 call    sub_407E2B
@@ -179,7 +193,7 @@ def extract_insn_number_features(f, bb, insn):
             return
 
         yield Number(v), insn.va
-        yield Number(v, arch=get_arch(f.vw)), insn.va
+        yield Number(v, bitness=get_bitness(f.vw)), insn.va
 
 
 def derefs(vw, p):
@@ -214,7 +228,7 @@ def derefs(vw, p):
         p = next
 
 
-def read_memory(vw, va, size):
+def read_memory(vw, va: int, size: int) -> bytes:
     # as documented in #176, vivisect will not readMemory() when the section is not marked readable.
     #
     # but here, we don't care about permissions.
@@ -227,10 +241,10 @@ def read_memory(vw, va, size):
             mva, msize, mperms, mfname = mmap
             offset = va - mva
             return mbytes[offset : offset + size]
-    raise envi.SegmentationViolation(va)
+    raise envi.exc.SegmentationViolation(va)
 
 
-def read_bytes(vw, va):
+def read_bytes(vw, va: int) -> bytes:
     """
     read up to MAX_BYTES_FEATURE_SIZE from the given address.
 
@@ -239,7 +253,7 @@ def read_bytes(vw, va):
     """
     segm = vw.getSegment(va)
     if not segm:
-        raise envi.SegmentationViolation(va)
+        raise envi.exc.SegmentationViolation(va)
 
     segm_end = segm[0] + segm[1]
     try:
@@ -248,7 +262,7 @@ def read_bytes(vw, va):
             return read_memory(vw, va, segm_end - va)
         else:
             return read_memory(vw, va, MAX_BYTES_FEATURE_SIZE)
-    except envi.SegmentationViolation:
+    except envi.exc.SegmentationViolation:
         raise
 
 
@@ -280,7 +294,7 @@ def extract_insn_bytes_features(f, bb, insn):
         for v in derefs(f.vw, v):
             try:
                 buf = read_bytes(f.vw, v)
-            except envi.SegmentationViolation:
+            except envi.exc.SegmentationViolation:
                 continue
 
             if capa.features.extractors.helpers.all_zeros(buf):
@@ -289,10 +303,10 @@ def extract_insn_bytes_features(f, bb, insn):
             yield Bytes(buf), insn.va
 
 
-def read_string(vw, offset):
+def read_string(vw, offset: int) -> str:
     try:
         alen = vw.detectString(offset)
-    except envi.SegmentationViolation:
+    except envi.exc.SegmentationViolation:
         pass
     else:
         if alen > 0:
@@ -300,7 +314,7 @@ def read_string(vw, offset):
 
     try:
         ulen = vw.detectUnicode(offset)
-    except envi.SegmentationViolation:
+    except envi.exc.SegmentationViolation:
         pass
     except IndexError:
         # potential vivisect bug detecting Unicode at segment end
@@ -361,21 +375,21 @@ def extract_insn_offset_features(f, bb, insn):
         #       reg   ^
         #             disp
         if isinstance(oper, envi.archs.i386.disasm.i386RegMemOper):
-            if oper.reg == envi.archs.i386.disasm.REG_ESP:
+            if oper.reg == envi.archs.i386.regs.REG_ESP:
                 continue
 
-            if oper.reg == envi.archs.i386.disasm.REG_EBP:
+            if oper.reg == envi.archs.i386.regs.REG_EBP:
                 continue
 
             # TODO: do x64 support for real.
-            if oper.reg == envi.archs.amd64.disasm.REG_RBP:
+            if oper.reg == envi.archs.amd64.regs.REG_RBP:
                 continue
 
             # viv already decodes offsets as signed
             v = oper.disp
 
             yield Offset(v), insn.va
-            yield Offset(v, arch=get_arch(f.vw)), insn.va
+            yield Offset(v, bitness=get_bitness(f.vw)), insn.va
 
         # like: [esi + ecx + 16384]
         #        reg   ^     ^
@@ -386,21 +400,21 @@ def extract_insn_offset_features(f, bb, insn):
             v = oper.disp
 
             yield Offset(v), insn.va
-            yield Offset(v, arch=get_arch(f.vw)), insn.va
+            yield Offset(v, bitness=get_bitness(f.vw)), insn.va
 
 
-def is_security_cookie(f, bb, insn):
+def is_security_cookie(f, bb, insn) -> bool:
     """
     check if an instruction is related to security cookie checks
     """
     # security cookie check should use SP or BP
     oper = insn.opers[1]
     if oper.isReg() and oper.reg not in [
-        envi.archs.i386.disasm.REG_ESP,
-        envi.archs.i386.disasm.REG_EBP,
+        envi.archs.i386.regs.REG_ESP,
+        envi.archs.i386.regs.REG_EBP,
         # TODO: do x64 support for real.
-        envi.archs.amd64.disasm.REG_RBP,
-        envi.archs.amd64.disasm.REG_RSP,
+        envi.archs.amd64.regs.REG_RBP,
+        envi.archs.amd64.regs.REG_RSP,
     ]:
         return False
 
@@ -476,7 +490,7 @@ def extract_insn_peb_access_characteristic_features(f, bb, insn):
 
 
 def extract_insn_segment_access_features(f, bb, insn):
-    """ parse the instruction for access to fs or gs """
+    """parse the instruction for access to fs or gs"""
     prefix = insn.getPrefixName()
 
     if prefix == "fs":
@@ -486,7 +500,7 @@ def extract_insn_segment_access_features(f, bb, insn):
         yield Characteristic("gs access"), insn.va
 
 
-def get_section(vw, va):
+def get_section(vw, va: int):
     for start, length, _, __ in vw.getMemoryMaps():
         if start <= va < start + length:
             return start
@@ -597,7 +611,7 @@ def extract_features(f, bb, insn):
       insn (vivisect...Instruction): the instruction to process.
 
     yields:
-      Feature, set[VA]: the features and their location found in this insn.
+      Tuple[Feature, int]: the features and their location found in this insn.
     """
     for insn_handler in INSTRUCTION_HANDLERS:
         for feature, va in insn_handler(f, bb, insn):

capa/features/file.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,22 +6,33 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-from capa.features import Feature
+from capa.features.common import Feature
 
 
 class Export(Feature):
-    def __init__(self, value, description=None):
+    def __init__(self, value: str, description=None):
         # value is export name
         super(Export, self).__init__(value, description=description)
 
 
 class Import(Feature):
-    def __init__(self, value, description=None):
+    def __init__(self, value: str, description=None):
         # value is import name
         super(Import, self).__init__(value, description=description)
 
 
 class Section(Feature):
-    def __init__(self, value, description=None):
+    def __init__(self, value: str, description=None):
         # value is section name
         super(Section, self).__init__(value, description=description)
+
+
+class FunctionName(Feature):
+    """recognized name for statically linked function"""
+
+    def __init__(self, name: str, description=None):
+        # value is function name
+        super(FunctionName, self).__init__(name, description=description)
+        # override the name property set by `capa.features.Feature`
+        # that would be `functionname` (note missing dash)
+        self.name = "function-name"

capa/features/freeze.py

@@ -8,17 +8,16 @@ json format:
       'base address': int(base address),
       'functions': {
         int(function va): {
-          'basic blocks': {
-            int(basic block va): {
-              'instructions': [instruction va, ...]
-            },
-            ...
-          },
+          int(basic block va): [int(instruction va), ...]
           ...
         },
         ...
       },
       'scopes': {
+        'global': [
+          (str(name), [any(arg), ...], int(va), ()),
+          ...
+        },
         'file': [
           (str(name), [any(arg), ...], int(va), ()),
           ...
@@ -41,7 +40,7 @@ json format:
       }
     }
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -53,11 +52,11 @@ import json
 import zlib
 import logging
 
-import capa.features
 import capa.features.file
 import capa.features.insn
+import capa.features.common
 import capa.features.basicblock
-import capa.features.extractors
+import capa.features.extractors.base_extractor
 from capa.helpers import hex
 
 logger = logging.getLogger(__name__)
@@ -67,7 +66,7 @@ def serialize_feature(feature):
     return feature.freeze_serialize()
 
 
-KNOWN_FEATURES = {F.__name__: F for F in capa.features.Feature.__subclasses__()}
+KNOWN_FEATURES = {F.__name__: F for F in capa.features.common.Feature.__subclasses__()}
 
 
 def deserialize_feature(doc):
@@ -80,7 +79,7 @@ def dumps(extractor):
     serialize the given extractor to a string
 
     args:
-      extractor: capa.features.extractor.FeatureExtractor:
+      extractor: capa.features.extractors.base_extractor.FeatureExtractor:
 
     returns:
       str: the serialized features.
@@ -90,12 +89,15 @@ def dumps(extractor):
         "base address": extractor.get_base_address(),
         "functions": {},
         "scopes": {
+            "global": [],
             "file": [],
             "function": [],
             "basic block": [],
             "instruction": [],
         },
     }
+    for feature, va in extractor.extract_global_features():
+        ret["scopes"]["global"].append(serialize_feature(feature) + (hex(va), ()))
 
     for feature, va in extractor.extract_file_features():
         ret["scopes"]["file"].append(serialize_feature(feature) + (hex(va), ()))
@@ -122,7 +124,7 @@ def dumps(extractor):
                 )
 
             for insnva, insn in sorted(
-                [(insn.__int__(), insn) for insn in extractor.get_instructions(f, bb)], key=lambda p: p[0]
+                [(int(insn), insn) for insn in extractor.get_instructions(f, bb)], key=lambda p: p[0]
             ):
                 ret["functions"][hex(f)][hex(bb)].append(hex(insnva))
 
@@ -150,6 +152,7 @@ def loads(s):
 
     features = {
         "base address": doc.get("base address"),
+        "global features": [],
         "file features": [],
         "functions": {},
     }
@@ -179,6 +182,12 @@ def loads(s):
     #     ('MatchedRule', ('foo', ), '0x401000', ('0x401000', ))
     #      ^^^^^^^^^^^^^  ^^^^^^^^^  ^^^^^^^^^^  ^^^^^^^^^^^^^^
     #      feature name   args       addr         func/bb/insn
+    for feature in doc.get("scopes", {}).get("global", []):
+        va, loc = feature[2:]
+        va = int(va, 0x10)
+        feature = deserialize_feature(feature[:2])
+        features["global features"].append((va, feature))
+
     for feature in doc.get("scopes", {}).get("file", []):
         va, loc = feature[2:]
         va = int(va, 0x10)
@@ -217,7 +226,7 @@ def loads(s):
         feature = deserialize_feature(feature[:2])
         features["functions"][loc[0]]["basic blocks"][loc[1]]["instructions"][loc[2]]["features"].append((va, feature))
 
-    return capa.features.extractors.NullFeatureExtractor(features)
+    return capa.features.extractors.base_extractor.NullFeatureExtractor(features)
 
 
 MAGIC = "capa0000".encode("ascii")
@@ -228,7 +237,7 @@ def dump(extractor):
     return MAGIC + zlib.compress(dumps(extractor).encode("utf-8"))
 
 
-def is_freeze(buf):
+def is_freeze(buf: bytes) -> bool:
     return buf[: len(MAGIC)] == MAGIC
 
 
@@ -248,45 +257,16 @@ def main(argv=None):
     if argv is None:
         argv = sys.argv[1:]
 
-    formats = [
-        ("auto", "(default) detect file type automatically"),
-        ("pe", "Windows PE file"),
-        ("sc32", "32-bit shellcode"),
-        ("sc64", "64-bit shellcode"),
-    ]
-    format_help = ", ".join(["%s: %s" % (f[0], f[1]) for f in formats])
-
     parser = argparse.ArgumentParser(description="save capa features to a file")
-    parser.add_argument("sample", type=str, help="Path to sample to analyze")
+    capa.main.install_common_args(parser, {"sample", "format", "backend", "signatures"})
     parser.add_argument("output", type=str, help="Path to output file")
-    parser.add_argument("-v", "--verbose", action="store_true", help="Enable verbose output")
-    parser.add_argument("-q", "--quiet", action="store_true", help="Disable all output but errors")
-    parser.add_argument(
-        "-f", "--format", choices=[f[0] for f in formats], default="auto", help="Select sample format, %s" % format_help
-    )
-    if sys.version_info >= (3, 0):
-        parser.add_argument(
-            "-b",
-            "--backend",
-            type=str,
-            help="select the backend to use",
-            choices=(capa.main.BACKEND_VIV, capa.main.BACKEND_SMDA),
-            default=capa.main.BACKEND_VIV,
-        )
     args = parser.parse_args(args=argv)
+    capa.main.handle_common_args(args)
 
-    if args.quiet:
-        logging.basicConfig(level=logging.ERROR)
-        logging.getLogger().setLevel(logging.ERROR)
-    elif args.verbose:
-        logging.basicConfig(level=logging.DEBUG)
-        logging.getLogger().setLevel(logging.DEBUG)
-    else:
-        logging.basicConfig(level=logging.INFO)
-        logging.getLogger().setLevel(logging.INFO)
+    sigpaths = capa.main.get_signatures(args.signatures)
+
+    extractor = capa.main.get_extractor(args.sample, args.format, args.backend, sigpaths, False)
 
-    backend = args.backend if sys.version_info > (3, 0) else capa.main.BACKEND_VIV
-    extractor = capa.main.get_extractor(args.sample, args.format, backend)
     with open(args.output, "wb") as f:
         f.write(dump(extractor))
 

capa/features/insn.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,11 +6,12 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-from capa.features import Feature
+import capa.render.utils
+from capa.features.common import Feature
 
 
 class API(Feature):
-    def __init__(self, name, description=None):
+    def __init__(self, name: str, description=None):
         # Downcase library name if given
         if "." in name:
             modname, _, impname = name.rpartition(".")
@@ -20,21 +21,21 @@ class API(Feature):
 
 
 class Number(Feature):
-    def __init__(self, value, arch=None, description=None):
-        super(Number, self).__init__(value, arch=arch, description=description)
+    def __init__(self, value: int, bitness=None, description=None):
+        super(Number, self).__init__(value, bitness=bitness, description=description)
 
     def get_value_str(self):
-        return "0x%X" % self.value
+        return capa.render.utils.hex(self.value)
 
 
 class Offset(Feature):
-    def __init__(self, value, arch=None, description=None):
-        super(Offset, self).__init__(value, arch=arch, description=description)
+    def __init__(self, value: int, bitness=None, description=None):
+        super(Offset, self).__init__(value, bitness=bitness, description=description)
 
     def get_value_str(self):
-        return "0x%X" % self.value
+        return capa.render.utils.hex(self.value)
 
 
 class Mnemonic(Feature):
-    def __init__(self, value, description=None):
+    def __init__(self, value: str, description=None):
         super(Mnemonic, self).__init__(value, description=description)

capa/helpers.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -7,30 +7,31 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 import os
+from typing import NoReturn
 
 _hex = hex
 
 
 def hex(i):
-    # under py2.7, long integers get formatted with a trailing `L`
-    # and this is not pretty. so strip it out.
-    return _hex(oint(i)).rstrip("L")
+    return _hex(int(i))
 
 
-def oint(i):
-    # there seems to be some trouble with using `int(viv_utils.Function)`
-    # with the black magic we do with binding the `__int__()` routine.
-    # i haven't had a chance to debug this yet (and i have no hotel wifi).
-    # so in the meantime, detect this, and call the method directly.
-    try:
-        return int(i)
-    except TypeError:
-        return i.__int__()
-
-
-def get_file_taste(sample_path):
+def get_file_taste(sample_path: str) -> bytes:
     if not os.path.exists(sample_path):
         raise IOError("sample path %s does not exist or cannot be accessed" % sample_path)
     with open(sample_path, "rb") as f:
         taste = f.read(8)
     return taste
+
+
+def is_runtime_ida():
+    try:
+        import idc
+    except ImportError:
+        return False
+    else:
+        return True
+
+
+def assert_never(value: NoReturn) -> NoReturn:
+    assert False, f"Unhandled value: {value} ({type(value).__name__})"

capa/ida/helpers.py

@@ -0,0 +1,163 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+import logging
+import datetime
+
+import idc
+import idaapi
+import idautils
+import ida_bytes
+import ida_loader
+
+import capa
+import capa.version
+import capa.features.common
+
+logger = logging.getLogger("capa")
+
+# file type as returned by idainfo.file_type
+SUPPORTED_FILE_TYPES = (
+    idaapi.f_PE,
+    idaapi.f_ELF,
+    idaapi.f_BIN,
+    # idaapi.f_MACHO,
+)
+
+# arch type as returned by idainfo.procname
+SUPPORTED_ARCH_TYPES = ("metapc",)
+
+
+def inform_user_ida_ui(message):
+    idaapi.info("%s. Please refer to IDA Output window for more information." % message)
+
+
+def is_supported_ida_version():
+    version = float(idaapi.get_kernel_version())
+    if version < 7.4 or version >= 8:
+        warning_msg = "This plugin does not support your IDA Pro version"
+        logger.warning(warning_msg)
+        logger.warning("Your IDA Pro version is: %s. Supported versions are: IDA >= 7.4 and IDA < 8.0." % version)
+        return False
+    return True
+
+
+def is_supported_file_type():
+    file_info = idaapi.get_inf_structure()
+    if file_info.filetype not in SUPPORTED_FILE_TYPES:
+        logger.error("-" * 80)
+        logger.error(" Input file does not appear to be a supported file type.")
+        logger.error(" ")
+        logger.error(
+            " capa currently only supports analyzing PE, ELF, or binary files containing x86 (32- and 64-bit) shellcode."
+        )
+        logger.error(" If you don't know the input file type, you can try using the `file` utility to guess it.")
+        logger.error("-" * 80)
+        return False
+    return True
+
+
+def is_supported_arch_type():
+    file_info = idaapi.get_inf_structure()
+    if file_info.procname not in SUPPORTED_ARCH_TYPES or not any((file_info.is_32bit(), file_info.is_64bit())):
+        logger.error("-" * 80)
+        logger.error(" Input file does not appear to target a supported architecture.")
+        logger.error(" ")
+        logger.error(" capa currently only supports analyzing x86 (32- and 64-bit).")
+        logger.error("-" * 80)
+        return False
+    return True
+
+
+def get_disasm_line(va):
+    """ """
+    return idc.generate_disasm_line(va, idc.GENDSM_FORCE_CODE)
+
+
+def is_func_start(ea):
+    """check if function stat exists at virtual address"""
+    f = idaapi.get_func(ea)
+    return f and f.start_ea == ea
+
+
+def get_func_start_ea(ea):
+    """ """
+    f = idaapi.get_func(ea)
+    return f if f is None else f.start_ea
+
+
+def get_file_md5():
+    """ """
+    md5 = idautils.GetInputFileMD5()
+    if not isinstance(md5, str):
+        md5 = capa.features.common.bytes_to_str(md5)
+    return md5
+
+
+def get_file_sha256():
+    """ """
+    sha256 = idaapi.retrieve_input_file_sha256()
+    if not isinstance(sha256, str):
+        sha256 = capa.features.common.bytes_to_str(sha256)
+    return sha256
+
+
+def collect_metadata():
+    """ """
+    md5 = get_file_md5()
+    sha256 = get_file_sha256()
+
+    return {
+        "timestamp": datetime.datetime.now().isoformat(),
+        # "argv" is not relevant here
+        "sample": {
+            "md5": md5,
+            "sha1": "",  # not easily accessible
+            "sha256": sha256,
+            "path": idaapi.get_input_file_path(),
+        },
+        "analysis": {
+            "format": idaapi.get_file_type_name(),
+            "extractor": "ida",
+            "base_address": idaapi.get_imagebase(),
+            "layout": {
+                # this is updated after capabilities have been collected.
+                # will look like:
+                #
+                # "functions": { 0x401000: { "matched_basic_blocks": [ 0x401000, 0x401005, ... ] }, ... }
+            },
+        },
+        "version": capa.version.__version__,
+    }
+
+
+class IDAIO:
+    """
+    An object that acts as a file-like object,
+    using bytes from the current IDB workspace.
+    """
+
+    def __init__(self):
+        super(IDAIO, self).__init__()
+        self.offset = 0
+
+    def seek(self, offset, whence=0):
+        assert whence == 0
+        self.offset = offset
+
+    def read(self, size):
+        ea = ida_loader.get_fileregion_ea(self.offset)
+        if ea == idc.BADADDR:
+            # best guess, such as if file is mapped at address 0x0.
+            ea = self.offset
+
+        logger.debug("reading 0x%x bytes at 0x%x (ea: 0x%x)", size, self.offset, ea)
+        return ida_bytes.get_bytes(ea, size)
+
+    def close(self):
+        return

capa/ida/helpers/__init__.py

@@ -1,121 +0,0 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-
-import logging
-import datetime
-
-import idc
-import six
-import idaapi
-import idautils
-
-import capa
-
-logger = logging.getLogger("capa")
-
-SUPPORTED_IDA_VERSIONS = [
-    "7.1",
-    "7.2",
-    "7.3",
-    "7.4",
-    "7.5",
-]
-
-# file type names as returned by idaapi.get_file_type_name()
-SUPPORTED_FILE_TYPES = [
-    "Portable executable for 80386 (PE)",
-    "Portable executable for AMD64 (PE)",
-    "Binary file",  # x86/AMD64 shellcode support
-]
-
-
-def inform_user_ida_ui(message):
-    idaapi.info("%s. Please refer to IDA Output window for more information." % message)
-
-
-def is_supported_ida_version():
-    version = idaapi.get_kernel_version()
-    if version not in SUPPORTED_IDA_VERSIONS:
-        warning_msg = "This plugin does not support your IDA Pro version"
-        logger.warning(warning_msg)
-        logger.warning(
-            "Your IDA Pro version is: %s. Supported versions are: %s." % (version, ", ".join(SUPPORTED_IDA_VERSIONS))
-        )
-        return False
-    return True
-
-
-def is_supported_file_type():
-    file_type = idaapi.get_file_type_name()
-    if file_type not in SUPPORTED_FILE_TYPES:
-        logger.error("-" * 80)
-        logger.error(" Input file does not appear to be a PE file.")
-        logger.error(" ")
-        logger.error(
-            " capa currently only supports analyzing PE files (or binary files containing x86/AMD64 shellcode) with IDA."
-        )
-        logger.error(" If you don't know the input file type, you can try using the `file` utility to guess it.")
-        logger.error("-" * 80)
-        return False
-    return True
-
-
-def get_disasm_line(va):
-    """ """
-    return idc.generate_disasm_line(va, idc.GENDSM_FORCE_CODE)
-
-
-def is_func_start(ea):
-    """ check if function stat exists at virtual address """
-    f = idaapi.get_func(ea)
-    return f and f.start_ea == ea
-
-
-def get_func_start_ea(ea):
-    """ """
-    f = idaapi.get_func(ea)
-    return f if f is None else f.start_ea
-
-
-def get_file_md5():
-    """ """
-    md5 = idautils.GetInputFileMD5()
-    if not isinstance(md5, six.string_types):
-        md5 = capa.features.bytes_to_str(md5)
-    return md5
-
-
-def get_file_sha256():
-    """ """
-    sha256 = idaapi.retrieve_input_file_sha256()
-    if not isinstance(sha256, six.string_types):
-        sha256 = capa.features.bytes_to_str(sha256)
-    return sha256
-
-
-def collect_metadata():
-    """ """
-    md5 = get_file_md5()
-    sha256 = get_file_sha256()
-
-    return {
-        "timestamp": datetime.datetime.now().isoformat(),
-        # "argv" is not relevant here
-        "sample": {
-            "md5": md5,
-            "sha1": "",  # not easily accessible
-            "sha256": sha256,
-            "path": idaapi.get_input_file_path(),
-        },
-        "analysis": {
-            "format": idaapi.get_file_type_name(),
-            "extractor": "ida",
-            "base_address": idaapi.get_imagebase(),
-        },
-        "version": capa.version.__version__,
-    }

capa/ida/plugin/README.md

@@ -34,18 +34,50 @@ For more information on the FLARE team's open-source framework, capa, check out
 
 ### Requirements
 
-capa explorer supports the following IDA setups:
+capa explorer supports Python versions >= 3.6.x and the following IDA Pro versions:
 
-* IDA Pro 7.4+ with Python 2.7 or Python 3.
+* IDA 7.4
+* IDA 7.5
+* IDA 7.6 (caveat below)
+* IDA 7.7
+
+capa explorer is however limited to the Python versions supported by your IDA installation (which may not include all Python versions >= 3.6.x). Based on our testing the following matrix shows the Python versions supported
+by each supported IDA version:
+
+| | IDA 7.4 | IDA 7.5 | IDA 7.6 |
+| --- | --- | --- | --- |
+| Python 3.6.x | Yes | Yes | Yes |
+| Python 3.7.x | Yes | Yes | Yes |
+| Python 3.8.x | Partial (see below) | Yes | Yes |
+| Python 3.9.x | No | Partial (see below) | Yes |
+
+To use capa explorer with IDA 7.4 and Python 3.8.x you must follow the instructions provided by hex-rays [here](https://hex-rays.com/blog/ida-7-4-and-python-3-8/).
+
+To use capa explorer with IDA 7.5 and Python 3.9.x you must follow the instructions provided by hex-rays [here](https://hex-rays.com/blog/python-3-9-support-for-ida-7-5/).
+
+If you encounter issues with your specific setup, please open a new [Issue](https://github.com/mandiant/capa/issues).
+
+#### IDA 7.6 caveat: IDA 7.6sp1 or patch required
+
+As described [here](https://www.hex-rays.com/blog/ida-7-6-empty-qtreeview-qtreewidget/):
+
+> A rather nasty issue evaded our testing and found its way into IDA 7.6: using the PyQt5 modules that are shipped with IDA, QTreeView (or QTreeWidget) instances will always fail to display contents.
+
+Therefore, in order to use capa under IDA 7.6 you need the [Service Pack 1 for IDA 7.6](https://www.hex-rays.com/products/ida/news/7_6sp1). Alternatively, you can download and install the fix corresponding to your IDA installation, replacing the original QtWidgets DLL with the one contained in the .zip file (links to Hex-Rays):
+
+
+  - Windows: [pyqt5_qtwidgets_win](https://www.hex-rays.com/wp-content/uploads/2021/04/pyqt5_qtwidgets_win.zip)
+  - Linux: [pyqt5_qtwidgets_linux](https://www.hex-rays.com/wp-content/uploads/2021/04/pyqt5_qtwidgets_linux.zip)
+  - MacOS (Intel): [pyqt5_qtwidgets_mac_x64](https://www.hex-rays.com/wp-content/uploads/2021/04/pyqt5_qtwidgets_mac_x64.zip)
+  - MacOS (AppleSilicon): [pyqt5_qtwidgets_mac_arm](https://www.hex-rays.com/wp-content/uploads/2021/04/pyqt5_qtwidgets_mac_arm.zip)
 
-If you encounter issues with your specific setup, please open a new [Issue](https://github.com/fireeye/capa/issues).
 
 ### Supported File Types
 
 capa explorer is limited to the file types supported by capa, which include:
 
-* Windows 32-bit and 64-bit PE files
-* Windows 32-bit and 64-bit shellcode
+* Windows x86 (32- and 64-bit) PE and ELF files
+* Windows x86 (32- and 64-bit) shellcode
 
 ### Installation
 
@@ -55,19 +87,20 @@ You can install capa explorer using the following steps:
     ```
     $ pip install flare-capa
     ```
-3. Download the [standard collection of capa rules](https://github.com/fireeye/capa-rules) (capa explorer needs capa rules to analyze a database)
-4. Copy [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory
+3. Download the [standard collection of capa rules](https://github.com/mandiant/capa-rules) (capa explorer needs capa rules to analyze a database)
+4. Copy [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory
 
 ### Usage
 
 1. Open IDA and analyze a supported file type (select the `Manual Load` and `Load Resources` options in IDA for best results)
 2. Open capa explorer in IDA by navigating to `Edit > Plugins > FLARE capa explorer` or using the keyboard shortcut `Alt+F5`
+   You can also use `ida_loader.load_and_run_plugin("capa_explorer", arg)`. `arg` is a bitflag for which setting the LSB enables automatic analysis. See `capa.ida.plugin.form.Options` for more details.
 3. Select the `Program Analysis` tab
 4. Click the `Analyze` button
 
 When running capa explorer for the first time you are prompted to select a file directory containing capa rules. The plugin conveniently
 remembers your selection for future runs; you can change this selection and other default settings by clicking `Settings`. We recommend 
-downloading and using the [standard collection of capa rules](https://github.com/fireeye/capa-rules) when getting started with the plugin.
+downloading and using the [standard collection of capa rules](https://github.com/mandiant/capa-rules) when getting started with the plugin.
 
 #### Tips for Program Analysis
 
@@ -93,15 +126,15 @@ downloading and using the [standard collection of capa rules](https://github.com
 ## Development
 
 capa explorer is packaged with capa so you will need to install capa locally for development. You can install capa locally by following the steps outlined in `Method 3: Inspecting the capa source code` of the [capa 
-installation guide](https://github.com/fireeye/capa/blob/master/doc/installation.md#method-3-inspecting-the-capa-source-code). Once installed, copy [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) 
+installation guide](https://github.com/mandiant/capa/blob/master/doc/installation.md#method-3-inspecting-the-capa-source-code). Once installed, copy [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) 
 to your plugins directory to install capa explorer in IDA.
 
 ### Components
 
 capa explorer consists of two main components:
 
-* An [feature extractor](https://github.com/fireeye/capa/tree/master/capa/features/extractors/ida) built on top of IDA's binary analysis engine
-  * This component uses IDAPython to extract [capa features](https://github.com/fireeye/capa-rules/blob/master/doc/format.md#extracted-features) from your IDBs such as strings, 
+* An [feature extractor](https://github.com/mandiant/capa/tree/master/capa/features/extractors/ida) built on top of IDA's binary analysis engine
+  * This component uses IDAPython to extract [capa features](https://github.com/mandiant/capa-rules/blob/master/doc/format.md#extracted-features) from your IDBs such as strings, 
 disassembly, and control flow; these extracted features are used by capa to find feature combinations that result in a rule match
-* An [interactive user interface](https://github.com/fireeye/capa/tree/master/capa/ida/plugin) for displaying and exploring capa rule matches
+* An [interactive user interface](https://github.com/mandiant/capa/tree/master/capa/ida/plugin) for displaying and exploring capa rule matches
   * This component integrates the feature extractor and capa, providing an interactive user interface to dissect rule matches found by capa using features extracted directly from your IDBs

capa/ida/plugin/__init__.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -11,7 +11,6 @@ import logging
 import idaapi
 import ida_kernwin
 
-from capa.ida.helpers import is_supported_file_type, is_supported_ida_version
 from capa.ida.plugin.form import CapaExplorerForm
 from capa.ida.plugin.icon import ICON
 
@@ -28,8 +27,8 @@ class CapaExplorerPlugin(idaapi.plugin_t):
     wanted_name = PLUGIN_NAME
     wanted_hotkey = "ALT-F5"
     comment = "IDA Pro plugin for the FLARE team's capa tool to identify capabilities in executable files."
-    website = "https://github.com/fireeye/capa"
-    help = "See https://github.com/fireeye/capa/blob/master/doc/usage.md"
+    website = "https://github.com/mandiant/capa"
+    help = "See https://github.com/mandiant/capa/blob/master/doc/usage.md"
     version = ""
     flags = 0
 
@@ -41,10 +40,14 @@ class CapaExplorerPlugin(idaapi.plugin_t):
         """called when IDA is loading the plugin"""
         logging.basicConfig(level=logging.INFO)
 
+        import capa.ida.helpers
+
         # do not load plugin if IDA version/file type not supported
-        if not is_supported_ida_version():
+        if not capa.ida.helpers.is_supported_ida_version():
             return idaapi.PLUGIN_SKIP
-        if not is_supported_file_type():
+        if not capa.ida.helpers.is_supported_file_type():
+            return idaapi.PLUGIN_SKIP
+        if not capa.ida.helpers.is_supported_arch_type():
             return idaapi.PLUGIN_SKIP
         return idaapi.PLUGIN_OK
 
@@ -53,8 +56,14 @@ class CapaExplorerPlugin(idaapi.plugin_t):
         pass
 
     def run(self, arg):
-        """called when IDA is running the plugin as a script"""
-        self.form = CapaExplorerForm(self.PLUGIN_NAME)
+        """
+        called when IDA is running the plugin as a script
+
+        args:
+          arg (int): bitflag. Setting LSB enables automatic analysis upon
+          loading. The other bits are currently undefined. See `form.Options`.
+        """
+        self.form = CapaExplorerForm(self.PLUGIN_NAME, arg)
         return True
 
 

capa/ida/plugin/capa_explorer.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/ida/plugin/form.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -20,9 +20,12 @@ from PyQt5 import QtGui, QtCore, QtWidgets
 
 import capa.main
 import capa.rules
+import capa.engine
 import capa.ida.helpers
-import capa.render.utils as rutils
-import capa.features.extractors.ida
+import capa.render.json
+import capa.features.common
+import capa.render.result_document
+import capa.features.extractors.ida.extractor
 from capa.ida.plugin.icon import QICON
 from capa.ida.plugin.view import (
     CapaExplorerQtreeView,
@@ -41,11 +44,16 @@ CAPA_SETTINGS_RULE_PATH = "rule_path"
 CAPA_SETTINGS_RULEGEN_AUTHOR = "rulegen_author"
 CAPA_SETTINGS_RULEGEN_SCOPE = "rulegen_scope"
 
+from enum import IntFlag
+
+
+class Options(IntFlag):
+    DEFAULT = 0
+    ANALYZE = 1  # Runs the analysis when starting the explorer
+
 
 def write_file(path, data):
     """ """
-    if os.path.exists(path) and 1 != idaapi.ask_yn(1, "The file already exists. Overwrite?"):
-        return
     with open(path, "wb") as save_file:
         save_file.write(data)
 
@@ -78,7 +86,7 @@ def find_func_features(f, extractor):
                 _bb_features[feature].add(ea)
                 func_features[feature].add(ea)
 
-        bb_features[capa.helpers.oint(bb)] = _bb_features
+        bb_features[int(bb)] = _bb_features
 
     return func_features, bb_features
 
@@ -97,10 +105,10 @@ def find_func_matches(f, ruleset, func_features, bb_features):
         for (name, res) in matches.items():
             bb_matches[name].extend(res)
             for (ea, _) in res:
-                func_features[capa.features.MatchedRule(name)].add(ea)
+                func_features[capa.features.common.MatchedRule(name)].add(ea)
 
     # find rule matches for function, function features include rule matches for basic blocks
-    _, matches = capa.engine.match(ruleset.function_rules, func_features, capa.helpers.oint(f))
+    _, matches = capa.engine.match(ruleset.function_rules, func_features, int(f))
     for (name, res) in matches.items():
         func_matches[name].extend(res)
 
@@ -155,7 +163,7 @@ class CapaExplorerProgressIndicator(QtCore.QObject):
         self.progress.emit("extracting features from %s" % text)
 
 
-class CapaExplorerFeatureExtractor(capa.features.extractors.ida.IdaFeatureExtractor):
+class CapaExplorerFeatureExtractor(capa.features.extractors.ida.extractor.IdaFeatureExtractor):
     """subclass the IdaFeatureExtractor
 
     track progress during feature extraction, also allow user to cancel feature extraction
@@ -227,7 +235,7 @@ class CapaSettingsInputDialog(QtWidgets.QDialog):
 class CapaExplorerForm(idaapi.PluginForm):
     """form element for plugin interface"""
 
-    def __init__(self, name):
+    def __init__(self, name, option=Options.DEFAULT):
         """initialize form elements"""
         super(CapaExplorerForm, self).__init__()
 
@@ -267,6 +275,7 @@ class CapaExplorerForm(idaapi.PluginForm):
         self.view_rulegen_editor = None
         self.view_rulegen_header_label = None
         self.view_rulegen_search = None
+        self.view_rulegen_limit_features_by_ea = None
         self.rulegen_current_function = None
         self.rulegen_bb_features_cache = {}
         self.rulegen_func_features_cache = {}
@@ -275,6 +284,9 @@ class CapaExplorerForm(idaapi.PluginForm):
 
         self.Show()
 
+        if (option & Options.ANALYZE) == Options.ANALYZE:
+            self.analyze_program()
+
     def OnCreate(self, form):
         """called when plugin form is created
 
@@ -454,6 +466,10 @@ class CapaExplorerForm(idaapi.PluginForm):
         label2.setText("Editor")
         label2.setFont(font)
 
+        self.view_rulegen_limit_features_by_ea = QtWidgets.QCheckBox("Limit features to current dissasembly address")
+        self.view_rulegen_limit_features_by_ea.setChecked(False)
+        self.view_rulegen_limit_features_by_ea.stateChanged.connect(self.slot_checkbox_limit_features_by_ea)
+
         self.view_rulegen_status_label = QtWidgets.QLabel()
         self.view_rulegen_status_label.setAlignment(QtCore.Qt.AlignLeft)
         self.view_rulegen_status_label.setText("")
@@ -484,6 +500,7 @@ class CapaExplorerForm(idaapi.PluginForm):
         layout3.addWidget(self.view_rulegen_editor, 65)
 
         layout2.addWidget(self.view_rulegen_header_label)
+        layout2.addWidget(self.view_rulegen_limit_features_by_ea)
         layout2.addWidget(self.view_rulegen_search)
         layout2.addWidget(self.view_rulegen_features)
 
@@ -548,6 +565,10 @@ class CapaExplorerForm(idaapi.PluginForm):
         self.limit_results_to_function(idaapi.get_func(ea))
         self.view_tree.reset_ui()
 
+    def update_rulegen_tree_limit_features_to_selection(self, ea):
+        """ """
+        self.view_rulegen_features.filter_items_by_ea(ea)
+
     def ida_hook_screen_ea_changed(self, widget, new_ea, old_ea):
         """function hook for IDA "screen ea changed" action
 
@@ -568,6 +589,9 @@ class CapaExplorerForm(idaapi.PluginForm):
         if not idaapi.get_func(new_ea):
             return
 
+        if self.view_tabs.currentIndex() == 1 and self.view_rulegen_limit_features_by_ea.isChecked():
+            return self.update_rulegen_tree_limit_features_to_selection(new_ea)
+
         if idaapi.get_func(new_ea) == idaapi.get_func(old_ea):
             # user navigated same function - ignore
             return
@@ -603,7 +627,7 @@ class CapaExplorerForm(idaapi.PluginForm):
                 path = self.ask_user_directory()
                 if not path:
                     logger.warning(
-                        "You must select a file directory containing capa rules before analysis can be run. The standard collection of capa rules can be downloaded from https://github.com/fireeye/capa-rules."
+                        "You must select a file directory containing capa rules before analysis can be run. The standard collection of capa rules can be downloaded from https://github.com/mandiant/capa-rules."
                     )
                     return False
                 settings.user[CAPA_SETTINGS_RULE_PATH] = path
@@ -670,7 +694,7 @@ class CapaExplorerForm(idaapi.PluginForm):
             )
             logger.error("Failed to load rules from %s (error: %s).", settings.user[CAPA_SETTINGS_RULE_PATH], e)
             logger.error(
-                "Make sure your file directory contains properly formatted capa rules. You can download the standard collection of capa rules from https://github.com/fireeye/capa-rules."
+                "Make sure your file directory contains properly formatted capa rules. You can download the standard collection of capa rules from https://github.com/mandiant/capa-rules."
             )
             settings.user[CAPA_SETTINGS_RULE_PATH] = ""
             return False
@@ -727,6 +751,7 @@ class CapaExplorerForm(idaapi.PluginForm):
                 meta = capa.ida.helpers.collect_metadata()
                 capabilities, counts = capa.main.find_capabilities(self.ruleset_cache, extractor, disable_progress=True)
                 meta["analysis"].update(counts)
+                meta["analysis"]["layout"] = capa.main.compute_layout(self.ruleset_cache, extractor, capabilities)
             except UserCancelledError:
                 logger.info("User cancelled analysis.")
                 return False
@@ -770,7 +795,9 @@ class CapaExplorerForm(idaapi.PluginForm):
             update_wait_box("rendering results")
 
             try:
-                self.doc = capa.render.convert_capabilities_to_result_document(meta, self.ruleset_cache, capabilities)
+                self.doc = capa.render.result_document.convert_capabilities_to_result_document(
+                    meta, self.ruleset_cache, capabilities
+                )
             except Exception as e:
                 logger.error("Failed to render results (error: %s)", e)
                 return False
@@ -865,7 +892,7 @@ class CapaExplorerForm(idaapi.PluginForm):
                         if rule.meta.get("capa/subscope-rule"):
                             continue
                         for (ea, _) in res:
-                            func_features[capa.features.MatchedRule(name)].add(ea)
+                            func_features[capa.features.common.MatchedRule(name)].add(ea)
                 except Exception as e:
                     logger.error("Failed to match function/basic block rule scope (error: %s)" % e)
                     return False
@@ -899,7 +926,7 @@ class CapaExplorerForm(idaapi.PluginForm):
                     if rule.meta.get("capa/subscope-rule"):
                         continue
                     for (ea, _) in res:
-                        file_features[capa.features.MatchedRule(name)].add(ea)
+                        file_features[capa.features.common.MatchedRule(name)].add(ea)
             except Exception as e:
                 logger.error("Failed to match file scope rules (error: %s)" % e)
                 return False
@@ -967,6 +994,7 @@ class CapaExplorerForm(idaapi.PluginForm):
         self.view_rulegen_editor.reset_view()
         self.view_rulegen_preview.reset_view()
         self.view_rulegen_search.clear()
+        self.view_rulegen_limit_features_by_ea.setChecked(False)
         self.set_rulegen_preview_border_neutral()
         self.rulegen_current_function = None
         self.rulegen_func_features_cache = {}
@@ -1005,7 +1033,7 @@ class CapaExplorerForm(idaapi.PluginForm):
 
     def update_rule_status(self, rule_text):
         """ """
-        if self.view_rulegen_editor.root is None:
+        if not self.view_rulegen_editor.invisibleRootItem().childCount():
             self.set_rulegen_preview_border_neutral()
             self.view_rulegen_status_label.clear()
             return
@@ -1123,9 +1151,9 @@ class CapaExplorerForm(idaapi.PluginForm):
             idaapi.info("No program analysis to save.")
             return
 
-        s = json.dumps(self.doc, sort_keys=True, cls=capa.render.CapaJsonObjectEncoder).encode("utf-8")
+        s = json.dumps(self.doc, sort_keys=True, cls=capa.render.json.CapaJsonObjectEncoder).encode("utf-8")
 
-        path = idaapi.ask_file(True, "*.json", "Choose file to save capa program analysis JSON")
+        path = self.ask_user_capa_json_file()
         if not path:
             return
 
@@ -1158,6 +1186,13 @@ class CapaExplorerForm(idaapi.PluginForm):
 
         self.view_tree.reset_ui()
 
+    def slot_checkbox_limit_features_by_ea(self, state):
+        """ """
+        if state == QtCore.Qt.Checked:
+            self.view_rulegen_features.filter_items_by_ea(idaapi.get_screen_ea())
+        else:
+            self.view_rulegen_features.show_all_items()
+
     def slot_checkbox_show_results_by_function_changed(self, state):
         """slot activated if checkbox clicked
 
@@ -1201,7 +1236,16 @@ class CapaExplorerForm(idaapi.PluginForm):
     def ask_user_capa_rule_file(self):
         """ """
         return QtWidgets.QFileDialog.getSaveFileName(
-            None, "Please select a capa rule to edit", settings.user.get(CAPA_SETTINGS_RULE_PATH, ""), "*.yml"
+            None,
+            "Please select a location to save capa rule file",
+            settings.user.get(CAPA_SETTINGS_RULE_PATH, ""),
+            "*.yml",
+        )[0]
+
+    def ask_user_capa_json_file(self):
+        """ """
+        return QtWidgets.QFileDialog.getSaveFileName(
+            None, "Please select a location to save capa JSON file", "", "*.json"
         )[0]
 
     def set_view_status_label(self, text):

capa/ida/plugin/hooks.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt

capa/ida/plugin/item.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,7 +6,6 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-import sys
 import codecs
 
 import idc
@@ -32,7 +31,7 @@ def location_to_hex(location):
     return "%08X" % location
 
 
-class CapaExplorerDataItem(object):
+class CapaExplorerDataItem:
     """store data for CapaExplorerDataModel"""
 
     def __init__(self, parent, data, can_check=True):
@@ -202,7 +201,7 @@ class CapaExplorerRuleMatchItem(CapaExplorerDataItem):
 
     @property
     def source(self):
-        """ return rule contents for display """
+        """return rule contents for display"""
         return self._source
 
 
@@ -328,14 +327,10 @@ class CapaExplorerByteViewItem(CapaExplorerFeatureItem):
         """
         byte_snap = idaapi.get_bytes(location, 32)
 
+        details = ""
         if byte_snap:
             byte_snap = codecs.encode(byte_snap, "hex").upper()
-            if sys.version_info >= (3, 0):
-                details = " ".join([byte_snap[i : i + 2].decode() for i in range(0, len(byte_snap), 2)])
-            else:
-                details = " ".join([byte_snap[i : i + 2] for i in range(0, len(byte_snap), 2)])
-        else:
-            details = ""
+            details = " ".join([byte_snap[i : i + 2].decode() for i in range(0, len(byte_snap), 2)])
 
         super(CapaExplorerByteViewItem, self).__init__(parent, display, location=location, details=details)
         self.ida_highlight = idc.get_color(location, idc.CIC_ITEM)

capa/ida/plugin/model.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -12,8 +12,10 @@ import idc
 import idaapi
 from PyQt5 import QtGui, QtCore
 
+import capa.rules
 import capa.ida.helpers
 import capa.render.utils as rutils
+import capa.features.common
 from capa.ida.plugin.item import (
     CapaExplorerDataItem,
     CapaExplorerRuleItem,
@@ -433,12 +435,18 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
             for ea in rule["matches"].keys():
                 ea = capa.ida.helpers.get_func_start_ea(ea)
                 if ea is None:
-                    # file scope, skip for rendering in this mode
+                    # file scope, skip rendering in this mode
                     continue
-                if None is matches_by_function.get(ea, None):
-                    matches_by_function[ea] = CapaExplorerFunctionItem(self.root_node, ea, can_check=False)
+                if not matches_by_function.get(ea, ()):
+                    # new function root
+                    matches_by_function[ea] = (CapaExplorerFunctionItem(self.root_node, ea, can_check=False), [])
+                function_root, match_cache = matches_by_function[ea]
+                if rule["meta"]["name"] in match_cache:
+                    # rule match already rendered for this function root, skip it
+                    continue
+                match_cache.append(rule["meta"]["name"])
                 CapaExplorerRuleItem(
-                    matches_by_function[ea],
+                    function_root,
                     rule["meta"]["name"],
                     rule["meta"].get("namespace"),
                     len(rule["matches"]),
@@ -492,7 +500,7 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
         value = feature[feature["type"]]
         if value:
             if key == "string":
-                value = '"%s"' % capa.features.escape_string(value)
+                value = '"%s"' % capa.features.common.escape_string(value)
             if feature.get("description", ""):
                 return "%s(%s = %s)" % (key, value, feature["description"])
             else:
@@ -554,10 +562,15 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
                 parent, display, source=doc["rules"].get(feature[feature["type"]], {}).get("source", "")
             )
 
-        if feature["type"] == "regex":
-            return CapaExplorerStringViewItem(
-                parent, display, location, '"%s"' % capa.features.escape_string(feature["match"])
-            )
+        if feature["type"] in ("regex", "substring"):
+            for s, locations in feature["matches"].items():
+                if location in locations:
+                    return CapaExplorerStringViewItem(
+                        parent, display, location, '"' + capa.features.common.escape_string(s) + '"'
+                    )
+
+            # programming error: the given location should always be found in the regex matches
+            raise ValueError("regex match at location not found")
 
         if feature["type"] == "basicblock":
             return CapaExplorerBlockItem(parent, location)
@@ -583,11 +596,14 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
         if feature["type"] in ("string",):
             # display string preview
             return CapaExplorerStringViewItem(
-                parent, display, location, '"%s"' % capa.features.escape_string(feature[feature["type"]])
+                parent, display, location, '"%s"' % capa.features.common.escape_string(feature[feature["type"]])
             )
 
-        if feature["type"] in ("import", "export"):
+        if feature["type"] in ("import", "export", "function-name"):
             # display no preview
+            return CapaExplorerFeatureItem(parent, location=location, display=display)
+
+        if feature["type"] in ("arch", "os", "format"):
             return CapaExplorerFeatureItem(parent, display=display)
 
         raise RuntimeError("unexpected feature type: " + str(feature["type"]))

capa/ida/plugin/proxy.py

@@ -1,11 +1,10 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
 # Unless required by applicable law or agreed to in writing, software distributed under the License
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
-import six
 from PyQt5 import QtCore
 from PyQt5.QtCore import Qt
 
@@ -208,7 +207,7 @@ class CapaExplorerSearchProxyModel(QtCore.QSortFilterProxyModel):
             if not data:
                 continue
 
-            if not isinstance(data, six.string_types):
+            if not isinstance(data, str):
                 # sanity check: should already be a string, but double check
                 continue
 

capa/ida/plugin/view.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -9,11 +9,13 @@ import re
 from collections import Counter
 
 import idc
+import idaapi
 from PyQt5 import QtGui, QtCore, QtWidgets
 
 import capa.rules
 import capa.engine
 import capa.ida.helpers
+import capa.features.common
 import capa.features.basicblock
 from capa.ida.plugin.item import CapaExplorerFunctionItem
 from capa.ida.plugin.model import CapaExplorerDataModel
@@ -25,7 +27,7 @@ COLOR_GREEN_RGB = (79, 121, 66)
 COLOR_BLUE_RGB = (37, 147, 215)
 
 
-def calc_level_by_indent(line, prev_level=0):
+def calc_indent_from_line(line, prev_level=0):
     """ """
     if not len(line.strip()):
         # blank line, which may occur for comments so we simply use the last level
@@ -35,10 +37,13 @@ def calc_level_by_indent(line, prev_level=0):
         # need to adjust two spaces when encountering string description
         line = line[2:]
     # calc line level based on preceding whitespace
-    return len(line) - len(stripped)
+    indent = len(line) - len(stripped)
+
+    # round up to nearest even number; helps keep parsing more sane
+    return indent + (indent % 2)
 
 
-def parse_feature_for_node(feature):
+def parse_yaml_line(feature):
     """ """
     description = ""
     comment = ""
@@ -111,30 +116,6 @@ def parse_node_for_feature(feature, description, comment, depth):
     return display if display.endswith("\n") else display + "\n"
 
 
-def yaml_to_nodes(s):
-    level = 0
-    for line in s.splitlines():
-        feature, description, comment = parse_feature_for_node(line.strip())
-
-        o = QtWidgets.QTreeWidgetItem(None)
-
-        # set node attributes
-        setattr(o, "capa_level", calc_level_by_indent(line, level))
-
-        if feature.startswith(("- and:", "- or:", "- not:", "- basic block:", "- optional:")):
-            setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_expression())
-        elif feature.startswith("#"):
-            setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_comment())
-        else:
-            setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_feature())
-
-        # set node text
-        for (i, v) in enumerate((feature, description, comment)):
-            o.setText(i, v)
-
-        yield o
-
-
 def iterate_tree(o):
     """ """
     itr = QtWidgets.QTreeWidgetItemIterator(o)
@@ -143,6 +124,13 @@ def iterate_tree(o):
         itr += 1
 
 
+def expand_tree(root):
+    """ """
+    for node in iterate_tree(root):
+        if node.childCount() and not node.isExpanded():
+            node.setExpanded(True)
+
+
 def calc_item_depth(o):
     """ """
     depth = 0
@@ -177,6 +165,13 @@ def build_context_menu(o, actions):
     return menu
 
 
+def resize_columns_to_content(header):
+    """ """
+    header.resizeSections(QtWidgets.QHeaderView.ResizeToContents)
+    if header.sectionSize(0) > MAX_SECTION_SIZE:
+        header.resizeSection(0, MAX_SECTION_SIZE)
+
+
 class CapaExplorerRulgenPreview(QtWidgets.QTextEdit):
 
     INDENT = " " * 2
@@ -318,7 +313,6 @@ class CapaExplorerRulgenEditor(QtWidgets.QTreeWidget):
         self.preview = preview
 
         self.setHeaderLabels(["Feature", "Description", "Comment"])
-        self.header().setSectionResizeMode(QtWidgets.QHeaderView.ResizeToContents)
         self.header().setStretchLastSection(False)
         self.setExpandsOnDoubleClick(False)
         self.setEditTriggers(QtWidgets.QAbstractItemView.NoEditTriggers)
@@ -326,6 +320,10 @@ class CapaExplorerRulgenEditor(QtWidgets.QTreeWidget):
         self.setSelectionMode(QtWidgets.QAbstractItemView.ExtendedSelection)
         self.setStyleSheet("QTreeView::item {padding-right: 15 px;padding-bottom: 2 px;}")
 
+        # configure view columns to auto-resize
+        for idx in range(3):
+            self.header().setSectionResizeMode(idx, QtWidgets.QHeaderView.Interactive)
+
         # enable drag and drop
         self.setDragEnabled(True)
         self.setAcceptDrops(True)
@@ -335,8 +333,9 @@ class CapaExplorerRulgenEditor(QtWidgets.QTreeWidget):
         self.itemChanged.connect(self.slot_item_changed)
         self.customContextMenuRequested.connect(self.slot_custom_context_menu_requested)
         self.itemDoubleClicked.connect(self.slot_item_double_clicked)
+        self.expanded.connect(self.slot_resize_columns_to_content)
+        self.collapsed.connect(self.slot_resize_columns_to_content)
 
-        self.root = None
         self.reset_view()
 
         self.is_editing = False
@@ -386,15 +385,17 @@ class CapaExplorerRulgenEditor(QtWidgets.QTreeWidget):
 
         super(CapaExplorerRulgenEditor, self).dropEvent(e)
 
-        # self.prune_expressions()
         self.update_preview()
-        self.expandAll()
+        expand_tree(self.invisibleRootItem())
 
     def reset_view(self):
         """ """
-        self.root = None
         self.clear()
 
+    def slot_resize_columns_to_content(self):
+        """ """
+        resize_columns_to_content(self.header())
+
     def slot_item_changed(self, item, column):
         """ """
         if self.is_editing:
@@ -404,16 +405,21 @@ class CapaExplorerRulgenEditor(QtWidgets.QTreeWidget):
     def slot_remove_selected(self, action):
         """ """
         for o in self.selectedItems():
-            if o == self.root:
+            if o.parent() is None:
+                # special handling for top-level items
                 self.takeTopLevelItem(self.indexOfTopLevelItem(o))
-                self.root = None
                 continue
             o.parent().removeChild(o)
 
     def slot_nest_features(self, action):
         """ """
-        # create a new parent under root node, by default; new node added last position in tree
-        new_parent = self.new_expression_node(self.root, (action.data()[0], ""))
+        # we don't want to add new features under the invisible root because capa rules should
+        # contain a single top-level node; this may not always be the case so we default to the last
+        # child node that was added to the invisible root
+        top_node = self.invisibleRootItem().child(self.invisibleRootItem().childCount() - 1)
+
+        # create a new parent under top-level node
+        new_parent = self.new_expression_node(top_node, (action.data()[0], ""))
 
         if "basic block" in action.data()[0]:
             # add default child expression when nesting under basic block
@@ -615,91 +621,138 @@ class CapaExplorerRulgenEditor(QtWidgets.QTreeWidget):
 
     def update_features(self, features):
         """ """
-        if not self.root:
-            # root node does not exist, create default node, set expanded
-            self.root = self.new_expression_node(self, ("- or:", ""))
+        if not self.invisibleRootItem().childCount():
+            # empty tree; add a default node
+            self.new_expression_node(self.invisibleRootItem(), ("- or:", ""))
+
+        # we don't want to add new features under the invisible root because capa rules should
+        # contain a single top-level node; this may not always be the case so we default to the last
+        # child node that was added to the invisible root
+        top_node = self.invisibleRootItem().child(self.invisibleRootItem().childCount() - 1)
 
         # build feature counts
         counted = list(zip(Counter(features).keys(), Counter(features).values()))
 
         # single features
         for (k, v) in filter(lambda t: t[1] == 1, counted):
-            if isinstance(k, (capa.features.String,)):
-                value = '"%s"' % capa.features.escape_string(k.get_value_str())
+            if isinstance(k, (capa.features.common.String,)):
+                value = '"%s"' % capa.features.common.escape_string(k.get_value_str())
             else:
                 value = k.get_value_str()
-            self.new_feature_node(self.root, ("- %s: %s" % (k.name.lower(), value), ""))
+            self.new_feature_node(top_node, ("- %s: %s" % (k.name.lower(), value), ""))
 
         # n > 1 features
         for (k, v) in filter(lambda t: t[1] > 1, counted):
             if k.value:
-                if isinstance(k, (capa.features.String,)):
-                    value = '"%s"' % capa.features.escape_string(k.get_value_str())
+                if isinstance(k, (capa.features.common.String,)):
+                    value = '"%s"' % capa.features.common.escape_string(k.get_value_str())
                 else:
                     value = k.get_value_str()
                 display = "- count(%s(%s)): %d" % (k.name.lower(), value, v)
             else:
                 display = "- count(%s): %d" % (k.name.lower(), v)
-            self.new_feature_node(self.root, (display, ""))
+            self.new_feature_node(top_node, (display, ""))
 
-        self.expandAll()
         self.update_preview()
+        expand_tree(self.invisibleRootItem())
+        resize_columns_to_content(self.header())
+
+    def make_child_node_from_feature(self, parent, feature):
+        """ """
+        feature, comment, description = feature
+
+        # we need special handling for the "description" tag; meaning we don't add a new node but simply
+        # set the "description" column for the appropriate parent node
+        if feature.startswith("description:"):
+            if not parent:
+                # we shouldn't have description without a parent; do nothing
+                return None
+
+            # we don't add a new node for description; either set description column of parent's last child
+            # or the parent itself
+            if parent.childCount():
+                parent.child(parent.childCount() - 1).setText(1, feature.lstrip("description:").lstrip())
+            else:
+                parent.setText(1, feature.lstrip("description:").lstrip())
+            return None
+        elif feature.startswith("- description:"):
+            if not parent:
+                # we shouldn't have a description without a parent; do nothing
+                return None
+
+            # we don't add a new node for description; set the description column of the parent instead
+            parent.setText(1, feature.lstrip("- description:").lstrip())
+            return None
+
+        node = QtWidgets.QTreeWidgetItem(parent)
+
+        # set node text to data parsed from feature
+        for (idx, text) in enumerate((feature, comment, description)):
+            node.setText(idx, text)
+
+        # we need to set our own type so we can control the GUI accordingly
+        if feature.startswith(("- and:", "- or:", "- not:", "- basic block:", "- optional:")):
+            setattr(node, "capa_type", CapaExplorerRulgenEditor.get_node_type_expression())
+        elif feature.startswith("#"):
+            setattr(node, "capa_type", CapaExplorerRulgenEditor.get_node_type_comment())
+        else:
+            setattr(node, "capa_type", CapaExplorerRulgenEditor.get_node_type_feature())
+
+        # format the node based on its type
+        (self.set_expression_node, self.set_feature_node, self.set_comment_node)[node.capa_type](node)
+
+        parent.addChild(node)
+
+        return node
 
     def load_features_from_yaml(self, rule_text, update_preview=False):
         """ """
-
-        def add_node(parent, node):
-            if node.text(0).startswith("description:"):
-                if parent.childCount():
-                    parent.child(parent.childCount() - 1).setText(1, node.text(0).lstrip("description:").lstrip())
-                else:
-                    parent.setText(1, node.text(0).lstrip("description:").lstrip())
-            elif node.text(0).startswith("- description:"):
-                parent.setText(1, node.text(0).lstrip("- description:").lstrip())
-            else:
-                parent.addChild(node)
-
-        def build(parent, nodes):
-            if nodes:
-                child_lvl = nodes[0].capa_level
-                while nodes:
-                    node = nodes.pop(0)
-                    if node.capa_level == child_lvl:
-                        add_node(parent, node)
-                    elif node.capa_level > child_lvl:
-                        nodes.insert(0, node)
-                        build(parent.child(parent.childCount() - 1), nodes)
-                    else:
-                        parent = parent.parent() if parent.parent() else parent
-                        add_node(parent, node)
-
         self.reset_view()
 
         # check for lack of features block
         if -1 == rule_text.find("features:"):
             return
 
-        rule_features = rule_text[rule_text.find("features:") + len("features:") :].strip()
-        rule_nodes = list(yaml_to_nodes(rule_features))
+        rule_features = rule_text[rule_text.find("features:") + len("features:") :].strip("\n")
 
-        # check for lack of nodes
-        if not rule_nodes:
+        if not rule_features:
+            # no features; nothing to do
             return
 
-        for o in rule_nodes:
-            (self.set_expression_node, self.set_feature_node, self.set_comment_node)[o.capa_type](o)
+        # build tree from yaml text using stack-based algorithm to build parent -> child edges
+        stack = [self.invisibleRootItem()]
+        for line in rule_features.splitlines():
+            if not len(line.strip()):
+                continue
 
-        self.root = rule_nodes.pop(0)
-        self.addTopLevelItem(self.root)
+            indent = calc_indent_from_line(line)
+
+            # we need to grow our stack to ensure proper parent -> child edges
+            if indent > len(stack):
+                stack.extend([None] * (indent - len(stack)))
+
+            # shave the stack; divide by 2 because even indent, add 1 to avoid shaving root node
+            stack[indent // 2 + 1 :] = []
+
+            # find our parent; should be last node in stack not None
+            parent = None
+            for o in stack[::-1]:
+                if o:
+                    parent = o
+                    break
+
+            node = self.make_child_node_from_feature(parent, parse_yaml_line(line.strip()))
+
+            # append our new node in case its a parent for another node
+            if node:
+                stack.append(node)
 
         if update_preview:
             self.preview.blockSignals(True)
             self.preview.setPlainText(rule_text)
             self.preview.blockSignals(False)
 
-        build(self.root, rule_nodes)
-
-        self.expandAll()
+        expand_tree(self.invisibleRootItem())
 
     def get_features(self, selected=False, ignore=()):
         """ """
@@ -735,9 +788,12 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
         self.editor = editor
 
         self.setHeaderLabels(["Feature", "Virtual Address"])
-        self.header().setSectionResizeMode(QtWidgets.QHeaderView.ResizeToContents)
         self.setStyleSheet("QTreeView::item {padding-right: 15 px;padding-bottom: 2 px;}")
 
+        # configure view columns to auto-resize
+        for idx in range(2):
+            self.header().setSectionResizeMode(idx, QtWidgets.QHeaderView.Interactive)
+
         self.setExpandsOnDoubleClick(False)
         self.setContextMenuPolicy(QtCore.Qt.CustomContextMenu)
         self.setSelectionMode(QtWidgets.QAbstractItemView.ExtendedSelection)
@@ -745,6 +801,8 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
         # connect slots
         self.itemDoubleClicked.connect(self.slot_item_double_clicked)
         self.customContextMenuRequested.connect(self.slot_custom_context_menu_requested)
+        self.expanded.connect(self.slot_resize_columns_to_content)
+        self.collapsed.connect(self.slot_resize_columns_to_content)
 
         self.reset_view()
 
@@ -772,12 +830,24 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
         """ """
         self.clear()
 
+    def slot_resize_columns_to_content(self):
+        """ """
+        resize_columns_to_content(self.header())
+
     def slot_add_selected_features(self, action):
         """ """
         selected = [item.data(0, 0x100) for item in self.selectedItems()]
         if selected:
             self.editor.update_features(selected)
 
+    def slot_add_n_bytes_feature(self, action):
+        """ """
+        count = idaapi.ask_long(16, f"Enter number of bytes (1-{capa.features.common.MAX_BYTES_FEATURE_SIZE}):")
+        if count and 1 <= count <= capa.features.common.MAX_BYTES_FEATURE_SIZE:
+            item = self.selectedItems()[0].data(0, 0x100)
+            item.value = item.value[:count]
+            self.editor.update_features([item])
+
     def slot_custom_context_menu_requested(self, pos):
         """ """
         actions = []
@@ -789,6 +859,8 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
 
         if selected_items_count == 1:
             action_add_features_fmt = "Add feature"
+            if isinstance(self.selectedItems()[0].data(0, 0x100), capa.features.common.Bytes):
+                actions.append(("Add n bytes...", (), self.slot_add_n_bytes_feature))
         else:
             action_add_features_fmt = "Add %d features" % selected_items_count
 
@@ -818,13 +890,58 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
                 if data:
                     to_match = data.get_value_str()
                     if not to_match or text.lower() not in to_match.lower():
-                        o.setHidden(True)
+                        if not o.isHidden():
+                            o.setHidden(True)
                         continue
-                o.setHidden(False)
-                o.setExpanded(True)
+                if o.isHidden():
+                    o.setHidden(False)
+                if o.childCount() and not o.isExpanded():
+                    o.setExpanded(True)
         else:
             self.show_all_items()
 
+    def filter_items_by_ea(self, min_ea, max_ea=None):
+        """ """
+        visited = []
+
+        def show_item_and_parents(_o):
+            """iteratively show and expand an item and its' parents"""
+            while _o:
+                visited.append(_o)
+                if _o.isHidden():
+                    _o.setHidden(False)
+                if _o.childCount() and not _o.isExpanded():
+                    _o.setExpanded(True)
+                _o = _o.parent()
+
+        for o in iterate_tree(self):
+            if o in visited:
+                # save some cycles, only visit item once
+                continue
+
+            # read ea from "Address" column
+            o_ea = o.text(CapaExplorerRulegenFeatures.get_column_address_index())
+
+            if o_ea == "":
+                # ea may be empty, hide by default
+                if not o.isHidden():
+                    o.setHidden(True)
+                continue
+
+            o_ea = int(o_ea, 16)
+
+            if max_ea is not None and min_ea <= o_ea <= max_ea:
+                show_item_and_parents(o)
+            elif o_ea == min_ea:
+                show_item_and_parents(o)
+            else:
+                # made it here, hide by default
+                if not o.isHidden():
+                    o.setHidden(True)
+
+        # resize the view for UX
+        resize_columns_to_content(self.header())
+
     def style_parent_node(self, o):
         """ """
         font = QtGui.QFont()
@@ -886,6 +1003,7 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
         self.parse_features_for_tree(self.new_parent_node(self, ("File Scope",)), file_features)
         if func_features:
             self.parse_features_for_tree(self.new_parent_node(self, ("Function/Basic Block Scope",)), func_features)
+        resize_columns_to_content(self.header())
 
     def parse_features_for_tree(self, parent, features):
         """ """
@@ -898,8 +1016,8 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
             """ """
             name = feature.name.lower()
             value = feature.get_value_str()
-            if isinstance(feature, (capa.features.String,)):
-                value = '"%s"' % capa.features.escape_string(value)
+            if isinstance(feature, (capa.features.common.String,)):
+                value = '"%s"' % capa.features.common.escape_string(value)
             return "%s(%s)" % (name, value)
 
         for (feature, eas) in sorted(features.items(), key=lambda k: sorted(k[1])):
@@ -930,7 +1048,11 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
                         self.parent_items[feature], (format_feature(feature), format_address(ea)), feature=feature
                     )
             else:
-                ea = eas.pop()
+                if eas:
+                    ea = eas.pop()
+                else:
+                    # some features may not have an address e.g. "format"
+                    ea = ""
                 for (i, v) in enumerate((format_feature(feature), format_address(ea))):
                     self.parent_items[feature].setText(i, v)
                 self.parent_items[feature].setData(0, 0x100, feature)
@@ -999,11 +1121,7 @@ class CapaExplorerQtreeView(QtWidgets.QTreeView):
     def slot_resize_columns_to_content(self):
         """reset view columns to contents"""
         if self.should_resize_columns:
-            self.header().resizeSections(QtWidgets.QHeaderView.ResizeToContents)
-
-            # limit size of first section
-            if self.header().sectionSize(0) > MAX_SECTION_SIZE:
-                self.header().resizeSection(0, MAX_SECTION_SIZE)
+            resize_columns_to_content(self.header())
 
     def map_index_to_source_item(self, model_index):
         """map proxy model index to source model item

capa/optimizer.py

@@ -0,0 +1,70 @@
+import logging
+
+import capa.engine as ceng
+import capa.features.common
+
+logger = logging.getLogger(__name__)
+
+
+def get_node_cost(node):
+    if isinstance(node, (capa.features.common.OS, capa.features.common.Arch, capa.features.common.Format)):
+        # we assume these are the most restrictive features:
+        # authors commonly use them at the start of rules to restrict the category of samples to inspect
+        return 0
+
+    # elif "everything else":
+    #   return 1
+    #
+    # this should be all hash-lookup features.
+    # see below.
+
+    elif isinstance(node, (capa.features.common.Substring, capa.features.common.Regex, capa.features.common.Bytes)):
+        # substring and regex features require a full scan of each string
+        # which we anticipate is more expensive then a hash lookup feature (e.g. mnemonic or count).
+        #
+        # TODO: compute the average cost of these feature relative to hash feature
+        # and adjust the factor accordingly.
+        return 2
+
+    elif isinstance(node, (ceng.Not, ceng.Range)):
+        # the cost of these nodes are defined by the complexity of their single child.
+        return 1 + get_node_cost(node.child)
+
+    elif isinstance(node, (ceng.And, ceng.Or, ceng.Some)):
+        # the cost of these nodes is the full cost of their children
+        # as this is the worst-case scenario.
+        return 1 + sum(map(get_node_cost, node.children))
+
+    else:
+        # this should be all hash-lookup features.
+        # we give this a arbitrary weight of 1.
+        # the only thing more "important" than this is checking OS/Arch/Format.
+        return 1
+
+
+def optimize_statement(statement):
+    # this routine operates in-place
+
+    if isinstance(statement, (ceng.And, ceng.Or, ceng.Some)):
+        # has .children
+        statement.children = sorted(statement.children, key=lambda n: get_node_cost(n))
+        return
+    elif isinstance(statement, (ceng.Not, ceng.Range)):
+        # has .child
+        optimize_statement(statement.child)
+        return
+    else:
+        # appears to be "simple"
+        return
+
+
+def optimize_rule(rule):
+    # this routine operates in-place
+    optimize_statement(rule.statement)
+
+
+def optimize_rules(rules):
+    logger.debug("optimizing %d rules", len(rules))
+    for rule in rules:
+        optimize_rule(rule)
+    return rules

capa/perf.py

@@ -0,0 +1,10 @@
+import collections
+from typing import Dict
+
+# this structure is unstable and may change before the next major release.
+counters: Dict[str, int] = collections.Counter()
+
+
+def reset():
+    global counters
+    counters = collections.Counter()

capa/render/__init__.py

@@ -1,266 +0,0 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-
-import json
-
-import six
-
-import capa.rules
-import capa.engine
-
-
-def convert_statement_to_result_document(statement):
-    """
-    "statement": {
-        "type": "or"
-    },
-
-    "statement": {
-        "max": 9223372036854775808,
-        "min": 2,
-        "type": "range"
-    },
-    """
-    statement_type = statement.name.lower()
-    result = {"type": statement_type}
-    if statement.description:
-        result["description"] = statement.description
-
-    if statement_type == "some" and statement.count == 0:
-        result["type"] = "optional"
-    elif statement_type == "some":
-        result["count"] = statement.count
-    elif statement_type == "range":
-        result["min"] = statement.min
-        result["max"] = statement.max
-        result["child"] = convert_feature_to_result_document(statement.child)
-    elif statement_type == "subscope":
-        result["subscope"] = statement.scope
-
-    return result
-
-
-def convert_feature_to_result_document(feature):
-    """
-    "feature": {
-        "number": 6,
-        "type": "number"
-    },
-
-    "feature": {
-        "api": "ws2_32.WSASocket",
-        "type": "api"
-    },
-
-    "feature": {
-        "match": "create TCP socket",
-        "type": "match"
-    },
-
-    "feature": {
-        "characteristic": [
-            "loop",
-            true
-        ],
-        "type": "characteristic"
-    },
-    """
-    result = {"type": feature.name, feature.name: feature.get_value_str()}
-    if feature.description:
-        result["description"] = feature.description
-    if feature.name == "regex":
-        result["match"] = feature.match
-    return result
-
-
-def convert_node_to_result_document(node):
-    """
-    "node": {
-        "type": "statement",
-        "statement": { ... }
-    },
-
-    "node": {
-        "type": "feature",
-        "feature": { ... }
-    },
-    """
-
-    if isinstance(node, capa.engine.Statement):
-        return {
-            "type": "statement",
-            "statement": convert_statement_to_result_document(node),
-        }
-    elif isinstance(node, capa.features.Feature):
-        return {
-            "type": "feature",
-            "feature": convert_feature_to_result_document(node),
-        }
-    else:
-        raise RuntimeError("unexpected match node type")
-
-
-def convert_match_to_result_document(rules, capabilities, result):
-    """
-    convert the given Result instance into a common, Python-native data structure.
-    this will become part of the "result document" format that can be emitted to JSON.
-    """
-    doc = {
-        "success": bool(result.success),
-        "node": convert_node_to_result_document(result.statement),
-        "children": [convert_match_to_result_document(rules, capabilities, child) for child in result.children],
-    }
-
-    # logic expression, like `and`, don't have locations - their children do.
-    # so only add `locations` to feature nodes.
-    if isinstance(result.statement, capa.features.Feature):
-        if bool(result.success):
-            doc["locations"] = result.locations
-    elif isinstance(result.statement, capa.rules.Range):
-        if bool(result.success):
-            doc["locations"] = result.locations
-
-    # if we have a `match` statement, then we're referencing another rule.
-    # this could an external rule (written by a human), or
-    #  rule generated to support a subscope (basic block, etc.)
-    # we still want to include the matching logic in this tree.
-    #
-    # so, we need to lookup the other rule results
-    # and then filter those down to the address used here.
-    # finally, splice that logic into this tree.
-    if (
-        doc["node"]["type"] == "feature"
-        and doc["node"]["feature"]["type"] == "match"
-        # only add subtree on success,
-        # because there won't be results for the other rule on failure.
-        and doc["success"]
-    ):
-
-        rule_name = doc["node"]["feature"]["match"]
-        rule = rules[rule_name]
-        rule_matches = {address: result for (address, result) in capabilities[rule_name]}
-
-        if rule.meta.get("capa/subscope-rule"):
-            # for a subscope rule, fixup the node to be a scope node, rather than a match feature node.
-            #
-            # e.g. `contain loop/30c4c78e29bf4d54894fc74f664c62e8` -> `basic block`
-            scope = rule.meta["scope"]
-            doc["node"] = {
-                "type": "statement",
-                "statement": {
-                    "type": "subscope",
-                    "subscope": scope,
-                },
-            }
-
-        for location in doc["locations"]:
-            doc["children"].append(convert_match_to_result_document(rules, capabilities, rule_matches[location]))
-
-    return doc
-
-
-def convert_capabilities_to_result_document(meta, rules, capabilities):
-    """
-    convert the given rule set and capabilities result to a common, Python-native data structure.
-    this format can be directly emitted to JSON, or passed to the other `render_*` routines
-     to render as text.
-
-    see examples of substructures in above routines.
-
-    schema:
-
-    ```json
-    {
-      "meta": {...},
-      "rules: {
-        $rule-name: {
-          "meta": {...copied from rule.meta...},
-          "matches: {
-            $address: {...match details...},
-            ...
-          }
-        },
-        ...
-      }
-    }
-    ```
-
-    Args:
-      meta (Dict[str, Any]):
-      rules (RuleSet):
-      capabilities (Dict[str, List[Tuple[int, Result]]]):
-    """
-    doc = {
-        "meta": meta,
-        "rules": {},
-    }
-
-    for rule_name, matches in capabilities.items():
-        rule = rules[rule_name]
-
-        if rule.meta.get("capa/subscope-rule"):
-            continue
-
-        doc["rules"][rule_name] = {
-            "meta": dict(rule.meta),
-            "source": rule.definition,
-            "matches": {
-                addr: convert_match_to_result_document(rules, capabilities, match) for (addr, match) in matches
-            },
-        }
-
-    return doc
-
-
-def render_vverbose(meta, rules, capabilities):
-    # there's an import loop here
-    # if capa.render imports capa.render.vverbose
-    # and capa.render.vverbose import capa.render (implicitly, as a submodule)
-    # so, defer the import until routine is called, breaking the import loop.
-    import capa.render.vverbose
-
-    doc = convert_capabilities_to_result_document(meta, rules, capabilities)
-    return capa.render.vverbose.render_vverbose(doc)
-
-
-def render_verbose(meta, rules, capabilities):
-    # break import loop
-    import capa.render.verbose
-
-    doc = convert_capabilities_to_result_document(meta, rules, capabilities)
-    return capa.render.verbose.render_verbose(doc)
-
-
-def render_default(meta, rules, capabilities):
-    # break import loop
-    import capa.render.default
-    import capa.render.verbose
-
-    doc = convert_capabilities_to_result_document(meta, rules, capabilities)
-    return capa.render.default.render_default(doc)
-
-
-class CapaJsonObjectEncoder(json.JSONEncoder):
-    """JSON encoder that emits Python sets as sorted lists"""
-
-    def default(self, obj):
-        if isinstance(obj, (list, dict, int, float, bool, type(None))) or isinstance(obj, six.string_types):
-            return json.JSONEncoder.default(self, obj)
-        elif isinstance(obj, set):
-            return list(sorted(obj))
-        else:
-            # probably will TypeError
-            return json.JSONEncoder.default(self, obj)
-
-
-def render_json(meta, rules, capabilities):
-    return json.dumps(
-        convert_capabilities_to_result_document(meta, rules, capabilities),
-        cls=CapaJsonObjectEncoder,
-        sort_keys=True,
-    )

capa/render/default.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -8,15 +8,18 @@
 
 import collections
 
-import six
 import tabulate
 
 import capa.render.utils as rutils
+import capa.render.result_document
+from capa.rules import RuleSet
+from capa.engine import MatchResults
+from capa.render.utils import StringIO
 
 tabulate.PRESERVE_WHITESPACE = True
 
 
-def width(s, character_count):
+def width(s: str, character_count: int) -> str:
     """pad the given string to at least `character_count`"""
     if len(s) < character_count:
         return s + " " * (character_count - len(s))
@@ -24,11 +27,14 @@ def width(s, character_count):
         return s
 
 
-def render_meta(doc, ostream):
+def render_meta(doc, ostream: StringIO):
     rows = [
         (width("md5", 22), width(doc["meta"]["sample"]["md5"], 82)),
         ("sha1", doc["meta"]["sample"]["sha1"]),
         ("sha256", doc["meta"]["sample"]["sha256"]),
+        ("os", doc["meta"]["analysis"]["os"]),
+        ("format", doc["meta"]["analysis"]["format"]),
+        ("arch", doc["meta"]["analysis"]["arch"]),
         ("path", doc["meta"]["sample"]["path"]),
     ]
 
@@ -64,7 +70,7 @@ def find_subrule_matches(doc):
     return matches
 
 
-def render_capabilities(doc, ostream):
+def render_capabilities(doc, ostream: StringIO):
     """
     example::
 
@@ -102,7 +108,7 @@ def render_capabilities(doc, ostream):
         ostream.writeln(rutils.bold("no capabilities found"))
 
 
-def render_attack(doc, ostream):
+def render_attack(doc, ostream: StringIO):
     """
     example::
 
@@ -124,27 +130,16 @@ def render_attack(doc, ostream):
             continue
 
         for attack in rule["meta"]["att&ck"]:
-            tactic, _, rest = attack.partition("::")
-            if "::" in rest:
-                technique, _, rest = rest.partition("::")
-                subtechnique, _, id = rest.rpartition(" ")
-                tactics[tactic].add((technique, subtechnique, id))
-            else:
-                technique, _, id = rest.rpartition(" ")
-                tactics[tactic].add((technique, id))
+            tactics[attack["tactic"]].add((attack["technique"], attack.get("subtechnique"), attack["id"]))
 
     rows = []
     for tactic, techniques in sorted(tactics.items()):
         inner_rows = []
-        for spec in sorted(techniques):
-            if len(spec) == 2:
-                technique, id = spec
+        for (technique, subtechnique, id) in sorted(techniques):
+            if subtechnique is None:
                 inner_rows.append("%s %s" % (rutils.bold(technique), id))
-            elif len(spec) == 3:
-                technique, subtechnique, id = spec
-                inner_rows.append("%s::%s %s" % (rutils.bold(technique), subtechnique, id))
             else:
-                raise RuntimeError("unexpected ATT&CK spec format")
+                inner_rows.append("%s::%s %s" % (rutils.bold(technique), subtechnique, id))
         rows.append(
             (
                 rutils.bold(tactic.upper()),
@@ -161,7 +156,7 @@ def render_attack(doc, ostream):
         ostream.write("\n")
 
 
-def render_mbc(doc, ostream):
+def render_mbc(doc, ostream: StringIO):
     """
     example::
 
@@ -180,32 +175,17 @@ def render_mbc(doc, ostream):
         if not rule["meta"].get("mbc"):
             continue
 
-        mbcs = rule["meta"]["mbc"]
-        if not isinstance(mbcs, list):
-            raise ValueError("invalid rule: MBC mapping is not a list")
-
-        for mbc in mbcs:
-            objective, _, rest = mbc.partition("::")
-            if "::" in rest:
-                behavior, _, rest = rest.partition("::")
-                method, _, id = rest.rpartition(" ")
-                objectives[objective].add((behavior, method, id))
-            else:
-                behavior, _, id = rest.rpartition(" ")
-                objectives[objective].add((behavior, id))
+        for mbc in rule["meta"]["mbc"]:
+            objectives[mbc["objective"]].add((mbc["behavior"], mbc.get("method"), mbc["id"]))
 
     rows = []
     for objective, behaviors in sorted(objectives.items()):
         inner_rows = []
-        for spec in sorted(behaviors):
-            if len(spec) == 2:
-                behavior, id = spec
-                inner_rows.append("%s %s" % (rutils.bold(behavior), id))
-            elif len(spec) == 3:
-                behavior, method, id = spec
-                inner_rows.append("%s::%s %s" % (rutils.bold(behavior), method, id))
+        for (behavior, method, id) in sorted(behaviors):
+            if method is None:
+                inner_rows.append("%s [%s]" % (rutils.bold(behavior), id))
             else:
-                raise RuntimeError("unexpected MBC spec format")
+                inner_rows.append("%s::%s [%s]" % (rutils.bold(behavior), method, id))
         rows.append(
             (
                 rutils.bold(objective.upper()),
@@ -232,3 +212,8 @@ def render_default(doc):
     render_capabilities(doc, ostream)
 
     return ostream.getvalue()
+
+
+def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
+    doc = capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities)
+    return render_default(doc)

capa/render/json.py

@@ -0,0 +1,33 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import json
+
+import capa.render.result_document
+from capa.rules import RuleSet
+from capa.engine import MatchResults
+
+
+class CapaJsonObjectEncoder(json.JSONEncoder):
+    """JSON encoder that emits Python sets as sorted lists"""
+
+    def default(self, obj):
+        if isinstance(obj, (list, dict, int, float, bool, type(None))) or isinstance(obj, str):
+            return json.JSONEncoder.default(self, obj)
+        elif isinstance(obj, set):
+            return list(sorted(obj))
+        else:
+            # probably will TypeError
+            return json.JSONEncoder.default(self, obj)
+
+
+def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
+    return json.dumps(
+        capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities),
+        cls=CapaJsonObjectEncoder,
+        sort_keys=True,
+    )

capa/render/result_document.py

@@ -0,0 +1,329 @@
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import copy
+
+import capa.rules
+import capa.engine
+import capa.render.utils
+import capa.features.common
+from capa.rules import RuleSet
+from capa.engine import MatchResults
+
+
+def convert_statement_to_result_document(statement):
+    """
+    "statement": {
+        "type": "or"
+    },
+
+    "statement": {
+        "max": 9223372036854775808,
+        "min": 2,
+        "type": "range"
+    },
+    """
+    statement_type = statement.name.lower()
+    result = {"type": statement_type}
+    if statement.description:
+        result["description"] = statement.description
+
+    if statement_type == "some" and statement.count == 0:
+        result["type"] = "optional"
+    elif statement_type == "some":
+        result["count"] = statement.count
+    elif statement_type == "range":
+        result["min"] = statement.min
+        result["max"] = statement.max
+        result["child"] = convert_feature_to_result_document(statement.child)
+    elif statement_type == "subscope":
+        result["subscope"] = statement.scope
+
+    return result
+
+
+def convert_feature_to_result_document(feature):
+    """
+    "feature": {
+        "number": 6,
+        "type": "number"
+    },
+
+    "feature": {
+        "api": "ws2_32.WSASocket",
+        "type": "api"
+    },
+
+    "feature": {
+        "match": "create TCP socket",
+        "type": "match"
+    },
+
+    "feature": {
+        "characteristic": [
+            "loop",
+            true
+        ],
+        "type": "characteristic"
+    },
+    """
+    result = {"type": feature.name, feature.name: feature.get_value_str()}
+    if feature.description:
+        result["description"] = feature.description
+    if feature.name in ("regex", "substring"):
+        result["matches"] = feature.matches
+    return result
+
+
+def convert_node_to_result_document(node):
+    """
+    "node": {
+        "type": "statement",
+        "statement": { ... }
+    },
+
+    "node": {
+        "type": "feature",
+        "feature": { ... }
+    },
+    """
+
+    if isinstance(node, capa.engine.Statement):
+        return {
+            "type": "statement",
+            "statement": convert_statement_to_result_document(node),
+        }
+    elif isinstance(node, capa.features.common.Feature):
+        return {
+            "type": "feature",
+            "feature": convert_feature_to_result_document(node),
+        }
+    else:
+        raise RuntimeError("unexpected match node type")
+
+
+def convert_match_to_result_document(rules, capabilities, result):
+    """
+    convert the given Result instance into a common, Python-native data structure.
+    this will become part of the "result document" format that can be emitted to JSON.
+    """
+    doc = {
+        "success": bool(result.success),
+        "node": convert_node_to_result_document(result.statement),
+        "children": [convert_match_to_result_document(rules, capabilities, child) for child in result.children],
+    }
+
+    # logic expression, like `and`, don't have locations - their children do.
+    # so only add `locations` to feature nodes.
+    if isinstance(result.statement, capa.features.common.Feature):
+        if bool(result.success):
+            doc["locations"] = result.locations
+    elif isinstance(result.statement, capa.engine.Range):
+        if bool(result.success):
+            doc["locations"] = result.locations
+
+    # if we have a `match` statement, then we're referencing another rule or namespace.
+    # this could an external rule (written by a human), or
+    #  rule generated to support a subscope (basic block, etc.)
+    # we still want to include the matching logic in this tree.
+    #
+    # so, we need to lookup the other rule results
+    # and then filter those down to the address used here.
+    # finally, splice that logic into this tree.
+    if (
+        doc["node"]["type"] == "feature"
+        and doc["node"]["feature"]["type"] == "match"
+        # only add subtree on success,
+        # because there won't be results for the other rule on failure.
+        and doc["success"]
+    ):
+
+        name = doc["node"]["feature"]["match"]
+
+        if name in rules:
+            # this is a rule that we're matching
+            #
+            # pull matches from the referenced rule into our tree here.
+            rule_name = doc["node"]["feature"]["match"]
+            rule = rules[rule_name]
+            rule_matches = {address: result for (address, result) in capabilities[rule_name]}
+
+            if rule.meta.get("capa/subscope-rule"):
+                # for a subscope rule, fixup the node to be a scope node, rather than a match feature node.
+                #
+                # e.g. `contain loop/30c4c78e29bf4d54894fc74f664c62e8` -> `basic block`
+                scope = rule.meta["scope"]
+                doc["node"] = {
+                    "type": "statement",
+                    "statement": {
+                        "type": "subscope",
+                        "subscope": scope,
+                    },
+                }
+
+            for location in doc["locations"]:
+                doc["children"].append(convert_match_to_result_document(rules, capabilities, rule_matches[location]))
+        else:
+            # this is a namespace that we're matching
+            #
+            # check for all rules in the namespace,
+            # seeing if they matched.
+            # if so, pull their matches into our match tree here.
+            ns_name = doc["node"]["feature"]["match"]
+            ns_rules = rules.rules_by_namespace[ns_name]
+
+            for rule in ns_rules:
+                if rule.name in capabilities:
+                    # the rule matched, so splice results into our tree here.
+                    #
+                    # note, there's a shortcoming in our result document schema here:
+                    # we lose the name of the rule that matched in a namespace.
+                    # for example, if we have a statement: `match: runtime/dotnet`
+                    # and we get matches, we can say the following:
+                    #
+                    #     match: runtime/dotnet @ 0x0
+                    #       or:
+                    #         import: mscoree._CorExeMain @ 0x402000
+                    #
+                    # however, we lose the fact that it was rule
+                    #   "compiled to the .NET platform"
+                    # that contained this logic and did the match.
+                    #
+                    # we could introduce an intermediate node here.
+                    # this would be a breaking change and require updates to the renderers.
+                    # in the meantime, the above might be sufficient.
+                    rule_matches = {address: result for (address, result) in capabilities[rule.name]}
+                    for location in doc["locations"]:
+                        # doc[locations] contains all matches for the given namespace.
+                        # for example, the feature might be `match: anti-analysis/packer`
+                        # which matches against "generic unpacker" and "UPX".
+                        # in this case, doc[locations] contains locations for *both* of thse.
+                        #
+                        # rule_matches contains the matches for the specific rule.
+                        # this is a subset of doc[locations].
+                        #
+                        # so, grab only the locations for current rule.
+                        if location in rule_matches:
+                            doc["children"].append(
+                                convert_match_to_result_document(rules, capabilities, rule_matches[location])
+                            )
+
+    return doc
+
+
+def convert_meta_to_result_document(meta):
+    # make a copy so that we don't modify the given parameter
+    meta = copy.deepcopy(meta)
+
+    attacks = meta.get("att&ck", [])
+    meta["att&ck"] = [parse_canonical_attack(attack) for attack in attacks]
+    mbcs = meta.get("mbc", [])
+    meta["mbc"] = [parse_canonical_mbc(mbc) for mbc in mbcs]
+    return meta
+
+
+def parse_canonical_attack(attack: str):
+    """
+    parse capa's canonical ATT&CK representation: `Tactic::Technique::Subtechnique [Identifier]`
+    """
+    tactic = ""
+    technique = ""
+    subtechnique = ""
+    parts, id = capa.render.utils.parse_parts_id(attack)
+    if len(parts) > 0:
+        tactic = parts[0]
+    if len(parts) > 1:
+        technique = parts[1]
+    if len(parts) > 2:
+        subtechnique = parts[2]
+
+    return {
+        "parts": parts,
+        "id": id,
+        "tactic": tactic,
+        "technique": technique,
+        "subtechnique": subtechnique,
+    }
+
+
+def parse_canonical_mbc(mbc: str):
+    """
+    parse capa's canonical MBC representation: `Objective::Behavior::Method [Identifier]`
+    """
+    objective = ""
+    behavior = ""
+    method = ""
+    parts, id = capa.render.utils.parse_parts_id(mbc)
+    if len(parts) > 0:
+        objective = parts[0]
+    if len(parts) > 1:
+        behavior = parts[1]
+    if len(parts) > 2:
+        method = parts[2]
+
+    return {
+        "parts": parts,
+        "id": id,
+        "objective": objective,
+        "behavior": behavior,
+        "method": method,
+    }
+
+
+def convert_capabilities_to_result_document(meta, rules: RuleSet, capabilities: MatchResults):
+    """
+    convert the given rule set and capabilities result to a common, Python-native data structure.
+    this format can be directly emitted to JSON, or passed to the other `capa.render.*.render()` routines
+     to render as text.
+
+    see examples of substructures in above routines.
+
+    schema:
+
+    ```json
+    {
+      "meta": {...},
+      "rules: {
+        $rule-name: {
+          "meta": {...copied from rule.meta...},
+          "matches: {
+            $address: {...match details...},
+            ...
+          }
+        },
+        ...
+      }
+    }
+    ```
+
+    Args:
+      meta (Dict[str, Any]):
+      rules (RuleSet):
+      capabilities (Dict[str, List[Tuple[int, Result]]]):
+    """
+    doc = {
+        "meta": meta,
+        "rules": {},
+    }
+
+    for rule_name, matches in capabilities.items():
+        rule = rules[rule_name]
+
+        if rule.meta.get("capa/subscope-rule"):
+            continue
+
+        rule_meta = convert_meta_to_result_document(rule.meta)
+
+        doc["rules"][rule_name] = {
+            "meta": rule_meta,
+            "source": rule.definition,
+            "matches": {
+                addr: convert_match_to_result_document(rules, capabilities, match) for (addr, match) in matches
+            },
+        }
+
+    return doc

capa/render/utils.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,21 +6,22 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-import six
+import io
+
 import termcolor
 
 
-def bold(s):
+def bold(s: str) -> str:
     """draw attention to the given string"""
     return termcolor.colored(s, "blue")
 
 
-def bold2(s):
+def bold2(s: str) -> str:
     """draw attention to the given string, within a `bold` section"""
     return termcolor.colored(s, "green")
 
 
-def hex(n):
+def hex(n: int) -> str:
     """render the given number using upper case hex, like: 0x123ABC"""
     if n < 0:
         return "-0x%X" % (-n)
@@ -28,6 +29,24 @@ def hex(n):
         return "0x%X" % n
 
 
+def parse_parts_id(s: str):
+    id = ""
+    parts = s.split("::")
+    if len(parts) > 0:
+        last = parts.pop()
+        last, _, id = last.rpartition(" ")
+        id = id.lstrip("[").rstrip("]")
+        parts.append(last)
+    return parts, id
+
+
+def format_parts_id(data):
+    """
+    format canonical representation of ATT&CK/MBC parts and ID
+    """
+    return "%s [%s]" % ("::".join(data["parts"]), data["id"])
+
+
 def capability_rules(doc):
     """enumerate the rules in (namespace, name) order that are 'capability' rules (not lib/subscope/disposition/etc)."""
     for (_, _, rule) in sorted(
@@ -41,6 +60,8 @@ def capability_rules(doc):
             continue
         if rule["meta"].get("maec/analysis-conclusion-ov"):
             continue
+        if rule["meta"].get("maec/malware-family"):
+            continue
         if rule["meta"].get("maec/malware-category"):
             continue
         if rule["meta"].get("maec/malware-category-ov"):
@@ -49,7 +70,7 @@ def capability_rules(doc):
         yield rule
 
 
-class StringIO(six.StringIO):
+class StringIO(io.StringIO):
     def writeln(self, s):
         self.write(s)
         self.write("\n")

capa/render/verbose.py

@@ -3,7 +3,7 @@ example::
 
     send data
     namespace    communication
-    author       william.ballenthin@fireeye.com
+    author       william.ballenthin@mandiant.com
     description  all known techniques for sending data to a potential C2 server
     scope        function
     examples     BFB9B5391A13D0AFD787E87AB90F14F5:0x13145D60
@@ -14,7 +14,7 @@ example::
                  0x10003415
                  0x10003797
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -26,6 +26,9 @@ import tabulate
 
 import capa.rules
 import capa.render.utils as rutils
+import capa.render.result_document
+from capa.rules import RuleSet
+from capa.engine import MatchResults
 
 
 def render_meta(ostream, doc):
@@ -38,7 +41,9 @@ def render_meta(ostream, doc):
         path                 /tmp/suspicious.dll_
         timestamp            2020-07-03T10:17:05.796933
         capa version         0.0.0
-        format               auto
+        os                   windows
+        format               pe
+        arch                 amd64
         extractor            VivisectFeatureExtractor
         base address         0x10000000
         rules                (embedded rules)
@@ -52,11 +57,14 @@ def render_meta(ostream, doc):
         ("path", doc["meta"]["sample"]["path"]),
         ("timestamp", doc["meta"]["timestamp"]),
         ("capa version", doc["meta"]["version"]),
+        ("os", doc["meta"]["analysis"]["os"]),
         ("format", doc["meta"]["analysis"]["format"]),
+        ("arch", doc["meta"]["analysis"]["arch"]),
         ("extractor", doc["meta"]["analysis"]["extractor"]),
         ("base address", hex(doc["meta"]["analysis"]["base_address"])),
         ("rules", doc["meta"]["analysis"]["rules"]),
         ("function count", len(doc["meta"]["analysis"]["feature_counts"]["functions"])),
+        ("library function count", len(doc["meta"]["analysis"]["library_functions"])),
         (
             "total feature count",
             doc["meta"]["analysis"]["feature_counts"]["file"]
@@ -119,3 +127,8 @@ def render_verbose(doc):
     ostream.write("\n")
 
     return ostream.getvalue()
+
+
+def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
+    doc = capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities)
+    return render_verbose(doc)

capa/render/vverbose.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,13 +6,15 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
-import collections
-
 import tabulate
 
 import capa.rules
 import capa.render.utils as rutils
 import capa.render.verbose
+import capa.features.common
+import capa.render.result_document
+from capa.rules import RuleSet
+from capa.engine import MatchResults
 
 
 def render_locations(ostream, match):
@@ -57,7 +59,7 @@ def render_statement(ostream, match, statement, indent=0):
 
         if child[child["type"]]:
             if child["type"] == "string":
-                value = '"%s"' % capa.features.escape_string(child[child["type"]])
+                value = '"%s"' % capa.features.common.escape_string(child[child["type"]])
             else:
                 value = child[child["type"]]
             value = rutils.bold2(value)
@@ -85,30 +87,51 @@ def render_statement(ostream, match, statement, indent=0):
         raise RuntimeError("unexpected match statement type: " + str(statement))
 
 
+def render_string_value(s):
+    return '"%s"' % capa.features.common.escape_string(s)
+
+
 def render_feature(ostream, match, feature, indent=0):
     ostream.write("  " * indent)
 
     key = feature["type"]
     value = feature[feature["type"]]
-    if key == "regex":
-        key = "string"  # render string for regex to mirror the rule source
-        value = feature["match"]  # the match provides more information than the value for regex
 
-    if key == "string":
-        value = '"%s"' % capa.features.escape_string(value)
+    if key not in ("regex", "substring"):
+        # like:
+        #   number: 10 = SOME_CONSTANT @ 0x401000
+        if key == "string":
+            value = render_string_value(value)
 
-    ostream.write(key)
-    ostream.write(": ")
+        ostream.write(key)
+        ostream.write(": ")
 
-    if value:
-        ostream.write(rutils.bold2(value))
+        if value:
+            ostream.write(rutils.bold2(value))
 
-        if "description" in feature:
-            ostream.write(capa.rules.DESCRIPTION_SEPARATOR)
-            ostream.write(feature["description"])
+            if "description" in feature:
+                ostream.write(capa.rules.DESCRIPTION_SEPARATOR)
+                ostream.write(feature["description"])
 
-    render_locations(ostream, match)
-    ostream.write("\n")
+        if key not in ("os", "arch"):
+            render_locations(ostream, match)
+        ostream.write("\n")
+    else:
+        # like:
+        #  regex: /blah/ = SOME_CONSTANT
+        #    - "foo blah baz" @ 0x401000
+        #    - "aaa blah bbb" @ 0x402000, 0x403400
+        ostream.write(key)
+        ostream.write(": ")
+        ostream.write(value)
+        ostream.write("\n")
+
+        for match, locations in sorted(feature["matches"].items(), key=lambda p: p[0]):
+            ostream.write("  " * (indent + 1))
+            ostream.write("- ")
+            ostream.write(rutils.bold2(render_string_value(match)))
+            render_locations(ostream, {"locations": locations})
+            ostream.write("\n")
 
 
 def render_node(ostream, match, node, indent=0):
@@ -170,7 +193,7 @@ def render_rules(ostream, doc):
         ## rules
         check for OutputDebugString error
         namespace  anti-analysis/anti-debugging/debugger-detection
-        author     michael.hunhoff@fireeye.com
+        author     michael.hunhoff@mandiant.com
         scope      function
         mbc        Anti-Behavioral Analysis::Detect Debugger::OutputDebugString
         examples   Practical Malware Analysis Lab 16-02.exe_:0x401020
@@ -180,6 +203,11 @@ def render_rules(ostream, doc):
             api: kernel32.GetLastError @ 0x10004A87
             api: kernel32.OutputDebugString @ 0x10004767, 0x10004787, 0x10004816, 0x10004895
     """
+    functions_by_bb = {}
+    for function, info in doc["meta"]["analysis"]["layout"]["functions"].items():
+        for bb in info["matched_basic_blocks"]:
+            functions_by_bb[bb] = function
+
     had_match = False
     for rule in rutils.capability_rules(doc):
         count = len(rule["matches"])
@@ -197,6 +225,12 @@ def render_rules(ostream, doc):
                 continue
 
             v = rule["meta"][key]
+            if not v:
+                continue
+
+            if key in ("att&ck", "mbc"):
+                v = [rutils.format_parts_id(vv) for vv in v]
+
             if isinstance(v, list) and len(v) == 1:
                 v = v[0]
             elif isinstance(v, list) and len(v) > 1:
@@ -218,7 +252,12 @@ def render_rules(ostream, doc):
             for location, match in sorted(doc["rules"][rule["meta"]["name"]]["matches"].items()):
                 ostream.write(rule["meta"]["scope"])
                 ostream.write(" @ ")
-                ostream.writeln(rutils.hex(location))
+                ostream.write(rutils.hex(location))
+
+                if rule["meta"]["scope"] == capa.rules.BASIC_BLOCK_SCOPE:
+                    ostream.write(" in function " + rutils.hex(functions_by_bb[location]))
+
+                ostream.write("\n")
                 render_match(ostream, match, indent=1)
         ostream.write("\n")
 
@@ -236,3 +275,8 @@ def render_vverbose(doc):
     ostream.write("\n")
 
     return ostream.getvalue()
+
+
+def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
+    doc = capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities)
+    return render_vverbose(doc)

capa/rules.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -6,29 +6,40 @@
 #  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and limitations under the License.
 
+import io
 import re
 import uuid
 import codecs
 import logging
 import binascii
 import functools
+import collections
+from enum import Enum
+
+from capa.helpers import assert_never
 
 try:
     from functools import lru_cache
 except ImportError:
-    from backports.functools_lru_cache import lru_cache
+    # need to type ignore this due to mypy bug here (duplicate name):
+    # https://github.com/python/mypy/issues/1153
+    from backports.functools_lru_cache import lru_cache  # type: ignore
+
+from typing import Any, Set, Dict, List, Tuple, Union, Iterator
 
-import six
 import yaml
 import ruamel.yaml
 
-import capa.engine
+import capa.perf
+import capa.engine as ceng
 import capa.features
+import capa.optimizer
 import capa.features.file
 import capa.features.insn
+import capa.features.common
 import capa.features.basicblock
-from capa.engine import *
-from capa.features import MAX_BYTES_FEATURE_SIZE
+from capa.engine import Statement, FeatureSet
+from capa.features.common import MAX_BYTES_FEATURE_SIZE, Feature
 
 logger = logging.getLogger(__name__)
 
@@ -40,6 +51,7 @@ META_KEYS = (
     "rule-category",
     "maec/analysis-conclusion",
     "maec/analysis-conclusion-ov",
+    "maec/malware-family",
     "maec/malware-category",
     "maec/malware-category-ov",
     "author",
@@ -58,44 +70,58 @@ META_KEYS = (
 HIDDEN_META_KEYS = ("capa/nursery", "capa/path")
 
 
-FILE_SCOPE = "file"
-FUNCTION_SCOPE = "function"
-BASIC_BLOCK_SCOPE = "basic block"
+class Scope(str, Enum):
+    FILE = "file"
+    FUNCTION = "function"
+    BASIC_BLOCK = "basic block"
+
+
+FILE_SCOPE = Scope.FILE.value
+FUNCTION_SCOPE = Scope.FUNCTION.value
+BASIC_BLOCK_SCOPE = Scope.BASIC_BLOCK.value
 
 
 SUPPORTED_FEATURES = {
     FILE_SCOPE: {
-        capa.features.MatchedRule,
+        capa.features.common.MatchedRule,
         capa.features.file.Export,
         capa.features.file.Import,
         capa.features.file.Section,
-        capa.features.Characteristic("embedded pe"),
-        capa.features.String,
+        capa.features.file.FunctionName,
+        capa.features.common.Characteristic("embedded pe"),
+        capa.features.common.String,
+        capa.features.common.Format,
+        capa.features.common.OS,
+        capa.features.common.Arch,
     },
     FUNCTION_SCOPE: {
         # plus basic block scope features, see below
         capa.features.basicblock.BasicBlock,
-        capa.features.Characteristic("calls from"),
-        capa.features.Characteristic("calls to"),
-        capa.features.Characteristic("loop"),
-        capa.features.Characteristic("recursive call"),
+        capa.features.common.Characteristic("calls from"),
+        capa.features.common.Characteristic("calls to"),
+        capa.features.common.Characteristic("loop"),
+        capa.features.common.Characteristic("recursive call"),
+        capa.features.common.OS,
+        capa.features.common.Arch,
     },
     BASIC_BLOCK_SCOPE: {
-        capa.features.MatchedRule,
+        capa.features.common.MatchedRule,
         capa.features.insn.API,
         capa.features.insn.Number,
-        capa.features.String,
-        capa.features.Bytes,
+        capa.features.common.String,
+        capa.features.common.Bytes,
         capa.features.insn.Offset,
         capa.features.insn.Mnemonic,
-        capa.features.Characteristic("nzxor"),
-        capa.features.Characteristic("peb access"),
-        capa.features.Characteristic("fs access"),
-        capa.features.Characteristic("gs access"),
-        capa.features.Characteristic("cross section flow"),
-        capa.features.Characteristic("tight loop"),
-        capa.features.Characteristic("stack string"),
-        capa.features.Characteristic("indirect call"),
+        capa.features.common.Characteristic("nzxor"),
+        capa.features.common.Characteristic("peb access"),
+        capa.features.common.Characteristic("fs access"),
+        capa.features.common.Characteristic("gs access"),
+        capa.features.common.Characteristic("cross section flow"),
+        capa.features.common.Characteristic("tight loop"),
+        capa.features.common.Characteristic("stack string"),
+        capa.features.common.Characteristic("indirect call"),
+        capa.features.common.OS,
+        capa.features.common.Arch,
     },
 }
 
@@ -138,22 +164,32 @@ class InvalidRuleSet(ValueError):
         return str(self)
 
 
-def ensure_feature_valid_for_scope(scope, feature):
-    if isinstance(feature, capa.features.Characteristic):
-        if capa.features.Characteristic(feature.value) not in SUPPORTED_FEATURES[scope]:
-            raise InvalidRule("feature %s not support for scope %s" % (feature, scope))
-    elif not isinstance(feature, tuple(filter(lambda t: isinstance(t, type), SUPPORTED_FEATURES[scope]))):
-        raise InvalidRule("feature %s not support for scope %s" % (feature, scope))
+def ensure_feature_valid_for_scope(scope: str, feature: Union[Feature, Statement]):
+    # if the given feature is a characteristic,
+    # check that is a valid characteristic for the given scope.
+    if (
+        isinstance(feature, capa.features.common.Characteristic)
+        and isinstance(feature.value, str)
+        and capa.features.common.Characteristic(feature.value) not in SUPPORTED_FEATURES[scope]
+    ):
+        raise InvalidRule("feature %s not supported for scope %s" % (feature, scope))
+
+    if not isinstance(feature, capa.features.common.Characteristic):
+        # features of this scope that are not Characteristics will be Type instances.
+        # check that the given feature is one of these types.
+        types_for_scope = filter(lambda t: isinstance(t, type), SUPPORTED_FEATURES[scope])
+        if not isinstance(feature, tuple(types_for_scope)):  # type: ignore
+            raise InvalidRule("feature %s not supported for scope %s" % (feature, scope))
 
 
-def parse_int(s):
+def parse_int(s: str) -> int:
     if s.startswith("0x"):
         return int(s, 0x10)
     else:
         return int(s, 10)
 
 
-def parse_range(s):
+def parse_range(s: str):
     """
     parse a string "(0, 1)" into a range (min, max).
     min and/or max may by None to indicate an unbound range.
@@ -166,23 +202,21 @@ def parse_range(s):
         raise InvalidRule("invalid range: %s" % (s))
 
     s = s[len("(") : -len(")")]
-    min, _, max = s.partition(",")
-    min = min.strip()
-    max = max.strip()
+    min_spec, _, max_spec = s.partition(",")
+    min_spec = min_spec.strip()
+    max_spec = max_spec.strip()
 
-    if min:
-        min = parse_int(min.strip())
+    min = None
+    if min_spec:
+        min = parse_int(min_spec)
         if min < 0:
             raise InvalidRule("range min less than zero")
-    else:
-        min = None
 
-    if max:
-        max = parse_int(max.strip())
+    max = None
+    if max_spec:
+        max = parse_int(max_spec)
         if max < 0:
             raise InvalidRule("range max less than zero")
-    else:
-        max = None
 
     if min is not None and max is not None:
         if max < min:
@@ -191,36 +225,38 @@ def parse_range(s):
     return min, max
 
 
-def parse_feature(key):
+def parse_feature(key: str):
     # keep this in sync with supported features
     if key == "api":
         return capa.features.insn.API
     elif key == "string":
-        return capa.features.StringFactory
+        return capa.features.common.StringFactory
+    elif key == "substring":
+        return capa.features.common.Substring
     elif key == "bytes":
-        return capa.features.Bytes
+        return capa.features.common.Bytes
     elif key == "number":
         return capa.features.insn.Number
     elif key.startswith("number/"):
-        arch = key.partition("/")[2]
+        bitness = key.partition("/")[2]
         # the other handlers here return constructors for features,
         # and we want to as well,
-        # however, we need to preconfigure one of the arguments (`arch`).
+        # however, we need to preconfigure one of the arguments (`bitness`).
         # so, instead we return a partially-applied function that
-        #  provides `arch` to the feature constructor.
+        #  provides `bitness` to the feature constructor.
         # it forwards any other arguments provided to the closure along to the constructor.
-        return functools.partial(capa.features.insn.Number, arch=arch)
+        return functools.partial(capa.features.insn.Number, bitness=bitness)
     elif key == "offset":
         return capa.features.insn.Offset
     elif key.startswith("offset/"):
-        arch = key.partition("/")[2]
-        return functools.partial(capa.features.insn.Offset, arch=arch)
+        bitness = key.partition("/")[2]
+        return functools.partial(capa.features.insn.Offset, bitness=bitness)
     elif key == "mnemonic":
         return capa.features.insn.Mnemonic
     elif key == "basic blocks":
         return capa.features.basicblock.BasicBlock
     elif key == "characteristic":
-        return capa.features.Characteristic
+        return capa.features.common.Characteristic
     elif key == "export":
         return capa.features.file.Export
     elif key == "import":
@@ -228,7 +264,16 @@ def parse_feature(key):
     elif key == "section":
         return capa.features.file.Section
     elif key == "match":
-        return capa.features.MatchedRule
+        return capa.features.common.MatchedRule
+    elif key == "function-name":
+        return capa.features.file.FunctionName
+    elif key == "os":
+        return capa.features.common.OS
+    elif key == "format":
+        return capa.features.common.Format
+    elif key == "arch":
+
+        return capa.features.common.Arch
     else:
         raise InvalidRule("unexpected statement: %s" % key)
 
@@ -240,39 +285,70 @@ def parse_feature(key):
 DESCRIPTION_SEPARATOR = " = "
 
 
-def parse_description(s, value_type, description=None):
-    """
-    s can be an int or a string
-    """
-    if value_type != "string" and isinstance(s, six.string_types) and DESCRIPTION_SEPARATOR in s:
-        if description:
-            raise InvalidRule(
-                'unexpected value: "%s", only one description allowed (inline description with `%s`)'
-                % (s, DESCRIPTION_SEPARATOR)
-            )
-        value, _, description = s.partition(DESCRIPTION_SEPARATOR)
-        if description == "":
-            raise InvalidRule('unexpected value: "%s", description cannot be empty' % s)
-    else:
+def parse_bytes(s: str) -> bytes:
+    try:
+        b = codecs.decode(s.replace(" ", "").encode("ascii"), "hex")
+    except binascii.Error:
+        raise InvalidRule('unexpected bytes value: must be a valid hex sequence: "%s"' % s)
+
+    if len(b) > MAX_BYTES_FEATURE_SIZE:
+        raise InvalidRule(
+            "unexpected bytes value: byte sequences must be no larger than %s bytes" % MAX_BYTES_FEATURE_SIZE
+        )
+
+    return b
+
+
+def parse_description(s: Union[str, int, bytes], value_type: str, description=None):
+    if value_type == "string":
+        # string features cannot have inline descriptions,
+        # so we assume the entire value is the string,
+        # like: `string: foo = bar` -> "foo = bar"
         value = s
+    else:
+        # other features can have inline descriptions, like `number: 10 = CONST_FOO`.
+        # in this case, the RHS will be like `10 = CONST_FOO` or some other string
+        if isinstance(s, str):
+            if DESCRIPTION_SEPARATOR in s:
+                if description:
+                    # there is already a description passed in as a sub node, like:
+                    #
+                    #     - number: 10 = CONST_FOO
+                    #       description: CONST_FOO
+                    raise InvalidRule(
+                        'unexpected value: "%s", only one description allowed (inline description with `%s`)'
+                        % (s, DESCRIPTION_SEPARATOR)
+                    )
 
-    if isinstance(value, six.string_types):
-        if value_type == "bytes":
-            try:
-                value = codecs.decode(value.replace(" ", ""), "hex")
-            # TODO: Remove TypeError when Python2 is not used anymore
-            except (TypeError, binascii.Error):
-                raise InvalidRule('unexpected bytes value: "%s", must be a valid hex sequence' % value)
+                value, _, description = s.partition(DESCRIPTION_SEPARATOR)
+                if description == "":
+                    # sanity check:
+                    # there is an empty description, like `number: 10 =`
+                    raise InvalidRule('unexpected value: "%s", description cannot be empty' % s)
+            else:
+                # this is a string, but there is no description,
+                # like: `api: CreateFileA`
+                value = s
 
-            if len(value) > MAX_BYTES_FEATURE_SIZE:
-                raise InvalidRule(
-                    "unexpected bytes value: byte sequences must be no larger than %s bytes" % MAX_BYTES_FEATURE_SIZE
-                )
-        elif value_type in ("number", "offset") or value_type.startswith(("number/", "offset/")):
-            try:
-                value = parse_int(value)
-            except ValueError:
-                raise InvalidRule('unexpected value: "%s", must begin with numerical value' % value)
+            # cast from the received string value to the appropriate type.
+            #
+            # without a description, this type would already be correct,
+            # but since we parsed the description from a string,
+            # we need to convert the value to the expected type.
+            #
+            # for example, from `number: 10 = CONST_FOO` we have
+            # the string "10" that needs to become the number 10.
+            if value_type == "bytes":
+                value = parse_bytes(value)
+            elif value_type in ("number", "offset") or value_type.startswith(("number/", "offset/")):
+                try:
+                    value = parse_int(value)
+                except ValueError:
+                    raise InvalidRule('unexpected value: "%s", must begin with numerical value' % value)
+
+        else:
+            # the value might be a number, like: `number: 10`
+            value = s
 
     return value, description
 
@@ -312,28 +388,28 @@ def pop_statement_description_entry(d):
     return description["description"]
 
 
-def build_statements(d, scope):
+def build_statements(d, scope: str):
     if len(d.keys()) > 2:
         raise InvalidRule("too many statements")
 
     key = list(d.keys())[0]
     description = pop_statement_description_entry(d[key])
     if key == "and":
-        return And([build_statements(dd, scope) for dd in d[key]], description=description)
+        return ceng.And([build_statements(dd, scope) for dd in d[key]], description=description)
     elif key == "or":
-        return Or([build_statements(dd, scope) for dd in d[key]], description=description)
+        return ceng.Or([build_statements(dd, scope) for dd in d[key]], description=description)
     elif key == "not":
         if len(d[key]) != 1:
             raise InvalidRule("not statement must have exactly one child statement")
-        return Not(build_statements(d[key][0], scope), description=description)
+        return ceng.Not(build_statements(d[key][0], scope), description=description)
     elif key.endswith(" or more"):
         count = int(key[: -len("or more")])
-        return Some(count, [build_statements(dd, scope) for dd in d[key]], description=description)
+        return ceng.Some(count, [build_statements(dd, scope) for dd in d[key]], description=description)
     elif key == "optional":
         # `optional` is an alias for `0 or more`
         # which is useful for documenting behaviors,
         # like with `write file`, we might say that `WriteFile` is optionally found alongside `CreateFileA`.
-        return Some(0, [build_statements(dd, scope) for dd in d[key]], description=description)
+        return ceng.Some(0, [build_statements(dd, scope) for dd in d[key]], description=description)
 
     elif key == "function":
         if scope != FILE_SCOPE:
@@ -342,7 +418,7 @@ def build_statements(d, scope):
         if len(d[key]) != 1:
             raise InvalidRule("subscope must have exactly one child statement")
 
-        return Subscope(FUNCTION_SCOPE, build_statements(d[key][0], FUNCTION_SCOPE))
+        return ceng.Subscope(FUNCTION_SCOPE, build_statements(d[key][0], FUNCTION_SCOPE))
 
     elif key == "basic block":
         if scope != FUNCTION_SCOPE:
@@ -351,7 +427,7 @@ def build_statements(d, scope):
         if len(d[key]) != 1:
             raise InvalidRule("subscope must have exactly one child statement")
 
-        return Subscope(BASIC_BLOCK_SCOPE, build_statements(d[key][0], BASIC_BLOCK_SCOPE))
+        return ceng.Subscope(BASIC_BLOCK_SCOPE, build_statements(d[key][0], BASIC_BLOCK_SCOPE))
 
     elif key.startswith("count(") and key.endswith(")"):
         # e.g.:
@@ -392,22 +468,28 @@ def build_statements(d, scope):
 
         count = d[key]
         if isinstance(count, int):
-            return Range(feature, min=count, max=count, description=description)
+            return ceng.Range(feature, min=count, max=count, description=description)
         elif count.endswith(" or more"):
             min = parse_int(count[: -len(" or more")])
             max = None
-            return Range(feature, min=min, max=max, description=description)
+            return ceng.Range(feature, min=min, max=max, description=description)
         elif count.endswith(" or fewer"):
             min = None
             max = parse_int(count[: -len(" or fewer")])
-            return Range(feature, min=min, max=max, description=description)
+            return ceng.Range(feature, min=min, max=max, description=description)
         elif count.startswith("("):
             min, max = parse_range(count)
-            return Range(feature, min=min, max=max, description=description)
+            return ceng.Range(feature, min=min, max=max, description=description)
         else:
             raise InvalidRule("unexpected range: %s" % (count))
-    elif key == "string" and not isinstance(d[key], six.string_types):
+    elif key == "string" and not isinstance(d[key], str):
         raise InvalidRule("ambiguous string value %s, must be defined as explicit string" % d[key])
+    elif (
+        (key == "os" and d[key] not in capa.features.common.VALID_OS)
+        or (key == "format" and d[key] not in capa.features.common.VALID_FORMAT)
+        or (key == "arch" and d[key] not in capa.features.common.VALID_ARCH)
+    ):
+        raise InvalidRule("unexpected %s value %s" % (key, d[key]))
     else:
         Feature = parse_feature(key)
         value, description = parse_description(d[key], key, d.get("description"))
@@ -419,16 +501,16 @@ def build_statements(d, scope):
         return feature
 
 
-def first(s):
+def first(s: List[Any]) -> Any:
     return s[0]
 
 
-def second(s):
+def second(s: List[Any]) -> Any:
     return s[1]
 
 
-class Rule(object):
-    def __init__(self, name, scope, statement, meta, definition=""):
+class Rule:
+    def __init__(self, name: str, scope: str, statement: Statement, meta, definition=""):
         super(Rule, self).__init__()
         self.name = name
         self.scope = scope
@@ -458,7 +540,7 @@ class Rule(object):
         deps = set([])
 
         def rec(statement):
-            if isinstance(statement, capa.features.MatchedRule):
+            if isinstance(statement, capa.features.common.MatchedRule):
                 # we're not sure at this point if the `statement.value` is
                 #  really a rule name or a namespace name (we use `MatchedRule` for both cases).
                 # we'll give precedence to namespaces, and then assume if that does work,
@@ -474,7 +556,7 @@ class Rule(object):
                     # not a namespace, assume its a rule name.
                     deps.add(statement.value)
 
-            elif isinstance(statement, Statement):
+            elif isinstance(statement, ceng.Statement):
                 for child in statement.get_children():
                     rec(child)
 
@@ -485,11 +567,9 @@ class Rule(object):
         return deps
 
     def _extract_subscope_rules_rec(self, statement):
-        if isinstance(statement, Statement):
+        if isinstance(statement, ceng.Statement):
             # for each child that is a subscope,
-            for subscope in filter(
-                lambda statement: isinstance(statement, capa.engine.Subscope), statement.get_children()
-            ):
+            for subscope in filter(lambda statement: isinstance(statement, ceng.Subscope), statement.get_children()):
 
                 # create a new rule from it.
                 # the name is a randomly generated, hopefully unique value.
@@ -514,7 +594,7 @@ class Rule(object):
                 )
 
                 # update the existing statement to `match` the new rule
-                new_node = capa.features.MatchedRule(name)
+                new_node = capa.features.common.MatchedRule(name)
                 statement.replace_child(subscope, new_node)
 
                 # and yield the new rule to our caller
@@ -551,15 +631,18 @@ class Rule(object):
         for new_rule in self._extract_subscope_rules_rec(self.statement):
             yield new_rule
 
-    def evaluate(self, features):
-        return self.statement.evaluate(features)
+    def evaluate(self, features: FeatureSet, short_circuit=True):
+        capa.perf.counters["evaluate.feature"] += 1
+        capa.perf.counters["evaluate.feature.rule"] += 1
+        return self.statement.evaluate(features, short_circuit=short_circuit)
 
     @classmethod
     def from_dict(cls, d, definition):
-        name = d["rule"]["meta"]["name"]
+        meta = d["rule"]["meta"]
+        name = meta["name"]
         # if scope is not specified, default to function scope.
         # this is probably the mode that rule authors will start with.
-        scope = d["rule"]["meta"].get("scope", FUNCTION_SCOPE)
+        scope = meta.get("scope", FUNCTION_SCOPE)
         statements = d["rule"]["features"]
 
         # the rule must start with a single logic node.
@@ -567,13 +650,19 @@ class Rule(object):
         if len(statements) != 1:
             raise InvalidRule("rule must begin with a single top level statement")
 
-        if isinstance(statements[0], capa.engine.Subscope):
+        if isinstance(statements[0], ceng.Subscope):
             raise InvalidRule("top level statement may not be a subscope")
 
         if scope not in SUPPORTED_FEATURES.keys():
             raise InvalidRule("{:s} is not a supported scope".format(scope))
 
-        return cls(name, scope, build_statements(statements[0], scope), d["rule"]["meta"], definition)
+        meta = d["rule"]["meta"]
+        if not isinstance(meta.get("att&ck", []), list):
+            raise InvalidRule("ATT&CK mapping must be a list")
+        if not isinstance(meta.get("mbc", []), list):
+            raise InvalidRule("MBC mapping must be a list")
+
+        return cls(name, scope, build_statements(statements[0], scope), meta, definition)
 
     @staticmethod
     @lru_cache()
@@ -699,7 +788,7 @@ class Rule(object):
         for key in hidden_meta.keys():
             del meta[key]
 
-        ostream = six.BytesIO()
+        ostream = io.BytesIO()
         self._get_ruamel_yaml_parser().dump(definition, ostream)
 
         for key, value in hidden_meta.items():
@@ -741,50 +830,37 @@ class Rule(object):
         return doc
 
 
-def get_rules_with_scope(rules, scope):
+def get_rules_with_scope(rules, scope) -> List[Rule]:
     """
     from the given collection of rules, select those with the given scope.
-
-    args:
-      rules (List[capa.rules.Rule]):
-      scope (str): one of the capa.rules.*_SCOPE constants.
-
-    returns:
-      List[capa.rules.Rule]:
+    `scope` is one of the capa.rules.*_SCOPE constants.
     """
     return list(rule for rule in rules if rule.scope == scope)
 
 
-def get_rules_and_dependencies(rules, rule_name):
+def get_rules_and_dependencies(rules: List[Rule], rule_name: str) -> Iterator[Rule]:
     """
     from the given collection of rules, select a rule and its dependencies (transitively).
-
-    args:
-      rules (List[Rule]):
-      rule_name (str):
-
-    yields:
-      Rule:
     """
     # we evaluate `rules` multiple times, so if its a generator, realize it into a list.
     rules = list(rules)
     namespaces = index_rules_by_namespace(rules)
-    rules = {rule.name: rule for rule in rules}
+    rules_by_name = {rule.name: rule for rule in rules}
     wanted = set([rule_name])
 
     def rec(rule):
         wanted.add(rule.name)
         for dep in rule.get_dependencies(namespaces):
-            rec(rules[dep])
+            rec(rules_by_name[dep])
 
-    rec(rules[rule_name])
+    rec(rules_by_name[rule_name])
 
-    for rule in rules.values():
+    for rule in rules_by_name.values():
         if rule.name in wanted:
             yield rule
 
 
-def ensure_rules_are_unique(rules):
+def ensure_rules_are_unique(rules: List[Rule]) -> None:
     seen = set([])
     for rule in rules:
         if rule.name in seen:
@@ -792,7 +868,7 @@ def ensure_rules_are_unique(rules):
         seen.add(rule.name)
 
 
-def ensure_rule_dependencies_are_met(rules):
+def ensure_rule_dependencies_are_met(rules: List[Rule]) -> None:
     """
     raise an exception if a rule dependency does not exist.
 
@@ -802,14 +878,14 @@ def ensure_rule_dependencies_are_met(rules):
     # we evaluate `rules` multiple times, so if its a generator, realize it into a list.
     rules = list(rules)
     namespaces = index_rules_by_namespace(rules)
-    rules = {rule.name: rule for rule in rules}
-    for rule in rules.values():
+    rules_by_name = {rule.name: rule for rule in rules}
+    for rule in rules_by_name.values():
         for dep in rule.get_dependencies(namespaces):
-            if dep not in rules:
+            if dep not in rules_by_name:
                 raise InvalidRule('rule "%s" depends on missing rule "%s"' % (rule.name, dep))
 
 
-def index_rules_by_namespace(rules):
+def index_rules_by_namespace(rules: List[Rule]) -> Dict[str, List[Rule]]:
     """
     compute the rules that fit into each namespace found within the given rules.
 
@@ -823,11 +899,6 @@ def index_rules_by_namespace(rules):
       c2/shell: [create reverse shell]
       c2/file-transfer: [download and write a file]
       c2: [create reverse shell, download and write a file]
-
-    Args:
-      rules (List[Rule]):
-
-    Returns: Dict[str, List[Rule]]
     """
     namespaces = collections.defaultdict(list)
 
@@ -843,7 +914,38 @@ def index_rules_by_namespace(rules):
     return dict(namespaces)
 
 
-class RuleSet(object):
+def topologically_order_rules(rules: List[Rule]) -> List[Rule]:
+    """
+    order the given rules such that dependencies show up before dependents.
+    this means that as we match rules, we can add features for the matches, and these
+     will be matched by subsequent rules if they follow this order.
+
+    assumes that the rule dependency graph is a DAG.
+    """
+    # we evaluate `rules` multiple times, so if its a generator, realize it into a list.
+    rules = list(rules)
+    namespaces = index_rules_by_namespace(rules)
+    rules_by_name = {rule.name: rule for rule in rules}
+    seen = set([])
+    ret = []
+
+    def rec(rule):
+        if rule.name in seen:
+            return
+
+        for dep in rule.get_dependencies(namespaces):
+            rec(rules_by_name[dep])
+
+        ret.append(rule)
+        seen.add(rule.name)
+
+    for rule in rules_by_name.values():
+        rec(rule)
+
+    return ret
+
+
+class RuleSet:
     """
     a ruleset is initialized with a collection of rules, which it verifies and sorts into scopes.
     each set of scoped rules is sorted topologically, which enables rules to match on past rule matches.
@@ -858,7 +960,7 @@ class RuleSet(object):
         capa.engine.match(ruleset.file_rules, ...)
     """
 
-    def __init__(self, rules):
+    def __init__(self, rules: List[Rule]):
         super(RuleSet, self).__init__()
 
         ensure_rules_are_unique(rules)
@@ -870,10 +972,22 @@ class RuleSet(object):
         if len(rules) == 0:
             raise InvalidRuleSet("no rules selected")
 
+        rules = capa.optimizer.optimize_rules(rules)
+
         self.file_rules = self._get_rules_for_scope(rules, FILE_SCOPE)
         self.function_rules = self._get_rules_for_scope(rules, FUNCTION_SCOPE)
         self.basic_block_rules = self._get_rules_for_scope(rules, BASIC_BLOCK_SCOPE)
         self.rules = {rule.name: rule for rule in rules}
+        self.rules_by_namespace = index_rules_by_namespace(rules)
+
+        # unstable
+        (self._easy_file_rules_by_feature, self._hard_file_rules) = self._index_rules_by_feature(self.file_rules)
+        (self._easy_function_rules_by_feature, self._hard_function_rules) = self._index_rules_by_feature(
+            self.function_rules
+        )
+        (self._easy_basic_block_rules_by_feature, self._hard_basic_block_rules) = self._index_rules_by_feature(
+            self.basic_block_rules
+        )
 
     def __len__(self):
         return len(self.rules)
@@ -881,6 +995,144 @@ class RuleSet(object):
     def __getitem__(self, rulename):
         return self.rules[rulename]
 
+    def __contains__(self, rulename):
+        return rulename in self.rules
+
+    @staticmethod
+    def _index_rules_by_feature(rules) -> Tuple[Dict[Feature, Set[str]], List[str]]:
+        """
+        split the given rules into two structures:
+          - "easy rules" are indexed by feature,
+            such that you can quickly find the rules that contain a given feature.
+          - "hard rules" are those that contain substring/regex/bytes features or match statements.
+            these continue to be ordered topologically.
+
+        a rule evaluator can use the "easy rule" index to restrict the
+        candidate rules that might match a given set of features.
+
+        at this time, a rule evaluator can't do anything special with
+        the "hard rules". it must still do a full top-down match of each
+        rule, in topological order.
+        """
+
+        # we'll do a couple phases:
+        #
+        #  1. recursively visit all nodes in all rules,
+        #    a. indexing all features
+        #    b. recording the types of features found per rule
+        #  2. compute the easy and hard rule sets
+        #  3. remove hard rules from the rules-by-feature index
+        #  4. construct the topologically ordered list of hard rules
+        rules_with_easy_features: Set[str] = set()
+        rules_with_hard_features: Set[str] = set()
+        rules_by_feature: Dict[Feature, Set[str]] = collections.defaultdict(set)
+
+        def rec(rule_name: str, node: Union[Feature, Statement]):
+            """
+            walk through a rule's logic tree, indexing the easy and hard rules,
+            and the features referenced by easy rules.
+            """
+            if isinstance(
+                node,
+                (
+                    # these are the "hard features"
+                    # substring: scanning feature
+                    capa.features.common.Substring,
+                    # regex: scanning feature
+                    capa.features.common.Regex,
+                    # bytes: scanning feature
+                    capa.features.common.Bytes,
+                    # match: dependency on another rule,
+                    # which we have to evaluate first,
+                    # and is therefore tricky.
+                    capa.features.common.MatchedRule,
+                ),
+            ):
+                # hard feature: requires scan or match lookup
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, capa.features.common.Feature):
+                # easy feature: hash lookup
+                rules_with_easy_features.add(rule_name)
+                rules_by_feature[node].add(rule_name)
+            elif isinstance(node, (ceng.Not)):
+                # `not:` statements are tricky to deal with.
+                #
+                # first, features found under a `not:` should not be indexed,
+                # because they're not wanted to be found.
+                # second, `not:` can be nested under another `not:`, or two, etc.
+                # third, `not:` at the root or directly under an `or:`
+                # means the rule will match against *anything* not specified there,
+                # which is a difficult set of things to compute and index.
+                #
+                # so, if a rule has a `not:` statement, its hard.
+                # as of writing, this is an uncommon statement, with only 6 instances in 740 rules.
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, (ceng.Some)) and node.count == 0:
+                # `optional:` and `0 or more:` are tricky to deal with.
+                #
+                # when a subtree is optional, it may match, but not matching
+                # doesn't have any impact either.
+                # now, our rule authors *should* not put this under `or:`
+                # and this is checked by the linter,
+                # but this could still happen (e.g. private rule set without linting)
+                # and would be hard to trace down.
+                #
+                # so better to be safe than sorry and consider this a hard case.
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, (ceng.Range)) and node.min == 0:
+                # `count(foo): 0 or more` are tricky to deal with.
+                # because the min is 0,
+                # this subtree *can* match just about any feature
+                # (except the given one)
+                # which is a difficult set of things to compute and index.
+                rules_with_hard_features.add(rule_name)
+            elif isinstance(node, (ceng.Range)):
+                rec(rule_name, node.child)
+            elif isinstance(node, (ceng.And, ceng.Or, ceng.Some)):
+                for child in node.children:
+                    rec(rule_name, child)
+            elif isinstance(node, ceng.Statement):
+                # unhandled type of statement.
+                # this should only happen if a new subtype of `Statement`
+                # has since been added to capa.
+                #
+                # ideally, we'd like to use mypy for exhaustiveness checking
+                # for all the subtypes of `Statement`.
+                # but, as far as i can tell, mypy does not support this type
+                # of checking.
+                #
+                # in a way, this makes some intuitive sense:
+                # the set of subtypes of type A is unbounded,
+                # because any user might come along and create a new subtype B,
+                # so mypy can't reason about this set of types.
+                assert False, f"Unhandled value: {node} ({type(node).__name__})"
+            else:
+                # programming error
+                assert_never(node)
+
+        for rule in rules:
+            rule_name = rule.meta["name"]
+            root = rule.statement
+            rec(rule_name, root)
+
+        # if a rule has a hard feature,
+        # dont consider it easy, and therefore,
+        # don't index any of its features.
+        #
+        # otherwise, its an easy rule, and index its features
+        for rules_with_feature in rules_by_feature.values():
+            rules_with_feature.difference_update(rules_with_hard_features)
+        easy_rules_by_feature = rules_by_feature
+
+        # `rules` is already topologically ordered,
+        # so extract our hard set into the topological ordering.
+        hard_rules = []
+        for rule in rules:
+            if rule.meta["name"] in rules_with_hard_features:
+                hard_rules.append(rule.meta["name"])
+
+        return (easy_rules_by_feature, hard_rules)
+
     @staticmethod
     def _get_rules_for_scope(rules, scope):
         """
@@ -901,7 +1153,7 @@ class RuleSet(object):
                 continue
 
             scope_rules.update(get_rules_and_dependencies(rules, rule.name))
-        return get_rules_with_scope(capa.engine.topologically_order_rules(scope_rules), scope)
+        return get_rules_with_scope(topologically_order_rules(list(scope_rules)), scope)
 
     @staticmethod
     def _extract_subscope_rules(rules):
@@ -925,7 +1177,7 @@ class RuleSet(object):
 
         return done
 
-    def filter_rules_by_meta(self, tag):
+    def filter_rules_by_meta(self, tag: str) -> "RuleSet":
         """
         return new rule set with rules filtered based on all meta field values, adds all dependency rules
         apply tag-based rule filter assuming that all required rules are loaded
@@ -934,12 +1186,75 @@ class RuleSet(object):
         TODO handle circular dependencies?
         TODO support -t=metafield <k>
         """
-        rules = self.rules.values()
+        rules = list(self.rules.values())
         rules_filtered = set([])
         for rule in rules:
             for k, v in rule.meta.items():
-                if isinstance(v, six.string_types) and tag in v:
+                if isinstance(v, str) and tag in v:
                     logger.debug('using rule "%s" and dependencies, found tag in meta.%s: %s', rule.name, k, v)
                     rules_filtered.update(set(capa.rules.get_rules_and_dependencies(rules, rule.name)))
                     break
         return RuleSet(list(rules_filtered))
+
+    def match(self, scope: Scope, features: FeatureSet, va: int) -> Tuple[FeatureSet, ceng.MatchResults]:
+        """
+        match rules from this ruleset at the given scope against the given features.
+
+        this routine should act just like `capa.engine.match`,
+        except that it may be more performant.
+        """
+        easy_rules_by_feature = {}
+        if scope is Scope.FILE:
+            easy_rules_by_feature = self._easy_file_rules_by_feature
+            hard_rule_names = self._hard_file_rules
+        elif scope is Scope.FUNCTION:
+            easy_rules_by_feature = self._easy_function_rules_by_feature
+            hard_rule_names = self._hard_function_rules
+        elif scope is Scope.BASIC_BLOCK:
+            easy_rules_by_feature = self._easy_basic_block_rules_by_feature
+            hard_rule_names = self._hard_basic_block_rules
+        else:
+            assert_never(scope)
+
+        candidate_rule_names = set()
+        for feature in features:
+            easy_rule_names = easy_rules_by_feature.get(feature)
+            if easy_rule_names:
+                candidate_rule_names.update(easy_rule_names)
+
+        # first, match against the set of rules that have at least one
+        # feature shared with our feature set.
+        candidate_rules = [self.rules[name] for name in candidate_rule_names]
+        features2, easy_matches = ceng.match(candidate_rules, features, va)
+
+        # note that we've stored the updated feature set in `features2`.
+        # this contains a superset of the features in `features`;
+        # it contains additional features for any easy rule matches.
+        # we'll pass this feature set to hard rule matching, since one
+        # of those rules might rely on an easy rule match.
+        #
+        # the updated feature set from hard matching will go into `features3`.
+        # this is a superset of `features2` is a superset of `features`.
+        # ultimately, this is what we'll return to the caller.
+        #
+        # in each case, we could have assigned the updated feature set back to `features`,
+        # but this is slightly more explicit how we're tracking the data.
+
+        # now, match against (topologically ordered) list of rules
+        # that we can't really make any guesses about.
+        # these are rules with hard features, like substring/regex/bytes and match statements.
+        hard_rules = [self.rules[name] for name in hard_rule_names]
+        features3, hard_matches = ceng.match(hard_rules, features2, va)
+
+        # note that above, we probably are skipping matching a bunch of
+        # rules that definitely would never hit.
+        # specifically, "easy rules" that don't share any features with
+        # feature set.
+
+        # MatchResults doesn't technically have an .update() method
+        # but a dict does.
+        matches = {}  # type: ignore
+        matches.update(easy_matches)
+        matches.update(hard_matches)
+
+        return (features3, matches)

capa/version.py

@@ -1 +1 @@
-__version__ = "1.6.1"
+__version__ = "3.1.0"

doc/installation.md

@@ -1,8 +1,8 @@
 # Installation
-You can install capa in a few different ways. First, if you simply want to use capa, just download the [standalone binary](https://github.com/fireeye/capa/releases). If you want to use capa as a Python library, you can install the package directly from GitHub using `pip`. If you'd like to contribute patches or features to capa, you can work with a local copy of the source code.
+You can install capa in a few different ways. First, if you simply want to use capa, just download the [standalone binary](https://github.com/mandiant/capa/releases). If you want to use capa as a Python library, you can install the package directly from GitHub using `pip`. If you'd like to contribute patches or features to capa, you can work with a local copy of the source code.
 
 ## Method 1: Standalone installation
-If you simply want to use capa, use the standalone binaries we host on GitHub: https://github.com/fireeye/capa/releases. These binary executable files contain all the source code, Python interpreter, and associated resources needed to make capa run. This means you can run it without any installation! Just invoke the file using your terminal shell to see the help documentation.
+If you simply want to use capa, use the standalone binaries we host on GitHub: https://github.com/mandiant/capa/releases. These binary executable files contain all the source code, Python interpreter, and associated resources needed to make capa run. This means you can run it without any installation! Just invoke the file using your terminal shell to see the help documentation.
 
 We use PyInstaller to create these packages.
 
@@ -26,7 +26,11 @@ To install capa as a Python library use `pip` to fetch the `flare-capa` module.
 
 #### *Note*:
 This method is appropriate for integrating capa in an existing project.
-This technique doesn't pull the default rule set, so you should check it out separately from [capa-rules](https://github.com/fireeye/capa-rules/) and pass the directory to the entrypoint using `-r` or set the rules path in the IDA Pro plugin.
+This technique doesn't pull the default rule set, so you should check it out separately from [capa-rules](https://github.com/mandiant/capa-rules/) and pass the directory to the entrypoint using `-r` or set the rules path in the IDA Pro plugin.
+This technique also doesn't set up the default library identification [signatures](https://github.com/mandiant/capa/tree/master/sigs). You can pass the signature directory using the `-s` argument.
+For example, to run capa with both a rule path and a signature path:
+
+    capa   -r /path/to/capa-rules   -s /path/to/capa-sigs  suspicious.exe
 Alternatively, see Method 3 below.
 
 ### 1. Install capa module
@@ -40,15 +44,16 @@ If you'd like to review and modify the capa source code, you'll need to check it
 
 ### 1. Check out source code
 Next, clone the capa git repository.
-We use submodules to separate [code](https://github.com/fireeye/capa), [rules](https://github.com/fireeye/capa-rules), and [test data](https://github.com/fireeye/capa-testfiles).
+We use submodules to separate [code](https://github.com/mandiant/capa), [rules](https://github.com/mandiant/capa-rules), and [test data](https://github.com/mandiant/capa-testfiles).
 To clone everything use the `--recurse-submodules` option:
-- `$ git clone --recurse-submodules https://github.com/fireeye/capa.git /local/path/to/src` (HTTPS)
-- `$ git clone --recurse-submodules git@github.com:fireeye/capa.git /local/path/to/src` (SSH)
+- CAUTION: The capa testfiles repository contains many malware samples. If you pull down everything using this method, you may want to install to a directory that won't trigger your anti-virus software.
+- `$ git clone --recurse-submodules https://github.com/mandiant/capa.git /local/path/to/src` (HTTPS)
+- `$ git clone --recurse-submodules git@github.com:mandiant/capa.git /local/path/to/src` (SSH)
 
 To only get the source code and our provided rules (common), follow these steps:
 - clone repository
-  - `$ git clone https://github.com/fireeye/capa.git /local/path/to/src` (HTTPS)
-  - `$ git clone git@github.com:fireeye/capa.git /local/path/to/src` (SSH)
+  - `$ git clone https://github.com/mandiant/capa.git /local/path/to/src` (HTTPS)
+  - `$ git clone git@github.com:mandiant/capa.git /local/path/to/src` (SSH)
 - `$ cd /local/path/to/src`
 - `$ git submodule update --init rules`
 
@@ -59,40 +64,57 @@ Use `pip` to install the source code in "editable" mode. This means that Python
 
 You'll find that the `capa.exe` (Windows) or `capa` (Linux/MacOS) executables in your path now invoke the capa binary from this directory.
 
+#### Development
+
+##### venv [optional]
+
+For development, we recommend to use [venv](https://docs.python.org/3/tutorial/venv.html). It allows you to create a virtual environment: a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages. This approach avoids conflicts between the requirements of different applications on your computer. It also ensures that you don't overlook to add a new requirement to `setup.up` using a library already installed on your system.
+
+To create an environment (in the parent directory, to avoid commiting it by accident or messing with the linters), run:
+`$ python3 -m venv ../capa-env`
+
+To activate `capa-env` in Linux or MacOS, run:
+`$ source ../capa-env/bin/activate`
+
+To activate `capa-env` in Windows, run:
+`$ ..\capa-env\Scripts\activate.bat`
+
+For more details about creating and using virtual environments, check out the [venv documentation](https://docs.python.org/3/tutorial/venv.html).
+
+##### Install development dependencies
+
 We use the following tools to ensure consistent code style and formatting:
   - [black](https://github.com/psf/black) code formatter, with `-l 120`
   - [isort 5](https://pypi.org/project/isort/) code formatter, with `--profile black --length-sort --line-width 120`
   - [dos2unix](https://linux.die.net/man/1/dos2unix) for UNIX-style LF newlines
-  - [capafmt](https://github.com/fireeye/capa/blob/master/scripts/capafmt.py) rule formatter
+  - [capafmt](https://github.com/mandiant/capa/blob/master/scripts/capafmt.py) rule formatter
 
 To install these development dependencies, run:
 
 `$ pip install -e /local/path/to/src[dev]`
 
-Note that some development dependencies (including the black code formatter) require Python 3.
-
 To check the code style, formatting and run the tests you can run the script `scripts/ci.sh`.
 You can run it with the argument `no_tests` to skip the tests and only run the code style and formatting: `scripts/ci.sh no_tests`
 
+##### Setup hooks [optional]
+
+If you plan to contribute to capa, you may want to setup the hooks.
+Run `scripts/setup-hooks.sh` to set the following hooks up:
+- The `pre-commit` hook runs checks before every `git commit`.
+  It runs `scripts/ci.sh no_tests` aborting the commit if there are code style or rule linter offenses you need to fix.
+- The `pre-push` hook runs checks before every `git push`.
+  It runs `scripts/ci.sh` aborting the push if there are code style or rule linter offenses or if the tests fail.
+  This way you can ensure everything is alright before sending a pull request.
+
+You can skip the checks by using the `--no-verify` git option.
+
 ### 3. Compile binary using PyInstaller
 We compile capa standalone binaries using PyInstaller. To reproduce the build process check out the source code as described above and follow these steps.
 
 #### Install PyInstaller:
-For Python 2.7: `$ pip install 'pyinstaller==3.*'` (PyInstaller 4 doesn't support Python 2.7)
-
-For Python 3: `$ pip install 'pyinstaller`
+`$ pip install pyinstaller` (Python 3)
 
 #### Run Pyinstaller
 `$ pyinstaller .github/pyinstaller/pyinstaller.spec`
 
 You can find the compiled binary in the created directory `dist/`.
-
-### 4. Setup hooks [optional]
-If you plan to contribute to capa, you may want to setup the hooks.
-Run `scripts/setup-hooks.sh` to set the following hooks up:
-- The `pre-commit` hook runs checks before every `git commit`.
-  It runs `scripts/ci.sh no_tests` aborting the commit if there are code style or rule linter offenses you need to fix.
-  You can skip this check by using the `--no-verify` git option.
-- The `pre-push` hook runs checks before every `git push`.
-  It runs `scripts/ci.sh` aborting the push if there are code style or rule linter offenses or if the tests fail.
-  This way you can ensure everything is alright before sending a pull request.

doc/limitations.md

@@ -46,6 +46,6 @@ We need more practical use cases and test samples to justify the additional work
 
 
 # ATT&CK, MAEC, MBC, and other capability tagging
-capa uses namespaces to group capabilities (see https://github.com/fireeye/capa-rules/tree/master#namespace-organization).
+capa uses namespaces to group capabilities (see https://github.com/mandiant/capa-rules/tree/master#namespace-organization).
 
-The `rule.meta` field also supports `att&ck`, `mbc`, and `maec` fields to associate rules with the respective taxonomy (see https://github.com/fireeye/capa-rules/blob/master/doc/format.md#meta-block).
+The `rule.meta` field also supports `att&ck`, `mbc`, and `maec` fields to associate rules with the respective taxonomy (see https://github.com/mandiant/capa-rules/blob/master/doc/format.md#meta-block).

doc/release.md

@@ -1,22 +1,16 @@
 # Release checklist
 
-- [ ] Ensure all [milestoned issues/PRs](https://github.com/fireeye/capa/milestones) are addressed, or reassign to a new milestone.
-- [ ] Add the `dont merge` label to all PRs that are close to be ready to merge (or merge them if they are ready) in [capa](https://github.com/fireeye/capa/pulls) and [capa-rules](https://github.com/fireeye/capa-rules/pulls).
-- [ ] Ensure the [CI workflow succeeds in master](https://github.com/fireeye/capa/actions/workflows/tests.yml?query=branch%3Amaster).
+- [ ] Ensure all [milestoned issues/PRs](https://github.com/mandiant/capa/milestones) are addressed, or reassign to a new milestone.
+- [ ] Add the `dont merge` label to all PRs that are close to be ready to merge (or merge them if they are ready) in [capa](https://github.com/mandiant/capa/pulls) and [capa-rules](https://github.com/mandiant/capa-rules/pulls).
+- [ ] Ensure the [CI workflow succeeds in master](https://github.com/mandiant/capa/actions/workflows/tests.yml?query=branch%3Amaster).
 - [ ] Ensure that `python scripts/lint.py rules/ --thorough` succeeds (only `missing examples`  offenses are allowed in the nursery).
 - [ ] Review changes
-  - capa https://github.com/fireeye/capa/compare/\<last-release\>...master
-  - capa-rules https://github.com/fireeye/capa-rules/compare/\<last-release>\...master
-- [ ] Update [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md)
+  - capa https://github.com/mandiant/capa/compare/\<last-release\>...master
+  - capa-rules https://github.com/mandiant/capa-rules/compare/\<last-release>\...master
+- [ ] Update [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md)
   - Do not forget to add a nice introduction thanking contributors
   - Remember that we need a major release if we introduce breaking changes
-  - Sections
-    - New Features
-    - New Rules
-    - Bug Fixes
-    - Changes
-    - Development
-    - Raw diffs
+  - Sections: see template below
   - Update `Raw diffs` links
   - Create placeholder for `master (unreleased)` section
     ```
@@ -24,21 +18,26 @@
 
     ### New Features
 
-    ### New Rules
+    ### Breaking Changes
+
+    ### New Rules (0)
+
+    -
 
     ### Bug Fixes
 
-    ### Changes
+    ### capa explorer IDA Pro plugin
 
     ### Development
 
     ### Raw diffs
-    - [capa <release>...master](https://github.com/fireeye/capa/compare/<release>...master)
-    - [capa-rules <release>...master](https://github.com/fireeye/capa-rules/compare/<release>...master)
+    - [capa <release>...master](https://github.com/mandiant/capa/compare/<release>...master)
+    - [capa-rules <release>...master](https://github.com/mandiant/capa-rules/compare/<release>...master)
     ```
-- [ ] Update [capa/version.py](https://github.com/fireeye/capa/blob/master/capa/version.py)
-- [ ] Create a PR with the updated [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md) and [capa/version.py](https://github.com/fireeye/capa/blob/master/capa/version.py). Copy this checklist in the PR description.
-- [ ] After PR review, merge the PR and [create the release in GH](https://github.com/fireeye/capa/releases/new) using text from the [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md).
-- [ ] Verify GH actions [upload artifacts](https://github.com/fireeye/capa/releases), [publish to PyPI](https://pypi.org/project/flare-capa) and [create a tag in capa rules](https://github.com/fireeye/capa-rules/tags) upon completion.
+- [ ] Update [capa/version.py](https://github.com/mandiant/capa/blob/master/capa/version.py)
+- [ ] Create a PR with the updated [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md) and [capa/version.py](https://github.com/mandiant/capa/blob/master/capa/version.py). Copy this checklist in the PR description.
+- [ ] After PR review, merge the PR and [create the release in GH](https://github.com/mandiant/capa/releases/new) using text from the [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md).
+- [ ] Verify GH actions [upload artifacts](https://github.com/mandiant/capa/releases), [publish to PyPI](https://pypi.org/project/flare-capa) and [create a tag in capa rules](https://github.com/mandiant/capa-rules/tags) upon completion.
 - [ ] [Spread the word](https://twitter.com)
+- [ ] Update internal service
 

doc/usage.md

@@ -11,3 +11,9 @@ For example, `capa -t william.ballenthin@mandiant.com` runs rules that reference
 
 ### IDA Pro plugin: capa explorer
 Please check out the [capa explorer documentation](/capa/ida/plugin/README.md).
+
+### save time by reusing .viv files
+Set the environment variable `CAPA_SAVE_WORKSPACE` to instruct the underlying analysis engine to 
+cache its intermediate results to the file system. For example, vivisect will create `.viv` files.
+Subsequently, capa may run faster when reprocessing the same input file.
+This is particularly useful during rule development as you repeatedly test a rule against a known sample.

scripts/bulk-process.py

@@ -47,7 +47,7 @@ usage:
                             parallelism factor
       --no-mp               disable subprocesses
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -55,6 +55,7 @@ Unless required by applicable law or agreed to in writing, software distributed
  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and limitations under the License.
 """
+import os
 import sys
 import json
 import logging
@@ -66,7 +67,7 @@ import multiprocessing.pool
 import capa
 import capa.main
 import capa.rules
-import capa.render
+import capa.render.json
 
 logger = logging.getLogger("capa")
 
@@ -77,6 +78,7 @@ def get_capa_results(args):
 
     args is a tuple, containing:
       rules (capa.rules.RuleSet): the rules to match
+      signatures (List[str]): list of file system paths to signature files
       format (str): the name of the sample file format
       path (str): the file system path to the sample to process
 
@@ -93,10 +95,13 @@ def get_capa_results(args):
       meta (dict): the meta analysis results
       capabilities (dict): the matched capabilities and their result objects
     """
-    rules, format, path = args
+    rules, sigpaths, format, path = args
+    should_save_workspace = os.environ.get("CAPA_SAVE_WORKSPACE") not in ("0", "no", "NO", "n", None)
     logger.info("computing capa results for: %s", path)
     try:
-        extractor = capa.main.get_extractor(path, format, capa.main.BACKEND_VIV, disable_progress=True)
+        extractor = capa.main.get_extractor(
+            path, format, capa.main.BACKEND_VIV, sigpaths, should_save_workspace, disable_progress=True
+        )
     except capa.main.UnsupportedFormatError:
         # i'm 100% sure if multiprocessing will reliably raise exceptions across process boundaries.
         # so instead, return an object with explicit success/failure status.
@@ -121,9 +126,10 @@ def get_capa_results(args):
             "error": "unexpected error: %s" % (e),
         }
 
-    meta = capa.main.collect_metadata("", path, "", format, extractor)
+    meta = capa.main.collect_metadata("", path, "", extractor)
     capabilities, counts = capa.main.find_capabilities(rules, extractor, disable_progress=True)
     meta["analysis"].update(counts)
+    meta["analysis"]["layout"] = capa.main.compute_layout(rules, extractor, capabilities)
 
     return {
         "path": path,
@@ -140,7 +146,7 @@ def main(argv=None):
         argv = sys.argv[1:]
 
         parser = argparse.ArgumentParser(description="detect capabilities in programs.")
-        capa.main.install_common_args(parser, wanted={"rules"})
+        capa.main.install_common_args(parser, wanted={"rules", "signatures"})
         parser.add_argument("input", type=str, help="Path to directory of files to recursively analyze")
         parser.add_argument(
             "-n", "--parallelism", type=int, default=multiprocessing.cpu_count(), help="parallelism factor"
@@ -149,14 +155,6 @@ def main(argv=None):
         args = parser.parse_args(args=argv)
         capa.main.handle_common_args(args)
 
-        if args.rules == "(embedded rules)":
-            logger.info("using default embedded rules")
-            logger.debug("detected running from source")
-            args.rules = os.path.join(os.path.dirname(__file__), "..", "rules")
-            logger.debug("default rule path (source method): %s", args.rules)
-        else:
-            logger.info("using rules path: %s", args.rules)
-
         try:
             rules = capa.main.get_rules(args.rules)
             rules = capa.rules.RuleSet(rules)
@@ -165,6 +163,12 @@ def main(argv=None):
             logger.error("%s", str(e))
             return -1
 
+        try:
+            sig_paths = capa.main.get_signatures(args.signatures)
+        except (IOError) as e:
+            logger.error("%s", str(e))
+            return -1
+
         samples = []
         for (base, directories, files) in os.walk(args.input):
             for file in files:
@@ -196,7 +200,7 @@ def main(argv=None):
 
         results = {}
         for result in mapper(
-            get_capa_results, [(rules, "pe", sample) for sample in samples], parallelism=args.parallelism
+            get_capa_results, [(rules, sig_paths, "pe", sample) for sample in samples], parallelism=args.parallelism
         ):
             if result["status"] == "error":
                 logger.warning(result["error"])
@@ -205,7 +209,7 @@ def main(argv=None):
                 capabilities = result["ok"]["capabilities"]
                 # our renderer expects to emit a json document for a single sample
                 # so we deserialize the json document, store it in a larger dict, and we'll subsequently re-encode.
-                results[result["path"]] = json.loads(capa.render.render_json(meta, rules, capabilities))
+                results[result["path"]] = json.loads(capa.render.json.render(meta, rules, capabilities))
             else:
                 raise ValueError("unexpected status: %s" % (result["status"]))
 

scripts/capa2yara.py

@@ -0,0 +1,760 @@
+"""
+Convert capa rules to YARA rules (where this is possible)
+
+    usage: capa2yara.py [-h] [--private] [--version] [-v] [-vv] [-d] [-q] [--color {auto,always,never}] [-t TAG] rules
+
+Capa to YARA rule converter
+
+positional arguments:
+  rules                 Path to rules
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --private, -p         Create private rules
+  --version             show program's version number and exit
+  -v, --verbose         enable verbose result document (no effect with --json)
+  -vv, --vverbose       enable very verbose result document (no effect with --json)
+  -d, --debug           enable debugging output on STDERR
+  -q, --quiet           disable all output but errors
+  --color {auto,always,never}
+                        enable ANSI color codes in results, default: only during interactive session
+  -t TAG, --tag TAG     filter on rule meta field values
+
+
+Copyright (C) 2020, 2021 Arnim Rupp (@ruppde) and Mandiant, Inc. All Rights Reserved.
+Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+You may obtain a copy of the License at: [package root]/LICENSE.txt
+Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and limitations under the License.
+"""
+
+import re
+import sys
+import string
+import logging
+import argparse
+import datetime
+import itertools
+
+import capa.main
+import capa.rules
+import capa.engine
+import capa.features
+import capa.features.insn
+from capa.features.common import BITNESS_X32, BITNESS_X64, String
+
+logger = logging.getLogger("capa2yara")
+
+today = str(datetime.date.today())
+
+# create unique variable names for each rule in case somebody wants to move/copy stuff around later
+var_names = ["".join(letters) for letters in itertools.product(string.ascii_lowercase, repeat=3)]
+
+
+# this have to be the internal names used by capa.py which are sometimes different to the ones written out in the rules, e.g. "2 or more" is "Some", count is Range
+unsupported = ["characteristic", "mnemonic", "offset", "subscope", "Range"]
+# TODO shorten this list, possible stuff:
+# - 2 or more strings: e.g.
+# -- https://github.com/mandiant/capa-rules/blob/master/collection/file-managers/gather-direct-ftp-information.yml
+# -- https://github.com/mandiant/capa-rules/blob/master/collection/browser/gather-firefox-profile-information.yml
+# - count(string    (1 rule: /executable/subfile/pe/contain-an-embedded-pe-file.yml)
+# - count(match( could be done by creating the referenced rule a 2nd time with the condition, that it hits x times (only 1 rule: ./anti-analysis/anti-disasm/contain-anti-disasm-techniques.yml)
+# - it would be technically possible to get the "basic blocks" working, but the rules contain mostly other non supported statements in there => not worth the effort.
+
+# collect all converted rules to be able to check if we have needed sub rules for match:
+converted_rules = []
+count_incomplete = 0
+
+default_tags = "CAPA "
+
+# minimum number of rounds to do be able to convert rules which depend on referenced rules in several levels of depth
+min_rounds = 5
+
+unsupported_capa_rules = open("unsupported_capa_rules.yml", "wb")
+unsupported_capa_rules_names = open("unsupported_capa_rules.txt", "wb")
+unsupported_capa_rules_list = []
+
+condition_header = """
+    capa_pe_file and
+"""
+
+condition_rule = """
+private rule capa_pe_file : CAPA {
+    meta:
+        description = "match in PE files. used by all further CAPA rules"
+        author = "Arnim Rupp"
+    condition:
+        uint16be(0) == 0x4d5a
+        or uint16be(0) == 0x558b
+        or uint16be(0) == 0x5649
+}
+"""
+
+
+def check_feature(statement, rulename):
+    if statement in unsupported:
+        logger.info("unsupported: " + statement + " in rule: " + rulename)
+        return True
+    else:
+        return False
+
+
+def get_rule_url(path):
+    path = re.sub(r"\.\.\/", "", path)
+    path = re.sub(r"capa-rules\/", "", path)
+    return "https://github.com/mandiant/capa-rules/blob/master/" + path
+
+
+def convert_capa_number_to_yara_bytes(number):
+    if not number.startswith("0x"):
+        print("TODO: fix decimal")
+        sys.exit()
+
+    number = re.sub(r"^0[xX]", "", number)
+    logger.info("number ok: " + repr(number))
+
+    # include spaces every 2 hex
+    bytesv = re.sub(r"(..)", r"\1 ", number)
+
+    # reverse order
+    bytesl = bytesv.split(" ")
+    bytesl.reverse()
+    bytesv = " ".join(bytesl)
+
+    # fix spaces
+    bytesv = bytesv[1:] + " "
+
+    return bytesv
+
+
+def convert_rule_name(rule_name):
+
+    # yara rule names: "Identifiers must follow the same lexical conventions of the C programming language, they can contain any alphanumeric character and the underscore character, but the first character cannot be a digit. Rule identifiers are case sensitive and cannot exceed 128 characters." so we replace any non-alpanum with _
+    rule_name = re.sub(r"\W", "_", rule_name)
+    rule_name = "capa_" + rule_name
+
+    return rule_name
+
+
+def convert_description(statement):
+    try:
+        desc = statement.description
+        if desc:
+            yara_desc = " // " + desc
+            logger.info("using desc: " + repr(yara_desc))
+            return yara_desc
+    except:
+        # no description
+        pass
+
+    return ""
+
+
+def convert_rule(rule, rulename, cround, depth):
+
+    depth += 1
+    logger.info("recursion depth: " + str(depth))
+
+    global var_names
+
+    def do_statement(s_type, kid):
+        yara_strings = ""
+        yara_condition = ""
+        if check_feature(s_type, rulename):
+            return "BREAK", s_type
+        elif s_type == "string":
+            string = kid.value
+            logger.info("doing string: " + repr(string))
+            string = string.replace("\\", "\\\\")
+            string = string.replace("\n", "\\n")
+            string = string.replace("\t", "\\t")
+            var_name = "str_" + var_names.pop(0)
+            yara_strings += "\t$" + var_name + ' = "' + string + '" ascii wide' + convert_description(kid) + "\n"
+            yara_condition += "\t$" + var_name + " "
+        elif s_type == "api" or s_type == "import":
+            # TODO: is it possible in YARA to make a difference between api & import?
+
+            # https://github.com/mandiant/capa-rules/blob/master/doc/format.md#api
+            api = kid.value
+            logger.info("doing api: " + repr(api))
+
+            #    e.g. kernel32.CreateNamedPipe => look for kernel32.dll and CreateNamedPipe
+            if "." in api:
+                dll, api = api.split(".")
+
+                # usage of regex is needed and /i because string search for "CreateMutex" in imports() doesn't look for e.g. CreateMutexA
+                yara_condition += "\tpe.imports(/" + dll + "/i, /" + api + "/) "
+
+            else:
+                # e.g. - api: 'CallNextHookEx'
+                # (from user32.dll)
+
+                # even looking for empty string in dll_regex doesn't work for some files (list below) with pe.imports so do just a string search
+                # yara_condition += '\tpe.imports(/.{0,30}/i, /' + api + '/) '
+                # 5fbbfeed28b258c42e0cfeb16718b31c, 2D3EDC218A90F03089CC01715A9F047F, 7EFF498DE13CC734262F87E6B3EF38AB, C91887D861D9BD4A5872249B641BC9F9, a70052c45e907820187c7e6bcdc7ecca, 0596C4EA5AA8DEF47F22C85D75AACA95
+                var_name = "api_" + var_names.pop(0)
+
+                # limit regex with word boundary \b but also search for appended A and W
+                # TODO: better use something like /(\\x00|\\x01|\\x02|\\x03|\\x04)' + api + '(A|W)?\\x00/  ???
+                yara_strings += "\t$" + var_name + " = /\\b" + api + "(A|W)?\\b/ ascii wide\n"
+                yara_condition += "\t$" + var_name + " "
+
+        elif s_type == "export":
+            export = kid.value
+            logger.info("doing export: " + repr(export))
+
+            yara_condition += '\tpe.exports("' + export + '") '
+
+        elif s_type == "section":
+            # https://github.com/mandiant/capa-rules/blob/master/doc/format.md#section
+            section = kid.value
+            logger.info("doing section: " + repr(section))
+
+            # e.g. - section: .rsrc
+            var_name_sec = var_names.pop(0)
+            # yeah, it would be better to make one loop out of multiple sections but we're in POC-land (and I guess it's not much of a performance hit, loop over short array?)
+            yara_condition += (
+                "\tfor any " + var_name_sec + " in pe.sections : ( " + var_name_sec + '.name == "' + section + '" ) '
+            )
+
+        elif s_type == "match":
+            # https://github.com/mandiant/capa-rules/blob/master/doc/format.md#matching-prior-rule-matches-and-namespaces
+            match = kid.value
+            logger.info("doing match: " + repr(match))
+
+            # e.g. - match: create process
+            #      - match: host-interaction/file-system/write
+            match_rule_name = convert_rule_name(match)
+
+            if match.startswith(rulename + "/"):
+                logger.info("Depending on myself = basic block: " + match)
+                return "BREAK", "Depending on myself = basic block"
+
+            if match_rule_name in converted_rules:
+                yara_condition += "\t" + match_rule_name + "\n"
+            else:
+                # don't complain in the early rounds as there should be 3+ rounds (if all rules are converted)
+                if cround > min_rounds - 2:
+                    logger.info("needed sub-rule not converted (yet, maybe in next round): " + repr(match))
+                    return "BREAK", "needed sub-rule not converted"
+                else:
+                    return "BREAK", "NOLOG"
+
+        elif s_type == "bytes":
+            bytesv = kid.get_value_str()
+            logger.info("doing bytes: " + repr(bytesv))
+            var_name = var_names.pop(0)
+
+            yara_strings += "\t$" + var_name + " = { " + bytesv + " }" + convert_description(kid) + "\n"
+            yara_condition += "\t$" + var_name + " "
+
+        elif s_type == "number":
+            number = kid.get_value_str()
+            logger.info("doing number: " + repr(number))
+
+            if len(number) < 10:
+                logger.info("too short for byte search (until I figure out how to do it properly)" + repr(number))
+                return "BREAK", "Number too short"
+
+            # there's just one rule which contains 0xFFFFFFF but yara gives a warning if if used
+            if number == "0xFFFFFFFF":
+                return "BREAK", "slow byte pattern for YARA search"
+
+            logger.info("number ok: " + repr(number))
+            number = convert_capa_number_to_yara_bytes(number)
+            logger.info("number ok: " + repr(number))
+
+            var_name = "num_" + var_names.pop(0)
+            yara_strings += "\t$" + var_name + " = { " + number + "}" + convert_description(kid) + "\n"
+            yara_condition += "$" + var_name + " "
+
+        elif s_type == "regex":
+            regex = kid.get_value_str()
+            logger.info("doing regex: " + repr(regex))
+
+            # change capas /xxx/i to yaras /xxx/ nocase, count will be used later to decide appending 'nocase'
+            regex, count = re.subn(r"/i$", "/", regex)
+
+            # remove / in the begining and end
+            regex = regex[1:-1]
+
+            # all .* in the regexes of capa look like they should be maximum 100 chars so take 1000 to speed up rules and prevent yara warnings on poor performance
+            regex = regex.replace(".*", ".{,1000}")
+            # strange: capa accepts regexes with unsescaped / like - string: /com/exe4j/runtime/exe4jcontroller/i in capa-rules/compiler/exe4j/compiled-with-exe4j.yml, needs a fix for yara:
+            # would assume that get_value_str() gives the raw string
+            regex = re.sub(r"(?<!\\)/", r"\/", regex)
+
+            # capa uses python regex which accepts /reg(|.exe)/ but yaras regex engine doesn't not => fix it
+            # /reg(|.exe)/ => /reg(.exe)?/
+            regex = re.sub(r"\(\|([^\)]+)\)", r"(\1)?", regex)
+
+            # change begining of line to null byte, e.g. /^open => /\x00open (not word boundary because we're not looking for the begining of a word in a text but usually a function name if there's ^ in a capa rule)
+            regex = re.sub(r"^\^", r"\\x00", regex)
+
+            # regex = re.sub(r"^\^", r"\\b", regex)
+
+            regex = "/" + regex + "/"
+            if count:
+                regex += " nocase"
+
+            # strange: if statement.name == "string", the string is as it is, if statement.name == "regex", the string has // around it, e.g. /regex/
+            var_name = "re_" + var_names.pop(0)
+            yara_strings += "\t" + "$" + var_name + " = " + regex + " ascii wide " + convert_description(kid) + "\n"
+            yara_condition += "\t" + "$" + var_name + " "
+        elif s_type == "Not" or s_type == "And" or s_type == "Or":
+            pass
+        else:
+            logger.info("something unhandled: " + repr(s_type))
+            sys.exit()
+
+        return yara_strings, yara_condition
+
+    ############################## end def do_statement
+
+    yara_strings_list = []
+    yara_condition_list = []
+    rule_comment = ""
+    incomplete = 0
+
+    statement = rule.name
+
+    logger.info("doing statement: " + statement)
+
+    if check_feature(statement, rulename):
+        return "BREAK", statement, rule_comment, incomplete
+
+    if statement == "And" or statement == "Or":
+        desc = convert_description(rule)
+        if desc:
+            logger.info("description of bool statement: " + repr(desc))
+            yara_strings_list.append("\t" * depth + desc + "\n")
+    elif statement == "Not":
+        logger.info("one of those seldom nots: " + rule.name)
+
+    # check for nested statements
+    try:
+        kids = rule.children
+        num_kids = len(kids)
+        logger.info("kids: " + kids)
+    except:
+        logger.info("no kids in rule: " + rule.name)
+
+        try:
+            # maybe it's "Not" = only one child:
+            kid = rule.child
+            kids = [kid]
+            num_kids = 1
+            logger.info("kid: %s", kids)
+        except:
+            logger.info("no kid in rule: %s", rule.name)
+
+    # just a single statement without 'and' or 'or' before it in this rule
+    if "kids" not in locals().keys():
+        logger.info("no kids: " + rule.name)
+
+        yara_strings_sub, yara_condition_sub = do_statement(statement, rule)
+
+        if yara_strings_sub == "BREAK":
+            logger.info("Unknown feature at1: " + rule.name)
+            return "BREAK", yara_condition_sub, rule_comment, incomplete
+        yara_strings_list.append(yara_strings_sub)
+        yara_condition_list.append(yara_condition_sub)
+
+    else:
+        x = 0
+        logger.info("doing kids: %r - len: %s", kids, num_kids)
+        for kid in kids:
+            s_type = kid.name
+            logger.info("doing type: " + s_type + " kidnum: " + str(x))
+
+            if s_type == "Some":
+                cmin = kid.count
+                logger.info("Some type with mininum: " + str(cmin))
+
+                if not cmin:
+                    logger.info("this is optional: which means, we can just ignore it")
+                    x += 1
+                    continue
+                elif statement == "Or":
+                    logger.info("we're inside an OR, we can just ignore it")
+                    x += 1
+                    continue
+                else:
+                    # this is "x or more". could be coded for strings TODO
+                    return "BREAK", "Some aka x or more (TODO)", rule_comment, incomplete
+
+            if s_type == "And" or s_type == "Or" or s_type == "Not" and not kid.name == "Some":
+                logger.info("doing bool with recursion: " + repr(kid))
+                logger.info("kid coming: " + repr(kid.name))
+                # logger.info("grandchildren: " + repr(kid.children))
+
+                ##### here we go into RECURSION  ##################################################################################
+                yara_strings_sub, yara_condition_sub, rule_comment_sub, incomplete_sub = convert_rule(
+                    kid, rulename, cround, depth
+                )
+
+                logger.info("coming out of this recursion, depth: " + repr(depth) + " s_type: " + s_type)
+
+                if yara_strings_sub == "BREAK":
+                    logger.info(
+                        "Unknown feature at2: " + rule.name + " - s_type: " + s_type + " - depth: " + str(depth)
+                    )
+
+                    # luckily this is only a killer, if we're inside an 'And', inside 'Or' we're just missing some coverage
+                    # only accept incomplete rules in rounds > 3 because the reason might be a reference to another rule not converted yet because of missing dependencies
+                    logger.info("rule.name,  depth,  cround: " + rule.name + ", " + str(depth) + ", " + str(cround))
+                    if rule.name == "Or" and depth == 1 and cround > min_rounds - 1:
+                        logger.info(
+                            "Unknown feature, just ignore this branch and keep the rest bec we're in Or (1): "
+                            + s_type
+                            + " - depth: "
+                            + str(depth)
+                        )
+                        # remove last 'or'
+                        # yara_condition = re.sub(r'\sor $', ' ', yara_condition)
+                        rule_comment += "This rule is incomplete because a branch inside an Or-statement had an unsupported feature and was skipped => coverage is reduced compared to the original capa rule. "
+                        x += 1
+                        incomplete = 1
+                        continue
+                    else:
+                        return "BREAK", yara_condition_sub, rule_comment, incomplete
+
+                rule_comment += rule_comment_sub
+                yara_strings_list.append(yara_strings_sub)
+                yara_condition_list.append(yara_condition_sub)
+
+                incomplete = incomplete or incomplete_sub
+
+            yara_strings_sub, yara_condition_sub = do_statement(s_type, kid)
+
+            if yara_strings_sub == "BREAK":
+                logger.info("Unknown feature at3: " + rule.name)
+                logger.info("rule.name,  depth,  cround: " + rule.name + ", " + str(depth) + ", " + str(cround))
+                if rule.name == "Or" and depth == 1 and cround > min_rounds - 1:
+                    logger.info(
+                        "Unknown feature, just ignore this branch and keep the rest bec we're in Or (2): "
+                        + s_type
+                        + " - depth: "
+                        + str(depth)
+                    )
+
+                    rule_comment += "This rule is incomplete because a branch inside an Or-statement had an unsupported feature and was skipped => coverage is reduced compared to the original capa rule. "
+                    x += 1
+                    incomplete = 1
+                    continue
+                else:
+                    return "BREAK", yara_condition_sub, rule_comment, incomplete
+
+            # don't append And or Or if we got no condition back from this kid from e.g. match in myself or unsupported feature inside 'Or'
+            if not yara_condition_sub:
+                continue
+
+            yara_strings_list.append(yara_strings_sub)
+            yara_condition_list.append(yara_condition_sub)
+            x += 1
+
+    # this might happen, if all conditions are inside "or" and none of them was supported
+    if not yara_condition_list:
+        return (
+            "BREAK",
+            'Multiple statements inside "- or:" where all unsupported, the last one was "' + s_type + '"',
+            rule_comment,
+            incomplete,
+        )
+
+    if statement == "And" or statement == "Or":
+        if yara_strings_list:
+            yara_strings = "".join(yara_strings_list)
+        else:
+            yara_strings = ""
+
+        yara_condition = " (\n\t\t" + ("\n\t\t" + statement.lower() + " ").join(yara_condition_list) + " \n\t) "
+
+    elif statement == "Some":
+        cmin = rule.count
+        logger.info("Some type with mininum at2: " + str(cmin))
+
+        if not cmin:
+            logger.info("this is optional: which means, we can just ignore it")
+        else:
+            # this is "x or more". could be coded for strings TODO
+            return "BREAK", "Some aka x or more (TODO)", rule_comment, incomplete
+    elif statement == "Not":
+        logger.info("Not")
+        yara_strings = "".join(yara_strings_list)
+        yara_condition = "not " + "".join(yara_condition_list) + " "
+    else:
+        if len(yara_condition_list) != 1:
+            logger.info("something wrong around here" + repr(yara_condition_list) + " - " + statement)
+            sys.exit()
+
+        # strings might be empty with only conditions
+        if yara_strings_list:
+            yara_strings = "\n\t" + yara_strings_list[0]
+
+        yara_condition = "\n\t" + yara_condition_list[0]
+
+    logger.info(
+        f"################# end of convert_rule() #strings: {len(yara_strings_list)} #conditions: {len(yara_condition_list)}"
+    )
+    logger.info(f"strings: {yara_strings} conditions: {yara_condition}")
+
+    return yara_strings, yara_condition, rule_comment, incomplete
+
+
+def output_yar(yara):
+    print(yara + "\n")
+
+
+def output_unsupported_capa_rules(yaml, capa_rulename, url, reason):
+
+    if reason != "NOLOG":
+        if capa_rulename not in unsupported_capa_rules_list:
+            logger.info("unsupported: " + capa_rulename + " - reason: " + reason + " - url: " + url)
+
+            unsupported_capa_rules_list.append(capa_rulename)
+            unsupported_capa_rules.write(yaml.encode("utf-8") + b"\n")
+            unsupported_capa_rules.write(
+                (
+                    "Reason: "
+                    + reason
+                    + " (there might be multiple unsupported things in this rule, this is the 1st one encountered)"
+                ).encode("utf-8")
+                + b"\n"
+            )
+            unsupported_capa_rules.write(url.encode("utf-8") + b"\n----------------------------------------------\n")
+            unsupported_capa_rules_names.write(capa_rulename.encode("utf-8") + b":")
+            unsupported_capa_rules_names.write(reason.encode("utf-8") + b":")
+            unsupported_capa_rules_names.write(url.encode("utf-8") + b"\n")
+
+
+def convert_rules(rules, namespaces, cround):
+    for rule in rules.rules.values():
+
+        rule_name = convert_rule_name(rule.name)
+
+        if rule.meta.get("capa/subscope-rule", False):
+            logger.info("skipping sub scope rule capa: " + rule.name)
+            continue
+
+        if rule_name in converted_rules:
+            logger.info("skipping already converted rule capa: " + rule.name + " - yara rule: " + rule_name)
+            continue
+
+        logger.info("-------------------------- DOING RULE CAPA: " + rule.name + " - yara rule: " + rule_name)
+        if "capa/path" in rule.meta:
+            url = get_rule_url(rule.meta["capa/path"])
+        else:
+            url = "no url"
+
+        logger.info("URL: " + url)
+        logger.info("statements: " + repr(rule.statement))
+
+        # don't really know what that passed empty string is good for :)
+        dependencies = rule.get_dependencies(namespaces)
+
+        if len(dependencies):
+            logger.info("Dependencies at4: " + rule.name + " - dep: " + str(dependencies))
+
+            for dep in dependencies:
+                logger.info("Dependencies at44: " + dep)
+                if not dep.startswith(rule.name + "/"):
+                    logger.info("Depending on another rule: " + dep)
+                    continue
+
+        yara_strings, yara_condition, rule_comment, incomplete = convert_rule(rule.statement, rule.name, cround, 0)
+
+        if yara_strings == "BREAK":
+            # only give up if in final extra round #9000
+            if cround == 9000:
+                output_unsupported_capa_rules(rule.to_yaml(), rule.name, url, yara_condition)
+            logger.info("Unknown feature at5: " + rule.name)
+        else:
+
+            yara_meta = ""
+            metas = rule.meta
+            rule_tags = ""
+
+            for meta in metas:
+                meta_name = meta
+                # e.g. 'examples:' can be a list
+                seen_hashes = []
+                if isinstance(metas[meta], list):
+                    if meta_name == "examples":
+                        meta_name = "hash"
+                    if meta_name == "att&ck":
+                        meta_name = "attack"
+                        for attack in list(metas[meta]):
+                            logger.info("attack:" + attack)
+                            # cut out tag in square brackets, e.g. Defense Evasion::Obfuscated Files or Information [T1027] => T1027
+                            r = re.search(r"\[(T[^\]]*)", attack)
+                            if r:
+                                tag = r.group(1)
+                                logger.info("attack tag:" + tag)
+                                tag = re.sub(r"\W", "_", tag)
+                                rule_tags += tag + " "
+                                # also add a line "attack = ..." to yaras 'meta:' to keep the long description:
+                                yara_meta += '\tattack = "' + attack + '"\n'
+                    elif meta_name == "mbc":
+                        for mbc in list(metas[meta]):
+                            logger.info("mbc:" + mbc)
+                            # cut out tag in square brackets, e.g. Cryptography::Encrypt Data::RC6 [C0027.010] => C0027.010
+                            r = re.search(r"\[(.[^\]]*)", mbc)
+                            if r:
+                                tag = r.group(1)
+                                logger.info("mbc tag:" + tag)
+                                tag = re.sub(r"\W", "_", tag)
+                                rule_tags += tag + " "
+
+                                # also add a line "mbc = ..." to yaras 'meta:' to keep the long description:
+                                yara_meta += '\tmbc = "' + mbc + '"\n'
+
+                    for value in metas[meta]:
+                        if meta_name == "hash":
+                            value = re.sub(r"^([0-9a-f]{20,64}):0x[0-9a-f]{1,10}$", r"\1", value, flags=re.IGNORECASE)
+
+                            # examples in capa can contain the same hash several times with different offset, so check if it's already there:
+                            # (keeping the offset might be interessting for some but breaks yara-ci for checking of the final rules
+                            if not value in seen_hashes:
+                                yara_meta += "\t" + meta_name + ' = "' + value + '"\n'
+                                seen_hashes.append(value)
+
+                else:
+                    # no list:
+                    if meta == "capa/path":
+                        url = get_rule_url(metas[meta])
+                        meta_name = "reference"
+                        meta_value = "This YARA rule converted from capa rule: " + url
+                    else:
+                        meta_value = metas[meta]
+
+                    if meta_name == "name":
+                        meta_name = "description"
+                        meta_value += " (converted from capa rule)"
+                    elif meta_name == "lib":
+                        meta_value = str(meta_value)
+                    elif meta_name == "capa/nursery":
+                        meta_name = "capa_nursery"
+                        meta_value = str(meta_value)
+
+                    # for the rest of the maec/malware-category names:
+                    meta_name = re.sub(r"\W", "_", meta_name)
+
+                    if meta_name and meta_value:
+                        yara_meta += "\t" + meta_name + ' = "' + meta_value + '"\n'
+
+            rule_name_bonus = ""
+            if rule_comment:
+                yara_meta += '\tcomment = "' + rule_comment + '"\n'
+            yara_meta += '\tdate = "' + today + '"\n'
+            yara_meta += '\tminimum_yara = "3.8"\n'
+            yara_meta += '\tlicense = "Apache-2.0 License"\n'
+
+            # check if there's some beef in condition:
+            tmp_yc = re.sub(r"(and|or|not)", "", yara_condition)
+            if re.search(r"\w", tmp_yc):
+
+                yara = ""
+                if make_priv:
+                    yara = "private "
+
+                # put yara rule tags here:
+                rule_tags = default_tags + rule_tags
+                yara += "rule " + rule_name + " : " + rule_tags + " { \n  meta: \n " + yara_meta + "\n"
+
+                if "$" in yara_strings:
+                    yara += "  strings: \n " + yara_strings + " \n"
+
+                yara += "  condition:" + condition_header + yara_condition + "\n}"
+
+                # TODO: now the rule is finished and could be automatically checked with the capa-testfile(s) named in meta (doing it for all of them using yara-ci upload at the moment)
+                output_yar(yara)
+                converted_rules.append(rule_name)
+                global count_incomplete
+                count_incomplete += incomplete
+            else:
+                output_unsupported_capa_rules(rule.to_yaml(), rule.name, url, yara_condition)
+                pass
+
+
+def main(argv=None):
+    if argv is None:
+        argv = sys.argv[1:]
+
+    parser = argparse.ArgumentParser(description="Capa to YARA rule converter")
+    parser.add_argument("rules", type=str, help="Path to rules")
+    parser.add_argument("--private", "-p", action="store_true", help="Create private rules", default=False)
+    capa.main.install_common_args(parser, wanted={"tag"})
+
+    args = parser.parse_args(args=argv)
+    global make_priv
+    make_priv = args.private
+
+    if args.verbose:
+        level = logging.DEBUG
+    elif args.quiet:
+        level = logging.ERROR
+    else:
+        level = logging.INFO
+
+    logging.basicConfig(level=level)
+    logging.getLogger("capa2yara").setLevel(level)
+
+    try:
+        rules = capa.main.get_rules(args.rules, disable_progress=True)
+        namespaces = capa.rules.index_rules_by_namespace(list(rules))
+        rules = capa.rules.RuleSet(rules)
+        logger.info("successfully loaded %s rules (including subscope rules which will be ignored)", len(rules))
+        if args.tag:
+            rules = rules.filter_rules_by_meta(args.tag)
+            logger.debug("selected %s rules", len(rules))
+            for i, r in enumerate(rules.rules, 1):
+                logger.debug(" %d. %s", i, r)
+    except (IOError, capa.rules.InvalidRule, capa.rules.InvalidRuleSet) as e:
+        logger.error("%s", str(e))
+        return -1
+
+    output_yar(
+        "// Rules from Mandiant's https://github.com/mandiant/capa-rules converted to YARA using https://github.com/mandiant/capa/blob/master/scripts/capa2yara.py by Arnim Rupp"
+    )
+    output_yar(
+        "// Beware: These are less rules than capa (because not all fit into YARA, stats at EOF) and is less precise because e.g. capas function scopes are applied to the whole file"
+    )
+    output_yar(
+        '// Beware: Some rules are incomplete because an optional branch was not supported by YARA. These rules are marked in a comment in meta: (search for "incomplete")'
+    )
+    output_yar("// Rule authors and license stay the same")
+    output_yar(
+        '// att&ck and MBC tags are put into YARA rule tags. All rules are tagged with "CAPA" for easy filtering'
+    )
+    output_yar("// The date = in meta: is the date of converting (there is no date in capa rules)")
+    output_yar("// Minimum YARA version is 3.8.0 plus PE module")
+    output_yar('\nimport "pe"')
+
+    output_yar(condition_rule)
+
+    # do several rounds of converting rules because some rules for match: might not be converted in the 1st run
+    num_rules = 9999999
+    cround = 0
+    while num_rules != len(converted_rules) or cround < min_rounds:
+        cround += 1
+        logger.info("doing convert_rules(), round: " + str(cround))
+        num_rules = len(converted_rules)
+        convert_rules(rules, namespaces, cround)
+
+    # one last round to collect all unconverted rules
+    convert_rules(rules, namespaces, 9000)
+
+    stats = "\n// converted rules              : " + str(len(converted_rules))
+    stats += "\n//   among those are incomplete : " + str(count_incomplete)
+    stats += "\n// unconverted rules            : " + str(len(unsupported_capa_rules_list)) + "\n"
+    logger.info(stats)
+    output_yar(stats)
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/capa_as_library.py

@@ -6,18 +6,18 @@ import collections
 import capa.main
 import capa.rules
 import capa.engine
-import capa.render
 import capa.features
+import capa.render.json
 import capa.render.utils as rutils
+import capa.render.default
+import capa.render.result_document
 from capa.engine import *
-from capa.render import convert_capabilities_to_result_document
 
 # edit this to set the path for file to analyze and rule directory
 RULES_PATH = "/tmp/capa/rules/"
 
 # load rules from disk
-rules = capa.main.get_rules(RULES_PATH, disable_progress=True)
-rules = capa.rules.RuleSet(rules)
+rules = capa.rules.RuleSet(capa.main.get_rules(RULES_PATH, disable_progress=True))
 
 # == Render ddictionary helpers
 def render_meta(doc, ostream):
@@ -104,28 +104,16 @@ def render_attack(doc, ostream):
     for rule in rutils.capability_rules(doc):
         if not rule["meta"].get("att&ck"):
             continue
-
         for attack in rule["meta"]["att&ck"]:
-            tactic, _, rest = attack.partition("::")
-            if "::" in rest:
-                technique, _, rest = rest.partition("::")
-                subtechnique, _, id = rest.rpartition(" ")
-                tactics[tactic].add((technique, subtechnique, id))
-            else:
-                technique, _, id = rest.rpartition(" ")
-                tactics[tactic].add((technique, id))
+            tactics[attack["tactic"]].add((attack["technique"], attack.get("subtechnique"), attack["id"]))
 
     for tactic, techniques in sorted(tactics.items()):
         inner_rows = []
-        for spec in sorted(techniques):
-            if len(spec) == 2:
-                technique, id = spec
+        for (technique, subtechnique, id) in sorted(techniques):
+            if subtechnique is None:
                 inner_rows.append("%s %s" % (technique, id))
-            elif len(spec) == 3:
-                technique, subtechnique, id = spec
-                inner_rows.append("%s::%s %s" % (technique, subtechnique, id))
             else:
-                raise RuntimeError("unexpected ATT&CK spec format")
+                inner_rows.append("%s::%s %s" % (technique, subtechnique, id))
         ostream["ATTCK"].setdefault(tactic.upper(), inner_rows)
 
 
@@ -150,31 +138,16 @@ def render_mbc(doc, ostream):
         if not rule["meta"].get("mbc"):
             continue
 
-        mbcs = rule["meta"]["mbc"]
-        if not isinstance(mbcs, list):
-            raise ValueError("invalid rule: MBC mapping is not a list")
-
-        for mbc in mbcs:
-            objective, _, rest = mbc.partition("::")
-            if "::" in rest:
-                behavior, _, rest = rest.partition("::")
-                method, _, id = rest.rpartition(" ")
-                objectives[objective].add((behavior, method, id))
-            else:
-                behavior, _, id = rest.rpartition(" ")
-                objectives[objective].add((behavior, id))
+        for mbc in rule["meta"]["mbc"]:
+            objectives[mbc["objective"]].add((mbc["behavior"], mbc.get("method"), mbc["id"]))
 
     for objective, behaviors in sorted(objectives.items()):
         inner_rows = []
-        for spec in sorted(behaviors):
-            if len(spec) == 2:
-                behavior, id = spec
-                inner_rows.append("%s %s" % (behavior, id))
-            elif len(spec) == 3:
-                behavior, method, id = spec
-                inner_rows.append("%s::%s %s" % (behavior, method, id))
+        for (behavior, method, id) in sorted(behaviors):
+            if method is None:
+                inner_rows.append("%s [%s]" % (behavior, id))
             else:
-                raise RuntimeError("unexpected MBC spec format")
+                inner_rows.append("%s::%s [%s]" % (behavior, method, id))
         ostream["MBC"].setdefault(objective.upper(), inner_rows)
 
 
@@ -190,26 +163,27 @@ def render_dictionary(doc):
 
 # ==== render dictionary helpers
 def capa_details(file_path, output_format="dictionary"):
+    # collect metadata (used only to make rendering more complete)
+    meta = capa.main.collect_metadata("", file_path, RULES_PATH, extractor)
 
     # extract features and find capabilities
-    extractor = capa.main.get_extractor(file_path, "auto", capa.main.BACKEND_VIV, disable_progress=True)
+    extractor = capa.main.get_extractor(file_path, "auto", capa.main.BACKEND_VIV, [], False, disable_progress=True)
     capabilities, counts = capa.main.find_capabilities(rules, extractor, disable_progress=True)
 
-    # collect metadata (used only to make rendering more complete)
-    meta = capa.main.collect_metadata("", file_path, RULES_PATH, "auto", extractor)
     meta["analysis"].update(counts)
+    meta["analysis"]["layout"] = capa.main.compute_layout(rules, extractor, capabilities)
 
     capa_output = False
     if output_format == "dictionary":
         # ...as python dictionary, simplified as textable but in dictionary
-        doc = convert_capabilities_to_result_document(meta, rules, capabilities)
+        doc = capa.render.result_document.convert_capabilities_to_result_document(meta, rules, capabilities)
         capa_output = render_dictionary(doc)
     elif output_format == "json":
         # render results
         # ...as json
-        capa_output = json.loads(capa.render.render_json(meta, rules, capabilities))
+        capa_output = json.loads(capa.render.json.render(meta, rules, capabilities))
     elif output_format == "texttable":
         # ...as human readable text table
-        capa_output = capa.render.render_default(meta, rules, capabilities)
+        capa_output = capa.render.default.render(meta, rules, capabilities)
 
     return capa_output

scripts/capafmt.py

@@ -6,7 +6,7 @@ Usage:
 
    $ python capafmt.py -i foo.yml
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt

scripts/ci.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -9,6 +9,7 @@
 # See the License for the specific language governing permissions and limitations under the License.
 
 # Use a console with emojis support for a better experience
+# Use venv to ensure that `python` calls the correct python version
 
 # Stash uncommited changes
 MSG="pre-push-$(date +%s)";
@@ -25,17 +26,8 @@ restore_stashed() {
   fi
 }
 
-python_3() {
-  case "$(uname -s)" in
-   CYGWIN*|MINGW32*|MSYS*|MINGW*)
-     py -3 -m $1 > $2 2>&1;;
-   *)
-     python3 -m $1 > $2 2>&1;;
-  esac
-}
-
 # Run isort and print state
-python_3 'isort --profile black --length-sort --line-width 120 -c .' 'isort-output.log';
+python -m isort --profile black --length-sort --line-width 120 -c . > isort-output.log 2>&1;
 if [ $? == 0 ]; then
   echo 'isort succeeded!! 💖';
 else
@@ -46,7 +38,7 @@ else
 fi
 
 # Run black and print state
-python_3 'black -l 120 --check .' 'black-output.log';
+python -m black -l 120 --check . > black-output.log 2>&1;
 if [ $? == 0 ]; then
   echo 'black succeeded!! 💝';
 else
@@ -70,7 +62,7 @@ fi
 # Run tests except if first argument is no_tests
 if [ "$1" != 'no_tests' ]; then
   echo 'Running tests, please wait ⌛';
-  pytest tests/ --maxfail=1;
+  python -m pytest tests/ --maxfail=1;
   if [ $? == 0 ]; then
     echo 'Tests succeed!! 🎉';
   else

scripts/detect-elf-os.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python2
+"""
+Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+You may obtain a copy of the License at: [package root]/LICENSE.txt
+Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and limitations under the License.
+
+detect-elf-os
+
+Attempt to detect the underlying OS that the given ELF file targets.
+"""
+import sys
+import logging
+import argparse
+import contextlib
+from typing import BinaryIO
+
+import capa.helpers
+import capa.features.extractors.elf
+
+logger = logging.getLogger("capa.detect-elf-os")
+
+
+def main(argv=None):
+    if capa.helpers.is_runtime_ida():
+        from capa.ida.helpers import IDAIO
+
+        f: BinaryIO = IDAIO()
+
+    else:
+        if argv is None:
+            argv = sys.argv[1:]
+
+        parser = argparse.ArgumentParser(description="Detect the underlying OS for the given ELF file")
+        parser.add_argument("sample", type=str, help="path to ELF file")
+
+        logging_group = parser.add_argument_group("logging arguments")
+
+        logging_group.add_argument("-d", "--debug", action="store_true", help="enable debugging output on STDERR")
+        logging_group.add_argument(
+            "-q", "--quiet", action="store_true", help="disable all status output except fatal errors"
+        )
+
+        args = parser.parse_args(args=argv)
+
+        if args.quiet:
+            logging.basicConfig(level=logging.WARNING)
+            logging.getLogger().setLevel(logging.WARNING)
+        elif args.debug:
+            logging.basicConfig(level=logging.DEBUG)
+            logging.getLogger().setLevel(logging.DEBUG)
+        else:
+            logging.basicConfig(level=logging.INFO)
+            logging.getLogger().setLevel(logging.INFO)
+
+        f = open(args.sample, "rb")
+
+    with contextlib.closing(f):
+        try:
+            print(capa.features.extractors.elf.detect_elf_os(f))
+            return 0
+        except capa.features.extractors.elf.CorruptElfFile as e:
+            logger.error("corrupt ELF file: %s", str(e.args[0]))
+            return -1
+
+
+if __name__ == "__main__":
+    if capa.helpers.is_runtime_ida():
+        main()
+    else:
+        sys.exit(main())

scripts/import-to-bn.py

@@ -20,12 +20,13 @@ Adapted for Binary Ninja by @psifertex
 This script will verify that the report matches the workspace.
 Check the log window for any errors, and/or the summary of changes.
 
-Derived from: https://github.com/fireeye/capa/blob/master/scripts/import-to-ida.py
+Derived from: https://github.com/mandiant/capa/blob/master/scripts/import-to-ida.py
 """
 import os
 import json
 
-from binaryninja import *
+import binaryninja
+import binaryninja.interaction
 
 
 def append_func_cmt(bv, va, cmt):
@@ -46,31 +47,31 @@ def append_func_cmt(bv, va, cmt):
 def load_analysis(bv):
     shortname = os.path.splitext(os.path.basename(bv.file.filename))[0]
     dirname = os.path.dirname(bv.file.filename)
-    log_info(f"dirname: {dirname}\nshortname: {shortname}\n")
+    binaryninja.log_info(f"dirname: {dirname}\nshortname: {shortname}\n")
     if os.access(os.path.join(dirname, shortname + ".js"), os.R_OK):
         path = os.path.join(dirname, shortname + ".js")
     elif os.access(os.path.join(dirname, shortname + ".json"), os.R_OK):
         path = os.path.join(dirname, shortname + ".json")
     else:
-        path = interaction.get_open_filename_input("capa report:", "JSON (*.js *.json);;All Files (*)")
+        path = binaryninja.interaction.get_open_filename_input("capa report:", "JSON (*.js *.json);;All Files (*)")
     if not path or not os.access(path, os.R_OK):
-        log_error("Invalid filename.")
+        binaryninja.log_error("Invalid filename.")
         return 0
-    log_info("Using capa file %s" % path)
+    binaryninja.log_info("Using capa file %s" % path)
 
     with open(path, "rb") as f:
         doc = json.loads(f.read().decode("utf-8"))
 
     if "meta" not in doc or "rules" not in doc:
-        log_error("doesn't appear to be a capa report")
+        binaryninja.log_error("doesn't appear to be a capa report")
         return -1
 
     a = doc["meta"]["sample"]["md5"].lower()
-    md5 = Transform["MD5"]
-    rawhex = Transform["RawHex"]
+    md5 = binaryninja.Transform["MD5"]
+    rawhex = binaryninja.Transform["RawHex"]
     b = rawhex.encode(md5.encode(bv.parent_view.read(bv.parent_view.start, bv.parent_view.end))).decode("utf-8")
     if not a == b:
-        log_error("sample mismatch")
+        binaryninja.log_error("sample mismatch")
         return -2
 
     rows = []
@@ -96,7 +97,7 @@ def load_analysis(bv):
         else:
             cmt = "%s" % (name,)
 
-        log_info("0x%x: %s" % (va, cmt))
+        binaryninja.log_info("0x%x: %s" % (va, cmt))
         try:
             # message will look something like:
             #
@@ -105,7 +106,7 @@ def load_analysis(bv):
         except ValueError:
             continue
 
-    log_info("ok")
+    binaryninja.log_info("ok")
 
 
-PluginCommand.register("Load capa file", "Loads an analysis file from capa", load_analysis)
+binaryninja.PluginCommand.register("Load capa file", "Loads an analysis file from capa", load_analysis)

scripts/import-to-ida.py

@@ -20,7 +20,7 @@ and then select the existing capa report from the file system.
 This script will verify that the report matches the workspace.
 Check the output window for any errors, and/or the summary of changes.
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt

scripts/lint.py

@@ -5,7 +5,7 @@ Usage:
 
    $ python scripts/lint.py rules/
 
-Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
 You may obtain a copy of the License at: [package root]/LICENSE.txt
@@ -13,38 +13,78 @@ Unless required by applicable law or agreed to in writing, software distributed
  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and limitations under the License.
 """
+import gc
 import os
 import sys
 import time
 import string
 import difflib
 import hashlib
+import inspect
 import logging
 import os.path
+import pathlib
 import argparse
 import itertools
 import posixpath
+import contextlib
+from typing import Set, Dict, List
+from pathlib import Path
+from dataclasses import field, dataclass
 
+import tqdm
+import termcolor
 import ruamel.yaml
+import tqdm.contrib.logging
 
 import capa.main
 import capa.rules
 import capa.engine
-import capa.features
 import capa.features.insn
+import capa.features.common
+from capa.rules import Rule, RuleSet
+from capa.features.common import Feature
 
-logger = logging.getLogger("capa.lint")
+logger = logging.getLogger("lint")
 
 
-class Lint(object):
-    WARN = "WARN"
-    FAIL = "FAIL"
+def red(s):
+    return termcolor.colored(s, "red")
+
+
+def orange(s):
+    return termcolor.colored(s, "yellow")
+
+
+def green(s):
+    return termcolor.colored(s, "green")
+
+
+@dataclass
+class Context:
+    """
+    attributes:
+      samples: mapping from content hash (MD5, SHA, etc.) to file path.
+      rules: rules to inspect
+      is_thorough: should inspect long-running lints
+      capabilities_by_sample: cache of results, indexed by file path.
+    """
+
+    samples: Dict[str, Path]
+    rules: RuleSet
+    is_thorough: bool
+    capabilities_by_sample: Dict[Path, Set[str]] = field(default_factory=dict)
+
+
+class Lint:
+    WARN = orange("WARN")
+    FAIL = red("FAIL")
 
     name = "lint"
     level = FAIL
     recommendation = ""
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         return False
 
 
@@ -52,7 +92,7 @@ class NameCasing(Lint):
     name = "rule name casing"
     recommendation = "Rename rule using to start with lower case letters"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         return rule.name[0] in string.ascii_uppercase and rule.name[1] not in string.ascii_uppercase
 
 
@@ -61,7 +101,7 @@ class FilenameDoesntMatchRuleName(Lint):
     recommendation = "Rename rule file to match the rule name"
     recommendation_template = 'Rename rule file to match the rule name, expected: "{:s}", found: "{:s}"'
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         expected = rule.name
         expected = expected.lower()
         expected = expected.replace(" ", "-")
@@ -83,7 +123,7 @@ class MissingNamespace(Lint):
     name = "missing rule namespace"
     recommendation = "Add meta.namespace so that the rule is emitted correctly"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         return (
             "namespace" not in rule.meta
             and not is_nursery_rule(rule)
@@ -96,7 +136,7 @@ class NamespaceDoesntMatchRulePath(Lint):
     name = "file path doesn't match rule namespace"
     recommendation = "Move rule to appropriate directory or update the namespace"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         # let the other lints catch namespace issues
         if "namespace" not in rule.meta:
             return False
@@ -114,7 +154,7 @@ class MissingScope(Lint):
     name = "missing scope"
     recommendation = "Add meta.scope so that the scope is explicit (defaults to `function`)"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         return "scope" not in rule.meta
 
 
@@ -122,7 +162,7 @@ class InvalidScope(Lint):
     name = "invalid scope"
     recommendation = "Use only file, function, or basic block rule scopes"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         return rule.meta.get("scope") not in ("file", "function", "basic block")
 
 
@@ -130,7 +170,7 @@ class MissingAuthor(Lint):
     name = "missing author"
     recommendation = "Add meta.author so that users know who to contact with questions"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         return "author" not in rule.meta
 
 
@@ -138,7 +178,7 @@ class MissingExamples(Lint):
     name = "missing examples"
     recommendation = "Add meta.examples so that the rule can be tested and verified"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         return (
             "examples" not in rule.meta
             or not isinstance(rule.meta["examples"], list)
@@ -151,7 +191,7 @@ class MissingExampleOffset(Lint):
     name = "missing example offset"
     recommendation = "Add offset of example function"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         if rule.meta.get("scope") in ("function", "basic block"):
             examples = rule.meta.get("examples")
             if isinstance(examples, list):
@@ -165,7 +205,7 @@ class ExampleFileDNE(Lint):
     name = "referenced example doesn't exist"
     recommendation = "Add the referenced example to samples directory ($capa-root/tests/data or supplied via --samples)"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         if not rule.meta.get("examples"):
             # let the MissingExamples lint catch this case, don't double report.
             return False
@@ -174,19 +214,59 @@ class ExampleFileDNE(Lint):
         for example in rule.meta.get("examples", []):
             if example:
                 example_id = example.partition(":")[0]
-                if example_id in ctx["samples"]:
+                if example_id in ctx.samples:
                     found = True
                     break
 
         return not found
 
 
+DEFAULT_SIGNATURES = capa.main.get_default_signatures()
+
+
+def get_sample_capabilities(ctx: Context, path: Path) -> Set[str]:
+    nice_path = os.path.abspath(str(path))
+    if path in ctx.capabilities_by_sample:
+        logger.debug("found cached results: %s: %d capabilities", nice_path, len(ctx.capabilities_by_sample[path]))
+        return ctx.capabilities_by_sample[path]
+
+    if nice_path.endswith(capa.main.EXTENSIONS_SHELLCODE_32):
+        format = "sc32"
+    elif nice_path.endswith(capa.main.EXTENSIONS_SHELLCODE_64):
+        format = "sc64"
+    else:
+        format = "auto"
+
+    logger.debug("analyzing sample: %s", nice_path)
+    extractor = capa.main.get_extractor(
+        nice_path, format, capa.main.BACKEND_VIV, DEFAULT_SIGNATURES, False, disable_progress=True
+    )
+
+    capabilities, _ = capa.main.find_capabilities(ctx.rules, extractor, disable_progress=True)
+    # mypy doesn't seem to be happy with the MatchResults type alias & set(...keys())?
+    # so we ignore a few types here.
+    capabilities = set(capabilities.keys())  # type: ignore
+    assert isinstance(capabilities, set)
+
+    logger.debug("computed results: %s: %d capabilities", nice_path, len(capabilities))
+    ctx.capabilities_by_sample[path] = capabilities
+
+    # when i (wb) run the linter in thorough mode locally,
+    # the OS occasionally kills the process due to memory usage.
+    # so, be extra aggressive in keeping memory usage down.
+    #
+    # tbh, im not sure this actually does anything, but maybe it helps?
+    gc.collect()
+
+    return capabilities
+
+
 class DoesntMatchExample(Lint):
     name = "doesn't match on referenced example"
     recommendation = "Fix the rule logic or provide a different example"
 
-    def check_rule(self, ctx, rule):
-        if not ctx["is_thorough"]:
+    def check_rule(self, ctx: Context, rule: Rule):
+        if not ctx.is_thorough:
             return False
 
         examples = rule.meta.get("examples", [])
@@ -196,29 +276,121 @@ class DoesntMatchExample(Lint):
         for example in examples:
             example_id = example.partition(":")[0]
             try:
-                path = ctx["samples"][example_id]
+                path = ctx.samples[example_id]
             except KeyError:
                 # lint ExampleFileDNE will catch this.
                 # don't double report.
                 continue
 
             try:
-                extractor = capa.main.get_extractor(path, "auto", capa.main.BACKEND_VIV, disable_progress=True)
-                capabilities, meta = capa.main.find_capabilities(ctx["rules"], extractor, disable_progress=True)
+                capabilities = get_sample_capabilities(ctx, path)
             except Exception as e:
-                logger.error("failed to extract capabilities: %s %s %s", rule.name, path, e)
+                logger.error("failed to extract capabilities: %s %s %s", rule.name, str(path), e, exc_info=True)
                 return True
 
             if rule.name not in capabilities:
                 return True
 
 
+class StatementWithSingleChildStatement(Lint):
+    name = "rule contains one or more statements with a single child statement"
+    recommendation = "remove the superfluous parent statement"
+    recommendation_template = "remove the superfluous parent statement: {:s}"
+    violation = False
+
+    def check_rule(self, ctx: Context, rule: Rule):
+        self.violation = False
+
+        def rec(statement, is_root=False):
+            if isinstance(statement, (capa.engine.And, capa.engine.Or)):
+                children = list(statement.get_children())
+                if not is_root and len(children) == 1 and isinstance(children[0], capa.engine.Statement):
+                    self.recommendation = self.recommendation_template.format(str(statement))
+                    self.violation = True
+                for child in children:
+                    rec(child)
+
+        rec(rule.statement, is_root=True)
+
+        return self.violation
+
+
+class OrStatementWithAlwaysTrueChild(Lint):
+    name = "rule contains an `or` statement that's always True because of an `optional` or other child statement that's always True"
+    recommendation = "clarify the rule logic, e.g. by moving the always True child statement"
+    recommendation_template = "clarify the rule logic, e.g. by moving the always True child statement: {:s}"
+    violation = False
+
+    def check_rule(self, ctx: Context, rule: Rule):
+        self.violation = False
+
+        def rec(statement):
+            if isinstance(statement, capa.engine.Or):
+                children = list(statement.get_children())
+                for child in children:
+                    # `Some` implements `optional` which is an alias for `0 or more`
+                    if isinstance(child, capa.engine.Some) and child.count == 0:
+                        self.recommendation = self.recommendation_template.format(str(child))
+                        self.violation = True
+                    rec(child)
+
+        rec(rule.statement)
+
+        return self.violation
+
+
+class NotNotUnderAnd(Lint):
+    name = "rule contains a `not` statement that's not found under an `and` statement"
+    recommendation = "clarify the rule logic and ensure `not` is always found under `and`"
+    violation = False
+
+    def check_rule(self, ctx: Context, rule: Rule):
+        self.violation = False
+
+        def rec(statement):
+            if isinstance(statement, capa.engine.Statement):
+                if not isinstance(statement, capa.engine.And):
+                    for child in statement.get_children():
+                        if isinstance(child, capa.engine.Not):
+                            self.violation = True
+
+                for child in statement.get_children():
+                    rec(child)
+
+        rec(rule.statement)
+
+        return self.violation
+
+
+class OptionalNotUnderAnd(Lint):
+    name = "rule contains an `optional` or `0 or more` statement that's not found under an `and` statement"
+    recommendation = "clarify the rule logic and ensure `optional` and `0 or more` is always found under `and`"
+    violation = False
+
+    def check_rule(self, ctx: Context, rule: Rule):
+        self.violation = False
+
+        def rec(statement):
+            if isinstance(statement, capa.engine.Statement):
+                if not isinstance(statement, capa.engine.And):
+                    for child in statement.get_children():
+                        if isinstance(child, capa.engine.Some) and child.count == 0:
+                            self.violation = True
+
+                for child in statement.get_children():
+                    rec(child)
+
+        rec(rule.statement)
+
+        return self.violation
+
+
 class UnusualMetaField(Lint):
     name = "unusual meta field"
     recommendation = "Remove the meta field"
     recommendation_template = 'Remove the meta field: "{:s}"'
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         for key in rule.meta.keys():
             if key in capa.rules.META_KEYS:
                 continue
@@ -234,7 +406,7 @@ class LibRuleNotInLibDirectory(Lint):
     name = "lib rule not found in lib directory"
     recommendation = "Move the rule to the `lib` subdirectory of the rules path"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         if is_nursery_rule(rule):
             return False
 
@@ -248,7 +420,7 @@ class LibRuleHasNamespace(Lint):
     name = "lib rule has a namespace"
     recommendation = "Remove the namespace from the rule"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         if "lib" not in rule.meta:
             return False
 
@@ -259,9 +431,10 @@ class FeatureStringTooShort(Lint):
     name = "feature string too short"
     recommendation = 'capa only extracts strings with length >= 4; will not match on "{:s}"'
 
-    def check_features(self, ctx, features):
+    def check_features(self, ctx: Context, features: List[Feature]):
         for feature in features:
-            if isinstance(feature, capa.features.String):
+            if isinstance(feature, (capa.features.common.String, capa.features.common.Substring)):
+                assert isinstance(feature.value, str)
                 if len(feature.value) < 4:
                     self.recommendation = self.recommendation.format(feature.value)
                     return True
@@ -276,9 +449,10 @@ class FeatureNegativeNumber(Lint):
         'representation; will not match on "{:d}"'
     )
 
-    def check_features(self, ctx, features):
+    def check_features(self, ctx: Context, features: List[Feature]):
         for feature in features:
             if isinstance(feature, (capa.features.insn.Number,)):
+                assert isinstance(feature.value, int)
                 if feature.value < 0:
                     self.recommendation = self.recommendation_template.format(feature.value)
                     return True
@@ -288,17 +462,61 @@ class FeatureNegativeNumber(Lint):
 class FeatureNtdllNtoskrnlApi(Lint):
     name = "feature api may overlap with ntdll and ntoskrnl"
     level = Lint.WARN
-    recommendation = (
+    recommendation_template = (
         "check if {:s} is exported by both ntdll and ntoskrnl; if true, consider removing {:s} "
         "module requirement to improve detection"
     )
 
-    def check_features(self, ctx, features):
+    def check_features(self, ctx: Context, features: List[Feature]):
         for feature in features:
             if isinstance(feature, capa.features.insn.API):
+                assert isinstance(feature.value, str)
                 modname, _, impname = feature.value.rpartition(".")
+
+                if modname == "ntdll":
+                    if impname in (
+                        "LdrGetProcedureAddress",
+                        "LdrLoadDll",
+                        "NtCreateThread",
+                        "NtCreatUserProcess",
+                        "NtLoadDriver",
+                        "NtQueryDirectoryObject",
+                        "NtResumeThread",
+                        "NtSuspendThread",
+                        "NtTerminateProcess",
+                        "NtWriteVirtualMemory",
+                        "RtlGetNativeSystemInformation",
+                        "NtCreateThreadEx",
+                        "NtCreateUserProcess",
+                        "NtOpenDirectoryObject",
+                        "NtQueueApcThread",
+                        "ZwResumeThread",
+                        "ZwSuspendThread",
+                        "ZwWriteVirtualMemory",
+                        "NtCreateProcess",
+                        "ZwCreateThread",
+                        "NtCreateProcessEx",
+                        "ZwCreateThreadEx",
+                        "ZwCreateProcess",
+                        "ZwCreateUserProcess",
+                        "RtlCreateUserProcess",
+                    ):
+                        # ntoskrnl.exe does not export these routines
+                        continue
+
+                if modname == "ntoskrnl":
+                    if impname in (
+                        "PsGetVersion",
+                        "PsLookupProcessByProcessId",
+                        "KeStackAttachProcess",
+                        "ObfDereferenceObject",
+                        "KeUnstackDetachProcess",
+                    ):
+                        # ntdll.dll does not export these routines
+                        continue
+
                 if modname in ("ntdll", "ntoskrnl"):
-                    self.recommendation = self.recommendation.format(impname, modname)
+                    self.recommendation = self.recommendation_template.format(impname, modname)
                     return True
         return False
 
@@ -307,7 +525,7 @@ class FormatLineFeedEOL(Lint):
     name = "line(s) end with CRLF (\\r\\n)"
     recommendation = "convert line endings to LF (\\n) for example using dos2unix"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         if len(rule.definition.split("\r\n")) > 0:
             return False
         return True
@@ -317,7 +535,7 @@ class FormatSingleEmptyLineEOF(Lint):
     name = "EOF format"
     recommendation = "end file with a single empty line"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         if rule.definition.endswith("\n") and not rule.definition.endswith("\n\n"):
             return False
         return True
@@ -327,7 +545,7 @@ class FormatIncorrect(Lint):
     name = "rule format incorrect"
     recommendation_template = "use scripts/capafmt.py or adjust as follows\n{:s}"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         actual = rule.definition
         expected = capa.rules.Rule.from_yaml(rule.definition, use_ruamel=True).to_yaml()
 
@@ -347,36 +565,53 @@ class FormatIncorrect(Lint):
 class FormatStringQuotesIncorrect(Lint):
     name = "rule string quotes incorrect"
 
-    def check_rule(self, ctx, rule):
+    def check_rule(self, ctx: Context, rule: Rule):
         events = capa.rules.Rule._get_ruamel_yaml_parser().parse(rule.definition)
         for key in events:
-            if not (isinstance(key, ruamel.yaml.ScalarEvent) and key.value == "string"):
+            if isinstance(key, ruamel.yaml.ScalarEvent) and key.value == "string":
+                value = next(events)  # assume value is next event
+                if not isinstance(value, ruamel.yaml.ScalarEvent):
+                    # ignore non-scalar
+                    continue
+                if value.value.startswith("/") and value.value.endswith(("/", "/i")):
+                    # ignore regex for now
+                    continue
+                if value.style is None:
+                    # no quotes
+                    self.recommendation = 'add double quotes to "%s"' % value.value
+                    return True
+                if value.style == "'":
+                    # single quote
+                    self.recommendation = 'change single quotes to double quotes for "%s"' % value.value
+                    return True
+
+            elif isinstance(key, ruamel.yaml.ScalarEvent) and key.value == "substring":
+                value = next(events)  # assume value is next event
+                if not isinstance(value, ruamel.yaml.ScalarEvent):
+                    # ignore non-scalar
+                    continue
+                if value.style is None:
+                    # no quotes
+                    self.recommendation = 'add double quotes to "%s"' % value.value
+                    return True
+                if value.style == "'":
+                    # single quote
+                    self.recommendation = 'change single quotes to double quotes for "%s"' % value.value
+                    return True
+
+            else:
                 continue
-            value = next(events)  # assume value is next event
-            if not isinstance(value, ruamel.yaml.ScalarEvent):
-                # ignore non-scalar
-                continue
-            if value.value.startswith("/") and value.value.endswith(("/", "/i")):
-                # ignore regex for now
-                continue
-            if value.style is None:
-                # no quotes
-                self.recommendation = 'add double quotes to "%s"' % value.value
-                return True
-            if value.style == "'":
-                # single quote
-                self.recommendation = 'change single quotes to double quotes for "%s"' % value.value
-                return True
+
         return False
 
 
-def run_lints(lints, ctx, rule):
+def run_lints(lints, ctx: Context, rule: Rule):
     for lint in lints:
         if lint.check_rule(ctx, rule):
             yield lint
 
 
-def run_feature_lints(lints, ctx, features):
+def run_feature_lints(lints, ctx: Context, features: List[Feature]):
     for lint in lints:
         if lint.check_features(ctx, features):
             yield lint
@@ -388,7 +623,7 @@ NAME_LINTS = (
 )
 
 
-def lint_name(ctx, rule):
+def lint_name(ctx: Context, rule: Rule):
     return run_lints(NAME_LINTS, ctx, rule)
 
 
@@ -398,7 +633,7 @@ SCOPE_LINTS = (
 )
 
 
-def lint_scope(ctx, rule):
+def lint_scope(ctx: Context, rule: Rule):
     return run_lints(SCOPE_LINTS, ctx, rule)
 
 
@@ -415,14 +650,14 @@ META_LINTS = (
 )
 
 
-def lint_meta(ctx, rule):
+def lint_meta(ctx: Context, rule: Rule):
     return run_lints(META_LINTS, ctx, rule)
 
 
 FEATURE_LINTS = (FeatureStringTooShort(), FeatureNegativeNumber(), FeatureNtdllNtoskrnlApi())
 
 
-def lint_features(ctx, rule):
+def lint_features(ctx: Context, rule: Rule):
     features = get_features(ctx, rule)
     return run_feature_lints(FEATURE_LINTS, ctx, features)
 
@@ -435,7 +670,7 @@ FORMAT_LINTS = (
 )
 
 
-def lint_format(ctx, rule):
+def lint_format(ctx: Context, rule: Rule):
     return run_lints(FORMAT_LINTS, ctx, rule)
 
 
@@ -443,11 +678,11 @@ def get_normpath(path):
     return posixpath.normpath(path).replace(os.sep, "/")
 
 
-def get_features(ctx, rule):
+def get_features(ctx: Context, rule: Rule):
     # get features from rule and all dependencies including subscopes and matched rules
     features = []
-    namespaces = capa.rules.index_rules_by_namespace([rule])
-    deps = [ctx["rules"].rules[dep] for dep in rule.get_dependencies(namespaces)]
+    namespaces = ctx.rules.rules_by_namespace
+    deps = [ctx.rules.rules[dep] for dep in rule.get_dependencies(namespaces)]
     for r in [rule] + deps:
         features.extend(get_rule_features(r))
     return features
@@ -467,10 +702,16 @@ def get_rule_features(rule):
     return features
 
 
-LOGIC_LINTS = (DoesntMatchExample(),)
+LOGIC_LINTS = (
+    DoesntMatchExample(),
+    StatementWithSingleChildStatement(),
+    OrStatementWithAlwaysTrueChild(),
+    NotNotUnderAnd(),
+    OptionalNotUnderAnd(),
+)
 
 
-def lint_logic(ctx, rule):
+def lint_logic(ctx: Context, rule: Rule):
     return run_lints(LOGIC_LINTS, ctx, rule)
 
 
@@ -483,7 +724,7 @@ def is_nursery_rule(rule):
     return rule.meta.get("capa/nursery")
 
 
-def lint_rule(ctx, rule):
+def lint_rule(ctx: Context, rule: Rule):
     logger.debug(rule.name)
 
     violations = list(
@@ -498,61 +739,124 @@ def lint_rule(ctx, rule):
     )
 
     if len(violations) > 0:
-        category = rule.meta.get("rule-category")
+        # don't show nursery rules with a single violation: needs examples.
+        # this is by far the most common reason to be in the nursery,
+        # and ends up just producing a lot of noise.
+        if not (is_nursery_rule(rule) and len(violations) == 1 and violations[0].name == "missing examples"):
+            category = rule.meta.get("rule-category")
 
-        print("")
-        print(
-            "%s%s %s"
-            % (
-                "    (nursery) " if is_nursery_rule(rule) else "",
-                rule.name,
-                ("(%s)" % category) if category else "",
-            )
-        )
-
-        for violation in violations:
+            print("")
             print(
-                "%s  %s: %s: %s"
+                "%s%s %s"
                 % (
-                    "    " if is_nursery_rule(rule) else "",
-                    Lint.WARN if is_nursery_rule(rule) else violation.level,
-                    violation.name,
-                    violation.recommendation,
+                    "    (nursery) " if is_nursery_rule(rule) else "",
+                    rule.name,
+                    ("(%s)" % category) if category else "",
                 )
             )
 
-        print("")
+            for violation in violations:
+                print(
+                    "%s  %s: %s: %s"
+                    % (
+                        "    " if is_nursery_rule(rule) else "",
+                        Lint.WARN if is_nursery_rule(rule) else violation.level,
+                        violation.name,
+                        violation.recommendation,
+                    )
+                )
 
-    lints_failed = any(map(lambda v: v.level == Lint.FAIL, violations))
+            print("")
 
-    if not lints_failed and is_nursery_rule(rule):
-        print("")
-        print("%s%s" % ("    (nursery) ", rule.name))
-        print("%s  %s: %s: %s" % ("    ", Lint.WARN, "no lint failures", "Graduate the rule"))
-        print("")
+    if is_nursery_rule(rule):
+        has_examples = not any(map(lambda v: v.level == Lint.FAIL and v.name == "missing examples", violations))
+        lints_failed = len(
+            tuple(
+                filter(
+                    lambda v: v.level == Lint.FAIL
+                    and not (v.name == "missing examples" or v.name == "referenced example doesn't exist"),
+                    violations,
+                )
+            )
+        )
+        lints_warned = len(
+            tuple(
+                filter(
+                    lambda v: v.level == Lint.WARN
+                    or (v.level == Lint.FAIL and v.name == "referenced example doesn't exist"),
+                    violations,
+                )
+            )
+        )
 
-    return lints_failed and not is_nursery_rule(rule)
+        if (not lints_failed) and (not lints_warned) and has_examples:
+            print("")
+            print("%s%s" % ("    (nursery) ", rule.name))
+            print("%s  %s: %s: %s" % ("    ", Lint.WARN, green("no lint failures"), "Graduate the rule"))
+            print("")
+    else:
+        lints_failed = len(tuple(filter(lambda v: v.level == Lint.FAIL, violations)))
+        lints_warned = len(tuple(filter(lambda v: v.level == Lint.WARN, violations)))
+
+    return (lints_failed, lints_warned)
 
 
-def lint(ctx, rules):
+def width(s, count):
+    if len(s) > count:
+        return s[: count - 3] + "..."
+    else:
+        return s.ljust(count)
+
+
+@contextlib.contextmanager
+def redirecting_print_to_tqdm():
     """
-    Args:
-      samples (Dict[string, string]): map from sample id to path.
-        for each sample, record sample id of sha256, md5, and filename.
-        see `collect_samples(path)`.
-      rules (List[Rule]): the rules to lint.
+    tqdm (progress bar) expects to have fairly tight control over console output.
+    so calls to `print()` will break the progress bar and make things look bad.
+    so, this context manager temporarily replaces the `print` implementation
+    with one that is compatible with tqdm.
+
+    via: https://stackoverflow.com/a/42424890/87207
     """
-    did_suggest_fix = False
-    for rule in rules.rules.values():
-        if rule.meta.get("capa/subscope-rule", False):
-            continue
+    old_print = print
 
-        did_suggest_fix = lint_rule(ctx, rule) or did_suggest_fix
+    def new_print(*args, **kwargs):
 
-    return did_suggest_fix
+        # If tqdm.tqdm.write raises error, use builtin print
+        try:
+            tqdm.tqdm.write(*args, **kwargs)
+        except:
+            old_print(*args, **kwargs)
+
+    try:
+        # Globaly replace print with new_print
+        inspect.builtins.print = new_print
+        yield
+    finally:
+        inspect.builtins.print = old_print
 
 
-def collect_samples(path):
+def lint(ctx: Context):
+    """
+    Returns: Dict[string, Tuple(int, int)]
+      - # lints failed
+      - # lints warned
+    """
+    ret = {}
+
+    with tqdm.contrib.logging.tqdm_logging_redirect(ctx.rules.rules.items(), unit="rule") as pbar:
+        with redirecting_print_to_tqdm():
+            for name, rule in pbar:
+                if rule.meta.get("capa/subscope-rule", False):
+                    continue
+
+                pbar.set_description(width("linting rule: %s" % (name), 48))
+                ret[name] = lint_rule(ctx, rule)
+
+    return ret
+
+
+def collect_samples(path) -> Dict[str, Path]:
     """
     recurse through the given path, collecting all file paths, indexed by their content sha256, md5, and filename.
     """
@@ -570,10 +874,10 @@ def collect_samples(path):
             if name.endswith(".fnames"):
                 continue
 
-            path = os.path.join(root, name)
+            path = pathlib.Path(os.path.join(root, name))
 
             try:
-                with open(path, "rb") as f:
+                with path.open("rb") as f:
                     buf = f.read()
             except IOError:
                 continue
@@ -640,22 +944,35 @@ def main(argv=None):
 
     samples = collect_samples(args.samples)
 
-    ctx = {
-        "samples": samples,
-        "rules": rules,
-        "is_thorough": args.thorough,
-    }
+    ctx = Context(samples=samples, rules=rules, is_thorough=args.thorough)
 
-    did_violate = lint(ctx, rules)
+    results_by_name = lint(ctx)
+    failed_rules = []
+    warned_rules = []
+    for name, (fail_count, warn_count) in results_by_name.items():
+        if fail_count > 0:
+            failed_rules.append(name)
+
+        if warn_count > 0:
+            warned_rules.append(name)
 
     min, sec = divmod(time.time() - time0, 60)
     logger.debug("lints ran for ~ %02d:%02dm", min, sec)
 
-    if not did_violate:
-        logger.info("no lints failed, nice!")
-        return 0
-    else:
+    if warned_rules:
+        print(orange("rules with WARN:"))
+        for warned_rule in sorted(warned_rules):
+            print("  - " + warned_rule)
+        print()
+
+    if failed_rules:
+        print(red("rules with FAIL:"))
+        for failed_rule in sorted(failed_rules):
+            print("  - " + failed_rule)
         return 1
+    else:
+        logger.info(green("no lints failed, nice!"))
+        return 0
 
 
 if __name__ == "__main__":

scripts/match-function-id.py

@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+"""
+Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+You may obtain a copy of the License at: [package root]/LICENSE.txt
+Unless required by applicable law or agreed to in writing, software distributed under the License
+ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and limitations under the License.
+
+match-function-id
+
+Show the names of functions as recognized by the function identification subsystem.
+This can help identify library functions statically linked into a program,
+such as when triaging false positive matches in capa rules.
+
+Example::
+
+    $ python scripts/match-function-id.py --signature sigs/vc6.pat.gz /tmp/suspicious.dll_
+    0x44cf30: ?GetPdbDll@@YAPAUHINSTANCE__@@XZ
+    0x44bb20: ?_strlen_priv@@YAIPBD@Z
+    0x44b6b0: ?invoke_main@@YAHXZ
+    0x44a5d0: ?find_pe_section@@YAPAU_IMAGE_SECTION_HEADER@@QAEI@Z
+    0x44a690: ?is_potentially_valid_image_base@@YA_NQAX@Z
+    0x44cbe0: ___get_entropy
+    0x44a4a0: __except_handler4
+    0x44b3d0: ?pre_cpp_initialization@@YAXXZ
+    0x44b2e0: ?pre_c_initialization@@YAHXZ
+    0x44b3c0: ?post_pgo_initialization@@YAHXZ
+    0x420156: ?
+    0x420270: ?
+    0x430dcd: ?
+    0x44d930: __except_handler4_noexcept
+    0x41e960: ?
+    0x44a1e0: @_RTC_AllocaHelper@12
+    0x44ba90: ?_getMemBlockDataString@@YAXPAD0PBDI@Z
+    0x44a220: @_RTC_CheckStackVars2@12
+    0x44a790: ___scrt_dllmain_after_initialize_c
+    0x44a7d0: ___scrt_dllmain_before_initialize_c
+    0x44a800: ___scrt_dllmain_crt_thread_attach
+    0x44a860: ___scrt_dllmain_exception_filter
+    0x44a900: ___scrt_dllmain_uninitialize_critical
+    0x44ad10: _at_quick_exit
+    0x44b940: ?_RTC_Failure@@YAXPAXH@Z
+    0x44be60: __RTC_UninitUse
+    0x44bfd0: __RTC_GetErrDesc
+    0x44c060: __RTC_SetErrorType
+    0x44cb60: ?
+    0x44cba0: __guard_icall_checks_enforced
+"""
+import sys
+import logging
+import argparse
+
+import flirt
+import viv_utils
+import viv_utils.flirt
+
+import capa.main
+import capa.rules
+import capa.engine
+import capa.helpers
+import capa.features
+import capa.features.freeze
+
+logger = logging.getLogger("capa.match-function-id")
+
+
+def main(argv=None):
+    if argv is None:
+        argv = sys.argv[1:]
+
+    parser = argparse.ArgumentParser(description="FLIRT match each function")
+    parser.add_argument("sample", type=str, help="Path to sample to analyze")
+    parser.add_argument(
+        "-F",
+        "--function",
+        type=lambda x: int(x, 0x10),
+        help="match a specific function by VA, rather than add functions",
+    )
+    parser.add_argument(
+        "--signature",
+        action="append",
+        dest="signatures",
+        type=str,
+        default=[],
+        help="use the given signatures to identify library functions, file system paths to .sig/.pat files.",
+    )
+    parser.add_argument("-d", "--debug", action="store_true", help="Enable debugging output on STDERR")
+    parser.add_argument("-q", "--quiet", action="store_true", help="Disable all output but errors")
+    args = parser.parse_args(args=argv)
+
+    if args.quiet:
+        logging.basicConfig(level=logging.ERROR)
+        logging.getLogger().setLevel(logging.ERROR)
+    elif args.debug:
+        logging.basicConfig(level=logging.DEBUG)
+        logging.getLogger().setLevel(logging.DEBUG)
+    else:
+        logging.basicConfig(level=logging.INFO)
+        logging.getLogger().setLevel(logging.INFO)
+
+    # disable vivisect-related logging, it's verbose and not relevant for capa users
+    capa.main.set_vivisect_log_level(logging.CRITICAL)
+
+    analyzers = []
+    for sigpath in args.signatures:
+        sigs = viv_utils.flirt.load_flirt_signature(sigpath)
+
+        with capa.main.timing("flirt: compiling sigs"):
+            matcher = flirt.compile(sigs)
+
+        analyzer = viv_utils.flirt.FlirtFunctionAnalyzer(matcher, sigpath)
+        logger.debug("registering viv function analyzer: %s", repr(analyzer))
+        analyzers.append(analyzer)
+
+    vw = viv_utils.getWorkspace(args.sample, analyze=True, should_save=False)
+
+    functions = vw.getFunctions()
+    if args.function:
+        functions = [args.function]
+
+    for function in functions:
+        logger.debug("matching function: 0x%04x", function)
+        for analyzer in analyzers:
+            name = viv_utils.flirt.match_function_flirt_signatures(analyzer.matcher, vw, function)
+            if name:
+                print("0x%04x: %s" % (function, name))
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/profile-memory.py

@@ -0,0 +1,73 @@
+import gc
+import linecache
+import tracemalloc
+
+tracemalloc.start()
+
+
+def display_top(snapshot, key_type="lineno", limit=10):
+    # via: https://docs.python.org/3/library/tracemalloc.html#pretty-top
+    snapshot = snapshot.filter_traces(
+        (
+            tracemalloc.Filter(False, "<frozen importlib._bootstrap_external>"),
+            tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
+            tracemalloc.Filter(False, "<unknown>"),
+        )
+    )
+    top_stats = snapshot.statistics(key_type)
+
+    print("Top %s lines" % limit)
+    for index, stat in enumerate(top_stats[:limit], 1):
+        frame = stat.traceback[0]
+        print("#%s: %s:%s: %.1f KiB" % (index, frame.filename, frame.lineno, stat.size / 1024))
+        line = linecache.getline(frame.filename, frame.lineno).strip()
+        if line:
+            print("    %s" % line)
+
+    other = top_stats[limit:]
+    if other:
+        size = sum(stat.size for stat in other)
+        print("%s other: %.1f KiB" % (len(other), size / 1024))
+    total = sum(stat.size for stat in top_stats)
+    print("Total allocated size: %.1f KiB" % (total / 1024))
+
+
+def main():
+    # import within main to keep isort happy
+    # while also invoking tracemalloc.start() immediately upon start.
+    import io
+    import os
+    import time
+    import contextlib
+
+    import psutil
+
+    import capa.main
+
+    count = int(os.environ.get("CAPA_PROFILE_COUNT", 1))
+    print("total iterations planned: %d (set via env var CAPA_PROFILE_COUNT)." % (count))
+    print()
+
+    for i in range(count):
+        print("iteration %d/%d..." % (i + 1, count))
+        with contextlib.redirect_stdout(io.StringIO()):
+            with contextlib.redirect_stderr(io.StringIO()):
+                t0 = time.time()
+                capa.main.main()
+                t1 = time.time()
+
+                gc.collect()
+
+        process = psutil.Process(os.getpid())
+        print("  duration: %0.02fs" % (t1 - t0))
+        print("  rss: %.1f MiB" % (process.memory_info().rss / 1024 / 1024))
+        print("  vms: %.1f MiB" % (process.memory_info().vms / 1024 / 1024))
+
+    print("done.")
+    gc.collect()
+
+    snapshot0 = tracemalloc.take_snapshot()
+    display_top(snapshot0)
+
+
+main()

scripts/profile-time.py

@@ -0,0 +1,150 @@
+"""
+Invoke capa multiple times and record profiling informations.
+Use the --number and --repeat options to change the number of iterations.
+By default, the script will emit a markdown table with a label pulled from git.
+
+Note: you can run this script against pre-generated .frz files to reduce the startup time.
+
+usage:
+
+    usage: profile-time.py [--number NUMBER] [--repeat REPEAT] [--label LABEL] sample
+
+    Profile capa performance
+
+    positional arguments:
+      sample                path to sample to analyze
+
+    optional arguments:
+      --number NUMBER       batch size of profile collection
+      --repeat REPEAT       batch count of profile collection
+      --label LABEL         description of the profile collection
+
+example:
+
+    $ python profile-time.py ./tests/data/kernel32.dll_.frz --number 1 --repeat 2
+
+    | label                                | count(evaluations)   | avg(time)   | min(time)   | max(time)   |
+    |--------------------------------------|----------------------|-------------|-------------|-------------|
+    | 18c30e4 main: remove perf debug msgs | 66,561,622           | 132.13s     | 125.14s     | 139.12s     |
+
+      ^^^ --label or git hash               
+"""
+import sys
+import timeit
+import logging
+import argparse
+import subprocess
+
+import tqdm
+import tabulate
+
+import capa.main
+import capa.perf
+import capa.rules
+import capa.engine
+import capa.helpers
+import capa.features
+import capa.features.common
+import capa.features.freeze
+
+logger = logging.getLogger("capa.profile")
+
+
+def main(argv=None):
+    if argv is None:
+        argv = sys.argv[1:]
+
+    label = subprocess.run(
+        "git show --pretty=oneline --abbrev-commit | head -n 1", shell=True, capture_output=True, text=True
+    ).stdout.strip()
+    is_dirty = (
+        subprocess.run(
+            "git status | grep 'modified: ' | grep -v 'rules' | grep -v 'tests/data'",
+            shell=True,
+            capture_output=True,
+            text=True,
+        ).stdout
+        != ""
+    )
+
+    if is_dirty:
+        label += " (dirty)"
+
+    parser = argparse.ArgumentParser(description="Profile capa performance")
+    capa.main.install_common_args(parser, wanted={"format", "sample", "signatures", "rules"})
+
+    parser.add_argument("--number", type=int, default=3, help="batch size of profile collection")
+    parser.add_argument("--repeat", type=int, default=30, help="batch count of profile collection")
+    parser.add_argument("--label", type=str, default=label, help="description of the profile collection")
+
+    args = parser.parse_args(args=argv)
+    capa.main.handle_common_args(args)
+
+    try:
+        taste = capa.helpers.get_file_taste(args.sample)
+    except IOError as e:
+        logger.error("%s", str(e))
+        return -1
+
+    try:
+        with capa.main.timing("load rules"):
+            rules = capa.rules.RuleSet(capa.main.get_rules(args.rules, disable_progress=True))
+    except (IOError) as e:
+        logger.error("%s", str(e))
+        return -1
+
+    try:
+        sig_paths = capa.main.get_signatures(args.signatures)
+    except (IOError) as e:
+        logger.error("%s", str(e))
+        return -1
+
+    if (args.format == "freeze") or (args.format == "auto" and capa.features.freeze.is_freeze(taste)):
+        with open(args.sample, "rb") as f:
+            extractor = capa.features.freeze.load(f.read())
+    else:
+        extractor = capa.main.get_extractor(
+            args.sample, args.format, capa.main.BACKEND_VIV, sig_paths, should_save_workspace=False
+        )
+
+    with tqdm.tqdm(total=args.number * args.repeat) as pbar:
+
+        def do_iteration():
+            capa.perf.reset()
+            capa.main.find_capabilities(rules, extractor, disable_progress=True)
+            pbar.update(1)
+
+        samples = timeit.repeat(do_iteration, number=args.number, repeat=args.repeat)
+
+    logger.debug("perf: find capabilities: min: %0.2fs" % (min(samples) / float(args.number)))
+    logger.debug("perf: find capabilities: avg: %0.2fs" % (sum(samples) / float(args.repeat) / float(args.number)))
+    logger.debug("perf: find capabilities: max: %0.2fs" % (max(samples) / float(args.number)))
+
+    for (counter, count) in capa.perf.counters.most_common():
+        logger.debug("perf: counter: {:}: {:,}".format(counter, count))
+
+    print(
+        tabulate.tabulate(
+            [
+                (
+                    args.label,
+                    "{:,}".format(capa.perf.counters["evaluate.feature"]),
+                    # python documentation indicates that min(samples) should be preferred,
+                    # so lets put that first.
+                    #
+                    # https://docs.python.org/3/library/timeit.html#timeit.Timer.repeat
+                    "%0.2fs" % (min(samples) / float(args.number)),
+                    "%0.2fs" % (sum(samples) / float(args.repeat) / float(args.number)),
+                    "%0.2fs" % (max(samples) / float(args.number)),
+                )
+            ],
+            headers=["label", "count(evaluations)", "min(time)", "avg(time)", "max(time)"],
+            tablefmt="github",
+        )
+    )
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/setup-hooks.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at: [package root]/LICENSE.txt