fix black errors

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

.bumpversion.toml

@@ -0,0 +1,27 @@
+[tool.bumpversion]
+current_version = "9.3.1"
+
+[[tool.bumpversion.files]]
+filename = "capa/version.py"
+search = '__version__ = "{current_version}"'
+replace = '__version__ = "{new_version}"'
+
+[[tool.bumpversion.files]]
+filename = "capa/ida/plugin/ida-plugin.json"
+search = '"version": "{current_version}"'
+replace = '"version": "{new_version}"'
+
+[[tool.bumpversion.files]]
+filename = "capa/ida/plugin/ida-plugin.json"
+search = '"flare-capa=={current_version}"'
+replace = '"flare-capa=={new_version}"'
+
+[[tool.bumpversion.files]]
+filename = "CHANGELOG.md"
+search = "v{current_version}...master"
+replace = "v{current_version}...{new_version}"
+
+[[tool.bumpversion.files]]
+filename = "CHANGELOG.md"
+search = "master (unreleased)"
+replace = "v{new_version}"

.github/CODE_OF_CONDUCT.md

@@ -1,46 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://contributor-covenant.org/version/1/4][version]
-
-[homepage]: https://contributor-covenant.org
-[version]: https://contributor-covenant.org/version/1/4/

.github/CONTRIBUTING.md

@@ -25,7 +25,7 @@ The following is a set of guidelines for contributing to capa and its packages,
 
 ## Code of Conduct
 
-This project and everyone participating in it is governed by the [Capa Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the maintainers.
+This project follows [Google's Open Source Community Guidelines](https://opensource.google/conduct).
 
 ## What should I know before I get started?
 
@@ -168,15 +168,17 @@ While the prerequisites above must be satisfied prior to having your pull reques
 
 ### Contributor License Agreement
 
-Contributions to this project must be accompanied by a Contributor License
-Agreement. You (or your employer) retain the copyright to your contribution,
-this simply gives us permission to use and redistribute your contributions as
-part of the project. Head over to <https://cla.developers.google.com/> to see
-your current agreements on file or to sign a new one.
+Contributions to this project must be accompanied by a
+[Contributor License Agreement](https://cla.developers.google.com/about) (CLA).
+You (or your employer) retain the copyright to your contribution; this simply
+gives us permission to use and redistribute your contributions as part of the
+project.
 
-You generally only need to submit a CLA once, so if you've already submitted one
-(even if it was for a different project), you probably don't need to do it
-again.
+If you or your current employer have already signed the Google CLA (even if it
+was for a different project), you probably don't need to do it again.
+
+Visit <https://cla.developers.google.com/> to see your current agreements or to
+sign a new one.
 
 ## Styleguides
 

.github/ISSUE_TEMPLATE/bug_report.md

@@ -10,8 +10,8 @@ We use submodules to separate code, rules and test data. If your issue is relate
 # Have you checked that your issue isn't already filed?
 Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
-# Have you read capa's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
+# Have you read Google's Code of Conduct?
+By filing an issue, you are expected to comply with it, including treating everyone with respect: https://opensource.google/conduct
 
 # Have you read capa's CONTRIBUTING guide?
 It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs

.github/ISSUE_TEMPLATE/feature_request.md

@@ -10,8 +10,8 @@ We use submodules to separate code, rules and test data. If your issue is relate
 # Have you checked that your issue isn't already filed?
 Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
-# Have you read capa's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
+# Have you read Google's Code of Conduct?
+By filing an issue, you are expected to comply with it, including treating everyone with respect: https://opensource.google/conduct
 
 # Have you read capa's CONTRIBUTING guide?
 It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements

.github/flake8.ini

@@ -40,4 +40,4 @@ per-file-ignores =
 
 copyright-check = True
 copyright-min-file-size = 1 
-copyright-regexp = Copyright \(C\) \d{4} Mandiant, Inc. All Rights Reserved.
+copyright-regexp = Copyright \d{4} Google LLC

.github/pyinstaller/hooks/hook-vivisect.py

@@ -1,4 +1,16 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 from PyInstaller.utils.hooks import copy_metadata
 

.github/pyinstaller/pyinstaller.spec

@@ -1,5 +1,18 @@
 # -*- mode: python -*-
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import sys
 
 import capa.rules.cache
@@ -61,6 +74,9 @@ a = Analysis(
         # only be installed locally.
         "binaryninja",
         "ida",
+        # remove once https://github.com/mandiant/capa/issues/2681 has
+        # been addressed by PyInstaller
+        "pkg_resources",
     ],
 )
 

.github/workflows/build.yml

@@ -9,6 +9,7 @@ on:
       - '**.md'
   release:
     types: [edited, published]
+  workflow_dispatch: # manual trigger for testing
 
 permissions:
   contents: write
@@ -22,24 +23,38 @@ jobs:
       fail-fast: true
       matrix:
         include:
-          - os: ubuntu-20.04
+          - os: ubuntu-22.04
             # use old linux so that the shared library versioning is more portable
             artifact_name: capa
             asset_name: linux
             python_version: '3.10'
-          - os: ubuntu-20.04
+          - os: ubuntu-22.04-arm
+            artifact_name: capa
+            asset_name: linux-arm64
+            python_version: '3.10'
+          - os: ubuntu-22.04
             artifact_name: capa
             asset_name: linux-py312
             python_version: '3.12'
-          - os: windows-2019
+          - os: windows-2022
             artifact_name: capa.exe
             asset_name: windows
             python_version: '3.10'
-          - os: macos-13
-            # use older macOS for assumed better portability
+          # Windows 11 ARM64 complains of conflicting package version
+          # Additionally, there is no ARM64 build of Python for Python 3.10 on Windows 11 ARM: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
+          #- os: windows-11-arm
+          #  artifact_name: capa.exe
+          #  asset_name: windows-arm64
+          #  python_version: '3.12'
+          - os: macos-15-intel
+            # macos-15-intel is the lowest native intel build
             artifact_name: capa
             asset_name: macos
             python_version: '3.10'
+          - os: macos-14
+            artifact_name: capa
+            asset_name: macos-arm64
+            python_version: '3.10'
     steps:
       - name: Checkout capa
         uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -49,7 +64,7 @@ jobs:
         uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
         with:
           python-version: ${{ matrix.python_version }}
-      - if: matrix.os == 'ubuntu-20.04'
+      - if: matrix.os == 'ubuntu-22.04' || matrix.os == 'ubuntu-22.04-arm'
         run: sudo apt-get install -y libyaml-dev
       - name: Upgrade pip, setuptools
         run: python -m pip install --upgrade pip setuptools
@@ -59,6 +74,28 @@ jobs:
           pip install -e .[build]
       - name: Build standalone executable
         run: pyinstaller --log-level DEBUG .github/pyinstaller/pyinstaller.spec
+      - name: Does it run without warnings or errors?
+        shell: bash
+        run: |
+          if [[ "${{ matrix.os }}" == "windows-2022" ]] || [[ "${{ matrix.os }}" == "windows-11-arm" ]]; then
+            EXECUTABLE=".\\dist\\capa"
+          else
+            EXECUTABLE="./dist/capa"
+          fi
+
+          output=$(${EXECUTABLE} --version 2>&1)
+          exit_code=$?
+
+          echo "${output}"
+          echo "${exit_code}"
+
+          if echo "${output}" | grep -iE 'error|warning'; then
+            exit 1
+          fi
+
+          if [[ "${exit_code}" -ne 0 ]]; then
+            exit 1
+          fi
       - name: Does it run (PE)?
         run: dist/capa -d "tests/data/Practical Malware Analysis Lab 01-01.dll_"
       - name: Does it run (Shellcode)?
@@ -74,34 +111,6 @@ jobs:
           name: ${{ matrix.asset_name }}
           path: dist/${{ matrix.artifact_name }}
 
-  test_run:
-    name: Test run on ${{ matrix.os }} / ${{ matrix.asset_name }}
-    runs-on: ${{ matrix.os }}
-    needs: [build]
-    strategy:
-      matrix:
-        include:
-          # OSs not already tested above
-          - os: ubuntu-22.04
-            artifact_name: capa
-            asset_name: linux
-          - os: ubuntu-22.04
-            artifact_name: capa
-            asset_name: linux-py312
-          - os: windows-2022
-            artifact_name: capa.exe
-            asset_name: windows
-    steps:
-      - name: Download ${{ matrix.asset_name }}
-        uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe # v4.1.2
-        with:
-          name: ${{ matrix.asset_name }}
-      - name: Set executable flag
-        if: matrix.os != 'windows-2022'
-        run: chmod +x ${{ matrix.artifact_name }}
-      - name: Run capa
-        run: ./${{ matrix.artifact_name }} -h
-
   zip_and_upload:
     # upload zipped binaries to Release page
     if: github.event_name == 'release'
@@ -113,12 +122,18 @@ jobs:
         include:
           - asset_name: linux
             artifact_name: capa
+          - asset_name: linux-arm64
+            artifact_name: capa
           - asset_name: linux-py312
             artifact_name: capa
           - asset_name: windows
             artifact_name: capa.exe
+          #- asset_name: windows-arm64
+          #  artifact_name: capa.exe
           - asset_name: macos
             artifact_name: capa
+          - asset_name: macos-arm64
+            artifact_name: capa
     steps:
       - name: Download ${{ matrix.asset_name }}
         uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe # v4.1.2

.github/workflows/publish.yml

@@ -35,7 +35,7 @@ jobs:
         with:
           path: dist/*
       - name: publish package
-        uses: pypa/gh-action-pypi-publish@f5622bde02b04381239da3573277701ceca8f6a0  # release/v1
+        uses: pypa/gh-action-pypi-publish@76f52bc884231f62b9a034ebfe128415bbaabdfc  # release/v1.12.4
         with:
           skip-existing: true
           verbose: true

.github/workflows/tests.yml

@@ -42,10 +42,10 @@ jobs:
     - name: Checkout capa
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
     # use latest available python to take advantage of best performance
-    - name: Set up Python 3.12
+    - name: Set up Python 3.13
       uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
       with:
-        python-version: "3.12"
+        python-version: "3.13"
     - name: Install dependencies
       run: |
         pip install -r requirements.txt
@@ -70,10 +70,10 @@ jobs:
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
       with:
         submodules: recursive
-    - name: Set up Python 3.12
+    - name: Set up Python 3.13
       uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
       with:
-        python-version: "3.12"
+        python-version: "3.13"
     - name: Install capa
       run: |
         pip install -r requirements.txt
@@ -88,16 +88,14 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-20.04, windows-2019, macos-13]
+        os: [ubuntu-22.04, ubuntu-22.04-arm, windows-2022, macos-15-intel, macos-14]
         # across all operating systems
-        python-version: ["3.10", "3.11"]
+        python-version: ["3.10", "3.13"]
         include:
           # on Ubuntu run these as well
-          - os: ubuntu-20.04
-            python-version: "3.10"
-          - os: ubuntu-20.04
+          - os: ubuntu-22.04
             python-version: "3.11"
-          - os: ubuntu-20.04
+          - os: ubuntu-22.04
             python-version: "3.12"
     steps:
     - name: Checkout capa with submodules
@@ -109,7 +107,7 @@ jobs:
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install pyyaml
-      if: matrix.os == 'ubuntu-20.04'
+      if: matrix.os == 'ubuntu-22.04'
       run: sudo apt-get install -y libyaml-dev
     - name: Install capa
       run: |
@@ -131,7 +129,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11"]
+        python-version: ["3.10", "3.13"]
     steps:
     - name: Checkout capa with submodules
       # do only run if BN_SERIAL is available, have to do this in every step, see https://github.com/orgs/community/discussions/26726#discussioncomment-3253118
@@ -168,16 +166,15 @@ jobs:
 
   ghidra-tests:
     name: Ghidra tests for ${{ matrix.python-version }}
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     needs: [tests]
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11"]
-        java-version: ["17"]
-        ghidra-version: ["11.0.1"]
-        public-version: ["PUBLIC_20240130"] # for ghidra releases
-        ghidrathon-version: ["4.0.0"] 
+        python-version: ["3.10", "3.13"]
+        java-version: ["21"]
+        ghidra-version: ["11.4"]
+        public-version: ["PUBLIC_20250620"] # for ghidra releases 
     steps:
     - name: Checkout capa with submodules
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -197,25 +194,13 @@ jobs:
         mkdir ./.github/ghidra
         wget "https://github.com/NationalSecurityAgency/ghidra/releases/download/Ghidra_${{ matrix.ghidra-version }}_build/ghidra_${{ matrix.ghidra-version }}_${{ matrix.public-version }}.zip" -O ./.github/ghidra/ghidra_${{ matrix.ghidra-version }}_PUBLIC.zip
         unzip .github/ghidra/ghidra_${{ matrix.ghidra-version }}_PUBLIC.zip -d .github/ghidra/
-    - name: Install Ghidrathon
-      run : |
-        mkdir ./.github/ghidrathon
-        wget "https://github.com/mandiant/Ghidrathon/releases/download/v${{ matrix.ghidrathon-version }}/Ghidrathon-v${{ matrix.ghidrathon-version}}.zip" -O ./.github/ghidrathon/ghidrathon-v${{ matrix.ghidrathon-version }}.zip
-        unzip .github/ghidrathon/ghidrathon-v${{ matrix.ghidrathon-version }}.zip -d .github/ghidrathon/
-        python -m pip install -r .github/ghidrathon/requirements.txt
-        python .github/ghidrathon/ghidrathon_configure.py $(pwd)/.github/ghidra/ghidra_${{ matrix.ghidra-version }}_PUBLIC
-        unzip .github/ghidrathon/Ghidrathon-v${{ matrix.ghidrathon-version }}.zip -d .github/ghidra/ghidra_${{ matrix.ghidra-version }}_PUBLIC/Ghidra/Extensions
     - name: Install pyyaml
       run: sudo apt-get install -y libyaml-dev
     - name: Install capa
       run: |
-        pip install -r requirements.txt
-        pip install -e .[dev,scripts]
+        pip install -e .[dev]
     - name: Run tests
-      run: | 
-        mkdir ./.github/ghidra/project
-        .github/ghidra/ghidra_${{ matrix.ghidra-version }}_PUBLIC/support/analyzeHeadless .github/ghidra/project ghidra_test -Import ./tests/data/mimikatz.exe_ -ScriptPath ./tests/ -PostScript test_ghidra_features.py > ../output.log
-        cat ../output.log
-        exit_code=$(cat ../output.log | grep exit | awk '{print $NF}')
-        exit $exit_code
+      env:
+        GHIDRA_INSTALL_DIR: ${{ github.workspace }}/.github/ghidra/ghidra_${{ matrix.ghidra-version }}_PUBLIC
+      run: pytest -v tests/test_ghidra_features.py
  

.github/workflows/web-release.yml

@@ -69,11 +69,35 @@ jobs:
       run: ls -t capa-explorer-web-v*.zip | tail -n +4 | xargs -r rm --
       working-directory: web/explorer/releases
 
-    - name: Commit and push release
+    - name: Stage release files
       run: |
         git config --local user.email "capa-dev@mandiant.com"
         git config --local user.name "Capa Bot"
         git add -f web/explorer/releases/${{ env.RELEASE_NAME }}.zip web/explorer/releases/CHANGELOG.md
         git add -u web/explorer/releases/
-        git commit -m ":robot: explorer web: add release ${{ env.RELEASE_NAME }}"
-        git push
+
+    - name: Create Pull Request
+      uses: peter-evans/create-pull-request@5e914681df9dc83aa4e4905692ca88beb2f9e91f # v7.0.5
+      with:
+        token: ${{ secrets.GITHUB_TOKEN }}
+        title: "explorer web: add release v${{ github.event.inputs.version }}"
+        body: |
+          This PR adds a new capa Explorer Web release v${{ github.event.inputs.version }}.
+
+          Release details:
+          - Name: ${{ env.RELEASE_NAME }}
+          - SHA256: ${{ env.RELEASE_SHA256 }}
+
+          This release is generated by the [web release](https://github.com/mandiant/capa/actions/workflows/web-release.yml) workflow.
+
+          - [x] No CHANGELOG update needed
+          - [x] No new tests needed
+          - [x] No documentation update needed
+        commit-message: ":robot: explorer web: add release ${{ env.RELEASE_NAME }}"
+        branch: release/web-v${{ github.event.inputs.version }}
+        add-paths: web/explorer/releases/${{ env.RELEASE_NAME }}.zip
+        base: master
+        labels: webui
+        delete-branch: true
+        committer: Capa Bot <capa-dev@mandiant.com>
+        author: Capa Bot <capa-dev@mandiant.com>

.gitignore

@@ -122,6 +122,7 @@ scripts/perf/*.zip
 */.DS_Store
 Pipfile
 Pipfile.lock
+uv.lock
 /cache/
 .github/binja/binaryninja
 .github/binja/download_headless.py

.pre-commit-config.yaml

@@ -25,7 +25,7 @@ repos:
     hooks:
     -   id: isort
         name: isort
-        stages: [commit, push, manual]
+        stages: [pre-commit, pre-push, manual]
         language: system
         entry: isort
         args: 
@@ -46,7 +46,7 @@ repos:
     hooks:
     -   id: black
         name: black
-        stages: [commit, push, manual]
+        stages: [pre-commit, pre-push, manual]
         language: system
         entry: black
         args: 
@@ -64,7 +64,7 @@ repos:
     hooks:
     -   id: ruff
         name: ruff
-        stages: [commit, push, manual]
+        stages: [pre-commit, pre-push, manual]
         language: system
         entry: ruff
         args: 
@@ -82,7 +82,7 @@ repos:
     hooks:
     -   id: flake8
         name: flake8
-        stages: [push, manual]
+        stages: [pre-push, manual]
         language: system
         entry: flake8
         args: 
@@ -101,7 +101,7 @@ repos:
     hooks:
     -   id: mypy
         name: mypy
-        stages: [push, manual]
+        stages: [pre-push, manual]
         language: system
         entry: mypy
         args: 
@@ -119,7 +119,7 @@ repos:
     hooks:
     -   id: deptry
         name: deptry
-        stages: [push, manual]
+        stages: [pre-push, manual]
         language: system
         entry: deptry .
         always_run: true

CHANGELOG.md

@@ -4,13 +4,279 @@
 
 ### New Features
 
+- ghidra: support PyGhidra @mike-hunhoff #2788
+
+### Breaking Changes
+
+### New Rules (4)
+
+- nursery/run-as-nodejs-native-module mehunhoff@google.com
+- nursery/inject-shellcode-using-thread-pool-work-insertion-with-tp_io still@teamt5.org
+- nursery/inject-shellcode-using-thread-pool-work-insertion-with-tp_timer still@teamt5.org
+- nursery/inject-shellcode-using-thread-pool-work-insertion-with-tp_work still@teamt5.org
+-
+
+### Bug Fixes
+- Fixed insecure deserialization vulnerability in YAML loading @0x1622 (#2770)
+
+### capa Explorer Web
+
+### capa Explorer IDA Pro plugin
+
+### Development
+
+- ci: deprecate macos-13 runner and use Python v3.13 for testing @mike-hunhoff #2777
+
+### Raw diffs
+- [capa v9.3.1...master](https://github.com/mandiant/capa/compare/v9.3.1...master)
+- [capa-rules v9.3.1...master](https://github.com/mandiant/capa-rules/compare/v9.3.1...master)
+
+## v9.3.1
+
+This patch release fixes a missing import for the capa explorer plugin for IDA Pro.
+
+### Bug Fixes
+
+- add missing ida-netnode dependency to project.toml @mike-hunhoff #2765
+
+### Development
+
+- ci: bump binja min version @mike-hunhoff #2763
+
+### Raw diffs
+- [capa v9.3.0...master](https://github.com/mandiant/capa/compare/v9.3.0...master)
+- [capa-rules v9.3.0...master](https://github.com/mandiant/capa-rules/compare/v9.3.0...master)
+
+## v9.3.0
+
+capa v9.3.0 comes with over 20 new and/or impoved rules.
+For IDA users the capa explorer plugin is now available via the IDA Pro plugin repository and contains Qt compatibility layer for PyQt5 and PySide6 support.
+Additionally a Binary Ninja bug has been fixed. Released binaries now include ARM64 binaries (Linux and macOS).
+
+### New Features
+
+- ci: add support for arm64 binary releases
+
+### Breaking Changes
+
+### New Rules (24)
+
+- anti-analysis/anti-vm/vm-detection/detect-mouse-movement-via-activity-checks-on-windows tevajdr@gmail.com
+- nursery/create-executable-heap moritz.raabe@mandiant.com
+- anti-analysis/packer/dxpack/packed-with-dxpack jakubjozwiak@google.com
+- anti-analysis/anti-av/patch-bitdefender-hooking-dll-function jakubjozwiak@google.com
+- nursery/acquire-load-driver-privileges mehunhoff@google.com
+- nursery/communicate-using-ftp mehunhoff@google.com
+- linking/static/eclipse-paho-mqtt-c/linked-against-eclipse-paho-mqtt-c jakubjozwiak@google.com
+- linking/static/qmqtt/linked-against-qmqtt jakubjozwiak@google.com
+- anti-analysis/anti-forensic/disable-powershell-transcription jakubjozwiak@google.com
+- host-interaction/powershell/bypass-powershell-constrained-language-mode-via-getsystemlockdownpolicy-patch jakubjozwiak@google.com
+- linking/static/grpc/linked-against-grpc jakubjozwiak@google.com
+- linking/static/hp-socket/linked-against-hp-socket jakubjozwiak@google.com
+- load-code/execute-jscript-via-vsaengine-in-dotnet jakubjozwiak@google.com
+- linking/static/funchook/linked-against-funchook jakubjozwiak@google.com
+- linking/static/plthook/linked-against-plthook jakubjozwiak@google.com
+- host-interaction/network/enumerate-tcp-connections-via-wmi-com-api jakubjozwiak@google.com
+- host-interaction/network/routing-table/create-routing-table-entry jakubjozwiak@google.com
+- host-interaction/network/routing-table/get-routing-table michael.hunhoff@mandiant.com
+- host-interaction/file-system/use-io_uring-io-interface-on-linux jakubjozwiak@google.com
+- collection/keylog/log-keystrokes-via-direct-input zeze-zeze
+- nursery/compiled-from-fsharp mehunhoff@google.com
+- nursery/decrypt-data-using-aes-via-dotnet mehunhoff@google.com
+- nursery/get-dotnet-assembly-entry-point mehunhoff@google.com
+
+### Bug Fixes
+
+- binja: fix a crash during feature extraction when the MLIL is unavailable @xusheng6 #2714 
+
+### capa Explorer Web
+
+### capa Explorer IDA Pro plugin
+
+- add `ida-plugin.json` for inclusion in the IDA Pro plugin repository @williballenthin
+- ida plugin: add Qt compatibility layer for PyQt5 and PySide6 support @williballenthin #2707
+- delay import to not load Qt* when running under idalib @mr-tz #2752
+
+### Development
+
+- ci: remove redundant "test_run" action from build workflow @mike-hunhoff #2692
+- dev: add bumpmyversion to bump and sync versions across the project @mr-tz
+
+### Raw diffs
+- [capa v9.2.1...9.3.0](https://github.com/mandiant/capa/compare/v9.2.1...9.3.0)
+- [capa-rules v9.2.1...9.3.0](https://github.com/mandiant/capa-rules/compare/v9.2.1...9.3.0)
+
+## v9.2.1
+
+This point release fixes bugs including removing an unnecessary PyInstaller warning message and enabling the standalone binary to execute on systems running older versions of glibc.
+
+### Bug Fixes
+
+- ci: exclude pkg_resources from PyInstaller build @mike-hunhoff #2684
+- ci: downgrade Ubuntu version to accommodate older glibc versions @mike-hunhoff #2684
+
+### Development
+
+- ci: upgrade Windows version to avoid deprecation @mike-hunhoff #2684
+- ci: check if build runs without warnings or errors @mike-hunhoff #2684
+
+### Raw diffs
+- [capa v9.2.0...v9.2.1](https://github.com/mandiant/capa/compare/v9.2.0...v9.2.1)
+- [capa-rules v9.2.0...v9.2.1](https://github.com/mandiant/capa-rules/compare/v9.2.0...v9.2.1)
+
+## v9.2.0
+
+This release improves a few aspects of dynamic analysis, including relaxing our validation on fields across many CAPE versions and processing additional VMRay submission file types, for example.
+It also includes an updated rule pack containing new rules and rule fixes.
+
+### New Features
+- vmray: do not restrict analysis to PE and ELF files, e.g. docx @mike-hunhoff #2672
+
+### Breaking Changes
+
+### New Rules (22)
+
+- communication/socket/connect-socket moritz.raabe@mandiant.com joakim@intezer.com mrhafizfarhad@gmail.com
+- communication/socket/udp/connect-udp-socket mrhafizfarhad@gmail.com
+- nursery/enter-debug-mode-in-dotnet @v1bh475u
+- nursery/decrypt-data-using-tripledes-in-dotnet 0xRavenspar
+- nursery/encrypt-data-using-tripledes-in-dotnet 0xRavenspar
+- nursery/disable-system-features-via-registry-on-windows mehunhoff@google.com
+- data-manipulation/encryption/chaskey/encrypt-data-using-chaskey still@teamt5.org
+- data-manipulation/encryption/speck/encrypt-data-using-speck still@teamt5.org
+- load-code/dotnet/load-assembly-via-iassembly still@teamt5.org
+- malware-family/donut-loader/load-shellcode-via-donut still@teamt5.org
+- nursery/disable-device-guard-features-via-registry-on-windows mehunhoff@google.com
+- nursery/disable-firewall-features-via-registry-on-windows mehunhoff@google.com
+- nursery/disable-system-restore-features-via-registry-on-windows mehunhoff@google.com
+- nursery/disable-windows-defender-features-via-registry-on-windows mehunhoff@google.com
+- host-interaction/file-system/write/clear-file-content jakeperalta7
+- host-interaction/filter/unload-minifilter-driver JakePeralta7
+- exploitation/enumeration/make-suspicious-ntquerysysteminformation-call zdw@google.com
+- exploitation/gadgets/load-ntoskrnl zdw@google.com
+- exploitation/gadgets/resolve-ntoskrnl-gadgets zdw@google.com
+- exploitation/spraying/make-suspicious-ntfscontrolfile-call zdw@google.com
+- anti-analysis/anti-forensic/unload-sysmon JakePeralta7
+
+### Bug Fixes
+- cape: make some fields optional @williballenthin #2631 #2632
+- lint: add WARN for regex features that contain unescaped dot #2635
+- lint: add ERROR for incomplete registry control set regex #2643
+- binja: update unit test core version #2670
+
+### Raw diffs
+- [capa v9.1.0...v9.2.0](https://github.com/mandiant/capa/compare/v9.1.0...v9.2.0)
+- [capa-rules v9.1.0...v9.2.0](https://github.com/mandiant/capa-rules/compare/v9.1.0...v9.2.0)
+
+## v9.1.0
+
+This release improves a few aspects of dynamic analysis, relaxing our validation on fields across many CAPE versions, for example.
+It also includes an updated rule pack in which many dynamic rules make better use of the "span of calls" scope.
+
+
+### New Rules (3)
+
+- host-interaction/registry/change-registry-key-timestamp wballenthin@google.com
+- host-interaction/mutex/check-mutex-and-terminate-process-on-windows @_re_fox moritz.raabe@mandiant.com mehunhoff@google.com
+- anti-analysis/anti-forensic/clear-logs/clear-windows-event-logs-remotely 99.elad.levi@gmail.com
+
+### Bug Fixes
+- only parse CAPE fields required for analysis @mike-hunhoff #2607
+- main: render result document without needing associated rules @williballenthin #2610
+- vmray: only verify process OS and monitor IDs match @mike-hunhoff #2613
+- render: don't assume prior matches exist within a thread @mike-hunhoff #2612
+
+### Raw diffs
+- [capa v9.0.0...v9.1.0](https://github.com/mandiant/capa/compare/v9.0.0...v9.1.0)
+- [capa-rules v9.0.0...v9.1.0](https://github.com/mandiant/capa-rules/compare/v9.0.0...v9.1.0)
+
+## v9.0.0
+
+This release introduces a new scope for dynamic analysis, "span of calls",
+ that matches features against a across a sliding window of API calls within a thread.
+Its useful for identifying behaviors that span multiple API calls,
+ such as `OpenFile`/`ReadFile`/`CloseFile`, without having to analyze an entire thread, which may be very long.
+
+The release also contains a number of bug fixes and enhancements by new contributors: @v1bh475u and @dhruvak001. Welcome and thank you!
+
+### New Features
+
+- add warning for dynamic .NET samples #1864 @v1bh475u
+- add lint for detecting duplicate features in capa-rules #2250 @v1bh475u
+- add span-of-calls scope to match features against a across a sliding window of API calls within a thread @williballenthin #2532
+- add lint to catch rules that depend on other rules with impossible scope @williballenthin #2124
+
+### Breaking Changes
+
+- remove `is_static_limitation` method from `capa.rules.Rule`
+- add span-of-calls scope to rule format
+- capabilities functions return dataclasses instead of tuples
+
+### New Rules (3)
+
+- data-manipulation/encryption/rsa/encrypt-data-using-rsa-via-embedded-library @Ana06
+- data-manipulation/encryption/use-bigint-function @Ana06
+- internal/limitation/dynamic/internal-dotnet-file-limitation @v1bh475u
+
+
+### Bug Fixes
+
+- dynamic: only check file limitations for static file formats @mr-tz
+- vmray: load more analysis archives @mr-tz
+- vmray: skip non-printable strings @mike-hunhoff
+- vmray: loosen file checks to enable processing more file types @mike-hunhoff #2571
+- strings: add type hints and fix uncovered bugs @williballenthin #2555
+- elffile: handle symbols without a name @williballenthin #2553
+- project: remove pytest-cov that wasn't used @williballenthin @2491
+- replace binascii methods with native Python methods @v1bh475u #2582
+- rules: scopes can now have subscope blocks with the same scope @williballenthin #2584
+
+### capa Explorer Web
+
+### capa Explorer IDA Pro plugin
+
+### Development
+
+- license & copyright: Correct LICENSE file and improve copyright and license information headers in the source code files @Ana06
+- documentation: Improve CLA and Code of Conduct information in CONTRIBUTING @Ana06
+
+### Raw diffs
+- [capa v8.0.1...v9.0.0](https://github.com/mandiant/capa/compare/v8.0.1...v9.0.0)
+- [capa-rules v8.0.1...v9.0.0](https://github.com/mandiant/capa-rules/compare/v8.0.1...v9.0.0)
+
+## v8.0.1
+
+This point release fixes an issue with the IDAPython API to now handle IDA Pro 8.3, 8.4, and 9.0 correctly.
+
+### Bug Fixes
+
+- handle IDA 8.3/8.4 vs. 9.0 API change @mr-tz
+
+### Raw diffs
+- [capa v8.0.0...v8.0.1](https://github.com/mandiant/capa/compare/v8.0.0...v8.0.1)
+- [capa-rules v8.0.0...v8.0.1](https://github.com/mandiant/capa-rules/compare/v8.0.0...v8.0.1)
+
+## v8.0.0
+
+capa version 8 adds support for IDA Pro 9.0 (and idalib). The release comes with various improvements and bug fixes for the Binary Ninja backend (including to load with database files) -- thanks to @xusheng6.
+
+Additional bug fixes improve the dynamic and BinExport backends.
+
+capa version 8 now requires Python 3.10 or newer.
+
+Special thanks to @Tamir-K, @harshit-wadhwani, @jorik-utwente for their great contributions.
+
+### New Features
+
 - allow call as valid subscope for call scoped rules @mr-tz
+- support loading and analyzing a Binary Ninja database #2496 @xusheng6
+- vmray: record process command line details @mr-tz
 
 ### Breaking Changes
 
 - remove support for Python 3.8 and use Python 3.10 as minimum now #1966 @mr-tz
 
-### New Rules (8)
+### New Rules (54)
 
 - nursery/get-shadow-password-file-entry-on-linux jonathanlepore@google.com
 - nursery/set-shadow-password-file-entry-on-linux jonathanlepore@google.com
@@ -20,7 +286,52 @@
 - nursery/persist-via-application-shimming j.j.vannielen@utwente.nl
 - nursery/persist-via-bits-job j.j.vannielen@utwente.nl
 - nursery/persist-via-print-processors-registry-key j.j.vannielen@utwente.nl
--
+- linking/static/touchsocket/linked-against-touchsocket still@teamt5.org
+- runtime/dotnet/compiled-with-dotnet-aot still@teamt5.org
+- nursery/persist-via-errorhandler-script j.j.vannielen@utwente.nl
+- nursery/persist-via-get-variable-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-iphlpapi-dll-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-lnk-shortcut j.j.vannielen@utwente.nl
+- nursery/persist-via-powershell-profile j.j.vannielen@utwente.nl
+- nursery/persist-via-windows-accessibility-tools j.j.vannielen@utwente.nl
+- nursery/persist-via-windows-terminal-profile j.j.vannielen@utwente.nl
+- nursery/write-to-browser-extension-directory j.j.vannielen@utwente.nl
+- nursery/persist-via-aedebug-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-amsi-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-app-paths-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-appcertdlls-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-appx-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-autodialdll-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-autoplayhandlers-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-bootverificationprogram-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-code-signing-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-com-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-command-processor-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-contextmenuhandlers-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-cor_profiler_path-registry-value j.j.vannielen@utwente.nl
+- nursery/persist-via-default-file-association-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-disk-cleanup-handler-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-dotnet-dbgmanageddebugger-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-dotnet_startup_hooks-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-explorer-tools-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-filter-handlers-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-group-policy-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-hhctrl-com-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-htmlhelp-author-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-image-file-execution-options-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-lsa-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-natural-language-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-netsh-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-network-provider-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-path-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-print-monitors-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-rdp-startup-programs-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-silentprocessexit-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-telemetrycontroller-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-timeproviders-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-ts-initialprogram-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-userinitmprlogonscript-registry-value j.j.vannielen@utwente.nl
+- nursery/persist-via-windows-error-reporting-registry-key j.j.vannielen@utwente.nl
 
 ### Bug Fixes
 
@@ -29,6 +340,13 @@
 - ghidra: fix saving of base address @mr-tz
 - binja: support loading raw x86/x86_64 shellcode #2489 @xusheng6
 - binja: fix crash when the IL of certain functions are not available. #2249 @xusheng6
+- binja: major performance improvement on the binja extractor. #1414 @xusheng6
+- cape: make Process model flexible and procmemory optional to load newest reports #2466 @mr-tz
+- binja: fix unit test failure by fixing up the analysis for file al-khaser_x64.exe_ #2507 @xusheng6
+- binja: move the stack string detection to function level #2516 @xusheng6
+- BinExport2: fix handling of incorrect thunk functions #2524 @williballenthin
+- BinExport2: more precise pruning of expressions @williballenthin
+- BinExport2: better handle weird expression trees from Ghidra #2528 #2530 @williballenthin
 
 ### capa Explorer Web
 
@@ -42,8 +360,8 @@
 - CI: update Binary Ninja version to 4.2 #2499 @xusheng6
 
 ### Raw diffs
-- [capa v7.4.0...master](https://github.com/mandiant/capa/compare/v7.4.0...master)
-- [capa-rules v7.4.0...master](https://github.com/mandiant/capa-rules/compare/v7.4.0...master)
+- [capa v7.4.0...v8.0.0](https://github.com/mandiant/capa/compare/v7.4.0...v8.0.0)
+- [capa-rules v7.4.0...v8.0.0](https://github.com/mandiant/capa-rules/compare/v7.4.0...v8.0.0)
 
 ## v7.4.0
 

LICENSE.txt

@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright (C) 2020 Mandiant, Inc.
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -38,49 +38,47 @@ Below you find a list of [our capa blog posts with more details.](#blog-posts)
 ```
 $ capa.exe suspicious.exe
 
-+------------------------+--------------------------------------------------------------------------------+
-| ATT&CK Tactic          | ATT&CK Technique                                                               |
-|------------------------+--------------------------------------------------------------------------------|
-| DEFENSE EVASION        | Obfuscated Files or Information [T1027]                                        |
-| DISCOVERY              | Query Registry [T1012]                                                         |
-|                        | System Information Discovery [T1082]                                           |
-| EXECUTION              | Command and Scripting Interpreter::Windows Command Shell [T1059.003]           |
-|                        | Shared Modules [T1129]                                                         |
-| EXFILTRATION           | Exfiltration Over C2 Channel [T1041]                                           |
-| PERSISTENCE            | Create or Modify System Process::Windows Service [T1543.003]                   |
-+------------------------+--------------------------------------------------------------------------------+
++--------------------+------------------------------------------------------------------------+
+| ATT&CK Tactic      | ATT&CK Technique                                                       |
+|--------------------+------------------------------------------------------------------------|
+| DEFENSE EVASION    | Obfuscated Files or Information [T1027]                                |
+| DISCOVERY          | Query Registry [T1012]                                                 |
+|                    | System Information Discovery [T1082]                                   |
+| EXECUTION          | Command and Scripting Interpreter::Windows Command Shell [T1059.003]   |
+|                    | Shared Modules [T1129]                                                 |
+| EXFILTRATION       | Exfiltration Over C2 Channel [T1041]                                   |
+| PERSISTENCE        | Create or Modify System Process::Windows Service [T1543.003]           |
++--------------------+------------------------------------------------------------------------+
 
-+-------------------------------------------------------+-------------------------------------------------+
-| CAPABILITY                                            | NAMESPACE                                       |
-|-------------------------------------------------------+-------------------------------------------------|
-| check for OutputDebugString error                     | anti-analysis/anti-debugging/debugger-detection |
-| read and send data from client to server              | c2/file-transfer                                |
-| execute shell command and capture output              | c2/shell                                        |
-| receive data (2 matches)                              | communication                                   |
-| send data (6 matches)                                 | communication                                   |
-| connect to HTTP server (3 matches)                    | communication/http/client                       |
-| send HTTP request (3 matches)                         | communication/http/client                       |
-| create pipe                                           | communication/named-pipe/create                 |
-| get socket status (2 matches)                         | communication/socket                            |
-| receive data on socket (2 matches)                    | communication/socket/receive                    |
-| send data on socket (3 matches)                       | communication/socket/send                       |
-| connect TCP socket                                    | communication/socket/tcp                        |
-| encode data using Base64                              | data-manipulation/encoding/base64               |
-| encode data using XOR (6 matches)                     | data-manipulation/encoding/xor                  |
-| run as a service                                      | executable/pe                                   |
-| get common file path (3 matches)                      | host-interaction/file-system                    |
-| read file                                             | host-interaction/file-system/read               |
-| write file (2 matches)                                | host-interaction/file-system/write              |
-| print debug messages (2 matches)                      | host-interaction/log/debug/write-event          |
-| resolve DNS                                           | host-interaction/network/dns/resolve            |
-| get hostname                                          | host-interaction/os/hostname                    |
-| create a process with modified I/O handles and window | host-interaction/process/create                 |
-| create process                                        | host-interaction/process/create                 |
-| create registry key                                   | host-interaction/registry/create                |
-| create service                                        | host-interaction/service/create                 |
-| create thread                                         | host-interaction/thread/create                  |
-| persist via Windows service                           | persistence/service                             |
-+-------------------------------------------------------+-------------------------------------------------+
++-------------------------------------------+-------------------------------------------------+
+| CAPABILITY                                | NAMESPACE                                       |
+|-------------------------------------------+-------------------------------------------------|
+| read and send data from client to server  | c2/file-transfer                               |
+| execute shell command and capture output  | c2/shell                                       |
+| receive data (2 matches)                  | communication                                   |
+| send data (6 matches)                     | communication                                   |
+| connect to HTTP server (3 matches)        | communication/http/client                       |
+| send HTTP request (3 matches)             | communication/http/client                       |
+| create pipe                               | communication/named-pipe/create                 |
+| get socket status (2 matches)             | communication/socket                            |
+| receive data on socket (2 matches)        | communication/socket/receive                    |
+| send data on socket (3 matches)           | communication/socket/send                       |
+| connect TCP socket                        | communication/socket/tcp                        |
+| encode data using Base64                  | data-manipulation/encoding/base64               |
+| encode data using XOR (6 matches)         | data-manipulation/encoding/xor                  |
+| run as a service                          | executable/pe                                   |
+| get common file path (3 matches)          | host-interaction/file-system                    |
+| read file                                 | host-interaction/file-system/read               |
+| write file (2 matches)                    | host-interaction/file-system/write              |
+| print debug messages (2 matches)          | host-interaction/log/debug/write-event          |
+| resolve DNS                               | host-interaction/network/dns/resolve            |
+| get hostname                              | host-interaction/os/hostname                    |
+| create process                            | host-interaction/process/create                 |
+| create registry key                       | host-interaction/registry/create                |
+| create service                            | host-interaction/service/create                 |
+| create thread                             | host-interaction/thread/create                  |
+| persist via Windows service               | persistence/service                             |
++-------------------------------------------+-------------------------------------------------+
 ```
 
 # download and usage
@@ -317,3 +315,6 @@ If you use Ghidra, then you can use the [capa + Ghidra integration](/capa/ghidra
 
 ## capa testfiles
 The [capa-testfiles repository](https://github.com/mandiant/capa-testfiles) contains the data we use to test capa's code and rules
+
+## mailing list
+Subscribe to the FLARE mailing list for community announcements! Email "subscribe" to [flare-external@google.com](mailto:flare-external@google.com?subject=subscribe).

capa/capabilities/common.py

@@ -1,25 +1,43 @@
 # -*- coding: utf-8 -*-
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 import itertools
 import collections
-from typing import Any
+from typing import Optional
+from dataclasses import dataclass
 
-from capa.rules import Scope, RuleSet
+from capa.rules import Rule, Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
 from capa.features.address import NO_ADDRESS
+from capa.render.result_document import LibraryFunction, StaticFeatureCounts, DynamicFeatureCounts
 from capa.features.extractors.base_extractor import FeatureExtractor, StaticFeatureExtractor, DynamicFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
-def find_file_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, function_features: FeatureSet):
+@dataclass
+class FileCapabilities:
+    features: FeatureSet
+    matches: MatchResults
+    feature_count: int
+
+
+def find_file_capabilities(
+    ruleset: RuleSet, extractor: FeatureExtractor, function_features: FeatureSet
+) -> FileCapabilities:
     file_features: FeatureSet = collections.defaultdict(set)
 
     for feature, va in itertools.chain(extractor.extract_file_features(), extractor.extract_global_features()):
@@ -36,35 +54,18 @@ def find_file_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, functi
 
     file_features.update(function_features)
 
-    _, matches = ruleset.match(Scope.FILE, file_features, NO_ADDRESS)
-    return matches, len(file_features)
+    features, matches = ruleset.match(Scope.FILE, file_features, NO_ADDRESS)
+    return FileCapabilities(features, matches, len(file_features))
 
 
-def has_file_limitation(rules: RuleSet, capabilities: MatchResults, is_standalone=True) -> bool:
-    file_limitation_rules = list(filter(lambda r: r.is_file_limitation_rule(), rules.rules.values()))
-
-    for file_limitation_rule in file_limitation_rules:
-        if file_limitation_rule.name not in capabilities:
-            continue
-
-        logger.warning("-" * 80)
-        for line in file_limitation_rule.meta.get("description", "").split("\n"):
-            logger.warning(" %s", line)
-        logger.warning(" Identified via rule: %s", file_limitation_rule.name)
-        if is_standalone:
-            logger.warning(" ")
-            logger.warning(" Use -v or -vv if you really want to see the capabilities identified by capa.")
-        logger.warning("-" * 80)
-
-        # bail on first file limitation
-        return True
-
-    return False
+@dataclass
+class Capabilities:
+    matches: MatchResults
+    feature_counts: StaticFeatureCounts | DynamicFeatureCounts
+    library_functions: Optional[tuple[LibraryFunction, ...]] = None
 
 
-def find_capabilities(
-    ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None, **kwargs
-) -> tuple[MatchResults, Any]:
+def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None, **kwargs) -> Capabilities:
     from capa.capabilities.static import find_static_capabilities
     from capa.capabilities.dynamic import find_dynamic_capabilities
 
@@ -77,3 +78,40 @@ def find_capabilities(
         return find_dynamic_capabilities(ruleset, extractor, disable_progress=disable_progress, **kwargs)
 
     raise ValueError(f"unexpected extractor type: {extractor.__class__.__name__}")
+
+
+def has_limitation(rules: list, capabilities: Capabilities | FileCapabilities, is_standalone: bool) -> bool:
+
+    for rule in rules:
+        if rule.name not in capabilities.matches:
+            continue
+        logger.warning("-" * 80)
+        for line in rule.meta.get("description", "").split("\n"):
+            logger.warning(" %s", line)
+        logger.warning(" Identified via rule: %s", rule.name)
+        if is_standalone:
+            logger.warning(" ")
+            logger.warning(" Use -v or -vv if you really want to see the capabilities identified by capa.")
+        logger.warning("-" * 80)
+
+        # bail on first file limitation
+        return True
+    return False
+
+
+def is_static_limitation_rule(r: Rule) -> bool:
+    return r.meta.get("namespace", "") == "internal/limitation/static"
+
+
+def has_static_limitation(rules: RuleSet, capabilities: Capabilities | FileCapabilities, is_standalone=True) -> bool:
+    file_limitation_rules = list(filter(lambda r: is_static_limitation_rule(r), rules.rules.values()))
+    return has_limitation(file_limitation_rules, capabilities, is_standalone)
+
+
+def is_dynamic_limitation_rule(r: Rule) -> bool:
+    return r.meta.get("namespace", "") == "internal/limitation/dynamic"
+
+
+def has_dynamic_limitation(rules: RuleSet, capabilities: Capabilities | FileCapabilities, is_standalone=True) -> bool:
+    dynamic_limitation_rules = list(filter(lambda r: is_dynamic_limitation_rule(r), rules.rules.values()))
+    return has_limitation(dynamic_limitation_rules, capabilities, is_standalone)

capa/capabilities/dynamic.py

@@ -1,34 +1,55 @@
 # -*- coding: utf-8 -*-
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 import itertools
 import collections
-from typing import Any
+from dataclasses import dataclass
 
 import capa.perf
+import capa.engine
+import capa.helpers
 import capa.features.freeze as frz
 import capa.render.result_document as rdoc
 from capa.rules import Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
-from capa.capabilities.common import find_file_capabilities
+from capa.features.address import _NoAddress
+from capa.capabilities.common import Capabilities, find_file_capabilities
 from capa.features.extractors.base_extractor import CallHandle, ThreadHandle, ProcessHandle, DynamicFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
+# The number of calls that make up a span of calls.
+#
+# The larger this is, the more calls are grouped together to match rule logic.
+# This means a longer chain can be recognized; however, its a bit more expensive.
+SPAN_SIZE = 20
+
+
+@dataclass
+class CallCapabilities:
+    features: FeatureSet
+    matches: MatchResults
+
+
 def find_call_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle
-) -> tuple[FeatureSet, MatchResults]:
+) -> CallCapabilities:
     """
     find matches for the given rules for the given call.
-
-    returns: tuple containing (features for call, match results for call)
     """
     # all features found for the call.
     features: FeatureSet = collections.defaultdict(set)
@@ -46,16 +67,105 @@ def find_call_capabilities(
         for addr, _ in res:
             capa.engine.index_rule_matches(features, rule, [addr])
 
-    return features, matches
+    return CallCapabilities(features, matches)
+
+
+@dataclass
+class ThreadCapabilities:
+    features: FeatureSet
+    thread_matches: MatchResults
+    span_matches: MatchResults
+    call_matches: MatchResults
+
+
+class SpanOfCallsMatcher:
+    def __init__(self, ruleset: RuleSet):
+        super().__init__()
+        self.ruleset = ruleset
+
+        # matches found at the span scope.
+        self.matches: MatchResults = collections.defaultdict(list)
+
+        # We match spans as the sliding window of calls with size SPAN_SIZE.
+        #
+        # For each call, we consider the window of SPAN_SIZE calls leading up to it,
+        #  merging all their features and doing a match.
+        #
+        # We track these features in two data structures:
+        #   1. a deque of those features found in the prior calls.
+        #      We'll append to it, and as it grows larger than SPAN_SIZE, the oldest items are removed.
+        #   2. a live set of features seen in the span.
+        #      As we pop from the deque, we remove features from the current set,
+        #      and as we push to the deque, we insert features to the current set.
+        # With this approach, our algorithm performance is independent of SPAN_SIZE.
+        # The naive algorithm, of merging all the trailing feature sets at each call, is dependent upon SPAN_SIZE
+        # (that is, runtime gets slower the larger SPAN_SIZE is).
+        self.current_feature_sets: collections.deque[FeatureSet] = collections.deque(maxlen=SPAN_SIZE)
+        self.current_features: FeatureSet = collections.defaultdict(set)
+
+        # the names of rules matched at the last span,
+        # so that we can deduplicate long strings of the same matches.
+        self.last_span_matches: set[str] = set()
+
+    def next(self, ch: CallHandle, call_features: FeatureSet):
+        # As we add items to the end of the deque, overflow and drop the oldest items (at the left end).
+        # While we could rely on `deque.append` with `maxlen` set (which we provide above),
+        # we want to use the dropped item first, to remove the old features, so we manually pop it here.
+        if len(self.current_feature_sets) == SPAN_SIZE:
+            overflowing_feature_set = self.current_feature_sets.popleft()
+
+            for feature, vas in overflowing_feature_set.items():
+                if len(vas) == 1 and isinstance(next(iter(vas)), _NoAddress):
+                    # `vas == { NO_ADDRESS }` without the garbage.
+                    #
+                    # ignore the common case of global features getting added/removed/trimmed repeatedly,
+                    # like arch/os/format.
+                    continue
+
+                self.current_features[feature] -= vas
+                if not self.current_features[feature]:
+                    del self.current_features[feature]
+
+        # update the deque and set of features with the latest call's worth of features.
+        self.current_feature_sets.append(call_features)
+        for feature, vas in call_features.items():
+            self.current_features[feature] |= vas
+
+        _, matches = self.ruleset.match(Scope.SPAN_OF_CALLS, self.current_features, ch.address)
+
+        newly_encountered_rules = set(matches.keys()) - self.last_span_matches
+
+        # don't emit match results for rules seen during the immediately preceeding spans.
+        #
+        # This means that we won't emit duplicate matches when there are multiple spans
+        #  that overlap a single matching event.
+        # It also handles the case of a tight loop containing matched logic;
+        #  only the first match will be recorded.
+        #
+        # In theory, this means the result document doesn't have *every* possible match location,
+        # but in practice, humans will only be interested in the first handful anyways.
+        suppressed_rules = set(self.last_span_matches)
+
+        # however, if a newly encountered rule depends on a suppressed rule,
+        # don't suppress that rule match, or we won't be able to reconstruct the vverbose output.
+        # see: https://github.com/mandiant/capa/pull/2532#issuecomment-2548508130
+        for new_rule in newly_encountered_rules:
+            suppressed_rules -= set(self.ruleset.rules[new_rule].get_dependencies(self.ruleset.rules_by_namespace))
+
+        for rule_name, res in matches.items():
+            if rule_name in suppressed_rules:
+                continue
+            self.matches[rule_name].extend(res)
+
+        self.last_span_matches = set(matches.keys())
 
 
 def find_thread_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle
-) -> tuple[FeatureSet, MatchResults, MatchResults]:
+) -> ThreadCapabilities:
     """
-    find matches for the given rules within the given thread.
-
-    returns: tuple containing (features for thread, match results for thread, match results for calls)
+    find matches for the given rules within the given thread,
+    which includes matches for all the spans and calls within it.
     """
     # all features found within this thread,
     # includes features found within calls.
@@ -65,14 +175,19 @@ def find_thread_capabilities(
     # might be found at different calls, that's ok.
     call_matches: MatchResults = collections.defaultdict(list)
 
-    for ch in extractor.get_calls(ph, th):
-        ifeatures, imatches = find_call_capabilities(ruleset, extractor, ph, th, ch)
-        for feature, vas in ifeatures.items():
+    span_matcher = SpanOfCallsMatcher(ruleset)
+
+    call_count = 0
+    for call_count, ch in enumerate(extractor.get_calls(ph, th)):  # noqa: B007
+        call_capabilities = find_call_capabilities(ruleset, extractor, ph, th, ch)
+        for feature, vas in call_capabilities.features.items():
             features[feature].update(vas)
 
-        for rule_name, res in imatches.items():
+        for rule_name, res in call_capabilities.matches.items():
             call_matches[rule_name].extend(res)
 
+        span_matcher.next(ch, call_capabilities.features)
+
     for feature, va in itertools.chain(extractor.extract_thread_features(ph, th), extractor.extract_global_features()):
         features[feature].add(va)
 
@@ -84,16 +199,31 @@ def find_thread_capabilities(
         for va, _ in res:
             capa.engine.index_rule_matches(features, rule, [va])
 
-    return features, matches, call_matches
+    logger.debug(
+        "analyzed thread %d[%d] with %d events, %d features, and %d matches",
+        th.address.process.pid,
+        th.address.tid,
+        call_count,
+        len(features),
+        len(matches) + len(span_matcher.matches) + len(call_matches),
+    )
+    return ThreadCapabilities(features, matches, span_matcher.matches, call_matches)
+
+
+@dataclass
+class ProcessCapabilities:
+    process_matches: MatchResults
+    thread_matches: MatchResults
+    span_matches: MatchResults
+    call_matches: MatchResults
+    feature_count: int
 
 
 def find_process_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle
-) -> tuple[MatchResults, MatchResults, MatchResults, int]:
+) -> ProcessCapabilities:
     """
     find matches for the given rules within the given process.
-
-    returns: tuple containing (match results for process, match results for threads, match results for calls, number of features)
     """
     # all features found within this process,
     # includes features found within threads (and calls).
@@ -103,33 +233,48 @@ def find_process_capabilities(
     # might be found at different threads, that's ok.
     thread_matches: MatchResults = collections.defaultdict(list)
 
+    # matches found at the span-of-calls scope.
+    # might be found at different spans, that's ok.
+    span_matches: MatchResults = collections.defaultdict(list)
+
     # matches found at the call scope.
     # might be found at different calls, that's ok.
     call_matches: MatchResults = collections.defaultdict(list)
 
     for th in extractor.get_threads(ph):
-        features, tmatches, cmatches = find_thread_capabilities(ruleset, extractor, ph, th)
-        for feature, vas in features.items():
+        thread_capabilities = find_thread_capabilities(ruleset, extractor, ph, th)
+        for feature, vas in thread_capabilities.features.items():
             process_features[feature].update(vas)
 
-        for rule_name, res in tmatches.items():
+        for rule_name, res in thread_capabilities.thread_matches.items():
             thread_matches[rule_name].extend(res)
 
-        for rule_name, res in cmatches.items():
+        for rule_name, res in thread_capabilities.span_matches.items():
+            span_matches[rule_name].extend(res)
+
+        for rule_name, res in thread_capabilities.call_matches.items():
             call_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(extractor.extract_process_features(ph), extractor.extract_global_features()):
         process_features[feature].add(va)
 
     _, process_matches = ruleset.match(Scope.PROCESS, process_features, ph.address)
-    return process_matches, thread_matches, call_matches, len(process_features)
+
+    logger.debug(
+        "analyzed process %d and extracted %d features with %d matches",
+        ph.address.pid,
+        len(process_features),
+        len(process_matches),
+    )
+    return ProcessCapabilities(process_matches, thread_matches, span_matches, call_matches, len(process_features))
 
 
 def find_dynamic_capabilities(
-    ruleset: RuleSet, extractor: DynamicFeatureExtractor, disable_progress=None
-) -> tuple[MatchResults, Any]:
+    ruleset: RuleSet, extractor: DynamicFeatureExtractor, disable_progress: bool = False
+) -> Capabilities:
     all_process_matches: MatchResults = collections.defaultdict(list)
     all_thread_matches: MatchResults = collections.defaultdict(list)
+    all_span_matches: MatchResults = collections.defaultdict(list)
     all_call_matches: MatchResults = collections.defaultdict(list)
 
     feature_counts = rdoc.DynamicFeatureCounts(file=0, processes=())
@@ -143,19 +288,20 @@ def find_dynamic_capabilities(
     ) as pbar:
         task = pbar.add_task("matching", total=n_processes, unit="processes")
         for p in processes:
-            process_matches, thread_matches, call_matches, feature_count = find_process_capabilities(
-                ruleset, extractor, p
-            )
+            process_capabilities = find_process_capabilities(ruleset, extractor, p)
             feature_counts.processes += (
-                rdoc.ProcessFeatureCount(address=frz.Address.from_capa(p.address), count=feature_count),
+                rdoc.ProcessFeatureCount(
+                    address=frz.Address.from_capa(p.address), count=process_capabilities.feature_count
+                ),
             )
-            logger.debug("analyzed %s and extracted %d features", p.address, feature_count)
 
-            for rule_name, res in process_matches.items():
+            for rule_name, res in process_capabilities.process_matches.items():
                 all_process_matches[rule_name].extend(res)
-            for rule_name, res in thread_matches.items():
+            for rule_name, res in process_capabilities.thread_matches.items():
                 all_thread_matches[rule_name].extend(res)
-            for rule_name, res in call_matches.items():
+            for rule_name, res in process_capabilities.span_matches.items():
+                all_span_matches[rule_name].extend(res)
+            for rule_name, res in process_capabilities.call_matches.items():
                 all_call_matches[rule_name].extend(res)
 
             pbar.advance(task)
@@ -164,29 +310,26 @@ def find_dynamic_capabilities(
     # mapping from feature (matched rule) to set of addresses at which it matched.
     process_and_lower_features: FeatureSet = collections.defaultdict(set)
     for rule_name, results in itertools.chain(
-        all_process_matches.items(), all_thread_matches.items(), all_call_matches.items()
+        all_process_matches.items(), all_thread_matches.items(), all_span_matches.items(), all_call_matches.items()
     ):
         locations = {p[0] for p in results}
         rule = ruleset[rule_name]
         capa.engine.index_rule_matches(process_and_lower_features, rule, locations)
 
-    all_file_matches, feature_count = find_file_capabilities(ruleset, extractor, process_and_lower_features)
-    feature_counts.file = feature_count
+    all_file_capabilities = find_file_capabilities(ruleset, extractor, process_and_lower_features)
+    feature_counts.file = all_file_capabilities.feature_count
 
     matches = dict(
         itertools.chain(
             # each rule exists in exactly one scope,
             # so there won't be any overlap among these following MatchResults,
             # and we can merge the dictionaries naively.
+            all_call_matches.items(),
+            all_span_matches.items(),
             all_thread_matches.items(),
             all_process_matches.items(),
-            all_call_matches.items(),
-            all_file_matches.items(),
+            all_file_capabilities.matches.items(),
         )
     )
 
-    meta = {
-        "feature_counts": feature_counts,
-    }
-
-    return matches, meta
+    return Capabilities(matches, feature_counts)

capa/capabilities/static.py

@@ -1,16 +1,23 @@
 # -*- coding: utf-8 -*-
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import time
 import logging
 import itertools
 import collections
-from typing import Any
+from dataclasses import dataclass
 
 import capa.perf
 import capa.helpers
@@ -18,19 +25,23 @@ import capa.features.freeze as frz
 import capa.render.result_document as rdoc
 from capa.rules import Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
-from capa.capabilities.common import find_file_capabilities
+from capa.capabilities.common import Capabilities, find_file_capabilities
 from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, StaticFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
+@dataclass
+class InstructionCapabilities:
+    features: FeatureSet
+    matches: MatchResults
+
+
 def find_instruction_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
-) -> tuple[FeatureSet, MatchResults]:
+) -> InstructionCapabilities:
     """
     find matches for the given rules for the given instruction.
-
-    returns: tuple containing (features for instruction, match results for instruction)
     """
     # all features found for the instruction.
     features: FeatureSet = collections.defaultdict(set)
@@ -48,16 +59,21 @@ def find_instruction_capabilities(
         for addr, _ in res:
             capa.engine.index_rule_matches(features, rule, [addr])
 
-    return features, matches
+    return InstructionCapabilities(features, matches)
+
+
+@dataclass
+class BasicBlockCapabilities:
+    features: FeatureSet
+    basic_block_matches: MatchResults
+    instruction_matches: MatchResults
 
 
 def find_basic_block_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle
-) -> tuple[FeatureSet, MatchResults, MatchResults]:
+) -> BasicBlockCapabilities:
     """
     find matches for the given rules within the given basic block.
-
-    returns: tuple containing (features for basic block, match results for basic block, match results for instructions)
     """
     # all features found within this basic block,
     # includes features found within instructions.
@@ -68,11 +84,11 @@ def find_basic_block_capabilities(
     insn_matches: MatchResults = collections.defaultdict(list)
 
     for insn in extractor.get_instructions(f, bb):
-        ifeatures, imatches = find_instruction_capabilities(ruleset, extractor, f, bb, insn)
-        for feature, vas in ifeatures.items():
+        instruction_capabilities = find_instruction_capabilities(ruleset, extractor, f, bb, insn)
+        for feature, vas in instruction_capabilities.features.items():
             features[feature].update(vas)
 
-        for rule_name, res in imatches.items():
+        for rule_name, res in instruction_capabilities.matches.items():
             insn_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(
@@ -88,16 +104,20 @@ def find_basic_block_capabilities(
         for va, _ in res:
             capa.engine.index_rule_matches(features, rule, [va])
 
-    return features, matches, insn_matches
+    return BasicBlockCapabilities(features, matches, insn_matches)
 
 
-def find_code_capabilities(
-    ruleset: RuleSet, extractor: StaticFeatureExtractor, fh: FunctionHandle
-) -> tuple[MatchResults, MatchResults, MatchResults, int]:
+@dataclass
+class CodeCapabilities:
+    function_matches: MatchResults
+    basic_block_matches: MatchResults
+    instruction_matches: MatchResults
+    feature_count: int
+
+
+def find_code_capabilities(ruleset: RuleSet, extractor: StaticFeatureExtractor, fh: FunctionHandle) -> CodeCapabilities:
     """
     find matches for the given rules within the given function.
-
-    returns: tuple containing (match results for function, match results for basic blocks, match results for instructions, number of features)
     """
     # all features found within this function,
     # includes features found within basic blocks (and instructions).
@@ -112,26 +132,26 @@ def find_code_capabilities(
     insn_matches: MatchResults = collections.defaultdict(list)
 
     for bb in extractor.get_basic_blocks(fh):
-        features, bmatches, imatches = find_basic_block_capabilities(ruleset, extractor, fh, bb)
-        for feature, vas in features.items():
+        basic_block_capabilities = find_basic_block_capabilities(ruleset, extractor, fh, bb)
+        for feature, vas in basic_block_capabilities.features.items():
             function_features[feature].update(vas)
 
-        for rule_name, res in bmatches.items():
+        for rule_name, res in basic_block_capabilities.basic_block_matches.items():
             bb_matches[rule_name].extend(res)
 
-        for rule_name, res in imatches.items():
+        for rule_name, res in basic_block_capabilities.instruction_matches.items():
             insn_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(extractor.extract_function_features(fh), extractor.extract_global_features()):
         function_features[feature].add(va)
 
     _, function_matches = ruleset.match(Scope.FUNCTION, function_features, fh.address)
-    return function_matches, bb_matches, insn_matches, len(function_features)
+    return CodeCapabilities(function_matches, bb_matches, insn_matches, len(function_features))
 
 
 def find_static_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, disable_progress=None
-) -> tuple[MatchResults, Any]:
+) -> Capabilities:
     all_function_matches: MatchResults = collections.defaultdict(list)
     all_bb_matches: MatchResults = collections.defaultdict(list)
     all_insn_matches: MatchResults = collections.defaultdict(list)
@@ -165,30 +185,36 @@ def find_static_capabilities(
                 pbar.advance(task)
                 continue
 
-            function_matches, bb_matches, insn_matches, feature_count = find_code_capabilities(ruleset, extractor, f)
+            code_capabilities = find_code_capabilities(ruleset, extractor, f)
             feature_counts.functions += (
-                rdoc.FunctionFeatureCount(address=frz.Address.from_capa(f.address), count=feature_count),
+                rdoc.FunctionFeatureCount(
+                    address=frz.Address.from_capa(f.address), count=code_capabilities.feature_count
+                ),
             )
             t1 = time.time()
 
             match_count = 0
-            for name, matches_ in itertools.chain(function_matches.items(), bb_matches.items(), insn_matches.items()):
+            for name, matches_ in itertools.chain(
+                code_capabilities.function_matches.items(),
+                code_capabilities.basic_block_matches.items(),
+                code_capabilities.instruction_matches.items(),
+            ):
                 if not ruleset.rules[name].is_subscope_rule():
                     match_count += len(matches_)
 
             logger.debug(
                 "analyzed function 0x%x and extracted %d features, %d matches in %0.02fs",
                 f.address,
-                feature_count,
+                code_capabilities.feature_count,
                 match_count,
                 t1 - t0,
             )
 
-            for rule_name, res in function_matches.items():
+            for rule_name, res in code_capabilities.function_matches.items():
                 all_function_matches[rule_name].extend(res)
-            for rule_name, res in bb_matches.items():
+            for rule_name, res in code_capabilities.basic_block_matches.items():
                 all_bb_matches[rule_name].extend(res)
-            for rule_name, res in insn_matches.items():
+            for rule_name, res in code_capabilities.instruction_matches.items():
                 all_insn_matches[rule_name].extend(res)
 
             pbar.advance(task)
@@ -203,8 +229,8 @@ def find_static_capabilities(
         rule = ruleset[rule_name]
         capa.engine.index_rule_matches(function_and_lower_features, rule, locations)
 
-    all_file_matches, feature_count = find_file_capabilities(ruleset, extractor, function_and_lower_features)
-    feature_counts.file = feature_count
+    all_file_capabilities = find_file_capabilities(ruleset, extractor, function_and_lower_features)
+    feature_counts.file = all_file_capabilities.feature_count
 
     matches: MatchResults = dict(
         itertools.chain(
@@ -214,13 +240,8 @@ def find_static_capabilities(
             all_insn_matches.items(),
             all_bb_matches.items(),
             all_function_matches.items(),
-            all_file_matches.items(),
+            all_file_capabilities.matches.items(),
         )
     )
 
-    meta = {
-        "feature_counts": feature_counts,
-        "library_functions": library_functions,
-    }
-
-    return matches, meta
+    return Capabilities(matches, feature_counts, library_functions)

capa/engine.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import copy
 import collections

capa/exceptions.py

@@ -1,10 +1,18 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
 class UnsupportedRuntimeError(RuntimeError):
     pass
 

capa/features/address.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import abc
 
 
@@ -107,8 +114,7 @@ class DynamicCallAddress(Address):
         return hash((self.thread, self.id))
 
     def __eq__(self, other):
-        assert isinstance(other, DynamicCallAddress)
-        return (self.thread, self.id) == (other.thread, other.id)
+        return isinstance(other, DynamicCallAddress) and (self.thread, self.id) == (other.thread, other.id)
 
     def __lt__(self, other):
         assert isinstance(other, DynamicCallAddress)

capa/features/basicblock.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from capa.features.common import Feature
 

capa/features/com/__init__.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from enum import Enum
 
 from capa.helpers import assert_never

capa/features/com/classes.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 COM_CLASSES: dict[str, list[str]] = {
     "ClusAppWiz": ["24F97150-6689-11D1-9AA7-00C04FB93A80"],

capa/features/com/interfaces.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 COM_INTERFACES: dict[str, list[str]] = {
     "IClusterApplicationWizard": ["24F97151-6689-11D1-9AA7-00C04FB93A80"],

capa/features/common.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import re
 import abc
@@ -85,7 +92,7 @@ class Result:
         self.success = success
         self.statement = statement
         self.children = children
-        self.locations = locations if locations is not None else set()
+        self.locations = frozenset(locations) if locations is not None else frozenset()
 
     def __eq__(self, other):
         if isinstance(other, bool):
@@ -98,6 +105,25 @@ class Result:
     def __nonzero__(self):
         return self.success
 
+    def __str__(self):
+        # as this object isn't user facing, this formatting is just to help with debugging
+
+        lines: list[str] = []
+
+        def rec(m: "Result", indent: int):
+            if isinstance(m.statement, capa.engine.Statement):
+                line = ("  " * indent) + str(m.statement.name) + " " + str(m.success)
+            else:
+                line = ("  " * indent) + str(m.statement) + " " + str(m.success) + " " + str(m.locations)
+
+            lines.append(line)
+
+            for child in m.children:
+                rec(child, indent + 1)
+
+        rec(self, 0)
+        return "\n".join(lines)
+
 
 class Feature(abc.ABC):  # noqa: B024
     # this is an abstract class, since we don't want anyone to instantiate it directly,
@@ -168,7 +194,11 @@ class Feature(abc.ABC):  # noqa: B024
     def evaluate(self, features: "capa.engine.FeatureSet", short_circuit=True) -> Result:
         capa.perf.counters["evaluate.feature"] += 1
         capa.perf.counters["evaluate.feature." + self.name] += 1
-        return Result(self in features, self, [], locations=features.get(self, set()))
+        success = self in features
+        if success:
+            return Result(True, self, [], locations=features[self])
+        else:
+            return Result(False, self, [], locations=None)
 
 
 class MatchedRule(Feature):
@@ -466,6 +496,7 @@ FORMAT_VMRAY = "vmray"
 FORMAT_BINEXPORT2 = "binexport2"
 FORMAT_FREEZE = "freeze"
 FORMAT_RESULT = "result"
+FORMAT_BINJA_DB = "binja_database"
 STATIC_FORMATS = {
     FORMAT_SC32,
     FORMAT_SC64,
@@ -475,6 +506,7 @@ STATIC_FORMATS = {
     FORMAT_FREEZE,
     FORMAT_RESULT,
     FORMAT_BINEXPORT2,
+    FORMAT_BINJA_DB,
 }
 DYNAMIC_FORMATS = {
     FORMAT_CAPE,

capa/features/extractors/base_extractor.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import abc
 import hashlib
@@ -481,11 +488,11 @@ class DynamicFeatureExtractor:
         raise NotImplementedError()
 
 
-def ProcessFilter(extractor: DynamicFeatureExtractor, processes: set) -> DynamicFeatureExtractor:
+def ProcessFilter(extractor: DynamicFeatureExtractor, pids: set[int]) -> DynamicFeatureExtractor:
     original_get_processes = extractor.get_processes
 
     def filtered_get_processes(self):
-        yield from (f for f in original_get_processes() if f.address.pid in processes)
+        yield from (f for f in original_get_processes() if f.address.pid in pids)
 
     # we make a copy of the original extractor object and then update its get_processes() method with the decorated filter one.
     # this is in order to preserve the original extractor object's get_processes() method, in case it is used elsewhere in the code.
@@ -497,4 +504,16 @@ def ProcessFilter(extractor: DynamicFeatureExtractor, processes: set) -> Dynamic
     return new_extractor
 
 
+def ThreadFilter(extractor: DynamicFeatureExtractor, threads: set[Address]) -> DynamicFeatureExtractor:
+    original_get_threads = extractor.get_threads
+
+    def filtered_get_threads(self, ph: ProcessHandle):
+        yield from (t for t in original_get_threads(ph) if t.address in threads)
+
+    new_extractor = copy(extractor)
+    new_extractor.get_threads = MethodType(filtered_get_threads, extractor)  # type: ignore
+
+    return new_extractor
+
+
 FeatureExtractor: TypeAlias = Union[StaticFeatureExtractor, DynamicFeatureExtractor]

capa/features/extractors/binexport2/__init__.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 """
 Proto files generated via protobuf v24.4:
 
@@ -280,11 +287,13 @@ class BinExport2Analysis:
             curr_idx: int = idx
             for _ in range(capa.features.common.THUNK_CHAIN_DEPTH_DELTA):
                 thunk_callees: list[int] = self.idx.callees_by_vertex_index[curr_idx]
-                # if this doesn't hold, then it doesn't seem like this is a thunk,
+                # If this doesn't hold, then it doesn't seem like this is a thunk,
                 # because either, len is:
-                #    0 and the thunk doesn't point to anything, or
+                #    0 and the thunk doesn't point to anything or is indirect, like `call eax`, or
                 #   >1 and the thunk may end up at many functions.
-                assert len(thunk_callees) == 1, f"thunk @ {hex(addr)} failed"
+                # In any case, this doesn't appear to be the sort of thunk we're looking for.
+                if len(thunk_callees) != 1:
+                    break
 
                 thunked_idx: int = thunk_callees[0]
                 thunked_vertex: BinExport2.CallGraph.Vertex = self.be2.call_graph.vertex[thunked_idx]

capa/features/extractors/binexport2/arch/arm/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
 

capa/features/extractors/binexport2/arch/arm/insn.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Iterator, Optional
 

capa/features/extractors/binexport2/arch/intel/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Optional
 from dataclasses import dataclass
 

capa/features/extractors/binexport2/arch/intel/insn.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Iterator
 

capa/features/extractors/binexport2/basicblock.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from typing import Iterator
 

capa/features/extractors/binexport2/extractor.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Iterator
 

capa/features/extractors/binexport2/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import io
 import logging
 from typing import Iterator

capa/features/extractors/binexport2/function.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 from capa.features.file import FunctionName

capa/features/extractors/binexport2/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import re
 from typing import Union, Iterator, Optional
 from collections import defaultdict
@@ -50,6 +57,25 @@ def is_vertex_type(vertex: BinExport2.CallGraph.Vertex, type_: BinExport2.CallGr
     return vertex.HasField("type") and vertex.type == type_
 
 
+# internal to `build_expression_tree`
+# this is unstable: it is subject to change, so don't rely on it!
+def _prune_expression_tree_references_to_tree_index(
+    expression_tree: list[list[int]],
+    tree_index: int,
+):
+    # `i` is the index of the tree node that we'll search for `tree_index`
+    # if we remove `tree_index` from it, and it is now empty,
+    # then we'll need to prune references to `i`.
+    for i, tree_node in enumerate(expression_tree):
+        if tree_index in tree_node:
+            tree_node.remove(tree_index)
+
+            if len(tree_node) == 0:
+                # if the parent node is now empty,
+                # remove references to that parent node.
+                _prune_expression_tree_references_to_tree_index(expression_tree, i)
+
+
 # internal to `build_expression_tree`
 # this is unstable: it is subject to change, so don't rely on it!
 def _prune_expression_tree_empty_shifts(
@@ -70,9 +96,7 @@ def _prune_expression_tree_empty_shifts(
             #
             # Which seems to be as if the shift wasn't there (shift of #0)
             # so we want to remove references to this node from any parent nodes.
-            for tree_node in expression_tree:
-                if tree_index in tree_node:
-                    tree_node.remove(tree_index)
+            _prune_expression_tree_references_to_tree_index(expression_tree, tree_index)
 
             return
 
@@ -82,7 +106,20 @@ def _prune_expression_tree_empty_shifts(
 
 # internal to `build_expression_tree`
 # this is unstable: it is subject to change, so don't rely on it!
-def _prune_expression_tree_empty_commas(
+def _fixup_expression_tree_references_to_tree_index(
+    expression_tree: list[list[int]],
+    existing_index: int,
+    new_index: int,
+):
+    for tree_node in expression_tree:
+        for i, index in enumerate(tree_node):
+            if index == existing_index:
+                tree_node[i] = new_index
+
+
+# internal to `build_expression_tree`
+# this is unstable: it is subject to change, so don't rely on it!
+def _fixup_expression_tree_lonely_commas(
     be2: BinExport2,
     operand: BinExport2.Operand,
     expression_tree: list[list[int]],
@@ -94,26 +131,12 @@ def _prune_expression_tree_empty_commas(
 
     if expression.type == BinExport2.Expression.OPERATOR:
         if len(children_tree_indexes) == 1 and expression.symbol == ",":
-            # Due to the above pruning of empty LSL or LSR expressions,
-            # the parents might need to be fixed up.
-            #
-            # Specifically, if the pruned node was part of a comma list with two children,
-            # now there's only a single child, which renders as an extra comma,
-            # so we replace references to the comma node with the immediate child.
-            #
-            # A more correct way of doing this might be to walk up the parents and do fixups,
-            # but I'm not quite sure how to do this yet. Just do two passes right now.
-            child = children_tree_indexes[0]
-
-            for tree_node in expression_tree:
-                tree_node.index
-                if tree_index in tree_node:
-                    tree_node[tree_node.index(tree_index)] = child
-
-            return
+            existing_index = tree_index
+            new_index = children_tree_indexes[0]
+            _fixup_expression_tree_references_to_tree_index(expression_tree, existing_index, new_index)
 
     for child_tree_index in children_tree_indexes:
-        _prune_expression_tree_empty_commas(be2, operand, expression_tree, child_tree_index)
+        _fixup_expression_tree_lonely_commas(be2, operand, expression_tree, child_tree_index)
 
 
 # internal to `build_expression_tree`
@@ -124,7 +147,7 @@ def _prune_expression_tree(
     expression_tree: list[list[int]],
 ):
     _prune_expression_tree_empty_shifts(be2, operand, expression_tree, 0)
-    _prune_expression_tree_empty_commas(be2, operand, expression_tree, 0)
+    _fixup_expression_tree_lonely_commas(be2, operand, expression_tree, 0)
 
 
 # this is unstable: it is subject to change, so don't rely on it!
@@ -173,7 +196,6 @@ def _build_expression_tree(
         tree.append(children)
 
     _prune_expression_tree(be2, operand, tree)
-    _prune_expression_tree(be2, operand, tree)
 
     return tree
 
@@ -193,9 +215,22 @@ def _fill_operand_expression_list(
     children_tree_indexes: list[int] = expression_tree[tree_index]
 
     if expression.type == BinExport2.Expression.REGISTER:
-        assert len(children_tree_indexes) == 0
+        assert len(children_tree_indexes) <= 1
         expression_list.append(expression)
-        return
+
+        if len(children_tree_indexes) == 0:
+            return
+        elif len(children_tree_indexes) == 1:
+            # like for aarch64 with vector instructions, indicating vector data size:
+            #
+            #     FADD V0.4S, V1.4S, V2.4S
+            #
+            # see: https://github.com/mandiant/capa/issues/2528
+            child_index = children_tree_indexes[0]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+            return
+        else:
+            raise NotImplementedError(len(children_tree_indexes))
 
     elif expression.type == BinExport2.Expression.SYMBOL:
         assert len(children_tree_indexes) <= 1
@@ -218,9 +253,23 @@ def _fill_operand_expression_list(
             raise NotImplementedError(len(children_tree_indexes))
 
     elif expression.type == BinExport2.Expression.IMMEDIATE_INT:
-        assert len(children_tree_indexes) == 0
+        assert len(children_tree_indexes) <= 1
         expression_list.append(expression)
-        return
+
+        if len(children_tree_indexes) == 0:
+            return
+        elif len(children_tree_indexes) == 1:
+            # the ghidra exporter can produce some weird expressions,
+            # particularly for MSRs, like for:
+            #
+            #     sreg(3, 0, c.0, c.4, 4)
+            #
+            # see: https://github.com/mandiant/capa/issues/2530
+            child_index = children_tree_indexes[0]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+            return
+        else:
+            raise NotImplementedError(len(children_tree_indexes))
 
     elif expression.type == BinExport2.Expression.SIZE_PREFIX:
         # like: b4

capa/features/extractors/binexport2/insn.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Iterator
 

capa/features/extractors/binja/basicblock.py

@@ -1,115 +1,32 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-import string
 from typing import Iterator
 
-from binaryninja import Function
 from binaryninja import BasicBlock as BinjaBasicBlock
-from binaryninja import (
-    BinaryView,
-    SymbolType,
-    RegisterValueType,
-    VariableSourceType,
-    MediumLevelILOperation,
-    MediumLevelILBasicBlock,
-    MediumLevelILInstruction,
-)
 
 from capa.features.common import Feature, Characteristic
 from capa.features.address import Address
 from capa.features.basicblock import BasicBlock
-from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
 from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
 
 
-def get_printable_len_ascii(s: bytes) -> int:
-    """Return string length if all operand bytes are ascii or utf16-le printable"""
-    count = 0
-    for c in s:
-        if c == 0:
-            return count
-        if c < 127 and chr(c) in string.printable:
-            count += 1
-    return count
-
-
-def get_printable_len_wide(s: bytes) -> int:
-    """Return string length if all operand bytes are ascii or utf16-le printable"""
-    if all(c == 0x00 for c in s[1::2]):
-        return get_printable_len_ascii(s[::2])
-    return 0
-
-
-def get_stack_string_len(f: Function, il: MediumLevelILInstruction) -> int:
-    bv: BinaryView = f.view
-
-    if il.operation != MediumLevelILOperation.MLIL_CALL:
-        return 0
-
-    target = il.dest
-    if target.operation not in [MediumLevelILOperation.MLIL_CONST, MediumLevelILOperation.MLIL_CONST_PTR]:
-        return 0
-
-    addr = target.value.value
-    sym = bv.get_symbol_at(addr)
-    if not sym or sym.type not in [SymbolType.LibraryFunctionSymbol, SymbolType.SymbolicFunctionSymbol]:
-        return 0
-
-    if sym.name not in ["__builtin_strncpy", "__builtin_strcpy", "__builtin_wcscpy"]:
-        return 0
-
-    if len(il.params) < 2:
-        return 0
-
-    dest = il.params[0]
-    if dest.operation in [MediumLevelILOperation.MLIL_ADDRESS_OF, MediumLevelILOperation.MLIL_VAR]:
-        var = dest.src
-    else:
-        return 0
-
-    if var.source_type != VariableSourceType.StackVariableSourceType:
-        return 0
-
-    src = il.params[1]
-    if src.value.type != RegisterValueType.ConstantDataAggregateValue:
-        return 0
-
-    s = f.get_constant_data(RegisterValueType.ConstantDataAggregateValue, src.value.value)
-    return max(get_printable_len_ascii(bytes(s)), get_printable_len_wide(bytes(s)))
-
-
-def bb_contains_stackstring(f: Function, bb: MediumLevelILBasicBlock) -> bool:
-    """check basic block for stackstring indicators
-
-    true if basic block contains enough moves of constant bytes to the stack
-    """
-    count = 0
-    for il in bb:
-        count += get_stack_string_len(f, il)
-        if count > MIN_STACKSTRING_LEN:
-            return True
-
-    return False
-
-
-def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
-    """extract stackstring indicators from basic block"""
-    bb: tuple[BinjaBasicBlock, MediumLevelILBasicBlock] = bbh.inner
-    if bb[1] is not None and bb_contains_stackstring(fh.inner, bb[1]):
-        yield Characteristic("stack string"), bbh.address
-
-
 def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """extract tight loop indicators from a basic block"""
-    bb: tuple[BinjaBasicBlock, MediumLevelILBasicBlock] = bbh.inner
-    for edge in bb[0].outgoing_edges:
-        if edge.target.start == bb[0].start:
+    bb: BinjaBasicBlock = bbh.inner
+    for edge in bb.outgoing_edges:
+        if edge.target.start == bb.start:
             yield Characteristic("tight loop"), bbh.address
 
 
@@ -121,7 +38,4 @@ def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Featur
     yield BasicBlock(), bbh.address
 
 
-BASIC_BLOCK_HANDLERS = (
-    extract_bb_tight_loop,
-    extract_bb_stackstring,
-)
+BASIC_BLOCK_HANDLERS = (extract_bb_tight_loop,)

capa/features/extractors/binja/extractor.py

@@ -1,14 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 import binaryninja as binja
-from binaryninja import ILException
 
 import capa.features.extractors.elf
 import capa.features.extractors.binja.file
@@ -54,23 +60,8 @@ class BinjaFeatureExtractor(StaticFeatureExtractor):
 
     def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
         f: binja.Function = fh.inner
-        # Set up a MLIL basic block dict look up to associate the disassembly basic block with its MLIL basic block
-        mlil_lookup = {}
-        try:
-            mlil = f.mlil
-        except ILException:
-            return
-
-        if mlil is None:
-            return
-
-        for mlil_bb in mlil.basic_blocks:
-            mlil_lookup[mlil_bb.source_block.start] = mlil_bb
-
         for bb in f.basic_blocks:
-            mlil_bb = mlil_lookup.get(bb.start)
-
-            yield BBHandle(address=AbsoluteVirtualAddress(bb.start), inner=(bb, mlil_bb))
+            yield BBHandle(address=AbsoluteVirtualAddress(bb.start), inner=bb)
 
     def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.binja.basicblock.extract_features(fh, bbh)
@@ -78,10 +69,10 @@ class BinjaFeatureExtractor(StaticFeatureExtractor):
     def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:
         import capa.features.extractors.binja.helpers as binja_helpers
 
-        bb: tuple[binja.BasicBlock, binja.MediumLevelILBasicBlock] = bbh.inner
-        addr = bb[0].start
+        bb: binja.BasicBlock = bbh.inner
+        addr = bb.start
 
-        for text, length in bb[0]:
+        for text, length in bb:
             insn = binja_helpers.DisassemblyInstruction(addr, length, text)
             yield InsnHandle(address=AbsoluteVirtualAddress(addr), inner=insn)
             addr += length

capa/features/extractors/binja/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 from binaryninja import Segment, BinaryView, SymbolType, SymbolBinding
@@ -18,6 +25,7 @@ from capa.features.common import (
     FORMAT_ELF,
     FORMAT_SC32,
     FORMAT_SC64,
+    FORMAT_BINJA_DB,
     Format,
     String,
     Feature,
@@ -137,6 +145,9 @@ def extract_file_function_names(bv: BinaryView) -> Iterator[tuple[Feature, Addre
 
 
 def extract_file_format(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
+    if bv.file.database is not None:
+        yield Format(FORMAT_BINJA_DB), NO_ADDRESS
+
     view_type = bv.view_type
     if view_type in ["PE", "COFF"]:
         yield Format(FORMAT_PE), NO_ADDRESS

capa/features/extractors/binja/find_binja_api.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import os
 import sys
 import logging
@@ -105,13 +112,13 @@ def find_binaryninja() -> Optional[Path]:
             logger.debug("detected OS: linux")
         elif sys.platform == "darwin":
             logger.warning("unsupported platform to find Binary Ninja: %s", sys.platform)
-            return False
+            return None
         elif sys.platform == "win32":
             logger.warning("unsupported platform to find Binary Ninja: %s", sys.platform)
-            return False
+            return None
         else:
             logger.warning("unsupported platform to find Binary Ninja: %s", sys.platform)
-            return False
+            return None
 
         desktop_entry = get_desktop_entry("com.vector35.binaryninja.desktop")
         if not desktop_entry:

capa/features/extractors/binja/function.py

@@ -1,18 +1,38 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import string
 from typing import Iterator
 
-from binaryninja import Function, BinaryView, SymbolType, ILException, RegisterValueType, LowLevelILOperation
+from binaryninja import (
+    Function,
+    BinaryView,
+    SymbolType,
+    RegisterValueType,
+    VariableSourceType,
+    LowLevelILOperation,
+    MediumLevelILOperation,
+    MediumLevelILBasicBlock,
+    MediumLevelILInstruction,
+)
 
 from capa.features.file import FunctionName
 from capa.features.common import Feature, Characteristic
 from capa.features.address import Address, AbsoluteVirtualAddress
 from capa.features.extractors import loops
+from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
+from capa.features.extractors.binja.helpers import get_llil_instr_at_addr
 from capa.features.extractors.base_extractor import FunctionHandle
 
 
@@ -24,14 +44,7 @@ def extract_function_calls_to(fh: FunctionHandle):
         # Everything that is a code reference to the current function is considered a caller, which actually includes
         # many other references that are NOT a caller. For example, an instruction `push function_start` will also be
         # considered a caller to the function
-        llil = None
-        try:
-            # Temporary fix for https://github.com/Vector35/binaryninja-api/issues/6020. Since `.llil` can throw an
-            # exception rather than returning None
-            llil = caller.llil
-        except ILException:
-            continue
-
+        llil = get_llil_instr_at_addr(func.view, caller.address)
         if (llil is None) or llil.operation not in [
             LowLevelILOperation.LLIL_CALL,
             LowLevelILOperation.LLIL_CALL_STACK_ADJUST,
@@ -40,14 +53,13 @@ def extract_function_calls_to(fh: FunctionHandle):
         ]:
             continue
 
-        if llil.dest.value.type not in [
-            RegisterValueType.ImportedAddressValue,
-            RegisterValueType.ConstantValue,
-            RegisterValueType.ConstantPointerValue,
+        if llil.dest.operation not in [
+            LowLevelILOperation.LLIL_CONST,
+            LowLevelILOperation.LLIL_CONST_PTR,
         ]:
             continue
 
-        address = llil.dest.value.value
+        address = llil.dest.constant
         if address != func.start:
             continue
 
@@ -102,10 +114,102 @@ def extract_function_name(fh: FunctionHandle):
             yield FunctionName(name[1:]), sym.address
 
 
+def get_printable_len_ascii(s: bytes) -> int:
+    """Return string length if all operand bytes are ascii or utf16-le printable"""
+    count = 0
+    for c in s:
+        if c == 0:
+            return count
+        if c < 127 and chr(c) in string.printable:
+            count += 1
+    return count
+
+
+def get_printable_len_wide(s: bytes) -> int:
+    """Return string length if all operand bytes are ascii or utf16-le printable"""
+    if all(c == 0x00 for c in s[1::2]):
+        return get_printable_len_ascii(s[::2])
+    return 0
+
+
+def get_stack_string_len(f: Function, il: MediumLevelILInstruction) -> int:
+    bv: BinaryView = f.view
+
+    if il.operation != MediumLevelILOperation.MLIL_CALL:
+        return 0
+
+    target = il.dest
+    if target.operation not in [MediumLevelILOperation.MLIL_CONST, MediumLevelILOperation.MLIL_CONST_PTR]:
+        return 0
+
+    addr = target.value.value
+    sym = bv.get_symbol_at(addr)
+    if not sym or sym.type not in [SymbolType.LibraryFunctionSymbol, SymbolType.SymbolicFunctionSymbol]:
+        return 0
+
+    if sym.name not in ["__builtin_strncpy", "__builtin_strcpy", "__builtin_wcscpy"]:
+        return 0
+
+    if len(il.params) < 2:
+        return 0
+
+    dest = il.params[0]
+    if dest.operation in [MediumLevelILOperation.MLIL_ADDRESS_OF, MediumLevelILOperation.MLIL_VAR]:
+        var = dest.src
+    else:
+        return 0
+
+    if var.source_type != VariableSourceType.StackVariableSourceType:
+        return 0
+
+    src = il.params[1]
+    if src.value.type != RegisterValueType.ConstantDataAggregateValue:
+        return 0
+
+    s = f.get_constant_data(RegisterValueType.ConstantDataAggregateValue, src.value.value)
+    return max(get_printable_len_ascii(bytes(s)), get_printable_len_wide(bytes(s)))
+
+
+def bb_contains_stackstring(f: Function, bb: MediumLevelILBasicBlock) -> bool:
+    """check basic block for stackstring indicators
+
+    true if basic block contains enough moves of constant bytes to the stack
+    """
+    count = 0
+    for il in bb:
+        count += get_stack_string_len(f, il)
+        if count > MIN_STACKSTRING_LEN:
+            return True
+
+    return False
+
+
+def extract_stackstring(fh: FunctionHandle):
+    """extract stackstring indicators"""
+    func: Function = fh.inner
+    bv: BinaryView = func.view
+    if bv is None:
+        return
+
+    mlil = func.mlil
+    if mlil is None:
+        return
+
+    for block in mlil.basic_blocks:
+        if bb_contains_stackstring(func, block):
+            yield Characteristic("stack string"), block.source_block.start
+
+
 def extract_features(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
     for func_handler in FUNCTION_HANDLERS:
         for feature, addr in func_handler(fh):
             yield feature, addr
 
 
-FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop, extract_recursive_call, extract_function_name)
+FUNCTION_HANDLERS = (
+    extract_function_calls_to,
+    extract_function_loop,
+    extract_recursive_call,
+    extract_function_name,
+    extract_stackstring,
+)

capa/features/extractors/binja/global_.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Iterator
 

capa/features/extractors/binja/helpers.py

@@ -1,15 +1,22 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import re
-from typing import Callable
+from typing import Callable, Optional
 from dataclasses import dataclass
 
-from binaryninja import BinaryView, LowLevelILInstruction
+from binaryninja import BinaryView, LowLevelILFunction, LowLevelILInstruction
 from binaryninja.architecture import InstructionTextToken
 
 
@@ -67,3 +74,13 @@ def read_c_string(bv: BinaryView, offset: int, max_len: int) -> str:
         s.append(chr(c))
 
     return "".join(s)
+
+
+def get_llil_instr_at_addr(bv: BinaryView, addr: int) -> Optional[LowLevelILInstruction]:
+    arch = bv.arch
+    buffer = bv.read(addr, arch.max_instr_length)
+    llil = LowLevelILFunction(arch=arch)
+    llil.current_address = addr
+    if arch.get_instruction_low_level_il(buffer, addr, llil) == 0:
+        return None
+    return llil[0]

capa/features/extractors/binja/insn.py

@@ -1,19 +1,25 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-from typing import Any, Iterator, Optional
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import Any, Optional
+from collections.abc import Iterator
 
-from binaryninja import Function
-from binaryninja import BasicBlock as BinjaBasicBlock
+import binaryninja as bn
 from binaryninja import (
+    Function,
     BinaryView,
     ILRegister,
     SymbolType,
-    ILException,
     BinaryReader,
     RegisterValueType,
     LowLevelILOperation,
@@ -24,7 +30,7 @@ import capa.features.extractors.helpers
 from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
 from capa.features.common import MAX_BYTES_FEATURE_SIZE, Bytes, String, Feature, Characteristic
 from capa.features.address import Address, AbsoluteVirtualAddress
-from capa.features.extractors.binja.helpers import DisassemblyInstruction, visit_llil_exprs
+from capa.features.extractors.binja.helpers import DisassemblyInstruction, visit_llil_exprs, get_llil_instr_at_addr
 from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
@@ -37,40 +43,24 @@ SECURITY_COOKIE_BYTES_DELTA = 0x40
 # 2. The function must only make one call/jump to another address
 # If the function being checked is a stub function, returns the target address. Otherwise, return None.
 def is_stub_function(bv: BinaryView, addr: int) -> Optional[int]:
-    funcs = bv.get_functions_at(addr)
-    for func in funcs:
-        if len(func.basic_blocks) != 1:
-            continue
+    llil = get_llil_instr_at_addr(bv, addr)
+    if llil is None or llil.operation not in [
+        LowLevelILOperation.LLIL_CALL,
+        LowLevelILOperation.LLIL_CALL_STACK_ADJUST,
+        LowLevelILOperation.LLIL_JUMP,
+        LowLevelILOperation.LLIL_TAILCALL,
+    ]:
+        return None
 
-        call_count = 0
-        call_target = None
-        try:
-            llil = func.llil
-        except ILException:
-            return None
+    # The LLIL instruction retrieved by `get_llil_instr_at_addr` did not go through a full analysis, so we cannot check
+    # `llil.dest.value.type` here
+    if llil.dest.operation not in [
+        LowLevelILOperation.LLIL_CONST,
+        LowLevelILOperation.LLIL_CONST_PTR,
+    ]:
+        return None
 
-        if llil is None:
-            continue
-
-        for il in llil.instructions:
-            if il.operation in [
-                LowLevelILOperation.LLIL_CALL,
-                LowLevelILOperation.LLIL_CALL_STACK_ADJUST,
-                LowLevelILOperation.LLIL_JUMP,
-                LowLevelILOperation.LLIL_TAILCALL,
-            ]:
-                call_count += 1
-                if il.dest.value.type in [
-                    RegisterValueType.ImportedAddressValue,
-                    RegisterValueType.ConstantValue,
-                    RegisterValueType.ConstantPointerValue,
-                ]:
-                    call_target = il.dest.value.value
-
-        if call_count == 1 and call_target is not None:
-            return call_target
-
-    return None
+    return llil.dest.constant
 
 
 def extract_insn_api_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
@@ -333,7 +323,7 @@ def extract_insn_offset_features(
     yield from results
 
 
-def is_nzxor_stack_cookie(f: Function, bb: BinjaBasicBlock, llil: LowLevelILInstruction) -> bool:
+def is_nzxor_stack_cookie(f: Function, bb: bn.BasicBlock, llil: LowLevelILInstruction) -> bool:
     """check if nzxor exists within stack cookie delta"""
     # TODO(xusheng): use LLIL SSA to do more accurate analysis
     # https://github.com/mandiant/capa/issues/1609
@@ -376,7 +366,7 @@ def extract_insn_nzxor_characteristic_features(
         # e.g., <llil: eax = 0>, (LLIL_SET_REG). So we do not need to check whether the two operands are the same.
         if il.operation == LowLevelILOperation.LLIL_XOR:
             # Exclude cases related to the stack cookie
-            if is_nzxor_stack_cookie(fh.inner, bbh.inner[0], il):
+            if is_nzxor_stack_cookie(fh.inner, bbh.inner, il):
                 return False
             results.append((Characteristic("nzxor"), ih.address))
             return False

capa/features/extractors/cape/call.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/cape/extractor.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Union, Iterator
@@ -15,8 +22,8 @@ import capa.features.extractors.cape.thread
 import capa.features.extractors.cape.global_
 import capa.features.extractors.cape.process
 from capa.exceptions import EmptyReportError, UnsupportedFormatError
-from capa.features.common import Feature, Characteristic
-from capa.features.address import NO_ADDRESS, Address, AbsoluteVirtualAddress, _NoAddress
+from capa.features.common import Feature
+from capa.features.address import Address, AbsoluteVirtualAddress, _NoAddress
 from capa.features.extractors.cape.models import Call, Static, Process, CapeReport
 from capa.features.extractors.base_extractor import (
     CallHandle,
@@ -47,7 +54,8 @@ class CapeExtractor(DynamicFeatureExtractor):
 
     def get_base_address(self) -> Union[AbsoluteVirtualAddress, _NoAddress, None]:
         # value according to the PE header, the actual trace may use a different imagebase
-        assert self.report.static is not None and self.report.static.pe is not None
+        assert self.report.static is not None
+        assert self.report.static.pe is not None
         return AbsoluteVirtualAddress(self.report.static.pe.imagebase)
 
     def extract_global_features(self) -> Iterator[tuple[Feature, Address]]:
@@ -70,11 +78,7 @@ class CapeExtractor(DynamicFeatureExtractor):
         yield from capa.features.extractors.cape.process.get_threads(ph)
 
     def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[tuple[Feature, Address]]:
-        if False:
-            # force this routine to be a generator,
-            # but we don't actually have any elements to generate.
-            yield Characteristic("never"), NO_ADDRESS
-        return
+        yield from []
 
     def get_calls(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[CallHandle]:
         yield from capa.features.extractors.cape.thread.get_calls(ph, th)

capa/features/extractors/cape/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator
@@ -81,31 +88,49 @@ def extract_file_strings(report: CapeReport) -> Iterator[tuple[Feature, Address]
 
 
 def extract_used_regkeys(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for regkey in report.behavior.summary.keys:
         yield String(regkey), NO_ADDRESS
 
 
 def extract_used_files(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for file in report.behavior.summary.files:
         yield String(file), NO_ADDRESS
 
 
 def extract_used_mutexes(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for mutex in report.behavior.summary.mutexes:
         yield String(mutex), NO_ADDRESS
 
 
 def extract_used_commands(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for cmd in report.behavior.summary.executed_commands:
         yield String(cmd), NO_ADDRESS
 
 
 def extract_used_apis(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for symbol in report.behavior.summary.resolved_apis:
         yield String(symbol), NO_ADDRESS
 
 
 def extract_used_services(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for svc in report.behavior.summary.created_services:
         yield String(svc), NO_ADDRESS
     for svc in report.behavior.summary.started_services:

capa/features/extractors/cape/global_.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/cape/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from typing import Any
 

capa/features/extractors/cape/models.py

@@ -1,12 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
-import binascii
-from typing import Any, Union, Literal, Optional, Annotated, TypeAlias
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Any, Union, Optional, Annotated, TypeAlias
 
 from pydantic import Field, BaseModel, ConfigDict
 from pydantic.functional_validators import BeforeValidator
@@ -20,7 +26,7 @@ def validate_hex_int(value):
 
 
 def validate_hex_bytes(value):
-    return binascii.unhexlify(value) if isinstance(value, str) else value
+    return bytes.fromhex(value) if isinstance(value, str) else value
 
 
 HexInt = Annotated[int, BeforeValidator(validate_hex_int)]
@@ -69,34 +75,37 @@ class Info(FlexibleModel):
     version: str
 
 
-class ImportedSymbol(ExactModel):
+class ImportedSymbol(FlexibleModel):
     address: HexInt
     name: Optional[str] = None
 
 
-class ImportedDll(ExactModel):
+class ImportedDll(FlexibleModel):
     dll: str
     imports: list[ImportedSymbol]
 
 
-class DirectoryEntry(ExactModel):
+"""
+class DirectoryEntry(FlexibleModel):
     name: str
     virtual_address: HexInt
     size: HexInt
+"""
 
 
-class Section(ExactModel):
+class Section(FlexibleModel):
     name: str
-    raw_address: HexInt
+    # raw_address: HexInt
     virtual_address: HexInt
-    virtual_size: HexInt
-    size_of_data: HexInt
-    characteristics: str
-    characteristics_raw: HexInt
-    entropy: float
+    # virtual_size: HexInt
+    # size_of_data: HexInt
+    # characteristics: str
+    # characteristics_raw: HexInt
+    # entropy: float
 
 
-class Resource(ExactModel):
+"""
+class Resource(FlexibleModel):
     name: str
     language: Optional[str] = None
     sublanguage: str
@@ -134,7 +143,7 @@ class DigitalSigner(FlexibleModel):
     extensions_subjectKeyIdentifier: Optional[str] = None
 
 
-class AuxSigner(ExactModel):
+class AuxSigner(FlexibleModel):
     name: str
     issued_to: str = Field(alias="Issued to")
     issued_by: str = Field(alias="Issued by")
@@ -142,7 +151,7 @@ class AuxSigner(ExactModel):
     sha1_hash: str = Field(alias="SHA1 hash")
 
 
-class Signer(ExactModel):
+class Signer(FlexibleModel):
     aux_sha1: Optional[str] = None
     aux_timestamp: Optional[str] = None
     aux_valid: Optional[bool] = None
@@ -151,60 +160,61 @@ class Signer(ExactModel):
     aux_signers: Optional[list[AuxSigner]] = None
 
 
-class Overlay(ExactModel):
+class Overlay(FlexibleModel):
     offset: HexInt
     size: HexInt
 
 
-class KV(ExactModel):
+class KV(FlexibleModel):
     name: str
     value: str
+"""
 
 
-class ExportedSymbol(ExactModel):
+class ExportedSymbol(FlexibleModel):
     address: HexInt
     name: str
-    ordinal: int
+    # ordinal: int
 
 
-class PE(ExactModel):
-    peid_signatures: TODO
+class PE(FlexibleModel):
+    # peid_signatures: TODO
     imagebase: HexInt
-    entrypoint: HexInt
-    reported_checksum: HexInt
-    actual_checksum: HexInt
-    osversion: str
-    pdbpath: Optional[str] = None
-    timestamp: str
+    # entrypoint: HexInt
+    # reported_checksum: HexInt
+    # actual_checksum: HexInt
+    # osversion: str
+    # pdbpath: Optional[str] = None
+    # timestamp: str
 
     # list[ImportedDll], or dict[basename(dll), ImportedDll]
-    imports: Union[list[ImportedDll], dict[str, ImportedDll]]
-    imported_dll_count: Optional[int] = None
-    imphash: str
+    imports: list[ImportedDll] | dict[str, ImportedDll] = Field(default_factory=list)  # type: ignore
+    # imported_dll_count: Optional[int] = None
+    # imphash: str
 
-    exported_dll_name: Optional[str] = None
-    exports: list[ExportedSymbol]
+    # exported_dll_name: Optional[str] = None
+    exports: list[ExportedSymbol] = Field(default_factory=list)
 
-    dirents: list[DirectoryEntry]
-    sections: list[Section]
+    # dirents: list[DirectoryEntry]
+    sections: list[Section] = Field(default_factory=list)
 
-    ep_bytes: Optional[HexBytes] = None
+    # ep_bytes: Optional[HexBytes] = None
 
-    overlay: Optional[Overlay] = None
-    resources: list[Resource]
-    versioninfo: list[KV]
+    # overlay: Optional[Overlay] = None
+    # resources: list[Resource]
+    # versioninfo: list[KV]
 
     # base64 encoded data
-    icon: Optional[str] = None
+    # icon: Optional[str] = None
     # MD5-like hash
-    icon_hash: Optional[str] = None
+    # icon_hash: Optional[str] = None
     # MD5-like hash
-    icon_fuzzy: Optional[str] = None
+    # icon_fuzzy: Optional[str] = None
     # short hex string
-    icon_dhash: Optional[str] = None
+    # icon_dhash: Optional[str] = None
 
-    digital_signers: list[DigitalSigner]
-    guest_signers: Signer
+    # digital_signers: list[DigitalSigner]
+    # guest_signers: Signer
 
 
 # TODO(mr-tz): target.file.dotnet, target.file.extracted_files, target.file.extracted_files_tool,
@@ -212,48 +222,49 @@ class PE(ExactModel):
 # https://github.com/mandiant/capa/issues/1814
 class File(FlexibleModel):
     type: str
-    cape_type_code: Optional[int] = None
-    cape_type: Optional[str] = None
+    # cape_type_code: Optional[int] = None
+    # cape_type: Optional[str] = None
 
-    pid: Optional[Union[int, Literal[""]]] = None
-    name: Union[list[str], str]
-    path: str
-    guest_paths: Union[list[str], str, None]
-    timestamp: Optional[str] = None
+    # pid: Optional[Union[int, Literal[""]]] = None
+    # name: Union[list[str], str]
+    # path: str
+    # guest_paths: Union[list[str], str, None]
+    # timestamp: Optional[str] = None
 
     #
     # hashes
     #
-    crc32: str
+    # crc32: str
     md5: str
     sha1: str
     sha256: str
-    sha512: str
-    sha3_384: Optional[str] = None
-    ssdeep: str
+    # sha512: str
+    # sha3_384: Optional[str] = None
+    # ssdeep: str
     # unsure why this would ever be "False"
-    tlsh: Optional[Union[str, bool]] = None
-    rh_hash: Optional[str] = None
+    # tlsh: Optional[Union[str, bool]] = None
+    # rh_hash: Optional[str] = None
 
     #
     # other metadata, static analysis
     #
-    size: int
+    # size: int
     pe: Optional[PE] = None
-    ep_bytes: Optional[HexBytes] = None
-    entrypoint: Optional[int] = None
-    data: Optional[str] = None
-    strings: Optional[list[str]] = None
+    # ep_bytes: Optional[HexBytes] = None
+    # entrypoint: Optional[int] = None
+    # data: Optional[str] = None
+    # strings: Optional[list[str]] = None
 
     #
     # detections (skip)
     #
-    yara: Skip = None
-    cape_yara: Skip = None
-    clamav: Skip = None
-    virustotal: Skip = None
+    # yara: Skip = None
+    # cape_yara: Skip = None
+    # clamav: Skip = None
+    # virustotal: Skip = None
 
 
+"""
 class ProcessFile(File):
     #
     # like a File, but also has dynamic analysis results
@@ -266,49 +277,54 @@ class ProcessFile(File):
     target_pid: Optional[Union[int, str]] = None
     target_path: Optional[str] = None
     target_process: Optional[str] = None
+"""
 
 
-class Argument(ExactModel):
+class Argument(FlexibleModel):
     name: str
     # unsure why empty list is provided here
     value: Union[HexInt, int, str, EmptyList]
     pretty_value: Optional[str] = None
 
 
-class Call(ExactModel):
-    timestamp: str
+class Call(FlexibleModel):
+    # timestamp: str
     thread_id: int
-    category: str
+    # category: str
 
     api: str
 
     arguments: list[Argument]
-    status: bool
+    # status: bool
     return_: HexInt = Field(alias="return")
     pretty_return: Optional[str] = None
 
-    repeated: int
+    # repeated: int
 
     # virtual addresses
-    caller: HexInt
-    parentcaller: HexInt
+    # caller: HexInt
+    # parentcaller: HexInt
 
     # index into calls array
-    id: int
+    # id: int
 
 
-class Process(ExactModel):
+# FlexibleModel to account for extended fields
+# refs: https://github.com/mandiant/capa/issues/2466
+# https://github.com/kevoreilly/CAPEv2/pull/2199
+class Process(FlexibleModel):
     process_id: int
     process_name: str
     parent_id: int
-    module_path: str
-    first_seen: str
+    # module_path: str
+    # first_seen: str
     calls: list[Call]
     threads: list[int]
     environ: dict[str, str]
 
 
-class ProcessTree(ExactModel):
+"""
+class ProcessTree(FlexibleModel):
     name: str
     pid: int
     parent_id: int
@@ -316,17 +332,18 @@ class ProcessTree(ExactModel):
     threads: list[int]
     environ: dict[str, str]
     children: list["ProcessTree"]
+"""
 
 
-class Summary(ExactModel):
+class Summary(FlexibleModel):
     files: list[str]
-    read_files: list[str]
-    write_files: list[str]
-    delete_files: list[str]
+    # read_files: list[str]
+    # write_files: list[str]
+    # delete_files: list[str]
     keys: list[str]
-    read_keys: list[str]
-    write_keys: list[str]
-    delete_keys: list[str]
+    # read_keys: list[str]
+    # write_keys: list[str]
+    # delete_keys: list[str]
     executed_commands: list[str]
     resolved_apis: list[str]
     mutexes: list[str]
@@ -334,7 +351,8 @@ class Summary(ExactModel):
     started_services: list[str]
 
 
-class EncryptedBuffer(ExactModel):
+"""
+class EncryptedBuffer(FlexibleModel):
     process_name: str
     pid: int
 
@@ -342,38 +360,41 @@ class EncryptedBuffer(ExactModel):
     buffer: str
     buffer_size: Optional[int] = None
     crypt_key: Optional[Union[HexInt, str]] = None
+"""
 
 
-class Behavior(ExactModel):
-    summary: Summary
+class Behavior(FlexibleModel):
+    summary: Summary | None = None
 
     # list of processes, of threads, of calls
     processes: list[Process]
     # tree of processes
-    processtree: list[ProcessTree]
+    # processtree: list[ProcessTree]
 
-    anomaly: list[str]
-    encryptedbuffers: list[EncryptedBuffer]
+    # anomaly: list[str]
+    # encryptedbuffers: list[EncryptedBuffer]
     # these are small objects that describe atomic events,
     # like file move, registry access.
     # we'll detect the same with our API call analysis.
-    enhanced: Skip = None
+    # enhanced: Skip = None
 
 
-class Target(ExactModel):
-    category: str
+class Target(FlexibleModel):
+    # category: str
     file: File
+    # pe: Optional[PE] = None
+
+
+class Static(FlexibleModel):
     pe: Optional[PE] = None
+    # flare_capa: Skip = None
 
 
-class Static(ExactModel):
-    pe: Optional[PE] = None
-    flare_capa: Skip = None
-
-
-class Cape(ExactModel):
+"""
+class Cape(FlexibleModel):
     payloads: list[ProcessFile]
     configs: Skip = None
+"""
 
 
 # flexible because there may be more sorts of analysis
@@ -396,15 +417,14 @@ class CapeReport(FlexibleModel):
     # post-processed results: process tree, anomalies, etc
     behavior: Behavior
 
-    # post-processed results: payloads and extracted configs
-    CAPE: Optional[Union[Cape, list]] = None
-    dropped: Optional[list[File]] = None
-    procdump: Optional[list[ProcessFile]] = None
-    procmemory: ListTODO
-
     # =========================================================================
     # information we won't use in capa
     #
+    # post-processed results: payloads and extracted configs
+    # CAPE: Optional[Union[Cape, list]] = None
+    # dropped: Optional[list[File]] = None
+    # procdump: Optional[list[ProcessFile]] = None
+    # procmemory: Optional[ListTODO] = None
 
     #
     # NBIs and HBIs
@@ -413,32 +433,32 @@ class CapeReport(FlexibleModel):
     #
     # if we come up with a future use for this, go ahead and re-enable!
     #
-    network: Skip = None
-    suricata: Skip = None
-    curtain: Skip = None
-    sysmon: Skip = None
-    url_analysis: Skip = None
+    # network: Skip = None
+    # suricata: Skip = None
+    # curtain: Skip = None
+    # sysmon: Skip = None
+    # url_analysis: Skip = None
 
     # screenshot hash values
-    deduplicated_shots: Skip = None
+    # deduplicated_shots: Skip = None
     # k-v pairs describing the time it took to run each stage.
-    statistics: Skip = None
+    # statistics: Skip = None
     # k-v pairs of ATT&CK ID to signature name or similar.
-    ttps: Skip = None
+    # ttps: Skip = None
     # debug log messages
-    debug: Skip = None
+    # debug: Skip = None
 
     # various signature matches
     # we could potentially extend capa to use this info one day,
     # though it would be quite sandbox-specific,
     # and more detection-oriented than capability detection.
-    signatures: Skip = None
-    malfamily_tag: Optional[str] = None
-    malscore: float
-    detections: Skip = None
-    detections2pid: Optional[dict[int, list[str]]] = None
+    # signatures: Skip = None
+    # malfamily_tag: Optional[str] = None
+    # malscore: float
+    # detections: Skip = None
+    # detections2pid: Optional[dict[int, list[str]]] = None
     # AV detections for the sample.
-    virustotal: Skip = None
+    # virustotal: Skip = None
 
     @classmethod
     def from_buf(cls, buf: bytes) -> "CapeReport":

capa/features/extractors/cape/process.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/cape/thread.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/common.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import io
 import re
 import logging

capa/features/extractors/dnfile/extractor.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/function.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/insn.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/types.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from typing import Optional
 

capa/features/extractors/dotnetfile.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Iterator
 from pathlib import Path

capa/features/extractors/drakvuf/call.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/extractor.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Union, Iterator
@@ -14,7 +21,7 @@ import capa.features.extractors.drakvuf.file
 import capa.features.extractors.drakvuf.thread
 import capa.features.extractors.drakvuf.global_
 import capa.features.extractors.drakvuf.process
-from capa.features.common import Feature, Characteristic
+from capa.features.common import Feature
 from capa.features.address import NO_ADDRESS, Address, ThreadAddress, ProcessAddress, AbsoluteVirtualAddress, _NoAddress
 from capa.features.extractors.base_extractor import (
     CallHandle,
@@ -67,11 +74,7 @@ class DrakvufExtractor(DynamicFeatureExtractor):
         yield from capa.features.extractors.drakvuf.process.get_threads(self.sorted_calls, ph)
 
     def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[tuple[Feature, Address]]:
-        if False:
-            # force this routine to be a generator,
-            # but we don't actually have any elements to generate.
-            yield Characteristic("never"), NO_ADDRESS
-        return
+        yield from []
 
     def get_calls(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[CallHandle]:
         yield from capa.features.extractors.drakvuf.thread.get_calls(self.sorted_calls, ph, th)

capa/features/extractors/drakvuf/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/global_.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import itertools
 

capa/features/extractors/drakvuf/models.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Any, Iterator
 

capa/features/extractors/drakvuf/process.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/thread.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/elf.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import struct
 import logging
 import itertools
@@ -1081,7 +1088,7 @@ def guess_os_from_go_buildinfo(elf: ELF) -> Optional[OS]:
     #  and the 32-byte header is followed by varint-prefixed string data
     #  for the two string values we care about.
     # https://github.com/mandiant/GoReSym/blob/0860a1b1b4f3495e9fb7e71eb4386bf3e0a7c500/buildinfo/buildinfo.go#L185-L193
-    BUILDINFO_MAGIC = b"\xFF Go buildinf:"
+    BUILDINFO_MAGIC = b"\xff Go buildinf:"
 
     try:
         index = buf.index(BUILDINFO_MAGIC)

capa/features/extractors/elffile.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import io
 import logging
 from typing import Iterator
@@ -73,8 +80,7 @@ def extract_file_export_names(elf: ELFFile, **kwargs):
 
 
 def extract_file_import_names(elf: ELFFile, **kwargs):
-    # Create a dictionary to store symbol names by their index
-    symbol_names = {}
+    symbol_name_by_index: dict[int, str] = {}
 
     # Extract symbol names and store them in the dictionary
     for segment in elf.iter_segments():
@@ -86,7 +92,7 @@ def extract_file_import_names(elf: ELFFile, **kwargs):
             logger.debug("Dynamic segment doesn't contain DT_SYMTAB")
             continue
 
-        for _, symbol in enumerate(segment.iter_symbols()):
+        for i, symbol in enumerate(segment.iter_symbols()):
             # The following conditions are based on the following article
             # http://www.m4b.io/elf/export/binary/analysis/2015/05/25/what-is-an-elf-export.html
             if not symbol.name:
@@ -100,7 +106,7 @@ def extract_file_import_names(elf: ELFFile, **kwargs):
             if symbol.entry.st_name == 0:
                 continue
 
-            symbol_names[_] = symbol.name
+            symbol_name_by_index[i] = symbol.name
 
     for segment in elf.iter_segments():
         if not isinstance(segment, DynamicSegment):
@@ -120,10 +126,17 @@ def extract_file_import_names(elf: ELFFile, **kwargs):
                     break
 
             for relocation in relocations:
-                # Extract the symbol name from the symbol table using the symbol index in the relocation
-                if relocation["r_info_sym"] not in symbol_names:
+                if "r_info_sym" not in relocation.entry or "r_offset" not in relocation.entry:
                     continue
-                yield Import(symbol_names[relocation["r_info_sym"]]), FileOffsetAddress(relocation["r_offset"])
+
+                symbol_address: int = relocation["r_offset"]
+                symbol_index: int = relocation["r_info_sym"]
+
+                if symbol_index not in symbol_name_by_index:
+                    continue
+                symbol_name = symbol_name_by_index[symbol_index]
+
+                yield Import(symbol_name), FileOffsetAddress(symbol_address)
 
 
 def extract_file_section_names(elf: ELFFile, **kwargs):

capa/features/extractors/ghidra/basicblock.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import string
 import struct
@@ -76,7 +83,7 @@ def bb_contains_stackstring(bb: ghidra.program.model.block.CodeBlock) -> bool:
     true if basic block contains enough moves of constant bytes to the stack
     """
     count = 0
-    for insn in currentProgram().getListing().getInstructions(bb, True):  # type: ignore [name-defined] # noqa: F821
+    for insn in capa.features.extractors.ghidra.helpers.get_current_program().getListing().getInstructions(bb, True):
         if is_mov_imm_to_stack(insn):
             count += get_printable_len(insn.getScalar(1))
         if count > MIN_STACKSTRING_LEN:
@@ -89,7 +96,9 @@ def _bb_has_tight_loop(bb: ghidra.program.model.block.CodeBlock):
     parse tight loops, true if last instruction in basic block branches to bb start
     """
     # Reverse Ordered, first InstructionDB
-    last_insn = currentProgram().getListing().getInstructions(bb, False).next()  # type: ignore [name-defined] # noqa: F821
+    last_insn = (
+        capa.features.extractors.ghidra.helpers.get_current_program().getListing().getInstructions(bb, False).next()
+    )
 
     if last_insn.getFlowType().isJump():
         return last_insn.getAddress(0) == bb.getMinAddress()
@@ -133,20 +142,3 @@ def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Featur
     for bb_handler in BASIC_BLOCK_HANDLERS:
         for feature, addr in bb_handler(fh, bbh):
             yield feature, addr
-
-
-def main():
-    features = []
-    from capa.features.extractors.ghidra.extractor import GhidraFeatureExtractor
-
-    for fh in GhidraFeatureExtractor().get_functions():
-        for bbh in capa.features.extractors.ghidra.helpers.get_function_blocks(fh):
-            features.extend(list(extract_features(fh, bbh)))
-
-    import pprint
-
-    pprint.pprint(features)  # noqa: T203
-
-
-if __name__ == "__main__":
-    main()

capa/features/extractors/ghidra/context.py

@@ -0,0 +1,36 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Optional
+
+
+class GhidraContext:
+    def __init__(self, program, flat_api, monitor):
+        self.program = program
+        self.flat_api = flat_api
+        self.monitor = monitor
+
+
+_context: Optional[GhidraContext] = None
+
+
+def set_context(program, flat_api, monitor):
+    global _context
+    _context = GhidraContext(program, flat_api, monitor)
+
+
+def get_context() -> GhidraContext:
+    if _context is None:
+        raise RuntimeError("GhidraContext not initialized")
+    return _context

capa/features/extractors/ghidra/extractor.py

@@ -1,10 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import contextlib
 from typing import Iterator
 
 import capa.features.extractors.ghidra.file
@@ -24,19 +32,21 @@ from capa.features.extractors.base_extractor import (
 
 
 class GhidraFeatureExtractor(StaticFeatureExtractor):
-    def __init__(self):
+    def __init__(self, ctx_manager=None, tmpdir=None):
+        self.ctx_manager = ctx_manager
+        self.tmpdir = tmpdir
         import capa.features.extractors.ghidra.helpers as ghidra_helpers
 
         super().__init__(
             SampleHashes(
-                md5=capa.ghidra.helpers.get_file_md5(),
+                md5=ghidra_helpers.get_current_program().getExecutableMD5(),
                 # ghidra doesn't expose this hash.
                 # https://ghidra.re/ghidra_docs/api/ghidra/program/model/listing/Program.html
                 #
                 # the hashes are stored in the database, not computed on the fly,
                 # so it's probably not trivial to add SHA1.
                 sha1="",
-                sha256=capa.ghidra.helpers.get_file_sha256(),
+                sha256=ghidra_helpers.get_current_program().getExecutableSHA256(),
             )
         )
 
@@ -49,7 +59,17 @@ class GhidraFeatureExtractor(StaticFeatureExtractor):
         self.fakes = ghidra_helpers.map_fake_import_addrs()
 
     def get_base_address(self):
-        return AbsoluteVirtualAddress(currentProgram().getImageBase().getOffset())  # type: ignore [name-defined] # noqa: F821
+        import capa.features.extractors.ghidra.helpers as ghidra_helpers
+
+        return AbsoluteVirtualAddress(ghidra_helpers.get_current_program().getImageBase().getOffset())
+
+    def __del__(self):
+        if hasattr(self, "ctx_manager") and self.ctx_manager:
+            with contextlib.suppress(Exception):
+                self.ctx_manager.__exit__(None, None, None)
+        if hasattr(self, "tmpdir") and self.tmpdir:
+            with contextlib.suppress(Exception):
+                self.tmpdir.cleanup()
 
     def extract_global_features(self):
         yield from self.global_features
@@ -70,7 +90,9 @@ class GhidraFeatureExtractor(StaticFeatureExtractor):
 
     @staticmethod
     def get_function(addr: int) -> FunctionHandle:
-        func = getFunctionContaining(toAddr(addr))  # type: ignore [name-defined] # noqa: F821
+        import capa.features.extractors.ghidra.helpers as ghidra_helpers
+
+        func = ghidra_helpers.get_flat_api().getFunctionContaining(ghidra_helpers.get_flat_api().toAddr(addr))
         return FunctionHandle(address=AbsoluteVirtualAddress(func.getEntryPoint().getOffset()), inner=func)
 
     def extract_function_features(self, fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:

capa/features/extractors/ghidra/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import re
 import struct
 from typing import Iterator
@@ -73,7 +80,7 @@ def extract_file_embedded_pe() -> Iterator[tuple[Feature, Address]]:
         for i in range(256)
     ]
 
-    for block in currentProgram().getMemory().getBlocks():  # type: ignore [name-defined] # noqa: F821
+    for block in capa.features.extractors.ghidra.helpers.get_current_program().getMemory().getBlocks():
         if not all((block.isLoaded(), block.isInitialized(), "Headers" not in block.getName())):
             continue
 
@@ -86,9 +93,37 @@ def extract_file_embedded_pe() -> Iterator[tuple[Feature, Address]]:
 
 def extract_file_export_names() -> Iterator[tuple[Feature, Address]]:
     """extract function exports"""
-    st = currentProgram().getSymbolTable()  # type: ignore [name-defined] # noqa: F821
+    program = capa.features.extractors.ghidra.helpers.get_current_program()
+    st = program.getSymbolTable()
+
     for addr in st.getExternalEntryPointIterator():
-        yield Export(st.getPrimarySymbol(addr).getName()), AbsoluteVirtualAddress(addr.getOffset())
+        sym = st.getPrimarySymbol(addr)
+        name = sym.getName()
+
+        # Check for forwarded export
+        is_forwarded = False
+        refs = program.getReferenceManager().getReferencesFrom(addr)
+        for ref in refs:
+            if ref.getToAddress().isExternalAddress():
+                ext_sym = st.getPrimarySymbol(ref.getToAddress())
+                if ext_sym:
+                    ext_loc = program.getExternalManager().getExternalLocation(ext_sym)
+                    if ext_loc:
+                        # It is a forwarded export
+                        libname = ext_loc.getLibraryName()
+                        if libname.lower().endswith(".dll"):
+                            libname = libname[:-4]
+
+                        forwarded_name = f"{libname}.{ext_loc.getLabel()}"
+                        forwarded_name = capa.features.extractors.helpers.reformat_forwarded_export_name(forwarded_name)
+
+                        yield Export(forwarded_name), AbsoluteVirtualAddress(addr.getOffset())
+                        yield Characteristic("forwarded export"), AbsoluteVirtualAddress(addr.getOffset())
+                        is_forwarded = True
+                        break
+
+        if not is_forwarded:
+            yield Export(name), AbsoluteVirtualAddress(addr.getOffset())
 
 
 def extract_file_import_names() -> Iterator[tuple[Feature, Address]]:
@@ -103,7 +138,7 @@ def extract_file_import_names() -> Iterator[tuple[Feature, Address]]:
      - importname
     """
 
-    for f in currentProgram().getFunctionManager().getExternalFunctions():  # type: ignore [name-defined] # noqa: F821
+    for f in capa.features.extractors.ghidra.helpers.get_current_program().getFunctionManager().getExternalFunctions():
         for r in f.getSymbol().getReferences():
             if r.getReferenceType().isData():
                 addr = r.getFromAddress().getOffset()  # gets pointer to fake external addr
@@ -119,14 +154,14 @@ def extract_file_import_names() -> Iterator[tuple[Feature, Address]]:
 def extract_file_section_names() -> Iterator[tuple[Feature, Address]]:
     """extract section names"""
 
-    for block in currentProgram().getMemory().getBlocks():  # type: ignore [name-defined] # noqa: F821
+    for block in capa.features.extractors.ghidra.helpers.get_current_program().getMemory().getBlocks():
         yield Section(block.getName()), AbsoluteVirtualAddress(block.getStart().getOffset())
 
 
 def extract_file_strings() -> Iterator[tuple[Feature, Address]]:
     """extract ASCII and UTF-16 LE strings"""
 
-    for block in currentProgram().getMemory().getBlocks():  # type: ignore [name-defined] # noqa: F821
+    for block in capa.features.extractors.ghidra.helpers.get_current_program().getMemory().getBlocks():
         if not block.isInitialized():
             continue
 
@@ -146,7 +181,8 @@ def extract_file_function_names() -> Iterator[tuple[Feature, Address]]:
     extract the names of statically-linked library functions.
     """
 
-    for sym in currentProgram().getSymbolTable().getAllSymbols(True):  # type: ignore [name-defined] # noqa: F821
+    for sym in capa.features.extractors.ghidra.helpers.get_current_program().getSymbolTable().getAllSymbols(True):
+
         # .isExternal() misses more than this config for the function symbols
         if sym.getSymbolType() == SymbolType.FUNCTION and sym.getSource() == SourceType.ANALYSIS and sym.isGlobal():
             name = sym.getName()  # starts to resolve names based on Ghidra's FidDB
@@ -163,7 +199,7 @@ def extract_file_function_names() -> Iterator[tuple[Feature, Address]]:
 
 
 def extract_file_format() -> Iterator[tuple[Feature, Address]]:
-    ef = currentProgram().getExecutableFormat()  # type: ignore [name-defined] # noqa: F821
+    ef = capa.features.extractors.ghidra.helpers.get_current_program().getExecutableFormat()
     if "PE" in ef:
         yield Format(FORMAT_PE), NO_ADDRESS
     elif "ELF" in ef:

capa/features/extractors/ghidra/function.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 import ghidra
@@ -19,18 +26,22 @@ from capa.features.extractors.base_extractor import FunctionHandle
 
 def extract_function_calls_to(fh: FunctionHandle):
     """extract callers to a function"""
-    f: ghidra.program.database.function.FunctionDB = fh.inner
+    f: "ghidra.program.database.function.FunctionDB" = fh.inner
     for ref in f.getSymbol().getReferences():
         if ref.getReferenceType().isCall():
             yield Characteristic("calls to"), AbsoluteVirtualAddress(ref.getFromAddress().getOffset())
 
 
 def extract_function_loop(fh: FunctionHandle):
-    f: ghidra.program.database.function.FunctionDB = fh.inner
+    f: "ghidra.program.database.function.FunctionDB" = fh.inner
 
     edges = []
-    for block in SimpleBlockIterator(BasicBlockModel(currentProgram()), f.getBody(), monitor()):  # type: ignore [name-defined] # noqa: F821
-        dests = block.getDestinations(monitor())  # type: ignore [name-defined] # noqa: F821
+    for block in SimpleBlockIterator(
+        BasicBlockModel(capa.features.extractors.ghidra.helpers.get_current_program()),
+        f.getBody(),
+        capa.features.extractors.ghidra.helpers.get_monitor(),
+    ):
+        dests = block.getDestinations(capa.features.extractors.ghidra.helpers.get_monitor())
         s_addrs = block.getStartAddresses()
 
         while dests.hasNext():  # For loop throws Python TypeError
@@ -42,16 +53,17 @@ def extract_function_loop(fh: FunctionHandle):
 
 
 def extract_recursive_call(fh: FunctionHandle):
-    f: ghidra.program.database.function.FunctionDB = fh.inner
+    f: "ghidra.program.database.function.FunctionDB" = fh.inner
 
-    for func in f.getCalledFunctions(monitor()):  # type: ignore [name-defined] # noqa: F821
+    for func in f.getCalledFunctions(capa.features.extractors.ghidra.helpers.get_monitor()):
         if func.getEntryPoint().getOffset() == f.getEntryPoint().getOffset():
             yield Characteristic("recursive call"), AbsoluteVirtualAddress(f.getEntryPoint().getOffset())
 
 
 def extract_features(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
-    for func_handler in FUNCTION_HANDLERS:
-        for feature, addr in func_handler(fh):
+    """extract function features"""
+    for function_handler in FUNCTION_HANDLERS:
+        for feature, addr in function_handler(fh):
             yield feature, addr
 
 

capa/features/extractors/ghidra/global_.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 import contextlib
 from typing import Iterator
@@ -19,7 +26,7 @@ logger = logging.getLogger(__name__)
 
 
 def extract_os() -> Iterator[tuple[Feature, Address]]:
-    format_name: str = currentProgram().getExecutableFormat()  # type: ignore [name-defined] # noqa: F821
+    format_name: str = capa.features.extractors.ghidra.helpers.get_current_program().getExecutableFormat()
 
     if "PE" in format_name:
         yield OS(OS_WINDOWS), NO_ADDRESS
@@ -46,7 +53,7 @@ def extract_os() -> Iterator[tuple[Feature, Address]]:
 
 
 def extract_arch() -> Iterator[tuple[Feature, Address]]:
-    lang_id = currentProgram().getMetadata().get("Language ID")  # type: ignore [name-defined] # noqa: F821
+    lang_id = capa.features.extractors.ghidra.helpers.get_current_program().getMetadata().get("Language ID")
 
     if "x86" in lang_id and "64" in lang_id:
         yield Arch(ARCH_AMD64), NO_ADDRESS

capa/features/extractors/ghidra/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 import ghidra
@@ -15,9 +22,22 @@ from ghidra.program.model.symbol import SourceType, SymbolType
 from ghidra.program.model.address import AddressSpace
 
 import capa.features.extractors.helpers
+import capa.features.extractors.ghidra.context as ghidra_context
 from capa.features.common import THUNK_CHAIN_DEPTH_DELTA
 from capa.features.address import AbsoluteVirtualAddress
-from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle
+
+
+def get_current_program():
+    return ghidra_context.get_context().program
+
+
+def get_monitor():
+    return ghidra_context.get_context().monitor
+
+
+def get_flat_api():
+    return ghidra_context.get_context().flat_api
 
 
 def ints_to_bytes(bytez: list[int]) -> bytes:
@@ -29,7 +49,7 @@ def ints_to_bytes(bytez: list[int]) -> bytes:
     return bytes([b & 0xFF for b in bytez])
 
 
-def find_byte_sequence(addr: ghidra.program.model.address.Address, seq: bytes) -> Iterator[int]:
+def find_byte_sequence(addr: "ghidra.program.model.address.Address", seq: bytes) -> Iterator[int]:
     """yield all ea of a given byte sequence
 
     args:
@@ -37,12 +57,12 @@ def find_byte_sequence(addr: ghidra.program.model.address.Address, seq: bytes) -
         seq: bytes to search e.g. b"\x01\x03"
     """
     seqstr = "".join([f"\\x{b:02x}" for b in seq])
-    eas = findBytes(addr, seqstr, java.lang.Integer.MAX_VALUE, 1)  # type: ignore [name-defined] # noqa: F821
+    eas = get_flat_api().findBytes(addr, seqstr, java.lang.Integer.MAX_VALUE, 1)
 
     yield from eas
 
 
-def get_bytes(addr: ghidra.program.model.address.Address, length: int) -> bytes:
+def get_bytes(addr: "ghidra.program.model.address.Address", length: int) -> bytes:
     """yield length bytes at addr
 
     args:
@@ -50,12 +70,12 @@ def get_bytes(addr: ghidra.program.model.address.Address, length: int) -> bytes:
         length: length of bytes to pull
     """
     try:
-        return ints_to_bytes(getBytes(addr, length))  # type: ignore [name-defined] # noqa: F821
-    except RuntimeError:
+        return ints_to_bytes(get_flat_api().getBytes(addr, int(length)))
+    except Exception:
         return b""
 
 
-def get_block_bytes(block: ghidra.program.model.mem.MemoryBlock) -> bytes:
+def get_block_bytes(block: "ghidra.program.model.mem.MemoryBlock") -> bytes:
     """yield all bytes in a given block
 
     args:
@@ -66,20 +86,21 @@ def get_block_bytes(block: ghidra.program.model.mem.MemoryBlock) -> bytes:
 
 def get_function_symbols():
     """yield all non-external function symbols"""
-    yield from currentProgram().getFunctionManager().getFunctionsNoStubs(True)  # type: ignore [name-defined] # noqa: F821
+    yield from get_current_program().getFunctionManager().getFunctionsNoStubs(True)
 
 
-def get_function_blocks(fh: FunctionHandle) -> Iterator[BBHandle]:
-    """yield BBHandle for each bb in a given function"""
+def get_function_blocks(fh: "capa.features.extractors.base_extractor.FunctionHandle") -> Iterator[BBHandle]:
+    """
+    yield the basic blocks of the function
+    """
 
-    func: ghidra.program.database.function.FunctionDB = fh.inner
-    for bb in SimpleBlockIterator(BasicBlockModel(currentProgram()), func.getBody(), monitor()):  # type: ignore [name-defined] # noqa: F821
-        yield BBHandle(address=AbsoluteVirtualAddress(bb.getMinAddress().getOffset()), inner=bb)
+    for block in SimpleBlockIterator(BasicBlockModel(get_current_program()), fh.inner.getBody(), get_monitor()):
+        yield BBHandle(address=AbsoluteVirtualAddress(block.getMinAddress().getOffset()), inner=block)
 
 
 def get_insn_in_range(bbh: BBHandle) -> Iterator[InsnHandle]:
     """yield InshHandle for each insn in a given basicblock"""
-    for insn in currentProgram().getListing().getInstructions(bbh.inner, True):  # type: ignore [name-defined] # noqa: F821
+    for insn in get_current_program().getListing().getInstructions(bbh.inner, True):
         yield InsnHandle(address=AbsoluteVirtualAddress(insn.getAddress().getOffset()), inner=insn)
 
 
@@ -88,7 +109,7 @@ def get_file_imports() -> dict[int, list[str]]:
 
     import_dict: dict[int, list[str]] = {}
 
-    for f in currentProgram().getFunctionManager().getExternalFunctions():  # type: ignore [name-defined] # noqa: F821
+    for f in get_current_program().getFunctionManager().getExternalFunctions():
         for r in f.getSymbol().getReferences():
             if r.getReferenceType().isData():
                 addr = r.getFromAddress().getOffset()  # gets pointer to fake external addr
@@ -126,7 +147,7 @@ def get_file_externs() -> dict[int, list[str]]:
 
     extern_dict: dict[int, list[str]] = {}
 
-    for sym in currentProgram().getSymbolTable().getAllSymbols(True):  # type: ignore [name-defined] # noqa: F821
+    for sym in get_current_program().getSymbolTable().getAllSymbols(True):
         # .isExternal() misses more than this config for the function symbols
         if sym.getSymbolType() == SymbolType.FUNCTION and sym.getSource() == SourceType.ANALYSIS and sym.isGlobal():
             name = sym.getName()  # starts to resolve names based on Ghidra's FidDB
@@ -164,7 +185,7 @@ def map_fake_import_addrs() -> dict[int, list[int]]:
     """
     fake_dict: dict[int, list[int]] = {}
 
-    for f in currentProgram().getFunctionManager().getExternalFunctions():  # type: ignore [name-defined] # noqa: F821
+    for f in get_current_program().getFunctionManager().getExternalFunctions():
         for r in f.getSymbol().getReferences():
             if r.getReferenceType().isData():
                 fake_dict.setdefault(f.getEntryPoint().getOffset(), []).append(r.getFromAddress().getOffset())
@@ -173,7 +194,7 @@ def map_fake_import_addrs() -> dict[int, list[int]]:
 
 
 def check_addr_for_api(
-    addr: ghidra.program.model.address.Address,
+    addr: "ghidra.program.model.address.Address",
     fakes: dict[int, list[int]],
     imports: dict[int, list[str]],
     externs: dict[int, list[str]],
@@ -195,18 +216,18 @@ def check_addr_for_api(
     return False
 
 
-def is_call_or_jmp(insn: ghidra.program.database.code.InstructionDB) -> bool:
+def is_call_or_jmp(insn: "ghidra.program.database.code.InstructionDB") -> bool:
     return any(mnem in insn.getMnemonicString() for mnem in ["CALL", "J"])  # JMP, JNE, JNZ, etc
 
 
-def is_sp_modified(insn: ghidra.program.database.code.InstructionDB) -> bool:
+def is_sp_modified(insn: "ghidra.program.database.code.InstructionDB") -> bool:
     for i in range(insn.getNumOperands()):
         if insn.getOperandType(i) == OperandType.REGISTER:
             return "SP" in insn.getRegister(i).getName() and insn.getOperandRefType(i).isWrite()
     return False
 
 
-def is_stack_referenced(insn: ghidra.program.database.code.InstructionDB) -> bool:
+def is_stack_referenced(insn: "ghidra.program.database.code.InstructionDB") -> bool:
     """generic catch-all for stack references"""
     for i in range(insn.getNumOperands()):
         if insn.getOperandType(i) == OperandType.REGISTER:
@@ -218,7 +239,7 @@ def is_stack_referenced(insn: ghidra.program.database.code.InstructionDB) -> boo
     return any(ref.isStackReference() for ref in insn.getReferencesFrom())
 
 
-def is_zxor(insn: ghidra.program.database.code.InstructionDB) -> bool:
+def is_zxor(insn: "ghidra.program.database.code.InstructionDB") -> bool:
     # assume XOR insn
     # XOR's against the same operand zero out
     ops = []
@@ -234,29 +255,29 @@ def is_zxor(insn: ghidra.program.database.code.InstructionDB) -> bool:
     return all(n == operands[0] for n in operands)
 
 
-def handle_thunk(addr: ghidra.program.model.address.Address):
+def handle_thunk(addr: "ghidra.program.model.address.Address"):
     """Follow thunk chains down to a reasonable depth"""
     ref = addr
     for _ in range(THUNK_CHAIN_DEPTH_DELTA):
-        thunk_jmp = getInstructionAt(ref)  # type: ignore [name-defined] # noqa: F821
+        thunk_jmp = get_flat_api().getInstructionAt(ref)
         if thunk_jmp and is_call_or_jmp(thunk_jmp):
             if OperandType.isAddress(thunk_jmp.getOperandType(0)):
                 ref = thunk_jmp.getAddress(0)
         else:
-            thunk_dat = getDataContaining(ref)  # type: ignore [name-defined] # noqa: F821
+            thunk_dat = get_flat_api().getDataContaining(ref)
             if thunk_dat and thunk_dat.isDefined() and thunk_dat.isPointer():
                 ref = thunk_dat.getValue()
                 break  # end of thunk chain reached
     return ref
 
 
-def dereference_ptr(insn: ghidra.program.database.code.InstructionDB):
+def dereference_ptr(insn: "ghidra.program.database.code.InstructionDB"):
     addr_code = OperandType.ADDRESS | OperandType.CODE
     to_deref = insn.getAddress(0)
-    dat = getDataContaining(to_deref)  # type: ignore [name-defined] # noqa: F821
+    dat = get_flat_api().getDataContaining(to_deref)
 
     if insn.getOperandType(0) == addr_code:
-        thfunc = getFunctionContaining(to_deref)  # type: ignore [name-defined] # noqa: F821
+        thfunc = get_flat_api().getFunctionContaining(to_deref)
         if thfunc and thfunc.isThunk():
             return handle_thunk(to_deref)
         else:
@@ -287,7 +308,7 @@ def find_data_references_from_insn(insn, max_depth: int = 10):
         to_addr = reference.getToAddress()
 
         for _ in range(max_depth - 1):
-            data = getDataAt(to_addr)  # type: ignore [name-defined] # noqa: F821
+            data = get_flat_api().getDataAt(to_addr)
             if data and data.isPointer():
                 ptr_value = data.getValue()
 

capa/features/extractors/ghidra/insn.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Any, Iterator
 
 import ghidra
@@ -227,7 +234,7 @@ def extract_insn_bytes_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
         push    offset iid_004118d4_IShellLinkA ; riid
     """
     for addr in capa.features.extractors.ghidra.helpers.find_data_references_from_insn(ih.inner):
-        data = getDataAt(addr)  # type: ignore [name-defined] # noqa: F821
+        data = capa.features.extractors.ghidra.helpers.get_flat_api().getDataAt(addr)
         if data and not data.hasStringValue():
             extracted_bytes = capa.features.extractors.ghidra.helpers.get_bytes(addr, MAX_BYTES_FEATURE_SIZE)
             if extracted_bytes and not capa.features.extractors.helpers.all_zeros(extracted_bytes):
@@ -242,9 +249,9 @@ def extract_insn_string_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandl
         push offset aAcr     ; "ACR  > "
     """
     for addr in capa.features.extractors.ghidra.helpers.find_data_references_from_insn(ih.inner):
-        data = getDataAt(addr)  # type: ignore [name-defined] # noqa: F821
+        data = capa.features.extractors.ghidra.helpers.get_flat_api().getDataAt(addr)
         if data and data.hasStringValue():
-            yield String(data.getValue()), ih.address
+            yield String(str(data.getValue())), ih.address
 
 
 def extract_insn_mnemonic_features(
@@ -354,8 +361,8 @@ def extract_insn_cross_section_cflow(
         if capa.features.extractors.ghidra.helpers.check_addr_for_api(ref, fakes, imports, externs):
             return
 
-    this_mem_block = getMemoryBlock(insn.getAddress())  # type: ignore [name-defined] # noqa: F821
-    ref_block = getMemoryBlock(ref)  # type: ignore [name-defined] # noqa: F821
+    this_mem_block = capa.features.extractors.ghidra.helpers.get_flat_api().getMemoryBlock(insn.getAddress())
+    ref_block = capa.features.extractors.ghidra.helpers.get_flat_api().getMemoryBlock(ref)
     if ref_block != this_mem_block:
         yield Characteristic("cross section flow"), ih.address
 
@@ -412,30 +419,29 @@ def extract_function_indirect_call_characteristic_features(
 def check_nzxor_security_cookie_delta(
     fh: ghidra.program.database.function.FunctionDB, insn: ghidra.program.database.code.InstructionDB
 ):
-    """Get the function containing the insn
-    Get the last block of the function that contains the insn
-
-    Check the bb containing the insn
-    Check the last bb of the function containing the insn
+    """
+    Get the first and last blocks of the function
+    Check if insn within first addr of first bb + delta
+    Check if insn within last addr of last bb - delta
     """
 
-    model = SimpleBlockModel(currentProgram())  # type: ignore [name-defined] # noqa: F821
+    model = SimpleBlockModel(capa.features.extractors.ghidra.helpers.get_current_program())
     insn_addr = insn.getAddress()
     func_asv = fh.getBody()
-    first_addr = func_asv.getMinAddress()
-    last_addr = func_asv.getMaxAddress()
 
-    if model.getFirstCodeBlockContaining(
-        first_addr, monitor()  # type: ignore [name-defined] # noqa: F821
-    ) == model.getFirstCodeBlockContaining(
-        last_addr, monitor()  # type: ignore [name-defined] # noqa: F821
-    ):
-        if insn_addr < first_addr.add(SECURITY_COOKIE_BYTES_DELTA):
+    first_addr = func_asv.getMinAddress()
+    if insn_addr < first_addr.add(SECURITY_COOKIE_BYTES_DELTA):
+        first_bb = model.getFirstCodeBlockContaining(first_addr, capa.features.extractors.ghidra.helpers.get_monitor())
+        if first_bb.contains(insn_addr):
             return True
-        else:
-            return insn_addr > last_addr.add(SECURITY_COOKIE_BYTES_DELTA * -1)
-    else:
-        return False
+
+    last_addr = func_asv.getMaxAddress()
+    if insn_addr > last_addr.add(SECURITY_COOKIE_BYTES_DELTA * -1):
+        last_bb = model.getFirstCodeBlockContaining(last_addr, capa.features.extractors.ghidra.helpers.get_monitor())
+        if last_bb.contains(insn_addr):
+            return True
+
+    return False
 
 
 def extract_insn_nzxor_characteristic_features(

capa/features/extractors/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import struct
 import builtins

capa/features/extractors/ida/basicblock.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import string
 import struct

capa/features/extractors/ida/extractor.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 import idaapi

capa/features/extractors/ida/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import struct
 from typing import Iterator

capa/features/extractors/ida/function.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 import idaapi

capa/features/extractors/ida/global_.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 import contextlib
 from typing import Iterator

capa/features/extractors/ida/helpers.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import functools
 from typing import Any, Iterator, Optional
 
@@ -41,7 +48,15 @@ if hasattr(ida_bytes, "parse_binpat_str"):
             return
 
         while True:
-            ea, _ = ida_bytes.bin_search(start, end, patterns, ida_bytes.BIN_SEARCH_FORWARD)
+            ea = ida_bytes.bin_search(start, end, patterns, ida_bytes.BIN_SEARCH_FORWARD)
+            if isinstance(ea, int):
+                # "ea_t" in IDA 8.4, 8.3
+                pass
+            elif isinstance(ea, tuple):
+                # "drc_t" in IDA 9
+                ea = ea[0]
+            else:
+                raise NotImplementedError(f"bin_search returned unhandled type: {type(ea)}")
             if ea == idaapi.BADADDR:
                 break
             start = ea + 1

capa/features/extractors/ida/idalib.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import os
 import sys
 import json

capa/features/extractors/ida/insn.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import re
 from typing import Any, Iterator, Optional
 

capa/features/extractors/loops.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import networkx
 from networkx.algorithms.components import strongly_connected_components

capa/features/extractors/null.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Union, TypeAlias
 from dataclasses import dataclass
 

capa/features/extractors/pefile.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from pathlib import Path
@@ -108,10 +115,7 @@ def extract_file_function_names(**kwargs):
     """
     extract the names of statically-linked library functions.
     """
-    if False:
-        # using a `yield` here to force this to be a generator, not function.
-        yield NotImplementedError("pefile doesn't have library matching")
-    return
+    yield from []
 
 
 def extract_file_os(**kwargs):

capa/features/extractors/strings.py

@@ -1,51 +1,98 @@
 # strings code from FLOSS, https://github.com/mandiant/flare-floss
 #
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 import re
+import string
 import contextlib
-from collections import namedtuple
+from dataclasses import dataclass
+from collections.abc import Iterator
 
 ASCII_BYTE = r" !\"#\$%&\'\(\)\*\+,-\./0123456789:;<=>\?@ABCDEFGHIJKLMNOPQRSTUVWXYZ\[\]\^_`abcdefghijklmnopqrstuvwxyz\{\|\}\\\~\t".encode(
     "ascii"
 )
 ASCII_RE_4 = re.compile(b"([%s]{%d,})" % (ASCII_BYTE, 4))
 UNICODE_RE_4 = re.compile(b"((?:[%s]\x00){%d,})" % (ASCII_BYTE, 4))
-REPEATS = [b"A", b"\x00", b"\xfe", b"\xff"]
+REPEATS = {ord("A"), 0x00, 0xFE, 0xFF}
 SLICE_SIZE = 4096
-
-String = namedtuple("String", ["s", "offset"])
+PRINTABLE_CHAR_SET = set(string.printable)
 
 
-def buf_filled_with(buf, character):
-    dupe_chunk = character * SLICE_SIZE
+@dataclass
+class String:
+    s: str
+    offset: int
+
+
+def buf_filled_with(buf: bytes, character: int) -> bool:
+    """Check if the given buffer is filled with the given character, repeatedly.
+
+    Args:
+        buf: The bytes buffer to check
+        character: The byte value (0-255) to check for
+
+    Returns:
+        True if all bytes in the buffer match the character, False otherwise.
+        The empty buffer contains no bytes, therefore always returns False.
+    """
+    if not buf:
+        return False
+
+    if not (0 <= character <= 255):
+        raise ValueError(f"Character value {character} outside valid byte range (0-255)")
+
+    if len(buf) < SLICE_SIZE:
+        return all(b == character for b in buf)
+
+    # single big allocation, re-used each loop
+    dupe_chunk = bytes(character) * SLICE_SIZE
+
     for offset in range(0, len(buf), SLICE_SIZE):
-        new_chunk = buf[offset : offset + SLICE_SIZE]
-        if dupe_chunk[: len(new_chunk)] != new_chunk:
-            return False
+        # bytes objects are immutable, so the slices share the underlying array,
+        # and therefore this is cheap.
+        current_chunk = buf[offset : offset + SLICE_SIZE]
+
+        if len(current_chunk) == SLICE_SIZE:
+            # chunk-aligned comparison
+
+            if dupe_chunk != current_chunk:
+                return False
+
+        else:
+            # last loop, final chunk size is not aligned
+            if not all(b == character for b in current_chunk):
+                return False
+
     return True
 
 
-def extract_ascii_strings(buf, n=4):
+def extract_ascii_strings(buf: bytes, n: int = 4) -> Iterator[String]:
     """
     Extract ASCII strings from the given binary data.
 
-    :param buf: A bytestring.
-    :type buf: str
-    :param n: The minimum length of strings to extract.
-    :type n: int
-    :rtype: Sequence[String]
+    Params:
+      buf: the bytes from which to extract strings
+      n: minimum string length
     """
 
     if not buf:
         return
 
+    if n < 1:
+        raise ValueError("minimum string length must be positive")
+
     if (buf[0] in REPEATS) and buf_filled_with(buf, buf[0]):
         return
 
@@ -59,20 +106,21 @@ def extract_ascii_strings(buf, n=4):
         yield String(match.group().decode("ascii"), match.start())
 
 
-def extract_unicode_strings(buf, n=4):
+def extract_unicode_strings(buf: bytes, n: int = 4) -> Iterator[String]:
     """
     Extract naive UTF-16 strings from the given binary data.
 
-    :param buf: A bytestring.
-    :type buf: str
-    :param n: The minimum length of strings to extract.
-    :type n: int
-    :rtype: Sequence[String]
+    Params:
+      buf: the bytes from which to extract strings
+      n: minimum string length
     """
 
     if not buf:
         return
 
+    if n < 1:
+        raise ValueError("minimum string length must be positive")
+
     if (buf[0] in REPEATS) and buf_filled_with(buf, buf[0]):
         return
 
@@ -84,3 +132,7 @@ def extract_unicode_strings(buf, n=4):
     for match in r.finditer(buf):
         with contextlib.suppress(UnicodeDecodeError):
             yield String(match.group().decode("utf-16"), match.start())
+
+
+def is_printable_str(s: str) -> bool:
+    return set(s).issubset(PRINTABLE_CHAR_SET)

capa/features/extractors/viv/basicblock.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import string
 import struct

capa/features/extractors/viv/extractor.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Any, Iterator
 from pathlib import Path

capa/features/extractors/viv/file.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 import PE.carve as pe_carve  # vivisect PE

capa/features/extractors/viv/function.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from typing import Iterator
 
 import envi

capa/features/extractors/viv/global_.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
-#  you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
-# Unless required by applicable law or agreed to in writing, software distributed under the License
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and limitations under the License.
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 from typing import Iterator