wip

.github/CODE_OF_CONDUCT.md

@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://contributor-covenant.org/version/1/4][version]
+
+[homepage]: https://contributor-covenant.org
+[version]: https://contributor-covenant.org/version/1/4/

.github/CONTRIBUTING.md

@@ -25,7 +25,7 @@ The following is a set of guidelines for contributing to capa and its packages,
 
 ## Code of Conduct
 
-This project follows [Google's Open Source Community Guidelines](https://opensource.google/conduct).
+This project and everyone participating in it is governed by the [Capa Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the maintainers.
 
 ## What should I know before I get started?
 
@@ -168,17 +168,15 @@ While the prerequisites above must be satisfied prior to having your pull reques
 
 ### Contributor License Agreement
 
-Contributions to this project must be accompanied by a
-[Contributor License Agreement](https://cla.developers.google.com/about) (CLA).
-You (or your employer) retain the copyright to your contribution; this simply
-gives us permission to use and redistribute your contributions as part of the
-project.
+Contributions to this project must be accompanied by a Contributor License
+Agreement. You (or your employer) retain the copyright to your contribution,
+this simply gives us permission to use and redistribute your contributions as
+part of the project. Head over to <https://cla.developers.google.com/> to see
+your current agreements on file or to sign a new one.
 
-If you or your current employer have already signed the Google CLA (even if it
-was for a different project), you probably don't need to do it again.
-
-Visit <https://cla.developers.google.com/> to see your current agreements or to
-sign a new one.
+You generally only need to submit a CLA once, so if you've already submitted one
+(even if it was for a different project), you probably don't need to do it
+again.
 
 ## Styleguides
 

.github/ISSUE_TEMPLATE/bug_report.md

@@ -10,8 +10,8 @@ We use submodules to separate code, rules and test data. If your issue is relate
 # Have you checked that your issue isn't already filed?
 Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
-# Have you read Google's Code of Conduct?
-By filing an issue, you are expected to comply with it, including treating everyone with respect: https://opensource.google/conduct
+# Have you read capa's Code of Conduct?
+By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
 
 # Have you read capa's CONTRIBUTING guide?
 It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs

.github/ISSUE_TEMPLATE/feature_request.md

@@ -10,8 +10,8 @@ We use submodules to separate code, rules and test data. If your issue is relate
 # Have you checked that your issue isn't already filed?
 Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
-# Have you read Google's Code of Conduct?
-By filing an issue, you are expected to comply with it, including treating everyone with respect: https://opensource.google/conduct
+# Have you read capa's Code of Conduct?
+By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
 
 # Have you read capa's CONTRIBUTING guide?
 It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements

.github/flake8.ini

@@ -40,4 +40,4 @@ per-file-ignores =
 
 copyright-check = True
 copyright-min-file-size = 1 
-copyright-regexp = Copyright \d{4} Google LLC
+copyright-regexp = Copyright \(C\) \d{4} Mandiant, Inc. All Rights Reserved.

.github/pyinstaller/hooks/hook-vivisect.py

@@ -1,16 +1,4 @@
-# Copyright 2020 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 
 from PyInstaller.utils.hooks import copy_metadata
 

.github/pyinstaller/pyinstaller.spec

@@ -1,18 +1,5 @@
 # -*- mode: python -*-
-# Copyright 2020 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 import sys
 
 import capa.rules.cache

.pre-commit-config.yaml

@@ -108,6 +108,7 @@ repos:
         -   "--check-untyped-defs"
         -   "--ignore-missing-imports"
         -   "--config-file=.github/mypy/mypy.ini"
+        -   "--enable-incomplete-feature=NewGenericSyntax"
         -   "capa/"
         -   "scripts/"
         -   "tests/"

CHANGELOG.md

@@ -6,101 +6,15 @@
 
 ### Breaking Changes
 
-### New Rules (15)
+### New Rules (0)
 
-- communication/socket/connect-socket moritz.raabe@mandiant.com joakim@intezer.com mrhafizfarhad@gmail.com
-- communication/socket/udp/connect-udp-socket mrhafizfarhad@gmail.com
-- nursery/enter-debug-mode-in-dotnet @v1bh475u
-- nursery/decrypt-data-using-tripledes-in-dotnet 0xRavenspar
-- nursery/encrypt-data-using-tripledes-in-dotnet 0xRavenspar
-- nursery/disable-system-features-via-registry-on-windows mehunhoff@google.com
-- data-manipulation/encryption/chaskey/encrypt-data-using-chaskey still@teamt5.org
-- data-manipulation/encryption/speck/encrypt-data-using-speck still@teamt5.org
-- load-code/dotnet/load-assembly-via-iassembly still@teamt5.org
-- malware-family/donut-loader/load-shellcode-via-donut still@teamt5.org
-- nursery/disable-device-guard-features-via-registry-on-windows mehunhoff@google.com
-- nursery/disable-firewall-features-via-registry-on-windows mehunhoff@google.com
-- nursery/disable-system-restore-features-via-registry-on-windows mehunhoff@google.com
-- nursery/disable-windows-defender-features-via-registry-on-windows mehunhoff@google.com
 -
 
 ### Bug Fixes
-- cape: make some fields optional @williballenthin #2631 #2632
-- lint: add WARN for regex features that contain unescaped dot #2635
-- lint: add ERROR for incomplete registry control set regex #2643
 
-### capa Explorer Web
-
-### capa Explorer IDA Pro plugin
-
-### Development
-
-### Raw diffs
-- [capa v9.1.0...master](https://github.com/mandiant/capa/compare/v9.1.0...master)
-- [capa-rules v9.1.0...master](https://github.com/mandiant/capa-rules/compare/v9.1.0...master)
-
-## v9.1.0
-
-This release improves a few aspects of dynamic analysis, relaxing our validation on fields across many CAPE versions, for example.
-It also includes an updated rule pack in which many dynamic rules make better use of the "span of calls" scope.
-
-
-### New Rules (3)
-
-- host-interaction/registry/change-registry-key-timestamp wballenthin@google.com
-- host-interaction/mutex/check-mutex-and-terminate-process-on-windows @_re_fox moritz.raabe@mandiant.com mehunhoff@google.com
-- anti-analysis/anti-forensic/clear-logs/clear-windows-event-logs-remotely 99.elad.levi@gmail.com
-
-### Bug Fixes
-- only parse CAPE fields required for analysis @mike-hunhoff #2607
-- main: render result document without needing associated rules @williballenthin #2610
-- vmray: only verify process OS and monitor IDs match @mike-hunhoff #2613
-- render: don't assume prior matches exist within a thread @mike-hunhoff #2612
-
-### Raw diffs
-- [capa v9.0.0...v9.1.0](https://github.com/mandiant/capa/compare/v9.0.0...v9.1.0)
-- [capa-rules v9.0.0...v9.1.0](https://github.com/mandiant/capa-rules/compare/v9.0.0...v9.1.0)
-
-## v9.0.0
-
-This release introduces a new scope for dynamic analysis, "span of calls",
- that matches features against a across a sliding window of API calls within a thread.
-Its useful for identifying behaviors that span multiple API calls,
- such as `OpenFile`/`ReadFile`/`CloseFile`, without having to analyze an entire thread, which may be very long.
-
-The release also contains a number of bug fixes and enhancements by new contributors: @v1bh475u and @dhruvak001. Welcome and thank you!
-
-### New Features
-
-- add warning for dynamic .NET samples #1864 @v1bh475u
-- add lint for detecting duplicate features in capa-rules #2250 @v1bh475u
-- add span-of-calls scope to match features against a across a sliding window of API calls within a thread @williballenthin #2532
-- add lint to catch rules that depend on other rules with impossible scope @williballenthin #2124
-
-### Breaking Changes
-
-- remove `is_static_limitation` method from `capa.rules.Rule`
-- add span-of-calls scope to rule format
-- capabilities functions return dataclasses instead of tuples
-
-### New Rules (3)
-
-- data-manipulation/encryption/rsa/encrypt-data-using-rsa-via-embedded-library @Ana06
-- data-manipulation/encryption/use-bigint-function @Ana06
-- internal/limitation/dynamic/internal-dotnet-file-limitation @v1bh475u
-
-
-### Bug Fixes
-
-- dynamic: only check file limitations for static file formats @mr-tz
 - vmray: load more analysis archives @mr-tz
+- dynamic: only check file limitations for static file formats @mr-tz
 - vmray: skip non-printable strings @mike-hunhoff
-- vmray: loosen file checks to enable processing more file types @mike-hunhoff #2571
-- strings: add type hints and fix uncovered bugs @williballenthin #2555
-- elffile: handle symbols without a name @williballenthin #2553
-- project: remove pytest-cov that wasn't used @williballenthin @2491
-- replace binascii methods with native Python methods @v1bh475u #2582
-- rules: scopes can now have subscope blocks with the same scope @williballenthin #2584
 
 ### capa Explorer Web
 
@@ -108,12 +22,9 @@ The release also contains a number of bug fixes and enhancements by new contribu
 
 ### Development
 
-- license & copyright: Correct LICENSE file and improve copyright and license information headers in the source code files @Ana06
-- documentation: Improve CLA and Code of Conduct information in CONTRIBUTING @Ana06
-
 ### Raw diffs
-- [capa v8.0.1...v9.0.0](https://github.com/mandiant/capa/compare/v8.0.1...v9.0.0)
-- [capa-rules v8.0.1...v9.0.0](https://github.com/mandiant/capa-rules/compare/v8.0.1...v9.0.0)
+- [capa v8.0.1...master](https://github.com/mandiant/capa/compare/v8.0.1...master)
+- [capa-rules v8.0.1...master](https://github.com/mandiant/capa-rules/compare/v8.0.1...master)
 
 ## v8.0.1
 

LICENSE.txt

@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright (C) 2020 Mandiant, Inc.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

capa/analysis/flirt.py

@@ -0,0 +1,38 @@
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+from pydantic import BaseModel
+
+import capa.features.extractors.ida.idalib as idalib
+
+if not idalib.has_idalib():
+    raise RuntimeError("cannot find IDA idalib module.")
+
+if not idalib.load_idalib():
+    raise RuntimeError("failed to load IDA idalib module.")
+
+import idaapi
+import idautils
+
+
+class FunctionId(BaseModel):
+    va: int
+    is_library: bool
+    name: str
+
+
+def get_flirt_matches(lib_only=True):
+    for fva in idautils.Functions():
+        f = idaapi.get_func(fva)
+        is_lib = bool(f.flags & idaapi.FUNC_LIB)
+        fname = idaapi.get_func_name(fva)
+
+        if lib_only and not is_lib:
+            continue
+
+        yield FunctionId(va=fva, is_library=is_lib, name=fname)

capa/analysis/libraries.py

@@ -0,0 +1,242 @@
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import io
+import sys
+import logging
+import argparse
+import tempfile
+import contextlib
+from enum import Enum
+from typing import List, Optional
+from pathlib import Path
+
+import rich
+from pydantic import BaseModel
+from rich.text import Text
+from rich.console import Console
+
+import capa.main
+import capa.helpers
+import capa.analysis.flirt
+import capa.analysis.strings
+import capa.features.extractors.ida.idalib as idalib
+
+if not idalib.has_idalib():
+    raise RuntimeError("cannot find IDA idalib module.")
+
+if not idalib.load_idalib():
+    raise RuntimeError("failed to load IDA idalib module.")
+
+import idaapi
+import idapro
+import ida_auto
+import idautils
+
+logger = logging.getLogger(__name__)
+
+
+class Classification(str, Enum):
+    USER = "user"
+    LIBRARY = "library"
+    UNKNOWN = "unknown"
+
+
+class Method(str, Enum):
+    FLIRT = "flirt"
+    STRINGS = "strings"
+    THUNK = "thunk"
+    ENTRYPOINT = "entrypoint"
+
+
+class FunctionClassification(BaseModel):
+    va: int
+    classification: Classification
+    # name per the disassembler/analysis tool
+    # may be combined with the recovered/suspected name TODO below
+    name: str
+
+    # if is library, this must be provided
+    method: Optional[Method]
+
+    # TODO if is library, recovered/suspected name?
+
+    # if is library, these can optionally be provided.
+    library_name: Optional[str] = None
+    library_version: Optional[str] = None
+
+
+class FunctionIdResults(BaseModel):
+    function_classifications: List[FunctionClassification]
+
+
+@contextlib.contextmanager
+def ida_session(input_path: Path, use_temp_dir=True):
+    if use_temp_dir:
+        t = Path(tempfile.mkdtemp(prefix="ida-")) / input_path.name
+    else:
+        t = input_path
+
+    logger.debug("using %s", str(t))
+    # stderr=True is used here to redirect the spinner banner to stderr,
+    # so that users can redirect capa's output.
+    console = Console(stderr=True, quiet=False)
+
+    try:
+        if use_temp_dir:
+            t.write_bytes(input_path.read_bytes())
+
+        # idalib writes to stdout (ugh), so we have to capture that
+        # so as not to screw up structured output.
+        with capa.helpers.stdout_redirector(io.BytesIO()):
+            idapro.enable_console_messages(False)
+            with capa.main.timing("analyze program"):
+                with console.status("analyzing program...", spinner="dots"):
+                    if idapro.open_database(str(t.absolute()), run_auto_analysis=True):
+                        raise RuntimeError("failed to analyze input file")
+
+            logger.debug("idalib: waiting for analysis...")
+            ida_auto.auto_wait()
+            logger.debug("idalib: opened database.")
+
+        yield
+    finally:
+        idapro.close_database()
+        if use_temp_dir:
+            t.unlink()
+
+
+def is_thunk_function(fva):
+    f = idaapi.get_func(fva)
+    return bool(f.flags & idaapi.FUNC_THUNK)
+
+
+def main(argv=None):
+    if argv is None:
+        argv = sys.argv[1:]
+
+    parser = argparse.ArgumentParser(description="Identify library functions using various strategies.")
+    capa.main.install_common_args(parser, wanted={"input_file"})
+    parser.add_argument("--store-idb", action="store_true", default=False, help="store IDA database file")
+    parser.add_argument("--min-string-length", type=int, default=8, help="minimum string length")
+    parser.add_argument("-j", "--json", action="store_true", help="emit JSON instead of text")
+    args = parser.parse_args(args=argv)
+
+    try:
+        capa.main.handle_common_args(args)
+    except capa.main.ShouldExitError as e:
+        return e.status_code
+
+    dbs = capa.analysis.strings.get_default_databases()
+    capa.analysis.strings.prune_databases(dbs, n=args.min_string_length)
+
+    function_classifications: List[FunctionClassification] = []
+    with ida_session(args.input_file, use_temp_dir=not args.store_idb):
+        with capa.main.timing("FLIRT-based library identification"):
+            # TODO: add more signature (files)
+            # TOOD: apply more signatures
+            for flirt_match in capa.analysis.flirt.get_flirt_matches():
+                function_classifications.append(
+                    FunctionClassification(
+                        va=flirt_match.va,
+                        name=flirt_match.name,
+                        classification=Classification.LIBRARY,
+                        method=Method.FLIRT,
+                        # note: we cannot currently include which signature matched per function via the IDA API
+                    )
+                )
+
+        # thunks
+        for fva in idautils.Functions():
+            if is_thunk_function(fva):
+                function_classifications.append(
+                    FunctionClassification(
+                        va=fva,
+                        name=idaapi.get_func_name(fva),
+                        classification=Classification.LIBRARY,
+                        method=Method.THUNK,
+                    )
+                )
+
+        with capa.main.timing("string-based library identification"):
+            for string_match in capa.analysis.strings.get_string_matches(dbs):
+                function_classifications.append(
+                    FunctionClassification(
+                        va=string_match.va,
+                        name=idaapi.get_func_name(string_match.va),
+                        classification=Classification.LIBRARY,
+                        method=Method.STRINGS,
+                        library_name=string_match.metadata.library_name,
+                        library_version=string_match.metadata.library_version,
+                    )
+                )
+
+        for va in idautils.Functions():
+            name = idaapi.get_func_name(va)
+            if name not in {
+                "WinMain",
+            }:
+                continue
+
+            function_classifications.append(
+                FunctionClassification(
+                    va=va,
+                    name=name,
+                    classification=Classification.USER,
+                    method=Method.ENTRYPOINT,
+                )
+            )
+
+        doc = FunctionIdResults(function_classifications=[])
+        classifications_by_va = capa.analysis.strings.create_index(function_classifications, "va")
+        for va in idautils.Functions():
+            if classifications := classifications_by_va.get(va):
+                doc.function_classifications.extend(classifications)
+            else:
+                doc.function_classifications.append(
+                    FunctionClassification(
+                        va=va,
+                        name=idaapi.get_func_name(va),
+                        classification=Classification.UNKNOWN,
+                        method=None,
+                    )
+                )
+
+        if args.json:
+            print(doc.model_dump_json())  # noqa: T201 print found
+
+        else:
+            table = rich.table.Table()
+            table.add_column("FVA")
+            table.add_column("CLASSIFICATION")
+            table.add_column("METHOD")
+            table.add_column("FNAME")
+            table.add_column("EXTRA INFO")
+
+            classifications_by_va = capa.analysis.strings.create_index(doc.function_classifications, "va", sorted_=True)
+            for va, classifications in classifications_by_va.items():
+                name = ", ".join({c.name for c in classifications})
+                if "sub_" in name:
+                    name = Text(name, style="grey53")
+
+                classification = {c.classification for c in classifications}
+                method = {c.method for c in classifications if c.method}
+                extra = {f"{c.library_name}@{c.library_version}" for c in classifications if c.library_name}
+
+                table.add_row(
+                    hex(va),
+                    ", ".join(classification) if classification != {"unknown"} else Text("unknown", style="grey53"),
+                    ", ".join(method),
+                    name,
+                    ", ".join(extra),
+                )
+
+            rich.print(table)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

capa/analysis/requirements.txt

@@ -0,0 +1,2 @@
+# temporary extra file to track dependencies of the analysis directory
+nltk==3.9.1

capa/analysis/strings/__init__.py

@@ -0,0 +1,269 @@
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+
+"""
+further requirements:
+  - nltk
+"""
+import gzip
+import logging
+import collections
+from typing import Any, Dict, Mapping
+from pathlib import Path
+from dataclasses import dataclass
+
+import msgspec
+
+import capa.features.extractors.strings
+
+logger = logging.getLogger(__name__)
+
+
+class LibraryString(msgspec.Struct):
+    string: str
+    library_name: str
+    library_version: str
+    file_path: str | None = None
+    function_name: str | None = None
+    line_number: int | None = None
+
+
+@dataclass
+class LibraryStringDatabase:
+    metadata_by_string: Dict[str, LibraryString]
+
+    def __len__(self) -> int:
+        return len(self.metadata_by_string)
+
+    @classmethod
+    def from_file(cls, path: Path) -> "LibraryStringDatabase":
+        metadata_by_string: Dict[str, LibraryString] = {}
+        decoder = msgspec.json.Decoder(type=LibraryString)
+        for line in gzip.decompress(path.read_bytes()).split(b"\n"):
+            if not line:
+                continue
+            s = decoder.decode(line)
+            metadata_by_string[s.string] = s
+
+        return cls(metadata_by_string=metadata_by_string)
+
+
+DEFAULT_FILENAMES = (
+    "brotli.jsonl.gz",
+    "bzip2.jsonl.gz",
+    "cryptopp.jsonl.gz",
+    "curl.jsonl.gz",
+    "detours.jsonl.gz",
+    "jemalloc.jsonl.gz",
+    "jsoncpp.jsonl.gz",
+    "kcp.jsonl.gz",
+    "liblzma.jsonl.gz",
+    "libsodium.jsonl.gz",
+    "libpcap.jsonl.gz",
+    "mbedtls.jsonl.gz",
+    "openssl.jsonl.gz",
+    "sqlite3.jsonl.gz",
+    "tomcrypt.jsonl.gz",
+    "wolfssl.jsonl.gz",
+    "zlib.jsonl.gz",
+)
+
+DEFAULT_PATHS = tuple(Path(__file__).parent / "data" / "oss" / filename for filename in DEFAULT_FILENAMES) + (
+    Path(__file__).parent / "data" / "crt" / "msvc_v143.jsonl.gz",
+)
+
+
+def get_default_databases() -> list[LibraryStringDatabase]:
+    return [LibraryStringDatabase.from_file(path) for path in DEFAULT_PATHS]
+
+
+@dataclass
+class WindowsApiStringDatabase:
+    dll_names: set[str]
+    api_names: set[str]
+
+    def __len__(self) -> int:
+        return len(self.dll_names) + len(self.api_names)
+
+    @classmethod
+    def from_dir(cls, path: Path) -> "WindowsApiStringDatabase":
+        dll_names: set[str] = set()
+        api_names: set[str] = set()
+
+        for line in gzip.decompress((path / "dlls.txt.gz").read_bytes()).decode("utf-8").splitlines():
+            if not line:
+                continue
+            dll_names.add(line)
+
+        for line in gzip.decompress((path / "apis.txt.gz").read_bytes()).decode("utf-8").splitlines():
+            if not line:
+                continue
+            api_names.add(line)
+
+        return cls(dll_names=dll_names, api_names=api_names)
+
+    @classmethod
+    def from_defaults(cls) -> "WindowsApiStringDatabase":
+        return cls.from_dir(Path(__file__).parent / "data" / "winapi")
+
+
+def extract_strings(buf, n=4):
+    yield from capa.features.extractors.strings.extract_ascii_strings(buf, n=n)
+    yield from capa.features.extractors.strings.extract_unicode_strings(buf, n=n)
+
+
+def prune_databases(dbs: list[LibraryStringDatabase], n=8):
+    """remove less trustyworthy database entries.
+
+    such as:
+      - those found in multiple databases
+      - those that are English words
+      - those that are too short
+      - Windows API and DLL names
+    """
+
+    # TODO: consider applying these filters directly to the persisted databases, not at load time.
+
+    winapi = WindowsApiStringDatabase.from_defaults()
+
+    try:
+        from nltk.corpus import words as nltk_words
+
+        nltk_words.words()
+    except (ImportError, LookupError):
+        # one-time download of dataset.
+        # this probably doesn't work well for embedded use.
+        import nltk
+
+        nltk.download("words")
+        from nltk.corpus import words as nltk_words
+    words = set(nltk_words.words())
+
+    counter: collections.Counter[str] = collections.Counter()
+    to_remove = set()
+    for db in dbs:
+        for string in db.metadata_by_string.keys():
+            counter[string] += 1
+
+            if string in words:
+                to_remove.add(string)
+                continue
+
+            if len(string) < n:
+                to_remove.add(string)
+                continue
+
+            if string in winapi.api_names:
+                to_remove.add(string)
+                continue
+
+            if string in winapi.dll_names:
+                to_remove.add(string)
+                continue
+
+    for string, count in counter.most_common():
+        if count <= 1:
+            break
+
+        # remove strings that are seen in more than one database
+        to_remove.add(string)
+
+    for db in dbs:
+        for string in to_remove:
+            if string in db.metadata_by_string:
+                del db.metadata_by_string[string]
+
+
+def get_function_strings():
+    import idaapi
+    import idautils
+
+    import capa.features.extractors.ida.helpers as ida_helpers
+
+    strings_by_function = collections.defaultdict(set)
+    for ea in idautils.Functions():
+        f = idaapi.get_func(ea)
+
+        # ignore library functions and thunk functions as identified by IDA
+        if f.flags & idaapi.FUNC_THUNK:
+            continue
+        if f.flags & idaapi.FUNC_LIB:
+            continue
+
+        for bb in ida_helpers.get_function_blocks(f):
+            for insn in ida_helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
+                ref = capa.features.extractors.ida.helpers.find_data_reference_from_insn(insn)
+                if ref == insn.ea:
+                    continue
+
+                string = capa.features.extractors.ida.helpers.find_string_at(ref)
+                if not string:
+                    continue
+
+                strings_by_function[ea].add(string)
+
+    return strings_by_function
+
+
+@dataclass
+class LibraryStringClassification:
+    va: int
+    string: str
+    library_name: str
+    metadata: LibraryString
+
+
+def create_index(s: list, k: str, sorted_: bool = False) -> Mapping[Any, list]:
+    """create an index of the elements in `s` using the key `k`, optionally sorted by `k`"""
+    if sorted_:
+        s = sorted(s, key=lambda x: getattr(x, k))
+
+    s_by_k = collections.defaultdict(list)
+    for v in s:
+        p = getattr(v, k)
+        s_by_k[p].append(v)
+    return s_by_k
+
+
+def get_string_matches(dbs: list[LibraryStringDatabase]) -> list[LibraryStringClassification]:
+    matches: list[LibraryStringClassification] = []
+
+    for function, strings in sorted(get_function_strings().items()):
+        for string in strings:
+            for db in dbs:
+                if metadata := db.metadata_by_string.get(string):
+                    matches.append(
+                        LibraryStringClassification(
+                            va=function,
+                            string=string,
+                            library_name=metadata.library_name,
+                            metadata=metadata,
+                        )
+                    )
+
+    # if there are less than N strings per library, ignore that library
+    matches_by_library = create_index(matches, "library_name")
+    for library_name, library_matches in matches_by_library.items():
+        if len(library_matches) > 5:
+            continue
+
+        logger.info("pruning library %s: only %d matched string", library_name, len(library_matches))
+        matches = [m for m in matches if m.library_name != library_name]
+
+    # if there are conflicts within a single function, don't label it
+    matches_by_function = create_index(matches, "va")
+    for va, function_matches in matches_by_function.items():
+        library_names = {m.library_name for m in function_matches}
+        if len(library_names) == 1:
+            continue
+
+        logger.info("conflicting matches: 0x%x: %s", va, sorted(library_names))
+        # this is potentially slow (O(n**2)) but hopefully fast enough in practice.
+        matches = [m for m in matches if m.va != va]
+
+    return matches

capa/analysis/strings/__main__.py

@@ -0,0 +1,130 @@
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import sys
+import logging
+import collections
+from pathlib import Path
+
+import rich
+from rich.text import Text
+
+import capa.analysis.strings
+import capa.features.extractors.strings
+import capa.features.extractors.ida.helpers as ida_helpers
+
+logger = logging.getLogger(__name__)
+
+
+def open_ida(input_path: Path):
+    import tempfile
+
+    import idapro
+
+    t = Path(tempfile.mkdtemp(prefix="ida-")) / input_path.name
+    t.write_bytes(input_path.read_bytes())
+    # resource leak: we should delete this upon exit
+
+    idapro.enable_console_messages(False)
+    idapro.open_database(str(t.absolute()), run_auto_analysis=True)
+
+    import ida_auto
+
+    ida_auto.auto_wait()
+
+
+def main():
+    logging.basicConfig(level=logging.DEBUG)
+
+    # use n=8 to ignore common words
+    N = 8
+
+    input_path = Path(sys.argv[1])
+
+    dbs = capa.analysis.strings.get_default_databases()
+    capa.analysis.strings.prune_databases(dbs, n=N)
+
+    strings_by_library = collections.defaultdict(set)
+    for string in capa.analysis.strings.extract_strings(input_path.read_bytes(), n=N):
+        for db in dbs:
+            if metadata := db.metadata_by_string.get(string.s):
+                strings_by_library[metadata.library_name].add(string.s)
+
+    console = rich.get_console()
+    console.print("found libraries:", style="bold")
+    for library, strings in sorted(strings_by_library.items(), key=lambda p: len(p[1]), reverse=True):
+        console.print(f"  - [b]{library}[/] ({len(strings)} strings)")
+
+        for string in sorted(strings)[:10]:
+            console.print(f"    - {string}", markup=False, style="grey37")
+
+        if len(strings) > 10:
+            console.print("    ...", style="grey37")
+
+    if not strings_by_library:
+        console.print("  (none)", style="grey37")
+        # since we're not going to find any strings
+        # return early and don't do IDA analysis
+        return
+
+    open_ida(input_path)
+
+    import idaapi
+    import idautils
+    import ida_funcs
+
+    strings_by_function = collections.defaultdict(set)
+    for ea in idautils.Functions():
+        f = idaapi.get_func(ea)
+
+        # ignore library functions and thunk functions as identified by IDA
+        if f.flags & idaapi.FUNC_THUNK:
+            continue
+        if f.flags & idaapi.FUNC_LIB:
+            continue
+
+        for bb in ida_helpers.get_function_blocks(f):
+            for insn in ida_helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
+                ref = capa.features.extractors.ida.helpers.find_data_reference_from_insn(insn)
+                if ref == insn.ea:
+                    continue
+
+                string = capa.features.extractors.ida.helpers.find_string_at(ref)
+                if not string:
+                    continue
+
+                for db in dbs:
+                    if metadata := db.metadata_by_string.get(string):
+                        strings_by_function[ea].add(string)
+
+    # ensure there are at least XXX functions renamed, or ignore those entries
+
+    console.print("functions:", style="bold")
+    for function, strings in sorted(strings_by_function.items()):
+        if strings:
+            name = ida_funcs.get_func_name(function)
+
+            console.print(f"  [b]{name}[/]@{function:08x}:")
+
+            for string in strings:
+                for db in dbs:
+                    if metadata := db.metadata_by_string.get(string):
+                        location = Text(
+                            f"{metadata.library_name}@{metadata.library_version}::{metadata.function_name}",
+                            style="grey37",
+                        )
+                        console.print("    - ", location, ": ", string.rstrip())
+
+    console.print()
+
+    console.print(
+        f"found {len(strings_by_function)} library functions across {len(list(idautils.Functions()))} functions"
+    )
+
+
+if __name__ == "__main__":
+    main()

capa/analysis/strings/data/oss/.gitignore

@@ -0,0 +1,3 @@
+*.csv
+*.jsonl
+*.jsonl.gz

capa/analysis/strings/data/oss/readme.md

@@ -0,0 +1,99 @@
+# Strings from Open Source libraries
+
+This directory contains databases of strings extracted from open soure software. 
+capa uses these databases to ignore functions that are likely library code.
+
+There is one file for each database. Each database is a gzip-compressed, JSONL (one JSON document per line) file.
+The JSON document looks like this:
+
+    string: "1.0.8, 13-Jul-2019"
+    library_name: "bzip2"
+    library_version: "1.0.8#3"
+    file_path: "CMakeFiles/bz2.dir/bzlib.c.obj"
+    function_name: "BZ2_bzlibVersion"
+    line_number: null
+
+The following databases were extracted via the vkpkg & jh technique:
+
+  - brotli 1.0.9#5
+  - bzip2 1.0.8#3
+  - cryptopp 8.7.0
+  - curl 7.86.0#1
+  - detours 4.0.1#7
+  - jemalloc 5.3.0#1
+  - jsoncpp 1.9.5
+  - kcp 1.7
+  - liblzma 5.2.5#6
+  - libsodium 1.0.18#8
+  - libpcap 1.10.1#3
+  - mbedtls 2.28.1
+  - openssl 3.0.7#1
+  - sqlite3 3.40.0#1
+  - tomcrypt 1.18.2#2
+  - wolfssl 5.5.0
+  - zlib 1.2.13
+
+This code was originally developed in FLOSS and imported into capa.
+
+## The vkpkg & jh technique
+
+Major steps:
+
+  1. build static libraries via vcpkg
+  2. extract features via jh
+  3. convert to JSONL format with `jh_to_qs.py`
+  4. compress with gzip
+
+### Build static libraries via vcpkg
+
+[vcpkg](https://vcpkg.io/en/) is a free C/C++ package manager for acquiring and managing libraries.
+We use it to easily build common open source libraries, like zlib.
+Use the triplet `x64-windows-static` to build static archives (.lib files that are AR archives containing COFF object files):
+
+```console
+PS > C:\vcpkg\vcpkg.exe install --triplet x64-windows-static zlib
+```
+
+### Extract features via jh
+
+[jh](https://github.com/williballenthin/lancelot/blob/master/bin/src/bin/jh.rs)
+is a lancelot-based utility that parses AR archives containing COFF object files,
+reconstructs their control flow, finds functions, and extracts features. 
+jh extracts numbers, API calls, and strings; we are only interested in the string features.
+
+For each feature, jh emits a CSV line with the fields 
+  - target triplet
+  - compiler 
+  - library
+  - version
+  - build profile
+  - path
+  - function
+  - feature type
+  - feature value
+
+For example:
+
+```csv
+x64-windows-static,msvc143,bzip2,1.0.8#3,release,CMakeFiles/bz2.dir/bzlib.c.obj,BZ2_bzBuffToBuffCompress,number,0x00000100
+```
+
+For example, to invoke jh:
+
+```console
+$ ~/lancelot/target/release/jh x64-windows-static msvc143 zlib 1.2.13 release /mnt/c/vcpkg/installed/x64-windows-static/lib/zlib.lib > ~/flare-floss/floss/qs/db/data/oss/zlib.csv
+```
+
+### Convert to OSS database format
+
+We use the script `jh_to_qs.py` to convert these CSV lines into JSONL file prepared for FLOSS:
+
+```console
+$ python3 jh_to_qs.py zlib.csv > zlib.jsonl
+```
+
+These files are then gzip'd:
+
+```console
+$  gzip -c zlib.jsonl > zlib.jsonl.gz
+```

capa/capabilities/common.py

@@ -1,43 +1,25 @@
 # -*- coding: utf-8 -*-
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 import itertools
 import collections
-from typing import Optional
-from dataclasses import dataclass
+from typing import Any
 
-from capa.rules import Rule, Scope, RuleSet
+from capa.rules import Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
 from capa.features.address import NO_ADDRESS
-from capa.render.result_document import LibraryFunction, StaticFeatureCounts, DynamicFeatureCounts
 from capa.features.extractors.base_extractor import FeatureExtractor, StaticFeatureExtractor, DynamicFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
-@dataclass
-class FileCapabilities:
-    features: FeatureSet
-    matches: MatchResults
-    feature_count: int
-
-
-def find_file_capabilities(
-    ruleset: RuleSet, extractor: FeatureExtractor, function_features: FeatureSet
-) -> FileCapabilities:
+def find_file_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, function_features: FeatureSet):
     file_features: FeatureSet = collections.defaultdict(set)
 
     for feature, va in itertools.chain(extractor.extract_file_features(), extractor.extract_global_features()):
@@ -54,18 +36,35 @@ def find_file_capabilities(
 
     file_features.update(function_features)
 
-    features, matches = ruleset.match(Scope.FILE, file_features, NO_ADDRESS)
-    return FileCapabilities(features, matches, len(file_features))
+    _, matches = ruleset.match(Scope.FILE, file_features, NO_ADDRESS)
+    return matches, len(file_features)
 
 
-@dataclass
-class Capabilities:
-    matches: MatchResults
-    feature_counts: StaticFeatureCounts | DynamicFeatureCounts
-    library_functions: Optional[tuple[LibraryFunction, ...]] = None
+def has_file_limitation(rules: RuleSet, capabilities: MatchResults, is_standalone=True) -> bool:
+    file_limitation_rules = list(filter(lambda r: r.is_file_limitation_rule(), rules.rules.values()))
+
+    for file_limitation_rule in file_limitation_rules:
+        if file_limitation_rule.name not in capabilities:
+            continue
+
+        logger.warning("-" * 80)
+        for line in file_limitation_rule.meta.get("description", "").split("\n"):
+            logger.warning(" %s", line)
+        logger.warning(" Identified via rule: %s", file_limitation_rule.name)
+        if is_standalone:
+            logger.warning(" ")
+            logger.warning(" Use -v or -vv if you really want to see the capabilities identified by capa.")
+        logger.warning("-" * 80)
+
+        # bail on first file limitation
+        return True
+
+    return False
 
 
-def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None, **kwargs) -> Capabilities:
+def find_capabilities(
+    ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None, **kwargs
+) -> tuple[MatchResults, Any]:
     from capa.capabilities.static import find_static_capabilities
     from capa.capabilities.dynamic import find_dynamic_capabilities
 
@@ -78,40 +77,3 @@ def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_pro
         return find_dynamic_capabilities(ruleset, extractor, disable_progress=disable_progress, **kwargs)
 
     raise ValueError(f"unexpected extractor type: {extractor.__class__.__name__}")
-
-
-def has_limitation(rules: list, capabilities: Capabilities | FileCapabilities, is_standalone: bool) -> bool:
-
-    for rule in rules:
-        if rule.name not in capabilities.matches:
-            continue
-        logger.warning("-" * 80)
-        for line in rule.meta.get("description", "").split("\n"):
-            logger.warning(" %s", line)
-        logger.warning(" Identified via rule: %s", rule.name)
-        if is_standalone:
-            logger.warning(" ")
-            logger.warning(" Use -v or -vv if you really want to see the capabilities identified by capa.")
-        logger.warning("-" * 80)
-
-        # bail on first file limitation
-        return True
-    return False
-
-
-def is_static_limitation_rule(r: Rule) -> bool:
-    return r.meta.get("namespace", "") == "internal/limitation/static"
-
-
-def has_static_limitation(rules: RuleSet, capabilities: Capabilities | FileCapabilities, is_standalone=True) -> bool:
-    file_limitation_rules = list(filter(lambda r: is_static_limitation_rule(r), rules.rules.values()))
-    return has_limitation(file_limitation_rules, capabilities, is_standalone)
-
-
-def is_dynamic_limitation_rule(r: Rule) -> bool:
-    return r.meta.get("namespace", "") == "internal/limitation/dynamic"
-
-
-def has_dynamic_limitation(rules: RuleSet, capabilities: Capabilities | FileCapabilities, is_standalone=True) -> bool:
-    dynamic_limitation_rules = list(filter(lambda r: is_dynamic_limitation_rule(r), rules.rules.values()))
-    return has_limitation(dynamic_limitation_rules, capabilities, is_standalone)

capa/capabilities/dynamic.py

@@ -1,55 +1,34 @@
 # -*- coding: utf-8 -*-
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 import itertools
 import collections
-from dataclasses import dataclass
+from typing import Any
 
 import capa.perf
-import capa.engine
-import capa.helpers
 import capa.features.freeze as frz
 import capa.render.result_document as rdoc
 from capa.rules import Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
-from capa.features.address import _NoAddress
-from capa.capabilities.common import Capabilities, find_file_capabilities
+from capa.capabilities.common import find_file_capabilities
 from capa.features.extractors.base_extractor import CallHandle, ThreadHandle, ProcessHandle, DynamicFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
-# The number of calls that make up a span of calls.
-#
-# The larger this is, the more calls are grouped together to match rule logic.
-# This means a longer chain can be recognized; however, its a bit more expensive.
-SPAN_SIZE = 20
-
-
-@dataclass
-class CallCapabilities:
-    features: FeatureSet
-    matches: MatchResults
-
-
 def find_call_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle
-) -> CallCapabilities:
+) -> tuple[FeatureSet, MatchResults]:
     """
     find matches for the given rules for the given call.
+
+    returns: tuple containing (features for call, match results for call)
     """
     # all features found for the call.
     features: FeatureSet = collections.defaultdict(set)
@@ -67,105 +46,16 @@ def find_call_capabilities(
         for addr, _ in res:
             capa.engine.index_rule_matches(features, rule, [addr])
 
-    return CallCapabilities(features, matches)
-
-
-@dataclass
-class ThreadCapabilities:
-    features: FeatureSet
-    thread_matches: MatchResults
-    span_matches: MatchResults
-    call_matches: MatchResults
-
-
-class SpanOfCallsMatcher:
-    def __init__(self, ruleset: RuleSet):
-        super().__init__()
-        self.ruleset = ruleset
-
-        # matches found at the span scope.
-        self.matches: MatchResults = collections.defaultdict(list)
-
-        # We match spans as the sliding window of calls with size SPAN_SIZE.
-        #
-        # For each call, we consider the window of SPAN_SIZE calls leading up to it,
-        #  merging all their features and doing a match.
-        #
-        # We track these features in two data structures:
-        #   1. a deque of those features found in the prior calls.
-        #      We'll append to it, and as it grows larger than SPAN_SIZE, the oldest items are removed.
-        #   2. a live set of features seen in the span.
-        #      As we pop from the deque, we remove features from the current set,
-        #      and as we push to the deque, we insert features to the current set.
-        # With this approach, our algorithm performance is independent of SPAN_SIZE.
-        # The naive algorithm, of merging all the trailing feature sets at each call, is dependent upon SPAN_SIZE
-        # (that is, runtime gets slower the larger SPAN_SIZE is).
-        self.current_feature_sets: collections.deque[FeatureSet] = collections.deque(maxlen=SPAN_SIZE)
-        self.current_features: FeatureSet = collections.defaultdict(set)
-
-        # the names of rules matched at the last span,
-        # so that we can deduplicate long strings of the same matches.
-        self.last_span_matches: set[str] = set()
-
-    def next(self, ch: CallHandle, call_features: FeatureSet):
-        # As we add items to the end of the deque, overflow and drop the oldest items (at the left end).
-        # While we could rely on `deque.append` with `maxlen` set (which we provide above),
-        # we want to use the dropped item first, to remove the old features, so we manually pop it here.
-        if len(self.current_feature_sets) == SPAN_SIZE:
-            overflowing_feature_set = self.current_feature_sets.popleft()
-
-            for feature, vas in overflowing_feature_set.items():
-                if len(vas) == 1 and isinstance(next(iter(vas)), _NoAddress):
-                    # `vas == { NO_ADDRESS }` without the garbage.
-                    #
-                    # ignore the common case of global features getting added/removed/trimmed repeatedly,
-                    # like arch/os/format.
-                    continue
-
-                self.current_features[feature] -= vas
-                if not self.current_features[feature]:
-                    del self.current_features[feature]
-
-        # update the deque and set of features with the latest call's worth of features.
-        self.current_feature_sets.append(call_features)
-        for feature, vas in call_features.items():
-            self.current_features[feature] |= vas
-
-        _, matches = self.ruleset.match(Scope.SPAN_OF_CALLS, self.current_features, ch.address)
-
-        newly_encountered_rules = set(matches.keys()) - self.last_span_matches
-
-        # don't emit match results for rules seen during the immediately preceeding spans.
-        #
-        # This means that we won't emit duplicate matches when there are multiple spans
-        #  that overlap a single matching event.
-        # It also handles the case of a tight loop containing matched logic;
-        #  only the first match will be recorded.
-        #
-        # In theory, this means the result document doesn't have *every* possible match location,
-        # but in practice, humans will only be interested in the first handful anyways.
-        suppressed_rules = set(self.last_span_matches)
-
-        # however, if a newly encountered rule depends on a suppressed rule,
-        # don't suppress that rule match, or we won't be able to reconstruct the vverbose output.
-        # see: https://github.com/mandiant/capa/pull/2532#issuecomment-2548508130
-        for new_rule in newly_encountered_rules:
-            suppressed_rules -= set(self.ruleset.rules[new_rule].get_dependencies(self.ruleset.rules_by_namespace))
-
-        for rule_name, res in matches.items():
-            if rule_name in suppressed_rules:
-                continue
-            self.matches[rule_name].extend(res)
-
-        self.last_span_matches = set(matches.keys())
+    return features, matches
 
 
 def find_thread_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle
-) -> ThreadCapabilities:
+) -> tuple[FeatureSet, MatchResults, MatchResults]:
     """
-    find matches for the given rules within the given thread,
-    which includes matches for all the spans and calls within it.
+    find matches for the given rules within the given thread.
+
+    returns: tuple containing (features for thread, match results for thread, match results for calls)
     """
     # all features found within this thread,
     # includes features found within calls.
@@ -175,19 +65,14 @@ def find_thread_capabilities(
     # might be found at different calls, that's ok.
     call_matches: MatchResults = collections.defaultdict(list)
 
-    span_matcher = SpanOfCallsMatcher(ruleset)
-
-    call_count = 0
-    for call_count, ch in enumerate(extractor.get_calls(ph, th)):  # noqa: B007
-        call_capabilities = find_call_capabilities(ruleset, extractor, ph, th, ch)
-        for feature, vas in call_capabilities.features.items():
+    for ch in extractor.get_calls(ph, th):
+        ifeatures, imatches = find_call_capabilities(ruleset, extractor, ph, th, ch)
+        for feature, vas in ifeatures.items():
             features[feature].update(vas)
 
-        for rule_name, res in call_capabilities.matches.items():
+        for rule_name, res in imatches.items():
             call_matches[rule_name].extend(res)
 
-        span_matcher.next(ch, call_capabilities.features)
-
     for feature, va in itertools.chain(extractor.extract_thread_features(ph, th), extractor.extract_global_features()):
         features[feature].add(va)
 
@@ -199,31 +84,16 @@ def find_thread_capabilities(
         for va, _ in res:
             capa.engine.index_rule_matches(features, rule, [va])
 
-    logger.debug(
-        "analyzed thread %d[%d] with %d events, %d features, and %d matches",
-        th.address.process.pid,
-        th.address.tid,
-        call_count,
-        len(features),
-        len(matches) + len(span_matcher.matches) + len(call_matches),
-    )
-    return ThreadCapabilities(features, matches, span_matcher.matches, call_matches)
-
-
-@dataclass
-class ProcessCapabilities:
-    process_matches: MatchResults
-    thread_matches: MatchResults
-    span_matches: MatchResults
-    call_matches: MatchResults
-    feature_count: int
+    return features, matches, call_matches
 
 
 def find_process_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle
-) -> ProcessCapabilities:
+) -> tuple[MatchResults, MatchResults, MatchResults, int]:
     """
     find matches for the given rules within the given process.
+
+    returns: tuple containing (match results for process, match results for threads, match results for calls, number of features)
     """
     # all features found within this process,
     # includes features found within threads (and calls).
@@ -233,48 +103,33 @@ def find_process_capabilities(
     # might be found at different threads, that's ok.
     thread_matches: MatchResults = collections.defaultdict(list)
 
-    # matches found at the span-of-calls scope.
-    # might be found at different spans, that's ok.
-    span_matches: MatchResults = collections.defaultdict(list)
-
     # matches found at the call scope.
     # might be found at different calls, that's ok.
     call_matches: MatchResults = collections.defaultdict(list)
 
     for th in extractor.get_threads(ph):
-        thread_capabilities = find_thread_capabilities(ruleset, extractor, ph, th)
-        for feature, vas in thread_capabilities.features.items():
+        features, tmatches, cmatches = find_thread_capabilities(ruleset, extractor, ph, th)
+        for feature, vas in features.items():
             process_features[feature].update(vas)
 
-        for rule_name, res in thread_capabilities.thread_matches.items():
+        for rule_name, res in tmatches.items():
             thread_matches[rule_name].extend(res)
 
-        for rule_name, res in thread_capabilities.span_matches.items():
-            span_matches[rule_name].extend(res)
-
-        for rule_name, res in thread_capabilities.call_matches.items():
+        for rule_name, res in cmatches.items():
             call_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(extractor.extract_process_features(ph), extractor.extract_global_features()):
         process_features[feature].add(va)
 
     _, process_matches = ruleset.match(Scope.PROCESS, process_features, ph.address)
-
-    logger.debug(
-        "analyzed process %d and extracted %d features with %d matches",
-        ph.address.pid,
-        len(process_features),
-        len(process_matches),
-    )
-    return ProcessCapabilities(process_matches, thread_matches, span_matches, call_matches, len(process_features))
+    return process_matches, thread_matches, call_matches, len(process_features)
 
 
 def find_dynamic_capabilities(
-    ruleset: RuleSet, extractor: DynamicFeatureExtractor, disable_progress: bool = False
-) -> Capabilities:
+    ruleset: RuleSet, extractor: DynamicFeatureExtractor, disable_progress=None
+) -> tuple[MatchResults, Any]:
     all_process_matches: MatchResults = collections.defaultdict(list)
     all_thread_matches: MatchResults = collections.defaultdict(list)
-    all_span_matches: MatchResults = collections.defaultdict(list)
     all_call_matches: MatchResults = collections.defaultdict(list)
 
     feature_counts = rdoc.DynamicFeatureCounts(file=0, processes=())
@@ -288,20 +143,19 @@ def find_dynamic_capabilities(
     ) as pbar:
         task = pbar.add_task("matching", total=n_processes, unit="processes")
         for p in processes:
-            process_capabilities = find_process_capabilities(ruleset, extractor, p)
-            feature_counts.processes += (
-                rdoc.ProcessFeatureCount(
-                    address=frz.Address.from_capa(p.address), count=process_capabilities.feature_count
-                ),
+            process_matches, thread_matches, call_matches, feature_count = find_process_capabilities(
+                ruleset, extractor, p
             )
+            feature_counts.processes += (
+                rdoc.ProcessFeatureCount(address=frz.Address.from_capa(p.address), count=feature_count),
+            )
+            logger.debug("analyzed %s and extracted %d features", p.address, feature_count)
 
-            for rule_name, res in process_capabilities.process_matches.items():
+            for rule_name, res in process_matches.items():
                 all_process_matches[rule_name].extend(res)
-            for rule_name, res in process_capabilities.thread_matches.items():
+            for rule_name, res in thread_matches.items():
                 all_thread_matches[rule_name].extend(res)
-            for rule_name, res in process_capabilities.span_matches.items():
-                all_span_matches[rule_name].extend(res)
-            for rule_name, res in process_capabilities.call_matches.items():
+            for rule_name, res in call_matches.items():
                 all_call_matches[rule_name].extend(res)
 
             pbar.advance(task)
@@ -310,26 +164,29 @@ def find_dynamic_capabilities(
     # mapping from feature (matched rule) to set of addresses at which it matched.
     process_and_lower_features: FeatureSet = collections.defaultdict(set)
     for rule_name, results in itertools.chain(
-        all_process_matches.items(), all_thread_matches.items(), all_span_matches.items(), all_call_matches.items()
+        all_process_matches.items(), all_thread_matches.items(), all_call_matches.items()
     ):
         locations = {p[0] for p in results}
         rule = ruleset[rule_name]
         capa.engine.index_rule_matches(process_and_lower_features, rule, locations)
 
-    all_file_capabilities = find_file_capabilities(ruleset, extractor, process_and_lower_features)
-    feature_counts.file = all_file_capabilities.feature_count
+    all_file_matches, feature_count = find_file_capabilities(ruleset, extractor, process_and_lower_features)
+    feature_counts.file = feature_count
 
     matches = dict(
         itertools.chain(
             # each rule exists in exactly one scope,
             # so there won't be any overlap among these following MatchResults,
             # and we can merge the dictionaries naively.
-            all_call_matches.items(),
-            all_span_matches.items(),
             all_thread_matches.items(),
             all_process_matches.items(),
-            all_file_capabilities.matches.items(),
+            all_call_matches.items(),
+            all_file_matches.items(),
         )
     )
 
-    return Capabilities(matches, feature_counts)
+    meta = {
+        "feature_counts": feature_counts,
+    }
+
+    return matches, meta

capa/capabilities/static.py

@@ -1,23 +1,16 @@
 # -*- coding: utf-8 -*-
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import time
 import logging
 import itertools
 import collections
-from dataclasses import dataclass
+from typing import Any
 
 import capa.perf
 import capa.helpers
@@ -25,23 +18,19 @@ import capa.features.freeze as frz
 import capa.render.result_document as rdoc
 from capa.rules import Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
-from capa.capabilities.common import Capabilities, find_file_capabilities
+from capa.capabilities.common import find_file_capabilities
 from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, StaticFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
-@dataclass
-class InstructionCapabilities:
-    features: FeatureSet
-    matches: MatchResults
-
-
 def find_instruction_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
-) -> InstructionCapabilities:
+) -> tuple[FeatureSet, MatchResults]:
     """
     find matches for the given rules for the given instruction.
+
+    returns: tuple containing (features for instruction, match results for instruction)
     """
     # all features found for the instruction.
     features: FeatureSet = collections.defaultdict(set)
@@ -59,21 +48,16 @@ def find_instruction_capabilities(
         for addr, _ in res:
             capa.engine.index_rule_matches(features, rule, [addr])
 
-    return InstructionCapabilities(features, matches)
-
-
-@dataclass
-class BasicBlockCapabilities:
-    features: FeatureSet
-    basic_block_matches: MatchResults
-    instruction_matches: MatchResults
+    return features, matches
 
 
 def find_basic_block_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle
-) -> BasicBlockCapabilities:
+) -> tuple[FeatureSet, MatchResults, MatchResults]:
     """
     find matches for the given rules within the given basic block.
+
+    returns: tuple containing (features for basic block, match results for basic block, match results for instructions)
     """
     # all features found within this basic block,
     # includes features found within instructions.
@@ -84,11 +68,11 @@ def find_basic_block_capabilities(
     insn_matches: MatchResults = collections.defaultdict(list)
 
     for insn in extractor.get_instructions(f, bb):
-        instruction_capabilities = find_instruction_capabilities(ruleset, extractor, f, bb, insn)
-        for feature, vas in instruction_capabilities.features.items():
+        ifeatures, imatches = find_instruction_capabilities(ruleset, extractor, f, bb, insn)
+        for feature, vas in ifeatures.items():
             features[feature].update(vas)
 
-        for rule_name, res in instruction_capabilities.matches.items():
+        for rule_name, res in imatches.items():
             insn_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(
@@ -104,20 +88,16 @@ def find_basic_block_capabilities(
         for va, _ in res:
             capa.engine.index_rule_matches(features, rule, [va])
 
-    return BasicBlockCapabilities(features, matches, insn_matches)
+    return features, matches, insn_matches
 
 
-@dataclass
-class CodeCapabilities:
-    function_matches: MatchResults
-    basic_block_matches: MatchResults
-    instruction_matches: MatchResults
-    feature_count: int
-
-
-def find_code_capabilities(ruleset: RuleSet, extractor: StaticFeatureExtractor, fh: FunctionHandle) -> CodeCapabilities:
+def find_code_capabilities(
+    ruleset: RuleSet, extractor: StaticFeatureExtractor, fh: FunctionHandle
+) -> tuple[MatchResults, MatchResults, MatchResults, int]:
     """
     find matches for the given rules within the given function.
+
+    returns: tuple containing (match results for function, match results for basic blocks, match results for instructions, number of features)
     """
     # all features found within this function,
     # includes features found within basic blocks (and instructions).
@@ -132,26 +112,26 @@ def find_code_capabilities(ruleset: RuleSet, extractor: StaticFeatureExtractor,
     insn_matches: MatchResults = collections.defaultdict(list)
 
     for bb in extractor.get_basic_blocks(fh):
-        basic_block_capabilities = find_basic_block_capabilities(ruleset, extractor, fh, bb)
-        for feature, vas in basic_block_capabilities.features.items():
+        features, bmatches, imatches = find_basic_block_capabilities(ruleset, extractor, fh, bb)
+        for feature, vas in features.items():
             function_features[feature].update(vas)
 
-        for rule_name, res in basic_block_capabilities.basic_block_matches.items():
+        for rule_name, res in bmatches.items():
             bb_matches[rule_name].extend(res)
 
-        for rule_name, res in basic_block_capabilities.instruction_matches.items():
+        for rule_name, res in imatches.items():
             insn_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(extractor.extract_function_features(fh), extractor.extract_global_features()):
         function_features[feature].add(va)
 
     _, function_matches = ruleset.match(Scope.FUNCTION, function_features, fh.address)
-    return CodeCapabilities(function_matches, bb_matches, insn_matches, len(function_features))
+    return function_matches, bb_matches, insn_matches, len(function_features)
 
 
 def find_static_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, disable_progress=None
-) -> Capabilities:
+) -> tuple[MatchResults, Any]:
     all_function_matches: MatchResults = collections.defaultdict(list)
     all_bb_matches: MatchResults = collections.defaultdict(list)
     all_insn_matches: MatchResults = collections.defaultdict(list)
@@ -185,36 +165,30 @@ def find_static_capabilities(
                 pbar.advance(task)
                 continue
 
-            code_capabilities = find_code_capabilities(ruleset, extractor, f)
+            function_matches, bb_matches, insn_matches, feature_count = find_code_capabilities(ruleset, extractor, f)
             feature_counts.functions += (
-                rdoc.FunctionFeatureCount(
-                    address=frz.Address.from_capa(f.address), count=code_capabilities.feature_count
-                ),
+                rdoc.FunctionFeatureCount(address=frz.Address.from_capa(f.address), count=feature_count),
             )
             t1 = time.time()
 
             match_count = 0
-            for name, matches_ in itertools.chain(
-                code_capabilities.function_matches.items(),
-                code_capabilities.basic_block_matches.items(),
-                code_capabilities.instruction_matches.items(),
-            ):
+            for name, matches_ in itertools.chain(function_matches.items(), bb_matches.items(), insn_matches.items()):
                 if not ruleset.rules[name].is_subscope_rule():
                     match_count += len(matches_)
 
             logger.debug(
                 "analyzed function 0x%x and extracted %d features, %d matches in %0.02fs",
                 f.address,
-                code_capabilities.feature_count,
+                feature_count,
                 match_count,
                 t1 - t0,
             )
 
-            for rule_name, res in code_capabilities.function_matches.items():
+            for rule_name, res in function_matches.items():
                 all_function_matches[rule_name].extend(res)
-            for rule_name, res in code_capabilities.basic_block_matches.items():
+            for rule_name, res in bb_matches.items():
                 all_bb_matches[rule_name].extend(res)
-            for rule_name, res in code_capabilities.instruction_matches.items():
+            for rule_name, res in insn_matches.items():
                 all_insn_matches[rule_name].extend(res)
 
             pbar.advance(task)
@@ -229,8 +203,8 @@ def find_static_capabilities(
         rule = ruleset[rule_name]
         capa.engine.index_rule_matches(function_and_lower_features, rule, locations)
 
-    all_file_capabilities = find_file_capabilities(ruleset, extractor, function_and_lower_features)
-    feature_counts.file = all_file_capabilities.feature_count
+    all_file_matches, feature_count = find_file_capabilities(ruleset, extractor, function_and_lower_features)
+    feature_counts.file = feature_count
 
     matches: MatchResults = dict(
         itertools.chain(
@@ -240,8 +214,13 @@ def find_static_capabilities(
             all_insn_matches.items(),
             all_bb_matches.items(),
             all_function_matches.items(),
-            all_file_capabilities.matches.items(),
+            all_file_matches.items(),
         )
     )
 
-    return Capabilities(matches, feature_counts, library_functions)
+    meta = {
+        "feature_counts": feature_counts,
+        "library_functions": library_functions,
+    }
+
+    return matches, meta

capa/engine.py

@@ -1,17 +1,10 @@
-# Copyright 2020 Google LLC
-#
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import copy
 import collections

capa/exceptions.py

@@ -1,18 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 class UnsupportedRuntimeError(RuntimeError):
     pass
 

capa/features/address.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import abc
 
 
@@ -114,7 +107,8 @@ class DynamicCallAddress(Address):
         return hash((self.thread, self.id))
 
     def __eq__(self, other):
-        return isinstance(other, DynamicCallAddress) and (self.thread, self.id) == (other.thread, other.id)
+        assert isinstance(other, DynamicCallAddress)
+        return (self.thread, self.id) == (other.thread, other.id)
 
     def __lt__(self, other):
         assert isinstance(other, DynamicCallAddress)

capa/features/basicblock.py

@@ -1,17 +1,10 @@
-# Copyright 2020 Google LLC
-#
+# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from capa.features.common import Feature
 

capa/features/com/__init__.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from enum import Enum
 
 from capa.helpers import assert_never

capa/features/com/classes.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 COM_CLASSES: dict[str, list[str]] = {
     "ClusAppWiz": ["24F97150-6689-11D1-9AA7-00C04FB93A80"],

capa/features/com/interfaces.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 COM_INTERFACES: dict[str, list[str]] = {
     "IClusterApplicationWizard": ["24F97151-6689-11D1-9AA7-00C04FB93A80"],

capa/features/common.py

@@ -1,17 +1,10 @@
-# Copyright 2021 Google LLC
-#
+# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import re
 import abc
@@ -92,7 +85,7 @@ class Result:
         self.success = success
         self.statement = statement
         self.children = children
-        self.locations = frozenset(locations) if locations is not None else frozenset()
+        self.locations = locations if locations is not None else set()
 
     def __eq__(self, other):
         if isinstance(other, bool):
@@ -105,25 +98,6 @@ class Result:
     def __nonzero__(self):
         return self.success
 
-    def __str__(self):
-        # as this object isn't user facing, this formatting is just to help with debugging
-
-        lines: list[str] = []
-
-        def rec(m: "Result", indent: int):
-            if isinstance(m.statement, capa.engine.Statement):
-                line = ("  " * indent) + str(m.statement.name) + " " + str(m.success)
-            else:
-                line = ("  " * indent) + str(m.statement) + " " + str(m.success) + " " + str(m.locations)
-
-            lines.append(line)
-
-            for child in m.children:
-                rec(child, indent + 1)
-
-        rec(self, 0)
-        return "\n".join(lines)
-
 
 class Feature(abc.ABC):  # noqa: B024
     # this is an abstract class, since we don't want anyone to instantiate it directly,
@@ -194,11 +168,7 @@ class Feature(abc.ABC):  # noqa: B024
     def evaluate(self, features: "capa.engine.FeatureSet", short_circuit=True) -> Result:
         capa.perf.counters["evaluate.feature"] += 1
         capa.perf.counters["evaluate.feature." + self.name] += 1
-        success = self in features
-        if success:
-            return Result(True, self, [], locations=features[self])
-        else:
-            return Result(False, self, [], locations=None)
+        return Result(self in features, self, [], locations=features.get(self, set()))
 
 
 class MatchedRule(Feature):

capa/features/extractors/base_extractor.py

@@ -1,17 +1,10 @@
-# Copyright 2021 Google LLC
-#
+# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import abc
 import hashlib
@@ -488,11 +481,11 @@ class DynamicFeatureExtractor:
         raise NotImplementedError()
 
 
-def ProcessFilter(extractor: DynamicFeatureExtractor, pids: set[int]) -> DynamicFeatureExtractor:
+def ProcessFilter(extractor: DynamicFeatureExtractor, processes: set) -> DynamicFeatureExtractor:
     original_get_processes = extractor.get_processes
 
     def filtered_get_processes(self):
-        yield from (f for f in original_get_processes() if f.address.pid in pids)
+        yield from (f for f in original_get_processes() if f.address.pid in processes)
 
     # we make a copy of the original extractor object and then update its get_processes() method with the decorated filter one.
     # this is in order to preserve the original extractor object's get_processes() method, in case it is used elsewhere in the code.
@@ -504,16 +497,4 @@ def ProcessFilter(extractor: DynamicFeatureExtractor, pids: set[int]) -> Dynamic
     return new_extractor
 
 
-def ThreadFilter(extractor: DynamicFeatureExtractor, threads: set[Address]) -> DynamicFeatureExtractor:
-    original_get_threads = extractor.get_threads
-
-    def filtered_get_threads(self, ph: ProcessHandle):
-        yield from (t for t in original_get_threads(ph) if t.address in threads)
-
-    new_extractor = copy(extractor)
-    new_extractor.get_threads = MethodType(filtered_get_threads, extractor)  # type: ignore
-
-    return new_extractor
-
-
 FeatureExtractor: TypeAlias = Union[StaticFeatureExtractor, DynamicFeatureExtractor]

capa/features/extractors/binexport2/__init__.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 """
 Proto files generated via protobuf v24.4:
 

capa/features/extractors/binexport2/arch/arm/helpers.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
 

capa/features/extractors/binexport2/arch/arm/insn.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 from typing import Iterator, Optional
 

capa/features/extractors/binexport2/arch/intel/helpers.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from typing import Optional
 from dataclasses import dataclass
 

capa/features/extractors/binexport2/arch/intel/insn.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 from typing import Iterator
 

capa/features/extractors/binexport2/basicblock.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from typing import Iterator
 

capa/features/extractors/binexport2/extractor.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 from typing import Iterator
 

capa/features/extractors/binexport2/file.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import io
 import logging
 from typing import Iterator

capa/features/extractors/binexport2/function.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from typing import Iterator
 
 from capa.features.file import FunctionName

capa/features/extractors/binexport2/helpers.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import re
 from typing import Union, Iterator, Optional
 from collections import defaultdict

capa/features/extractors/binexport2/insn.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 from typing import Iterator
 

capa/features/extractors/binja/basicblock.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from typing import Iterator
 
 from binaryninja import BasicBlock as BinjaBasicBlock

capa/features/extractors/binja/extractor.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from typing import Iterator
 
 import binaryninja as binja

capa/features/extractors/binja/file.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from typing import Iterator
 
 from binaryninja import Segment, BinaryView, SymbolType, SymbolBinding

capa/features/extractors/binja/find_binja_api.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import os
 import sys
 import logging

capa/features/extractors/binja/function.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import string
 from typing import Iterator
 

capa/features/extractors/binja/global_.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 from typing import Iterator
 

capa/features/extractors/binja/helpers.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import re
 from typing import Callable, Optional
 from dataclasses import dataclass

capa/features/extractors/binja/insn.py

@@ -1,22 +1,15 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-from typing import Any, Optional
-from collections.abc import Iterator
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+from typing import Any, Iterator, Optional
 
-import binaryninja as bn
+from binaryninja import Function
+from binaryninja import BasicBlock as BinjaBasicBlock
 from binaryninja import (
-    Function,
     BinaryView,
     ILRegister,
     SymbolType,
@@ -323,7 +316,7 @@ def extract_insn_offset_features(
     yield from results
 
 
-def is_nzxor_stack_cookie(f: Function, bb: bn.BasicBlock, llil: LowLevelILInstruction) -> bool:
+def is_nzxor_stack_cookie(f: Function, bb: BinjaBasicBlock, llil: LowLevelILInstruction) -> bool:
     """check if nzxor exists within stack cookie delta"""
     # TODO(xusheng): use LLIL SSA to do more accurate analysis
     # https://github.com/mandiant/capa/issues/1609

capa/features/extractors/cape/call.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/cape/extractor.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Union, Iterator
@@ -22,8 +15,8 @@ import capa.features.extractors.cape.thread
 import capa.features.extractors.cape.global_
 import capa.features.extractors.cape.process
 from capa.exceptions import EmptyReportError, UnsupportedFormatError
-from capa.features.common import Feature
-from capa.features.address import Address, AbsoluteVirtualAddress, _NoAddress
+from capa.features.common import Feature, Characteristic
+from capa.features.address import NO_ADDRESS, Address, AbsoluteVirtualAddress, _NoAddress
 from capa.features.extractors.cape.models import Call, Static, Process, CapeReport
 from capa.features.extractors.base_extractor import (
     CallHandle,
@@ -54,8 +47,7 @@ class CapeExtractor(DynamicFeatureExtractor):
 
     def get_base_address(self) -> Union[AbsoluteVirtualAddress, _NoAddress, None]:
         # value according to the PE header, the actual trace may use a different imagebase
-        assert self.report.static is not None
-        assert self.report.static.pe is not None
+        assert self.report.static is not None and self.report.static.pe is not None
         return AbsoluteVirtualAddress(self.report.static.pe.imagebase)
 
     def extract_global_features(self) -> Iterator[tuple[Feature, Address]]:
@@ -78,7 +70,11 @@ class CapeExtractor(DynamicFeatureExtractor):
         yield from capa.features.extractors.cape.process.get_threads(ph)
 
     def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[tuple[Feature, Address]]:
-        yield from []
+        if False:
+            # force this routine to be a generator,
+            # but we don't actually have any elements to generate.
+            yield Characteristic("never"), NO_ADDRESS
+        return
 
     def get_calls(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[CallHandle]:
         yield from capa.features.extractors.cape.thread.get_calls(ph, th)

capa/features/extractors/cape/file.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator
@@ -88,49 +81,31 @@ def extract_file_strings(report: CapeReport) -> Iterator[tuple[Feature, Address]
 
 
 def extract_used_regkeys(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
-    if not report.behavior.summary:
-        return
-
     for regkey in report.behavior.summary.keys:
         yield String(regkey), NO_ADDRESS
 
 
 def extract_used_files(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
-    if not report.behavior.summary:
-        return
-
     for file in report.behavior.summary.files:
         yield String(file), NO_ADDRESS
 
 
 def extract_used_mutexes(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
-    if not report.behavior.summary:
-        return
-
     for mutex in report.behavior.summary.mutexes:
         yield String(mutex), NO_ADDRESS
 
 
 def extract_used_commands(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
-    if not report.behavior.summary:
-        return
-
     for cmd in report.behavior.summary.executed_commands:
         yield String(cmd), NO_ADDRESS
 
 
 def extract_used_apis(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
-    if not report.behavior.summary:
-        return
-
     for symbol in report.behavior.summary.resolved_apis:
         yield String(symbol), NO_ADDRESS
 
 
 def extract_used_services(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
-    if not report.behavior.summary:
-        return
-
     for svc in report.behavior.summary.created_services:
         yield String(svc), NO_ADDRESS
     for svc in report.behavior.summary.started_services:

capa/features/extractors/cape/global_.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/cape/helpers.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from typing import Any
 

capa/features/extractors/cape/models.py

@@ -1,18 +1,12 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from typing import Any, Union, Optional, Annotated, TypeAlias
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
+import binascii
+from typing import Any, Union, Literal, Optional, Annotated, TypeAlias
 
 from pydantic import Field, BaseModel, ConfigDict
 from pydantic.functional_validators import BeforeValidator
@@ -26,7 +20,7 @@ def validate_hex_int(value):
 
 
 def validate_hex_bytes(value):
-    return bytes.fromhex(value) if isinstance(value, str) else value
+    return binascii.unhexlify(value) if isinstance(value, str) else value
 
 
 HexInt = Annotated[int, BeforeValidator(validate_hex_int)]
@@ -75,37 +69,34 @@ class Info(FlexibleModel):
     version: str
 
 
-class ImportedSymbol(FlexibleModel):
+class ImportedSymbol(ExactModel):
     address: HexInt
     name: Optional[str] = None
 
 
-class ImportedDll(FlexibleModel):
+class ImportedDll(ExactModel):
     dll: str
     imports: list[ImportedSymbol]
 
 
-"""
-class DirectoryEntry(FlexibleModel):
+class DirectoryEntry(ExactModel):
     name: str
     virtual_address: HexInt
     size: HexInt
-"""
 
 
-class Section(FlexibleModel):
+class Section(ExactModel):
     name: str
-    # raw_address: HexInt
+    raw_address: HexInt
     virtual_address: HexInt
-    # virtual_size: HexInt
-    # size_of_data: HexInt
-    # characteristics: str
-    # characteristics_raw: HexInt
-    # entropy: float
+    virtual_size: HexInt
+    size_of_data: HexInt
+    characteristics: str
+    characteristics_raw: HexInt
+    entropy: float
 
 
-"""
-class Resource(FlexibleModel):
+class Resource(ExactModel):
     name: str
     language: Optional[str] = None
     sublanguage: str
@@ -143,7 +134,7 @@ class DigitalSigner(FlexibleModel):
     extensions_subjectKeyIdentifier: Optional[str] = None
 
 
-class AuxSigner(FlexibleModel):
+class AuxSigner(ExactModel):
     name: str
     issued_to: str = Field(alias="Issued to")
     issued_by: str = Field(alias="Issued by")
@@ -151,7 +142,7 @@ class AuxSigner(FlexibleModel):
     sha1_hash: str = Field(alias="SHA1 hash")
 
 
-class Signer(FlexibleModel):
+class Signer(ExactModel):
     aux_sha1: Optional[str] = None
     aux_timestamp: Optional[str] = None
     aux_valid: Optional[bool] = None
@@ -160,61 +151,60 @@ class Signer(FlexibleModel):
     aux_signers: Optional[list[AuxSigner]] = None
 
 
-class Overlay(FlexibleModel):
+class Overlay(ExactModel):
     offset: HexInt
     size: HexInt
 
 
-class KV(FlexibleModel):
+class KV(ExactModel):
     name: str
     value: str
-"""
 
 
-class ExportedSymbol(FlexibleModel):
+class ExportedSymbol(ExactModel):
     address: HexInt
     name: str
-    # ordinal: int
+    ordinal: int
 
 
-class PE(FlexibleModel):
-    # peid_signatures: TODO
+class PE(ExactModel):
+    peid_signatures: TODO
     imagebase: HexInt
-    # entrypoint: HexInt
-    # reported_checksum: HexInt
-    # actual_checksum: HexInt
-    # osversion: str
-    # pdbpath: Optional[str] = None
-    # timestamp: str
+    entrypoint: HexInt
+    reported_checksum: HexInt
+    actual_checksum: HexInt
+    osversion: str
+    pdbpath: Optional[str] = None
+    timestamp: str
 
     # list[ImportedDll], or dict[basename(dll), ImportedDll]
-    imports: list[ImportedDll] | dict[str, ImportedDll] = Field(default_factory=list)  # type: ignore
-    # imported_dll_count: Optional[int] = None
-    # imphash: str
+    imports: Union[list[ImportedDll], dict[str, ImportedDll]]
+    imported_dll_count: Optional[int] = None
+    imphash: str
 
-    # exported_dll_name: Optional[str] = None
-    exports: list[ExportedSymbol] = Field(default_factory=list)
+    exported_dll_name: Optional[str] = None
+    exports: list[ExportedSymbol]
 
-    # dirents: list[DirectoryEntry]
-    sections: list[Section] = Field(default_factory=list)
+    dirents: list[DirectoryEntry]
+    sections: list[Section]
 
-    # ep_bytes: Optional[HexBytes] = None
+    ep_bytes: Optional[HexBytes] = None
 
-    # overlay: Optional[Overlay] = None
-    # resources: list[Resource]
-    # versioninfo: list[KV]
+    overlay: Optional[Overlay] = None
+    resources: list[Resource]
+    versioninfo: list[KV]
 
     # base64 encoded data
-    # icon: Optional[str] = None
+    icon: Optional[str] = None
     # MD5-like hash
-    # icon_hash: Optional[str] = None
+    icon_hash: Optional[str] = None
     # MD5-like hash
-    # icon_fuzzy: Optional[str] = None
+    icon_fuzzy: Optional[str] = None
     # short hex string
-    # icon_dhash: Optional[str] = None
+    icon_dhash: Optional[str] = None
 
-    # digital_signers: list[DigitalSigner]
-    # guest_signers: Signer
+    digital_signers: list[DigitalSigner]
+    guest_signers: Signer
 
 
 # TODO(mr-tz): target.file.dotnet, target.file.extracted_files, target.file.extracted_files_tool,
@@ -222,49 +212,48 @@ class PE(FlexibleModel):
 # https://github.com/mandiant/capa/issues/1814
 class File(FlexibleModel):
     type: str
-    # cape_type_code: Optional[int] = None
-    # cape_type: Optional[str] = None
+    cape_type_code: Optional[int] = None
+    cape_type: Optional[str] = None
 
-    # pid: Optional[Union[int, Literal[""]]] = None
-    # name: Union[list[str], str]
-    # path: str
-    # guest_paths: Union[list[str], str, None]
-    # timestamp: Optional[str] = None
+    pid: Optional[Union[int, Literal[""]]] = None
+    name: Union[list[str], str]
+    path: str
+    guest_paths: Union[list[str], str, None]
+    timestamp: Optional[str] = None
 
     #
     # hashes
     #
-    # crc32: str
+    crc32: str
     md5: str
     sha1: str
     sha256: str
-    # sha512: str
-    # sha3_384: Optional[str] = None
-    # ssdeep: str
+    sha512: str
+    sha3_384: Optional[str] = None
+    ssdeep: str
     # unsure why this would ever be "False"
-    # tlsh: Optional[Union[str, bool]] = None
-    # rh_hash: Optional[str] = None
+    tlsh: Optional[Union[str, bool]] = None
+    rh_hash: Optional[str] = None
 
     #
     # other metadata, static analysis
     #
-    # size: int
+    size: int
     pe: Optional[PE] = None
-    # ep_bytes: Optional[HexBytes] = None
-    # entrypoint: Optional[int] = None
-    # data: Optional[str] = None
-    # strings: Optional[list[str]] = None
+    ep_bytes: Optional[HexBytes] = None
+    entrypoint: Optional[int] = None
+    data: Optional[str] = None
+    strings: Optional[list[str]] = None
 
     #
     # detections (skip)
     #
-    # yara: Skip = None
-    # cape_yara: Skip = None
-    # clamav: Skip = None
-    # virustotal: Skip = None
+    yara: Skip = None
+    cape_yara: Skip = None
+    clamav: Skip = None
+    virustotal: Skip = None
 
 
-"""
 class ProcessFile(File):
     #
     # like a File, but also has dynamic analysis results
@@ -277,36 +266,35 @@ class ProcessFile(File):
     target_pid: Optional[Union[int, str]] = None
     target_path: Optional[str] = None
     target_process: Optional[str] = None
-"""
 
 
-class Argument(FlexibleModel):
+class Argument(ExactModel):
     name: str
     # unsure why empty list is provided here
     value: Union[HexInt, int, str, EmptyList]
     pretty_value: Optional[str] = None
 
 
-class Call(FlexibleModel):
-    # timestamp: str
+class Call(ExactModel):
+    timestamp: str
     thread_id: int
-    # category: str
+    category: str
 
     api: str
 
     arguments: list[Argument]
-    # status: bool
+    status: bool
     return_: HexInt = Field(alias="return")
     pretty_return: Optional[str] = None
 
-    # repeated: int
+    repeated: int
 
     # virtual addresses
-    # caller: HexInt
-    # parentcaller: HexInt
+    caller: HexInt
+    parentcaller: HexInt
 
     # index into calls array
-    # id: int
+    id: int
 
 
 # FlexibleModel to account for extended fields
@@ -316,15 +304,14 @@ class Process(FlexibleModel):
     process_id: int
     process_name: str
     parent_id: int
-    # module_path: str
-    # first_seen: str
+    module_path: str
+    first_seen: str
     calls: list[Call]
     threads: list[int]
     environ: dict[str, str]
 
 
-"""
-class ProcessTree(FlexibleModel):
+class ProcessTree(ExactModel):
     name: str
     pid: int
     parent_id: int
@@ -332,18 +319,17 @@ class ProcessTree(FlexibleModel):
     threads: list[int]
     environ: dict[str, str]
     children: list["ProcessTree"]
-"""
 
 
-class Summary(FlexibleModel):
+class Summary(ExactModel):
     files: list[str]
-    # read_files: list[str]
-    # write_files: list[str]
-    # delete_files: list[str]
+    read_files: list[str]
+    write_files: list[str]
+    delete_files: list[str]
     keys: list[str]
-    # read_keys: list[str]
-    # write_keys: list[str]
-    # delete_keys: list[str]
+    read_keys: list[str]
+    write_keys: list[str]
+    delete_keys: list[str]
     executed_commands: list[str]
     resolved_apis: list[str]
     mutexes: list[str]
@@ -351,8 +337,7 @@ class Summary(FlexibleModel):
     started_services: list[str]
 
 
-"""
-class EncryptedBuffer(FlexibleModel):
+class EncryptedBuffer(ExactModel):
     process_name: str
     pid: int
 
@@ -360,41 +345,38 @@ class EncryptedBuffer(FlexibleModel):
     buffer: str
     buffer_size: Optional[int] = None
     crypt_key: Optional[Union[HexInt, str]] = None
-"""
 
 
-class Behavior(FlexibleModel):
-    summary: Summary | None = None
+class Behavior(ExactModel):
+    summary: Summary
 
     # list of processes, of threads, of calls
     processes: list[Process]
     # tree of processes
-    # processtree: list[ProcessTree]
+    processtree: list[ProcessTree]
 
-    # anomaly: list[str]
-    # encryptedbuffers: list[EncryptedBuffer]
+    anomaly: list[str]
+    encryptedbuffers: list[EncryptedBuffer]
     # these are small objects that describe atomic events,
     # like file move, registry access.
     # we'll detect the same with our API call analysis.
-    # enhanced: Skip = None
+    enhanced: Skip = None
 
 
-class Target(FlexibleModel):
-    # category: str
+class Target(ExactModel):
+    category: str
     file: File
-    # pe: Optional[PE] = None
-
-
-class Static(FlexibleModel):
     pe: Optional[PE] = None
-    # flare_capa: Skip = None
 
 
-"""
-class Cape(FlexibleModel):
+class Static(ExactModel):
+    pe: Optional[PE] = None
+    flare_capa: Skip = None
+
+
+class Cape(ExactModel):
     payloads: list[ProcessFile]
     configs: Skip = None
-"""
 
 
 # flexible because there may be more sorts of analysis
@@ -417,14 +399,15 @@ class CapeReport(FlexibleModel):
     # post-processed results: process tree, anomalies, etc
     behavior: Behavior
 
+    # post-processed results: payloads and extracted configs
+    CAPE: Optional[Union[Cape, list]] = None
+    dropped: Optional[list[File]] = None
+    procdump: Optional[list[ProcessFile]] = None
+    procmemory: Optional[ListTODO] = None
+
     # =========================================================================
     # information we won't use in capa
     #
-    # post-processed results: payloads and extracted configs
-    # CAPE: Optional[Union[Cape, list]] = None
-    # dropped: Optional[list[File]] = None
-    # procdump: Optional[list[ProcessFile]] = None
-    # procmemory: Optional[ListTODO] = None
 
     #
     # NBIs and HBIs
@@ -433,32 +416,32 @@ class CapeReport(FlexibleModel):
     #
     # if we come up with a future use for this, go ahead and re-enable!
     #
-    # network: Skip = None
-    # suricata: Skip = None
-    # curtain: Skip = None
-    # sysmon: Skip = None
-    # url_analysis: Skip = None
+    network: Skip = None
+    suricata: Skip = None
+    curtain: Skip = None
+    sysmon: Skip = None
+    url_analysis: Skip = None
 
     # screenshot hash values
-    # deduplicated_shots: Skip = None
+    deduplicated_shots: Skip = None
     # k-v pairs describing the time it took to run each stage.
-    # statistics: Skip = None
+    statistics: Skip = None
     # k-v pairs of ATT&CK ID to signature name or similar.
-    # ttps: Skip = None
+    ttps: Skip = None
     # debug log messages
-    # debug: Skip = None
+    debug: Skip = None
 
     # various signature matches
     # we could potentially extend capa to use this info one day,
     # though it would be quite sandbox-specific,
     # and more detection-oriented than capability detection.
-    # signatures: Skip = None
-    # malfamily_tag: Optional[str] = None
-    # malscore: float
-    # detections: Skip = None
-    # detections2pid: Optional[dict[int, list[str]]] = None
+    signatures: Skip = None
+    malfamily_tag: Optional[str] = None
+    malscore: float
+    detections: Skip = None
+    detections2pid: Optional[dict[int, list[str]]] = None
     # AV detections for the sample.
-    # virustotal: Skip = None
+    virustotal: Skip = None
 
     @classmethod
     def from_buf(cls, buf: bytes) -> "CapeReport":

capa/features/extractors/cape/process.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/cape/thread.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/common.py

@@ -1,17 +1,10 @@
-# Copyright 2021 Google LLC
-#
+# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import io
 import re
 import logging

capa/features/extractors/dnfile/extractor.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/file.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/function.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/helpers.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/insn.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from __future__ import annotations
 

capa/features/extractors/dnfile/types.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 from typing import Optional
 

capa/features/extractors/dotnetfile.py

@@ -1,17 +1,10 @@
-# Copyright 2022 Google LLC
-#
+# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 from typing import Iterator
 from pathlib import Path

capa/features/extractors/drakvuf/call.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/extractor.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Union, Iterator
@@ -21,7 +14,7 @@ import capa.features.extractors.drakvuf.file
 import capa.features.extractors.drakvuf.thread
 import capa.features.extractors.drakvuf.global_
 import capa.features.extractors.drakvuf.process
-from capa.features.common import Feature
+from capa.features.common import Feature, Characteristic
 from capa.features.address import NO_ADDRESS, Address, ThreadAddress, ProcessAddress, AbsoluteVirtualAddress, _NoAddress
 from capa.features.extractors.base_extractor import (
     CallHandle,
@@ -74,7 +67,11 @@ class DrakvufExtractor(DynamicFeatureExtractor):
         yield from capa.features.extractors.drakvuf.process.get_threads(self.sorted_calls, ph)
 
     def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[tuple[Feature, Address]]:
-        yield from []
+        if False:
+            # force this routine to be a generator,
+            # but we don't actually have any elements to generate.
+            yield Characteristic("never"), NO_ADDRESS
+        return
 
     def get_calls(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[CallHandle]:
         yield from capa.features.extractors.drakvuf.thread.get_calls(self.sorted_calls, ph, th)

capa/features/extractors/drakvuf/file.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/global_.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/helpers.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import itertools
 

capa/features/extractors/drakvuf/models.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 from typing import Any, Iterator
 

capa/features/extractors/drakvuf/process.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/drakvuf/thread.py

@@ -1,17 +1,10 @@
-# Copyright 2024 Google LLC
-#
+# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import logging
 from typing import Iterator

capa/features/extractors/elf.py

@@ -1,17 +1,10 @@
-# Copyright 2021 Google LLC
-#
+# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import struct
 import logging
 import itertools
@@ -1088,7 +1081,7 @@ def guess_os_from_go_buildinfo(elf: ELF) -> Optional[OS]:
     #  and the 32-byte header is followed by varint-prefixed string data
     #  for the two string values we care about.
     # https://github.com/mandiant/GoReSym/blob/0860a1b1b4f3495e9fb7e71eb4386bf3e0a7c500/buildinfo/buildinfo.go#L185-L193
-    BUILDINFO_MAGIC = b"\xff Go buildinf:"
+    BUILDINFO_MAGIC = b"\xFF Go buildinf:"
 
     try:
         index = buf.index(BUILDINFO_MAGIC)

capa/features/extractors/elffile.py

@@ -1,17 +1,10 @@
-# Copyright 2021 Google LLC
-#
+# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import io
 import logging
 from typing import Iterator
@@ -80,7 +73,8 @@ def extract_file_export_names(elf: ELFFile, **kwargs):
 
 
 def extract_file_import_names(elf: ELFFile, **kwargs):
-    symbol_name_by_index: dict[int, str] = {}
+    # Create a dictionary to store symbol names by their index
+    symbol_names = {}
 
     # Extract symbol names and store them in the dictionary
     for segment in elf.iter_segments():
@@ -92,7 +86,7 @@ def extract_file_import_names(elf: ELFFile, **kwargs):
             logger.debug("Dynamic segment doesn't contain DT_SYMTAB")
             continue
 
-        for i, symbol in enumerate(segment.iter_symbols()):
+        for _, symbol in enumerate(segment.iter_symbols()):
             # The following conditions are based on the following article
             # http://www.m4b.io/elf/export/binary/analysis/2015/05/25/what-is-an-elf-export.html
             if not symbol.name:
@@ -106,7 +100,7 @@ def extract_file_import_names(elf: ELFFile, **kwargs):
             if symbol.entry.st_name == 0:
                 continue
 
-            symbol_name_by_index[i] = symbol.name
+            symbol_names[_] = symbol.name
 
     for segment in elf.iter_segments():
         if not isinstance(segment, DynamicSegment):
@@ -126,17 +120,10 @@ def extract_file_import_names(elf: ELFFile, **kwargs):
                     break
 
             for relocation in relocations:
-                if "r_info_sym" not in relocation.entry or "r_offset" not in relocation.entry:
+                # Extract the symbol name from the symbol table using the symbol index in the relocation
+                if relocation["r_info_sym"] not in symbol_names:
                     continue
-
-                symbol_address: int = relocation["r_offset"]
-                symbol_index: int = relocation["r_info_sym"]
-
-                if symbol_index not in symbol_name_by_index:
-                    continue
-                symbol_name = symbol_name_by_index[symbol_index]
-
-                yield Import(symbol_name), FileOffsetAddress(symbol_address)
+                yield Import(symbol_names[relocation["r_info_sym"]]), FileOffsetAddress(relocation["r_offset"])
 
 
 def extract_file_section_names(elf: ELFFile, **kwargs):

capa/features/extractors/ghidra/basicblock.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 
 import string
 import struct

capa/features/extractors/ghidra/extractor.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from typing import Iterator
 
 import capa.features.extractors.ghidra.file

capa/features/extractors/ghidra/file.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import re
 import struct
 from typing import Iterator

capa/features/extractors/ghidra/function.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 from typing import Iterator
 
 import ghidra

capa/features/extractors/ghidra/global_.py

@@ -1,17 +1,10 @@
-# Copyright 2023 Google LLC
-#
+# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
 # Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
+#  you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# Unless required by applicable law or agreed to in writing, software distributed under the License
+#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and limitations under the License.
 import logging
 import contextlib
 from typing import Iterator