mirror of
https://github.com/mandiant/capa.git
synced 2025-12-10 06:40:36 -08:00
Compare commits
12 Commits
v4.0.1
...
master-py2
| Author | SHA1 | Date | |
|---|---|---|---|
|
c547519ee4 | ||
|
|
b65286a435 | ||
|
|
3eef5c8773 | ||
|
|
f70b046ed4 | ||
|
|
ce8370931e | ||
|
|
8f58ccc8ae | ||
|
|
92cd6c6726 | ||
|
|
eea0e1e738 | ||
|
|
60834e3ecd | ||
|
|
54f8f6d162 | ||
|
|
62743e1363 | ||
|
|
b34d791d05 |
@@ -1,21 +0,0 @@
|
||||
# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.233.0/containers/python-3/.devcontainer/base.Dockerfile
|
||||
|
||||
# [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.10, 3.9, 3.8, 3.7, 3.6, 3-bullseye, 3.10-bullseye, 3.9-bullseye, 3.8-bullseye, 3.7-bullseye, 3.6-bullseye, 3-buster, 3.10-buster, 3.9-buster, 3.8-buster, 3.7-buster, 3.6-buster
|
||||
ARG VARIANT="3.10-bullseye"
|
||||
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
|
||||
|
||||
# [Choice] Node.js version: none, lts/*, 16, 14, 12, 10
|
||||
ARG NODE_VERSION="none"
|
||||
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi
|
||||
|
||||
# [Optional] If your pip requirements rarely change, uncomment this section to add them to the image.
|
||||
# COPY requirements.txt /tmp/pip-tmp/
|
||||
# RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
|
||||
# && rm -rf /tmp/pip-tmp
|
||||
|
||||
# [Optional] Uncomment this section to install additional OS packages.
|
||||
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
|
||||
# && apt-get -y install --no-install-recommends <your-package-list-here>
|
||||
|
||||
# [Optional] Uncomment this line to install global node packages.
|
||||
# RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && npm install -g <your-package-here>" 2>&1
|
||||
@@ -1,51 +0,0 @@
|
||||
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
|
||||
// https://github.com/microsoft/vscode-dev-containers/tree/v0.233.0/containers/python-3
|
||||
{
|
||||
"name": "Python 3",
|
||||
"build": {
|
||||
"dockerfile": "Dockerfile",
|
||||
"context": "..",
|
||||
"args": {
|
||||
// Update 'VARIANT' to pick a Python version: 3, 3.10, 3.9, 3.8, 3.7, 3.6
|
||||
// Append -bullseye or -buster to pin to an OS version.
|
||||
// Use -bullseye variants on local on arm64/Apple Silicon.
|
||||
"VARIANT": "3.10",
|
||||
// Options
|
||||
"NODE_VERSION": "none"
|
||||
}
|
||||
},
|
||||
|
||||
// Set *default* container specific settings.json values on container create.
|
||||
"settings": {
|
||||
"python.defaultInterpreterPath": "/usr/local/bin/python",
|
||||
"python.linting.enabled": true,
|
||||
"python.linting.pylintEnabled": true,
|
||||
"python.formatting.autopep8Path": "/usr/local/py-utils/bin/autopep8",
|
||||
"python.formatting.blackPath": "/usr/local/py-utils/bin/black",
|
||||
"python.formatting.yapfPath": "/usr/local/py-utils/bin/yapf",
|
||||
"python.linting.banditPath": "/usr/local/py-utils/bin/bandit",
|
||||
"python.linting.flake8Path": "/usr/local/py-utils/bin/flake8",
|
||||
"python.linting.mypyPath": "/usr/local/py-utils/bin/mypy",
|
||||
"python.linting.pycodestylePath": "/usr/local/py-utils/bin/pycodestyle",
|
||||
"python.linting.pydocstylePath": "/usr/local/py-utils/bin/pydocstyle",
|
||||
"python.linting.pylintPath": "/usr/local/py-utils/bin/pylint"
|
||||
},
|
||||
|
||||
// Add the IDs of extensions you want installed when the container is created.
|
||||
"extensions": [
|
||||
"ms-python.python",
|
||||
"ms-python.vscode-pylance"
|
||||
],
|
||||
|
||||
// Use 'forwardPorts' to make a list of ports inside the container available locally.
|
||||
// "forwardPorts": [],
|
||||
|
||||
// Use 'postCreateCommand' to run commands after the container is created.
|
||||
"postCreateCommand": "git submodule update --init && pip3 install --user -e .[dev]",
|
||||
|
||||
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
|
||||
"remoteUser": "vscode",
|
||||
"features": {
|
||||
"git": "latest"
|
||||
}
|
||||
}
|
||||
30
.github/CONTRIBUTING.md
vendored
30
.github/CONTRIBUTING.md
vendored
@@ -2,7 +2,7 @@
|
||||
|
||||
First off, thanks for taking the time to contribute!
|
||||
|
||||
The following is a set of guidelines for contributing to capa and its packages, which are hosted in the [Mandiant Organization](https://github.com/mandiant) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
|
||||
The following is a set of guidelines for contributing to capa and its packages, which are hosted in the [FireEye Organization](https://github.com/fireeye) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
|
||||
|
||||
#### Table Of Contents
|
||||
|
||||
@@ -32,9 +32,9 @@ This project and everyone participating in it is governed by the [Capa Code of C
|
||||
### Capa and its repositories
|
||||
|
||||
We host the capa project as three Github repositories:
|
||||
- [capa](https://github.com/mandiant/capa)
|
||||
- [capa-rules](https://github.com/mandiant/capa-rules)
|
||||
- [capa-testfiles](https://github.com/mandiant/capa-testfiles)
|
||||
- [capa](https://github.com/fireeye/capa)
|
||||
- [capa-rules](https://github.com/fireeye/capa-rules)
|
||||
- [capa-testfiles](https://github.com/fireeye/capa-testfiles)
|
||||
|
||||
The command line tools, logic engine, and other Python source code are found in the `capa` repository.
|
||||
This is the repository to fork when you want to enhance the features, performance, or user interface of capa.
|
||||
@@ -54,7 +54,7 @@ These are files you'll need in order to run the linter (in `--thorough` mode) an
|
||||
### Design Decisions
|
||||
|
||||
When we make a significant decision in how we maintain the project and what we can or cannot support,
|
||||
we will document it in the [capa issues tracker](https://github.com/mandiant/capa/issues).
|
||||
we will document it in the [capa issues tracker](https://github.com/fireeye/capa/issues).
|
||||
This is the best place review our discussions about what/how/why we do things in the project.
|
||||
If you have a question, check to see if it is documented there.
|
||||
If it is *not* documented there, or you can't find an answer, please open a issue.
|
||||
@@ -78,7 +78,7 @@ Fill out [the required template](./ISSUE_TEMPLATE/bug_report.md),
|
||||
#### Before Submitting A Bug Report
|
||||
|
||||
* **Determine [which repository the problem should be reported in](#capa-and-its-repositories)**.
|
||||
* **Perform a [cursory search](https://github.com/mandiant/capa/issues?q=is%3Aissue)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one.
|
||||
* **Perform a [cursory search](https://github.com/fireeye/capa/issues?q=is%3Aissue)** to see if the problem has already been reported. If it has **and the issue is still open**, add a comment to the existing issue instead of opening a new one.
|
||||
|
||||
#### How Do I Submit A (Good) Bug Report?
|
||||
|
||||
@@ -101,7 +101,7 @@ Explain the problem and include additional details to help maintainers reproduce
|
||||
Provide more context by answering these questions:
|
||||
|
||||
* **Did the problem start happening recently** (e.g. after updating to a new version of capa) or was this always a problem?
|
||||
* If the problem started happening recently, **can you reproduce the problem in an older version of capa?** What's the most recent version in which the problem doesn't happen? You can download older versions of capa from [the releases page](https://github.com/mandiant/capa/releases).
|
||||
* If the problem started happening recently, **can you reproduce the problem in an older version of capa?** What's the most recent version in which the problem doesn't happen? You can download older versions of capa from [the releases page](https://github.com/fireeye/capa/releases).
|
||||
* **Can you reliably reproduce the issue?** If not, provide details about how often the problem happens and under which conditions it normally happens.
|
||||
* If the problem is related to working with files (e.g. opening and editing files), **does the problem happen for all files and projects or only some?** Does the problem happen only when working with local or remote files (e.g. on network drives), with files of a specific type (e.g. only JavaScript or Python files), with large files or files with very long lines, or with files in a specific encoding? Is there anything else special about the files you are using?
|
||||
|
||||
@@ -119,7 +119,7 @@ Before creating enhancement suggestions, please check [this list](#before-submit
|
||||
#### Before Submitting An Enhancement Suggestion
|
||||
|
||||
* **Determine [which repository the enhancement should be suggested in](#capa-and-its-repositories).**
|
||||
* **Perform a [cursory search](https://github.com/mandiant/capa/issues?q=is%3Aissue)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
|
||||
* **Perform a [cursory search](https://github.com/fireeye/capa/issues?q=is%3Aissue)** to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
|
||||
|
||||
#### How Do I Submit A (Good) Enhancement Suggestion?
|
||||
|
||||
@@ -138,15 +138,15 @@ Enhancement suggestions are tracked as [GitHub issues](https://guides.github.com
|
||||
|
||||
Unsure where to begin contributing to capa? You can start by looking through these `good-first-issue` and `rule-idea` issues:
|
||||
|
||||
* [good-first-issue](https://github.com/mandiant/capa/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - issues which should only require a few lines of code, and a test or two.
|
||||
* [rule-idea](https://github.com/mandiant/capa-rules/issues?q=is%3Aissue+is%3Aopen+label%3A%22rule+idea%22) - issues that describe potential new rule ideas.
|
||||
* [good-first-issue](https://github.com/fireeye/capa/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - issues which should only require a few lines of code, and a test or two.
|
||||
* [rule-idea](https://github.com/fireeye/capa-rules/issues?q=is%3Aissue+is%3Aopen+label%3A%22rule+idea%22) - issues that describe potential new rule ideas.
|
||||
|
||||
Both issue lists are sorted by total number of comments. While not perfect, number of comments is a reasonable proxy for impact a given change will have.
|
||||
|
||||
#### Local development
|
||||
|
||||
capa and all its resources can be developed locally.
|
||||
For instructions on how to do this, see the "Method 3" section of the [installation guide](https://github.com/mandiant/capa/blob/master/doc/installation.md).
|
||||
For instructions on how to do this, see the "Method 3" section of the [installation guide](https://github.com/fireeye/capa/blob/master/doc/installation.md).
|
||||
|
||||
### Pull Requests
|
||||
|
||||
@@ -159,8 +159,8 @@ The process described here has several goals:
|
||||
|
||||
Please follow these steps to have your contribution considered by the maintainers:
|
||||
|
||||
1. Follow the [styleguides](#styleguides)
|
||||
2. Update the CHANGELOG and add tests and documentation. In case they are not needed, indicate it in [the PR template](pull_request_template.md).
|
||||
1. Follow all instructions in [the template](PULL_REQUEST_TEMPLATE.md)
|
||||
2. Follow the [styleguides](#styleguides)
|
||||
3. After you submit your pull request, verify that all [status checks](https://help.github.com/articles/about-status-checks/) are passing <details><summary>What if the status checks are failing? </summary>If a status check is failing, and you believe that the failure is unrelated to your change, please leave a comment on the pull request explaining why you believe the failure is unrelated. A maintainer will re-run the status check for you. If we conclude that the failure was a false positive, then we will open an issue to track that problem with our status check suite.</details>
|
||||
|
||||
While the prerequisites above must be satisfied prior to having your pull request reviewed, the reviewer(s) may ask you to complete additional design work, tests, or other changes before your pull request can be ultimately accepted.
|
||||
@@ -190,8 +190,8 @@ Our CI pipeline will reformat and enforce the Python styleguide.
|
||||
|
||||
All (non-nursery) capa rules must:
|
||||
|
||||
1. pass the [linter](https://github.com/mandiant/capa/blob/master/scripts/lint.py), and
|
||||
2. be formatted with [capafmt](https://github.com/mandiant/capa/blob/master/scripts/capafmt.py)
|
||||
1. pass the [linter](https://github.com/fireeye/capa/blob/master/scripts/lint.py), and
|
||||
2. be formatted with [capafmt](https://github.com/fireeye/capa/blob/master/scripts/capafmt.py)
|
||||
|
||||
This ensures that all rules meet the same minimum level of quality and are structured in a consistent way.
|
||||
Our CI pipeline will reformat and enforce the capa rules styleguide.
|
||||
|
||||
8
.github/ISSUE_TEMPLATE/bug_report.md
vendored
8
.github/ISSUE_TEMPLATE/bug_report.md
vendored
@@ -5,16 +5,16 @@ about: Create a report to help us improve
|
||||
---
|
||||
<!--
|
||||
# Is your bug report related to capa rules (for example a false positive)?
|
||||
We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/mandiant/capa-rules/issues.
|
||||
We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/fireeye/capa-rules/issues.
|
||||
|
||||
# Have you checked that your issue isn't already filed?
|
||||
Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
|
||||
Please search if there is a similar issue at https://github.com/fireeye/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
|
||||
|
||||
# Have you read capa's Code of Conduct?
|
||||
By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
|
||||
By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/fireeye/capa/blob/master/.github/CODE_OF_CONDUCT.md
|
||||
|
||||
# Have you read capa's CONTRIBUTING guide?
|
||||
It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs
|
||||
It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs
|
||||
-->
|
||||
|
||||
### Description
|
||||
|
||||
8
.github/ISSUE_TEMPLATE/feature_request.md
vendored
8
.github/ISSUE_TEMPLATE/feature_request.md
vendored
@@ -5,16 +5,16 @@ about: Suggest an idea for capa
|
||||
---
|
||||
<!--
|
||||
# Is your issue related to capa rules (for example an idea for a new rule)?
|
||||
We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/mandiant/capa-rules/issues.
|
||||
We use submodules to separate code, rules and test data. If your issue is related to capa rules, please report it at https://github.com/fireeye/capa-rules/issues.
|
||||
|
||||
# Have you checked that your issue isn't already filed?
|
||||
Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
|
||||
Please search if there is a similar issue at https://github.com/fireeye/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
|
||||
|
||||
# Have you read capa's Code of Conduct?
|
||||
By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
|
||||
By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/fireeye/capa/blob/master/.github/CODE_OF_CONDUCT.md
|
||||
|
||||
# Have you read capa's CONTRIBUTING guide?
|
||||
It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements
|
||||
It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements
|
||||
-->
|
||||
|
||||
### Summary
|
||||
|
||||
79
.github/mypy/mypy.ini
vendored
79
.github/mypy/mypy.ini
vendored
@@ -1,79 +0,0 @@
|
||||
[mypy]
|
||||
|
||||
[mypy-halo.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-tqdm.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-ruamel.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-networkx.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-pefile.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-viv_utils.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-flirt.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-smda.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-lief.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-idc.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-vivisect.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-envi.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-PE.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-idaapi.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-idautils.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-ida_bytes.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-ida_kernwin.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-ida_settings.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-ida_funcs.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-ida_loader.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-PyQt5.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-binaryninja.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-pytest.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-devtools.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-elftools.*]
|
||||
ignore_missing_imports = True
|
||||
|
||||
[mypy-dncil.*]
|
||||
ignore_missing_imports = True
|
||||
36
.github/pull_request_template.md
vendored
36
.github/pull_request_template.md
vendored
@@ -1,22 +1,32 @@
|
||||
|
||||
<!--
|
||||
Thank you for contributing to capa! <3
|
||||
Thank you for contributing to capa! :heart:
|
||||
|
||||
IMPORTANT NOTE
|
||||
It's most important that you submit your improvements. So even if you don't use this complete template we look forward to collaborating!
|
||||
|
||||
Please read capa's CONTRIBUTING guide if you haven't done so already.
|
||||
It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md
|
||||
It contains helpful information about how to contribute to capa. Check https://github.com/fireeye/capa/blob/master/.github/CONTRIBUTING.md
|
||||
|
||||
Please describe the changes in this pull request (PR). Include your motivation and context to help us review.
|
||||
|
||||
Please mention the issue your PR addresses (if any):
|
||||
closes #issue_number
|
||||
PR template based on https://embeddedartistry.com/blog/2017/08/04/a-github-pull-request-template-for-your-projects/
|
||||
-->
|
||||
|
||||
### Description
|
||||
|
||||
### Checklist
|
||||
<!-- Please describe the changes in this PR. Including your motivation and context helps us to review. -->
|
||||
|
||||
<!-- CHANGELOG.md has a `master (unreleased)` section. Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning in the release notes to this file. -->
|
||||
- [ ] No CHANGELOG update needed
|
||||
<!-- Tests prove that your fix/work as expected and ensure it doesn't break on the feature. -->
|
||||
closes # (issue)
|
||||
|
||||
### Type of change
|
||||
|
||||
Please update the [CHANGELOG.md](/CHANGELOG.md)
|
||||
|
||||
- [ ] Bug fix (non-breaking change which fixes an issue)
|
||||
- [ ] New feature (non-breaking change which adds functionality)
|
||||
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
|
||||
- [ ] This change requires a documentation update
|
||||
- [ ] I have made the corresponding changes to the documentation
|
||||
|
||||
### Tests
|
||||
|
||||
- [ ] I have added tests that prove my fix is effective or that my feature works
|
||||
- [ ] No new tests needed
|
||||
<!-- Please help us keeping capa documentation up-to-date -->
|
||||
- [ ] No documentation update needed
|
||||
|
||||
2
.github/pyinstaller/hooks/hook-smda.py
vendored
2
.github/pyinstaller/hooks/hook-smda.py
vendored
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
import PyInstaller.utils.hooks
|
||||
|
||||
# ref: https://groups.google.com/g/pyinstaller/c/amWi0-66uZI/m/miPoKfWjBAAJ
|
||||
|
||||
4
.github/pyinstaller/hooks/hook-vivisect.py
vendored
4
.github/pyinstaller/hooks/hook-vivisect.py
vendored
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
|
||||
from PyInstaller.utils.hooks import copy_metadata
|
||||
|
||||
@@ -45,8 +45,8 @@ hiddenimports = [
|
||||
"vivisect.analysis.crypto",
|
||||
"vivisect.analysis.crypto.constants",
|
||||
"vivisect.analysis.elf",
|
||||
"vivisect.analysis.elf",
|
||||
"vivisect.analysis.elf.elfplt",
|
||||
"vivisect.analysis.elf.elfplt_late",
|
||||
"vivisect.analysis.elf.libc_start_main",
|
||||
"vivisect.analysis.generic",
|
||||
"vivisect.analysis.generic",
|
||||
|
||||
94
.github/pyinstaller/pyinstaller.spec
vendored
94
.github/pyinstaller/pyinstaller.spec
vendored
@@ -1,64 +1,55 @@
|
||||
# -*- mode: python -*-
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
import os.path
|
||||
import subprocess
|
||||
|
||||
import wcwidth
|
||||
|
||||
|
||||
# git output will look like:
|
||||
#
|
||||
# tags/v1.0.0-0-g3af38dc
|
||||
# ------- tag
|
||||
# - commits since
|
||||
# g------- git hash fragment
|
||||
version = (
|
||||
subprocess.check_output(["git", "describe", "--always", "--tags", "--long"])
|
||||
.decode("utf-8")
|
||||
.strip()
|
||||
.replace("tags/", "")
|
||||
)
|
||||
# when invoking pyinstaller from the project root, this gets run from the project root.
|
||||
with open("./capa/version.py", "r", encoding="utf-8") as f:
|
||||
lines = f.read()
|
||||
# version.py contains the version string and other helper functions
|
||||
# here we manually replace the version value substring with the result of the above git output
|
||||
VERSION_DEF = "__version__ = "
|
||||
s = lines.index(VERSION_DEF)
|
||||
e = s + len(VERSION_DEF)
|
||||
off_rest_file = e + lines[e:].index("\n")
|
||||
lines = lines[s:e] + f'"{version}"' + lines[off_rest_file:]
|
||||
with open("./capa/version.py", "w", encoding="utf-8") as f:
|
||||
f.write(lines)
|
||||
# when invoking pyinstaller from the project root,
|
||||
# this gets run from the project root.
|
||||
with open('./capa/version.py', 'wb') as f:
|
||||
# git output will look like:
|
||||
#
|
||||
# tags/v1.0.0-0-g3af38dc
|
||||
# ------- tag
|
||||
# - commits since
|
||||
# g------- git hash fragment
|
||||
version = (subprocess.check_output(["git", "describe", "--always", "--tags", "--long"])
|
||||
.decode("utf-8")
|
||||
.strip()
|
||||
.replace("tags/", ""))
|
||||
f.write(("__version__ = '%s'" % version).encode("utf-8"))
|
||||
|
||||
a = Analysis(
|
||||
# when invoking pyinstaller from the project root,
|
||||
# this gets invoked from the directory of the spec file,
|
||||
# i.e. ./.github/pyinstaller
|
||||
["../../capa/main.py"],
|
||||
pathex=["capa"],
|
||||
['../../capa/main.py'],
|
||||
pathex=['capa'],
|
||||
binaries=None,
|
||||
datas=[
|
||||
# when invoking pyinstaller from the project root,
|
||||
# this gets invoked from the directory of the spec file,
|
||||
# i.e. ./.github/pyinstaller
|
||||
("../../rules", "rules"),
|
||||
("../../sigs", "sigs"),
|
||||
('../../rules', 'rules'),
|
||||
|
||||
# capa.render.default uses tabulate that depends on wcwidth.
|
||||
# it seems wcwidth uses a json file `version.json`
|
||||
# and this doesn't get picked up by pyinstaller automatically.
|
||||
# so we manually embed the wcwidth resources here.
|
||||
#
|
||||
# ref: https://stackoverflow.com/a/62278462/87207
|
||||
(os.path.dirname(wcwidth.__file__), "wcwidth"),
|
||||
(os.path.dirname(wcwidth.__file__), 'wcwidth')
|
||||
],
|
||||
# when invoking pyinstaller from the project root,
|
||||
# this gets run from the project root.
|
||||
hookspath=[".github/pyinstaller/hooks"],
|
||||
hookspath=['.github/pyinstaller/hooks'],
|
||||
runtime_hooks=None,
|
||||
excludes=[
|
||||
# ignore packages that would otherwise be bundled with the .exe.
|
||||
# review: build/pyinstaller/xref-pyinstaller.html
|
||||
|
||||
# we don't do any GUI stuff, so ignore these modules
|
||||
"tkinter",
|
||||
"_tkinter",
|
||||
@@ -68,6 +59,7 @@ a = Analysis(
|
||||
# since we don't spawn a notebook, we can safely remove these.
|
||||
"IPython",
|
||||
"ipywidgets",
|
||||
|
||||
# these are pulled in by networkx
|
||||
# but we don't need to compute the strongly connected components.
|
||||
"numpy",
|
||||
@@ -75,6 +67,7 @@ a = Analysis(
|
||||
"matplotlib",
|
||||
"pandas",
|
||||
"pytest",
|
||||
|
||||
# deps from viv that we don't use.
|
||||
# this duplicates the entries in `hook-vivisect`,
|
||||
# but works better this way.
|
||||
@@ -84,35 +77,36 @@ a = Analysis(
|
||||
"PyQt5",
|
||||
"qt5",
|
||||
"pyqtwebengine",
|
||||
"pyasn1",
|
||||
],
|
||||
)
|
||||
"pyasn1"
|
||||
])
|
||||
|
||||
a.binaries = a.binaries - TOC([("tcl85.dll", None, None), ("tk85.dll", None, None), ("_tkinter", None, None)])
|
||||
a.binaries = a.binaries - TOC([
|
||||
('tcl85.dll', None, None),
|
||||
('tk85.dll', None, None),
|
||||
('_tkinter', None, None)])
|
||||
|
||||
pyz = PYZ(a.pure, a.zipped_data)
|
||||
|
||||
exe = EXE(
|
||||
pyz,
|
||||
a.scripts,
|
||||
a.binaries,
|
||||
a.zipfiles,
|
||||
a.datas,
|
||||
exclude_binaries=False,
|
||||
name="capa",
|
||||
icon="logo.ico",
|
||||
debug=False,
|
||||
strip=None,
|
||||
upx=True,
|
||||
console=True,
|
||||
)
|
||||
exe = EXE(pyz,
|
||||
a.scripts,
|
||||
a.binaries,
|
||||
a.zipfiles,
|
||||
a.datas,
|
||||
exclude_binaries=False,
|
||||
name='capa',
|
||||
icon='logo.ico',
|
||||
debug=False,
|
||||
strip=None,
|
||||
upx=True,
|
||||
console=True )
|
||||
|
||||
# enable the following to debug the contents of the .exe
|
||||
#
|
||||
# coll = COLLECT(exe,
|
||||
#coll = COLLECT(exe,
|
||||
# a.binaries,
|
||||
# a.zipfiles,
|
||||
# a.datas,
|
||||
# strip=None,
|
||||
# upx=True,
|
||||
# name='capa-dat')
|
||||
|
||||
|
||||
64
.github/workflows/build.yml
vendored
64
.github/workflows/build.yml
vendored
@@ -1,8 +1,6 @@
|
||||
name: build
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [ master ]
|
||||
release:
|
||||
types: [edited, published]
|
||||
|
||||
@@ -11,19 +9,16 @@ jobs:
|
||||
name: PyInstaller for ${{ matrix.os }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
# set to false for debugging
|
||||
fail-fast: true
|
||||
matrix:
|
||||
include:
|
||||
- os: ubuntu-18.04
|
||||
- os: ubuntu-16.04
|
||||
# use old linux so that the shared library versioning is more portable
|
||||
artifact_name: capa
|
||||
asset_name: linux
|
||||
- os: windows-2022
|
||||
- os: windows-2019
|
||||
artifact_name: capa.exe
|
||||
asset_name: windows
|
||||
- os: macos-10.15
|
||||
# use older macOS for assumed better portability
|
||||
artifact_name: capa
|
||||
asset_name: macos
|
||||
steps:
|
||||
@@ -31,64 +26,29 @@ jobs:
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
submodules: true
|
||||
# using Python 3.8 to support running across multiple operating systems including Windows 7
|
||||
- name: Set up Python 3.8
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: 3.8
|
||||
- if: matrix.os == 'ubuntu-18.04'
|
||||
- if: matrix.os == 'ubuntu-16.04'
|
||||
run: sudo apt-get install -y libyaml-dev
|
||||
- name: Upgrade pip, setuptools
|
||||
run: python -m pip install --upgrade pip setuptools
|
||||
- name: Install capa with build requirements
|
||||
run: pip install -e .[build]
|
||||
- name: Install PyInstaller
|
||||
run: pip install 'pyinstaller==4.2'
|
||||
- name: Install capa
|
||||
run: pip install -e .
|
||||
- name: Build standalone executable
|
||||
run: pyinstaller --log-level DEBUG .github/pyinstaller/pyinstaller.spec
|
||||
- name: Does it run (PE)?
|
||||
run: pyinstaller .github/pyinstaller/pyinstaller.spec
|
||||
- name: Does it run?
|
||||
run: dist/capa "tests/data/Practical Malware Analysis Lab 01-01.dll_"
|
||||
- name: Does it run (Shellcode)?
|
||||
run: dist/capa "tests/data/499c2a85f6e8142c3f48d4251c9c7cd6.raw32"
|
||||
- name: Does it run (ELF)?
|
||||
run: dist/capa "tests/data/7351f8a40c5450557b24622417fc478d.elf_"
|
||||
- uses: actions/upload-artifact@v2
|
||||
with:
|
||||
name: ${{ matrix.asset_name }}
|
||||
path: dist/${{ matrix.artifact_name }}
|
||||
|
||||
test_run:
|
||||
name: Test run on ${{ matrix.os }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
needs: [build]
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
# OSs not already tested above
|
||||
- os: ubuntu-18.04
|
||||
artifact_name: capa
|
||||
asset_name: linux
|
||||
- os: ubuntu-20.04
|
||||
artifact_name: capa
|
||||
asset_name: linux
|
||||
- os: windows-2022
|
||||
artifact_name: capa.exe
|
||||
asset_name: windows
|
||||
steps:
|
||||
- name: Download ${{ matrix.asset_name }}
|
||||
uses: actions/download-artifact@v2
|
||||
with:
|
||||
name: ${{ matrix.asset_name }}
|
||||
- name: Set executable flag
|
||||
if: matrix.os != 'windows-2022'
|
||||
run: chmod +x ${{ matrix.artifact_name }}
|
||||
- name: Run capa
|
||||
run: ./${{ matrix.artifact_name }} -h
|
||||
|
||||
zip_and_upload:
|
||||
# upload zipped binaries to Release page
|
||||
if: github.event_name == 'release'
|
||||
name: zip and upload ${{ matrix.asset_name }}
|
||||
zip:
|
||||
name: zip ${{ matrix.asset_name }}
|
||||
runs-on: ubuntu-20.04
|
||||
needs: [build]
|
||||
needs: build
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
|
||||
41
.github/workflows/changelog.yml
vendored
41
.github/workflows/changelog.yml
vendored
@@ -1,41 +0,0 @@
|
||||
name: changelog
|
||||
|
||||
on:
|
||||
# We need pull_request_target instead of pull_request because a write
|
||||
# repository token is needed to add a review to a PR. DO NOT BUILD
|
||||
# OR RUN UNTRUSTED CODE FROM PRs IN THIS ACTION
|
||||
pull_request_target:
|
||||
types: [opened, edited, synchronize]
|
||||
|
||||
jobs:
|
||||
check_changelog:
|
||||
# no need to check for dependency updates via dependabot
|
||||
if: github.actor != 'dependabot[bot]' && github.actor != 'dependabot-preview[bot]'
|
||||
runs-on: ubuntu-20.04
|
||||
env:
|
||||
NO_CHANGELOG: '[x] No CHANGELOG update needed'
|
||||
steps:
|
||||
- name: Get changed files
|
||||
id: files
|
||||
uses: Ana06/get-changed-files@v1.2
|
||||
- name: check changelog updated
|
||||
id: changelog_updated
|
||||
env:
|
||||
PR_BODY: ${{ github.event.pull_request.body }}
|
||||
FILES: ${{ steps.files.outputs.modified }}
|
||||
run: |
|
||||
echo $FILES | grep -qF 'CHANGELOG.md' || echo $PR_BODY | grep -qiF "$NO_CHANGELOG"
|
||||
- name: Reject pull request if no CHANGELOG update
|
||||
if: ${{ always() && steps.changelog_updated.outcome == 'failure' }}
|
||||
uses: Ana06/automatic-pull-request-review@v0.1.0
|
||||
with:
|
||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
event: REQUEST_CHANGES
|
||||
body: "Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the `master (unreleased)` section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: `${{ env.NO_CHANGELOG }}`"
|
||||
allow_duplicate: false
|
||||
- name: Dismiss previous review if CHANGELOG update
|
||||
uses: Ana06/automatic-pull-request-review@v0.1.0
|
||||
with:
|
||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
event: DISMISS
|
||||
body: "CHANGELOG updated or no update needed, thanks! :smile:"
|
||||
3
.github/workflows/publish.yml
vendored
3
.github/workflows/publish.yml
vendored
@@ -15,7 +15,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: '3.7'
|
||||
python-version: '2.7'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
@@ -27,4 +27,3 @@ jobs:
|
||||
run: |
|
||||
python setup.py sdist bdist_wheel
|
||||
twine upload --skip-existing dist/*
|
||||
|
||||
|
||||
12
.github/workflows/tag.yml
vendored
12
.github/workflows/tag.yml
vendored
@@ -12,19 +12,13 @@ jobs:
|
||||
- name: Checkout capa-rules
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
repository: mandiant/capa-rules
|
||||
repository: fireeye/capa-rules
|
||||
token: ${{ secrets.CAPA_TOKEN }}
|
||||
- name: Tag capa-rules
|
||||
run: |
|
||||
# user information is needed to create annotated tags (with a message)
|
||||
git config user.email 'capa-dev@mandiant.com'
|
||||
git config user.name 'Capa Bot'
|
||||
name=${{ github.event.release.tag_name }}
|
||||
git tag $name -m "https://github.com/mandiant/capa/releases/$name"
|
||||
# TODO update branch name-major=${name%%.*}
|
||||
run: git tag ${{ github.event.release.tag_name }}
|
||||
- name: Push tag to capa-rules
|
||||
uses: ad-m/github-push-action@master
|
||||
with:
|
||||
repository: mandiant/capa-rules
|
||||
repository: fireeye/capa-rules
|
||||
github_token: ${{ secrets.CAPA_TOKEN }}
|
||||
tags: true
|
||||
|
||||
48
.github/workflows/tests.yml
vendored
48
.github/workflows/tests.yml
vendored
@@ -2,26 +2,11 @@ name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ master ]
|
||||
branches: [ master, master-py2 ]
|
||||
pull_request:
|
||||
branches: [ master ]
|
||||
|
||||
# save workspaces to speed up testing
|
||||
env:
|
||||
CAPA_SAVE_WORKSPACE: "True"
|
||||
branches: [ master, master-py2 ]
|
||||
|
||||
jobs:
|
||||
changelog_format:
|
||||
runs-on: ubuntu-20.04
|
||||
steps:
|
||||
- name: Checkout capa
|
||||
uses: actions/checkout@v2
|
||||
# The sync GH action in capa-rules relies on a single '- *$' in the CHANGELOG file
|
||||
- name: Ensure CHANGELOG has '- *$'
|
||||
run: |
|
||||
number=$(grep '\- *$' CHANGELOG.md | wc -l)
|
||||
if [ $number != 1 ]; then exit 1; fi
|
||||
|
||||
code_style:
|
||||
runs-on: ubuntu-20.04
|
||||
steps:
|
||||
@@ -30,29 +15,25 @@ jobs:
|
||||
- name: Set up Python 3.8
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: "3.8"
|
||||
python-version: 3.8
|
||||
- name: Install dependencies
|
||||
run: pip install -e .[dev]
|
||||
- name: Lint with isort
|
||||
run: isort --profile black --length-sort --line-width 120 -c .
|
||||
- name: Lint with black
|
||||
run: black -l 120 --check .
|
||||
- name: Lint with pycodestyle
|
||||
run: pycodestyle --show-source capa/ scripts/ tests/
|
||||
- name: Check types with mypy
|
||||
run: mypy --config-file .github/mypy/mypy.ini capa/ scripts/ tests/
|
||||
|
||||
rule_linter:
|
||||
runs-on: ubuntu-20.04
|
||||
steps:
|
||||
- name: Checkout capa with submodules
|
||||
- name: Checkout capa with rules submodule
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
submodules: recursive
|
||||
submodules: true
|
||||
- name: Set up Python 3.8
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: "3.8"
|
||||
python-version: 3.8
|
||||
- name: Install capa
|
||||
run: pip install -e .
|
||||
- name: Run rule linter
|
||||
@@ -65,20 +46,22 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-20.04, windows-2019, macos-11]
|
||||
os: [ubuntu-20.04, windows-2019, macos-10.15]
|
||||
# across all operating systems
|
||||
python-version: ["3.7", "3.10"]
|
||||
python-version: [3.6, 3.9]
|
||||
include:
|
||||
# on Ubuntu run these as well
|
||||
- os: ubuntu-20.04
|
||||
python-version: "3.8"
|
||||
python-version: 2.7
|
||||
- os: ubuntu-20.04
|
||||
python-version: "3.9"
|
||||
python-version: 3.7
|
||||
- os: ubuntu-20.04
|
||||
python-version: 3.8
|
||||
steps:
|
||||
- name: Checkout capa with submodules
|
||||
uses: actions/checkout@v2
|
||||
with:
|
||||
submodules: recursive
|
||||
submodules: true
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
@@ -86,7 +69,10 @@ jobs:
|
||||
- name: Install pyyaml
|
||||
if: matrix.os == 'ubuntu-20.04'
|
||||
run: sudo apt-get install -y libyaml-dev
|
||||
- name: Install Microsoft Visual C++ 9.0
|
||||
if: matrix.os == 'windows-2019' && matrix.python-version == '2.7'
|
||||
run: choco install vcpython27
|
||||
- name: Install capa
|
||||
run: pip install -e .[dev]
|
||||
- name: Run tests
|
||||
run: pytest -v tests/
|
||||
run: pytest tests/
|
||||
|
||||
8
.gitignore
vendored
8
.gitignore
vendored
@@ -114,11 +114,3 @@ venv.bak/
|
||||
isort-output.log
|
||||
black-output.log
|
||||
rule-linter-output.log
|
||||
.vscode
|
||||
scripts/perf/*.txt
|
||||
scripts/perf/*.svg
|
||||
scripts/perf/*.zip
|
||||
.direnv
|
||||
.envrc
|
||||
.DS_Store
|
||||
*/.DS_Store
|
||||
|
||||
651
CHANGELOG.md
651
CHANGELOG.md
@@ -1,596 +1,5 @@
|
||||
# Change Log
|
||||
|
||||
## master (unreleased)
|
||||
|
||||
### New Features
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
### New Rules (0)
|
||||
|
||||
-
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
|
||||
### capa explorer IDA Pro plugin
|
||||
|
||||
### Development
|
||||
|
||||
### Raw diffs
|
||||
- [capa v4.0.1...master](https://github.com/mandiant/capa/compare/v4.0.1...master)
|
||||
- [capa-rules v4.0.1...master](https://github.com/mandiant/capa-rules/compare/v4.0.1...master)
|
||||
|
||||
|
||||
## v4.0.1 (2022-08-15)
|
||||
Some rules contained invalid metadata fields that caused an error when rendering rule hits. We've updated all rules and enhanced the rule linter to catch such issues.
|
||||
|
||||
### New Rules (1)
|
||||
|
||||
- anti-analysis/obfuscation/obfuscated-with-vs-obfuscation jakub.jozwiak@mandiant.com
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
- linter: use pydantic to validate rule metadata #1141 @mike-hunhoff
|
||||
- build binaries using PyInstaller no longer overwrites functions in version.py #1136 @mr-tz
|
||||
|
||||
### Raw diffs
|
||||
- [capa v4.0.0...v4.0.1](https://github.com/mandiant/capa/compare/v4.0.0...v4.0.1)
|
||||
- [capa-rules v4.0.0...v4.0.1](https://github.com/mandiant/capa-rules/compare/v4.0.0...v4.0.1)
|
||||
|
||||
## v4.0.0 (2022-08-10)
|
||||
Version 4 adds support for analyzing .NET executables. capa will autodetect .NET modules, or you can explicitly invoke the new feature extractor via `--format dotnet`. We've also extended the rule syntax for .NET features including `namespace` and `class`.
|
||||
|
||||
Additionally, new `instruction` scope and `operand` features enable users to create more explicit rules. These features are not backwards compatible. We removed the previously used `/x32` and `/x64` flavors of number and operand features.
|
||||
|
||||
We updated 49 existing rules and added 22 new rules leveraging these new features and characteristics to detect capabilities seen in .NET malware.
|
||||
|
||||
More breaking changes include updates to the JSON results document, freeze file format schema (now format version v2), and the internal handling of addresses.
|
||||
|
||||
Thanks for all the support, especially to @htnhan, @jtothej, @sara-rn, @anushkavirgaonkar, and @_re_fox!
|
||||
|
||||
*Deprecation warning: v4.0 will be the last capa version to support the SMDA backend.*
|
||||
|
||||
### New Features
|
||||
|
||||
- add new scope "instruction" for matching mnemonics and operands #767 @williballenthin
|
||||
- add new feature "operand[{0, 1, 2}].number" for matching instruction operand immediate values #767 @williballenthin
|
||||
- add new feature "operand[{0, 1, 2}].offset" for matching instruction operand offsets #767 @williballenthin
|
||||
- extract additional offset/number features in certain circumstances #320 @williballenthin
|
||||
- add detection and basic feature extraction for dotnet #987 @mr-tz, @mike-hunhoff, @williballenthin
|
||||
- add file string extraction for dotnet files #1012 @mike-hunhoff
|
||||
- add file function-name extraction for dotnet files #1015 @mike-hunhoff
|
||||
- add unmanaged call characteristic for dotnet files #1023 @mike-hunhoff
|
||||
- add mixed mode characteristic feature extraction for dotnet files #1024 @mike-hunhoff
|
||||
- emit class and namespace features for dotnet files #1030 @mike-hunhoff
|
||||
- render: support Addresses that aren't simple integers, like .NET token+offset #981 @williballenthin
|
||||
- document rule tags and branches #1006 @williballenthin, @mr-tz
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- instruction scope and operand feature are new and are not backwards compatible with older versions of capa
|
||||
- Python 3.7 is now the minimum supported Python version #866 @williballenthin
|
||||
- remove /x32 and /x64 flavors of number and operand features #932 @williballenthin
|
||||
- the tool now accepts multiple paths to rules, and JSON doc updated accordingly @williballenthin
|
||||
- extractors must use handles to identify functions/basic blocks/instructions #981 @williballenthin
|
||||
- the freeze file format schema was updated, including format version bump to v2 #986 @williballenthin
|
||||
|
||||
Deprecation notice: as described in [#937](https://github.com/mandiant/capa/issues/937), we plan to remove the SMDA backend for v5. If you rely on this backend, please reach out so we can discuss extending the support for SMDA or transitioning your workflow to use vivisect.
|
||||
|
||||
### New Rules (30)
|
||||
|
||||
- data-manipulation/encryption/aes/manually-build-aes-constants huynh.t.nhan@gmail.com
|
||||
- nursery/get-process-image-filename michael.hunhoff@mandiant.com
|
||||
- compiler/v/compiled-with-v jakub.jozwiak@mandiant.com
|
||||
- compiler/zig/compiled-with-zig jakub.jozwiak@mandiant.com
|
||||
- anti-analysis/packer/huan/packed-with-huan jakub.jozwiak@mandiant.com
|
||||
- internal/limitation/file/internal-dotnet-file-limitation william.ballenthin@mandiant.com
|
||||
- nursery/get-os-information-via-kuser_shared_data @mr-tz
|
||||
- load-code/pe/resolve-function-by-parsing-PE-exports @sara-rn
|
||||
- anti-analysis/packer/huan/packed-with-huan jakub.jozwiak@mandiant.com
|
||||
- nursery/execute-dotnet-assembly anushka.virgaonkar@mandiant.com
|
||||
- nursery/invoke-dotnet-assembly-method anushka.virgaonkar@mandiant.com
|
||||
- collection/screenshot/capture-screenshot-via-keybd-event @_re_fox
|
||||
- collection/browser/gather-chrome-based-browser-login-information @_re_fox
|
||||
- nursery/power-down-monitor michael.hunhoff@mandiant.com
|
||||
- nursery/hash-data-using-aphash @_re_fox
|
||||
- nursery/hash-data-using-jshash @_re_fox
|
||||
- host-interaction/file-system/files/list/enumerate-files-on-windows moritz.raabe@mandiant.com anushka.virgaonkar@mandiant.com
|
||||
- nursery/check-clipboard-data anushka.virgaonkar@mandiant.com
|
||||
- nursery/clear-clipboard-data anushka.virgaonkar@mandiant.com
|
||||
- nursery/compile-dotnet-assembly anushka.virgaonkar@mandiant.com
|
||||
- nursery/create-process-via-wmi anushka.virgaonkar@mandiant.com
|
||||
- nursery/display-service-notification-message-box anushka.virgaonkar@mandiant.com
|
||||
- nursery/find-process-by-name anushka.virgaonkar@mandiant.com
|
||||
- nursery/generate-random-numbers-in-dotnet anushka.virgaonkar@mandiant.com
|
||||
- nursery/send-keystrokes anushka.virgaonkar@mandiant.com
|
||||
- nursery/send-request-in-dotnet anushka.virgaonakr@mandiant.com
|
||||
- nursery/terminate-process-by-name-in-dotnet anushka.virgaonkar@mandiant.com
|
||||
- nursery/hash-data-using-rshash @_re_fox
|
||||
- persistence/authentication-process/act-as-credential-manager-dll jakub.jozwiak@mandiant.com
|
||||
- persistence/authentication-process/act-as-password-filter-dll jakub.jozwiak@mandiant.com
|
||||
|
||||
### Bug Fixes
|
||||
- improve handling _ prefix compile/link artifact #924 @mike-hunhoff
|
||||
- better detect OS in ELF samples #988 @williballenthin
|
||||
- display number feature zero in vverbose #1097 @mike-hunhoff
|
||||
|
||||
### capa explorer IDA Pro plugin
|
||||
- improve file format extraction #918 @mike-hunhoff
|
||||
- remove decorators added by IDA to ELF imports #919 @mike-hunhoff
|
||||
- bug fixes for Address abstraction #1091 @mike-hunhoff
|
||||
|
||||
### Development
|
||||
|
||||
### Raw diffs
|
||||
- [capa v3.2.0...v4.0.0](https://github.com/mandiant/capa/compare/v3.2.0...master)
|
||||
- [capa-rules v3.2.0...v4.0.0](https://github.com/mandiant/capa-rules/compare/v3.2.0...master)
|
||||
|
||||
## v3.2.1 (2022-06-06)
|
||||
This out-of-band release bumps the SMDA dependency version to enable installation on Python 3.10.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- update SMDA dependency @mike-hunhoff #922
|
||||
|
||||
### Raw diffs
|
||||
- [capa v3.2.0...v3.2.1](https://github.com/mandiant/capa/compare/v3.2.0...v3.2.1)
|
||||
- [capa-rules v3.2.0...v3.2.1](https://github.com/mandiant/capa-rules/compare/v3.2.0...v3.2.1)
|
||||
|
||||
## v3.2.0 (2022-03-03)
|
||||
This release adds a new characteristic `characteristic: call $+5` enabling users to create more explicit rules. The linter now also validates ATT&CK and MBC categories. Additionally, many dependencies, including the vivisect backend, have been updated.
|
||||
|
||||
One rule has been added and many more have been improved.
|
||||
|
||||
Thanks for all the support, especially to @kn0wl3dge and first time contributor @uckelman-sf!
|
||||
|
||||
### New Features
|
||||
|
||||
- linter: validate ATT&CK/MBC categories and IDs #103 @kn0wl3dge
|
||||
- extractor: add characteristic "call $+5" feature #366 @kn0wl3dge
|
||||
|
||||
### New Rules (1)
|
||||
|
||||
- anti-analysis/obfuscation/obfuscated-with-advobfuscator jakub.jozwiak@mandiant.com
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- remove typing package as a requirement for Python 3.7+ compatibility #901 @uckelman-sf
|
||||
- elf: fix OS detection for Linux kernel modules #867 @williballenthin
|
||||
|
||||
### Raw diffs
|
||||
- [capa v3.1.0...v3.2.0](https://github.com/mandiant/capa/compare/v3.1.0...v3.2.0)
|
||||
- [capa-rules v3.1.0...v3.2.0](https://github.com/mandiant/capa-rules/compare/v3.1.0...v3.2.0)
|
||||
|
||||
## v3.1.0 (2022-01-10)
|
||||
This release improves the performance of capa while also adding 23 new rules and many code quality enhancements. We profiled capa's CPU usage and optimized the way that it matches rules, such as by short circuiting when appropriate. According to our testing, the matching phase is approximately 66% faster than v3.0.3! We also added support for Python 3.10, aarch64 builds, and additional MAEC metadata in the rule headers.
|
||||
|
||||
This release adds 23 new rules, including nine by Jakub Jozwiak of Mandiant. @ryantxu1 and @dzbeck updated the ATT&CK and MBC mappings for many rules. Thank you!
|
||||
|
||||
And as always, welcome first time contributors!
|
||||
|
||||
- @kn0wl3dge
|
||||
- @jtothej
|
||||
- @cl30
|
||||
|
||||
|
||||
### New Features
|
||||
|
||||
- engine: short circuit logic nodes for better performance #824 @williballenthin
|
||||
- engine: add optimizer the order faster nodes first #829 @williballenthin
|
||||
- engine: optimize rule evaluation by skipping rules that can't match #830 @williballenthin
|
||||
- support python 3.10 #816 @williballenthin
|
||||
- support aarch64 #683 @williballenthin
|
||||
- rules: support maec/malware-family meta #841 @mr-tz
|
||||
- engine: better type annotations/exhaustiveness checking #839 @cl30
|
||||
|
||||
### Breaking Changes: None
|
||||
|
||||
### New Rules (23)
|
||||
|
||||
- nursery/delete-windows-backup-catalog michael.hunhoff@mandiant.com
|
||||
- nursery/disable-automatic-windows-recovery-features michael.hunhoff@mandiant.com
|
||||
- nursery/capture-webcam-video @johnk3r
|
||||
- nursery/create-registry-key-via-stdregprov michael.hunhoff@mandiant.com
|
||||
- nursery/delete-registry-key-via-stdregprov michael.hunhoff@mandiant.com
|
||||
- nursery/delete-registry-value-via-stdregprov michael.hunhoff@mandiant.com
|
||||
- nursery/query-or-enumerate-registry-key-via-stdregprov michael.hunhoff@mandiant.com
|
||||
- nursery/query-or-enumerate-registry-value-via-stdregprov michael.hunhoff@mandiant.com
|
||||
- nursery/set-registry-value-via-stdregprov michael.hunhoff@mandiant.com
|
||||
- data-manipulation/compression/decompress-data-using-ucl jakub.jozwiak@mandiant.com
|
||||
- linking/static/wolfcrypt/linked-against-wolfcrypt jakub.jozwiak@mandiant.com
|
||||
- linking/static/wolfssl/linked-against-wolfssl jakub.jozwiak@mandiant.com
|
||||
- anti-analysis/packer/pespin/packed-with-pespin jakub.jozwiak@mandiant.com
|
||||
- load-code/shellcode/execute-shellcode-via-windows-fibers jakub.jozwiak@mandiant.com
|
||||
- load-code/shellcode/execute-shellcode-via-enumuilanguages jakub.jozwiak@mandiant.com
|
||||
- anti-analysis/packer/themida/packed-with-themida william.ballenthin@mandiant.com
|
||||
- load-code/shellcode/execute-shellcode-via-createthreadpoolwait jakub.jozwiak@mandiant.com
|
||||
- host-interaction/process/inject/inject-shellcode-using-a-file-mapping-object jakub.jozwiak@mandiant.com
|
||||
- load-code/shellcode/execute-shellcode-via-copyfile2 jakub.jozwiak@mandiant.com
|
||||
- malware-family/plugx/match-known-plugx-module still@teamt5.org
|
||||
|
||||
### Rule Changes
|
||||
|
||||
- update ATT&CK mappings by @ryantxu1
|
||||
- update ATT&CK and MBC mappings by @dzbeck
|
||||
- aplib detection by @cdong1012
|
||||
- golang runtime detection by @stevemk14eber
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- fix circular import error #825 @williballenthin
|
||||
- fix smda negative number extraction #430 @kn0wl3dge
|
||||
|
||||
### capa explorer IDA Pro plugin
|
||||
|
||||
- pin supported versions to >= 7.4 and < 8.0 #849 @mike-hunhoff
|
||||
|
||||
### Development
|
||||
|
||||
- add profiling infrastructure #828 @williballenthin
|
||||
- linter: detect shellcode extension #820 @mr-tz
|
||||
- show features script: add backend flag #430 @kn0wl3dge
|
||||
|
||||
### Raw diffs
|
||||
- [capa v3.0.3...v3.1.0](https://github.com/mandiant/capa/compare/v3.0.3...v3.1.0)
|
||||
- [capa-rules v3.0.3...v3.1.0](https://github.com/mandiant/capa-rules/compare/v3.0.3...v3.1.0)
|
||||
|
||||
|
||||
## v3.0.3 (2021-10-27)
|
||||
|
||||
This is primarily a rule maintenance release:
|
||||
- eight new rules, including all relevant techniques from [ATT&CK v10](https://medium.com/mitre-attack/introducing-attack-v10-7743870b37e3), and
|
||||
- two rules removed, due to the prevalence of false positives
|
||||
|
||||
We've also tweaked the status codes returned by capa.exe to be more specific and added a bit more metadata to the JSON output format.
|
||||
|
||||
As always, welcome first time contributors!
|
||||
- still@teamt5.org
|
||||
- zander.work@mandiant.com
|
||||
|
||||
|
||||
### New Features
|
||||
|
||||
- show in which function a BB match is #130 @williballenthin
|
||||
- main: exit with unique error codes when bailing #802 @williballenthin
|
||||
|
||||
### New Rules (8)
|
||||
|
||||
- nursery/resolve-function-by-fnv-1a-hash still@teamt5.org
|
||||
- data-manipulation/encryption/encrypt-data-using-memfrob-from-glibc zander.work@mandiant.com
|
||||
- collection/group-policy/discover-group-policy-via-gpresult william.ballenthin@mandiant.com
|
||||
- host-interaction/bootloader/manipulate-safe-mode-programs william.ballenthin@mandiant.com
|
||||
- nursery/enable-safe-mode-boot william.ballenthin@mandiant.com
|
||||
- persistence/iis/persist-via-iis-module william.ballenthin@mandiant.com
|
||||
- persistence/iis/persist-via-isapi-extension william.ballenthin@mandiant.com
|
||||
- targeting/language/identify-system-language-via-api william.ballenthin@mandiant.com
|
||||
|
||||
## Removed rules (2)
|
||||
- load-code/pe/parse-pe-exports: too many false positives in unrelated structure accesses
|
||||
- anti-analysis/anti-vm/vm-detection/execute-anti-vm-instructions: too many false positives in junk code
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- update references from FireEye to Mandiant
|
||||
|
||||
### Raw diffs
|
||||
- [capa v3.0.2...v3.0.3](https://github.com/fireeye/capa/compare/v3.0.2...v3.0.3)
|
||||
- [capa-rules v3.0.2...v3.0.3](https://github.com/fireeye/capa-rules/compare/v3.0.2...v3.0.3)
|
||||
|
||||
## v3.0.2 (2021-09-28)
|
||||
|
||||
This release fixes an issue with the standalone executables built with PyInstaller when running capa against ELF files.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- fix bug in PyInstaller config preventing ELF analysis #795 @mr-tz
|
||||
|
||||
### Raw diffs
|
||||
- [capa v3.0.1...v3.0.2](https://github.com/fireeye/capa/compare/v3.0.1...v3.0.2)
|
||||
- [capa-rules v3.0.1...v3.0.2](https://github.com/fireeye/capa-rules/compare/v3.0.1...v3.0.2)
|
||||
|
||||
## v3.0.1 (2021-09-27)
|
||||
|
||||
This version updates the version of vivisect used by capa. Users will experience fewer bugs and find improved analysis results.
|
||||
|
||||
Thanks to the community for highlighting issues and analysis misses. Your feedback is crucial to further improve capa.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- fix many underlying bugs in vivisect analysis and update to version v1.0.5 #786 @williballenthin
|
||||
|
||||
### Raw diffs
|
||||
- [capa v3.0.0...v3.0.1](https://github.com/fireeye/capa/compare/v3.0.0...v3.0.1)
|
||||
- [capa-rules v3.0.0...v3.0.1](https://github.com/fireeye/capa-rules/compare/v3.0.0...v3.0.1)
|
||||
|
||||
## v3.0.0 (2021-09-15)
|
||||
|
||||
We are excited to announce version 3.0! :tada:
|
||||
|
||||
capa 3.0:
|
||||
- adds support for ELF files targeting Linux thanks to [Intezer](https://www.intezer.com/)
|
||||
- adds new features to specify OS, CPU architecture, and file format
|
||||
- fixes a few bugs that may have led to false negatives (missed capabilities) in older versions
|
||||
- adds 80 new rules, including 36 describing techniques for Linux
|
||||
|
||||
A huge thanks to everyone who submitted issues, provided feedback, and contributed code and rules.
|
||||
Special acknowledgement to @Adir-Shemesh and @TcM1911 of [Intezer](https://www.intezer.com/) for contributing the code to enable ELF support.
|
||||
Also, welcome first time contributors:
|
||||
- @jaredscottwilson
|
||||
- @cdong1012
|
||||
- @jlepore-fe
|
||||
|
||||
### New Features
|
||||
|
||||
- all: add support for ELF files #700 @Adir-Shemesh @TcM1911
|
||||
- rule format: add feature `format: ` for file format, like `format: pe` #723 @williballenthin
|
||||
- rule format: add feature `arch: ` for architecture, like `arch: amd64` #723 @williballenthin
|
||||
- rule format: add feature `os: ` for operating system, like `os: windows` #723 @williballenthin
|
||||
- rule format: add feature `substring: ` for verbatim strings with leading/trailing wildcards #737 @williballenthin
|
||||
- scripts: add `profile-memory.py` for profiling memory usage #736 @williballenthin
|
||||
- main: add light weight ELF file feature extractor to detect file limitations #770 @mr-tz
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- rules using `format`, `arch`, `os`, or `substring` features cannot be used by capa versions prior to v3
|
||||
- legacy term `arch` (i.e., "x32") is now called `bitness` @williballenthin
|
||||
- freeze format gains new section for "global" features #759 @williballenthin
|
||||
|
||||
### New Rules (80)
|
||||
|
||||
- collection/webcam/capture-webcam-image @johnk3r
|
||||
- nursery/list-drag-and-drop-files michael.hunhoff@mandiant.com
|
||||
- nursery/monitor-clipboard-content michael.hunhoff@mandiant.com
|
||||
- nursery/monitor-local-ipv4-address-changes michael.hunhoff@mandiant.com
|
||||
- nursery/load-windows-common-language-runtime michael.hunhoff@mandiant.com
|
||||
- nursery/resize-volume-shadow-copy-storage michael.hunhoff@mandiant.com
|
||||
- nursery/add-user-account-group michael.hunhoff@mandiant.com
|
||||
- nursery/add-user-account-to-group michael.hunhoff@mandiant.com
|
||||
- nursery/add-user-account michael.hunhoff@mandiant.com
|
||||
- nursery/change-user-account-password michael.hunhoff@mandiant.com
|
||||
- nursery/delete-user-account-from-group michael.hunhoff@mandiant.com
|
||||
- nursery/delete-user-account-group michael.hunhoff@mandiant.com
|
||||
- nursery/delete-user-account michael.hunhoff@mandiant.com
|
||||
- nursery/list-domain-servers michael.hunhoff@mandiant.com
|
||||
- nursery/list-groups-for-user-account michael.hunhoff@mandiant.com
|
||||
- nursery/list-user-account-groups michael.hunhoff@mandiant.com
|
||||
- nursery/list-user-accounts-for-group michael.hunhoff@mandiant.com
|
||||
- nursery/list-user-accounts michael.hunhoff@mandiant.com
|
||||
- nursery/parse-url michael.hunhoff@mandiant.com
|
||||
- nursery/register-raw-input-devices michael.hunhoff@mandiant.com
|
||||
- anti-analysis/packer/gopacker/packed-with-gopacker jared.wilson@mandiant.com
|
||||
- host-interaction/driver/create-device-object @mr-tz
|
||||
- host-interaction/process/create/execute-command @mr-tz
|
||||
- data-manipulation/encryption/create-new-key-via-cryptacquirecontext chuong.dong@mandiant.com
|
||||
- host-interaction/log/clfs/append-data-to-clfs-log-container blaine.stancill@mandiant.com
|
||||
- host-interaction/log/clfs/read-data-from-clfs-log-container blaine.stancill@mandiant.com
|
||||
- data-manipulation/encryption/hc-128/encrypt-data-using-hc-128-via-wolfssl blaine.stancill@mandiant.com
|
||||
- c2/shell/create-unix-reverse-shell joakim@intezer.com
|
||||
- c2/shell/execute-shell-command-received-from-socket joakim@intezer.com
|
||||
- collection/get-current-user joakim@intezer.com
|
||||
- host-interaction/file-system/change-file-permission joakim@intezer.com
|
||||
- host-interaction/hardware/memory/get-memory-information joakim@intezer.com
|
||||
- host-interaction/mutex/lock-file joakim@intezer.com
|
||||
- host-interaction/os/version/get-kernel-version joakim@intezer.com
|
||||
- host-interaction/os/version/get-linux-distribution joakim@intezer.com
|
||||
- host-interaction/process/terminate/terminate-process-via-kill joakim@intezer.com
|
||||
- lib/duplicate-stdin-and-stdout joakim@intezer.com
|
||||
- nursery/capture-network-configuration-via-ifconfig joakim@intezeer.com
|
||||
- nursery/collect-ssh-keys joakim@intezer.com
|
||||
- nursery/enumerate-processes-via-procfs joakim@intezer.com
|
||||
- nursery/interact-with-iptables joakim@intezer.com
|
||||
- persistence/persist-via-desktop-autostart joakim@intezer.com
|
||||
- persistence/persist-via-shell-profile-or-rc-file joakim@intezer.com
|
||||
- persistence/service/persist-via-rc-script joakim@intezer.com
|
||||
- collection/get-current-user-on-linux joakim@intezer.com
|
||||
- collection/network/get-mac-address-on-windows moritz.raabe@mandiant.com
|
||||
- host-interaction/file-system/read/read-file-on-linux moritz.raabe@mandiant.com joakim@intezer.com
|
||||
- host-interaction/file-system/read/read-file-on-windows moritz.raabe@mandiant.com
|
||||
- host-interaction/file-system/write/write-file-on-windows william.ballenthin@mandiant.com
|
||||
- host-interaction/os/info/get-system-information-on-windows moritz.raabe@mandiant.com joakim@intezer.com
|
||||
- host-interaction/process/create/create-process-on-windows moritz.raabe@mandiant.com
|
||||
- linking/runtime-linking/link-function-at-runtime-on-windows moritz.raabe@mandiant.com
|
||||
- nursery/create-process-on-linux joakim@intezer.com
|
||||
- nursery/enumerate-files-on-linux william.ballenthin@mandiant.com
|
||||
- nursery/get-mac-address-on-linux joakim@intezer.com
|
||||
- nursery/get-system-information-on-linux joakim@intezer.com
|
||||
- nursery/link-function-at-runtime-on-linux joakim@intezer.com
|
||||
- nursery/write-file-on-linux joakim@intezer.com
|
||||
- communication/socket/tcp/send/obtain-transmitpackets-callback-function-via-wsaioctl jonathan.lepore@mandiant.com
|
||||
- nursery/linked-against-cpp-http-library @mr-tz
|
||||
- nursery/linked-against-cpp-json-library @mr-tz
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- main: fix `KeyError: 0` when reporting results @williballehtin #703
|
||||
- main: fix potential false negatives due to namespaces across scopes @williballenthin #721
|
||||
- linter: suppress some warnings about imports from ntdll/ntoskrnl @williballenthin #743
|
||||
- linter: suppress some warnings about missing examples in the nursery @williballenthin #747
|
||||
|
||||
### capa explorer IDA Pro plugin
|
||||
|
||||
- explorer: add additional filter logic when displaying matches by function #686 @mike-hunhoff
|
||||
- explorer: remove duplicate check when saving file #687 @mike-hunhoff
|
||||
- explorer: update IDA extractor to use non-canon mnemonics #688 @mike-hunhoff
|
||||
- explorer: allow user to add specified number of bytes when adding a Bytes feature in the Rule Generator #689 @mike-hunhoff
|
||||
- explorer: enforce max column width Features and Editor panes #691 @mike-hunhoff
|
||||
- explorer: add option to limit features to currently selected disassembly address #692 @mike-hunhoff
|
||||
- explorer: update support documentation and runtime checks #741 @mike-hunhoff
|
||||
- explorer: small performance boost to rule generator search functionality #742 @mike-hunhoff
|
||||
- explorer: add support for arch, os, and format features #758 @mike-hunhoff
|
||||
- explorer: improve parsing algorithm for rule generator feature editor #768 @mike-hunhoff
|
||||
|
||||
### Development
|
||||
|
||||
### Raw diffs
|
||||
- [capa v2.0.0...v3.0.0](https://github.com/mandiant/capa/compare/v2.0.0...v3.0.0)
|
||||
- [capa-rules v2.0.0...v3.0.0](https://github.com/mandiant/capa-rules/compare/v2.0.0...v3.0.0)
|
||||
|
||||
|
||||
## v2.0.0 (2021-07-19)
|
||||
|
||||
We are excited to announce version 2.0! :tada:
|
||||
capa 2.0:
|
||||
- enables anyone to contribute rules more easily
|
||||
- is the first Python 3 ONLY version
|
||||
- provides more concise and relevant result via identification of library functions using FLIRT
|
||||
|
||||
- includes many features and enhancements for the capa explorer IDA plugin
|
||||
- adds 93 new rules, including all new techniques introduced in MITRE ATT&CK v9
|
||||
|
||||
A huge thanks to everyone who submitted issues, provided feedback, and contributed code and rules. Many colleagues across dozens of organizations have volunteered their experience to improve this tool! :heart:
|
||||
|
||||
|
||||
### New Features
|
||||
|
||||
- rules: update ATT&CK and MBC mappings https://github.com/mandiant/capa-rules/pull/317 @williballenthin
|
||||
- main: use FLIRT signatures to identify and ignore library code #446 @williballenthin
|
||||
- tests: update test cases and caching #545 @mr-tz
|
||||
- scripts: capa2yara.py convert capa rules to YARA rules #561 @ruppde
|
||||
- rule: add file-scope feature (`function-name`) for recognized library functions #567 @williballenthin
|
||||
- main: auto detect shellcode based on file extension #516 @mr-tz
|
||||
- main: more detailed progress bar output when matching functions #562 @mr-tz
|
||||
- main: detect file limitations without doing code analysis for better performance #583 @williballenthin
|
||||
- show-features: don't show features from library functions #569 @williballenthin
|
||||
- linter: summarize results at the end #571 @williballenthin
|
||||
- linter: check for `or` with always true child statement, e.g. `optional`, colors #348 @mr-tz
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- py3: drop Python 2 support #480 @Ana06
|
||||
- meta: added `library_functions` field, `feature_counts.functions` does not include library functions any more #562 @mr-tz
|
||||
- json: results document now contains parsed ATT&CK and MBC fields instead of canonical representation #526 @mr-tz
|
||||
- json: record all matching strings for regex #159 @williballenthin
|
||||
- main: implement file limitations via rules not code #390 @williballenthin
|
||||
- json: correctly render negative offsets #619 @williballenthin
|
||||
- library: remove logic from `__init__.py` throughout #622 @williballenthin
|
||||
|
||||
### New Rules (93)
|
||||
|
||||
- anti-analysis/packer/amber/packed-with-amber @gormaniac
|
||||
- collection/file-managers/gather-3d-ftp-information @re-fox
|
||||
- collection/file-managers/gather-alftp-information @re-fox
|
||||
- collection/file-managers/gather-bitkinex-information @re-fox
|
||||
- collection/file-managers/gather-blazeftp-information @re-fox
|
||||
- collection/file-managers/gather-bulletproof-ftp-information @re-fox
|
||||
- collection/file-managers/gather-classicftp-information @re-fox
|
||||
- collection/file-managers/gather-coreftp-information @re-fox
|
||||
- collection/file-managers/gather-cuteftp-information @re-fox
|
||||
- collection/file-managers/gather-cyberduck-information @re-fox
|
||||
- collection/file-managers/gather-direct-ftp-information @re-fox
|
||||
- collection/file-managers/gather-directory-opus-information @re-fox
|
||||
- collection/file-managers/gather-expandrive-information @re-fox
|
||||
- collection/file-managers/gather-faststone-browser-information @re-fox
|
||||
- collection/file-managers/gather-fasttrack-ftp-information @re-fox
|
||||
- collection/file-managers/gather-ffftp-information @re-fox
|
||||
- collection/file-managers/gather-filezilla-information @re-fox
|
||||
- collection/file-managers/gather-flashfxp-information @re-fox
|
||||
- collection/file-managers/gather-fling-ftp-information @re-fox
|
||||
- collection/file-managers/gather-freshftp-information @re-fox
|
||||
- collection/file-managers/gather-frigate3-information @re-fox
|
||||
- collection/file-managers/gather-ftp-commander-information @re-fox
|
||||
- collection/file-managers/gather-ftp-explorer-information @re-fox
|
||||
- collection/file-managers/gather-ftp-voyager-information @re-fox
|
||||
- collection/file-managers/gather-ftpgetter-information @re-fox
|
||||
- collection/file-managers/gather-ftpinfo-information @re-fox
|
||||
- collection/file-managers/gather-ftpnow-information @re-fox
|
||||
- collection/file-managers/gather-ftprush-information @re-fox
|
||||
- collection/file-managers/gather-ftpshell-information @re-fox
|
||||
- collection/file-managers/gather-global-downloader-information @re-fox
|
||||
- collection/file-managers/gather-goftp-information @re-fox
|
||||
- collection/file-managers/gather-leapftp-information @re-fox
|
||||
- collection/file-managers/gather-netdrive-information @re-fox
|
||||
- collection/file-managers/gather-nexusfile-information @re-fox
|
||||
- collection/file-managers/gather-nova-ftp-information @re-fox
|
||||
- collection/file-managers/gather-robo-ftp-information @re-fox
|
||||
- collection/file-managers/gather-securefx-information @re-fox
|
||||
- collection/file-managers/gather-smart-ftp-information @re-fox
|
||||
- collection/file-managers/gather-softx-ftp-information @re-fox
|
||||
- collection/file-managers/gather-southriver-webdrive-information @re-fox
|
||||
- collection/file-managers/gather-staff-ftp-information @re-fox
|
||||
- collection/file-managers/gather-total-commander-information @re-fox
|
||||
- collection/file-managers/gather-turbo-ftp-information @re-fox
|
||||
- collection/file-managers/gather-ultrafxp-information @re-fox
|
||||
- collection/file-managers/gather-winscp-information @re-fox
|
||||
- collection/file-managers/gather-winzip-information @re-fox
|
||||
- collection/file-managers/gather-wise-ftp-information @re-fox
|
||||
- collection/file-managers/gather-ws-ftp-information @re-fox
|
||||
- collection/file-managers/gather-xftp-information @re-fox
|
||||
- data-manipulation/compression/decompress-data-using-aplib @r3c0nst @mr-tz
|
||||
- host-interaction/bootloader/disable-code-signing @williballenthin
|
||||
- host-interaction/bootloader/manipulate-boot-configuration @williballenthin
|
||||
- host-interaction/driver/disable-driver-code-integrity @williballenthin
|
||||
- host-interaction/file-system/bypass-mark-of-the-web @williballenthin
|
||||
- host-interaction/network/domain/get-domain-information @recvfrom
|
||||
- host-interaction/session/get-logon-sessions @recvfrom
|
||||
- linking/runtime-linking/resolve-function-by-fin8-fasthash @r3c0nst @mr-tz
|
||||
- nursery/build-docker-image @williballenthin
|
||||
- nursery/create-container @williballenthin
|
||||
- nursery/encrypt-data-using-fakem-cipher @mike-hunhoff
|
||||
- nursery/list-containers @williballenthin
|
||||
- nursery/run-in-container @williballenthin
|
||||
- persistence/registry/appinitdlls/disable-appinit_dlls-code-signature-enforcement @williballenthin
|
||||
- collection/password-manager/steal-keepass-passwords-using-keefarce @Ana06
|
||||
- host-interaction/network/connectivity/check-internet-connectivity-via-wininet matthew.williams@mandiant.com michael.hunhoff@mandiant.com
|
||||
- nursery/create-bits-job @mr-tz
|
||||
- nursery/execute-syscall-instruction @kulinacs @mr-tz
|
||||
- nursery/connect-to-wmi-namespace-via-wbemlocator michael.hunhoff@mandiant.com
|
||||
- anti-analysis/obfuscation/obfuscated-with-callobfuscator johnk3r
|
||||
- executable/installer/inno-setup/packaged-as-an-inno-setup-installer awillia2@cisco.com
|
||||
- data-manipulation/hashing/djb2/hash-data-using-djb2 awillia2@cisco.com
|
||||
- data-manipulation/encoding/base64/decode-data-using-base64-via-dword-translation-table gilbert.elliot@mandiant.com
|
||||
- nursery/list-tcp-connections-and-listeners michael.hunhoff@mandiant.com
|
||||
- nursery/list-udp-connections-and-listeners michael.hunhoff@mandiant.com
|
||||
- nursery/log-keystrokes-via-raw-input-data michael.hunhoff@mandiant.com
|
||||
- nursery/register-http-server-url michael.hunhoff@mandiant.com
|
||||
- internal/limitation/file/internal-autoit-file-limitation.yml william.ballenthin@mandiant.com
|
||||
- internal/limitation/file/internal-dotnet-file-limitation.yml william.ballenthin@mandiant.com
|
||||
- internal/limitation/file/internal-installer-file-limitation.yml william.ballenthin@mandiant.com
|
||||
- internal/limitation/file/internal-packer-file-limitation.yml william.ballenthin@mandiant.com
|
||||
- host-interaction/network/domain/enumerate-domain-computers-via-ldap awillia2@cisco.com
|
||||
- host-interaction/network/domain/get-domain-controller-name awillia2@cisco.com
|
||||
- internal/limitation/file/internal-visual-basic-file-limitation @mr-tz
|
||||
- data-manipulation/hashing/md5/hash-data-with-md5 moritz.raabe@mandiant.com
|
||||
- compiler/autohotkey/compiled-with-autohotkey awillia2@cisco.com
|
||||
- internal/limitation/file/internal-autohotkey-file-limitation @mr-tz
|
||||
- host-interaction/process/dump/create-process-memory-minidump michael.hunhoff@mandiant.com
|
||||
- nursery/get-storage-device-properties michael.hunhoff@mandiant.com
|
||||
- nursery/execute-shell-command-via-windows-remote-management michael.hunhoff@mandiant.com
|
||||
- nursery/get-token-privileges michael.hunhoff@mandiant.com
|
||||
- nursery/prompt-user-for-credentials michael.hunhoff@mandiant.com
|
||||
- nursery/spoof-parent-pid michael.hunhoff@mandiant.com
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- build: use Python 3.8 for PyInstaller to support consistently running across multiple operating systems including Windows 7 #505 @mr-tz
|
||||
- main: correctly match BB-scope matches at file scope #605 @williballenthin
|
||||
- main: do not process non-PE files even when --format explicitly provided #664 @mr-tz
|
||||
|
||||
### capa explorer IDA Pro plugin
|
||||
- explorer: IDA 7.6 support #497 @williballenthin
|
||||
- explorer: explain how to install IDA 7.6 patch to enable the plugin #528 @williballenthin
|
||||
- explorer: document IDA 7.6sp1 as alternative to the patch #536 @Ana06
|
||||
- explorer: add support for function-name feature #618 @mike-hunhoff
|
||||
- explorer: circular import workaround #654 @mike-hunhoff
|
||||
- explorer: add argument to control whether to automatically analyze when running capa explorer #548 @Ana06
|
||||
- explorer: extract API features via function names recognized by IDA/FLIRT #661 @mr-tz
|
||||
|
||||
### Development
|
||||
|
||||
- ci: add capa release link to capa-rules tag #517 @Ana06
|
||||
- ci, changelog: update `New Rules` section in CHANGELOG automatically https://github.com/mandiant/capa-rules/pull/374 #549 #604 @Ana06
|
||||
- ci, changelog: support multiple author in sync GH https://github.com/mandiant/capa-rules/pull/378 @Ana06
|
||||
- ci, lint: check statements for single child statements #563 @mr-tz
|
||||
- ci: reject PRs without CHANGELOG update to ensure CHANGELOG is kept up-to-date #584 @Ana06
|
||||
- ci: test that scripts run #660 @mr-tz
|
||||
|
||||
### Raw diffs
|
||||
|
||||
<!-- The diff uses v1.6.1 because master doesn't include v1.6.2 and v1.6.3 -->
|
||||
- [capa v1.6.1...v2.0.0](https://github.com/mandiant/capa/compare/v1.6.1...v2.0.0)
|
||||
- [capa-rules v1.6.1...v2.0.0](https://github.com/mandiant/capa-rules/compare/v1.6.1...v2.0.0)
|
||||
|
||||
|
||||
## v1.6.3 (2021-04-29)
|
||||
|
||||
This release adds IDA 7.6 support to capa.
|
||||
@@ -601,7 +10,7 @@ This release adds IDA 7.6 support to capa.
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.6.2...v1.6.3](https://github.com/mandiant/capa/compare/v1.6.2...v1.6.3)
|
||||
- [capa v1.6.2...v1.6.3](https://github.com/fireeye/capa/compare/v1.6.2...v1.6.3)
|
||||
|
||||
|
||||
## v1.6.2 (2021-04-13)
|
||||
@@ -614,7 +23,7 @@ This release backports a fix to capa 1.6: The Windows binary was built with Pyth
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.6.1...v1.6.2](https://github.com/mandiant/capa/compare/v1.6.1...v1.6.2)
|
||||
- [capa v1.6.1...v1.6.2](https://github.com/fireeye/capa/compare/v1.6.1...v1.6.2)
|
||||
|
||||
|
||||
## v1.6.1 (2021-04-07)
|
||||
@@ -682,8 +91,8 @@ This release includes several bug fixes, such as a vivisect issue that prevented
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.6.0...v1.6.1](https://github.com/mandiant/capa/compare/v1.6.0...v1.6.1)
|
||||
- [capa-rules v1.6.0...v1.6.1](https://github.com/mandiant/capa-rules/compare/v1.6.0...v1.6.1)
|
||||
- [capa v1.6.0...v1.6.1](https://github.com/fireeye/capa/compare/v1.6.0...v1.6.1)
|
||||
- [capa-rules v1.6.0...v1.6.1](https://github.com/fireeye/capa-rules/compare/v1.6.0...v1.6.1)
|
||||
|
||||
|
||||
## v1.6.0 (2021-03-09)
|
||||
@@ -692,7 +101,7 @@ This release adds the capa explorer rule generator plugin for IDA Pro, vivisect
|
||||
|
||||
### Rule Generator IDA Plugin
|
||||
|
||||
The capa explorer IDA plugin now helps you quickly build new capa rules using features extracted directly from your IDA database. Without leaving the plugin interface you can use the features extracted by capa explorer to develop and test new rules and save your work directly to your capa rules directory. To get started select the new `Rule Generator` tab, navigate to a function in the IDA `Disassembly` view, and click `Analyze`. For more information check out the capa explorer [readme](https://github.com/mandiant/capa/blob/master/capa/ida/plugin/README.md).
|
||||
The capa explorer IDA plugin now helps you quickly build new capa rules using features extracted directly from your IDA database. Without leaving the plugin interface you can use the features extracted by capa explorer to develop and test new rules and save your work directly to your capa rules directory. To get started select the new `Rule Generator` tab, navigate to a function in the IDA `Disassembly` view, and click `Analyze`. For more information check out the capa explorer [readme](https://github.com/fireeye/capa/blob/master/capa/ida/plugin/README.md).
|
||||
|
||||
|
||||
|
||||
@@ -754,8 +163,8 @@ If you have workflows that rely on the Python 2 version and need future maintena
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.5.1...v1.6.0](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
|
||||
- [capa-rules v1.5.1...v1.6.0](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
|
||||
- [capa v1.5.1...v1.6.0](https://github.com/fireeye/capa/compare/v1.5.1...v1.6.0)
|
||||
- [capa-rules v1.5.1...v1.6.0](https://github.com/fireeye/capa-rules/compare/v1.5.1...v1.6.0)
|
||||
|
||||
|
||||
## v1.5.1 (2021-02-09)
|
||||
@@ -768,8 +177,8 @@ This release fixes the version number that we forgot to update for v1.5.0 (there
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.5.0...v1.5.1](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
|
||||
- [capa-rules v1.5.0...v1.5.1](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
|
||||
- [capa v1.5.0...v1.5.1](https://github.com/fireeye/capa/compare/v1.5.1...v1.6.0)
|
||||
- [capa-rules v1.5.0...v1.5.1](https://github.com/fireeye/capa-rules/compare/v1.5.1...v1.6.0)
|
||||
|
||||
|
||||
## v1.5.0 (2021-02-05)
|
||||
@@ -784,7 +193,7 @@ This release brings support for running capa under Python 3 via [SMDA](https://g
|
||||
|
||||
@dzbeck also added [Malware Behavior Catalog](https://github.com/MBCProject/mbc-markdown) (MBC) and ATT&CK mappings for many rules.
|
||||
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
|
||||
|
||||
|
||||
### New Features
|
||||
@@ -867,8 +276,8 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.4.1...v1.5.0](https://github.com/mandiant/capa/compare/v1.4.1...v1.5.0)
|
||||
- [capa-rules v1.4.0...v1.5.0](https://github.com/mandiant/capa-rules/compare/v1.4.0...v1.5.0)
|
||||
- [capa v1.4.1...v1.5.0](https://github.com/fireeye/capa/compare/v1.4.1...v1.5.0)
|
||||
- [capa-rules v1.4.0...v1.5.0](https://github.com/fireeye/capa-rules/compare/v1.4.0...v1.5.0)
|
||||
|
||||
## v1.4.1 (2020-10-23)
|
||||
|
||||
@@ -880,8 +289,8 @@ This release fixes an issue building capa on our CI server, which prevented us f
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.4.0...v1.4.1](https://github.com/mandiant/capa/compare/v1.4.0...v1.4.1)
|
||||
- [capa-rules v1.4.0...v1.4.1](https://github.com/mandiant/capa-rules/compare/v1.4.0...v1.4.1)
|
||||
- [capa v1.4.0...v1.4.1](https://github.com/fireeye/capa/compare/v1.4.0...v1.4.1)
|
||||
- [capa-rules v1.4.0...v1.4.1](https://github.com/fireeye/capa-rules/compare/v1.4.0...v1.4.1)
|
||||
|
||||
## v1.4.0 (2020-10-23)
|
||||
|
||||
@@ -892,7 +301,7 @@ This capa release includes changes to the rule parsing, enhanced feature extract
|
||||
|
||||
@dzbeck added [Malware Behavior Catalog](https://github.com/MBCProject/mbc-markdown) (MBC) and ATT&CK mappings for 86 rules.
|
||||
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
|
||||
|
||||
### New features
|
||||
|
||||
@@ -995,8 +404,8 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.3.0...v1.4.0](https://github.com/mandiant/capa/compare/v1.3.0...v1.4.0)
|
||||
- [capa-rules v1.3.0...v1.4.0](https://github.com/mandiant/capa-rules/compare/v1.3.0...v1.4.0)
|
||||
- [capa v1.3.0...v1.4.0](https://github.com/fireeye/capa/compare/v1.3.0...v1.4.0)
|
||||
- [capa-rules v1.3.0...v1.4.0](https://github.com/fireeye/capa-rules/compare/v1.3.0...v1.4.0)
|
||||
|
||||
## v1.3.0 (2020-09-14)
|
||||
|
||||
@@ -1010,7 +419,7 @@ This release brings newly updated mappings to the [Malware Behavior Catalog vers
|
||||
- @weslambert
|
||||
- @stevemk14ebr
|
||||
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
|
||||
|
||||
### Key changes to IDA Plugin
|
||||
|
||||
@@ -1020,9 +429,9 @@ The IDA Pro integration is now distributed as a real plugin, instead of a script
|
||||
- updates distributed PyPI/`pip install --upgrade` without touching your `%IDADIR%`
|
||||
- generally doing thing the "right way"
|
||||
|
||||
How to get this new version? Its easy: download [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory and update your capa installation (incidentally, this is a good opportunity to migrate to `pip install flare-capa` instead of git checkouts). Now you should see the plugin listed in the `Edit > Plugins > FLARE capa explorer` menu in IDA.
|
||||
How to get this new version? Its easy: download [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory and update your capa installation (incidentally, this is a good opportunity to migrate to `pip install flare-capa` instead of git checkouts). Now you should see the plugin listed in the `Edit > Plugins > FLARE capa explorer` menu in IDA.
|
||||
|
||||
Please refer to the plugin [readme](https://github.com/mandiant/capa/blob/master/capa/ida/plugin/README.md) for additional information on installing and using the IDA Pro plugin.
|
||||
Please refer to the plugin [readme](https://github.com/fireeye/capa/blob/master/capa/ida/plugin/README.md) for additional information on installing and using the IDA Pro plugin.
|
||||
|
||||
Please open an issue in this repository if you notice anything weird.
|
||||
|
||||
@@ -1067,8 +476,8 @@ Please open an issue in this repository if you notice anything weird.
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.2.0...v1.3.0](https://github.com/mandiant/capa/compare/v1.2.0...v1.3.0)
|
||||
- [capa-rules v1.2.0...v1.3.0](https://github.com/mandiant/capa-rules/compare/v1.2.0...v1.3.0)
|
||||
- [capa v1.2.0...v1.3.0](https://github.com/fireeye/capa/compare/v1.2.0...v1.3.0)
|
||||
- [capa-rules v1.2.0...v1.3.0](https://github.com/fireeye/capa-rules/compare/v1.2.0...v1.3.0)
|
||||
|
||||
## v1.2.0 (2020-08-31)
|
||||
|
||||
@@ -1084,9 +493,9 @@ We received contributions from ten reverse engineers, including five new ones:
|
||||
- @edeca
|
||||
- @winniepe
|
||||
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/).
|
||||
Report issues on our [issue tracker](https://github.com/mandiant/capa/issues)
|
||||
and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/).
|
||||
Report issues on our [issue tracker](https://github.com/fireeye/capa/issues)
|
||||
and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
|
||||
|
||||
### New features
|
||||
|
||||
@@ -1165,8 +574,8 @@ and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.1.0...v1.2.0](https://github.com/mandiant/capa/compare/v1.1.0...v1.2.0)
|
||||
- [capa-rules v1.1.0...v1.2.0](https://github.com/mandiant/capa-rules/compare/v1.1.0...v1.2.0)
|
||||
- [capa v1.1.0...v1.2.0](https://github.com/fireeye/capa/compare/v1.1.0...v1.2.0)
|
||||
- [capa-rules v1.1.0...v1.2.0](https://github.com/fireeye/capa-rules/compare/v1.1.0...v1.2.0)
|
||||
|
||||
## v1.1.0 (2020-08-05)
|
||||
|
||||
@@ -1179,7 +588,7 @@ We received contributions from eight reverse engineers, including four new ones:
|
||||
- @bitsofbinary
|
||||
- @threathive
|
||||
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
|
||||
Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/fireeye/capa/). Report issues on our [issue tracker](https://github.com/fireeye/capa/issues) and contribute new rules at [capa-rules](https://github.com/fireeye/capa-rules/).
|
||||
|
||||
### New features
|
||||
|
||||
@@ -1252,5 +661,5 @@ Download a standalone binary below and checkout the readme [here on GitHub](http
|
||||
|
||||
### Raw diffs
|
||||
|
||||
- [capa v1.0.0...v1.1.0](https://github.com/mandiant/capa/compare/v1.0.0...v1.1.0)
|
||||
- [capa-rules v1.0.0...v1.1.0](https://github.com/mandiant/capa-rules/compare/v1.0.0...v1.1.0)
|
||||
- [capa v1.0.0...v1.1.0](https://github.com/fireeye/capa/compare/v1.0.0...v1.1.0)
|
||||
- [capa-rules v1.0.0...v1.1.0](https://github.com/fireeye/capa-rules/compare/v1.0.0...v1.1.0)
|
||||
|
||||
@@ -187,7 +187,7 @@
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright (C) 2020 Mandiant, Inc.
|
||||
Copyright (C) 2020 FireEye, Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
103
README.md
103
README.md
@@ -1,21 +1,17 @@
|
||||
|
||||
|
||||
|
||||
[](https://pypi.org/project/flare-capa)
|
||||
[](https://github.com/mandiant/capa/releases)
|
||||
[](https://github.com/mandiant/capa-rules)
|
||||
[](https://github.com/mandiant/capa/actions?query=workflow%3ACI+event%3Apush+branch%3Amaster)
|
||||
[](https://github.com/mandiant/capa/releases)
|
||||
[](https://github.com/fireeye/capa/releases)
|
||||
[](https://github.com/fireeye/capa-rules)
|
||||
[](https://github.com/fireeye/capa/actions?query=workflow%3ACI+event%3Apush+branch%3Amaster)
|
||||
[](https://github.com/fireeye/capa/releases)
|
||||
[](LICENSE.txt)
|
||||
|
||||
capa detects capabilities in executable files.
|
||||
You run it against a PE, ELF, .NET module, or shellcode file and it tells you what it thinks the program can do.
|
||||
You run it against a PE file or shellcode and it tells you what it thinks the program can do.
|
||||
For example, it might suggest that the file is a backdoor, is capable of installing services, or relies on HTTP to communicate.
|
||||
|
||||
Check out:
|
||||
- the overview in our first [capa blog post](https://www.mandiant.com/resources/capa-automatically-identify-malware-capabilities)
|
||||
- the major version 2.0 updates described in our [second blog post](https://www.mandiant.com/resources/capa-2-better-stronger-faster)
|
||||
- the major version 3.0 (ELF support) described in the [third blog post](https://www.mandiant.com/resources/elfant-in-the-room-capa-v3)
|
||||
- the major version 4.0 (.NET support) described in the [fourth blog post](https://www.mandiant.com/resources/blog/capa-v4-casting-wider-net)
|
||||
Check out the overview in our first [capa blog post](https://www.fireeye.com/blog/threat-research/2020/07/capa-automatically-identify-malware-capabilities.html).
|
||||
|
||||
```
|
||||
$ capa.exe suspicious.exe
|
||||
@@ -67,11 +63,18 @@ $ capa.exe suspicious.exe
|
||||
|
||||
# download and usage
|
||||
|
||||
Download stable releases of the standalone capa binaries [here](https://github.com/mandiant/capa/releases). You can run the standalone binaries without installation. capa is a command line tool that should be run from the terminal.
|
||||
Download stable releases of the standalone capa binaries [here](https://github.com/fireeye/capa/releases). You can run the standalone binaries without installation. capa is a command line tool that should be run from the terminal.
|
||||
|
||||
To use capa as a library or integrate with another tool, see [doc/installation.md](https://github.com/mandiant/capa/blob/master/doc/installation.md) for further setup instructions.
|
||||
<!--
|
||||
Alternatively, you can fetch a nightly build of a standalone binary from one of the following links. These are built using the latest development branch.
|
||||
- Windows 64bit: TODO
|
||||
- Linux: TODO
|
||||
- OSX: TODO
|
||||
-->
|
||||
|
||||
For more information about how to use capa, see [doc/usage.md](https://github.com/mandiant/capa/blob/master/doc/usage.md).
|
||||
To use capa as a library or integrate with another tool, see [doc/installation.md](doc/installation.md) for further setup instructions.
|
||||
|
||||
For more information about how to use capa, see [doc/usage.md](doc/usage.md).
|
||||
|
||||
# example
|
||||
|
||||
@@ -88,40 +91,31 @@ This is useful for at least two reasons:
|
||||
- it shows where within the binary an experienced analyst might study with IDA Pro
|
||||
|
||||
```
|
||||
$ capa.exe suspicious.exe -vv
|
||||
λ capa.exe suspicious.exe -vv
|
||||
...
|
||||
execute shell command and capture output
|
||||
namespace c2/shell
|
||||
author matthew.williams@mandiant.com
|
||||
author matthew.williams@fireeye.com
|
||||
scope function
|
||||
att&ck Execution::Command and Scripting Interpreter::Windows Command Shell [T1059.003]
|
||||
references https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/ns-processthreadsapi-startupinfoa
|
||||
function @ 0x4011C0
|
||||
examples Practical Malware Analysis Lab 14-02.exe_:0x4011C0
|
||||
function @ 0x10003A13
|
||||
and:
|
||||
match: create a process with modified I/O handles and window @ 0x4011C0
|
||||
match: create a process with modified I/O handles and window @ 0x10003A13
|
||||
and:
|
||||
number: 257 = STARTF_USESTDHANDLES | STARTF_USESHOWWINDOW @ 0x4012B8
|
||||
or:
|
||||
number: 68 = StartupInfo.cb (size) @ 0x401282
|
||||
or: = API functions that accept a pointer to a STARTUPINFO structure
|
||||
api: kernel32.CreateProcess @ 0x401343
|
||||
match: create pipe @ 0x4011C0
|
||||
api: kernel32.CreateProcess @ 0x10003D6D
|
||||
number: 0x101 @ 0x10003B03
|
||||
or:
|
||||
number: 0x44 @ 0x10003ADC
|
||||
optional:
|
||||
api: kernel32.GetStartupInfo @ 0x10003AE4
|
||||
match: create pipe @ 0x10003A13
|
||||
or:
|
||||
api: kernel32.CreatePipe @ 0x40126F, 0x401280
|
||||
optional:
|
||||
match: create thread @ 0x40136A, 0x4013BA
|
||||
or:
|
||||
and:
|
||||
os: windows
|
||||
or:
|
||||
api: kernel32.CreateThread @ 0x4013D7
|
||||
or:
|
||||
and:
|
||||
os: windows
|
||||
or:
|
||||
api: kernel32.CreateThread @ 0x401395
|
||||
api: kernel32.CreatePipe @ 0x10003ACB
|
||||
or:
|
||||
string: "cmd.exe" @ 0x4012FD
|
||||
string: cmd.exe /c @ 0x10003AED
|
||||
...
|
||||
```
|
||||
|
||||
@@ -137,49 +131,36 @@ rule:
|
||||
meta:
|
||||
name: hash data with CRC32
|
||||
namespace: data-manipulation/checksum/crc32
|
||||
authors:
|
||||
- moritz.raabe@mandiant.com
|
||||
author: moritz.raabe@fireeye.com
|
||||
scope: function
|
||||
mbc:
|
||||
- Data::Checksum::CRC32 [C0032.001]
|
||||
examples:
|
||||
- 2D3EDC218A90F03089CC01715A9F047F:0x403CBD
|
||||
- 7D28CB106CB54876B2A5C111724A07CD:0x402350 # RtlComputeCrc32
|
||||
- 7EFF498DE13CC734262F87E6B3EF38AB:0x100084A6
|
||||
features:
|
||||
- or:
|
||||
- and:
|
||||
- mnemonic: shr
|
||||
- or:
|
||||
- number: 0xEDB88320
|
||||
- bytes: 00 00 00 00 96 30 07 77 2C 61 0E EE BA 51 09 99 19 C4 6D 07 8F F4 6A 70 35 A5 63 E9 A3 95 64 9E = crc32_tab
|
||||
- number: 0xEDB88320
|
||||
- number: 8
|
||||
- characteristic: nzxor
|
||||
- and:
|
||||
- number: 0x8320
|
||||
- number: 0xEDB8
|
||||
- characteristic: nzxor
|
||||
- api: RtlComputeCrc32
|
||||
```
|
||||
|
||||
The [github.com/mandiant/capa-rules](https://github.com/mandiant/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
|
||||
The [github.com/fireeye/capa-rules](https://github.com/fireeye/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
|
||||
Please learn to write rules and contribute new entries as you find interesting techniques in malware.
|
||||
|
||||
If you use IDA Pro, then you can use the [capa explorer](https://github.com/mandiant/capa/tree/master/capa/ida/plugin) plugin.
|
||||
If you use IDA Pro, then you can use the [capa explorer](capa/ida/plugin/) plugin.
|
||||
capa explorer helps you identify interesting areas of a program and build new capa rules using features extracted directly from your IDA Pro database.
|
||||
|
||||
|
||||
|
||||
|
||||
# further information
|
||||
## capa
|
||||
- [Installation](https://github.com/mandiant/capa/blob/master/doc/installation.md)
|
||||
- [Usage](https://github.com/mandiant/capa/blob/master/doc/usage.md)
|
||||
- [Limitations](https://github.com/mandiant/capa/blob/master/doc/limitations.md)
|
||||
- [Contributing Guide](https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md)
|
||||
- [doc/installation](doc/installation.md)
|
||||
- [doc/usage](doc/usage.md)
|
||||
- [doc/limitations](doc/limitations.md)
|
||||
- [Contributing Guide](.github/CONTRIBUTING.md)
|
||||
|
||||
## capa rules
|
||||
- [capa-rules repository](https://github.com/mandiant/capa-rules)
|
||||
- [capa-rules rule format](https://github.com/mandiant/capa-rules/blob/master/doc/format.md)
|
||||
|
||||
## capa testfiles
|
||||
The [capa-testfiles repository](https://github.com/mandiant/capa-testfiles) contains the data we use to test capa's code and rules
|
||||
- [capa-rules repository](https://github.com/fireeye/capa-rules)
|
||||
- [capa-rules rule format](https://github.com/fireeye/capa-rules/blob/master/doc/format.md)
|
||||
|
||||
300
capa/engine.py
300
capa/engine.py
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -8,29 +8,11 @@
|
||||
|
||||
import copy
|
||||
import collections
|
||||
from typing import TYPE_CHECKING, Set, Dict, List, Tuple, Mapping, Iterable
|
||||
|
||||
import capa.perf
|
||||
import capa.features.common
|
||||
from capa.features.common import Result, Feature
|
||||
from capa.features.address import Address
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# circular import, otherwise
|
||||
import capa.rules
|
||||
import capa.features
|
||||
|
||||
|
||||
# a collection of features and the locations at which they are found.
|
||||
#
|
||||
# used throughout matching as the context in which features are searched:
|
||||
# to check if a feature exists, do: `Number(0x10) in features`.
|
||||
# to collect the locations of a feature, do: `features[Number(0x10)]`
|
||||
#
|
||||
# aliased here so that the type can be documented and xref'd.
|
||||
FeatureSet = Dict[Feature, Set[Address]]
|
||||
|
||||
|
||||
class Statement:
|
||||
class Statement(object):
|
||||
"""
|
||||
superclass for structural nodes, such as and/or/not.
|
||||
this exists to provide a default impl for `__str__` and `__repr__`,
|
||||
@@ -51,12 +33,15 @@ class Statement:
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
def evaluate(self, features: FeatureSet, short_circuit=True) -> Result:
|
||||
def evaluate(self, ctx):
|
||||
"""
|
||||
classes that inherit `Statement` must implement `evaluate`
|
||||
|
||||
args:
|
||||
short_circuit (bool): if true, then statements like and/or/some may short circuit.
|
||||
ctx (defaultdict[Feature, set[VA]])
|
||||
|
||||
returns:
|
||||
Result
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@@ -65,7 +50,7 @@ class Statement:
|
||||
yield self.child
|
||||
|
||||
if hasattr(self, "children"):
|
||||
for child in getattr(self, "children"):
|
||||
for child in self.children:
|
||||
yield child
|
||||
|
||||
def replace_child(self, existing, new):
|
||||
@@ -74,76 +59,75 @@ class Statement:
|
||||
self.child = new
|
||||
|
||||
if hasattr(self, "children"):
|
||||
children = getattr(self, "children")
|
||||
for i, child in enumerate(children):
|
||||
for i, child in enumerate(self.children):
|
||||
if child is existing:
|
||||
children[i] = new
|
||||
self.children[i] = new
|
||||
|
||||
|
||||
class Result(object):
|
||||
"""
|
||||
represents the results of an evaluation of statements against features.
|
||||
|
||||
instances of this class should behave like a bool,
|
||||
e.g. `assert Result(True, ...) == True`
|
||||
|
||||
instances track additional metadata about evaluation results.
|
||||
they contain references to the statement node (e.g. an And statement),
|
||||
as well as the children Result instances.
|
||||
|
||||
we need this so that we can render the tree of expressions and their results.
|
||||
"""
|
||||
|
||||
def __init__(self, success, statement, children, locations=None):
|
||||
"""
|
||||
args:
|
||||
success (bool)
|
||||
statement (capa.engine.Statement or capa.features.Feature)
|
||||
children (list[Result])
|
||||
locations (iterable[VA])
|
||||
"""
|
||||
super(Result, self).__init__()
|
||||
self.success = success
|
||||
self.statement = statement
|
||||
self.children = children
|
||||
self.locations = locations if locations is not None else ()
|
||||
|
||||
def __eq__(self, other):
|
||||
if isinstance(other, bool):
|
||||
return self.success == other
|
||||
return False
|
||||
|
||||
def __bool__(self):
|
||||
return self.success
|
||||
|
||||
def __nonzero__(self):
|
||||
return self.success
|
||||
|
||||
|
||||
class And(Statement):
|
||||
"""
|
||||
match if all of the children evaluate to True.
|
||||
|
||||
the order of evaluation is dictated by the property
|
||||
`And.children` (type: List[Statement|Feature]).
|
||||
a query optimizer may safely manipulate the order of these children.
|
||||
"""
|
||||
"""match if all of the children evaluate to True."""
|
||||
|
||||
def __init__(self, children, description=None):
|
||||
super(And, self).__init__(description=description)
|
||||
self.children = children
|
||||
|
||||
def evaluate(self, ctx, short_circuit=True):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.and"] += 1
|
||||
|
||||
if short_circuit:
|
||||
results = []
|
||||
for child in self.children:
|
||||
result = child.evaluate(ctx, short_circuit=short_circuit)
|
||||
results.append(result)
|
||||
if not result:
|
||||
# short circuit
|
||||
return Result(False, self, results)
|
||||
|
||||
return Result(True, self, results)
|
||||
else:
|
||||
results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
|
||||
success = all(results)
|
||||
return Result(success, self, results)
|
||||
def evaluate(self, ctx):
|
||||
results = [child.evaluate(ctx) for child in self.children]
|
||||
success = all(results)
|
||||
return Result(success, self, results)
|
||||
|
||||
|
||||
class Or(Statement):
|
||||
"""
|
||||
match if any of the children evaluate to True.
|
||||
|
||||
the order of evaluation is dictated by the property
|
||||
`Or.children` (type: List[Statement|Feature]).
|
||||
a query optimizer may safely manipulate the order of these children.
|
||||
"""
|
||||
"""match if any of the children evaluate to True."""
|
||||
|
||||
def __init__(self, children, description=None):
|
||||
super(Or, self).__init__(description=description)
|
||||
self.children = children
|
||||
|
||||
def evaluate(self, ctx, short_circuit=True):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.or"] += 1
|
||||
|
||||
if short_circuit:
|
||||
results = []
|
||||
for child in self.children:
|
||||
result = child.evaluate(ctx, short_circuit=short_circuit)
|
||||
results.append(result)
|
||||
if result:
|
||||
# short circuit as soon as we hit one match
|
||||
return Result(True, self, results)
|
||||
|
||||
return Result(False, self, results)
|
||||
else:
|
||||
results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
|
||||
success = any(results)
|
||||
return Result(success, self, results)
|
||||
def evaluate(self, ctx):
|
||||
results = [child.evaluate(ctx) for child in self.children]
|
||||
success = any(results)
|
||||
return Result(success, self, results)
|
||||
|
||||
|
||||
class Not(Statement):
|
||||
@@ -153,55 +137,28 @@ class Not(Statement):
|
||||
super(Not, self).__init__(description=description)
|
||||
self.child = child
|
||||
|
||||
def evaluate(self, ctx, short_circuit=True):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.not"] += 1
|
||||
|
||||
results = [self.child.evaluate(ctx, short_circuit=short_circuit)]
|
||||
def evaluate(self, ctx):
|
||||
results = [self.child.evaluate(ctx)]
|
||||
success = not results[0]
|
||||
return Result(success, self, results)
|
||||
|
||||
|
||||
class Some(Statement):
|
||||
"""
|
||||
match if at least N of the children evaluate to True.
|
||||
|
||||
the order of evaluation is dictated by the property
|
||||
`Some.children` (type: List[Statement|Feature]).
|
||||
a query optimizer may safely manipulate the order of these children.
|
||||
"""
|
||||
"""match if at least N of the children evaluate to True."""
|
||||
|
||||
def __init__(self, count, children, description=None):
|
||||
super(Some, self).__init__(description=description)
|
||||
self.count = count
|
||||
self.children = children
|
||||
|
||||
def evaluate(self, ctx, short_circuit=True):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.some"] += 1
|
||||
|
||||
if short_circuit:
|
||||
results = []
|
||||
satisfied_children_count = 0
|
||||
for child in self.children:
|
||||
result = child.evaluate(ctx, short_circuit=short_circuit)
|
||||
results.append(result)
|
||||
if result:
|
||||
satisfied_children_count += 1
|
||||
|
||||
if satisfied_children_count >= self.count:
|
||||
# short circuit as soon as we hit the threshold
|
||||
return Result(True, self, results)
|
||||
|
||||
return Result(False, self, results)
|
||||
else:
|
||||
results = [child.evaluate(ctx, short_circuit=short_circuit) for child in self.children]
|
||||
# note that here we cast the child result as a bool
|
||||
# because we've overridden `__bool__` above.
|
||||
#
|
||||
# we can't use `if child is True` because the instance is not True.
|
||||
success = sum([1 for child in results if bool(child) is True]) >= self.count
|
||||
return Result(success, self, results)
|
||||
def evaluate(self, ctx):
|
||||
results = [child.evaluate(ctx) for child in self.children]
|
||||
# note that here we cast the child result as a bool
|
||||
# because we've overridden `__bool__` above.
|
||||
#
|
||||
# we can't use `if child is True` because the instance is not True.
|
||||
success = sum([1 for child in results if bool(child) is True]) >= self.count
|
||||
return Result(success, self, results)
|
||||
|
||||
|
||||
class Range(Statement):
|
||||
@@ -213,10 +170,7 @@ class Range(Statement):
|
||||
self.min = min if min is not None else 0
|
||||
self.max = max if max is not None else (1 << 64 - 1)
|
||||
|
||||
def evaluate(self, ctx, **kwargs):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.range"] += 1
|
||||
|
||||
def evaluate(self, ctx):
|
||||
count = len(ctx.get(self.child, []))
|
||||
if self.min == 0 and count == 0:
|
||||
return Result(True, self, [])
|
||||
@@ -236,66 +190,59 @@ class Subscope(Statement):
|
||||
the engine should preprocess rules to extract subscope statements into their own rules.
|
||||
"""
|
||||
|
||||
def __init__(self, scope, child, description=None):
|
||||
super(Subscope, self).__init__(description=description)
|
||||
def __init__(self, scope, child):
|
||||
super(Subscope, self).__init__()
|
||||
self.scope = scope
|
||||
self.child = child
|
||||
|
||||
def evaluate(self, ctx, **kwargs):
|
||||
def evaluate(self, ctx):
|
||||
raise ValueError("cannot evaluate a subscope directly!")
|
||||
|
||||
|
||||
# mapping from rule name to list of: (location of match, result object)
|
||||
#
|
||||
# used throughout matching and rendering to collection the results
|
||||
# of statement evaluation and their locations.
|
||||
#
|
||||
# to check if a rule matched, do: `"TCP client" in matches`.
|
||||
# to find where a rule matched, do: `map(first, matches["TCP client"])`
|
||||
# to see how a rule matched, do:
|
||||
#
|
||||
# for address, match_details in matches["TCP client"]:
|
||||
# inspect(match_details)
|
||||
#
|
||||
# aliased here so that the type can be documented and xref'd.
|
||||
MatchResults = Mapping[str, List[Tuple[Address, Result]]]
|
||||
|
||||
|
||||
def index_rule_matches(features: FeatureSet, rule: "capa.rules.Rule", locations: Iterable[Address]):
|
||||
def topologically_order_rules(rules):
|
||||
"""
|
||||
record into the given featureset that the given rule matched at the given locations.
|
||||
order the given rules such that dependencies show up before dependents.
|
||||
this means that as we match rules, we can add features for the matches, and these
|
||||
will be matched by subsequent rules if they follow this order.
|
||||
|
||||
naively, this is just adding a MatchedRule feature;
|
||||
however, we also want to record matches for the rule's namespaces.
|
||||
|
||||
updates `features` in-place. doesn't modify the remaining arguments.
|
||||
assumes that the rule dependency graph is a DAG.
|
||||
"""
|
||||
features[capa.features.common.MatchedRule(rule.name)].update(locations)
|
||||
namespace = rule.meta.get("namespace")
|
||||
if namespace:
|
||||
while namespace:
|
||||
features[capa.features.common.MatchedRule(namespace)].update(locations)
|
||||
namespace, _, _ = namespace.rpartition("/")
|
||||
# we evaluate `rules` multiple times, so if its a generator, realize it into a list.
|
||||
rules = list(rules)
|
||||
namespaces = capa.rules.index_rules_by_namespace(rules)
|
||||
rules = {rule.name: rule for rule in rules}
|
||||
seen = set([])
|
||||
ret = []
|
||||
|
||||
def rec(rule):
|
||||
if rule.name in seen:
|
||||
return
|
||||
|
||||
for dep in rule.get_dependencies(namespaces):
|
||||
rec(rules[dep])
|
||||
|
||||
ret.append(rule)
|
||||
seen.add(rule.name)
|
||||
|
||||
for rule in rules.values():
|
||||
rec(rule)
|
||||
|
||||
return ret
|
||||
|
||||
|
||||
def match(rules: List["capa.rules.Rule"], features: FeatureSet, addr: Address) -> Tuple[FeatureSet, MatchResults]:
|
||||
def match(rules, features, va):
|
||||
"""
|
||||
match the given rules against the given features,
|
||||
returning an updated set of features and the matches.
|
||||
Args:
|
||||
rules (List[capa.rules.Rule]): these must already be ordered topologically by dependency.
|
||||
features (Mapping[capa.features.Feature, int]):
|
||||
va (int): location of the features
|
||||
|
||||
the updated features are just like the input,
|
||||
but extended to include the match features (e.g. names of rules that matched).
|
||||
the given feature set is not modified; an updated copy is returned.
|
||||
|
||||
the given list of rules must be ordered topologically by dependency,
|
||||
or else `match` statements will not be handled correctly.
|
||||
|
||||
this routine should be fairly optimized, but is not guaranteed to be the fastest matcher possible.
|
||||
it has a particularly convenient signature: (rules, features) -> matches
|
||||
other strategies can be imagined that match differently; implement these elsewhere.
|
||||
specifically, this routine does "top down" matching of the given rules against the feature set.
|
||||
Returns:
|
||||
Tuple[List[capa.features.Feature], Dict[str, Tuple[int, capa.engine.Result]]]: two-tuple with entries:
|
||||
- list of features used for matching (which may be greater than argument, due to rule match features), and
|
||||
- mapping from rule name to (location of match, result object)
|
||||
"""
|
||||
results = collections.defaultdict(list) # type: MatchResults
|
||||
results = collections.defaultdict(list)
|
||||
|
||||
# copy features so that we can modify it
|
||||
# without affecting the caller (keep this function pure)
|
||||
@@ -304,22 +251,15 @@ def match(rules: List["capa.rules.Rule"], features: FeatureSet, addr: Address) -
|
||||
features = collections.defaultdict(set, copy.copy(features))
|
||||
|
||||
for rule in rules:
|
||||
res = rule.evaluate(features, short_circuit=True)
|
||||
res = rule.evaluate(features)
|
||||
if res:
|
||||
# we first matched the rule with short circuiting enabled.
|
||||
# this is much faster than without short circuiting.
|
||||
# however, we want to collect all results thoroughly,
|
||||
# so once we've found a match quickly,
|
||||
# go back and capture results without short circuiting.
|
||||
res = rule.evaluate(features, short_circuit=False)
|
||||
results[rule.name].append((va, res))
|
||||
features[capa.features.MatchedRule(rule.name)].add(va)
|
||||
|
||||
# sanity check
|
||||
assert bool(res) is True
|
||||
|
||||
results[rule.name].append((addr, res))
|
||||
# we need to update the current `features`
|
||||
# because subsequent iterations of this loop may use newly added features,
|
||||
# such as rule or namespace matches.
|
||||
index_rule_matches(features, rule, [addr])
|
||||
namespace = rule.meta.get("namespace")
|
||||
if namespace:
|
||||
while namespace:
|
||||
features[capa.features.MatchedRule(namespace)].add(va)
|
||||
namespace, _, _ = namespace.rpartition("/")
|
||||
|
||||
return (features, results)
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
class UnsupportedRuntimeError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class UnsupportedFormatError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class UnsupportedArchError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class UnsupportedOSError(ValueError):
|
||||
pass
|
||||
@@ -0,0 +1,236 @@
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import re
|
||||
import sys
|
||||
import codecs
|
||||
import logging
|
||||
|
||||
import capa.engine
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
MAX_BYTES_FEATURE_SIZE = 0x100
|
||||
|
||||
# thunks may be chained so we specify a delta to control the depth to which these chains are explored
|
||||
THUNK_CHAIN_DEPTH_DELTA = 5
|
||||
|
||||
# identifiers for supported architectures names that tweak a feature
|
||||
# for example, offset/x32
|
||||
ARCH_X32 = "x32"
|
||||
ARCH_X64 = "x64"
|
||||
VALID_ARCH = (ARCH_X32, ARCH_X64)
|
||||
|
||||
|
||||
def bytes_to_str(b):
|
||||
if sys.version_info[0] >= 3:
|
||||
return str(codecs.encode(b, "hex").decode("utf-8"))
|
||||
else:
|
||||
return codecs.encode(b, "hex")
|
||||
|
||||
|
||||
def hex_string(h):
|
||||
""" render hex string e.g. "0a40b1" as "0A 40 B1" """
|
||||
return " ".join(h[i : i + 2] for i in range(0, len(h), 2)).upper()
|
||||
|
||||
|
||||
def escape_string(s):
|
||||
"""escape special characters"""
|
||||
s = repr(s)
|
||||
if not s.startswith(('"', "'")):
|
||||
# u'hello\r\nworld' -> hello\\r\\nworld
|
||||
s = s[2:-1]
|
||||
else:
|
||||
# 'hello\r\nworld' -> hello\\r\\nworld
|
||||
s = s[1:-1]
|
||||
s = s.replace("\\'", "'") # repr() may escape "'" in some edge cases, remove
|
||||
s = s.replace('"', '\\"') # repr() does not escape '"', add
|
||||
return s
|
||||
|
||||
|
||||
class Feature(object):
|
||||
def __init__(self, value, arch=None, description=None):
|
||||
"""
|
||||
Args:
|
||||
value (any): the value of the feature, such as the number or string.
|
||||
arch (str): one of the VALID_ARCH values, or None.
|
||||
When None, then the feature applies to any architecture.
|
||||
Modifies the feature name from `feature` to `feature/arch`, like `offset/x32`.
|
||||
description (str): a human-readable description that explains the feature value.
|
||||
"""
|
||||
super(Feature, self).__init__()
|
||||
|
||||
if arch is not None:
|
||||
if arch not in VALID_ARCH:
|
||||
raise ValueError("arch '%s' must be one of %s" % (arch, VALID_ARCH))
|
||||
self.name = self.__class__.__name__.lower() + "/" + arch
|
||||
else:
|
||||
self.name = self.__class__.__name__.lower()
|
||||
|
||||
self.value = value
|
||||
self.arch = arch
|
||||
self.description = description
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.name, self.value, self.arch))
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.name == other.name and self.value == other.value and self.arch == other.arch
|
||||
|
||||
def get_value_str(self):
|
||||
"""
|
||||
render the value of this feature, for use by `__str__` and friends.
|
||||
subclasses should override to customize the rendering.
|
||||
|
||||
Returns: any
|
||||
"""
|
||||
return self.value
|
||||
|
||||
def __str__(self):
|
||||
if self.value is not None:
|
||||
if self.description:
|
||||
return "%s(%s = %s)" % (self.name, self.get_value_str(), self.description)
|
||||
else:
|
||||
return "%s(%s)" % (self.name, self.get_value_str())
|
||||
else:
|
||||
return "%s" % self.name
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
def evaluate(self, ctx):
|
||||
return capa.engine.Result(self in ctx, self, [], locations=ctx.get(self, []))
|
||||
|
||||
def freeze_serialize(self):
|
||||
if self.arch is not None:
|
||||
return (self.__class__.__name__, [self.value, {"arch": self.arch}])
|
||||
else:
|
||||
return (self.__class__.__name__, [self.value])
|
||||
|
||||
@classmethod
|
||||
def freeze_deserialize(cls, args):
|
||||
# as you can see below in code,
|
||||
# if the last argument is a dictionary,
|
||||
# consider it to be kwargs passed to the feature constructor.
|
||||
if len(args) == 1:
|
||||
return cls(*args)
|
||||
elif isinstance(args[-1], dict):
|
||||
kwargs = args[-1]
|
||||
args = args[:-1]
|
||||
return cls(*args, **kwargs)
|
||||
|
||||
|
||||
class MatchedRule(Feature):
|
||||
def __init__(self, value, description=None):
|
||||
super(MatchedRule, self).__init__(value, description=description)
|
||||
self.name = "match"
|
||||
|
||||
|
||||
class Characteristic(Feature):
|
||||
def __init__(self, value, description=None):
|
||||
super(Characteristic, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class String(Feature):
|
||||
def __init__(self, value, description=None):
|
||||
super(String, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class Regex(String):
|
||||
def __init__(self, value, description=None):
|
||||
super(Regex, self).__init__(value, description=description)
|
||||
pat = self.value[len("/") : -len("/")]
|
||||
flags = re.DOTALL
|
||||
if value.endswith("/i"):
|
||||
pat = self.value[len("/") : -len("/i")]
|
||||
flags |= re.IGNORECASE
|
||||
try:
|
||||
self.re = re.compile(pat, flags)
|
||||
except re.error:
|
||||
if value.endswith("/i"):
|
||||
value = value[: -len("i")]
|
||||
raise ValueError(
|
||||
"invalid regular expression: %s it should use Python syntax, try it at https://pythex.org" % value
|
||||
)
|
||||
|
||||
def evaluate(self, ctx):
|
||||
for feature, locations in ctx.items():
|
||||
if not isinstance(feature, (capa.features.String,)):
|
||||
continue
|
||||
|
||||
# `re.search` finds a match anywhere in the given string
|
||||
# which implies leading and/or trailing whitespace.
|
||||
# using this mode cleans is more convenient for rule authors,
|
||||
# so that they don't have to prefix/suffix their terms like: /.*foo.*/.
|
||||
if self.re.search(feature.value):
|
||||
# unlike other features, we cannot return put a reference to `self` directly in a `Result`.
|
||||
# this is because `self` may match on many strings, so we can't stuff the matched value into it.
|
||||
# instead, return a new instance that has a reference to both the regex and the matched value.
|
||||
# see #262.
|
||||
return capa.engine.Result(True, _MatchedRegex(self, feature.value), [], locations=locations)
|
||||
|
||||
return capa.engine.Result(False, _MatchedRegex(self, None), [])
|
||||
|
||||
def __str__(self):
|
||||
return "regex(string =~ %s)" % self.value
|
||||
|
||||
|
||||
class _MatchedRegex(Regex):
|
||||
"""
|
||||
this represents a specific instance of a regular expression feature match.
|
||||
treat it the same as a `Regex` except it has the `match` field that contains the complete string that matched.
|
||||
|
||||
note: this type should only ever be constructed by `Regex.evaluate()`. it is not part of the public API.
|
||||
"""
|
||||
|
||||
def __init__(self, regex, match):
|
||||
"""
|
||||
args:
|
||||
regex (Regex): the regex feature that matches
|
||||
match (string|None): the matching string or None if it doesn't match
|
||||
"""
|
||||
super(_MatchedRegex, self).__init__(regex.value, description=regex.description)
|
||||
# we want this to collide with the name of `Regex` above,
|
||||
# so that it works nicely with the renderers.
|
||||
self.name = "regex"
|
||||
# this may be None if the regex doesn't match
|
||||
self.match = match
|
||||
|
||||
def __str__(self):
|
||||
return 'regex(string =~ %s, matched = "%s")' % (self.value, self.match)
|
||||
|
||||
|
||||
class StringFactory(object):
|
||||
def __new__(self, value, description=None):
|
||||
if value.startswith("/") and (value.endswith("/") or value.endswith("/i")):
|
||||
return Regex(value, description=description)
|
||||
return String(value, description=description)
|
||||
|
||||
|
||||
class Bytes(Feature):
|
||||
def __init__(self, value, description=None):
|
||||
super(Bytes, self).__init__(value, description=description)
|
||||
|
||||
def evaluate(self, ctx):
|
||||
for feature, locations in ctx.items():
|
||||
if not isinstance(feature, (capa.features.Bytes,)):
|
||||
continue
|
||||
|
||||
if feature.value.startswith(self.value):
|
||||
return capa.engine.Result(True, self, [], locations=locations)
|
||||
|
||||
return capa.engine.Result(False, self, [])
|
||||
|
||||
def get_value_str(self):
|
||||
return hex_string(bytes_to_str(self.value))
|
||||
|
||||
def freeze_serialize(self):
|
||||
return (self.__class__.__name__, [bytes_to_str(self.value).upper()])
|
||||
|
||||
@classmethod
|
||||
def freeze_deserialize(cls, args):
|
||||
return cls(*[codecs.decode(x, "hex") for x in args])
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
import abc
|
||||
|
||||
from dncil.clr.token import Token
|
||||
|
||||
|
||||
class Address(abc.ABC):
|
||||
@abc.abstractmethod
|
||||
def __eq__(self, other):
|
||||
...
|
||||
|
||||
@abc.abstractmethod
|
||||
def __lt__(self, other):
|
||||
# implement < so that addresses can be sorted from low to high
|
||||
...
|
||||
|
||||
@abc.abstractmethod
|
||||
def __hash__(self):
|
||||
# implement hash so that addresses can be used in sets and dicts
|
||||
...
|
||||
|
||||
@abc.abstractmethod
|
||||
def __repr__(self):
|
||||
# implement repr to help during debugging
|
||||
...
|
||||
|
||||
|
||||
class AbsoluteVirtualAddress(int, Address):
|
||||
"""an absolute memory address"""
|
||||
|
||||
def __new__(cls, v):
|
||||
assert v >= 0
|
||||
return int.__new__(cls, v)
|
||||
|
||||
def __repr__(self):
|
||||
return f"absolute(0x{self:x})"
|
||||
|
||||
|
||||
class RelativeVirtualAddress(int, Address):
|
||||
"""a memory address relative to a base address"""
|
||||
|
||||
def __repr__(self):
|
||||
return f"relative(0x{self:x})"
|
||||
|
||||
|
||||
class FileOffsetAddress(int, Address):
|
||||
"""an address relative to the start of a file"""
|
||||
|
||||
def __new__(cls, v):
|
||||
assert v >= 0
|
||||
return int.__new__(cls, v)
|
||||
|
||||
def __repr__(self):
|
||||
return f"file(0x{self:x})"
|
||||
|
||||
|
||||
class DNTokenAddress(Address):
|
||||
"""a .NET token"""
|
||||
|
||||
def __init__(self, token: Token):
|
||||
self.token = token
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.token.value == other.token.value
|
||||
|
||||
def __lt__(self, other):
|
||||
return self.token.value < other.token.value
|
||||
|
||||
def __hash__(self):
|
||||
return hash(self.token.value)
|
||||
|
||||
def __repr__(self):
|
||||
return f"token(0x{self.token.value:x})"
|
||||
|
||||
def __index__(self):
|
||||
# returns the object converted to an integer
|
||||
return self.token.value
|
||||
|
||||
|
||||
class DNTokenOffsetAddress(Address):
|
||||
"""an offset into an object specified by a .NET token"""
|
||||
|
||||
def __init__(self, token: Token, offset: int):
|
||||
assert offset >= 0
|
||||
self.token = token
|
||||
self.offset = offset
|
||||
|
||||
def __eq__(self, other):
|
||||
return (self.token.value, self.offset) == (other.token.value, other.offset)
|
||||
|
||||
def __lt__(self, other):
|
||||
return (self.token.value, self.offset) < (other.token.value, other.offset)
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.token.value, self.offset))
|
||||
|
||||
def __repr__(self):
|
||||
return f"token(0x{self.token.value:x})+(0x{self.offset:x})"
|
||||
|
||||
def __index__(self):
|
||||
return self.token.value + self.offset
|
||||
|
||||
|
||||
class _NoAddress(Address):
|
||||
def __eq__(self, other):
|
||||
return True
|
||||
|
||||
def __lt__(self, other):
|
||||
return False
|
||||
|
||||
def __hash__(self):
|
||||
return hash(0)
|
||||
|
||||
def __repr__(self):
|
||||
return "no address"
|
||||
|
||||
|
||||
NO_ADDRESS = _NoAddress()
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,15 +6,22 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from capa.features.common import Feature
|
||||
from capa.features import Feature
|
||||
|
||||
|
||||
class BasicBlock(Feature):
|
||||
def __init__(self, description=None):
|
||||
super(BasicBlock, self).__init__(None, description=description)
|
||||
def __init__(self):
|
||||
super(BasicBlock, self).__init__(None)
|
||||
|
||||
def __str__(self):
|
||||
return "basic block"
|
||||
|
||||
def get_value_str(self):
|
||||
return ""
|
||||
|
||||
def freeze_serialize(self):
|
||||
return (self.__class__.__name__, [])
|
||||
|
||||
@classmethod
|
||||
def freeze_deserialize(cls, args):
|
||||
return cls()
|
||||
|
||||
@@ -1,431 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import re
|
||||
import abc
|
||||
import codecs
|
||||
import logging
|
||||
import collections
|
||||
from typing import TYPE_CHECKING, Set, Dict, List, Union, Optional, Sequence
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# circular import, otherwise
|
||||
import capa.engine
|
||||
|
||||
import capa.perf
|
||||
import capa.features
|
||||
import capa.features.extractors.elf
|
||||
from capa.features.address import Address
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
MAX_BYTES_FEATURE_SIZE = 0x100
|
||||
|
||||
# thunks may be chained so we specify a delta to control the depth to which these chains are explored
|
||||
THUNK_CHAIN_DEPTH_DELTA = 5
|
||||
|
||||
|
||||
def bytes_to_str(b: bytes) -> str:
|
||||
return str(codecs.encode(b, "hex").decode("utf-8"))
|
||||
|
||||
|
||||
def hex_string(h: str) -> str:
|
||||
"""render hex string e.g. "0a40b1" as "0A 40 B1" """
|
||||
return " ".join(h[i : i + 2] for i in range(0, len(h), 2)).upper()
|
||||
|
||||
|
||||
def escape_string(s: str) -> str:
|
||||
"""escape special characters"""
|
||||
s = repr(s)
|
||||
if not s.startswith(('"', "'")):
|
||||
# u'hello\r\nworld' -> hello\\r\\nworld
|
||||
s = s[2:-1]
|
||||
else:
|
||||
# 'hello\r\nworld' -> hello\\r\\nworld
|
||||
s = s[1:-1]
|
||||
s = s.replace("\\'", "'") # repr() may escape "'" in some edge cases, remove
|
||||
s = s.replace('"', '\\"') # repr() does not escape '"', add
|
||||
return s
|
||||
|
||||
|
||||
class Result:
|
||||
"""
|
||||
represents the results of an evaluation of statements against features.
|
||||
|
||||
instances of this class should behave like a bool,
|
||||
e.g. `assert Result(True, ...) == True`
|
||||
|
||||
instances track additional metadata about evaluation results.
|
||||
they contain references to the statement node (e.g. an And statement),
|
||||
as well as the children Result instances.
|
||||
|
||||
we need this so that we can render the tree of expressions and their results.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
success: bool,
|
||||
statement: Union["capa.engine.Statement", "Feature"],
|
||||
children: List["Result"],
|
||||
locations: Optional[Set[Address]] = None,
|
||||
):
|
||||
super(Result, self).__init__()
|
||||
self.success = success
|
||||
self.statement = statement
|
||||
self.children = children
|
||||
self.locations = locations if locations is not None else set()
|
||||
|
||||
def __eq__(self, other):
|
||||
if isinstance(other, bool):
|
||||
return self.success == other
|
||||
return False
|
||||
|
||||
def __bool__(self):
|
||||
return self.success
|
||||
|
||||
def __nonzero__(self):
|
||||
return self.success
|
||||
|
||||
|
||||
class Feature(abc.ABC):
|
||||
def __init__(self, value: Union[str, int, float, bytes], description=None):
|
||||
"""
|
||||
Args:
|
||||
value (any): the value of the feature, such as the number or string.
|
||||
description (str): a human-readable description that explains the feature value.
|
||||
"""
|
||||
super(Feature, self).__init__()
|
||||
self.name = self.__class__.__name__.lower()
|
||||
|
||||
self.value = value
|
||||
self.description = description
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.name, self.value))
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.name == other.name and self.value == other.value
|
||||
|
||||
def __lt__(self, other):
|
||||
# TODO: this is a huge hack!
|
||||
import capa.features.freeze.features
|
||||
|
||||
return (
|
||||
capa.features.freeze.features.feature_from_capa(self).json()
|
||||
< capa.features.freeze.features.feature_from_capa(other).json()
|
||||
)
|
||||
|
||||
def get_value_str(self) -> str:
|
||||
"""
|
||||
render the value of this feature, for use by `__str__` and friends.
|
||||
subclasses should override to customize the rendering.
|
||||
|
||||
Returns: any
|
||||
"""
|
||||
return str(self.value)
|
||||
|
||||
def __str__(self):
|
||||
if self.value is not None:
|
||||
if self.description:
|
||||
return "%s(%s = %s)" % (self.name, self.get_value_str(), self.description)
|
||||
else:
|
||||
return "%s(%s)" % (self.name, self.get_value_str())
|
||||
else:
|
||||
return "%s" % self.name
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
def evaluate(self, ctx: Dict["Feature", Set[Address]], **kwargs) -> Result:
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature." + self.name] += 1
|
||||
return Result(self in ctx, self, [], locations=ctx.get(self, set()))
|
||||
|
||||
|
||||
class MatchedRule(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(MatchedRule, self).__init__(value, description=description)
|
||||
self.name = "match"
|
||||
|
||||
|
||||
class Characteristic(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(Characteristic, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class String(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(String, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class Class(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(Class, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class Namespace(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(Namespace, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class Substring(String):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(Substring, self).__init__(value, description=description)
|
||||
self.value = value
|
||||
|
||||
def evaluate(self, ctx, short_circuit=True):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.substring"] += 1
|
||||
|
||||
# mapping from string value to list of locations.
|
||||
# will unique the locations later on.
|
||||
matches = collections.defaultdict(list)
|
||||
|
||||
for feature, locations in ctx.items():
|
||||
if not isinstance(feature, (String,)):
|
||||
continue
|
||||
|
||||
if not isinstance(feature.value, str):
|
||||
# this is a programming error: String should only contain str
|
||||
raise ValueError("unexpected feature value type")
|
||||
|
||||
if self.value in feature.value:
|
||||
matches[feature.value].extend(locations)
|
||||
if short_circuit:
|
||||
# we found one matching string, thats sufficient to match.
|
||||
# don't collect other matching strings in this mode.
|
||||
break
|
||||
|
||||
if matches:
|
||||
# finalize: defaultdict -> dict
|
||||
# which makes json serialization easier
|
||||
matches = dict(matches)
|
||||
|
||||
# collect all locations
|
||||
locations = set()
|
||||
for s in matches.keys():
|
||||
matches[s] = list(set(matches[s]))
|
||||
locations.update(matches[s])
|
||||
|
||||
# unlike other features, we cannot return put a reference to `self` directly in a `Result`.
|
||||
# this is because `self` may match on many strings, so we can't stuff the matched value into it.
|
||||
# instead, return a new instance that has a reference to both the substring and the matched values.
|
||||
return Result(True, _MatchedSubstring(self, matches), [], locations=locations)
|
||||
else:
|
||||
return Result(False, _MatchedSubstring(self, {}), [])
|
||||
|
||||
def __str__(self):
|
||||
return "substring(%s)" % self.value
|
||||
|
||||
|
||||
class _MatchedSubstring(Substring):
|
||||
"""
|
||||
this represents specific match instances of a substring feature.
|
||||
treat it the same as a `Substring` except it has the `matches` field that contains the complete strings that matched.
|
||||
|
||||
note: this type should only ever be constructed by `Substring.evaluate()`. it is not part of the public API.
|
||||
"""
|
||||
|
||||
def __init__(self, substring: Substring, matches: Dict[str, Set[Address]]):
|
||||
"""
|
||||
args:
|
||||
substring: the substring feature that matches.
|
||||
match: mapping from matching string to its locations.
|
||||
"""
|
||||
super(_MatchedSubstring, self).__init__(str(substring.value), description=substring.description)
|
||||
# we want this to collide with the name of `Substring` above,
|
||||
# so that it works nicely with the renderers.
|
||||
self.name = "substring"
|
||||
# this may be None if the substring doesn't match
|
||||
self.matches = matches
|
||||
|
||||
def __str__(self):
|
||||
return 'substring("%s", matches = %s)' % (
|
||||
self.value,
|
||||
", ".join(map(lambda s: '"' + s + '"', (self.matches or {}).keys())),
|
||||
)
|
||||
|
||||
|
||||
class Regex(String):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(Regex, self).__init__(value, description=description)
|
||||
self.value = value
|
||||
|
||||
pat = self.value[len("/") : -len("/")]
|
||||
flags = re.DOTALL
|
||||
if value.endswith("/i"):
|
||||
pat = self.value[len("/") : -len("/i")]
|
||||
flags |= re.IGNORECASE
|
||||
try:
|
||||
self.re = re.compile(pat, flags)
|
||||
except re.error:
|
||||
if value.endswith("/i"):
|
||||
value = value[: -len("i")]
|
||||
raise ValueError(
|
||||
"invalid regular expression: %s it should use Python syntax, try it at https://pythex.org" % value
|
||||
)
|
||||
|
||||
def evaluate(self, ctx, short_circuit=True):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.regex"] += 1
|
||||
|
||||
# mapping from string value to list of locations.
|
||||
# will unique the locations later on.
|
||||
matches = collections.defaultdict(list)
|
||||
|
||||
for feature, locations in ctx.items():
|
||||
if not isinstance(feature, (String,)):
|
||||
continue
|
||||
|
||||
if not isinstance(feature.value, str):
|
||||
# this is a programming error: String should only contain str
|
||||
raise ValueError("unexpected feature value type")
|
||||
|
||||
# `re.search` finds a match anywhere in the given string
|
||||
# which implies leading and/or trailing whitespace.
|
||||
# using this mode cleans is more convenient for rule authors,
|
||||
# so that they don't have to prefix/suffix their terms like: /.*foo.*/.
|
||||
if self.re.search(feature.value):
|
||||
matches[feature.value].extend(locations)
|
||||
if short_circuit:
|
||||
# we found one matching string, thats sufficient to match.
|
||||
# don't collect other matching strings in this mode.
|
||||
break
|
||||
|
||||
if matches:
|
||||
# finalize: defaultdict -> dict
|
||||
# which makes json serialization easier
|
||||
matches = dict(matches)
|
||||
|
||||
# collect all locations
|
||||
locations = set()
|
||||
for s in matches.keys():
|
||||
matches[s] = list(set(matches[s]))
|
||||
locations.update(matches[s])
|
||||
|
||||
# unlike other features, we cannot return put a reference to `self` directly in a `Result`.
|
||||
# this is because `self` may match on many strings, so we can't stuff the matched value into it.
|
||||
# instead, return a new instance that has a reference to both the regex and the matched values.
|
||||
# see #262.
|
||||
return Result(True, _MatchedRegex(self, matches), [], locations=locations)
|
||||
else:
|
||||
return Result(False, _MatchedRegex(self, {}), [])
|
||||
|
||||
def __str__(self):
|
||||
return "regex(string =~ %s)" % self.value
|
||||
|
||||
|
||||
class _MatchedRegex(Regex):
|
||||
"""
|
||||
this represents specific match instances of a regular expression feature.
|
||||
treat it the same as a `Regex` except it has the `matches` field that contains the complete strings that matched.
|
||||
|
||||
note: this type should only ever be constructed by `Regex.evaluate()`. it is not part of the public API.
|
||||
"""
|
||||
|
||||
def __init__(self, regex: Regex, matches: Dict[str, Set[Address]]):
|
||||
"""
|
||||
args:
|
||||
regex: the regex feature that matches.
|
||||
matches: mapping from matching string to its locations.
|
||||
"""
|
||||
super(_MatchedRegex, self).__init__(str(regex.value), description=regex.description)
|
||||
# we want this to collide with the name of `Regex` above,
|
||||
# so that it works nicely with the renderers.
|
||||
self.name = "regex"
|
||||
# this may be None if the regex doesn't match
|
||||
self.matches = matches
|
||||
|
||||
def __str__(self):
|
||||
return "regex(string =~ %s, matches = %s)" % (
|
||||
self.value,
|
||||
", ".join(map(lambda s: '"' + s + '"', (self.matches or {}).keys())),
|
||||
)
|
||||
|
||||
|
||||
class StringFactory:
|
||||
def __new__(cls, value: str, description=None):
|
||||
if value.startswith("/") and (value.endswith("/") or value.endswith("/i")):
|
||||
return Regex(value, description=description)
|
||||
return String(value, description=description)
|
||||
|
||||
|
||||
class Bytes(Feature):
|
||||
def __init__(self, value: bytes, description=None):
|
||||
super(Bytes, self).__init__(value, description=description)
|
||||
self.value = value
|
||||
|
||||
def evaluate(self, ctx, **kwargs):
|
||||
capa.perf.counters["evaluate.feature"] += 1
|
||||
capa.perf.counters["evaluate.feature.bytes"] += 1
|
||||
|
||||
for feature, locations in ctx.items():
|
||||
if not isinstance(feature, (Bytes,)):
|
||||
continue
|
||||
|
||||
if feature.value.startswith(self.value):
|
||||
return Result(True, self, [], locations=locations)
|
||||
|
||||
return Result(False, self, [])
|
||||
|
||||
def get_value_str(self):
|
||||
return hex_string(bytes_to_str(self.value))
|
||||
|
||||
|
||||
# other candidates here: https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#machine-types
|
||||
ARCH_I386 = "i386"
|
||||
ARCH_AMD64 = "amd64"
|
||||
# dotnet
|
||||
ARCH_ANY = "any"
|
||||
VALID_ARCH = (ARCH_I386, ARCH_AMD64, ARCH_ANY)
|
||||
|
||||
|
||||
class Arch(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(Arch, self).__init__(value, description=description)
|
||||
self.name = "arch"
|
||||
|
||||
|
||||
OS_WINDOWS = "windows"
|
||||
OS_LINUX = "linux"
|
||||
OS_MACOS = "macos"
|
||||
# dotnet
|
||||
OS_ANY = "any"
|
||||
VALID_OS = {os.value for os in capa.features.extractors.elf.OS}
|
||||
VALID_OS.update({OS_WINDOWS, OS_LINUX, OS_MACOS, OS_ANY})
|
||||
|
||||
|
||||
class OS(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(OS, self).__init__(value, description=description)
|
||||
self.name = "os"
|
||||
|
||||
|
||||
FORMAT_PE = "pe"
|
||||
FORMAT_ELF = "elf"
|
||||
FORMAT_DOTNET = "dotnet"
|
||||
VALID_FORMAT = (FORMAT_PE, FORMAT_ELF, FORMAT_DOTNET)
|
||||
# internal only, not to be used in rules
|
||||
FORMAT_AUTO = "auto"
|
||||
FORMAT_SC32 = "sc32"
|
||||
FORMAT_SC64 = "sc64"
|
||||
FORMAT_FREEZE = "freeze"
|
||||
FORMAT_UNKNOWN = "unknown"
|
||||
|
||||
|
||||
class Format(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
super(Format, self).__init__(value, description=description)
|
||||
self.name = "format"
|
||||
|
||||
|
||||
def is_global_feature(feature):
|
||||
"""
|
||||
is this a feature that is extracted at every scope?
|
||||
today, these are OS and arch features.
|
||||
"""
|
||||
return isinstance(feature, (OS, Arch))
|
||||
@@ -0,0 +1,286 @@
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import abc
|
||||
|
||||
|
||||
class FeatureExtractor(object):
|
||||
"""
|
||||
FeatureExtractor defines the interface for fetching features from a sample.
|
||||
|
||||
There may be multiple backends that support fetching features for capa.
|
||||
For example, we use vivisect by default, but also want to support saving
|
||||
and restoring features from a JSON file.
|
||||
When we restore the features, we'd like to use exactly the same matching logic
|
||||
to find matching rules.
|
||||
Therefore, we can define a FeatureExtractor that provides features from the
|
||||
serialized JSON file and do matching without a binary analysis pass.
|
||||
Also, this provides a way to hook in an IDA backend.
|
||||
|
||||
This class is not instantiated directly; it is the base class for other implementations.
|
||||
"""
|
||||
|
||||
__metaclass__ = abc.ABCMeta
|
||||
|
||||
def __init__(self):
|
||||
#
|
||||
# note: a subclass should define ctor parameters for its own use.
|
||||
# for example, the Vivisect feature extract might require the vw and/or path.
|
||||
# this base class doesn't know what to do with that info, though.
|
||||
#
|
||||
super(FeatureExtractor, self).__init__()
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_base_address(self):
|
||||
"""
|
||||
fetch the preferred load address at which the sample was analyzed.
|
||||
|
||||
returns: int
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_file_features(self):
|
||||
"""
|
||||
extract file-scope features.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for feature, va in extractor.get_file_features():
|
||||
print('0x%x: %s', va, feature)
|
||||
|
||||
yields:
|
||||
Tuple[capa.features.Feature, int]: feature and its location
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_functions(self):
|
||||
"""
|
||||
enumerate the functions and provide opaque values that will
|
||||
subsequently be provided to `.extract_function_features()`, etc.
|
||||
|
||||
by "opaque value", we mean that this can be any object, as long as it
|
||||
provides enough context to `.extract_function_features()`.
|
||||
|
||||
the opaque value should support casting to int (`__int__`) for the function start address.
|
||||
|
||||
yields:
|
||||
any: the opaque function value.
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_function_features(self, f):
|
||||
"""
|
||||
extract function-scope features.
|
||||
the arguments are opaque values previously provided by `.get_functions()`, etc.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for function in extractor.get_functions():
|
||||
for feature, va in extractor.extract_function_features(function):
|
||||
print('0x%x: %s', va, feature)
|
||||
|
||||
args:
|
||||
f [any]: an opaque value previously fetched from `.get_functions()`.
|
||||
|
||||
yields:
|
||||
Tuple[capa.features.Feature, int]: feature and its location
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_basic_blocks(self, f):
|
||||
"""
|
||||
enumerate the basic blocks in the given function and provide opaque values that will
|
||||
subsequently be provided to `.extract_basic_block_features()`, etc.
|
||||
|
||||
by "opaque value", we mean that this can be any object, as long as it
|
||||
provides enough context to `.extract_basic_block_features()`.
|
||||
|
||||
the opaque value should support casting to int (`__int__`) for the basic block start address.
|
||||
|
||||
yields:
|
||||
any: the opaque basic block value.
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
"""
|
||||
extract basic block-scope features.
|
||||
the arguments are opaque values previously provided by `.get_functions()`, etc.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for function in extractor.get_functions():
|
||||
for bb in extractor.get_basic_blocks(function):
|
||||
for feature, va in extractor.extract_basic_block_features(function, bb):
|
||||
print('0x%x: %s', va, feature)
|
||||
|
||||
args:
|
||||
f [any]: an opaque value previously fetched from `.get_functions()`.
|
||||
bb [any]: an opaque value previously fetched from `.get_basic_blocks()`.
|
||||
|
||||
yields:
|
||||
Tuple[capa.features.Feature, int]: feature and its location
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_instructions(self, f, bb):
|
||||
"""
|
||||
enumerate the instructions in the given basic block and provide opaque values that will
|
||||
subsequently be provided to `.extract_insn_features()`, etc.
|
||||
|
||||
by "opaque value", we mean that this can be any object, as long as it
|
||||
provides enough context to `.extract_insn_features()`.
|
||||
|
||||
the opaque value should support casting to int (`__int__`) for the instruction address.
|
||||
|
||||
yields:
|
||||
any: the opaque function value.
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
"""
|
||||
extract instruction-scope features.
|
||||
the arguments are opaque values previously provided by `.get_functions()`, etc.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for function in extractor.get_functions():
|
||||
for bb in extractor.get_basic_blocks(function):
|
||||
for insn in extractor.get_instructions(function, bb):
|
||||
for feature, va in extractor.extract_insn_features(function, bb, insn):
|
||||
print('0x%x: %s', va, feature)
|
||||
|
||||
args:
|
||||
f [any]: an opaque value previously fetched from `.get_functions()`.
|
||||
bb [any]: an opaque value previously fetched from `.get_basic_blocks()`.
|
||||
insn [any]: an opaque value previously fetched from `.get_instructions()`.
|
||||
|
||||
yields:
|
||||
Tuple[capa.features.Feature, int]: feature and its location
|
||||
"""
|
||||
raise NotImplemented
|
||||
|
||||
|
||||
class NullFeatureExtractor(FeatureExtractor):
|
||||
"""
|
||||
An extractor that extracts some user-provided features.
|
||||
The structure of the single parameter is demonstrated in the example below.
|
||||
|
||||
This is useful for testing, as we can provide expected values and see if matching works.
|
||||
Also, this is how we represent features deserialized from a freeze file.
|
||||
|
||||
example::
|
||||
|
||||
extractor = NullFeatureExtractor({
|
||||
'base address: 0x401000,
|
||||
'file features': [
|
||||
(0x402345, capa.features.Characteristic('embedded pe')),
|
||||
],
|
||||
'functions': {
|
||||
0x401000: {
|
||||
'features': [
|
||||
(0x401000, capa.features.Characteristic('nzxor')),
|
||||
],
|
||||
'basic blocks': {
|
||||
0x401000: {
|
||||
'features': [
|
||||
(0x401000, capa.features.Characteristic('tight-loop')),
|
||||
],
|
||||
'instructions': {
|
||||
0x401000: {
|
||||
'features': [
|
||||
(0x401000, capa.features.Characteristic('nzxor')),
|
||||
],
|
||||
},
|
||||
0x401002: ...
|
||||
}
|
||||
},
|
||||
0x401005: ...
|
||||
}
|
||||
},
|
||||
0x40200: ...
|
||||
}
|
||||
)
|
||||
"""
|
||||
|
||||
def __init__(self, features):
|
||||
super(NullFeatureExtractor, self).__init__()
|
||||
self.features = features
|
||||
|
||||
def get_base_address(self):
|
||||
return self.features["base address"]
|
||||
|
||||
def extract_file_features(self):
|
||||
for p in self.features.get("file features", []):
|
||||
va, feature = p
|
||||
yield feature, va
|
||||
|
||||
def get_functions(self):
|
||||
for va in sorted(self.features["functions"].keys()):
|
||||
yield va
|
||||
|
||||
def extract_function_features(self, f):
|
||||
for p in self.features.get("functions", {}).get(f, {}).get("features", []): # noqa: E127 line over-indented
|
||||
va, feature = p
|
||||
yield feature, va
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
for va in sorted(
|
||||
self.features.get("functions", {}) # noqa: E127 line over-indented
|
||||
.get(f, {})
|
||||
.get("basic blocks", {})
|
||||
.keys()
|
||||
):
|
||||
yield va
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
for p in (
|
||||
self.features.get("functions", {}) # noqa: E127 line over-indented
|
||||
.get(f, {})
|
||||
.get("basic blocks", {})
|
||||
.get(bb, {})
|
||||
.get("features", [])
|
||||
):
|
||||
va, feature = p
|
||||
yield feature, va
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
for va in sorted(
|
||||
self.features.get("functions", {}) # noqa: E127 line over-indented
|
||||
.get(f, {})
|
||||
.get("basic blocks", {})
|
||||
.get(bb, {})
|
||||
.get("instructions", {})
|
||||
.keys()
|
||||
):
|
||||
yield va
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
for p in (
|
||||
self.features.get("functions", {}) # noqa: E127 line over-indented
|
||||
.get(f, {})
|
||||
.get("basic blocks", {})
|
||||
.get(bb, {})
|
||||
.get("instructions", {})
|
||||
.get(insn, {})
|
||||
.get("features", [])
|
||||
):
|
||||
va, feature = p
|
||||
yield feature, va
|
||||
|
||||
@@ -1,264 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import abc
|
||||
import dataclasses
|
||||
from typing import Any, Dict, Tuple, Union, Iterator
|
||||
from dataclasses import dataclass
|
||||
|
||||
import capa.features.address
|
||||
from capa.features.common import Feature
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
|
||||
# feature extractors may reference functions, BBs, insns by opaque handle values.
|
||||
# you can use the `.address` property to get and render the address of the feature.
|
||||
#
|
||||
# these handles are only consumed by routines on
|
||||
# the feature extractor from which they were created.
|
||||
|
||||
|
||||
@dataclass
|
||||
class FunctionHandle:
|
||||
"""reference to a function recognized by a feature extractor.
|
||||
|
||||
Attributes:
|
||||
address: the address of the function.
|
||||
inner: extractor-specific data.
|
||||
ctx: a context object for the extractor.
|
||||
"""
|
||||
|
||||
address: Address
|
||||
inner: Any
|
||||
ctx: Dict[str, Any] = dataclasses.field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class BBHandle:
|
||||
"""reference to a basic block recognized by a feature extractor.
|
||||
|
||||
Attributes:
|
||||
address: the address of the basic block start address.
|
||||
inner: extractor-specific data.
|
||||
"""
|
||||
|
||||
address: Address
|
||||
inner: Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class InsnHandle:
|
||||
"""reference to a instruction recognized by a feature extractor.
|
||||
|
||||
Attributes:
|
||||
address: the address of the instruction address.
|
||||
inner: extractor-specific data.
|
||||
"""
|
||||
|
||||
address: Address
|
||||
inner: Any
|
||||
|
||||
|
||||
class FeatureExtractor:
|
||||
"""
|
||||
FeatureExtractor defines the interface for fetching features from a sample.
|
||||
|
||||
There may be multiple backends that support fetching features for capa.
|
||||
For example, we use vivisect by default, but also want to support saving
|
||||
and restoring features from a JSON file.
|
||||
When we restore the features, we'd like to use exactly the same matching logic
|
||||
to find matching rules.
|
||||
Therefore, we can define a FeatureExtractor that provides features from the
|
||||
serialized JSON file and do matching without a binary analysis pass.
|
||||
Also, this provides a way to hook in an IDA backend.
|
||||
|
||||
This class is not instantiated directly; it is the base class for other implementations.
|
||||
"""
|
||||
|
||||
__metaclass__ = abc.ABCMeta
|
||||
|
||||
def __init__(self):
|
||||
#
|
||||
# note: a subclass should define ctor parameters for its own use.
|
||||
# for example, the Vivisect feature extract might require the vw and/or path.
|
||||
# this base class doesn't know what to do with that info, though.
|
||||
#
|
||||
super(FeatureExtractor, self).__init__()
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_base_address(self) -> Union[AbsoluteVirtualAddress, capa.features.address._NoAddress]:
|
||||
"""
|
||||
fetch the preferred load address at which the sample was analyzed.
|
||||
|
||||
when the base address is `NO_ADDRESS`, then the loader has no concept of a preferred load address.
|
||||
such as: shellcode, .NET modules, etc.
|
||||
in these scenarios, RelativeVirtualAddresses aren't used.
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_global_features(self) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
extract features found at every scope ("global").
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for feature, va in extractor.get_global_features():
|
||||
print('0x%x: %s', va, feature)
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: feature and its location
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_file_features(self) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
extract file-scope features.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for feature, va in extractor.get_file_features():
|
||||
print('0x%x: %s', va, feature)
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: feature and its location
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_functions(self) -> Iterator[FunctionHandle]:
|
||||
"""
|
||||
enumerate the functions and provide opaque values that will
|
||||
subsequently be provided to `.extract_function_features()`, etc.
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
def is_library_function(self, addr: Address) -> bool:
|
||||
"""
|
||||
is the given address a library function?
|
||||
the backend may implement its own function matching algorithm, or none at all.
|
||||
we accept an address here, rather than function object,
|
||||
to handle addresses identified in instructions.
|
||||
|
||||
this information is used to:
|
||||
- filter out matches in library functions (by default), and
|
||||
- recognize when to fetch symbol names for called (non-API) functions
|
||||
|
||||
args:
|
||||
addr (Address): the address of a function.
|
||||
|
||||
returns:
|
||||
bool: True if the given address is the start of a library function.
|
||||
"""
|
||||
return False
|
||||
|
||||
def get_function_name(self, addr: Address) -> str:
|
||||
"""
|
||||
fetch any recognized name for the given address.
|
||||
this is only guaranteed to return a value when the given function is a recognized library function.
|
||||
we accept a VA here, rather than function object, to handle addresses identified in instructions.
|
||||
|
||||
args:
|
||||
addr (Address): the address of a function.
|
||||
|
||||
returns:
|
||||
str: the function name
|
||||
|
||||
raises:
|
||||
KeyError: when the given function does not have a name.
|
||||
"""
|
||||
raise KeyError(addr)
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_function_features(self, f: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
extract function-scope features.
|
||||
the arguments are opaque values previously provided by `.get_functions()`, etc.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for function in extractor.get_functions():
|
||||
for feature, address in extractor.extract_function_features(function):
|
||||
print('0x%x: %s', address, feature)
|
||||
|
||||
args:
|
||||
f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: feature and its location
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_basic_blocks(self, f: FunctionHandle) -> Iterator[BBHandle]:
|
||||
"""
|
||||
enumerate the basic blocks in the given function and provide opaque values that will
|
||||
subsequently be provided to `.extract_basic_block_features()`, etc.
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_basic_block_features(self, f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
extract basic block-scope features.
|
||||
the arguments are opaque values previously provided by `.get_functions()`, etc.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for function in extractor.get_functions():
|
||||
for bb in extractor.get_basic_blocks(function):
|
||||
for feature, address in extractor.extract_basic_block_features(function, bb):
|
||||
print('0x%x: %s', address, feature)
|
||||
|
||||
args:
|
||||
f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
|
||||
bb [BBHandle]: an opaque value previously fetched from `.get_basic_blocks()`.
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: feature and its location
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@abc.abstractmethod
|
||||
def get_instructions(self, f: FunctionHandle, bb: BBHandle) -> Iterator[InsnHandle]:
|
||||
"""
|
||||
enumerate the instructions in the given basic block and provide opaque values that will
|
||||
subsequently be provided to `.extract_insn_features()`, etc.
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
@abc.abstractmethod
|
||||
def extract_insn_features(
|
||||
self, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
extract instruction-scope features.
|
||||
the arguments are opaque values previously provided by `.get_functions()`, etc.
|
||||
|
||||
example::
|
||||
|
||||
extractor = VivisectFeatureExtractor(vw, path)
|
||||
for function in extractor.get_functions():
|
||||
for bb in extractor.get_basic_blocks(function):
|
||||
for insn in extractor.get_instructions(function, bb):
|
||||
for feature, address in extractor.extract_insn_features(function, bb, insn):
|
||||
print('0x%x: %s', address, feature)
|
||||
|
||||
args:
|
||||
f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
|
||||
bb [BBHandle]: an opaque value previously fetched from `.get_basic_blocks()`.
|
||||
insn [InsnHandle]: an opaque value previously fetched from `.get_instructions()`.
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: feature and its location
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
@@ -1,100 +0,0 @@
|
||||
import io
|
||||
import logging
|
||||
import binascii
|
||||
import contextlib
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import pefile
|
||||
|
||||
import capa.features
|
||||
import capa.features.extractors.elf
|
||||
import capa.features.extractors.pefile
|
||||
from capa.features.common import OS, FORMAT_PE, FORMAT_ELF, OS_WINDOWS, FORMAT_FREEZE, Arch, Format, String, Feature
|
||||
from capa.features.freeze import is_freeze
|
||||
from capa.features.address import NO_ADDRESS, Address, FileOffsetAddress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_file_strings(buf, **kwargs) -> Iterator[Tuple[String, Address]]:
|
||||
"""
|
||||
extract ASCII and UTF-16 LE strings from file
|
||||
"""
|
||||
for s in capa.features.extractors.strings.extract_ascii_strings(buf):
|
||||
yield String(s.s), FileOffsetAddress(s.offset)
|
||||
|
||||
for s in capa.features.extractors.strings.extract_unicode_strings(buf):
|
||||
yield String(s.s), FileOffsetAddress(s.offset)
|
||||
|
||||
|
||||
def extract_format(buf) -> Iterator[Tuple[Feature, Address]]:
|
||||
if buf.startswith(b"MZ"):
|
||||
yield Format(FORMAT_PE), NO_ADDRESS
|
||||
elif buf.startswith(b"\x7fELF"):
|
||||
yield Format(FORMAT_ELF), NO_ADDRESS
|
||||
elif is_freeze(buf):
|
||||
yield Format(FORMAT_FREEZE), NO_ADDRESS
|
||||
else:
|
||||
# we likely end up here:
|
||||
# 1. handling a file format (e.g. macho)
|
||||
#
|
||||
# for (1), this logic will need to be updated as the format is implemented.
|
||||
logger.debug("unsupported file format: %s", binascii.hexlify(buf[:4]).decode("ascii"))
|
||||
return
|
||||
|
||||
|
||||
def extract_arch(buf) -> Iterator[Tuple[Feature, Address]]:
|
||||
if buf.startswith(b"MZ"):
|
||||
yield from capa.features.extractors.pefile.extract_file_arch(pe=pefile.PE(data=buf))
|
||||
|
||||
elif buf.startswith(b"\x7fELF"):
|
||||
with contextlib.closing(io.BytesIO(buf)) as f:
|
||||
arch = capa.features.extractors.elf.detect_elf_arch(f)
|
||||
|
||||
if arch not in capa.features.common.VALID_ARCH:
|
||||
logger.debug("unsupported arch: %s", arch)
|
||||
return
|
||||
|
||||
yield Arch(arch), NO_ADDRESS
|
||||
|
||||
else:
|
||||
# we likely end up here:
|
||||
# 1. handling shellcode, or
|
||||
# 2. handling a new file format (e.g. macho)
|
||||
#
|
||||
# for (1) we can't do much - its shellcode and all bets are off.
|
||||
# we could maybe accept a futher CLI argument to specify the arch,
|
||||
# but i think this would be rarely used.
|
||||
# rules that rely on arch conditions will fail to match on shellcode.
|
||||
#
|
||||
# for (2), this logic will need to be updated as the format is implemented.
|
||||
logger.debug("unsupported file format: %s, will not guess Arch", binascii.hexlify(buf[:4]).decode("ascii"))
|
||||
return
|
||||
|
||||
|
||||
def extract_os(buf) -> Iterator[Tuple[Feature, Address]]:
|
||||
if buf.startswith(b"MZ"):
|
||||
yield OS(OS_WINDOWS), NO_ADDRESS
|
||||
elif buf.startswith(b"\x7fELF"):
|
||||
with contextlib.closing(io.BytesIO(buf)) as f:
|
||||
os = capa.features.extractors.elf.detect_elf_os(f)
|
||||
|
||||
if os not in capa.features.common.VALID_OS:
|
||||
logger.debug("unsupported os: %s", os)
|
||||
return
|
||||
|
||||
yield OS(os), NO_ADDRESS
|
||||
|
||||
else:
|
||||
# we likely end up here:
|
||||
# 1. handling shellcode, or
|
||||
# 2. handling a new file format (e.g. macho)
|
||||
#
|
||||
# for (1) we can't do much - its shellcode and all bets are off.
|
||||
# we could maybe accept a futher CLI argument to specify the OS,
|
||||
# but i think this would be rarely used.
|
||||
# rules that rely on OS conditions will fail to match on shellcode.
|
||||
#
|
||||
# for (2), this logic will need to be updated as the format is implemented.
|
||||
logger.debug("unsupported file format: %s, will not guess OS", binascii.hexlify(buf[:4]).decode("ascii"))
|
||||
return
|
||||
@@ -1,71 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import List, Tuple, Iterator
|
||||
|
||||
import dnfile
|
||||
from dncil.clr.token import Token
|
||||
|
||||
import capa.features.extractors
|
||||
import capa.features.extractors.dnfile.file
|
||||
import capa.features.extractors.dnfile.insn
|
||||
from capa.features.common import Feature
|
||||
from capa.features.address import NO_ADDRESS, Address, DNTokenAddress, DNTokenOffsetAddress, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
|
||||
from capa.features.extractors.dnfile.helpers import get_dotnet_managed_method_bodies
|
||||
|
||||
|
||||
class DnfileFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, path: str):
|
||||
super(DnfileFeatureExtractor, self).__init__()
|
||||
self.pe: dnfile.dnPE = dnfile.dnPE(path)
|
||||
|
||||
# pre-compute these because we'll yield them at *every* scope.
|
||||
self.global_features: List[Tuple[Feature, Address]] = []
|
||||
self.global_features.extend(capa.features.extractors.dotnetfile.extract_file_os(pe=self.pe))
|
||||
self.global_features.extend(capa.features.extractors.dotnetfile.extract_file_arch(pe=self.pe))
|
||||
|
||||
def get_base_address(self):
|
||||
return NO_ADDRESS
|
||||
|
||||
def extract_global_features(self):
|
||||
yield from self.global_features
|
||||
|
||||
def extract_file_features(self):
|
||||
yield from capa.features.extractors.dnfile.file.extract_features(self.pe)
|
||||
|
||||
def get_functions(self) -> Iterator[FunctionHandle]:
|
||||
for token, f in get_dotnet_managed_method_bodies(self.pe):
|
||||
yield FunctionHandle(address=DNTokenAddress(Token(token)), inner=f, ctx={"pe": self.pe})
|
||||
|
||||
def extract_function_features(self, f):
|
||||
# TODO
|
||||
yield from []
|
||||
|
||||
def get_basic_blocks(self, f) -> Iterator[BBHandle]:
|
||||
# each dotnet method is considered 1 basic block
|
||||
yield BBHandle(
|
||||
address=f.address,
|
||||
inner=f.inner,
|
||||
)
|
||||
|
||||
def extract_basic_block_features(self, fh, bbh):
|
||||
# we don't support basic block features
|
||||
yield from []
|
||||
|
||||
def get_instructions(self, fh, bbh):
|
||||
for insn in bbh.inner.instructions:
|
||||
yield InsnHandle(
|
||||
address=DNTokenOffsetAddress(bbh.address.token, insn.offset - (fh.inner.offset + fh.inner.header_size)),
|
||||
inner=insn,
|
||||
)
|
||||
|
||||
def extract_insn_features(self, fh, bbh, ih) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.dnfile.insn.extract_features(fh, bbh, ih)
|
||||
@@ -1,63 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import dnfile
|
||||
|
||||
import capa.features.extractors.dotnetfile
|
||||
from capa.features.file import Import, FunctionName
|
||||
from capa.features.common import Class, Format, String, Feature, Namespace, Characteristic
|
||||
from capa.features.address import Address
|
||||
|
||||
|
||||
def extract_file_import_names(pe: dnfile.dnPE) -> Iterator[Tuple[Import, Address]]:
|
||||
yield from capa.features.extractors.dotnetfile.extract_file_import_names(pe=pe)
|
||||
|
||||
|
||||
def extract_file_format(pe: dnfile.dnPE) -> Iterator[Tuple[Format, Address]]:
|
||||
yield from capa.features.extractors.dotnetfile.extract_file_format(pe=pe)
|
||||
|
||||
|
||||
def extract_file_function_names(pe: dnfile.dnPE) -> Iterator[Tuple[FunctionName, Address]]:
|
||||
yield from capa.features.extractors.dotnetfile.extract_file_function_names(pe=pe)
|
||||
|
||||
|
||||
def extract_file_strings(pe: dnfile.dnPE) -> Iterator[Tuple[String, Address]]:
|
||||
yield from capa.features.extractors.dotnetfile.extract_file_strings(pe=pe)
|
||||
|
||||
|
||||
def extract_file_mixed_mode_characteristic_features(pe: dnfile.dnPE) -> Iterator[Tuple[Characteristic, Address]]:
|
||||
yield from capa.features.extractors.dotnetfile.extract_file_mixed_mode_characteristic_features(pe=pe)
|
||||
|
||||
|
||||
def extract_file_namespace_features(pe: dnfile.dnPE) -> Iterator[Tuple[Namespace, Address]]:
|
||||
yield from capa.features.extractors.dotnetfile.extract_file_namespace_features(pe=pe)
|
||||
|
||||
|
||||
def extract_file_class_features(pe: dnfile.dnPE) -> Iterator[Tuple[Class, Address]]:
|
||||
yield from capa.features.extractors.dotnetfile.extract_file_class_features(pe=pe)
|
||||
|
||||
|
||||
def extract_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for (feature, address) in file_handler(pe):
|
||||
yield feature, address
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
extract_file_import_names,
|
||||
extract_file_function_names,
|
||||
extract_file_strings,
|
||||
extract_file_format,
|
||||
extract_file_mixed_mode_characteristic_features,
|
||||
extract_file_namespace_features,
|
||||
extract_file_class_features,
|
||||
)
|
||||
@@ -1,261 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Tuple, Iterator, Optional
|
||||
|
||||
import dnfile
|
||||
from dncil.cil.body import CilMethodBody
|
||||
from dncil.cil.error import MethodBodyFormatError
|
||||
from dncil.clr.token import Token, StringToken, InvalidToken
|
||||
from dncil.cil.body.reader import CilMethodBodyReaderBase
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# key indexes to dotnet metadata tables
|
||||
DOTNET_META_TABLES_BY_INDEX = {table.value: table.name for table in dnfile.enums.MetadataTables}
|
||||
|
||||
|
||||
class DnfileMethodBodyReader(CilMethodBodyReaderBase):
|
||||
def __init__(self, pe: dnfile.dnPE, row: dnfile.mdtable.MethodDefRow):
|
||||
self.pe: dnfile.dnPE = pe
|
||||
self.offset: int = self.pe.get_offset_from_rva(row.Rva)
|
||||
|
||||
def read(self, n: int) -> bytes:
|
||||
data: bytes = self.pe.get_data(self.pe.get_rva_from_offset(self.offset), n)
|
||||
self.offset += n
|
||||
return data
|
||||
|
||||
def tell(self) -> int:
|
||||
return self.offset
|
||||
|
||||
def seek(self, offset: int) -> int:
|
||||
self.offset = offset
|
||||
return self.offset
|
||||
|
||||
|
||||
class DnClass(object):
|
||||
def __init__(self, token: int, namespace: str, classname: str):
|
||||
self.token: int = token
|
||||
self.namespace: str = namespace
|
||||
self.classname: str = classname
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.token,))
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.token == other.token
|
||||
|
||||
def __str__(self):
|
||||
return DnClass.format_name(self.namespace, self.classname)
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
@staticmethod
|
||||
def format_name(namespace: str, classname: str):
|
||||
name: str = classname
|
||||
if namespace:
|
||||
# like System.IO.File::OpenRead
|
||||
name = f"{namespace}.{name}"
|
||||
return name
|
||||
|
||||
|
||||
class DnMethod(DnClass):
|
||||
def __init__(self, token: int, namespace: str, classname: str, methodname: str):
|
||||
super(DnMethod, self).__init__(token, namespace, classname)
|
||||
self.methodname: str = methodname
|
||||
|
||||
def __str__(self):
|
||||
return DnMethod.format_name(self.namespace, self.classname, self.methodname)
|
||||
|
||||
@staticmethod
|
||||
def format_name(namespace: str, classname: str, methodname: str): # type: ignore
|
||||
# like File::OpenRead
|
||||
name: str = f"{classname}::{methodname}"
|
||||
if namespace:
|
||||
# like System.IO.File::OpenRead
|
||||
name = f"{namespace}.{name}"
|
||||
return name
|
||||
|
||||
|
||||
class DnUnmanagedMethod:
|
||||
def __init__(self, token: int, modulename: str, methodname: str):
|
||||
self.token: int = token
|
||||
self.modulename: str = modulename
|
||||
self.methodname: str = methodname
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.token,))
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.token == other.token
|
||||
|
||||
def __str__(self):
|
||||
return DnUnmanagedMethod.format_name(self.modulename, self.methodname)
|
||||
|
||||
def __repr__(self):
|
||||
return str(self)
|
||||
|
||||
@staticmethod
|
||||
def format_name(modulename, methodname):
|
||||
return f"{modulename}.{methodname}"
|
||||
|
||||
|
||||
def resolve_dotnet_token(pe: dnfile.dnPE, token: Token) -> Any:
|
||||
"""map generic token to string or table row"""
|
||||
if isinstance(token, StringToken):
|
||||
user_string: Optional[str] = read_dotnet_user_string(pe, token)
|
||||
if user_string is None:
|
||||
return InvalidToken(token.value)
|
||||
return user_string
|
||||
|
||||
table_name: str = DOTNET_META_TABLES_BY_INDEX.get(token.table, "")
|
||||
if not table_name:
|
||||
# table_index is not valid
|
||||
return InvalidToken(token.value)
|
||||
|
||||
table: Any = getattr(pe.net.mdtables, table_name, None)
|
||||
if table is None:
|
||||
# table index is valid but table is not present
|
||||
return InvalidToken(token.value)
|
||||
|
||||
try:
|
||||
return table.rows[token.rid - 1]
|
||||
except IndexError:
|
||||
# table index is valid but row index is not valid
|
||||
return InvalidToken(token.value)
|
||||
|
||||
|
||||
def read_dotnet_method_body(pe: dnfile.dnPE, row: dnfile.mdtable.MethodDefRow) -> Optional[CilMethodBody]:
|
||||
"""read dotnet method body"""
|
||||
try:
|
||||
return CilMethodBody(DnfileMethodBodyReader(pe, row))
|
||||
except MethodBodyFormatError as e:
|
||||
logger.warn("failed to parse managed method body @ 0x%08x (%s)" % (row.Rva, e))
|
||||
return None
|
||||
|
||||
|
||||
def read_dotnet_user_string(pe: dnfile.dnPE, token: StringToken) -> Optional[str]:
|
||||
"""read user string from #US stream"""
|
||||
try:
|
||||
user_string: Optional[dnfile.stream.UserString] = pe.net.user_strings.get_us(token.rid)
|
||||
except UnicodeDecodeError as e:
|
||||
logger.warn("failed to decode #US stream index 0x%08x (%s)" % (token.rid, e))
|
||||
return None
|
||||
|
||||
if user_string is None:
|
||||
return None
|
||||
|
||||
return user_string.value
|
||||
|
||||
|
||||
def get_dotnet_managed_imports(pe: dnfile.dnPE) -> Iterator[DnMethod]:
|
||||
"""get managed imports from MemberRef table
|
||||
|
||||
see https://www.ntcore.com/files/dotnetformat.htm
|
||||
|
||||
10 - MemberRef Table
|
||||
Each row represents an imported method
|
||||
Class (index into the TypeRef, ModuleRef, MethodDef, TypeSpec or TypeDef tables)
|
||||
Name (index into String heap)
|
||||
01 - TypeRef Table
|
||||
Each row represents an imported class, its namespace and the assembly which contains it
|
||||
TypeName (index into String heap)
|
||||
TypeNamespace (index into String heap)
|
||||
"""
|
||||
for (rid, row) in enumerate(iter_dotnet_table(pe, "MemberRef")):
|
||||
if not isinstance(row.Class.row, dnfile.mdtable.TypeRefRow):
|
||||
continue
|
||||
|
||||
token: int = calculate_dotnet_token_value(pe.net.mdtables.MemberRef.number, rid + 1)
|
||||
yield DnMethod(token, row.Class.row.TypeNamespace, row.Class.row.TypeName, row.Name)
|
||||
|
||||
|
||||
def get_dotnet_managed_methods(pe: dnfile.dnPE) -> Iterator[DnMethod]:
|
||||
"""get managed method names from TypeDef table
|
||||
|
||||
see https://www.ntcore.com/files/dotnetformat.htm
|
||||
|
||||
02 - TypeDef Table
|
||||
Each row represents a class in the current assembly.
|
||||
TypeName (index into String heap)
|
||||
TypeNamespace (index into String heap)
|
||||
MethodList (index into MethodDef table; it marks the first of a continguous run of Methods owned by this Type)
|
||||
"""
|
||||
for row in iter_dotnet_table(pe, "TypeDef"):
|
||||
for index in row.MethodList:
|
||||
token = calculate_dotnet_token_value(index.table.number, index.row_index)
|
||||
yield DnMethod(token, row.TypeNamespace, row.TypeName, index.row.Name)
|
||||
|
||||
|
||||
def get_dotnet_managed_method_bodies(pe: dnfile.dnPE) -> Iterator[Tuple[int, CilMethodBody]]:
|
||||
"""get managed methods from MethodDef table"""
|
||||
if not hasattr(pe.net.mdtables, "MethodDef"):
|
||||
return
|
||||
|
||||
for (rid, row) in enumerate(pe.net.mdtables.MethodDef):
|
||||
if not row.ImplFlags.miIL or any((row.Flags.mdAbstract, row.Flags.mdPinvokeImpl)):
|
||||
# skip methods that do not have a method body
|
||||
continue
|
||||
|
||||
body: Optional[CilMethodBody] = read_dotnet_method_body(pe, row)
|
||||
if body is None:
|
||||
continue
|
||||
|
||||
token: int = calculate_dotnet_token_value(dnfile.enums.MetadataTables.MethodDef.value, rid + 1)
|
||||
yield token, body
|
||||
|
||||
|
||||
def get_dotnet_unmanaged_imports(pe: dnfile.dnPE) -> Iterator[DnUnmanagedMethod]:
|
||||
"""get unmanaged imports from ImplMap table
|
||||
|
||||
see https://www.ntcore.com/files/dotnetformat.htm
|
||||
|
||||
28 - ImplMap Table
|
||||
ImplMap table holds information about unmanaged methods that can be reached from managed code, using PInvoke dispatch
|
||||
MemberForwarded (index into the Field or MethodDef table; more precisely, a MemberForwarded coded index)
|
||||
ImportName (index into the String heap)
|
||||
ImportScope (index into the ModuleRef table)
|
||||
"""
|
||||
for row in iter_dotnet_table(pe, "ImplMap"):
|
||||
modulename: str = row.ImportScope.row.Name
|
||||
methodname: str = row.ImportName
|
||||
|
||||
# ECMA says "Each row of the ImplMap table associates a row in the MethodDef table (MemberForwarded) with the
|
||||
# name of a routine (ImportName) in some unmanaged DLL (ImportScope)"; so we calculate and map the MemberForwarded
|
||||
# MethodDef table token to help us later record native import method calls made from CIL
|
||||
token: int = calculate_dotnet_token_value(row.MemberForwarded.table.number, row.MemberForwarded.row_index)
|
||||
|
||||
# like Kernel32.dll
|
||||
if modulename and "." in modulename:
|
||||
modulename = modulename.split(".")[0]
|
||||
|
||||
# like kernel32.CreateFileA
|
||||
yield DnUnmanagedMethod(token, modulename, methodname)
|
||||
|
||||
|
||||
def calculate_dotnet_token_value(table: int, rid: int) -> int:
|
||||
return ((table & 0xFF) << Token.TABLE_SHIFT) | (rid & Token.RID_MASK)
|
||||
|
||||
|
||||
def is_dotnet_table_valid(pe: dnfile.dnPE, table_name: str) -> bool:
|
||||
return bool(getattr(pe.net.mdtables, table_name, None))
|
||||
|
||||
|
||||
def is_dotnet_mixed_mode(pe: dnfile.dnPE) -> bool:
|
||||
return not bool(pe.net.Flags.CLR_ILONLY)
|
||||
|
||||
|
||||
def iter_dotnet_table(pe: dnfile.dnPE, name: str) -> Iterator[Any]:
|
||||
if not is_dotnet_table_valid(pe, name):
|
||||
return
|
||||
for row in getattr(pe.net.mdtables, name):
|
||||
yield row
|
||||
@@ -1,182 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict, Tuple, Union, Iterator, Optional
|
||||
|
||||
import dnfile
|
||||
from dncil.cil.body import CilMethodBody
|
||||
from dncil.clr.token import Token, StringToken, InvalidToken
|
||||
from dncil.cil.opcode import OpCodes
|
||||
from dncil.cil.instruction import Instruction
|
||||
|
||||
import capa.features.extractors.helpers
|
||||
from capa.features.insn import API, Number
|
||||
from capa.features.common import Class, String, Feature, Namespace, Characteristic
|
||||
from capa.features.address import Address
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
|
||||
from capa.features.extractors.dnfile.helpers import (
|
||||
DnClass,
|
||||
DnMethod,
|
||||
DnUnmanagedMethod,
|
||||
resolve_dotnet_token,
|
||||
read_dotnet_user_string,
|
||||
get_dotnet_managed_imports,
|
||||
get_dotnet_managed_methods,
|
||||
get_dotnet_unmanaged_imports,
|
||||
)
|
||||
|
||||
|
||||
def get_managed_imports(ctx: Dict) -> Dict:
|
||||
if "managed_imports_cache" not in ctx:
|
||||
ctx["managed_imports_cache"] = {}
|
||||
for method in get_dotnet_managed_imports(ctx["pe"]):
|
||||
ctx["managed_imports_cache"][method.token] = method
|
||||
return ctx["managed_imports_cache"]
|
||||
|
||||
|
||||
def get_unmanaged_imports(ctx: Dict) -> Dict:
|
||||
if "unmanaged_imports_cache" not in ctx:
|
||||
ctx["unmanaged_imports_cache"] = {}
|
||||
for imp in get_dotnet_unmanaged_imports(ctx["pe"]):
|
||||
ctx["unmanaged_imports_cache"][imp.token] = imp
|
||||
return ctx["unmanaged_imports_cache"]
|
||||
|
||||
|
||||
def get_methods(ctx: Dict) -> Dict:
|
||||
if "methods_cache" not in ctx:
|
||||
ctx["methods_cache"] = {}
|
||||
for method in get_dotnet_managed_methods(ctx["pe"]):
|
||||
ctx["methods_cache"][method.token] = method
|
||||
return ctx["methods_cache"]
|
||||
|
||||
|
||||
def get_callee(ctx: Dict, token: int) -> Union[DnMethod, DnUnmanagedMethod, None]:
|
||||
"""map dotnet token to un/managed method"""
|
||||
callee: Union[DnMethod, DnUnmanagedMethod, None] = get_managed_imports(ctx).get(token, None)
|
||||
if not callee:
|
||||
# we must check unmanaged imports before managed methods because we map forwarded managed methods
|
||||
# to their unmanaged imports; we prefer a forwarded managed method be mapped to its unmanaged import for analysis
|
||||
callee = get_unmanaged_imports(ctx).get(token, None)
|
||||
if not callee:
|
||||
callee = get_methods(ctx).get(token, None)
|
||||
return callee
|
||||
|
||||
|
||||
def extract_insn_api_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse instruction API features"""
|
||||
insn: Instruction = ih.inner
|
||||
|
||||
if insn.opcode not in (OpCodes.Call, OpCodes.Callvirt, OpCodes.Jmp, OpCodes.Calli):
|
||||
return
|
||||
|
||||
callee: Union[DnMethod, DnUnmanagedMethod, None] = get_callee(fh.ctx, insn.operand.value)
|
||||
if callee is None:
|
||||
return
|
||||
|
||||
if isinstance(callee, DnUnmanagedMethod):
|
||||
# like kernel32.CreateFileA
|
||||
for name in capa.features.extractors.helpers.generate_symbols(callee.modulename, callee.methodname):
|
||||
yield API(name), ih.address
|
||||
else:
|
||||
# like System.IO.File::Delete
|
||||
yield API(str(callee)), ih.address
|
||||
|
||||
|
||||
def extract_insn_class_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Class, Address]]:
|
||||
"""parse instruction class features"""
|
||||
if ih.inner.opcode not in (OpCodes.Call, OpCodes.Callvirt, OpCodes.Jmp, OpCodes.Calli):
|
||||
return
|
||||
|
||||
row: Any = resolve_dotnet_token(fh.ctx["pe"], Token(ih.inner.operand.value))
|
||||
|
||||
if not isinstance(row, dnfile.mdtable.MemberRefRow):
|
||||
return
|
||||
if not isinstance(row.Class.row, (dnfile.mdtable.TypeRefRow, dnfile.mdtable.TypeDefRow)):
|
||||
return
|
||||
|
||||
yield Class(DnClass.format_name(row.Class.row.TypeNamespace, row.Class.row.TypeName)), ih.address
|
||||
|
||||
|
||||
def extract_insn_namespace_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Namespace, Address]]:
|
||||
"""parse instruction namespace features"""
|
||||
if ih.inner.opcode not in (OpCodes.Call, OpCodes.Callvirt, OpCodes.Jmp, OpCodes.Calli):
|
||||
return
|
||||
|
||||
row: Any = resolve_dotnet_token(fh.ctx["pe"], Token(ih.inner.operand.value))
|
||||
|
||||
if not isinstance(row, dnfile.mdtable.MemberRefRow):
|
||||
return
|
||||
if not isinstance(row.Class.row, (dnfile.mdtable.TypeRefRow, dnfile.mdtable.TypeDefRow)):
|
||||
return
|
||||
if not row.Class.row.TypeNamespace:
|
||||
return
|
||||
|
||||
yield Namespace(row.Class.row.TypeNamespace), ih.address
|
||||
|
||||
|
||||
def extract_insn_number_features(fh, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse instruction number features"""
|
||||
insn: Instruction = ih.inner
|
||||
|
||||
if insn.is_ldc():
|
||||
yield Number(insn.get_ldc()), ih.address
|
||||
|
||||
|
||||
def extract_insn_string_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse instruction string features"""
|
||||
f: CilMethodBody = fh.inner
|
||||
insn: Instruction = ih.inner
|
||||
|
||||
if not insn.is_ldstr():
|
||||
return
|
||||
|
||||
if not isinstance(insn.operand, StringToken):
|
||||
return
|
||||
|
||||
user_string: Optional[str] = read_dotnet_user_string(fh.ctx["pe"], insn.operand)
|
||||
if user_string is None:
|
||||
return
|
||||
|
||||
yield String(user_string), ih.address
|
||||
|
||||
|
||||
def extract_unmanaged_call_characteristic_features(
|
||||
fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Characteristic, Address]]:
|
||||
insn: Instruction = ih.inner
|
||||
if insn.opcode not in (OpCodes.Call, OpCodes.Callvirt, OpCodes.Jmp, OpCodes.Calli):
|
||||
return
|
||||
|
||||
token: Any = resolve_dotnet_token(fh.ctx["pe"], insn.operand)
|
||||
if isinstance(token, InvalidToken):
|
||||
return
|
||||
if not isinstance(token, dnfile.mdtable.MethodDefRow):
|
||||
return
|
||||
|
||||
if any((token.Flags.mdPinvokeImpl, token.ImplFlags.miUnmanaged, token.ImplFlags.miNative)):
|
||||
yield Characteristic("unmanaged call"), ih.address
|
||||
|
||||
|
||||
def extract_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""extract instruction features"""
|
||||
for inst_handler in INSTRUCTION_HANDLERS:
|
||||
for (feature, addr) in inst_handler(fh, bbh, ih):
|
||||
assert isinstance(addr, Address)
|
||||
yield feature, addr
|
||||
|
||||
|
||||
INSTRUCTION_HANDLERS = (
|
||||
extract_insn_api_features,
|
||||
extract_insn_number_features,
|
||||
extract_insn_string_features,
|
||||
extract_insn_namespace_features,
|
||||
extract_insn_class_features,
|
||||
extract_unmanaged_call_characteristic_features,
|
||||
)
|
||||
@@ -1,116 +0,0 @@
|
||||
import logging
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import dnfile
|
||||
import pefile
|
||||
|
||||
from capa.features.common import OS, OS_ANY, ARCH_ANY, ARCH_I386, ARCH_AMD64, FORMAT_DOTNET, Arch, Format, Feature
|
||||
from capa.features.address import NO_ADDRESS, Address, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import FeatureExtractor
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_file_format(**kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield Format(FORMAT_DOTNET), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_os(**kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield OS(OS_ANY), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_arch(pe, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
# to distinguish in more detail, see https://stackoverflow.com/a/23614024/10548020
|
||||
# .NET 4.5 added option: any CPU, 32-bit preferred
|
||||
if pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE:
|
||||
yield Arch(ARCH_I386), NO_ADDRESS
|
||||
elif not pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE_PLUS:
|
||||
yield Arch(ARCH_AMD64), NO_ADDRESS
|
||||
else:
|
||||
yield Arch(ARCH_ANY), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for feature, address in file_handler(pe=pe): # type: ignore
|
||||
yield feature, address
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
# extract_file_export_names,
|
||||
# extract_file_import_names,
|
||||
# extract_file_section_names,
|
||||
# extract_file_strings,
|
||||
# extract_file_function_names,
|
||||
extract_file_format,
|
||||
)
|
||||
|
||||
|
||||
def extract_global_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
|
||||
for handler in GLOBAL_HANDLERS:
|
||||
for feature, addr in handler(pe=pe): # type: ignore
|
||||
yield feature, addr
|
||||
|
||||
|
||||
GLOBAL_HANDLERS = (
|
||||
extract_file_os,
|
||||
extract_file_arch,
|
||||
)
|
||||
|
||||
|
||||
class DnfileFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, path: str):
|
||||
super(DnfileFeatureExtractor, self).__init__()
|
||||
self.path: str = path
|
||||
self.pe: dnfile.dnPE = dnfile.dnPE(path)
|
||||
|
||||
def get_base_address(self) -> AbsoluteVirtualAddress:
|
||||
return AbsoluteVirtualAddress(0x0)
|
||||
|
||||
def get_entry_point(self) -> int:
|
||||
# self.pe.net.Flags.CLT_NATIVE_ENTRYPOINT
|
||||
# True: native EP: Token
|
||||
# False: managed EP: RVA
|
||||
return self.pe.net.struct.EntryPointTokenOrRva
|
||||
|
||||
def extract_global_features(self):
|
||||
yield from extract_global_features(self.pe)
|
||||
|
||||
def extract_file_features(self):
|
||||
yield from extract_file_features(self.pe)
|
||||
|
||||
def is_dotnet_file(self) -> bool:
|
||||
return bool(self.pe.net)
|
||||
|
||||
def is_mixed_mode(self) -> bool:
|
||||
return not bool(self.pe.net.Flags.CLR_ILONLY)
|
||||
|
||||
def get_runtime_version(self) -> Tuple[int, int]:
|
||||
return self.pe.net.struct.MajorRuntimeVersion, self.pe.net.struct.MinorRuntimeVersion
|
||||
|
||||
def get_meta_version_string(self) -> str:
|
||||
return self.pe.net.metadata.struct.Version.rstrip(b"\x00").decode("utf-8")
|
||||
|
||||
def get_functions(self):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_function_features(self, f):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def is_library_function(self, va):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_function_name(self, va):
|
||||
raise NotImplementedError("DnfileFeatureExtractor can only be used to extract file features")
|
||||
@@ -1,203 +0,0 @@
|
||||
import logging
|
||||
import itertools
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import dnfile
|
||||
import pefile
|
||||
from dncil.clr.token import Token
|
||||
|
||||
import capa.features.extractors.helpers
|
||||
from capa.features.file import Import, FunctionName
|
||||
from capa.features.common import (
|
||||
OS,
|
||||
OS_ANY,
|
||||
ARCH_ANY,
|
||||
ARCH_I386,
|
||||
ARCH_AMD64,
|
||||
FORMAT_DOTNET,
|
||||
Arch,
|
||||
Class,
|
||||
Format,
|
||||
String,
|
||||
Feature,
|
||||
Namespace,
|
||||
Characteristic,
|
||||
)
|
||||
from capa.features.address import NO_ADDRESS, Address, DNTokenAddress, DNTokenOffsetAddress, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import FeatureExtractor
|
||||
from capa.features.extractors.dnfile.helpers import (
|
||||
DnClass,
|
||||
DnMethod,
|
||||
iter_dotnet_table,
|
||||
is_dotnet_mixed_mode,
|
||||
get_dotnet_managed_imports,
|
||||
get_dotnet_managed_methods,
|
||||
calculate_dotnet_token_value,
|
||||
get_dotnet_unmanaged_imports,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_file_format(**kwargs) -> Iterator[Tuple[Format, Address]]:
|
||||
yield Format(FORMAT_DOTNET), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_import_names(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Import, Address]]:
|
||||
for method in get_dotnet_managed_imports(pe):
|
||||
# like System.IO.File::OpenRead
|
||||
yield Import(str(method)), DNTokenAddress(Token(method.token))
|
||||
|
||||
for imp in get_dotnet_unmanaged_imports(pe):
|
||||
# like kernel32.CreateFileA
|
||||
for name in capa.features.extractors.helpers.generate_symbols(imp.modulename, imp.methodname):
|
||||
yield Import(name), DNTokenAddress(Token(imp.token))
|
||||
|
||||
|
||||
def extract_file_function_names(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[FunctionName, Address]]:
|
||||
for method in get_dotnet_managed_methods(pe):
|
||||
yield FunctionName(str(method)), DNTokenAddress(Token(method.token))
|
||||
|
||||
|
||||
def extract_file_namespace_features(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Namespace, Address]]:
|
||||
"""emit namespace features from TypeRef and TypeDef tables"""
|
||||
|
||||
# namespaces may be referenced multiple times, so we need to filter
|
||||
namespaces = set()
|
||||
|
||||
for row in iter_dotnet_table(pe, "TypeDef"):
|
||||
namespaces.add(row.TypeNamespace)
|
||||
|
||||
for row in iter_dotnet_table(pe, "TypeRef"):
|
||||
namespaces.add(row.TypeNamespace)
|
||||
|
||||
# namespaces may be empty, discard
|
||||
namespaces.discard("")
|
||||
|
||||
for namespace in namespaces:
|
||||
# namespace do not have an associated token, so we yield 0x0
|
||||
yield Namespace(namespace), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_class_features(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Class, Address]]:
|
||||
"""emit class features from TypeRef and TypeDef tables"""
|
||||
for (rid, row) in enumerate(iter_dotnet_table(pe, "TypeDef")):
|
||||
token = calculate_dotnet_token_value(pe.net.mdtables.TypeDef.number, rid + 1)
|
||||
yield Class(DnClass.format_name(row.TypeNamespace, row.TypeName)), DNTokenAddress(Token(token))
|
||||
|
||||
for (rid, row) in enumerate(iter_dotnet_table(pe, "TypeRef")):
|
||||
token = calculate_dotnet_token_value(pe.net.mdtables.TypeRef.number, rid + 1)
|
||||
yield Class(DnClass.format_name(row.TypeNamespace, row.TypeName)), DNTokenAddress(Token(token))
|
||||
|
||||
|
||||
def extract_file_os(**kwargs) -> Iterator[Tuple[OS, Address]]:
|
||||
yield OS(OS_ANY), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_arch(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Arch, Address]]:
|
||||
# to distinguish in more detail, see https://stackoverflow.com/a/23614024/10548020
|
||||
# .NET 4.5 added option: any CPU, 32-bit preferred
|
||||
if pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE:
|
||||
yield Arch(ARCH_I386), NO_ADDRESS
|
||||
elif not pe.net.Flags.CLR_32BITREQUIRED and pe.PE_TYPE == pefile.OPTIONAL_HEADER_MAGIC_PE_PLUS:
|
||||
yield Arch(ARCH_AMD64), NO_ADDRESS
|
||||
else:
|
||||
yield Arch(ARCH_ANY), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_strings(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[String, Address]]:
|
||||
yield from capa.features.extractors.common.extract_file_strings(pe.__data__)
|
||||
|
||||
|
||||
def extract_file_mixed_mode_characteristic_features(
|
||||
pe: dnfile.dnPE, **kwargs
|
||||
) -> Iterator[Tuple[Characteristic, Address]]:
|
||||
if is_dotnet_mixed_mode(pe):
|
||||
yield Characteristic("mixed mode"), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for feature, addr in file_handler(pe=pe): # type: ignore
|
||||
yield feature, addr
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
extract_file_import_names,
|
||||
extract_file_function_names,
|
||||
extract_file_strings,
|
||||
extract_file_format,
|
||||
extract_file_mixed_mode_characteristic_features,
|
||||
extract_file_namespace_features,
|
||||
extract_file_class_features,
|
||||
)
|
||||
|
||||
|
||||
def extract_global_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
|
||||
for handler in GLOBAL_HANDLERS:
|
||||
for feature, va in handler(pe=pe): # type: ignore
|
||||
yield feature, va
|
||||
|
||||
|
||||
GLOBAL_HANDLERS = (
|
||||
extract_file_os,
|
||||
extract_file_arch,
|
||||
)
|
||||
|
||||
|
||||
class DotnetFileFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, path: str):
|
||||
super(DotnetFileFeatureExtractor, self).__init__()
|
||||
self.path: str = path
|
||||
self.pe: dnfile.dnPE = dnfile.dnPE(path)
|
||||
|
||||
def get_base_address(self):
|
||||
return NO_ADDRESS
|
||||
|
||||
def get_entry_point(self) -> int:
|
||||
# self.pe.net.Flags.CLT_NATIVE_ENTRYPOINT
|
||||
# True: native EP: Token
|
||||
# False: managed EP: RVA
|
||||
return self.pe.net.struct.EntryPointTokenOrRva
|
||||
|
||||
def extract_global_features(self):
|
||||
yield from extract_global_features(self.pe)
|
||||
|
||||
def extract_file_features(self):
|
||||
yield from extract_file_features(self.pe)
|
||||
|
||||
def is_dotnet_file(self) -> bool:
|
||||
return bool(self.pe.net)
|
||||
|
||||
def is_mixed_mode(self) -> bool:
|
||||
return is_dotnet_mixed_mode(self.pe)
|
||||
|
||||
def get_runtime_version(self) -> Tuple[int, int]:
|
||||
return self.pe.net.struct.MajorRuntimeVersion, self.pe.net.struct.MinorRuntimeVersion
|
||||
|
||||
def get_meta_version_string(self) -> str:
|
||||
return self.pe.net.metadata.struct.Version.rstrip(b"\x00").decode("utf-8")
|
||||
|
||||
def get_functions(self):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_function_features(self, f):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def is_library_function(self, va):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_function_name(self, va):
|
||||
raise NotImplementedError("DotnetFileFeatureExtractor can only be used to extract file features")
|
||||
@@ -1,350 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
import struct
|
||||
import logging
|
||||
from enum import Enum
|
||||
from typing import BinaryIO
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def align(v, alignment):
|
||||
remainder = v % alignment
|
||||
if remainder == 0:
|
||||
return v
|
||||
else:
|
||||
return v + (alignment - remainder)
|
||||
|
||||
|
||||
class CorruptElfFile(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class OS(str, Enum):
|
||||
HPUX = "hpux"
|
||||
NETBSD = "netbsd"
|
||||
LINUX = "linux"
|
||||
HURD = "hurd"
|
||||
_86OPEN = "86open"
|
||||
SOLARIS = "solaris"
|
||||
AIX = "aix"
|
||||
IRIX = "irix"
|
||||
FREEBSD = "freebsd"
|
||||
TRU64 = "tru64"
|
||||
MODESTO = "modesto"
|
||||
OPENBSD = "openbsd"
|
||||
OPENVMS = "openvms"
|
||||
NSK = "nsk"
|
||||
AROS = "aros"
|
||||
FENIXOS = "fenixos"
|
||||
CLOUD = "cloud"
|
||||
SYLLABLE = "syllable"
|
||||
NACL = "nacl"
|
||||
|
||||
|
||||
# via readelf: https://github.com/bminor/binutils-gdb/blob/c0e94211e1ac05049a4ce7c192c9d14d1764eb3e/binutils/readelf.c#L19635-L19658
|
||||
# and here: https://github.com/bminor/binutils-gdb/blob/34c54daa337da9fadf87d2706d6a590ae1f88f4d/include/elf/common.h#L933-L939
|
||||
GNU_ABI_TAG = {
|
||||
0: OS.LINUX,
|
||||
1: OS.HURD,
|
||||
2: OS.SOLARIS,
|
||||
3: OS.FREEBSD,
|
||||
4: OS.NETBSD,
|
||||
5: OS.SYLLABLE,
|
||||
6: OS.NACL,
|
||||
}
|
||||
|
||||
|
||||
def detect_elf_os(f) -> str:
|
||||
"""
|
||||
f: type Union[BinaryIO, IDAIO]
|
||||
"""
|
||||
f.seek(0x0)
|
||||
file_header = f.read(0x40)
|
||||
|
||||
# we'll set this to the detected OS
|
||||
# prefer the first heuristics,
|
||||
# but rather than short circuiting,
|
||||
# we'll still parse out the remainder, for debugging.
|
||||
ret = None
|
||||
|
||||
if not file_header.startswith(b"\x7fELF"):
|
||||
raise CorruptElfFile("missing magic header")
|
||||
|
||||
ei_class, ei_data = struct.unpack_from("BB", file_header, 4)
|
||||
logger.debug("ei_class: 0x%02x ei_data: 0x%02x", ei_class, ei_data)
|
||||
if ei_class == 1:
|
||||
bitness = 32
|
||||
elif ei_class == 2:
|
||||
bitness = 64
|
||||
else:
|
||||
raise CorruptElfFile("invalid ei_class: 0x%02x" % ei_class)
|
||||
|
||||
if ei_data == 1:
|
||||
endian = "<"
|
||||
elif ei_data == 2:
|
||||
endian = ">"
|
||||
else:
|
||||
raise CorruptElfFile("not an ELF file: invalid ei_data: 0x%02x" % ei_data)
|
||||
|
||||
if bitness == 32:
|
||||
(e_phoff, e_shoff) = struct.unpack_from(endian + "II", file_header, 0x1C)
|
||||
e_phentsize, e_phnum = struct.unpack_from(endian + "HH", file_header, 0x2A)
|
||||
e_shentsize, e_shnum = struct.unpack_from(endian + "HH", file_header, 0x2E)
|
||||
elif bitness == 64:
|
||||
(e_phoff, e_shoff) = struct.unpack_from(endian + "QQ", file_header, 0x20)
|
||||
e_phentsize, e_phnum = struct.unpack_from(endian + "HH", file_header, 0x36)
|
||||
e_shentsize, e_shnum = struct.unpack_from(endian + "HH", file_header, 0x3A)
|
||||
else:
|
||||
raise NotImplementedError()
|
||||
|
||||
logger.debug("e_phoff: 0x%02x e_phentsize: 0x%02x e_phnum: %d", e_phoff, e_phentsize, e_phnum)
|
||||
|
||||
(ei_osabi,) = struct.unpack_from(endian + "B", file_header, 7)
|
||||
OSABI = {
|
||||
# via pyelftools: https://github.com/eliben/pyelftools/blob/0664de05ed2db3d39041e2d51d19622a8ef4fb0f/elftools/elf/enums.py#L35-L58
|
||||
# some candidates are commented out because the are not useful values,
|
||||
# at least when guessing OSes
|
||||
# 0: "SYSV", # too often used when OS is not SYSV
|
||||
1: OS.HPUX,
|
||||
2: OS.NETBSD,
|
||||
3: OS.LINUX,
|
||||
4: OS.HURD,
|
||||
5: OS._86OPEN,
|
||||
6: OS.SOLARIS,
|
||||
7: OS.AIX,
|
||||
8: OS.IRIX,
|
||||
9: OS.FREEBSD,
|
||||
10: OS.TRU64,
|
||||
11: OS.MODESTO,
|
||||
12: OS.OPENBSD,
|
||||
13: OS.OPENVMS,
|
||||
14: OS.NSK,
|
||||
15: OS.AROS,
|
||||
16: OS.FENIXOS,
|
||||
17: OS.CLOUD,
|
||||
# 53: "SORTFIX", # i can't find any reference to this OS, i dont think it exists
|
||||
# 64: "ARM_AEABI", # not an OS
|
||||
# 97: "ARM", # not an OS
|
||||
# 255: "STANDALONE", # not an OS
|
||||
}
|
||||
logger.debug("ei_osabi: 0x%02x (%s)", ei_osabi, OSABI.get(ei_osabi, "unknown"))
|
||||
|
||||
# os_osabi == 0 is commonly set even when the OS is not SYSV.
|
||||
# other values are unused or unknown.
|
||||
if ei_osabi in OSABI and ei_osabi != 0x0:
|
||||
# subsequent strategies may overwrite this value
|
||||
ret = OSABI[ei_osabi]
|
||||
|
||||
f.seek(e_phoff)
|
||||
program_header_size = e_phnum * e_phentsize
|
||||
program_headers = f.read(program_header_size)
|
||||
if len(program_headers) != program_header_size:
|
||||
logger.warning("failed to read program headers")
|
||||
e_phnum = 0
|
||||
|
||||
# search for PT_NOTE sections that specify an OS
|
||||
# for example, on Linux there is a GNU section with minimum kernel version
|
||||
for i in range(e_phnum):
|
||||
offset = i * e_phentsize
|
||||
phent = program_headers[offset : offset + e_phentsize]
|
||||
|
||||
PT_NOTE = 0x4
|
||||
|
||||
(p_type,) = struct.unpack_from(endian + "I", phent, 0x0)
|
||||
logger.debug("ph:p_type: 0x%04x", p_type)
|
||||
if p_type != PT_NOTE:
|
||||
continue
|
||||
|
||||
if bitness == 32:
|
||||
p_offset, _, _, p_filesz = struct.unpack_from(endian + "IIII", phent, 0x4)
|
||||
elif bitness == 64:
|
||||
p_offset, _, _, p_filesz = struct.unpack_from(endian + "QQQQ", phent, 0x8)
|
||||
else:
|
||||
raise NotImplementedError()
|
||||
|
||||
logger.debug("ph:p_offset: 0x%02x p_filesz: 0x%04x", p_offset, p_filesz)
|
||||
|
||||
f.seek(p_offset)
|
||||
note = f.read(p_filesz)
|
||||
if len(note) != p_filesz:
|
||||
logger.warning("failed to read note content")
|
||||
continue
|
||||
|
||||
namesz, descsz, type_ = struct.unpack_from(endian + "III", note, 0x0)
|
||||
name_offset = 0xC
|
||||
desc_offset = name_offset + align(namesz, 0x4)
|
||||
|
||||
logger.debug("ph:namesz: 0x%02x descsz: 0x%02x type: 0x%04x", namesz, descsz, type_)
|
||||
|
||||
name = note[name_offset : name_offset + namesz].partition(b"\x00")[0].decode("ascii")
|
||||
logger.debug("name: %s", name)
|
||||
|
||||
if type_ != 1:
|
||||
continue
|
||||
|
||||
if name == "GNU":
|
||||
if descsz < 16:
|
||||
continue
|
||||
|
||||
desc = note[desc_offset : desc_offset + descsz]
|
||||
abi_tag, kmajor, kminor, kpatch = struct.unpack_from(endian + "IIII", desc, 0x0)
|
||||
logger.debug("GNU_ABI_TAG: 0x%02x", abi_tag)
|
||||
|
||||
if abi_tag in GNU_ABI_TAG:
|
||||
# update only if not set
|
||||
# so we can get the debugging output of subsequent strategies
|
||||
ret = GNU_ABI_TAG[abi_tag] if not ret else ret
|
||||
logger.debug("abi tag: %s earliest compatible kernel: %d.%d.%d", ret, kmajor, kminor, kpatch)
|
||||
elif name == "OpenBSD":
|
||||
logger.debug("note owner: %s", "OPENBSD")
|
||||
ret = OS.OPENBSD if not ret else ret
|
||||
elif name == "NetBSD":
|
||||
logger.debug("note owner: %s", "NETBSD")
|
||||
ret = OS.NETBSD if not ret else ret
|
||||
elif name == "FreeBSD":
|
||||
logger.debug("note owner: %s", "FREEBSD")
|
||||
ret = OS.FREEBSD if not ret else ret
|
||||
|
||||
# search for recognizable dynamic linkers (interpreters)
|
||||
# for example, on linux, we see file paths like: /lib64/ld-linux-x86-64.so.2
|
||||
for i in range(e_phnum):
|
||||
offset = i * e_phentsize
|
||||
phent = program_headers[offset : offset + e_phentsize]
|
||||
|
||||
PT_INTERP = 0x3
|
||||
|
||||
(p_type,) = struct.unpack_from(endian + "I", phent, 0x0)
|
||||
if p_type != PT_INTERP:
|
||||
continue
|
||||
|
||||
if bitness == 32:
|
||||
p_offset, _, _, p_filesz = struct.unpack_from(endian + "IIII", phent, 0x4)
|
||||
elif bitness == 64:
|
||||
p_offset, _, _, p_filesz = struct.unpack_from(endian + "QQQQ", phent, 0x8)
|
||||
else:
|
||||
raise NotImplementedError()
|
||||
|
||||
f.seek(p_offset)
|
||||
interp = f.read(p_filesz)
|
||||
if len(interp) != p_filesz:
|
||||
logger.warning("failed to read interp content")
|
||||
continue
|
||||
|
||||
linker = interp.partition(b"\x00")[0].decode("ascii")
|
||||
logger.debug("linker: %s", linker)
|
||||
if "ld-linux" in linker:
|
||||
# update only if not set
|
||||
# so we can get the debugging output of subsequent strategies
|
||||
ret = OS.LINUX if ret is None else ret
|
||||
|
||||
f.seek(e_shoff)
|
||||
section_header_size = e_shnum * e_shentsize
|
||||
section_headers = f.read(section_header_size)
|
||||
if len(section_headers) != section_header_size:
|
||||
logger.warning("failed to read section headers")
|
||||
e_shnum = 0
|
||||
|
||||
# search for notes stored in sections that aren't visible in program headers.
|
||||
# e.g. .note.Linux in Linux kernel modules.
|
||||
for i in range(e_shnum):
|
||||
offset = i * e_shentsize
|
||||
shent = section_headers[offset : offset + e_shentsize]
|
||||
|
||||
if bitness == 32:
|
||||
sh_name, sh_type, _, sh_addr, sh_offset, sh_size = struct.unpack_from(endian + "IIIIII", shent, 0x0)
|
||||
elif bitness == 64:
|
||||
sh_name, sh_type, _, sh_addr, sh_offset, sh_size = struct.unpack_from(endian + "IIQQQQ", shent, 0x0)
|
||||
else:
|
||||
raise NotImplementedError()
|
||||
|
||||
SHT_NOTE = 0x7
|
||||
if sh_type != SHT_NOTE:
|
||||
continue
|
||||
|
||||
logger.debug("sh:sh_offset: 0x%02x sh_size: 0x%04x", sh_offset, sh_size)
|
||||
|
||||
f.seek(sh_offset)
|
||||
note = f.read(sh_size)
|
||||
if len(note) != sh_size:
|
||||
logger.warning("failed to read note content")
|
||||
continue
|
||||
|
||||
namesz, descsz, type_ = struct.unpack_from(endian + "III", note, 0x0)
|
||||
name_offset = 0xC
|
||||
desc_offset = name_offset + align(namesz, 0x4)
|
||||
|
||||
logger.debug("sh:namesz: 0x%02x descsz: 0x%02x type: 0x%04x", namesz, descsz, type_)
|
||||
|
||||
name = note[name_offset : name_offset + namesz].partition(b"\x00")[0].decode("ascii")
|
||||
logger.debug("name: %s", name)
|
||||
|
||||
if name == "Linux":
|
||||
logger.debug("note owner: %s", "LINUX")
|
||||
ret = OS.LINUX if not ret else ret
|
||||
elif name == "OpenBSD":
|
||||
logger.debug("note owner: %s", "OPENBSD")
|
||||
ret = OS.OPENBSD if not ret else ret
|
||||
elif name == "NetBSD":
|
||||
logger.debug("note owner: %s", "NETBSD")
|
||||
ret = OS.NETBSD if not ret else ret
|
||||
elif name == "FreeBSD":
|
||||
logger.debug("note owner: %s", "FREEBSD")
|
||||
ret = OS.FREEBSD if not ret else ret
|
||||
elif name == "GNU":
|
||||
if descsz < 16:
|
||||
continue
|
||||
|
||||
desc = note[desc_offset : desc_offset + descsz]
|
||||
abi_tag, kmajor, kminor, kpatch = struct.unpack_from(endian + "IIII", desc, 0x0)
|
||||
logger.debug("GNU_ABI_TAG: 0x%02x", abi_tag)
|
||||
|
||||
if abi_tag in GNU_ABI_TAG:
|
||||
# update only if not set
|
||||
# so we can get the debugging output of subsequent strategies
|
||||
ret = GNU_ABI_TAG[abi_tag] if not ret else ret
|
||||
logger.debug("abi tag: %s earliest compatible kernel: %d.%d.%d", ret, kmajor, kminor, kpatch)
|
||||
|
||||
return ret.value if ret is not None else "unknown"
|
||||
|
||||
|
||||
class Arch(str, Enum):
|
||||
I386 = "i386"
|
||||
AMD64 = "amd64"
|
||||
|
||||
|
||||
def detect_elf_arch(f: BinaryIO) -> str:
|
||||
f.seek(0x0)
|
||||
file_header = f.read(0x40)
|
||||
|
||||
if not file_header.startswith(b"\x7fELF"):
|
||||
raise CorruptElfFile("missing magic header")
|
||||
|
||||
(ei_data,) = struct.unpack_from("B", file_header, 5)
|
||||
logger.debug("ei_data: 0x%02x", ei_data)
|
||||
|
||||
if ei_data == 1:
|
||||
endian = "<"
|
||||
elif ei_data == 2:
|
||||
endian = ">"
|
||||
else:
|
||||
raise CorruptElfFile("not an ELF file: invalid ei_data: 0x%02x" % ei_data)
|
||||
|
||||
(ei_machine,) = struct.unpack_from(endian + "H", file_header, 0x12)
|
||||
logger.debug("ei_machine: 0x%02x", ei_machine)
|
||||
|
||||
EM_386 = 0x3
|
||||
EM_X86_64 = 0x3E
|
||||
if ei_machine == EM_386:
|
||||
return Arch.I386
|
||||
elif ei_machine == EM_X86_64:
|
||||
return Arch.AMD64
|
||||
else:
|
||||
# not really unknown, but unsupport at the moment:
|
||||
# https://github.com/eliben/pyelftools/blob/ab444d982d1849191e910299a985989857466620/elftools/elf/enums.py#L73
|
||||
return "unknown"
|
||||
@@ -1,160 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
import io
|
||||
import logging
|
||||
import contextlib
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
from elftools.elf.elffile import ELFFile, SymbolTableSection
|
||||
|
||||
import capa.features.extractors.common
|
||||
from capa.features.file import Import, Section
|
||||
from capa.features.common import OS, FORMAT_ELF, Arch, Format, Feature
|
||||
from capa.features.address import NO_ADDRESS, FileOffsetAddress, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.elf import Arch as ElfArch
|
||||
from capa.features.extractors.base_extractor import FeatureExtractor
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_file_import_names(elf, **kwargs):
|
||||
# see https://github.com/eliben/pyelftools/blob/0664de05ed2db3d39041e2d51d19622a8ef4fb0f/scripts/readelf.py#L372
|
||||
symbol_tables = [(idx, s) for idx, s in enumerate(elf.iter_sections()) if isinstance(s, SymbolTableSection)]
|
||||
|
||||
for section_index, section in symbol_tables:
|
||||
if not isinstance(section, SymbolTableSection):
|
||||
continue
|
||||
|
||||
if section["sh_entsize"] == 0:
|
||||
logger.debug("Symbol table '%s' has a sh_entsize of zero!" % (section.name))
|
||||
continue
|
||||
|
||||
logger.debug("Symbol table '%s' contains %s entries:" % (section.name, section.num_symbols()))
|
||||
|
||||
for nsym, symbol in enumerate(section.iter_symbols()):
|
||||
if symbol.name and symbol.entry.st_info.type == "STT_FUNC":
|
||||
# TODO symbol address
|
||||
# TODO symbol version info?
|
||||
yield Import(symbol.name), FileOffsetAddress(0x0)
|
||||
|
||||
|
||||
def extract_file_section_names(elf, **kwargs):
|
||||
for section in elf.iter_sections():
|
||||
if section.name:
|
||||
yield Section(section.name), AbsoluteVirtualAddress(section.header.sh_addr)
|
||||
elif section.is_null():
|
||||
yield Section("NULL"), AbsoluteVirtualAddress(section.header.sh_addr)
|
||||
|
||||
|
||||
def extract_file_strings(buf, **kwargs):
|
||||
yield from capa.features.extractors.common.extract_file_strings(buf)
|
||||
|
||||
|
||||
def extract_file_os(elf, buf, **kwargs):
|
||||
# our current approach does not always get an OS value, e.g. for packed samples
|
||||
# for file limitation purposes, we're more lax here
|
||||
try:
|
||||
os_tuple = next(capa.features.extractors.common.extract_os(buf))
|
||||
yield os_tuple
|
||||
except StopIteration:
|
||||
yield OS("unknown"), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_format(**kwargs):
|
||||
yield Format(FORMAT_ELF), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_arch(elf, **kwargs):
|
||||
# TODO merge with capa.features.extractors.elf.detect_elf_arch()
|
||||
arch = elf.get_machine_arch()
|
||||
if arch == "x86":
|
||||
yield Arch(ElfArch.I386), NO_ADDRESS
|
||||
elif arch == "x64":
|
||||
yield Arch(ElfArch.AMD64), NO_ADDRESS
|
||||
else:
|
||||
logger.warning("unsupported architecture: %s", arch)
|
||||
|
||||
|
||||
def extract_file_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for feature, addr in file_handler(elf=elf, buf=buf): # type: ignore
|
||||
yield feature, addr
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
# TODO extract_file_export_names,
|
||||
extract_file_import_names,
|
||||
extract_file_section_names,
|
||||
extract_file_strings,
|
||||
# no library matching
|
||||
extract_file_format,
|
||||
)
|
||||
|
||||
|
||||
def extract_global_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
|
||||
for global_handler in GLOBAL_HANDLERS:
|
||||
for feature, addr in global_handler(elf=elf, buf=buf): # type: ignore
|
||||
yield feature, addr
|
||||
|
||||
|
||||
GLOBAL_HANDLERS = (
|
||||
extract_file_os,
|
||||
extract_file_arch,
|
||||
)
|
||||
|
||||
|
||||
class ElfFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, path: str):
|
||||
super(ElfFeatureExtractor, self).__init__()
|
||||
self.path = path
|
||||
with open(self.path, "rb") as f:
|
||||
self.elf = ELFFile(io.BytesIO(f.read()))
|
||||
|
||||
def get_base_address(self):
|
||||
# virtual address of the first segment with type LOAD
|
||||
for segment in self.elf.iter_segments():
|
||||
if segment.header.p_type == "PT_LOAD":
|
||||
return AbsoluteVirtualAddress(segment.header.p_vaddr)
|
||||
|
||||
def extract_global_features(self):
|
||||
with open(self.path, "rb") as f:
|
||||
buf = f.read()
|
||||
|
||||
for feature, addr in extract_global_features(self.elf, buf):
|
||||
yield feature, addr
|
||||
|
||||
def extract_file_features(self):
|
||||
with open(self.path, "rb") as f:
|
||||
buf = f.read()
|
||||
|
||||
for feature, addr in extract_file_features(self.elf, buf):
|
||||
yield feature, addr
|
||||
|
||||
def get_functions(self):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_function_features(self, f):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def is_library_function(self, va):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
|
||||
def get_function_name(self, va):
|
||||
raise NotImplementedError("ElfFeatureExtractor can only be used to extract file features")
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,18 +6,23 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import struct
|
||||
import sys
|
||||
import builtins
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
from capa.features.file import Import
|
||||
from capa.features.insn import API
|
||||
|
||||
MIN_STACKSTRING_LEN = 8
|
||||
|
||||
|
||||
def xor_static(data: bytes, i: int) -> bytes:
|
||||
return bytes(c ^ i for c in data)
|
||||
def xor_static(data, i):
|
||||
if sys.version_info >= (3, 0):
|
||||
return bytes(c ^ i for c in data)
|
||||
else:
|
||||
return "".join(chr(ord(c) ^ i) for c in data)
|
||||
|
||||
|
||||
def is_aw_function(symbol: str) -> bool:
|
||||
def is_aw_function(symbol):
|
||||
"""
|
||||
is the given function name an A/W function?
|
||||
these are variants of functions that, on Windows, accept either a narrow or wide string.
|
||||
@@ -29,10 +34,11 @@ def is_aw_function(symbol: str) -> bool:
|
||||
if symbol[-1] not in ("A", "W"):
|
||||
return False
|
||||
|
||||
return True
|
||||
# second to last character should be lowercase letter
|
||||
return "a" <= symbol[-2] <= "z" or "0" <= symbol[-2] <= "9"
|
||||
|
||||
|
||||
def is_ordinal(symbol: str) -> bool:
|
||||
def is_ordinal(symbol):
|
||||
"""
|
||||
is the given symbol an ordinal that is prefixed by "#"?
|
||||
"""
|
||||
@@ -41,7 +47,7 @@ def is_ordinal(symbol: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def generate_symbols(dll: str, symbol: str) -> Iterator[str]:
|
||||
def generate_symbols(dll, symbol):
|
||||
"""
|
||||
for a given dll and symbol name, generate variants.
|
||||
we over-generate features to make matching easier.
|
||||
@@ -51,9 +57,6 @@ def generate_symbols(dll: str, symbol: str) -> Iterator[str]:
|
||||
- CreateFileA
|
||||
- CreateFile
|
||||
"""
|
||||
# normalize dll name
|
||||
dll = dll.lower()
|
||||
|
||||
# kernel32.CreateFileA
|
||||
yield "%s.%s" % (dll, symbol)
|
||||
|
||||
@@ -70,11 +73,11 @@ def generate_symbols(dll: str, symbol: str) -> Iterator[str]:
|
||||
yield symbol[:-1]
|
||||
|
||||
|
||||
def all_zeros(bytez: bytes) -> bool:
|
||||
def all_zeros(bytez):
|
||||
return all(b == 0 for b in builtins.bytes(bytez))
|
||||
|
||||
|
||||
def twos_complement(val: int, bits: int) -> int:
|
||||
def twos_complement(val, bits):
|
||||
"""
|
||||
compute the 2's complement of int value val
|
||||
|
||||
@@ -87,49 +90,3 @@ def twos_complement(val: int, bits: int) -> int:
|
||||
else:
|
||||
# return positive value as is
|
||||
return val
|
||||
|
||||
|
||||
def carve_pe(pbytes: bytes, offset: int = 0) -> Iterator[Tuple[int, int]]:
|
||||
"""
|
||||
Generate (offset, key) tuples of embedded PEs
|
||||
|
||||
Based on the version from vivisect:
|
||||
https://github.com/vivisect/vivisect/blob/7be4037b1cecc4551b397f840405a1fc606f9b53/PE/carve.py#L19
|
||||
And its IDA adaptation:
|
||||
capa/features/extractors/ida/file.py
|
||||
"""
|
||||
mz_xor = [
|
||||
(
|
||||
xor_static(b"MZ", key),
|
||||
xor_static(b"PE", key),
|
||||
key,
|
||||
)
|
||||
for key in range(256)
|
||||
]
|
||||
|
||||
pblen = len(pbytes)
|
||||
todo = [(pbytes.find(mzx, offset), mzx, pex, key) for mzx, pex, key in mz_xor]
|
||||
todo = [(off, mzx, pex, key) for (off, mzx, pex, key) in todo if off != -1]
|
||||
|
||||
while len(todo):
|
||||
|
||||
off, mzx, pex, key = todo.pop()
|
||||
|
||||
# The MZ header has one field we will check
|
||||
# e_lfanew is at 0x3c
|
||||
e_lfanew = off + 0x3C
|
||||
if pblen < (e_lfanew + 4):
|
||||
continue
|
||||
|
||||
newoff = struct.unpack("<I", xor_static(pbytes[e_lfanew : e_lfanew + 4], key))[0]
|
||||
|
||||
nextres = pbytes.find(mzx, off + 1)
|
||||
if nextres != -1:
|
||||
todo.append((nextres, mzx, pex, key))
|
||||
|
||||
peoff = off + newoff
|
||||
if pblen < (peoff + 2):
|
||||
continue
|
||||
|
||||
if pbytes[peoff : peoff + 2] == pex:
|
||||
yield (off, key)
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import idaapi
|
||||
|
||||
import capa.features.extractors.ida.file
|
||||
import capa.features.extractors.ida.insn
|
||||
import capa.features.extractors.ida.function
|
||||
import capa.features.extractors.ida.basicblock
|
||||
from capa.features.extractors import FeatureExtractor
|
||||
|
||||
|
||||
def get_ea(self):
|
||||
""" """
|
||||
if isinstance(self, (idaapi.BasicBlock, idaapi.func_t)):
|
||||
return self.start_ea
|
||||
if isinstance(self, idaapi.insn_t):
|
||||
return self.ea
|
||||
raise TypeError
|
||||
|
||||
|
||||
def add_ea_int_cast(o):
|
||||
"""
|
||||
dynamically add a cast-to-int (`__int__`) method to the given object
|
||||
that returns the value of the `.ea` property.
|
||||
this bit of skullduggery lets use cast viv-utils objects as ints.
|
||||
the correct way of doing this is to update viv-utils (or subclass the objects here).
|
||||
"""
|
||||
if sys.version_info[0] >= 3:
|
||||
setattr(o, "__int__", types.MethodType(get_ea, o))
|
||||
else:
|
||||
setattr(o, "__int__", types.MethodType(get_ea, o, type(o)))
|
||||
return o
|
||||
|
||||
|
||||
class IdaFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self):
|
||||
super(IdaFeatureExtractor, self).__init__()
|
||||
|
||||
def get_base_address(self):
|
||||
return idaapi.get_imagebase()
|
||||
|
||||
def extract_file_features(self):
|
||||
for (feature, ea) in capa.features.extractors.ida.file.extract_features():
|
||||
yield feature, ea
|
||||
|
||||
def get_functions(self):
|
||||
import capa.features.extractors.ida.helpers as ida_helpers
|
||||
|
||||
# data structure shared across functions yielded here.
|
||||
# useful for caching analysis relevant across a single workspace.
|
||||
ctx = {}
|
||||
|
||||
# ignore library functions and thunk functions as identified by IDA
|
||||
for f in ida_helpers.get_functions(skip_thunks=True, skip_libs=True):
|
||||
setattr(f, "ctx", ctx)
|
||||
yield add_ea_int_cast(f)
|
||||
|
||||
@staticmethod
|
||||
def get_function(ea):
|
||||
f = idaapi.get_func(ea)
|
||||
setattr(f, "ctx", {})
|
||||
return add_ea_int_cast(f)
|
||||
|
||||
def extract_function_features(self, f):
|
||||
for (feature, ea) in capa.features.extractors.ida.function.extract_features(f):
|
||||
yield feature, ea
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
for bb in capa.features.extractors.ida.helpers.get_function_blocks(f):
|
||||
yield add_ea_int_cast(bb)
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
for (feature, ea) in capa.features.extractors.ida.basicblock.extract_features(f, bb):
|
||||
yield feature, ea
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
import capa.features.extractors.ida.helpers as ida_helpers
|
||||
|
||||
for insn in ida_helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
|
||||
yield add_ea_int_cast(insn)
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
for (feature, ea) in capa.features.extractors.ida.insn.extract_features(f, bb, insn):
|
||||
yield feature, ea
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,23 +6,25 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import sys
|
||||
import string
|
||||
import struct
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import idaapi
|
||||
|
||||
import capa.features.extractors.ida.helpers
|
||||
from capa.features.common import Feature, Characteristic
|
||||
from capa.features.address import Address
|
||||
from capa.features import Characteristic
|
||||
from capa.features.basicblock import BasicBlock
|
||||
from capa.features.extractors.ida import helpers
|
||||
from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
|
||||
from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
|
||||
|
||||
|
||||
def get_printable_len(op: idaapi.op_t) -> int:
|
||||
"""Return string length if all operand bytes are ascii or utf16-le printable"""
|
||||
def get_printable_len(op):
|
||||
"""Return string length if all operand bytes are ascii or utf16-le printable
|
||||
|
||||
args:
|
||||
op (IDA op_t)
|
||||
"""
|
||||
op_val = capa.features.extractors.ida.helpers.mask_op_val(op)
|
||||
|
||||
if op.dtype == idaapi.dt_byte:
|
||||
@@ -36,12 +38,19 @@ def get_printable_len(op: idaapi.op_t) -> int:
|
||||
else:
|
||||
raise ValueError("Unhandled operand data type 0x%x." % op.dtype)
|
||||
|
||||
def is_printable_ascii(chars_: bytes):
|
||||
return all(c < 127 and chr(c) in string.printable for c in chars_)
|
||||
def is_printable_ascii(chars):
|
||||
if sys.version_info[0] >= 3:
|
||||
return all(c < 127 and chr(c) in string.printable for c in chars)
|
||||
else:
|
||||
return all(ord(c) < 127 and c in string.printable for c in chars)
|
||||
|
||||
def is_printable_utf16le(chars_: bytes):
|
||||
if all(c == 0x00 for c in chars_[1::2]):
|
||||
return is_printable_ascii(chars_[::2])
|
||||
def is_printable_utf16le(chars):
|
||||
if sys.version_info[0] >= 3:
|
||||
if all(c == 0x00 for c in chars[1::2]):
|
||||
return is_printable_ascii(chars[::2])
|
||||
else:
|
||||
if all(c == "\x00" for c in chars[1::2]):
|
||||
return is_printable_ascii(chars[::2])
|
||||
|
||||
if is_printable_ascii(chars):
|
||||
return idaapi.get_dtype_size(op.dtype)
|
||||
@@ -52,8 +61,12 @@ def get_printable_len(op: idaapi.op_t) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def is_mov_imm_to_stack(insn: idaapi.insn_t) -> bool:
|
||||
"""verify instruction moves immediate onto stack"""
|
||||
def is_mov_imm_to_stack(insn):
|
||||
"""verify instruction moves immediate onto stack
|
||||
|
||||
args:
|
||||
insn (IDA insn_t)
|
||||
"""
|
||||
if insn.Op2.type != idaapi.o_imm:
|
||||
return False
|
||||
|
||||
@@ -66,10 +79,14 @@ def is_mov_imm_to_stack(insn: idaapi.insn_t) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def bb_contains_stackstring(f: idaapi.func_t, bb: idaapi.BasicBlock) -> bool:
|
||||
def bb_contains_stackstring(f, bb):
|
||||
"""check basic block for stackstring indicators
|
||||
|
||||
true if basic block contains enough moves of constant bytes to the stack
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
"""
|
||||
count = 0
|
||||
for insn in capa.features.extractors.ida.helpers.get_instructions_in_range(bb.start_ea, bb.end_ea):
|
||||
@@ -80,24 +97,39 @@ def bb_contains_stackstring(f: idaapi.func_t, bb: idaapi.BasicBlock) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""extract stackstring indicators from basic block"""
|
||||
if bb_contains_stackstring(fh.inner, bbh.inner):
|
||||
yield Characteristic("stack string"), bbh.address
|
||||
def extract_bb_stackstring(f, bb):
|
||||
"""extract stackstring indicators from basic block
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
"""
|
||||
if bb_contains_stackstring(f, bb):
|
||||
yield Characteristic("stack string"), bb.start_ea
|
||||
|
||||
|
||||
def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""extract tight loop indicators from a basic block"""
|
||||
if capa.features.extractors.ida.helpers.is_basic_block_tight_loop(bbh.inner):
|
||||
yield Characteristic("tight loop"), bbh.address
|
||||
def extract_bb_tight_loop(f, bb):
|
||||
"""extract tight loop indicators from a basic block
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
"""
|
||||
if capa.features.extractors.ida.helpers.is_basic_block_tight_loop(bb):
|
||||
yield Characteristic("tight loop"), bb.start_ea
|
||||
|
||||
|
||||
def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""extract basic block features"""
|
||||
def extract_features(f, bb):
|
||||
"""extract basic block features
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
"""
|
||||
for bb_handler in BASIC_BLOCK_HANDLERS:
|
||||
for (feature, addr) in bb_handler(fh, bbh):
|
||||
yield feature, addr
|
||||
yield BasicBlock(), bbh.address
|
||||
for (feature, ea) in bb_handler(f, bb):
|
||||
yield feature, ea
|
||||
yield BasicBlock(), bb.start_ea
|
||||
|
||||
|
||||
BASIC_BLOCK_HANDLERS = (
|
||||
@@ -108,10 +140,9 @@ BASIC_BLOCK_HANDLERS = (
|
||||
|
||||
def main():
|
||||
features = []
|
||||
for fhandle in helpers.get_functions(skip_thunks=True, skip_libs=True):
|
||||
f: idaapi.func_t = fhandle.inner
|
||||
for f in helpers.get_functions(skip_thunks=True, skip_libs=True):
|
||||
for bb in idaapi.FlowChart(f, flags=idaapi.FC_PREDS):
|
||||
features.extend(list(extract_features(fhandle, bb)))
|
||||
features.extend(list(extract_features(f, bb)))
|
||||
|
||||
import pprint
|
||||
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import List, Tuple, Iterator
|
||||
|
||||
import idaapi
|
||||
|
||||
import capa.ida.helpers
|
||||
import capa.features.extractors.elf
|
||||
import capa.features.extractors.ida.file
|
||||
import capa.features.extractors.ida.insn
|
||||
import capa.features.extractors.ida.global_
|
||||
import capa.features.extractors.ida.function
|
||||
import capa.features.extractors.ida.basicblock
|
||||
from capa.features.common import Feature
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
|
||||
|
||||
|
||||
class IdaFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self):
|
||||
super(IdaFeatureExtractor, self).__init__()
|
||||
self.global_features: List[Tuple[Feature, Address]] = []
|
||||
self.global_features.extend(capa.features.extractors.ida.global_.extract_os())
|
||||
self.global_features.extend(capa.features.extractors.ida.global_.extract_arch())
|
||||
|
||||
def get_base_address(self):
|
||||
return AbsoluteVirtualAddress(idaapi.get_imagebase())
|
||||
|
||||
def extract_global_features(self):
|
||||
yield from self.global_features
|
||||
|
||||
def extract_file_features(self):
|
||||
yield from capa.features.extractors.ida.file.extract_features()
|
||||
|
||||
def get_functions(self) -> Iterator[FunctionHandle]:
|
||||
import capa.features.extractors.ida.helpers as ida_helpers
|
||||
|
||||
# ignore library functions and thunk functions as identified by IDA
|
||||
yield from ida_helpers.get_functions(skip_thunks=True, skip_libs=True)
|
||||
|
||||
@staticmethod
|
||||
def get_function(ea: int) -> FunctionHandle:
|
||||
f = idaapi.get_func(ea)
|
||||
return FunctionHandle(address=AbsoluteVirtualAddress(f.start_ea), inner=f)
|
||||
|
||||
def extract_function_features(self, fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.ida.function.extract_features(fh)
|
||||
|
||||
def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
|
||||
import capa.features.extractors.ida.helpers as ida_helpers
|
||||
|
||||
for bb in ida_helpers.get_function_blocks(fh.inner):
|
||||
yield BBHandle(address=AbsoluteVirtualAddress(bb.start_ea), inner=bb)
|
||||
|
||||
def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.ida.basicblock.extract_features(fh, bbh)
|
||||
|
||||
def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:
|
||||
import capa.features.extractors.ida.helpers as ida_helpers
|
||||
|
||||
for insn in ida_helpers.get_instructions_in_range(bbh.inner.start_ea, bbh.inner.end_ea):
|
||||
yield InsnHandle(address=AbsoluteVirtualAddress(insn.ea), inner=insn)
|
||||
|
||||
def extract_insn_features(self, fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle):
|
||||
yield from capa.features.extractors.ida.insn.extract_features(fh, bbh, ih)
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -7,26 +7,26 @@
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import struct
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import idc
|
||||
import idaapi
|
||||
import idautils
|
||||
|
||||
import capa.features.extractors.common
|
||||
import capa.features.extractors.helpers
|
||||
import capa.features.extractors.strings
|
||||
import capa.features.extractors.ida.helpers
|
||||
from capa.features.file import Export, Import, Section, FunctionName
|
||||
from capa.features.common import FORMAT_PE, FORMAT_ELF, Format, String, Feature, Characteristic
|
||||
from capa.features.address import NO_ADDRESS, Address, FileOffsetAddress, AbsoluteVirtualAddress
|
||||
from capa.features import String, Characteristic
|
||||
from capa.features.file import Export, Import, Section
|
||||
|
||||
|
||||
def check_segment_for_pe(seg: idaapi.segment_t) -> Iterator[Tuple[int, int]]:
|
||||
def check_segment_for_pe(seg):
|
||||
"""check segment for embedded PE
|
||||
|
||||
adapted for IDA from:
|
||||
https://github.com/vivisect/vivisect/blob/7be4037b1cecc4551b397f840405a1fc606f9b53/PE/carve.py#L19
|
||||
|
||||
args:
|
||||
seg (IDA segment_t)
|
||||
"""
|
||||
seg_max = seg.end_ea
|
||||
mz_xor = [
|
||||
@@ -59,13 +59,13 @@ def check_segment_for_pe(seg: idaapi.segment_t) -> Iterator[Tuple[int, int]]:
|
||||
continue
|
||||
|
||||
if idc.get_bytes(peoff, 2) == pex:
|
||||
yield off, i
|
||||
yield (off, i)
|
||||
|
||||
for nextres in capa.features.extractors.ida.helpers.find_byte_sequence(off + 1, seg.end_ea, mzx):
|
||||
todo.append((nextres, mzx, pex, i))
|
||||
|
||||
|
||||
def extract_file_embedded_pe() -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_file_embedded_pe():
|
||||
"""extract embedded PE features
|
||||
|
||||
IDA must load resource sections for this to be complete
|
||||
@@ -74,16 +74,16 @@ def extract_file_embedded_pe() -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
for seg in capa.features.extractors.ida.helpers.get_segments(skip_header_segments=True):
|
||||
for (ea, _) in check_segment_for_pe(seg):
|
||||
yield Characteristic("embedded pe"), FileOffsetAddress(ea)
|
||||
yield Characteristic("embedded pe"), ea
|
||||
|
||||
|
||||
def extract_file_export_names() -> Iterator[Tuple[Feature, Address]]:
|
||||
"""extract function exports"""
|
||||
def extract_file_export_names():
|
||||
""" extract function exports """
|
||||
for (_, _, ea, name) in idautils.Entries():
|
||||
yield Export(name), AbsoluteVirtualAddress(ea)
|
||||
yield Export(name), ea
|
||||
|
||||
|
||||
def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_file_import_names():
|
||||
"""extract function imports
|
||||
|
||||
1. imports by ordinal:
|
||||
@@ -95,12 +95,11 @@ def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
|
||||
- importname
|
||||
"""
|
||||
for (ea, info) in capa.features.extractors.ida.helpers.get_file_imports().items():
|
||||
addr = AbsoluteVirtualAddress(ea)
|
||||
if info[1] and info[2]:
|
||||
# e.g. in mimikatz: ('cabinet', 'FCIAddFile', 11L)
|
||||
# extract by name here and by ordinal below
|
||||
for name in capa.features.extractors.helpers.generate_symbols(info[0], info[1]):
|
||||
yield Import(name), addr
|
||||
yield Import(name), ea
|
||||
dll = info[0]
|
||||
symbol = "#%d" % (info[2])
|
||||
elif info[1]:
|
||||
@@ -113,10 +112,10 @@ def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
|
||||
continue
|
||||
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
|
||||
yield Import(name), addr
|
||||
yield Import(name), ea
|
||||
|
||||
|
||||
def extract_file_section_names() -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_file_section_names():
|
||||
"""extract section names
|
||||
|
||||
IDA must load resource sections for this to be complete
|
||||
@@ -124,10 +123,10 @@ def extract_file_section_names() -> Iterator[Tuple[Feature, Address]]:
|
||||
- Check 'Load resource sections' when opening binary in IDA manually
|
||||
"""
|
||||
for seg in capa.features.extractors.ida.helpers.get_segments(skip_header_segments=True):
|
||||
yield Section(idaapi.get_segm_name(seg)), AbsoluteVirtualAddress(seg.start_ea)
|
||||
yield Section(idaapi.get_segm_name(seg)), seg.start_ea
|
||||
|
||||
|
||||
def extract_file_strings() -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_file_strings():
|
||||
"""extract ASCII and UTF-16 LE strings
|
||||
|
||||
IDA must load resource sections for this to be complete
|
||||
@@ -137,50 +136,18 @@ def extract_file_strings() -> Iterator[Tuple[Feature, Address]]:
|
||||
for seg in capa.features.extractors.ida.helpers.get_segments():
|
||||
seg_buff = capa.features.extractors.ida.helpers.get_segment_buffer(seg)
|
||||
|
||||
# differing to common string extractor factor in segment offset here
|
||||
for s in capa.features.extractors.strings.extract_ascii_strings(seg_buff):
|
||||
yield String(s.s), FileOffsetAddress(seg.start_ea + s.offset)
|
||||
yield String(s.s), (seg.start_ea + s.offset)
|
||||
|
||||
for s in capa.features.extractors.strings.extract_unicode_strings(seg_buff):
|
||||
yield String(s.s), FileOffsetAddress(seg.start_ea + s.offset)
|
||||
yield String(s.s), (seg.start_ea + s.offset)
|
||||
|
||||
|
||||
def extract_file_function_names() -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
extract the names of statically-linked library functions.
|
||||
"""
|
||||
for ea in idautils.Functions():
|
||||
addr = AbsoluteVirtualAddress(ea)
|
||||
if idaapi.get_func(ea).flags & idaapi.FUNC_LIB:
|
||||
name = idaapi.get_name(ea)
|
||||
yield FunctionName(name), addr
|
||||
if name.startswith("_"):
|
||||
# some linkers may prefix linked routines with a `_` to avoid name collisions.
|
||||
# extract features for both the mangled and un-mangled representations.
|
||||
# e.g. `_fwrite` -> `fwrite`
|
||||
# see: https://stackoverflow.com/a/2628384/87207
|
||||
yield FunctionName(name[1:]), addr
|
||||
|
||||
|
||||
def extract_file_format() -> Iterator[Tuple[Feature, Address]]:
|
||||
file_info = idaapi.get_inf_structure()
|
||||
|
||||
if file_info.filetype == idaapi.f_PE:
|
||||
yield Format(FORMAT_PE), NO_ADDRESS
|
||||
elif file_info.filetype == idaapi.f_ELF:
|
||||
yield Format(FORMAT_ELF), NO_ADDRESS
|
||||
elif file_info.filetype == idaapi.f_BIN:
|
||||
# no file type to return when processing a binary file, but we want to continue processing
|
||||
return
|
||||
else:
|
||||
raise NotImplementedError("file format: %d" % file_info.filetype)
|
||||
|
||||
|
||||
def extract_features() -> Iterator[Tuple[Feature, Address]]:
|
||||
"""extract file features"""
|
||||
def extract_features():
|
||||
""" extract file features """
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for feature, addr in file_handler():
|
||||
yield feature, addr
|
||||
for feature, va in file_handler():
|
||||
yield feature, va
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
@@ -189,8 +156,6 @@ FILE_HANDLERS = (
|
||||
extract_file_strings,
|
||||
extract_file_section_names,
|
||||
extract_file_embedded_pe,
|
||||
extract_file_function_names,
|
||||
extract_file_format,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,31 +1,35 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import idaapi
|
||||
import idautils
|
||||
|
||||
import capa.features.extractors.ida.helpers
|
||||
from capa.features.common import Feature, Characteristic
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features import Characteristic
|
||||
from capa.features.extractors import loops
|
||||
from capa.features.extractors.base_extractor import FunctionHandle
|
||||
|
||||
|
||||
def extract_function_calls_to(fh: FunctionHandle):
|
||||
"""extract callers to a function"""
|
||||
for ea in idautils.CodeRefsTo(fh.inner.start_ea, True):
|
||||
yield Characteristic("calls to"), AbsoluteVirtualAddress(ea)
|
||||
def extract_function_calls_to(f):
|
||||
"""extract callers to a function
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
"""
|
||||
for ea in idautils.CodeRefsTo(f.start_ea, True):
|
||||
yield Characteristic("calls to"), ea
|
||||
|
||||
|
||||
def extract_function_loop(fh: FunctionHandle):
|
||||
"""extract loop indicators from a function"""
|
||||
f: idaapi.func_t = fh.inner
|
||||
def extract_function_loop(f):
|
||||
"""extract loop indicators from a function
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
"""
|
||||
edges = []
|
||||
|
||||
# construct control flow graph
|
||||
@@ -34,19 +38,28 @@ def extract_function_loop(fh: FunctionHandle):
|
||||
edges.append((bb.start_ea, succ.start_ea))
|
||||
|
||||
if loops.has_loop(edges):
|
||||
yield Characteristic("loop"), fh.address
|
||||
yield Characteristic("loop"), f.start_ea
|
||||
|
||||
|
||||
def extract_recursive_call(fh: FunctionHandle):
|
||||
"""extract recursive function call"""
|
||||
if capa.features.extractors.ida.helpers.is_function_recursive(fh.inner):
|
||||
yield Characteristic("recursive call"), fh.address
|
||||
def extract_recursive_call(f):
|
||||
"""extract recursive function call
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
"""
|
||||
if capa.features.extractors.ida.helpers.is_function_recursive(f):
|
||||
yield Characteristic("recursive call"), f.start_ea
|
||||
|
||||
|
||||
def extract_features(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_features(f):
|
||||
"""extract function features
|
||||
|
||||
arg:
|
||||
f (IDA func_t)
|
||||
"""
|
||||
for func_handler in FUNCTION_HANDLERS:
|
||||
for (feature, addr) in func_handler(fh):
|
||||
yield feature, addr
|
||||
for (feature, ea) in func_handler(f):
|
||||
yield feature, ea
|
||||
|
||||
|
||||
FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop, extract_recursive_call)
|
||||
@@ -55,8 +68,8 @@ FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop, extract_r
|
||||
def main():
|
||||
""" """
|
||||
features = []
|
||||
for fhandle in capa.features.extractors.ida.helpers.get_functions(skip_thunks=True, skip_libs=True):
|
||||
features.extend(list(extract_features(fhandle)))
|
||||
for f in capa.features.extractors.ida.get_functions(skip_thunks=True, skip_libs=True):
|
||||
features.extend(list(extract_features(f)))
|
||||
|
||||
import pprint
|
||||
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
import logging
|
||||
import contextlib
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import idaapi
|
||||
import ida_loader
|
||||
|
||||
import capa.ida.helpers
|
||||
import capa.features.extractors.elf
|
||||
from capa.features.common import OS, ARCH_I386, ARCH_AMD64, OS_WINDOWS, Arch, Feature
|
||||
from capa.features.address import NO_ADDRESS, Address
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_os() -> Iterator[Tuple[Feature, Address]]:
|
||||
format_name: str = ida_loader.get_file_type_name()
|
||||
|
||||
if "PE" in format_name:
|
||||
yield OS(OS_WINDOWS), NO_ADDRESS
|
||||
|
||||
elif "ELF" in format_name:
|
||||
with contextlib.closing(capa.ida.helpers.IDAIO()) as f:
|
||||
os = capa.features.extractors.elf.detect_elf_os(f)
|
||||
|
||||
yield OS(os), NO_ADDRESS
|
||||
|
||||
else:
|
||||
# we likely end up here:
|
||||
# 1. handling shellcode, or
|
||||
# 2. handling a new file format (e.g. macho)
|
||||
#
|
||||
# for (1) we can't do much - its shellcode and all bets are off.
|
||||
# we could maybe accept a futher CLI argument to specify the OS,
|
||||
# but i think this would be rarely used.
|
||||
# rules that rely on OS conditions will fail to match on shellcode.
|
||||
#
|
||||
# for (2), this logic will need to be updated as the format is implemented.
|
||||
logger.debug("unsupported file format: %s, will not guess OS", format_name)
|
||||
return
|
||||
|
||||
|
||||
def extract_arch() -> Iterator[Tuple[Feature, Address]]:
|
||||
info: idaapi.idainfo = idaapi.get_inf_structure()
|
||||
if info.procname == "metapc" and info.is_64bit():
|
||||
yield Arch(ARCH_AMD64), NO_ADDRESS
|
||||
elif info.procname == "metapc" and info.is_32bit():
|
||||
yield Arch(ARCH_I386), NO_ADDRESS
|
||||
elif info.procname == "metapc":
|
||||
logger.debug("unsupported architecture: non-32-bit nor non-64-bit intel")
|
||||
return
|
||||
else:
|
||||
# we likely end up here:
|
||||
# 1. handling a new architecture (e.g. aarch64)
|
||||
#
|
||||
# for (1), this logic will need to be updated as the format is implemented.
|
||||
logger.debug("unsupported architecture: %s", info.procname)
|
||||
return
|
||||
@@ -1,22 +1,21 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import Any, Dict, Tuple, Iterator
|
||||
|
||||
import sys
|
||||
import string
|
||||
|
||||
import idc
|
||||
import idaapi
|
||||
import idautils
|
||||
import ida_bytes
|
||||
|
||||
from capa.features.address import AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import FunctionHandle
|
||||
|
||||
|
||||
def find_byte_sequence(start: int, end: int, seq: bytes) -> Iterator[int]:
|
||||
def find_byte_sequence(start, end, seq):
|
||||
"""yield all ea of a given byte sequence
|
||||
|
||||
args:
|
||||
@@ -24,32 +23,36 @@ def find_byte_sequence(start: int, end: int, seq: bytes) -> Iterator[int]:
|
||||
end: max virtual address
|
||||
seq: bytes to search e.g. b"\x01\x03"
|
||||
"""
|
||||
seqstr = " ".join(["%02x" % b for b in seq])
|
||||
if sys.version_info[0] >= 3:
|
||||
seq = " ".join(["%02x" % b for b in seq])
|
||||
else:
|
||||
seq = " ".join(["%02x" % ord(b) for b in seq])
|
||||
|
||||
while True:
|
||||
# TODO find_binary: Deprecated. Please use ida_bytes.bin_search() instead.
|
||||
ea = idaapi.find_binary(start, end, seqstr, 0, idaapi.SEARCH_DOWN)
|
||||
ea = idaapi.find_binary(start, end, seq, 0, idaapi.SEARCH_DOWN)
|
||||
if ea == idaapi.BADADDR:
|
||||
break
|
||||
start = ea + 1
|
||||
yield ea
|
||||
|
||||
|
||||
def get_functions(
|
||||
start: int = None, end: int = None, skip_thunks: bool = False, skip_libs: bool = False
|
||||
) -> Iterator[FunctionHandle]:
|
||||
def get_functions(start=None, end=None, skip_thunks=False, skip_libs=False):
|
||||
"""get functions, range optional
|
||||
|
||||
args:
|
||||
start: min virtual address
|
||||
end: max virtual address
|
||||
|
||||
ret:
|
||||
yield func_t*
|
||||
"""
|
||||
for ea in idautils.Functions(start=start, end=end):
|
||||
f = idaapi.get_func(ea)
|
||||
if not (skip_thunks and (f.flags & idaapi.FUNC_THUNK) or skip_libs and (f.flags & idaapi.FUNC_LIB)):
|
||||
yield FunctionHandle(address=AbsoluteVirtualAddress(ea), inner=f)
|
||||
yield f
|
||||
|
||||
|
||||
def get_segments(skip_header_segments=False) -> Iterator[idaapi.segment_t]:
|
||||
def get_segments(skip_header_segments=False):
|
||||
"""get list of segments (sections) in the binary image
|
||||
|
||||
args:
|
||||
@@ -61,7 +64,7 @@ def get_segments(skip_header_segments=False) -> Iterator[idaapi.segment_t]:
|
||||
yield seg
|
||||
|
||||
|
||||
def get_segment_buffer(seg: idaapi.segment_t) -> bytes:
|
||||
def get_segment_buffer(seg):
|
||||
"""return bytes stored in a given segment
|
||||
|
||||
decrease buffer size until IDA is able to read bytes from the segment
|
||||
@@ -79,8 +82,8 @@ def get_segment_buffer(seg: idaapi.segment_t) -> bytes:
|
||||
return buff if buff else b""
|
||||
|
||||
|
||||
def get_file_imports() -> Dict[int, Tuple[str, str, int]]:
|
||||
"""get file imports"""
|
||||
def get_file_imports():
|
||||
""" get file imports """
|
||||
imports = {}
|
||||
|
||||
for idx in range(idaapi.get_import_module_qty()):
|
||||
@@ -89,18 +92,10 @@ def get_file_imports() -> Dict[int, Tuple[str, str, int]]:
|
||||
if not library:
|
||||
continue
|
||||
|
||||
# IDA uses section names for the library of ELF imports, like ".dynsym"
|
||||
library = library.lstrip(".")
|
||||
|
||||
def inspect_import(ea, function, ordinal):
|
||||
if function and function.startswith("__imp_"):
|
||||
# handle mangled PE imports
|
||||
# handle mangled names starting
|
||||
function = function[len("__imp_") :]
|
||||
|
||||
if function and "@@" in function:
|
||||
# handle mangled ELF imports, like "fopen@@glibc_2.2.5"
|
||||
function, _, _ = function.partition("@@")
|
||||
|
||||
imports[ea] = (library.lower(), function, ordinal)
|
||||
return True
|
||||
|
||||
@@ -109,12 +104,14 @@ def get_file_imports() -> Dict[int, Tuple[str, str, int]]:
|
||||
return imports
|
||||
|
||||
|
||||
def get_instructions_in_range(start: int, end: int) -> Iterator[idaapi.insn_t]:
|
||||
def get_instructions_in_range(start, end):
|
||||
"""yield instructions in range
|
||||
|
||||
args:
|
||||
start: virtual address (inclusive)
|
||||
end: virtual address (exclusive)
|
||||
yield:
|
||||
(insn_t*)
|
||||
"""
|
||||
for head in idautils.Heads(start, end):
|
||||
insn = idautils.DecodeInstruction(head)
|
||||
@@ -122,8 +119,8 @@ def get_instructions_in_range(start: int, end: int) -> Iterator[idaapi.insn_t]:
|
||||
yield insn
|
||||
|
||||
|
||||
def is_operand_equal(op1: idaapi.op_t, op2: idaapi.op_t) -> bool:
|
||||
"""compare two IDA op_t"""
|
||||
def is_operand_equal(op1, op2):
|
||||
""" compare two IDA op_t """
|
||||
if op1.flags != op2.flags:
|
||||
return False
|
||||
|
||||
@@ -148,8 +145,8 @@ def is_operand_equal(op1: idaapi.op_t, op2: idaapi.op_t) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def is_basic_block_equal(bb1: idaapi.BasicBlock, bb2: idaapi.BasicBlock) -> bool:
|
||||
"""compare two IDA BasicBlock"""
|
||||
def is_basic_block_equal(bb1, bb2):
|
||||
""" compare two IDA BasicBlock """
|
||||
if bb1.start_ea != bb2.start_ea:
|
||||
return False
|
||||
|
||||
@@ -162,12 +159,12 @@ def is_basic_block_equal(bb1: idaapi.BasicBlock, bb2: idaapi.BasicBlock) -> bool
|
||||
return True
|
||||
|
||||
|
||||
def basic_block_size(bb: idaapi.BasicBlock) -> int:
|
||||
"""calculate size of basic block"""
|
||||
def basic_block_size(bb):
|
||||
""" calculate size of basic block """
|
||||
return bb.end_ea - bb.start_ea
|
||||
|
||||
|
||||
def read_bytes_at(ea: int, count: int) -> bytes:
|
||||
def read_bytes_at(ea, count):
|
||||
""" """
|
||||
# check if byte has a value, see get_wide_byte doc
|
||||
if not idc.is_loaded(ea):
|
||||
@@ -180,10 +177,10 @@ def read_bytes_at(ea: int, count: int) -> bytes:
|
||||
return idc.get_bytes(ea, count)
|
||||
|
||||
|
||||
def find_string_at(ea: int, min_: int = 4) -> str:
|
||||
"""check if ASCII string exists at a given virtual address"""
|
||||
def find_string_at(ea, min=4):
|
||||
""" check if ASCII string exists at a given virtual address """
|
||||
found = idaapi.get_strlit_contents(ea, -1, idaapi.STRTYPE_C)
|
||||
if found and len(found) > min_:
|
||||
if found and len(found) > min:
|
||||
try:
|
||||
found = found.decode("ascii")
|
||||
# hacky check for IDA bug; get_strlit_contents also reads Unicode as
|
||||
@@ -197,7 +194,7 @@ def find_string_at(ea: int, min_: int = 4) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def get_op_phrase_info(op: idaapi.op_t) -> Dict:
|
||||
def get_op_phrase_info(op):
|
||||
"""parse phrase features from operand
|
||||
|
||||
Pretty much dup of sark's implementation:
|
||||
@@ -234,24 +231,24 @@ def get_op_phrase_info(op: idaapi.op_t) -> Dict:
|
||||
return {"base": base, "index": index, "scale": scale, "offset": offset}
|
||||
|
||||
|
||||
def is_op_write(insn: idaapi.insn_t, op: idaapi.op_t) -> bool:
|
||||
"""Check if an operand is written to (destination operand)"""
|
||||
def is_op_write(insn, op):
|
||||
""" Check if an operand is written to (destination operand) """
|
||||
return idaapi.has_cf_chg(insn.get_canon_feature(), op.n)
|
||||
|
||||
|
||||
def is_op_read(insn: idaapi.insn_t, op: idaapi.op_t) -> bool:
|
||||
"""Check if an operand is read from (source operand)"""
|
||||
def is_op_read(insn, op):
|
||||
""" Check if an operand is read from (source operand) """
|
||||
return idaapi.has_cf_use(insn.get_canon_feature(), op.n)
|
||||
|
||||
|
||||
def is_op_offset(insn: idaapi.insn_t, op: idaapi.op_t) -> bool:
|
||||
"""Check is an operand has been marked as an offset (by auto-analysis or manually)"""
|
||||
def is_op_offset(insn, op):
|
||||
""" Check is an operand has been marked as an offset (by auto-analysis or manually) """
|
||||
flags = idaapi.get_flags(insn.ea)
|
||||
return ida_bytes.is_off(flags, op.n)
|
||||
|
||||
|
||||
def is_sp_modified(insn: idaapi.insn_t) -> bool:
|
||||
"""determine if instruction modifies SP, ESP, RSP"""
|
||||
def is_sp_modified(insn):
|
||||
""" determine if instruction modifies SP, ESP, RSP """
|
||||
for op in get_insn_ops(insn, target_ops=(idaapi.o_reg,)):
|
||||
if op.reg == idautils.procregs.sp.reg and is_op_write(insn, op):
|
||||
# register is stack and written
|
||||
@@ -259,8 +256,8 @@ def is_sp_modified(insn: idaapi.insn_t) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def is_bp_modified(insn: idaapi.insn_t) -> bool:
|
||||
"""check if instruction modifies BP, EBP, RBP"""
|
||||
def is_bp_modified(insn):
|
||||
""" check if instruction modifies BP, EBP, RBP """
|
||||
for op in get_insn_ops(insn, target_ops=(idaapi.o_reg,)):
|
||||
if op.reg == idautils.procregs.bp.reg and is_op_write(insn, op):
|
||||
# register is base and written
|
||||
@@ -268,13 +265,13 @@ def is_bp_modified(insn: idaapi.insn_t) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def is_frame_register(reg: int) -> bool:
|
||||
"""check if register is sp or bp"""
|
||||
def is_frame_register(reg):
|
||||
""" check if register is sp or bp """
|
||||
return reg in (idautils.procregs.sp.reg, idautils.procregs.bp.reg)
|
||||
|
||||
|
||||
def get_insn_ops(insn: idaapi.insn_t, target_ops: Tuple[Any] = None) -> idaapi.op_t:
|
||||
"""yield op_t for instruction, filter on type if specified"""
|
||||
def get_insn_ops(insn, target_ops=()):
|
||||
""" yield op_t for instruction, filter on type if specified """
|
||||
for op in insn.ops:
|
||||
if op.type == idaapi.o_void:
|
||||
# avoid looping all 6 ops if only subset exists
|
||||
@@ -284,12 +281,12 @@ def get_insn_ops(insn: idaapi.insn_t, target_ops: Tuple[Any] = None) -> idaapi.o
|
||||
yield op
|
||||
|
||||
|
||||
def is_op_stack_var(ea: int, index: int) -> bool:
|
||||
"""check if operand is a stack variable"""
|
||||
def is_op_stack_var(ea, index):
|
||||
""" check if operand is a stack variable """
|
||||
return idaapi.is_stkvar(idaapi.get_flags(ea), index)
|
||||
|
||||
|
||||
def mask_op_val(op: idaapi.op_t) -> int:
|
||||
def mask_op_val(op):
|
||||
"""mask value by data type
|
||||
|
||||
necessary due to a bug in AMD64
|
||||
@@ -309,18 +306,26 @@ def mask_op_val(op: idaapi.op_t) -> int:
|
||||
return masks.get(op.dtype, op.value) & op.value
|
||||
|
||||
|
||||
def is_function_recursive(f: idaapi.func_t) -> bool:
|
||||
"""check if function is recursive"""
|
||||
def is_function_recursive(f):
|
||||
"""check if function is recursive
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
"""
|
||||
for ref in idautils.CodeRefsTo(f.start_ea, True):
|
||||
if f.contains(ref):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def is_basic_block_tight_loop(bb: idaapi.BasicBlock) -> bool:
|
||||
def is_basic_block_tight_loop(bb):
|
||||
"""check basic block loops to self
|
||||
|
||||
true if last instruction in basic block branches to basic block start
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
"""
|
||||
bb_end = idc.prev_head(bb.end_ea)
|
||||
if bb.start_ea < bb_end:
|
||||
@@ -330,8 +335,8 @@ def is_basic_block_tight_loop(bb: idaapi.BasicBlock) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def find_data_reference_from_insn(insn: idaapi.insn_t, max_depth: int = 10) -> int:
|
||||
"""search for data reference from instruction, return address of instruction if no reference exists"""
|
||||
def find_data_reference_from_insn(insn, max_depth=10):
|
||||
""" search for data reference from instruction, return address of instruction if no reference exists """
|
||||
depth = 0
|
||||
ea = insn.ea
|
||||
|
||||
@@ -360,18 +365,19 @@ def find_data_reference_from_insn(insn: idaapi.insn_t, max_depth: int = 10) -> i
|
||||
return ea
|
||||
|
||||
|
||||
def get_function_blocks(f: idaapi.func_t) -> Iterator[idaapi.BasicBlock]:
|
||||
"""yield basic blocks contained in specified function"""
|
||||
def get_function_blocks(f):
|
||||
"""yield basic blocks contained in specified function
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
yield:
|
||||
block (IDA BasicBlock)
|
||||
"""
|
||||
# leverage idaapi.FC_NOEXT flag to ignore useless external blocks referenced by the function
|
||||
for block in idaapi.FlowChart(f, flags=(idaapi.FC_PREDS | idaapi.FC_NOEXT)):
|
||||
yield block
|
||||
|
||||
|
||||
def is_basic_block_return(bb: idaapi.BasicBlock) -> bool:
|
||||
"""check if basic block is return block"""
|
||||
def is_basic_block_return(bb):
|
||||
""" check if basic block is return block """
|
||||
return bb.type == idaapi.fcb_ret
|
||||
|
||||
|
||||
def has_sib(oper: idaapi.op_t) -> bool:
|
||||
# via: https://reverseengineering.stackexchange.com/a/14300
|
||||
return oper.specflag1 == 1
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import Any, Dict, Tuple, Iterator
|
||||
|
||||
import idc
|
||||
import idaapi
|
||||
@@ -13,24 +12,51 @@ import idautils
|
||||
|
||||
import capa.features.extractors.helpers
|
||||
import capa.features.extractors.ida.helpers
|
||||
from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
|
||||
from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Feature, Characteristic
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
|
||||
from capa.features import (
|
||||
ARCH_X32,
|
||||
ARCH_X64,
|
||||
MAX_BYTES_FEATURE_SIZE,
|
||||
THUNK_CHAIN_DEPTH_DELTA,
|
||||
Bytes,
|
||||
String,
|
||||
Characteristic,
|
||||
)
|
||||
from capa.features.insn import API, Number, Offset, Mnemonic
|
||||
|
||||
# security cookie checks may perform non-zeroing XORs, these are expected within a certain
|
||||
# byte range within the first and returning basic blocks, this helps to reduce FP features
|
||||
SECURITY_COOKIE_BYTES_DELTA = 0x40
|
||||
|
||||
|
||||
def get_imports(ctx: Dict[str, Any]) -> Dict[str, Any]:
|
||||
def get_arch(ctx):
|
||||
"""
|
||||
fetch the ARCH_* constant for the currently open workspace.
|
||||
|
||||
via Tamir Bahar/@tmr232
|
||||
https://reverseengineering.stackexchange.com/a/11398/17194
|
||||
"""
|
||||
if "arch" not in ctx:
|
||||
info = idaapi.get_inf_structure()
|
||||
if info.is_64bit():
|
||||
ctx["arch"] = ARCH_X64
|
||||
elif info.is_32bit():
|
||||
ctx["arch"] = ARCH_X32
|
||||
else:
|
||||
raise ValueError("unexpected architecture")
|
||||
return ctx["arch"]
|
||||
|
||||
|
||||
def get_imports(ctx):
|
||||
if "imports_cache" not in ctx:
|
||||
ctx["imports_cache"] = capa.features.extractors.ida.helpers.get_file_imports()
|
||||
return ctx["imports_cache"]
|
||||
|
||||
|
||||
def check_for_api_call(ctx: Dict[str, Any], insn: idaapi.insn_t) -> Iterator[str]:
|
||||
"""check instruction for API call"""
|
||||
def check_for_api_call(ctx, insn):
|
||||
""" check instruction for API call """
|
||||
if not insn.get_canon_mnem() in ("call", "jmp"):
|
||||
return
|
||||
|
||||
info = ()
|
||||
ref = insn.ea
|
||||
|
||||
@@ -58,55 +84,34 @@ def check_for_api_call(ctx: Dict[str, Any], insn: idaapi.insn_t) -> Iterator[str
|
||||
yield "%s.%s" % (info[0], info[1])
|
||||
|
||||
|
||||
def extract_insn_api_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse instruction API features
|
||||
def extract_insn_api_features(f, bb, insn):
|
||||
"""parse instruction API features
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
|
||||
example:
|
||||
call dword [0x00473038]
|
||||
call dword [0x00473038]
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
if not insn.get_canon_mnem() in ("call", "jmp"):
|
||||
return
|
||||
|
||||
for api in check_for_api_call(fh.ctx, insn):
|
||||
for api in check_for_api_call(f.ctx, insn):
|
||||
dll, _, symbol = api.rpartition(".")
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
|
||||
yield API(name), ih.address
|
||||
|
||||
# extract IDA/FLIRT recognized API functions
|
||||
targets = tuple(idautils.CodeRefsFrom(insn.ea, False))
|
||||
if not targets:
|
||||
return
|
||||
|
||||
target = targets[0]
|
||||
target_func = idaapi.get_func(target)
|
||||
if not target_func or target_func.start_ea != target:
|
||||
# not a function (start)
|
||||
return
|
||||
|
||||
if target_func.flags & idaapi.FUNC_LIB:
|
||||
name = idaapi.get_name(target_func.start_ea)
|
||||
yield API(name), ih.address
|
||||
if name.startswith("_"):
|
||||
# some linkers may prefix linked routines with a `_` to avoid name collisions.
|
||||
# extract features for both the mangled and un-mangled representations.
|
||||
# e.g. `_fwrite` -> `fwrite`
|
||||
# see: https://stackoverflow.com/a/2628384/87207
|
||||
yield API(name[1:]), ih.address
|
||||
yield API(name), insn.ea
|
||||
|
||||
|
||||
def extract_insn_number_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse instruction number features
|
||||
def extract_insn_number_features(f, bb, insn):
|
||||
"""parse instruction number features
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
|
||||
example:
|
||||
push 3136B0h ; dwControlCode
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
if idaapi.is_ret_insn(insn):
|
||||
# skip things like:
|
||||
# .text:0042250E retn 8
|
||||
@@ -117,11 +122,7 @@ def extract_insn_number_features(
|
||||
# .text:00401145 add esp, 0Ch
|
||||
return
|
||||
|
||||
for i, op in enumerate(insn.ops):
|
||||
if op.type == idaapi.o_void:
|
||||
break
|
||||
if op.type not in (idaapi.o_imm, idaapi.o_mem):
|
||||
continue
|
||||
for op in capa.features.extractors.ida.helpers.get_insn_ops(insn, target_ops=(idaapi.o_imm, idaapi.o_mem)):
|
||||
# skip things like:
|
||||
# .text:00401100 shr eax, offset loc_C
|
||||
if capa.features.extractors.ida.helpers.is_op_offset(insn, op):
|
||||
@@ -132,27 +133,21 @@ def extract_insn_number_features(
|
||||
else:
|
||||
const = op.addr
|
||||
|
||||
yield Number(const), ih.address
|
||||
yield OperandNumber(i, const), ih.address
|
||||
|
||||
if insn.itype == idaapi.NN_add and 0 < const < MAX_STRUCTURE_SIZE and op.type == idaapi.o_imm:
|
||||
# for pattern like:
|
||||
#
|
||||
# add eax, 0x10
|
||||
#
|
||||
# assume 0x10 is also an offset (imagine eax is a pointer).
|
||||
yield Offset(const), ih.address
|
||||
yield OperandOffset(i, const), ih.address
|
||||
yield Number(const), insn.ea
|
||||
yield Number(const, arch=get_arch(f.ctx)), insn.ea
|
||||
|
||||
|
||||
def extract_insn_bytes_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse referenced byte sequences
|
||||
def extract_insn_bytes_features(f, bb, insn):
|
||||
"""parse referenced byte sequences
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
|
||||
example:
|
||||
push offset iid_004118d4_IShellLinkA ; riid
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
if idaapi.is_call_insn(insn):
|
||||
return
|
||||
|
||||
@@ -160,46 +155,41 @@ def extract_insn_bytes_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandl
|
||||
if ref != insn.ea:
|
||||
extracted_bytes = capa.features.extractors.ida.helpers.read_bytes_at(ref, MAX_BYTES_FEATURE_SIZE)
|
||||
if extracted_bytes and not capa.features.extractors.helpers.all_zeros(extracted_bytes):
|
||||
yield Bytes(extracted_bytes), ih.address
|
||||
yield Bytes(extracted_bytes), insn.ea
|
||||
|
||||
|
||||
def extract_insn_string_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse instruction string features
|
||||
def extract_insn_string_features(f, bb, insn):
|
||||
"""parse instruction string features
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
|
||||
example:
|
||||
push offset aAcr ; "ACR > "
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
ref = capa.features.extractors.ida.helpers.find_data_reference_from_insn(insn)
|
||||
if ref != insn.ea:
|
||||
found = capa.features.extractors.ida.helpers.find_string_at(ref)
|
||||
if found:
|
||||
yield String(found), ih.address
|
||||
yield String(found), insn.ea
|
||||
|
||||
|
||||
def extract_insn_offset_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse instruction structure offset features
|
||||
def extract_insn_offset_features(f, bb, insn):
|
||||
"""parse instruction structure offset features
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
|
||||
example:
|
||||
.text:0040112F cmp [esi+4], ebx
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
for i, op in enumerate(insn.ops):
|
||||
if op.type == idaapi.o_void:
|
||||
break
|
||||
if op.type not in (idaapi.o_phrase, idaapi.o_displ):
|
||||
continue
|
||||
for op in capa.features.extractors.ida.helpers.get_insn_ops(insn, target_ops=(idaapi.o_phrase, idaapi.o_displ)):
|
||||
if capa.features.extractors.ida.helpers.is_op_stack_var(insn.ea, op.n):
|
||||
continue
|
||||
|
||||
p_info = capa.features.extractors.ida.helpers.get_op_phrase_info(op)
|
||||
op_off = p_info.get("offset", 0)
|
||||
if idaapi.is_mapped(op_off):
|
||||
@@ -212,32 +202,12 @@ def extract_insn_offset_features(
|
||||
# https://stackoverflow.com/questions/31853189/x86-64-assembly-why-displacement-not-64-bits
|
||||
op_off = capa.features.extractors.helpers.twos_complement(op_off, 32)
|
||||
|
||||
yield Offset(op_off), ih.address
|
||||
yield OperandOffset(i, op_off), ih.address
|
||||
|
||||
if (
|
||||
insn.itype == idaapi.NN_lea
|
||||
and i == 1
|
||||
# o_displ is used for both:
|
||||
# [eax+1]
|
||||
# [eax+ebx+2]
|
||||
and op.type == idaapi.o_displ
|
||||
# but the SIB is only present for [eax+ebx+2]
|
||||
# which we don't want
|
||||
and not capa.features.extractors.ida.helpers.has_sib(op)
|
||||
):
|
||||
# for pattern like:
|
||||
#
|
||||
# lea eax, [ebx + 1]
|
||||
#
|
||||
# assume 1 is also an offset (imagine ebx is a zero register).
|
||||
yield Number(op_off), ih.address
|
||||
yield OperandNumber(i, op_off), ih.address
|
||||
yield Offset(op_off), insn.ea
|
||||
yield Offset(op_off, arch=get_arch(f.ctx)), insn.ea
|
||||
|
||||
|
||||
def contains_stack_cookie_keywords(s: str) -> bool:
|
||||
"""
|
||||
check if string contains stack cookie keywords
|
||||
def contains_stack_cookie_keywords(s):
|
||||
"""check if string contains stack cookie keywords
|
||||
|
||||
Examples:
|
||||
xor ecx, ebp ; StackCookie
|
||||
@@ -251,7 +221,7 @@ def contains_stack_cookie_keywords(s: str) -> bool:
|
||||
return any(keyword in s for keyword in ("stack", "security"))
|
||||
|
||||
|
||||
def bb_stack_cookie_registers(bb: idaapi.BasicBlock) -> Iterator[int]:
|
||||
def bb_stack_cookie_registers(bb):
|
||||
"""scan basic block for stack cookie operations
|
||||
|
||||
yield registers ids that may have been used for stack cookie operations
|
||||
@@ -285,8 +255,8 @@ def bb_stack_cookie_registers(bb: idaapi.BasicBlock) -> Iterator[int]:
|
||||
yield op.reg
|
||||
|
||||
|
||||
def is_nzxor_stack_cookie_delta(f: idaapi.func_t, bb: idaapi.BasicBlock, insn: idaapi.insn_t) -> bool:
|
||||
"""check if nzxor exists within stack cookie delta"""
|
||||
def is_nzxor_stack_cookie_delta(f, bb, insn):
|
||||
""" check if nzxor exists within stack cookie delta """
|
||||
# security cookie check should use SP or BP
|
||||
if not capa.features.extractors.ida.helpers.is_frame_register(insn.Op2.reg):
|
||||
return False
|
||||
@@ -308,8 +278,8 @@ def is_nzxor_stack_cookie_delta(f: idaapi.func_t, bb: idaapi.BasicBlock, insn: i
|
||||
return False
|
||||
|
||||
|
||||
def is_nzxor_stack_cookie(f: idaapi.func_t, bb: idaapi.BasicBlock, insn: idaapi.insn_t) -> bool:
|
||||
"""check if nzxor is related to stack cookie"""
|
||||
def is_nzxor_stack_cookie(f, bb, insn):
|
||||
""" check if nzxor is related to stack cookie """
|
||||
if contains_stack_cookie_keywords(idaapi.get_cmt(insn.ea, False)):
|
||||
# Example:
|
||||
# xor ecx, ebp ; StackCookie
|
||||
@@ -325,49 +295,37 @@ def is_nzxor_stack_cookie(f: idaapi.func_t, bb: idaapi.BasicBlock, insn: idaapi.
|
||||
return False
|
||||
|
||||
|
||||
def extract_insn_nzxor_characteristic_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse instruction non-zeroing XOR instruction
|
||||
ignore expected non-zeroing XORs, e.g. security cookies
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
def extract_insn_nzxor_characteristic_features(f, bb, insn):
|
||||
"""parse instruction non-zeroing XOR instruction
|
||||
|
||||
ignore expected non-zeroing XORs, e.g. security cookies
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
"""
|
||||
if insn.itype not in (idaapi.NN_xor, idaapi.NN_xorpd, idaapi.NN_xorps, idaapi.NN_pxor):
|
||||
return
|
||||
if capa.features.extractors.ida.helpers.is_operand_equal(insn.Op1, insn.Op2):
|
||||
return
|
||||
if is_nzxor_stack_cookie(fh.inner, bbh.inner, insn):
|
||||
if is_nzxor_stack_cookie(f, bb, insn):
|
||||
return
|
||||
yield Characteristic("nzxor"), ih.address
|
||||
yield Characteristic("nzxor"), insn.ea
|
||||
|
||||
|
||||
def extract_insn_mnemonic_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse instruction mnemonic features"""
|
||||
yield Mnemonic(idc.print_insn_mnem(ih.inner.ea)), ih.address
|
||||
def extract_insn_mnemonic_features(f, bb, insn):
|
||||
"""parse instruction mnemonic features
|
||||
|
||||
|
||||
def extract_insn_obfs_call_plus_5_characteristic_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
"""
|
||||
parse call $+5 instruction from the given instruction.
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
if not idaapi.is_call_insn(insn):
|
||||
return
|
||||
|
||||
if insn.ea + 5 == idc.get_operand_value(insn.ea, 0):
|
||||
yield Characteristic("call $+5"), ih.address
|
||||
yield Mnemonic(insn.get_canon_mnem()), insn.ea
|
||||
|
||||
|
||||
def extract_insn_peb_access_characteristic_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_peb_access_characteristic_features(f, bb, insn):
|
||||
"""parse instruction peb access
|
||||
|
||||
fs:[0x30] on x86, gs:[0x60] on x64
|
||||
@@ -375,8 +333,6 @@ def extract_insn_peb_access_characteristic_features(
|
||||
TODO:
|
||||
IDA should be able to do this..
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
if insn.itype not in (idaapi.NN_push, idaapi.NN_mov):
|
||||
return
|
||||
|
||||
@@ -388,19 +344,15 @@ def extract_insn_peb_access_characteristic_features(
|
||||
|
||||
if " fs:30h" in disasm or " gs:60h" in disasm:
|
||||
# TODO: replace above with proper IDA
|
||||
yield Characteristic("peb access"), ih.address
|
||||
yield Characteristic("peb access"), insn.ea
|
||||
|
||||
|
||||
def extract_insn_segment_access_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_segment_access_features(f, bb, insn):
|
||||
"""parse instruction fs or gs access
|
||||
|
||||
TODO:
|
||||
IDA should be able to do this...
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
if all(map(lambda op: op.type != idaapi.o_mem, insn.ops)):
|
||||
# try to optimize for only memory references
|
||||
return
|
||||
@@ -409,21 +361,23 @@ def extract_insn_segment_access_features(
|
||||
|
||||
if " fs:" in disasm:
|
||||
# TODO: replace above with proper IDA
|
||||
yield Characteristic("fs access"), ih.address
|
||||
yield Characteristic("fs access"), insn.ea
|
||||
|
||||
if " gs:" in disasm:
|
||||
# TODO: replace above with proper IDA
|
||||
yield Characteristic("gs access"), ih.address
|
||||
yield Characteristic("gs access"), insn.ea
|
||||
|
||||
|
||||
def extract_insn_cross_section_cflow(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""inspect the instruction for a CALL or JMP that crosses section boundaries"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
def extract_insn_cross_section_cflow(f, bb, insn):
|
||||
"""inspect the instruction for a CALL or JMP that crosses section boundaries
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
"""
|
||||
for ref in idautils.CodeRefsFrom(insn.ea, False):
|
||||
if ref in get_imports(fh.ctx).keys():
|
||||
if ref in get_imports(f.ctx).keys():
|
||||
# ignore API calls
|
||||
continue
|
||||
if not idaapi.getseg(ref):
|
||||
@@ -431,40 +385,50 @@ def extract_insn_cross_section_cflow(
|
||||
continue
|
||||
if idaapi.getseg(ref) == idaapi.getseg(insn.ea):
|
||||
continue
|
||||
yield Characteristic("cross section flow"), ih.address
|
||||
yield Characteristic("cross section flow"), insn.ea
|
||||
|
||||
|
||||
def extract_function_calls_from(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_function_calls_from(f, bb, insn):
|
||||
"""extract functions calls from features
|
||||
|
||||
most relevant at the function scope, however, its most efficient to extract at the instruction scope
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
"""
|
||||
if idaapi.is_call_insn(insn):
|
||||
for ref in idautils.CodeRefsFrom(insn.ea, False):
|
||||
yield Characteristic("calls from"), AbsoluteVirtualAddress(ref)
|
||||
yield Characteristic("calls from"), ref
|
||||
|
||||
|
||||
def extract_function_indirect_call_characteristic_features(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_function_indirect_call_characteristic_features(f, bb, insn):
|
||||
"""extract indirect function calls (e.g., call eax or call dword ptr [edx+4])
|
||||
does not include calls like => call ds:dword_ABD4974
|
||||
|
||||
most relevant at the function or basic block scope;
|
||||
however, its most efficient to extract at the instruction scope
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
"""
|
||||
insn: idaapi.insn_t = ih.inner
|
||||
|
||||
if idaapi.is_call_insn(insn) and idc.get_operand_type(insn.ea, 0) in (idc.o_reg, idc.o_phrase, idc.o_displ):
|
||||
yield Characteristic("indirect call"), ih.address
|
||||
yield Characteristic("indirect call"), insn.ea
|
||||
|
||||
|
||||
def extract_features(f: FunctionHandle, bbh: BBHandle, insn: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""extract instruction features"""
|
||||
def extract_features(f, bb, insn):
|
||||
"""extract instruction features
|
||||
|
||||
args:
|
||||
f (IDA func_t)
|
||||
bb (IDA BasicBlock)
|
||||
insn (IDA insn_t)
|
||||
"""
|
||||
for inst_handler in INSTRUCTION_HANDLERS:
|
||||
for (feature, ea) in inst_handler(f, bbh, insn):
|
||||
for (feature, ea) in inst_handler(f, bb, insn):
|
||||
yield feature, ea
|
||||
|
||||
|
||||
@@ -476,7 +440,6 @@ INSTRUCTION_HANDLERS = (
|
||||
extract_insn_offset_features,
|
||||
extract_insn_nzxor_characteristic_features,
|
||||
extract_insn_mnemonic_features,
|
||||
extract_insn_obfs_call_plus_5_characteristic_features,
|
||||
extract_insn_peb_access_characteristic_features,
|
||||
extract_insn_cross_section_cflow,
|
||||
extract_insn_segment_access_features,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,7 +6,7 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import networkx
|
||||
from networkx import nx
|
||||
from networkx.algorithms.components import strongly_connected_components
|
||||
|
||||
|
||||
@@ -20,6 +20,6 @@ def has_loop(edges, threshold=2):
|
||||
returns:
|
||||
bool
|
||||
"""
|
||||
g = networkx.DiGraph()
|
||||
g = nx.DiGraph()
|
||||
g.add_edges_from(edges)
|
||||
return any(len(comp) >= threshold for comp in strongly_connected_components(g))
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
from typing import Dict, List, Tuple
|
||||
from dataclasses import dataclass
|
||||
|
||||
from capa.features.common import Feature
|
||||
from capa.features.address import NO_ADDRESS, Address
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
|
||||
|
||||
|
||||
@dataclass
|
||||
class InstructionFeatures:
|
||||
features: List[Tuple[Address, Feature]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class BasicBlockFeatures:
|
||||
features: List[Tuple[Address, Feature]]
|
||||
instructions: Dict[Address, InstructionFeatures]
|
||||
|
||||
|
||||
@dataclass
|
||||
class FunctionFeatures:
|
||||
features: List[Tuple[Address, Feature]]
|
||||
basic_blocks: Dict[Address, BasicBlockFeatures]
|
||||
|
||||
|
||||
@dataclass
|
||||
class NullFeatureExtractor(FeatureExtractor):
|
||||
"""
|
||||
An extractor that extracts some user-provided features.
|
||||
|
||||
This is useful for testing, as we can provide expected values and see if matching works.
|
||||
"""
|
||||
|
||||
base_address: Address
|
||||
global_features: List[Feature]
|
||||
file_features: List[Tuple[Address, Feature]]
|
||||
functions: Dict[Address, FunctionFeatures]
|
||||
|
||||
def get_base_address(self):
|
||||
return self.base_address
|
||||
|
||||
def extract_global_features(self):
|
||||
for feature in self.global_features:
|
||||
yield feature, NO_ADDRESS
|
||||
|
||||
def extract_file_features(self):
|
||||
for address, feature in self.file_features:
|
||||
yield feature, address
|
||||
|
||||
def get_functions(self):
|
||||
for address in sorted(self.functions.keys()):
|
||||
yield FunctionHandle(address, None)
|
||||
|
||||
def extract_function_features(self, f):
|
||||
for address, feature in self.functions.get(f.address, {}).features:
|
||||
yield feature, address
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
for address in sorted(self.functions.get(f.address, {}).basic_blocks.keys()):
|
||||
yield BBHandle(address, None)
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
for address, feature in self.functions.get(f.address, {}).basic_blocks.get(bb.address, {}).features:
|
||||
yield feature, address
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
for address in sorted(self.functions.get(f.address, {}).basic_blocks.get(bb.address, {}).instructions.keys()):
|
||||
yield InsnHandle(address, None)
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
for address, feature in (
|
||||
self.functions.get(f.address, {})
|
||||
.basic_blocks.get(bb.address, {})
|
||||
.instructions.get(insn.address, {})
|
||||
.features
|
||||
):
|
||||
yield feature, address
|
||||
@@ -1,216 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import logging
|
||||
|
||||
import pefile
|
||||
|
||||
import capa.features.common
|
||||
import capa.features.extractors
|
||||
import capa.features.extractors.common
|
||||
import capa.features.extractors.helpers
|
||||
import capa.features.extractors.strings
|
||||
from capa.features.file import Export, Import, Section
|
||||
from capa.features.common import OS, ARCH_I386, FORMAT_PE, ARCH_AMD64, OS_WINDOWS, Arch, Format, Characteristic
|
||||
from capa.features.address import NO_ADDRESS, FileOffsetAddress, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import FeatureExtractor
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_file_embedded_pe(buf, **kwargs):
|
||||
for offset, _ in capa.features.extractors.helpers.carve_pe(buf, 1):
|
||||
yield Characteristic("embedded pe"), FileOffsetAddress(offset)
|
||||
|
||||
|
||||
def extract_file_export_names(pe, **kwargs):
|
||||
base_address = pe.OPTIONAL_HEADER.ImageBase
|
||||
|
||||
if hasattr(pe, "DIRECTORY_ENTRY_EXPORT"):
|
||||
for export in pe.DIRECTORY_ENTRY_EXPORT.symbols:
|
||||
if not export.name:
|
||||
continue
|
||||
try:
|
||||
name = export.name.partition(b"\x00")[0].decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
va = base_address + export.address
|
||||
yield Export(name), AbsoluteVirtualAddress(va)
|
||||
|
||||
|
||||
def extract_file_import_names(pe, **kwargs):
|
||||
"""
|
||||
extract imported function names
|
||||
1. imports by ordinal:
|
||||
- modulename.#ordinal
|
||||
2. imports by name, results in two features to support importname-only matching:
|
||||
- modulename.importname
|
||||
- importname
|
||||
"""
|
||||
if hasattr(pe, "DIRECTORY_ENTRY_IMPORT"):
|
||||
for dll in pe.DIRECTORY_ENTRY_IMPORT:
|
||||
try:
|
||||
modname = dll.dll.partition(b"\x00")[0].decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
|
||||
# strip extension
|
||||
modname = modname.rpartition(".")[0].lower()
|
||||
|
||||
for imp in dll.imports:
|
||||
if imp.import_by_ordinal:
|
||||
impname = "#%s" % imp.ordinal
|
||||
else:
|
||||
try:
|
||||
impname = imp.name.partition(b"\x00")[0].decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
|
||||
for name in capa.features.extractors.helpers.generate_symbols(modname, impname):
|
||||
yield Import(name), AbsoluteVirtualAddress(imp.address)
|
||||
|
||||
|
||||
def extract_file_section_names(pe, **kwargs):
|
||||
base_address = pe.OPTIONAL_HEADER.ImageBase
|
||||
|
||||
for section in pe.sections:
|
||||
try:
|
||||
name = section.Name.partition(b"\x00")[0].decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
|
||||
yield Section(name), AbsoluteVirtualAddress(base_address + section.VirtualAddress)
|
||||
|
||||
|
||||
def extract_file_strings(buf, **kwargs):
|
||||
yield from capa.features.extractors.common.extract_file_strings(buf)
|
||||
|
||||
|
||||
def extract_file_function_names(**kwargs):
|
||||
"""
|
||||
extract the names of statically-linked library functions.
|
||||
"""
|
||||
if False:
|
||||
# using a `yield` here to force this to be a generator, not function.
|
||||
yield NotImplementedError("pefile doesn't have library matching")
|
||||
return
|
||||
|
||||
|
||||
def extract_file_os(**kwargs):
|
||||
# assuming PE -> Windows
|
||||
# though i suppose they're also used by UEFI
|
||||
yield OS(OS_WINDOWS), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_format(**kwargs):
|
||||
yield Format(FORMAT_PE), NO_ADDRESS
|
||||
|
||||
|
||||
def extract_file_arch(pe, **kwargs):
|
||||
if pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_I386"]:
|
||||
yield Arch(ARCH_I386), NO_ADDRESS
|
||||
elif pe.FILE_HEADER.Machine == pefile.MACHINE_TYPE["IMAGE_FILE_MACHINE_AMD64"]:
|
||||
yield Arch(ARCH_AMD64), NO_ADDRESS
|
||||
else:
|
||||
logger.warning("unsupported architecture: %s", pefile.MACHINE_TYPE[pe.FILE_HEADER.Machine])
|
||||
|
||||
|
||||
def extract_file_features(pe, buf):
|
||||
"""
|
||||
extract file features from given workspace
|
||||
|
||||
args:
|
||||
pe (pefile.PE): the parsed PE
|
||||
buf: the raw sample bytes
|
||||
|
||||
yields:
|
||||
Tuple[Feature, VA]: a feature and its location.
|
||||
"""
|
||||
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for feature, va in file_handler(pe=pe, buf=buf):
|
||||
yield feature, va
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
extract_file_embedded_pe,
|
||||
extract_file_export_names,
|
||||
extract_file_import_names,
|
||||
extract_file_section_names,
|
||||
extract_file_strings,
|
||||
extract_file_function_names,
|
||||
extract_file_format,
|
||||
)
|
||||
|
||||
|
||||
def extract_global_features(pe, buf):
|
||||
"""
|
||||
extract global features from given workspace
|
||||
|
||||
args:
|
||||
pe (pefile.PE): the parsed PE
|
||||
buf: the raw sample bytes
|
||||
|
||||
yields:
|
||||
Tuple[Feature, VA]: a feature and its location.
|
||||
"""
|
||||
for handler in GLOBAL_HANDLERS:
|
||||
for feature, va in handler(pe=pe, buf=buf):
|
||||
yield feature, va
|
||||
|
||||
|
||||
GLOBAL_HANDLERS = (
|
||||
extract_file_os,
|
||||
extract_file_arch,
|
||||
)
|
||||
|
||||
|
||||
class PefileFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, path: str):
|
||||
super(PefileFeatureExtractor, self).__init__()
|
||||
self.path = path
|
||||
self.pe = pefile.PE(path)
|
||||
|
||||
def get_base_address(self):
|
||||
return AbsoluteVirtualAddress(self.pe.OPTIONAL_HEADER.ImageBase)
|
||||
|
||||
def extract_global_features(self):
|
||||
with open(self.path, "rb") as f:
|
||||
buf = f.read()
|
||||
|
||||
yield from extract_global_features(self.pe, buf)
|
||||
|
||||
def extract_file_features(self):
|
||||
with open(self.path, "rb") as f:
|
||||
buf = f.read()
|
||||
|
||||
yield from extract_file_features(self.pe, buf)
|
||||
|
||||
def get_functions(self):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
|
||||
def extract_function_features(self, f):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
|
||||
def is_library_function(self, va):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
|
||||
def get_function_name(self, va):
|
||||
raise NotImplementedError("PefileFeatureExtract can only be used to extract file features")
|
||||
@@ -0,0 +1,52 @@
|
||||
import sys
|
||||
import types
|
||||
|
||||
from smda.common.SmdaReport import SmdaReport
|
||||
from smda.common.SmdaInstruction import SmdaInstruction
|
||||
|
||||
import capa.features.extractors.smda.file
|
||||
import capa.features.extractors.smda.insn
|
||||
import capa.features.extractors.smda.function
|
||||
import capa.features.extractors.smda.basicblock
|
||||
from capa.main import UnsupportedRuntimeError
|
||||
from capa.features.extractors import FeatureExtractor
|
||||
|
||||
|
||||
class SmdaFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, smda_report: SmdaReport, path):
|
||||
super(SmdaFeatureExtractor, self).__init__()
|
||||
if sys.version_info < (3, 0):
|
||||
raise UnsupportedRuntimeError("SMDA should only be used with Python 3.")
|
||||
self.smda_report = smda_report
|
||||
self.path = path
|
||||
|
||||
def get_base_address(self):
|
||||
return self.smda_report.base_addr
|
||||
|
||||
def extract_file_features(self):
|
||||
for feature, va in capa.features.extractors.smda.file.extract_features(self.smda_report, self.path):
|
||||
yield feature, va
|
||||
|
||||
def get_functions(self):
|
||||
for function in self.smda_report.getFunctions():
|
||||
yield function
|
||||
|
||||
def extract_function_features(self, f):
|
||||
for feature, va in capa.features.extractors.smda.function.extract_features(f):
|
||||
yield feature, va
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
for bb in f.getBlocks():
|
||||
yield bb
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
for feature, va in capa.features.extractors.smda.basicblock.extract_features(f, bb):
|
||||
yield feature, va
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
for smda_ins in bb.getInstructions():
|
||||
yield smda_ins
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
for feature, va in capa.features.extractors.smda.insn.extract_features(f, bb, insn):
|
||||
yield feature, va
|
||||
|
||||
@@ -1,12 +1,10 @@
|
||||
import sys
|
||||
import string
|
||||
import struct
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
from capa.features.common import Feature, Characteristic
|
||||
from capa.features.address import Address
|
||||
from capa.features import Characteristic
|
||||
from capa.features.basicblock import BasicBlock
|
||||
from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
|
||||
from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
|
||||
|
||||
|
||||
def _bb_has_tight_loop(f, bb):
|
||||
@@ -16,10 +14,10 @@ def _bb_has_tight_loop(f, bb):
|
||||
return bb.offset in f.blockrefs[bb.offset] if bb.offset in f.blockrefs else False
|
||||
|
||||
|
||||
def extract_bb_tight_loop(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""check basic block for tight loop indicators"""
|
||||
if _bb_has_tight_loop(f.inner, bb.inner):
|
||||
yield Characteristic("tight loop"), bb.address
|
||||
def extract_bb_tight_loop(f, bb):
|
||||
""" check basic block for tight loop indicators """
|
||||
if _bb_has_tight_loop(f, bb):
|
||||
yield Characteristic("tight loop"), bb.offset
|
||||
|
||||
|
||||
def _bb_has_stackstring(f, bb):
|
||||
@@ -40,10 +38,10 @@ def get_operands(smda_ins):
|
||||
return [o.strip() for o in smda_ins.operands.split(",")]
|
||||
|
||||
|
||||
def extract_stackstring(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""check basic block for stackstring indicators"""
|
||||
if _bb_has_stackstring(f.inner, bb.inner):
|
||||
yield Characteristic("stack string"), bb.address
|
||||
def extract_stackstring(f, bb):
|
||||
""" check basic block for stackstring indicators """
|
||||
if _bb_has_stackstring(f, bb):
|
||||
yield Characteristic("stack string"), bb.offset
|
||||
|
||||
|
||||
def is_mov_imm_to_stack(smda_ins):
|
||||
@@ -110,21 +108,21 @@ def get_printable_len(instr):
|
||||
return 0
|
||||
|
||||
|
||||
def extract_features(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_features(f, bb):
|
||||
"""
|
||||
extract features from the given basic block.
|
||||
|
||||
args:
|
||||
f: the function from which to extract features
|
||||
bb: the basic block to process.
|
||||
f (smda.common.SmdaFunction): the function from which to extract features
|
||||
bb (smda.common.SmdaBasicBlock): the basic block to process.
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: the features and their location found in this basic block.
|
||||
Feature, set[VA]: the features and their location found in this basic block.
|
||||
"""
|
||||
yield BasicBlock(), bb.address
|
||||
yield BasicBlock(), bb.offset
|
||||
for bb_handler in BASIC_BLOCK_HANDLERS:
|
||||
for feature, addr in bb_handler(f, bb):
|
||||
yield feature, addr
|
||||
for feature, va in bb_handler(f, bb):
|
||||
yield feature, va
|
||||
|
||||
|
||||
BASIC_BLOCK_HANDLERS = (
|
||||
|
||||
@@ -1,57 +0,0 @@
|
||||
from typing import List, Tuple
|
||||
|
||||
from smda.common.SmdaReport import SmdaReport
|
||||
|
||||
import capa.features.extractors.common
|
||||
import capa.features.extractors.smda.file
|
||||
import capa.features.extractors.smda.insn
|
||||
import capa.features.extractors.smda.global_
|
||||
import capa.features.extractors.smda.function
|
||||
import capa.features.extractors.smda.basicblock
|
||||
from capa.features.common import Feature
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
|
||||
|
||||
|
||||
class SmdaFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, smda_report: SmdaReport, path):
|
||||
super(SmdaFeatureExtractor, self).__init__()
|
||||
self.smda_report = smda_report
|
||||
self.path = path
|
||||
with open(self.path, "rb") as f:
|
||||
self.buf = f.read()
|
||||
|
||||
# pre-compute these because we'll yield them at *every* scope.
|
||||
self.global_features: List[Tuple[Feature, Address]] = []
|
||||
self.global_features.extend(capa.features.extractors.common.extract_os(self.buf))
|
||||
self.global_features.extend(capa.features.extractors.smda.global_.extract_arch(self.smda_report))
|
||||
|
||||
def get_base_address(self):
|
||||
return AbsoluteVirtualAddress(self.smda_report.base_addr)
|
||||
|
||||
def extract_global_features(self):
|
||||
yield from self.global_features
|
||||
|
||||
def extract_file_features(self):
|
||||
yield from capa.features.extractors.smda.file.extract_features(self.smda_report, self.buf)
|
||||
|
||||
def get_functions(self):
|
||||
for function in self.smda_report.getFunctions():
|
||||
yield FunctionHandle(address=AbsoluteVirtualAddress(function.offset), inner=function)
|
||||
|
||||
def extract_function_features(self, fh):
|
||||
yield from capa.features.extractors.smda.function.extract_features(fh)
|
||||
|
||||
def get_basic_blocks(self, fh):
|
||||
for bb in fh.inner.getBlocks():
|
||||
yield BBHandle(address=AbsoluteVirtualAddress(bb.offset), inner=bb)
|
||||
|
||||
def extract_basic_block_features(self, fh, bbh):
|
||||
yield from capa.features.extractors.smda.basicblock.extract_features(fh, bbh)
|
||||
|
||||
def get_instructions(self, fh, bbh):
|
||||
for smda_ins in bbh.inner.getInstructions():
|
||||
yield InsnHandle(address=AbsoluteVirtualAddress(smda_ins.offset), inner=smda_ins)
|
||||
|
||||
def extract_insn_features(self, fh, bbh, ih):
|
||||
yield from capa.features.extractors.smda.insn.extract_features(fh, bbh, ih)
|
||||
@@ -1,95 +1,133 @@
|
||||
import struct
|
||||
|
||||
# if we have SMDA we definitely have lief
|
||||
import lief
|
||||
|
||||
import capa.features.extractors.common
|
||||
import capa.features.extractors.helpers
|
||||
import capa.features.extractors.strings
|
||||
from capa.features import String, Characteristic
|
||||
from capa.features.file import Export, Import, Section
|
||||
from capa.features.common import String, Characteristic
|
||||
from capa.features.address import FileOffsetAddress, AbsoluteVirtualAddress
|
||||
|
||||
|
||||
def extract_file_embedded_pe(buf, **kwargs):
|
||||
for offset, _ in capa.features.extractors.helpers.carve_pe(buf, 1):
|
||||
yield Characteristic("embedded pe"), FileOffsetAddress(offset)
|
||||
def carve(pbytes, offset=0):
|
||||
"""
|
||||
Return a list of (offset, size, xor) tuples of embedded PEs
|
||||
|
||||
Based on the version from vivisect:
|
||||
https://github.com/vivisect/vivisect/blob/7be4037b1cecc4551b397f840405a1fc606f9b53/PE/carve.py#L19
|
||||
And its IDA adaptation:
|
||||
capa/features/extractors/ida/file.py
|
||||
"""
|
||||
mz_xor = [
|
||||
(
|
||||
capa.features.extractors.helpers.xor_static(b"MZ", i),
|
||||
capa.features.extractors.helpers.xor_static(b"PE", i),
|
||||
i,
|
||||
)
|
||||
for i in range(256)
|
||||
]
|
||||
|
||||
pblen = len(pbytes)
|
||||
todo = [(pbytes.find(mzx, offset), mzx, pex, i) for mzx, pex, i in mz_xor]
|
||||
todo = [(off, mzx, pex, i) for (off, mzx, pex, i) in todo if off != -1]
|
||||
|
||||
while len(todo):
|
||||
|
||||
off, mzx, pex, i = todo.pop()
|
||||
|
||||
# The MZ header has one field we will check
|
||||
# e_lfanew is at 0x3c
|
||||
e_lfanew = off + 0x3C
|
||||
if pblen < (e_lfanew + 4):
|
||||
continue
|
||||
|
||||
newoff = struct.unpack("<I", capa.features.extractors.helpers.xor_static(pbytes[e_lfanew : e_lfanew + 4], i))[0]
|
||||
|
||||
nextres = pbytes.find(mzx, off + 1)
|
||||
if nextres != -1:
|
||||
todo.append((nextres, mzx, pex, i))
|
||||
|
||||
peoff = off + newoff
|
||||
if pblen < (peoff + 2):
|
||||
continue
|
||||
|
||||
if pbytes[peoff : peoff + 2] == pex:
|
||||
yield (off, i)
|
||||
|
||||
|
||||
def extract_file_export_names(buf, **kwargs):
|
||||
lief_binary = lief.parse(buf)
|
||||
def extract_file_embedded_pe(smda_report, file_path):
|
||||
with open(file_path, "rb") as f:
|
||||
fbytes = f.read()
|
||||
|
||||
for offset, i in carve(fbytes, 1):
|
||||
yield Characteristic("embedded pe"), offset
|
||||
|
||||
|
||||
def extract_file_export_names(smda_report, file_path):
|
||||
lief_binary = lief.parse(file_path)
|
||||
if lief_binary is not None:
|
||||
for function in lief_binary.exported_functions:
|
||||
yield Export(function.name), AbsoluteVirtualAddress(function.address)
|
||||
yield Export(function.name), function.address
|
||||
|
||||
|
||||
def extract_file_import_names(smda_report, buf):
|
||||
def extract_file_import_names(smda_report, file_path):
|
||||
# extract import table info via LIEF
|
||||
lief_binary = lief.parse(buf)
|
||||
lief_binary = lief.parse(file_path)
|
||||
if not isinstance(lief_binary, lief.PE.Binary):
|
||||
return
|
||||
for imported_library in lief_binary.imports:
|
||||
library_name = imported_library.name.lower()
|
||||
library_name = library_name[:-4] if library_name.endswith(".dll") else library_name
|
||||
for func in imported_library.entries:
|
||||
va = func.iat_address + smda_report.base_addr
|
||||
if func.name:
|
||||
va = func.iat_address + smda_report.base_addr
|
||||
for name in capa.features.extractors.helpers.generate_symbols(library_name, func.name):
|
||||
yield Import(name), AbsoluteVirtualAddress(va)
|
||||
yield Import(name), va
|
||||
elif func.is_ordinal:
|
||||
for name in capa.features.extractors.helpers.generate_symbols(library_name, "#%s" % func.ordinal):
|
||||
yield Import(name), AbsoluteVirtualAddress(va)
|
||||
yield Import(name), va
|
||||
|
||||
|
||||
def extract_file_section_names(buf, **kwargs):
|
||||
lief_binary = lief.parse(buf)
|
||||
def extract_file_section_names(smda_report, file_path):
|
||||
lief_binary = lief.parse(file_path)
|
||||
if not isinstance(lief_binary, lief.PE.Binary):
|
||||
return
|
||||
if lief_binary and lief_binary.sections:
|
||||
base_address = lief_binary.optional_header.imagebase
|
||||
for section in lief_binary.sections:
|
||||
yield Section(section.name), AbsoluteVirtualAddress(base_address + section.virtual_address)
|
||||
yield Section(section.name), base_address + section.virtual_address
|
||||
|
||||
|
||||
def extract_file_strings(buf, **kwargs):
|
||||
def extract_file_strings(smda_report, file_path):
|
||||
"""
|
||||
extract ASCII and UTF-16 LE strings from file
|
||||
"""
|
||||
for s in capa.features.extractors.strings.extract_ascii_strings(buf):
|
||||
yield String(s.s), FileOffsetAddress(s.offset)
|
||||
with open(file_path, "rb") as f:
|
||||
b = f.read()
|
||||
|
||||
for s in capa.features.extractors.strings.extract_unicode_strings(buf):
|
||||
yield String(s.s), FileOffsetAddress(s.offset)
|
||||
for s in capa.features.extractors.strings.extract_ascii_strings(b):
|
||||
yield String(s.s), s.offset
|
||||
|
||||
for s in capa.features.extractors.strings.extract_unicode_strings(b):
|
||||
yield String(s.s), s.offset
|
||||
|
||||
|
||||
def extract_file_function_names(smda_report, **kwargs):
|
||||
"""
|
||||
extract the names of statically-linked library functions.
|
||||
"""
|
||||
if False:
|
||||
# using a `yield` here to force this to be a generator, not function.
|
||||
yield NotImplementedError("SMDA doesn't have library matching")
|
||||
return
|
||||
|
||||
|
||||
def extract_file_format(buf, **kwargs):
|
||||
yield from capa.features.extractors.common.extract_format(buf)
|
||||
|
||||
|
||||
def extract_features(smda_report, buf):
|
||||
def extract_features(smda_report, file_path):
|
||||
"""
|
||||
extract file features from given workspace
|
||||
|
||||
args:
|
||||
smda_report (smda.common.SmdaReport): a SmdaReport
|
||||
buf: the raw bytes of the sample
|
||||
file_path: path to the input file
|
||||
|
||||
yields:
|
||||
Tuple[Feature, VA]: a feature and its location.
|
||||
"""
|
||||
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for feature, addr in file_handler(smda_report=smda_report, buf=buf):
|
||||
yield feature, addr
|
||||
result = file_handler(smda_report, file_path)
|
||||
for feature, va in file_handler(smda_report, file_path):
|
||||
yield feature, va
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
@@ -98,6 +136,4 @@ FILE_HANDLERS = (
|
||||
extract_file_import_names,
|
||||
extract_file_section_names,
|
||||
extract_file_strings,
|
||||
extract_file_function_names,
|
||||
extract_file_format,
|
||||
)
|
||||
|
||||
@@ -1,42 +1,38 @@
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
from capa.features.common import Feature, Characteristic
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features import Characteristic
|
||||
from capa.features.extractors import loops
|
||||
from capa.features.extractors.base_extractor import FunctionHandle
|
||||
|
||||
|
||||
def extract_function_calls_to(f: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
for inref in f.inner.inrefs:
|
||||
yield Characteristic("calls to"), AbsoluteVirtualAddress(inref)
|
||||
def extract_function_calls_to(f):
|
||||
for inref in f.inrefs:
|
||||
yield Characteristic("calls to"), inref
|
||||
|
||||
|
||||
def extract_function_loop(f: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_function_loop(f):
|
||||
"""
|
||||
parse if a function has a loop
|
||||
"""
|
||||
edges = []
|
||||
for bb_from, bb_tos in f.inner.blockrefs.items():
|
||||
for bb_from, bb_tos in f.blockrefs.items():
|
||||
for bb_to in bb_tos:
|
||||
edges.append((bb_from, bb_to))
|
||||
|
||||
if edges and loops.has_loop(edges):
|
||||
yield Characteristic("loop"), f.address
|
||||
yield Characteristic("loop"), f.offset
|
||||
|
||||
|
||||
def extract_features(f: FunctionHandle):
|
||||
def extract_features(f):
|
||||
"""
|
||||
extract features from the given function.
|
||||
|
||||
args:
|
||||
f: the function from which to extract features
|
||||
f (smda.common.SmdaFunction): the function from which to extract features
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: the features and their location found in this function.
|
||||
Feature, set[VA]: the features and their location found in this function.
|
||||
"""
|
||||
for func_handler in FUNCTION_HANDLERS:
|
||||
for feature, addr in func_handler(f):
|
||||
yield feature, addr
|
||||
for feature, va in func_handler(f):
|
||||
yield feature, va
|
||||
|
||||
|
||||
FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop)
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
import logging
|
||||
|
||||
from capa.features.common import ARCH_I386, ARCH_AMD64, Arch
|
||||
from capa.features.address import NO_ADDRESS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_arch(smda_report):
|
||||
if smda_report.architecture == "intel":
|
||||
if smda_report.bitness == 32:
|
||||
yield Arch(ARCH_I386), NO_ADDRESS
|
||||
elif smda_report.bitness == 64:
|
||||
yield Arch(ARCH_AMD64), NO_ADDRESS
|
||||
else:
|
||||
# we likely end up here:
|
||||
# 1. handling a new architecture (e.g. aarch64)
|
||||
#
|
||||
# for (1), this logic will need to be updated as the format is implemented.
|
||||
logger.debug("unsupported architecture: %s", smda_report.architecture)
|
||||
return
|
||||
@@ -1,15 +1,20 @@
|
||||
import re
|
||||
import string
|
||||
import struct
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import smda
|
||||
from smda.common.SmdaReport import SmdaReport
|
||||
|
||||
import capa.features.extractors.helpers
|
||||
from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
|
||||
from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Feature, Characteristic
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
|
||||
from capa.features import (
|
||||
ARCH_X32,
|
||||
ARCH_X64,
|
||||
MAX_BYTES_FEATURE_SIZE,
|
||||
THUNK_CHAIN_DEPTH_DELTA,
|
||||
Bytes,
|
||||
String,
|
||||
Characteristic,
|
||||
)
|
||||
from capa.features.insn import API, Number, Offset, Mnemonic
|
||||
|
||||
# security cookie checks may perform non-zeroing XORs, these are expected within a certain
|
||||
# byte range within the first and returning basic blocks, this helps to reduce FP features
|
||||
@@ -18,20 +23,27 @@ PATTERN_HEXNUM = re.compile(r"[+\-] (?P<num>0x[a-fA-F0-9]+)")
|
||||
PATTERN_SINGLENUM = re.compile(r"[+\-] (?P<num>[0-9])")
|
||||
|
||||
|
||||
def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse API features from the given instruction."""
|
||||
f: smda.Function = fh.inner
|
||||
insn: smda.Insn = ih.inner
|
||||
def get_arch(smda_report):
|
||||
if smda_report.architecture == "intel":
|
||||
if smda_report.bitness == 32:
|
||||
return ARCH_X32
|
||||
elif smda_report.bitness == 64:
|
||||
return ARCH_X64
|
||||
else:
|
||||
raise NotImplementedError
|
||||
|
||||
if ih.address in f.apirefs:
|
||||
api_entry = f.apirefs[ih.address]
|
||||
|
||||
def extract_insn_api_features(f, bb, insn):
|
||||
"""parse API features from the given instruction."""
|
||||
if insn.offset in f.apirefs:
|
||||
api_entry = f.apirefs[insn.offset]
|
||||
# reformat
|
||||
dll_name, api_name = api_entry.split("!")
|
||||
dll_name = dll_name.split(".")[0]
|
||||
dll_name = dll_name.lower()
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll_name, api_name):
|
||||
yield API(name), ih.address
|
||||
elif ih.address in f.outrefs:
|
||||
yield API(name), insn.offset
|
||||
elif insn.offset in f.outrefs:
|
||||
current_function = f
|
||||
current_instruction = insn
|
||||
for index in range(THUNK_CHAIN_DEPTH_DELTA):
|
||||
@@ -50,7 +62,7 @@ def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterato
|
||||
dll_name = dll_name.split(".")[0]
|
||||
dll_name = dll_name.lower()
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll_name, api_name):
|
||||
yield API(name), ih.address
|
||||
yield API(name), insn.offset
|
||||
elif referenced_function.num_instructions == 1 and referenced_function.num_outrefs == 1:
|
||||
current_function = referenced_function
|
||||
current_instruction = [i for i in referenced_function.getInstructions()][0]
|
||||
@@ -58,14 +70,11 @@ def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterato
|
||||
return
|
||||
|
||||
|
||||
def extract_insn_number_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_number_features(f, bb, insn):
|
||||
"""parse number features from the given instruction."""
|
||||
# example:
|
||||
#
|
||||
# push 3136B0h ; dwControlCode
|
||||
f: smda.Function = fh.inner
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
operands = [o.strip() for o in insn.operands.split(",")]
|
||||
if insn.mnemonic == "add" and operands[0] in ["esp", "rsp"]:
|
||||
# skip things like:
|
||||
@@ -73,25 +82,12 @@ def extract_insn_number_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iter
|
||||
# .text:00401140 call sub_407E2B
|
||||
# .text:00401145 add esp, 0Ch
|
||||
return
|
||||
for i, operand in enumerate(operands):
|
||||
for operand in operands:
|
||||
try:
|
||||
# The result of bitwise operations is calculated as though carried out
|
||||
# in two’s complement with an infinite number of sign bits
|
||||
value = int(operand, 16) & ((1 << f.smda_report.bitness) - 1)
|
||||
except ValueError:
|
||||
yield Number(int(operand, 16)), insn.offset
|
||||
yield Number(int(operand, 16), arch=get_arch(f.smda_report)), insn.offset
|
||||
except:
|
||||
continue
|
||||
else:
|
||||
yield Number(value), ih.address
|
||||
yield OperandNumber(i, value), ih.address
|
||||
|
||||
if insn.mnemonic == "add" and 0 < value < MAX_STRUCTURE_SIZE:
|
||||
# for pattern like:
|
||||
#
|
||||
# add eax, 0x10
|
||||
#
|
||||
# assume 0x10 is also an offset (imagine eax is a pointer).
|
||||
yield Offset(value), ih.address
|
||||
yield OperandOffset(i, value), ih.address
|
||||
|
||||
|
||||
def read_bytes(smda_report, va, num_bytes=None):
|
||||
@@ -101,7 +97,7 @@ def read_bytes(smda_report, va, num_bytes=None):
|
||||
|
||||
rva = va - smda_report.base_addr
|
||||
if smda_report.buffer is None:
|
||||
raise ValueError("buffer is empty")
|
||||
return
|
||||
buffer_end = len(smda_report.buffer)
|
||||
max_bytes = num_bytes if num_bytes is not None else MAX_BYTES_FEATURE_SIZE
|
||||
if rva + max_bytes > buffer_end:
|
||||
@@ -140,15 +136,12 @@ def derefs(smda_report, p):
|
||||
p = val
|
||||
|
||||
|
||||
def extract_insn_bytes_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_bytes_features(f, bb, insn):
|
||||
"""
|
||||
parse byte sequence features from the given instruction.
|
||||
example:
|
||||
# push offset iid_004118d4_IShellLinkA ; riid
|
||||
"""
|
||||
f: smda.Function = fh.inner
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
for data_ref in insn.getDataRefs():
|
||||
for v in derefs(f.smda_report, data_ref):
|
||||
bytes_read = read_bytes(f.smda_report, v)
|
||||
@@ -157,7 +150,7 @@ def extract_insn_bytes_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Itera
|
||||
if capa.features.extractors.helpers.all_zeros(bytes_read):
|
||||
continue
|
||||
|
||||
yield Bytes(bytes_read), ih.address
|
||||
yield Bytes(bytes_read), insn.offset
|
||||
|
||||
|
||||
def detect_ascii_len(smda_report, offset):
|
||||
@@ -201,34 +194,30 @@ def read_string(smda_report, offset):
|
||||
return read_bytes(smda_report, offset, ulen).decode("utf-16")
|
||||
|
||||
|
||||
def extract_insn_string_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_string_features(f, bb, insn):
|
||||
"""parse string features from the given instruction."""
|
||||
# example:
|
||||
#
|
||||
# push offset aAcr ; "ACR > "
|
||||
f: smda.Function = fh.inner
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
for data_ref in insn.getDataRefs():
|
||||
for v in derefs(f.smda_report, data_ref):
|
||||
string_read = read_string(f.smda_report, v)
|
||||
if string_read:
|
||||
yield String(string_read.rstrip("\x00")), ih.address
|
||||
yield String(string_read.rstrip("\x00")), insn.offset
|
||||
|
||||
|
||||
def extract_insn_offset_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_offset_features(f, bb, insn):
|
||||
"""parse structure offset features from the given instruction."""
|
||||
# examples:
|
||||
#
|
||||
# mov eax, [esi + 4]
|
||||
# mov eax, [esi + ecx + 16384]
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
operands = [o.strip() for o in insn.operands.split(",")]
|
||||
for i, operand in enumerate(operands):
|
||||
for operand in operands:
|
||||
if not "ptr" in operand:
|
||||
continue
|
||||
if "esp" in operand or "ebp" in operand or "rbp" in operand:
|
||||
continue
|
||||
|
||||
number = 0
|
||||
number_hex = re.search(PATTERN_HEXNUM, operand)
|
||||
number_int = re.search(PATTERN_SINGLENUM, operand)
|
||||
@@ -238,26 +227,8 @@ def extract_insn_offset_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Featur
|
||||
elif number_int:
|
||||
number = int(number_int.group("num"))
|
||||
number = -1 * number if number_int.group().startswith("-") else number
|
||||
|
||||
if "ptr" not in operand:
|
||||
if (
|
||||
insn.mnemonic == "lea"
|
||||
and i == 1
|
||||
and (operand.count("+") + operand.count("-")) == 1
|
||||
and operand.count("*") == 0
|
||||
):
|
||||
# for pattern like:
|
||||
#
|
||||
# lea eax, [ebx + 1]
|
||||
#
|
||||
# assume 1 is also an offset (imagine ebx is a zero register).
|
||||
yield Number(number), ih.address
|
||||
yield OperandNumber(i, number), ih.address
|
||||
|
||||
continue
|
||||
|
||||
yield Offset(number), ih.address
|
||||
yield OperandOffset(i, number), ih.address
|
||||
yield Offset(number), insn.offset
|
||||
yield Offset(number, arch=get_arch(f.smda_report)), insn.offset
|
||||
|
||||
|
||||
def is_security_cookie(f, bb, insn):
|
||||
@@ -281,16 +252,11 @@ def is_security_cookie(f, bb, insn):
|
||||
return False
|
||||
|
||||
|
||||
def extract_insn_nzxor_characteristic_features(
|
||||
fh: FunctionHandle, bh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_nzxor_characteristic_features(f, bb, insn):
|
||||
"""
|
||||
parse non-zeroing XOR instruction from the given instruction.
|
||||
ignore expected non-zeroing XORs, e.g. security cookies.
|
||||
"""
|
||||
f: smda.Function = fh.inner
|
||||
bb: smda.BasicBlock = bh.inner
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
if insn.mnemonic not in ("xor", "xorpd", "xorps", "pxor"):
|
||||
return
|
||||
@@ -302,35 +268,18 @@ def extract_insn_nzxor_characteristic_features(
|
||||
if is_security_cookie(f, bb, insn):
|
||||
return
|
||||
|
||||
yield Characteristic("nzxor"), ih.address
|
||||
yield Characteristic("nzxor"), insn.offset
|
||||
|
||||
|
||||
def extract_insn_mnemonic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_mnemonic_features(f, bb, insn):
|
||||
"""parse mnemonic features from the given instruction."""
|
||||
yield Mnemonic(ih.inner.mnemonic), ih.address
|
||||
yield Mnemonic(insn.mnemonic), insn.offset
|
||||
|
||||
|
||||
def extract_insn_obfs_call_plus_5_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse call $+5 instruction from the given instruction.
|
||||
"""
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
if insn.mnemonic != "call":
|
||||
return
|
||||
|
||||
if not insn.operands.startswith("0x"):
|
||||
return
|
||||
|
||||
if int(insn.operands, 16) == insn.offset + 5:
|
||||
yield Characteristic("call $+5"), ih.address
|
||||
|
||||
|
||||
def extract_insn_peb_access_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_peb_access_characteristic_features(f, bb, insn):
|
||||
"""
|
||||
parse peb access from the given function. fs:[0x30] on x86, gs:[0x60] on x64
|
||||
"""
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
if insn.mnemonic not in ["push", "mov"]:
|
||||
return
|
||||
@@ -338,75 +287,65 @@ def extract_insn_peb_access_characteristic_features(f, bb, ih: InsnHandle) -> It
|
||||
operands = [o.strip() for o in insn.operands.split(",")]
|
||||
for operand in operands:
|
||||
if "fs:" in operand and "0x30" in operand:
|
||||
yield Characteristic("peb access"), ih.address
|
||||
yield Characteristic("peb access"), insn.offset
|
||||
elif "gs:" in operand and "0x60" in operand:
|
||||
yield Characteristic("peb access"), ih.address
|
||||
yield Characteristic("peb access"), insn.offset
|
||||
|
||||
|
||||
def extract_insn_segment_access_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse the instruction for access to fs or gs"""
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
def extract_insn_segment_access_features(f, bb, insn):
|
||||
""" parse the instruction for access to fs or gs """
|
||||
operands = [o.strip() for o in insn.operands.split(",")]
|
||||
for operand in operands:
|
||||
if "fs:" in operand:
|
||||
yield Characteristic("fs access"), ih.address
|
||||
yield Characteristic("fs access"), insn.offset
|
||||
elif "gs:" in operand:
|
||||
yield Characteristic("gs access"), ih.address
|
||||
yield Characteristic("gs access"), insn.offset
|
||||
|
||||
|
||||
def extract_insn_cross_section_cflow(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_cross_section_cflow(f, bb, insn):
|
||||
"""
|
||||
inspect the instruction for a CALL or JMP that crosses section boundaries.
|
||||
"""
|
||||
f: smda.Function = fh.inner
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
if insn.mnemonic in ["call", "jmp"]:
|
||||
if ih.address in f.apirefs:
|
||||
if insn.offset in f.apirefs:
|
||||
return
|
||||
|
||||
smda_report = insn.smda_function.smda_report
|
||||
if ih.address in f.outrefs:
|
||||
for target in f.outrefs[ih.address]:
|
||||
if smda_report.getSection(ih.address) != smda_report.getSection(target):
|
||||
yield Characteristic("cross section flow"), ih.address
|
||||
if insn.offset in f.outrefs:
|
||||
for target in f.outrefs[insn.offset]:
|
||||
if smda_report.getSection(insn.offset) != smda_report.getSection(target):
|
||||
yield Characteristic("cross section flow"), insn.offset
|
||||
elif insn.operands.startswith("0x"):
|
||||
target = int(insn.operands, 16)
|
||||
if smda_report.getSection(ih.address) != smda_report.getSection(target):
|
||||
yield Characteristic("cross section flow"), ih.address
|
||||
if smda_report.getSection(insn.offset) != smda_report.getSection(target):
|
||||
yield Characteristic("cross section flow"), insn.offset
|
||||
|
||||
|
||||
# this is a feature that's most relevant at the function scope,
|
||||
# however, its most efficient to extract at the instruction scope.
|
||||
def extract_function_calls_from(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
f: smda.Function = fh.inner
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
def extract_function_calls_from(f, bb, insn):
|
||||
if insn.mnemonic != "call":
|
||||
return
|
||||
|
||||
if ih.address in f.outrefs:
|
||||
for outref in f.outrefs[ih.address]:
|
||||
yield Characteristic("calls from"), AbsoluteVirtualAddress(outref)
|
||||
if insn.offset in f.outrefs:
|
||||
for outref in f.outrefs[insn.offset]:
|
||||
yield Characteristic("calls from"), outref
|
||||
|
||||
if outref == f.offset:
|
||||
# if we found a jump target and it's the function address
|
||||
# mark as recursive
|
||||
yield Characteristic("recursive call"), AbsoluteVirtualAddress(outref)
|
||||
if ih.address in f.apirefs:
|
||||
yield Characteristic("calls from"), ih.address
|
||||
yield Characteristic("recursive call"), outref
|
||||
if insn.offset in f.apirefs:
|
||||
yield Characteristic("calls from"), insn.offset
|
||||
|
||||
|
||||
# this is a feature that's most relevant at the function or basic block scope,
|
||||
# however, its most efficient to extract at the instruction scope.
|
||||
def extract_function_indirect_call_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_function_indirect_call_characteristic_features(f, bb, insn):
|
||||
"""
|
||||
extract indirect function call characteristic (e.g., call eax or call dword ptr [edx+4])
|
||||
does not include calls like => call ds:dword_ABD4974
|
||||
"""
|
||||
insn: smda.Insn = ih.inner
|
||||
|
||||
if insn.mnemonic != "call":
|
||||
return
|
||||
if insn.operands.startswith("0x"):
|
||||
@@ -418,7 +357,7 @@ def extract_function_indirect_call_characteristic_features(f, bb, ih: InsnHandle
|
||||
# call edx
|
||||
# call dword ptr [eax+50h]
|
||||
# call qword ptr [rsp+78h]
|
||||
yield Characteristic("indirect call"), ih.address
|
||||
yield Characteristic("indirect call"), insn.offset
|
||||
|
||||
|
||||
def extract_features(f, bb, insn):
|
||||
@@ -426,16 +365,16 @@ def extract_features(f, bb, insn):
|
||||
extract features from the given insn.
|
||||
|
||||
args:
|
||||
f: the function to process.
|
||||
bb: the basic block to process.
|
||||
insn: the instruction to process.
|
||||
f (smda.common.SmdaFunction): the function to process.
|
||||
bb (smda.common.SmdaBasicBlock): the basic block to process.
|
||||
insn (smda.common.SmdaInstruction): the instruction to process.
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: the features and their location found in this insn.
|
||||
Feature, set[VA]: the features and their location found in this insn.
|
||||
"""
|
||||
for insn_handler in INSTRUCTION_HANDLERS:
|
||||
for feature, addr in insn_handler(f, bb, insn):
|
||||
yield feature, addr
|
||||
for feature, va in insn_handler(f, bb, insn):
|
||||
yield feature, va
|
||||
|
||||
|
||||
INSTRUCTION_HANDLERS = (
|
||||
@@ -446,7 +385,6 @@ INSTRUCTION_HANDLERS = (
|
||||
extract_insn_offset_features,
|
||||
extract_insn_nzxor_characteristic_features,
|
||||
extract_insn_mnemonic_features,
|
||||
extract_insn_obfs_call_plus_5_characteristic_features,
|
||||
extract_insn_peb_access_characteristic_features,
|
||||
extract_insn_cross_section_cflow,
|
||||
extract_insn_segment_access_features,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# strings code from FLOSS, https://github.com/mandiant/flare-floss
|
||||
# strings code from FLOSS, https://github.com/fireeye/flare-floss
|
||||
#
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import types
|
||||
|
||||
import viv_utils
|
||||
|
||||
import capa.features.extractors
|
||||
import capa.features.extractors.viv.file
|
||||
import capa.features.extractors.viv.insn
|
||||
import capa.features.extractors.viv.function
|
||||
import capa.features.extractors.viv.basicblock
|
||||
from capa.features.extractors import FeatureExtractor
|
||||
|
||||
__all__ = ["file", "function", "basicblock", "insn"]
|
||||
|
||||
|
||||
def get_va(self):
|
||||
try:
|
||||
# vivisect type
|
||||
return self.va
|
||||
except AttributeError:
|
||||
pass
|
||||
|
||||
raise TypeError()
|
||||
|
||||
|
||||
def add_va_int_cast(o):
|
||||
"""
|
||||
dynamically add a cast-to-int (`__int__`) method to the given object
|
||||
that returns the value of the `.va` property.
|
||||
|
||||
this bit of skullduggery lets use cast viv-utils objects as ints.
|
||||
the correct way of doing this is to update viv-utils (or subclass the objects here).
|
||||
"""
|
||||
setattr(o, "__int__", types.MethodType(get_va, o))
|
||||
return o
|
||||
|
||||
|
||||
class VivisectFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, vw, path):
|
||||
super(VivisectFeatureExtractor, self).__init__()
|
||||
self.vw = vw
|
||||
self.path = path
|
||||
|
||||
def get_base_address(self):
|
||||
# assume there is only one file loaded into the vw
|
||||
return list(self.vw.filemeta.values())[0]["imagebase"]
|
||||
|
||||
def extract_file_features(self):
|
||||
for feature, va in capa.features.extractors.viv.file.extract_features(self.vw, self.path):
|
||||
yield feature, va
|
||||
|
||||
def get_functions(self):
|
||||
for va in sorted(self.vw.getFunctions()):
|
||||
yield add_va_int_cast(viv_utils.Function(self.vw, va))
|
||||
|
||||
def extract_function_features(self, f):
|
||||
for feature, va in capa.features.extractors.viv.function.extract_features(f):
|
||||
yield feature, va
|
||||
|
||||
def get_basic_blocks(self, f):
|
||||
for bb in f.basic_blocks:
|
||||
yield add_va_int_cast(bb)
|
||||
|
||||
def extract_basic_block_features(self, f, bb):
|
||||
for feature, va in capa.features.extractors.viv.basicblock.extract_features(f, bb):
|
||||
yield feature, va
|
||||
|
||||
def get_instructions(self, f, bb):
|
||||
for insn in bb.instructions:
|
||||
yield add_va_int_cast(insn)
|
||||
|
||||
def extract_insn_features(self, f, bb, insn):
|
||||
for feature, va in capa.features.extractors.viv.insn.extract_features(f, bb, insn):
|
||||
yield feature, va
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -8,30 +8,27 @@
|
||||
|
||||
import string
|
||||
import struct
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import envi
|
||||
import envi.archs.i386.disasm
|
||||
import vivisect.const
|
||||
|
||||
from capa.features.common import Feature, Characteristic
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features import Characteristic
|
||||
from capa.features.basicblock import BasicBlock
|
||||
from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
|
||||
from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
|
||||
|
||||
|
||||
def interface_extract_basic_block_XXX(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def interface_extract_basic_block_XXX(f, bb):
|
||||
"""
|
||||
parse features from the given basic block.
|
||||
|
||||
args:
|
||||
f: the function to process.
|
||||
bb: the basic block to process.
|
||||
f (viv_utils.Function): the function to process.
|
||||
bb (viv_utils.BasicBlock): the basic block to process.
|
||||
|
||||
yields:
|
||||
(Feature, Address): the feature and the address at which its found.
|
||||
(Feature, int): the feature and the address at which its found.
|
||||
"""
|
||||
...
|
||||
yield NotImplementedError("feature"), NotImplementedError("virtual address")
|
||||
|
||||
|
||||
def _bb_has_tight_loop(f, bb):
|
||||
@@ -40,17 +37,17 @@ def _bb_has_tight_loop(f, bb):
|
||||
"""
|
||||
if len(bb.instructions) > 0:
|
||||
for bva, bflags in bb.instructions[-1].getBranches():
|
||||
if bflags & envi.BR_COND:
|
||||
if bflags & vivisect.envi.BR_COND:
|
||||
if bva == bb.va:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def extract_bb_tight_loop(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""check basic block for tight loop indicators"""
|
||||
if _bb_has_tight_loop(f, bb.inner):
|
||||
yield Characteristic("tight loop"), bb.address
|
||||
def extract_bb_tight_loop(f, bb):
|
||||
""" check basic block for tight loop indicators """
|
||||
if _bb_has_tight_loop(f, bb):
|
||||
yield Characteristic("tight loop"), bb.va
|
||||
|
||||
|
||||
def _bb_has_stackstring(f, bb):
|
||||
@@ -70,13 +67,13 @@ def _bb_has_stackstring(f, bb):
|
||||
return False
|
||||
|
||||
|
||||
def extract_stackstring(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""check basic block for stackstring indicators"""
|
||||
if _bb_has_stackstring(f, bb.inner):
|
||||
yield Characteristic("stack string"), bb.address
|
||||
def extract_stackstring(f, bb):
|
||||
""" check basic block for stackstring indicators """
|
||||
if _bb_has_stackstring(f, bb):
|
||||
yield Characteristic("stack string"), bb.va
|
||||
|
||||
|
||||
def is_mov_imm_to_stack(instr: envi.archs.i386.disasm.i386Opcode) -> bool:
|
||||
def is_mov_imm_to_stack(instr):
|
||||
"""
|
||||
Return if instruction moves immediate onto stack
|
||||
"""
|
||||
@@ -108,7 +105,7 @@ def is_mov_imm_to_stack(instr: envi.archs.i386.disasm.i386Opcode) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def get_printable_len(oper: envi.archs.i386.disasm.i386ImmOper) -> int:
|
||||
def get_printable_len(oper):
|
||||
"""
|
||||
Return string length if all operand bytes are ascii or utf16-le printable
|
||||
"""
|
||||
@@ -120,18 +117,14 @@ def get_printable_len(oper: envi.archs.i386.disasm.i386ImmOper) -> int:
|
||||
chars = struct.pack("<I", oper.imm)
|
||||
elif oper.tsize == 8:
|
||||
chars = struct.pack("<Q", oper.imm)
|
||||
else:
|
||||
raise ValueError("unexpected oper.tsize: %d" % (oper.tsize))
|
||||
|
||||
if is_printable_ascii(chars):
|
||||
return oper.tsize
|
||||
elif is_printable_utf16le(chars):
|
||||
if is_printable_utf16le(chars):
|
||||
return oper.tsize / 2
|
||||
else:
|
||||
return 0
|
||||
return 0
|
||||
|
||||
|
||||
def is_printable_ascii(chars: bytes) -> bool:
|
||||
def is_printable_ascii(chars):
|
||||
try:
|
||||
chars_str = chars.decode("ascii")
|
||||
except UnicodeDecodeError:
|
||||
@@ -140,13 +133,12 @@ def is_printable_ascii(chars: bytes) -> bool:
|
||||
return all(c in string.printable for c in chars_str)
|
||||
|
||||
|
||||
def is_printable_utf16le(chars: bytes) -> bool:
|
||||
def is_printable_utf16le(chars):
|
||||
if all(c == b"\x00" for c in chars[1::2]):
|
||||
return is_printable_ascii(chars[::2])
|
||||
return False
|
||||
|
||||
|
||||
def extract_features(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_features(f, bb):
|
||||
"""
|
||||
extract features from the given basic block.
|
||||
|
||||
@@ -155,12 +147,12 @@ def extract_features(f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature,
|
||||
bb (viv_utils.BasicBlock): the basic block to process.
|
||||
|
||||
yields:
|
||||
Tuple[Feature, int]: the features and their location found in this basic block.
|
||||
Feature, set[VA]: the features and their location found in this basic block.
|
||||
"""
|
||||
yield BasicBlock(), AbsoluteVirtualAddress(bb.inner.va)
|
||||
yield BasicBlock(), bb.va
|
||||
for bb_handler in BASIC_BLOCK_HANDLERS:
|
||||
for feature, addr in bb_handler(f, bb):
|
||||
yield feature, addr
|
||||
for feature, va in bb_handler(f, bb):
|
||||
yield feature, va
|
||||
|
||||
|
||||
BASIC_BLOCK_HANDLERS = (
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
import logging
|
||||
from typing import List, Tuple, Iterator
|
||||
|
||||
import viv_utils
|
||||
import viv_utils.flirt
|
||||
|
||||
import capa.features.extractors.common
|
||||
import capa.features.extractors.viv.file
|
||||
import capa.features.extractors.viv.insn
|
||||
import capa.features.extractors.viv.global_
|
||||
import capa.features.extractors.viv.function
|
||||
import capa.features.extractors.viv.basicblock
|
||||
from capa.features.common import Feature
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, FeatureExtractor
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class VivisectFeatureExtractor(FeatureExtractor):
|
||||
def __init__(self, vw, path):
|
||||
super(VivisectFeatureExtractor, self).__init__()
|
||||
self.vw = vw
|
||||
self.path = path
|
||||
with open(self.path, "rb") as f:
|
||||
self.buf = f.read()
|
||||
|
||||
# pre-compute these because we'll yield them at *every* scope.
|
||||
self.global_features: List[Tuple[Feature, Address]] = []
|
||||
self.global_features.extend(capa.features.extractors.common.extract_os(self.buf))
|
||||
self.global_features.extend(capa.features.extractors.viv.global_.extract_arch(self.vw))
|
||||
|
||||
def get_base_address(self):
|
||||
# assume there is only one file loaded into the vw
|
||||
return AbsoluteVirtualAddress(list(self.vw.filemeta.values())[0]["imagebase"])
|
||||
|
||||
def extract_global_features(self):
|
||||
yield from self.global_features
|
||||
|
||||
def extract_file_features(self):
|
||||
yield from capa.features.extractors.viv.file.extract_features(self.vw, self.buf)
|
||||
|
||||
def get_functions(self) -> Iterator[FunctionHandle]:
|
||||
for va in sorted(self.vw.getFunctions()):
|
||||
yield FunctionHandle(address=AbsoluteVirtualAddress(va), inner=viv_utils.Function(self.vw, va))
|
||||
|
||||
def extract_function_features(self, fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.viv.function.extract_features(fh)
|
||||
|
||||
def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
|
||||
f: viv_utils.Function = fh.inner
|
||||
for bb in f.basic_blocks:
|
||||
yield BBHandle(address=AbsoluteVirtualAddress(bb.va), inner=bb)
|
||||
|
||||
def extract_basic_block_features(self, fh: FunctionHandle, bbh) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.viv.basicblock.extract_features(fh, bbh)
|
||||
|
||||
def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:
|
||||
bb: viv_utils.BasicBlock = bbh.inner
|
||||
for insn in bb.instructions:
|
||||
yield InsnHandle(address=AbsoluteVirtualAddress(insn.va), inner=insn)
|
||||
|
||||
def extract_insn_features(
|
||||
self, fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.viv.insn.extract_features(fh, bbh, ih)
|
||||
|
||||
def is_library_function(self, addr):
|
||||
return viv_utils.flirt.is_library_function(self.vw, addr)
|
||||
|
||||
def get_function_name(self, addr):
|
||||
return viv_utils.get_function_name(self.vw, addr)
|
||||
@@ -1,36 +1,33 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import PE.carve as pe_carve # vivisect PE
|
||||
import viv_utils
|
||||
import viv_utils.flirt
|
||||
|
||||
import capa.features.insn
|
||||
import capa.features.extractors.common
|
||||
import capa.features.extractors.helpers
|
||||
import capa.features.extractors.strings
|
||||
from capa.features.file import Export, Import, Section, FunctionName
|
||||
from capa.features.common import String, Feature, Characteristic
|
||||
from capa.features.address import Address, FileOffsetAddress, AbsoluteVirtualAddress
|
||||
from capa.features import String, Characteristic
|
||||
from capa.features.file import Export, Import, Section
|
||||
|
||||
|
||||
def extract_file_embedded_pe(buf, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
for offset, _ in pe_carve.carve(buf, 1):
|
||||
yield Characteristic("embedded pe"), FileOffsetAddress(offset)
|
||||
def extract_file_embedded_pe(vw, file_path):
|
||||
with open(file_path, "rb") as f:
|
||||
fbytes = f.read()
|
||||
|
||||
for offset, i in pe_carve.carve(fbytes, 1):
|
||||
yield Characteristic("embedded pe"), offset
|
||||
|
||||
|
||||
def extract_file_export_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
for va, _, name, _ in vw.getExports():
|
||||
yield Export(name), AbsoluteVirtualAddress(va)
|
||||
def extract_file_export_names(vw, file_path):
|
||||
for va, etype, name, _ in vw.getExports():
|
||||
yield Export(name), va
|
||||
|
||||
|
||||
def extract_file_import_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_file_import_names(vw, file_path):
|
||||
"""
|
||||
extract imported function names
|
||||
1. imports by ordinal:
|
||||
@@ -41,17 +38,16 @@ def extract_file_import_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]
|
||||
"""
|
||||
for va, _, _, tinfo in vw.getImports():
|
||||
# vivisect source: tinfo = "%s.%s" % (libname, impname)
|
||||
modname, impname = tinfo.split(".", 1)
|
||||
modname, impname = tinfo.split(".")
|
||||
if is_viv_ord_impname(impname):
|
||||
# replace ord prefix with #
|
||||
impname = "#%s" % impname[len("ord") :]
|
||||
|
||||
addr = AbsoluteVirtualAddress(va)
|
||||
for name in capa.features.extractors.helpers.generate_symbols(modname, impname):
|
||||
yield Import(name), addr
|
||||
yield Import(name), va
|
||||
|
||||
|
||||
def is_viv_ord_impname(impname: str) -> bool:
|
||||
def is_viv_ord_impname(impname):
|
||||
"""
|
||||
return if import name matches vivisect's ordinal naming scheme `'ord%d' % ord`
|
||||
"""
|
||||
@@ -65,51 +61,40 @@ def is_viv_ord_impname(impname: str) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def extract_file_section_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_file_section_names(vw, file_path):
|
||||
for va, _, segname, _ in vw.getSegments():
|
||||
yield Section(segname), AbsoluteVirtualAddress(va)
|
||||
yield Section(segname), va
|
||||
|
||||
|
||||
def extract_file_strings(buf, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.common.extract_file_strings(buf)
|
||||
|
||||
|
||||
def extract_file_function_names(vw, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_file_strings(vw, file_path):
|
||||
"""
|
||||
extract the names of statically-linked library functions.
|
||||
extract ASCII and UTF-16 LE strings from file
|
||||
"""
|
||||
for va in sorted(vw.getFunctions()):
|
||||
addr = AbsoluteVirtualAddress(va)
|
||||
if viv_utils.flirt.is_library_function(vw, va):
|
||||
name = viv_utils.get_function_name(vw, va)
|
||||
yield FunctionName(name), addr
|
||||
if name.startswith("_"):
|
||||
# some linkers may prefix linked routines with a `_` to avoid name collisions.
|
||||
# extract features for both the mangled and un-mangled representations.
|
||||
# e.g. `_fwrite` -> `fwrite`
|
||||
# see: https://stackoverflow.com/a/2628384/87207
|
||||
yield FunctionName(name[1:]), addr
|
||||
with open(file_path, "rb") as f:
|
||||
b = f.read()
|
||||
|
||||
for s in capa.features.extractors.strings.extract_ascii_strings(b):
|
||||
yield String(s.s), s.offset
|
||||
|
||||
for s in capa.features.extractors.strings.extract_unicode_strings(b):
|
||||
yield String(s.s), s.offset
|
||||
|
||||
|
||||
def extract_file_format(buf, **kwargs) -> Iterator[Tuple[Feature, Address]]:
|
||||
yield from capa.features.extractors.common.extract_format(buf)
|
||||
|
||||
|
||||
def extract_features(vw, buf: bytes) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_features(vw, file_path):
|
||||
"""
|
||||
extract file features from given workspace
|
||||
|
||||
args:
|
||||
vw (vivisect.VivWorkspace): the vivisect workspace
|
||||
buf: the raw input file bytes
|
||||
file_path: path to the input file
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: a feature and its location.
|
||||
Tuple[Feature, VA]: a feature and its location.
|
||||
"""
|
||||
|
||||
for file_handler in FILE_HANDLERS:
|
||||
for feature, addr in file_handler(vw=vw, buf=buf): # type: ignore
|
||||
yield feature, addr
|
||||
for feature, va in file_handler(vw, file_path):
|
||||
yield feature, va
|
||||
|
||||
|
||||
FILE_HANDLERS = (
|
||||
@@ -118,6 +103,4 @@ FILE_HANDLERS = (
|
||||
extract_file_import_names,
|
||||
extract_file_section_names,
|
||||
extract_file_strings,
|
||||
extract_file_function_names,
|
||||
extract_file_format,
|
||||
)
|
||||
|
||||
@@ -1,47 +1,39 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import envi
|
||||
import viv_utils
|
||||
import vivisect.const
|
||||
|
||||
from capa.features.common import Feature, Characteristic
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features import Characteristic
|
||||
from capa.features.extractors import loops
|
||||
from capa.features.extractors.base_extractor import FunctionHandle
|
||||
|
||||
|
||||
def interface_extract_function_XXX(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def interface_extract_function_XXX(f):
|
||||
"""
|
||||
parse features from the given function.
|
||||
|
||||
args:
|
||||
f: the function to process.
|
||||
f (viv_utils.Function): the function to process.
|
||||
|
||||
yields:
|
||||
(Feature, Address): the feature and the address at which its found.
|
||||
(Feature, int): the feature and the address at which its found.
|
||||
"""
|
||||
...
|
||||
yield NotImplementedError("feature"), NotImplementedError("virtual address")
|
||||
|
||||
|
||||
def extract_function_calls_to(fhandle: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
f: viv_utils.Function = fhandle.inner
|
||||
def extract_function_calls_to(f):
|
||||
for src, _, _, _ in f.vw.getXrefsTo(f.va, rtype=vivisect.const.REF_CODE):
|
||||
yield Characteristic("calls to"), AbsoluteVirtualAddress(src)
|
||||
yield Characteristic("calls to"), src
|
||||
|
||||
|
||||
def extract_function_loop(fhandle: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_function_loop(f):
|
||||
"""
|
||||
parse if a function has a loop
|
||||
"""
|
||||
f: viv_utils.Function = fhandle.inner
|
||||
|
||||
edges = []
|
||||
|
||||
for bb in f.basic_blocks:
|
||||
@@ -49,30 +41,30 @@ def extract_function_loop(fhandle: FunctionHandle) -> Iterator[Tuple[Feature, Ad
|
||||
for bva, bflags in bb.instructions[-1].getBranches():
|
||||
# vivisect does not set branch flags for non-conditional jmp so add explicit check
|
||||
if (
|
||||
bflags & envi.BR_COND
|
||||
or bflags & envi.BR_FALL
|
||||
or bflags & envi.BR_TABLE
|
||||
bflags & vivisect.envi.BR_COND
|
||||
or bflags & vivisect.envi.BR_FALL
|
||||
or bflags & vivisect.envi.BR_TABLE
|
||||
or bb.instructions[-1].mnem == "jmp"
|
||||
):
|
||||
edges.append((bb.va, bva))
|
||||
|
||||
if edges and loops.has_loop(edges):
|
||||
yield Characteristic("loop"), fhandle.address
|
||||
yield Characteristic("loop"), f.va
|
||||
|
||||
|
||||
def extract_features(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_features(f):
|
||||
"""
|
||||
extract features from the given function.
|
||||
|
||||
args:
|
||||
fh: the function handle from which to extract features
|
||||
f (viv_utils.Function): the function from which to extract features
|
||||
|
||||
yields:
|
||||
Tuple[Feature, int]: the features and their location found in this function.
|
||||
Feature, set[VA]: the features and their location found in this function.
|
||||
"""
|
||||
for func_handler in FUNCTION_HANDLERS:
|
||||
for feature, addr in func_handler(fh):
|
||||
yield feature, addr
|
||||
for feature, va in func_handler(f):
|
||||
yield feature, va
|
||||
|
||||
|
||||
FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop)
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
import logging
|
||||
from typing import Tuple, Iterator
|
||||
|
||||
import envi.archs.i386
|
||||
import envi.archs.amd64
|
||||
|
||||
from capa.features.common import ARCH_I386, ARCH_AMD64, Arch, Feature
|
||||
from capa.features.address import NO_ADDRESS, Address
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def extract_arch(vw) -> Iterator[Tuple[Feature, Address]]:
|
||||
if isinstance(vw.arch, envi.archs.amd64.Amd64Module):
|
||||
yield Arch(ARCH_AMD64), NO_ADDRESS
|
||||
|
||||
elif isinstance(vw.arch, envi.archs.i386.i386Module):
|
||||
yield Arch(ARCH_I386), NO_ADDRESS
|
||||
|
||||
else:
|
||||
# we likely end up here:
|
||||
# 1. handling a new architecture (e.g. aarch64)
|
||||
#
|
||||
# for (1), this logic will need to be updated as the format is implemented.
|
||||
logger.debug("unsupported architecture: %s", vw.arch.__class__.__name__)
|
||||
return
|
||||
@@ -1,17 +1,14 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import Optional
|
||||
|
||||
from vivisect import VivWorkspace
|
||||
from vivisect.const import XR_TO, REF_CODE
|
||||
|
||||
|
||||
def get_coderef_from(vw: VivWorkspace, va: int) -> Optional[int]:
|
||||
def get_coderef_from(vw, va):
|
||||
"""
|
||||
return first code `tova` whose origin is the specified va
|
||||
return None if no code reference is found
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -7,13 +7,11 @@
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import collections
|
||||
from typing import Set, List, Deque, Tuple, Union, Optional
|
||||
|
||||
import envi
|
||||
import vivisect.const
|
||||
import envi.archs.i386.disasm
|
||||
import envi.archs.amd64.disasm
|
||||
from vivisect import VivWorkspace
|
||||
|
||||
# pull out consts for lookup performance
|
||||
i386RegOper = envi.archs.i386.disasm.i386RegOper
|
||||
@@ -28,7 +26,7 @@ FAR_BRANCH_MASK = envi.BR_PROC | envi.BR_DEREF | envi.BR_ARCH
|
||||
DESTRUCTIVE_MNEMONICS = ("mov", "lea", "pop", "xor")
|
||||
|
||||
|
||||
def get_previous_instructions(vw: VivWorkspace, va: int) -> List[int]:
|
||||
def get_previous_instructions(vw, va):
|
||||
"""
|
||||
collect the instructions that flow to the given address, local to the current function.
|
||||
|
||||
@@ -45,14 +43,12 @@ def get_previous_instructions(vw: VivWorkspace, va: int) -> List[int]:
|
||||
# ensure that it fallsthrough to this one.
|
||||
loc = vw.getPrevLocation(va, adjacent=True)
|
||||
if loc is not None:
|
||||
ploc = vw.getPrevLocation(va, adjacent=True)
|
||||
if ploc is not None:
|
||||
# from vivisect.const:
|
||||
# location: (L_VA, L_SIZE, L_LTYPE, L_TINFO)
|
||||
(pva, _, ptype, pinfo) = ploc
|
||||
# from vivisect.const:
|
||||
# location: (L_VA, L_SIZE, L_LTYPE, L_TINFO)
|
||||
(pva, _, ptype, pinfo) = vw.getPrevLocation(va, adjacent=True)
|
||||
|
||||
if ptype == LOC_OP and not (pinfo & IF_NOFALL):
|
||||
ret.append(pva)
|
||||
if ptype == LOC_OP and not (pinfo & IF_NOFALL):
|
||||
ret.append(pva)
|
||||
|
||||
# find any code refs, e.g. jmp, to this location.
|
||||
# ignore any calls.
|
||||
@@ -71,7 +67,7 @@ class NotFoundError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def find_definition(vw: VivWorkspace, va: int, reg: int) -> Tuple[int, Union[int, None]]:
|
||||
def find_definition(vw, va, reg):
|
||||
"""
|
||||
scan backwards from the given address looking for assignments to the given register.
|
||||
if a constant, return that value.
|
||||
@@ -87,8 +83,8 @@ def find_definition(vw: VivWorkspace, va: int, reg: int) -> Tuple[int, Union[int
|
||||
raises:
|
||||
NotFoundError: when the definition cannot be found.
|
||||
"""
|
||||
q = collections.deque() # type: Deque[int]
|
||||
seen = set([]) # type: Set[int]
|
||||
q = collections.deque()
|
||||
seen = set([])
|
||||
|
||||
q.extend(get_previous_instructions(vw, va))
|
||||
while q:
|
||||
@@ -132,14 +128,14 @@ def find_definition(vw: VivWorkspace, va: int, reg: int) -> Tuple[int, Union[int
|
||||
raise NotFoundError()
|
||||
|
||||
|
||||
def is_indirect_call(vw: VivWorkspace, va: int, insn: envi.Opcode) -> bool:
|
||||
def is_indirect_call(vw, va, insn=None):
|
||||
if insn is None:
|
||||
insn = vw.parseOpcode(va)
|
||||
|
||||
return insn.mnem in ("call", "jmp") and isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegOper)
|
||||
|
||||
|
||||
def resolve_indirect_call(vw: VivWorkspace, va: int, insn: envi.Opcode) -> Tuple[int, Optional[int]]:
|
||||
def resolve_indirect_call(vw, va, insn=None):
|
||||
"""
|
||||
inspect the given indirect call instruction and attempt to resolve the target address.
|
||||
|
||||
|
||||
@@ -1,28 +1,26 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
from typing import List, Tuple, Callable, Iterator
|
||||
|
||||
import envi
|
||||
import envi.exc
|
||||
import viv_utils
|
||||
import envi.memory
|
||||
import viv_utils.flirt
|
||||
import envi.archs.i386.regs
|
||||
import envi.archs.amd64.regs
|
||||
import envi.archs.i386.disasm
|
||||
import envi.archs.amd64.disasm
|
||||
|
||||
import capa.features.extractors.helpers
|
||||
import capa.features.extractors.viv.helpers
|
||||
from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
|
||||
from capa.features.common import MAX_BYTES_FEATURE_SIZE, THUNK_CHAIN_DEPTH_DELTA, Bytes, String, Feature, Characteristic
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
|
||||
from capa.features import (
|
||||
ARCH_X32,
|
||||
ARCH_X64,
|
||||
MAX_BYTES_FEATURE_SIZE,
|
||||
THUNK_CHAIN_DEPTH_DELTA,
|
||||
Bytes,
|
||||
String,
|
||||
Characteristic,
|
||||
)
|
||||
from capa.features.insn import API, Number, Offset, Mnemonic
|
||||
from capa.features.extractors.viv.indirect_calls import NotFoundError, resolve_indirect_call
|
||||
|
||||
# security cookie checks may perform non-zeroing XORs, these are expected within a certain
|
||||
@@ -30,21 +28,27 @@ from capa.features.extractors.viv.indirect_calls import NotFoundError, resolve_i
|
||||
SECURITY_COOKIE_BYTES_DELTA = 0x40
|
||||
|
||||
|
||||
def interface_extract_instruction_XXX(
|
||||
fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
def get_arch(vw):
|
||||
arch = vw.getMeta("Architecture")
|
||||
if arch == "i386":
|
||||
return ARCH_X32
|
||||
elif arch == "amd64":
|
||||
return ARCH_X64
|
||||
|
||||
|
||||
def interface_extract_instruction_XXX(f, bb, insn):
|
||||
"""
|
||||
parse features from the given instruction.
|
||||
|
||||
args:
|
||||
fh: the function handle to process.
|
||||
bbh: the basic block handle to process.
|
||||
ih: the instruction handle to process.
|
||||
f (viv_utils.Function): the function to process.
|
||||
bb (viv_utils.BasicBlock): the basic block to process.
|
||||
insn (vivisect...Instruction): the instruction to process.
|
||||
|
||||
yields:
|
||||
(Feature, Address): the feature and the address at which its found.
|
||||
(Feature, int): the feature and the address at which its found.
|
||||
"""
|
||||
...
|
||||
yield NotImplementedError("feature"), NotImplementedError("virtual address")
|
||||
|
||||
|
||||
def get_imports(vw):
|
||||
@@ -64,15 +68,13 @@ def get_imports(vw):
|
||||
return imports
|
||||
|
||||
|
||||
def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse API features from the given instruction.
|
||||
def extract_insn_api_features(f, bb, insn):
|
||||
"""parse API features from the given instruction."""
|
||||
|
||||
# example:
|
||||
#
|
||||
# call dword [0x00473038]
|
||||
|
||||
example:
|
||||
call dword [0x00473038]
|
||||
"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
if insn.mnem not in ("call", "jmp"):
|
||||
return
|
||||
|
||||
@@ -89,12 +91,12 @@ def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterato
|
||||
if target in imports:
|
||||
dll, symbol = imports[target]
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
|
||||
yield API(name), ih.address
|
||||
yield API(name), insn.va
|
||||
|
||||
# call via thunk on x86,
|
||||
# see 9324d1a8ae37a36ae560c37448c9705a at 0x407985
|
||||
#
|
||||
# this is also how calls to internal functions may be decoded on x32 and x64.
|
||||
# this is also how calls to internal functions may be decoded on x64.
|
||||
# see Lab21-01.exe_:0x140001178
|
||||
#
|
||||
# follow chained thunks, e.g. in 82bf6347acf15e5d883715dc289d8a2b at 0x14005E0FF in
|
||||
@@ -109,26 +111,11 @@ def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterato
|
||||
if not target:
|
||||
return
|
||||
|
||||
if viv_utils.flirt.is_library_function(f.vw, target):
|
||||
name = viv_utils.get_function_name(f.vw, target)
|
||||
yield API(name), ih.address
|
||||
if name.startswith("_"):
|
||||
# some linkers may prefix linked routines with a `_` to avoid name collisions.
|
||||
# extract features for both the mangled and un-mangled representations.
|
||||
# e.g. `_fwrite` -> `fwrite`
|
||||
# see: https://stackoverflow.com/a/2628384/87207
|
||||
yield API(name[1:]), ih.address
|
||||
return
|
||||
|
||||
for _ in range(THUNK_CHAIN_DEPTH_DELTA):
|
||||
if target in imports:
|
||||
dll, symbol = imports[target]
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
|
||||
yield API(name), ih.address
|
||||
|
||||
# if jump leads to an ENDBRANCH instruction, skip it
|
||||
if f.vw.getByteDef(target)[1].startswith(b"\xf3\x0f\x1e"):
|
||||
target += 4
|
||||
yield API(name), insn.va
|
||||
|
||||
target = capa.features.extractors.viv.helpers.get_coderef_from(f.vw, target)
|
||||
if not target:
|
||||
@@ -144,7 +131,7 @@ def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterato
|
||||
if target in imports:
|
||||
dll, symbol = imports[target]
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
|
||||
yield API(name), ih.address
|
||||
yield API(name), insn.va
|
||||
|
||||
elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegOper):
|
||||
try:
|
||||
@@ -161,7 +148,38 @@ def extract_insn_api_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterato
|
||||
if target in imports:
|
||||
dll, symbol = imports[target]
|
||||
for name in capa.features.extractors.helpers.generate_symbols(dll, symbol):
|
||||
yield API(name), ih.address
|
||||
yield API(name), insn.va
|
||||
|
||||
|
||||
def extract_insn_number_features(f, bb, insn):
|
||||
"""parse number features from the given instruction."""
|
||||
# example:
|
||||
#
|
||||
# push 3136B0h ; dwControlCode
|
||||
for oper in insn.opers:
|
||||
# this is for both x32 and x64
|
||||
if not isinstance(oper, (envi.archs.i386.disasm.i386ImmOper, envi.archs.i386.disasm.i386ImmMemOper)):
|
||||
continue
|
||||
|
||||
if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
|
||||
v = oper.getOperValue(oper)
|
||||
else:
|
||||
v = oper.getOperAddr(oper)
|
||||
|
||||
if f.vw.probeMemory(v, 1, envi.memory.MM_READ):
|
||||
# this is a valid address
|
||||
# assume its not also a constant.
|
||||
continue
|
||||
|
||||
if insn.mnem == "add" and insn.opers[0].isReg() and insn.opers[0].reg == envi.archs.i386.disasm.REG_ESP:
|
||||
# skip things like:
|
||||
#
|
||||
# .text:00401140 call sub_407E2B
|
||||
# .text:00401145 add esp, 0Ch
|
||||
return
|
||||
|
||||
yield Number(v), insn.va
|
||||
yield Number(v, arch=get_arch(f.vw)), insn.va
|
||||
|
||||
|
||||
def derefs(vw, p):
|
||||
@@ -196,7 +214,7 @@ def derefs(vw, p):
|
||||
p = next
|
||||
|
||||
|
||||
def read_memory(vw, va: int, size: int) -> bytes:
|
||||
def read_memory(vw, va, size):
|
||||
# as documented in #176, vivisect will not readMemory() when the section is not marked readable.
|
||||
#
|
||||
# but here, we don't care about permissions.
|
||||
@@ -209,10 +227,10 @@ def read_memory(vw, va: int, size: int) -> bytes:
|
||||
mva, msize, mperms, mfname = mmap
|
||||
offset = va - mva
|
||||
return mbytes[offset : offset + size]
|
||||
raise envi.exc.SegmentationViolation(va)
|
||||
raise envi.SegmentationViolation(va)
|
||||
|
||||
|
||||
def read_bytes(vw, va: int) -> bytes:
|
||||
def read_bytes(vw, va):
|
||||
"""
|
||||
read up to MAX_BYTES_FEATURE_SIZE from the given address.
|
||||
|
||||
@@ -221,7 +239,7 @@ def read_bytes(vw, va: int) -> bytes:
|
||||
"""
|
||||
segm = vw.getSegment(va)
|
||||
if not segm:
|
||||
raise envi.exc.SegmentationViolation(va)
|
||||
raise envi.SegmentationViolation(va)
|
||||
|
||||
segm_end = segm[0] + segm[1]
|
||||
try:
|
||||
@@ -230,19 +248,16 @@ def read_bytes(vw, va: int) -> bytes:
|
||||
return read_memory(vw, va, segm_end - va)
|
||||
else:
|
||||
return read_memory(vw, va, MAX_BYTES_FEATURE_SIZE)
|
||||
except envi.exc.SegmentationViolation:
|
||||
except envi.SegmentationViolation:
|
||||
raise
|
||||
|
||||
|
||||
def extract_insn_bytes_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_bytes_features(f, bb, insn):
|
||||
"""
|
||||
parse byte sequence features from the given instruction.
|
||||
example:
|
||||
# push offset iid_004118d4_IShellLinkA ; riid
|
||||
"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
|
||||
if insn.mnem == "call":
|
||||
return
|
||||
|
||||
@@ -265,19 +280,19 @@ def extract_insn_bytes_features(fh: FunctionHandle, bb, ih: InsnHandle) -> Itera
|
||||
for v in derefs(f.vw, v):
|
||||
try:
|
||||
buf = read_bytes(f.vw, v)
|
||||
except envi.exc.SegmentationViolation:
|
||||
except envi.SegmentationViolation:
|
||||
continue
|
||||
|
||||
if capa.features.extractors.helpers.all_zeros(buf):
|
||||
continue
|
||||
|
||||
yield Bytes(buf), ih.address
|
||||
yield Bytes(buf), insn.va
|
||||
|
||||
|
||||
def read_string(vw, offset: int) -> str:
|
||||
def read_string(vw, offset):
|
||||
try:
|
||||
alen = vw.detectString(offset)
|
||||
except envi.exc.SegmentationViolation:
|
||||
except envi.SegmentationViolation:
|
||||
pass
|
||||
else:
|
||||
if alen > 0:
|
||||
@@ -285,7 +300,7 @@ def read_string(vw, offset: int) -> str:
|
||||
|
||||
try:
|
||||
ulen = vw.detectUnicode(offset)
|
||||
except envi.exc.SegmentationViolation:
|
||||
except envi.SegmentationViolation:
|
||||
pass
|
||||
except IndexError:
|
||||
# potential vivisect bug detecting Unicode at segment end
|
||||
@@ -305,18 +320,87 @@ def read_string(vw, offset: int) -> str:
|
||||
raise ValueError("not a string", offset)
|
||||
|
||||
|
||||
def is_security_cookie(f, bb, insn) -> bool:
|
||||
def extract_insn_string_features(f, bb, insn):
|
||||
"""parse string features from the given instruction."""
|
||||
# example:
|
||||
#
|
||||
# push offset aAcr ; "ACR > "
|
||||
|
||||
for oper in insn.opers:
|
||||
if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
|
||||
v = oper.getOperValue(oper)
|
||||
elif isinstance(oper, envi.archs.i386.disasm.i386ImmMemOper):
|
||||
# like 0x10056CB4 in `lea eax, dword [0x10056CB4]`
|
||||
v = oper.imm
|
||||
elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
|
||||
# like 0x401000 in `mov eax, 0x401000[2 * ebx]`
|
||||
v = oper.imm
|
||||
elif isinstance(oper, envi.archs.amd64.disasm.Amd64RipRelOper):
|
||||
v = oper.getOperAddr(insn)
|
||||
else:
|
||||
continue
|
||||
|
||||
for v in derefs(f.vw, v):
|
||||
try:
|
||||
s = read_string(f.vw, v)
|
||||
except ValueError:
|
||||
continue
|
||||
else:
|
||||
yield String(s.rstrip("\x00")), insn.va
|
||||
|
||||
|
||||
def extract_insn_offset_features(f, bb, insn):
|
||||
"""parse structure offset features from the given instruction."""
|
||||
# example:
|
||||
#
|
||||
# .text:0040112F cmp [esi+4], ebx
|
||||
for oper in insn.opers:
|
||||
|
||||
# this is for both x32 and x64
|
||||
# like [esi + 4]
|
||||
# reg ^
|
||||
# disp
|
||||
if isinstance(oper, envi.archs.i386.disasm.i386RegMemOper):
|
||||
if oper.reg == envi.archs.i386.disasm.REG_ESP:
|
||||
continue
|
||||
|
||||
if oper.reg == envi.archs.i386.disasm.REG_EBP:
|
||||
continue
|
||||
|
||||
# TODO: do x64 support for real.
|
||||
if oper.reg == envi.archs.amd64.disasm.REG_RBP:
|
||||
continue
|
||||
|
||||
# viv already decodes offsets as signed
|
||||
v = oper.disp
|
||||
|
||||
yield Offset(v), insn.va
|
||||
yield Offset(v, arch=get_arch(f.vw)), insn.va
|
||||
|
||||
# like: [esi + ecx + 16384]
|
||||
# reg ^ ^
|
||||
# index ^
|
||||
# disp
|
||||
elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
|
||||
# viv already decodes offsets as signed
|
||||
v = oper.disp
|
||||
|
||||
yield Offset(v), insn.va
|
||||
yield Offset(v, arch=get_arch(f.vw)), insn.va
|
||||
|
||||
|
||||
def is_security_cookie(f, bb, insn):
|
||||
"""
|
||||
check if an instruction is related to security cookie checks
|
||||
"""
|
||||
# security cookie check should use SP or BP
|
||||
oper = insn.opers[1]
|
||||
if oper.isReg() and oper.reg not in [
|
||||
envi.archs.i386.regs.REG_ESP,
|
||||
envi.archs.i386.regs.REG_EBP,
|
||||
envi.archs.i386.disasm.REG_ESP,
|
||||
envi.archs.i386.disasm.REG_EBP,
|
||||
# TODO: do x64 support for real.
|
||||
envi.archs.amd64.regs.REG_RBP,
|
||||
envi.archs.amd64.regs.REG_RSP,
|
||||
envi.archs.amd64.disasm.REG_RBP,
|
||||
envi.archs.amd64.disasm.REG_RSP,
|
||||
]:
|
||||
return False
|
||||
|
||||
@@ -333,17 +417,11 @@ def is_security_cookie(f, bb, insn) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def extract_insn_nzxor_characteristic_features(
|
||||
fh: FunctionHandle, bbhandle: BBHandle, ih: InsnHandle
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_nzxor_characteristic_features(f, bb, insn):
|
||||
"""
|
||||
parse non-zeroing XOR instruction from the given instruction.
|
||||
ignore expected non-zeroing XORs, e.g. security cookies.
|
||||
"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
bb: viv_utils.BasicBlock = bbhandle.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
|
||||
if insn.mnem not in ("xor", "xorpd", "xorps", "pxor"):
|
||||
return
|
||||
|
||||
@@ -353,40 +431,19 @@ def extract_insn_nzxor_characteristic_features(
|
||||
if is_security_cookie(f, bb, insn):
|
||||
return
|
||||
|
||||
yield Characteristic("nzxor"), ih.address
|
||||
yield Characteristic("nzxor"), insn.va
|
||||
|
||||
|
||||
def extract_insn_mnemonic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_mnemonic_features(f, bb, insn):
|
||||
"""parse mnemonic features from the given instruction."""
|
||||
yield Mnemonic(ih.inner.mnem), ih.address
|
||||
yield Mnemonic(insn.mnem), insn.va
|
||||
|
||||
|
||||
def extract_insn_obfs_call_plus_5_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""
|
||||
parse call $+5 instruction from the given instruction.
|
||||
"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
|
||||
if insn.mnem != "call":
|
||||
return
|
||||
|
||||
if isinstance(insn.opers[0], envi.archs.i386.disasm.i386PcRelOper):
|
||||
if insn.va + 5 == insn.opers[0].getOperValue(insn):
|
||||
yield Characteristic("call $+5"), ih.address
|
||||
|
||||
if isinstance(insn.opers[0], envi.archs.i386.disasm.i386ImmMemOper) or isinstance(
|
||||
insn.opers[0], envi.archs.amd64.disasm.Amd64RipRelOper
|
||||
):
|
||||
if insn.va + 5 == insn.opers[0].getOperAddr(insn):
|
||||
yield Characteristic("call $+5"), ih.address
|
||||
|
||||
|
||||
def extract_insn_peb_access_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_peb_access_characteristic_features(f, bb, insn):
|
||||
"""
|
||||
parse peb access from the given function. fs:[0x30] on x86, gs:[0x60] on x64
|
||||
"""
|
||||
# TODO handle where fs/gs are loaded into a register or onto the stack and used later
|
||||
insn: envi.Opcode = ih.inner
|
||||
|
||||
if insn.mnem not in ["push", "mov"]:
|
||||
return
|
||||
@@ -405,7 +462,7 @@ def extract_insn_peb_access_characteristic_features(f, bb, ih: InsnHandle) -> It
|
||||
if (isinstance(oper, envi.archs.i386.disasm.i386RegMemOper) and oper.disp == 0x30) or (
|
||||
isinstance(oper, envi.archs.i386.disasm.i386ImmMemOper) and oper.imm == 0x30
|
||||
):
|
||||
yield Characteristic("peb access"), ih.address
|
||||
yield Characteristic("peb access"), insn.va
|
||||
elif "gs" in prefix:
|
||||
for oper in insn.opers:
|
||||
if (
|
||||
@@ -413,25 +470,23 @@ def extract_insn_peb_access_characteristic_features(f, bb, ih: InsnHandle) -> It
|
||||
or (isinstance(oper, envi.archs.amd64.disasm.i386SibOper) and oper.imm == 0x60)
|
||||
or (isinstance(oper, envi.archs.amd64.disasm.i386ImmMemOper) and oper.imm == 0x60)
|
||||
):
|
||||
yield Characteristic("peb access"), ih.address
|
||||
yield Characteristic("peb access"), insn.va
|
||||
else:
|
||||
pass
|
||||
|
||||
|
||||
def extract_insn_segment_access_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse the instruction for access to fs or gs"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
|
||||
def extract_insn_segment_access_features(f, bb, insn):
|
||||
""" parse the instruction for access to fs or gs """
|
||||
prefix = insn.getPrefixName()
|
||||
|
||||
if prefix == "fs":
|
||||
yield Characteristic("fs access"), ih.address
|
||||
yield Characteristic("fs access"), insn.va
|
||||
|
||||
if prefix == "gs":
|
||||
yield Characteristic("gs access"), ih.address
|
||||
yield Characteristic("gs access"), insn.va
|
||||
|
||||
|
||||
def get_section(vw, va: int):
|
||||
def get_section(vw, va):
|
||||
for start, length, _, __ in vw.getMemoryMaps():
|
||||
if start <= va < start + length:
|
||||
return start
|
||||
@@ -439,13 +494,10 @@ def get_section(vw, va: int):
|
||||
raise KeyError(va)
|
||||
|
||||
|
||||
def extract_insn_cross_section_cflow(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_insn_cross_section_cflow(f, bb, insn):
|
||||
"""
|
||||
inspect the instruction for a CALL or JMP that crosses section boundaries.
|
||||
"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
|
||||
for va, flags in insn.getBranches():
|
||||
if va is None:
|
||||
# va may be none for dynamic branches that haven't been resolved, such as `jmp eax`.
|
||||
@@ -472,7 +524,7 @@ def extract_insn_cross_section_cflow(fh: FunctionHandle, bb, ih: InsnHandle) ->
|
||||
continue
|
||||
|
||||
if get_section(f.vw, insn.va) != get_section(f.vw, va):
|
||||
yield Characteristic("cross section flow"), ih.address
|
||||
yield Characteristic("cross section flow"), insn.va
|
||||
|
||||
except KeyError:
|
||||
continue
|
||||
@@ -480,10 +532,7 @@ def extract_insn_cross_section_cflow(fh: FunctionHandle, bb, ih: InsnHandle) ->
|
||||
|
||||
# this is a feature that's most relevant at the function scope,
|
||||
# however, its most efficient to extract at the instruction scope.
|
||||
def extract_function_calls_from(fh: FunctionHandle, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
insn: envi.Opcode = ih.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
|
||||
def extract_function_calls_from(f, bb, insn):
|
||||
if insn.mnem != "call":
|
||||
return
|
||||
|
||||
@@ -493,7 +542,7 @@ def extract_function_calls_from(fh: FunctionHandle, bb, ih: InsnHandle) -> Itera
|
||||
if isinstance(insn.opers[0], envi.archs.i386.disasm.i386ImmMemOper):
|
||||
oper = insn.opers[0]
|
||||
target = oper.getOperAddr(insn)
|
||||
yield Characteristic("calls from"), AbsoluteVirtualAddress(target)
|
||||
yield Characteristic("calls from"), target
|
||||
|
||||
# call via thunk on x86,
|
||||
# see 9324d1a8ae37a36ae560c37448c9705a at 0x407985
|
||||
@@ -502,191 +551,43 @@ def extract_function_calls_from(fh: FunctionHandle, bb, ih: InsnHandle) -> Itera
|
||||
# see Lab21-01.exe_:0x140001178
|
||||
elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386PcRelOper):
|
||||
target = insn.opers[0].getOperValue(insn)
|
||||
if target >= 0:
|
||||
yield Characteristic("calls from"), AbsoluteVirtualAddress(target)
|
||||
yield Characteristic("calls from"), target
|
||||
|
||||
# call via IAT, x64
|
||||
elif isinstance(insn.opers[0], envi.archs.amd64.disasm.Amd64RipRelOper):
|
||||
op = insn.opers[0]
|
||||
target = op.getOperAddr(insn)
|
||||
yield Characteristic("calls from"), AbsoluteVirtualAddress(target)
|
||||
yield Characteristic("calls from"), target
|
||||
|
||||
if target and target == f.va:
|
||||
# if we found a jump target and it's the function address
|
||||
# mark as recursive
|
||||
yield Characteristic("recursive call"), AbsoluteVirtualAddress(target)
|
||||
yield Characteristic("recursive call"), target
|
||||
|
||||
|
||||
# this is a feature that's most relevant at the function or basic block scope,
|
||||
# however, its most efficient to extract at the instruction scope.
|
||||
def extract_function_indirect_call_characteristic_features(f, bb, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_function_indirect_call_characteristic_features(f, bb, insn):
|
||||
"""
|
||||
extract indirect function call characteristic (e.g., call eax or call dword ptr [edx+4])
|
||||
does not include calls like => call ds:dword_ABD4974
|
||||
"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
|
||||
if insn.mnem != "call":
|
||||
return
|
||||
|
||||
# Checks below work for x86 and x64
|
||||
if isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegOper):
|
||||
# call edx
|
||||
yield Characteristic("indirect call"), ih.address
|
||||
yield Characteristic("indirect call"), insn.va
|
||||
elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386RegMemOper):
|
||||
# call dword ptr [eax+50h]
|
||||
yield Characteristic("indirect call"), ih.address
|
||||
yield Characteristic("indirect call"), insn.va
|
||||
elif isinstance(insn.opers[0], envi.archs.i386.disasm.i386SibOper):
|
||||
# call qword ptr [rsp+78h]
|
||||
yield Characteristic("indirect call"), ih.address
|
||||
yield Characteristic("indirect call"), insn.va
|
||||
|
||||
|
||||
def extract_op_number_features(
|
||||
fh: FunctionHandle, bb, ih: InsnHandle, i, oper: envi.Operand
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse number features from the given operand.
|
||||
|
||||
example:
|
||||
push 3136B0h ; dwControlCode
|
||||
"""
|
||||
insn: envi.Opcode = ih.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
|
||||
# this is for both x32 and x64
|
||||
if not isinstance(oper, (envi.archs.i386.disasm.i386ImmOper, envi.archs.i386.disasm.i386ImmMemOper)):
|
||||
return
|
||||
|
||||
if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
|
||||
v = oper.getOperValue(oper)
|
||||
else:
|
||||
v = oper.getOperAddr(oper)
|
||||
|
||||
if f.vw.probeMemory(v, 1, envi.memory.MM_READ):
|
||||
# this is a valid address
|
||||
# assume its not also a constant.
|
||||
return
|
||||
|
||||
if insn.mnem == "add" and insn.opers[0].isReg() and insn.opers[0].reg == envi.archs.i386.regs.REG_ESP:
|
||||
# skip things like:
|
||||
#
|
||||
# .text:00401140 call sub_407E2B
|
||||
# .text:00401145 add esp, 0Ch
|
||||
return
|
||||
|
||||
yield Number(v), ih.address
|
||||
yield OperandNumber(i, v), ih.address
|
||||
|
||||
if insn.mnem == "add" and 0 < v < MAX_STRUCTURE_SIZE and isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
|
||||
# for pattern like:
|
||||
#
|
||||
# add eax, 0x10
|
||||
#
|
||||
# assume 0x10 is also an offset (imagine eax is a pointer).
|
||||
yield Offset(v), ih.address
|
||||
yield OperandOffset(i, v), ih.address
|
||||
|
||||
|
||||
def extract_op_offset_features(
|
||||
fh: FunctionHandle, bb, ih: InsnHandle, i, oper: envi.Operand
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse structure offset features from the given operand."""
|
||||
# example:
|
||||
#
|
||||
# .text:0040112F cmp [esi+4], ebx
|
||||
insn: envi.Opcode = ih.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
|
||||
# this is for both x32 and x64
|
||||
# like [esi + 4]
|
||||
# reg ^
|
||||
# disp
|
||||
if isinstance(oper, envi.archs.i386.disasm.i386RegMemOper):
|
||||
if oper.reg == envi.archs.i386.regs.REG_ESP:
|
||||
return
|
||||
|
||||
if oper.reg == envi.archs.i386.regs.REG_EBP:
|
||||
return
|
||||
|
||||
# TODO: do x64 support for real.
|
||||
if oper.reg == envi.archs.amd64.regs.REG_RBP:
|
||||
return
|
||||
|
||||
# viv already decodes offsets as signed
|
||||
v = oper.disp
|
||||
|
||||
yield Offset(v), ih.address
|
||||
yield OperandOffset(i, v), ih.address
|
||||
|
||||
if insn.mnem == "lea" and i == 1 and not f.vw.probeMemory(v, 1, envi.memory.MM_READ):
|
||||
# for pattern like:
|
||||
#
|
||||
# lea eax, [ebx + 1]
|
||||
#
|
||||
# assume 1 is also an offset (imagine ebx is a zero register).
|
||||
yield Number(v), ih.address
|
||||
yield OperandNumber(i, v), ih.address
|
||||
|
||||
# like: [esi + ecx + 16384]
|
||||
# reg ^ ^
|
||||
# index ^
|
||||
# disp
|
||||
elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
|
||||
# viv already decodes offsets as signed
|
||||
v = oper.disp
|
||||
|
||||
yield Offset(v), ih.address
|
||||
yield OperandOffset(i, v), ih.address
|
||||
|
||||
|
||||
def extract_op_string_features(
|
||||
fh: FunctionHandle, bb, ih: InsnHandle, i, oper: envi.Operand
|
||||
) -> Iterator[Tuple[Feature, Address]]:
|
||||
"""parse string features from the given operand."""
|
||||
# example:
|
||||
#
|
||||
# push offset aAcr ; "ACR > "
|
||||
insn: envi.Opcode = ih.inner
|
||||
f: viv_utils.Function = fh.inner
|
||||
|
||||
if isinstance(oper, envi.archs.i386.disasm.i386ImmOper):
|
||||
v = oper.getOperValue(oper)
|
||||
elif isinstance(oper, envi.archs.i386.disasm.i386ImmMemOper):
|
||||
# like 0x10056CB4 in `lea eax, dword [0x10056CB4]`
|
||||
v = oper.imm
|
||||
elif isinstance(oper, envi.archs.i386.disasm.i386SibOper):
|
||||
# like 0x401000 in `mov eax, 0x401000[2 * ebx]`
|
||||
v = oper.imm
|
||||
elif isinstance(oper, envi.archs.amd64.disasm.Amd64RipRelOper):
|
||||
v = oper.getOperAddr(insn)
|
||||
else:
|
||||
return
|
||||
|
||||
for v in derefs(f.vw, v):
|
||||
try:
|
||||
s = read_string(f.vw, v)
|
||||
except ValueError:
|
||||
continue
|
||||
else:
|
||||
yield String(s.rstrip("\x00")), ih.address
|
||||
|
||||
|
||||
def extract_operand_features(f: FunctionHandle, bb, insn: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
|
||||
for i, oper in enumerate(insn.inner.opers):
|
||||
for op_handler in OPERAND_HANDLERS:
|
||||
for feature, addr in op_handler(f, bb, insn, i, oper):
|
||||
yield feature, addr
|
||||
|
||||
|
||||
OPERAND_HANDLERS: List[
|
||||
Callable[[FunctionHandle, BBHandle, InsnHandle, int, envi.Operand], Iterator[Tuple[Feature, Address]]]
|
||||
] = [
|
||||
extract_op_number_features,
|
||||
extract_op_offset_features,
|
||||
extract_op_string_features,
|
||||
]
|
||||
|
||||
|
||||
def extract_features(f, bb, insn) -> Iterator[Tuple[Feature, Address]]:
|
||||
def extract_features(f, bb, insn):
|
||||
"""
|
||||
extract features from the given insn.
|
||||
|
||||
@@ -696,23 +597,24 @@ def extract_features(f, bb, insn) -> Iterator[Tuple[Feature, Address]]:
|
||||
insn (vivisect...Instruction): the instruction to process.
|
||||
|
||||
yields:
|
||||
Tuple[Feature, Address]: the features and their location found in this insn.
|
||||
Feature, set[VA]: the features and their location found in this insn.
|
||||
"""
|
||||
for insn_handler in INSTRUCTION_HANDLERS:
|
||||
for feature, addr in insn_handler(f, bb, insn):
|
||||
yield feature, addr
|
||||
for feature, va in insn_handler(f, bb, insn):
|
||||
yield feature, va
|
||||
|
||||
|
||||
INSTRUCTION_HANDLERS: List[Callable[[FunctionHandle, BBHandle, InsnHandle], Iterator[Tuple[Feature, Address]]]] = [
|
||||
INSTRUCTION_HANDLERS = (
|
||||
extract_insn_api_features,
|
||||
extract_insn_number_features,
|
||||
extract_insn_string_features,
|
||||
extract_insn_bytes_features,
|
||||
extract_insn_offset_features,
|
||||
extract_insn_nzxor_characteristic_features,
|
||||
extract_insn_mnemonic_features,
|
||||
extract_insn_obfs_call_plus_5_characteristic_features,
|
||||
extract_insn_peb_access_characteristic_features,
|
||||
extract_insn_cross_section_cflow,
|
||||
extract_insn_segment_access_features,
|
||||
extract_function_calls_from,
|
||||
extract_function_indirect_call_characteristic_features,
|
||||
extract_operand_features,
|
||||
]
|
||||
)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,33 +6,22 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from capa.features.common import Feature
|
||||
from capa.features import Feature
|
||||
|
||||
|
||||
class Export(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
def __init__(self, value, description=None):
|
||||
# value is export name
|
||||
super(Export, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class Import(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
def __init__(self, value, description=None):
|
||||
# value is import name
|
||||
super(Import, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class Section(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
def __init__(self, value, description=None):
|
||||
# value is section name
|
||||
super(Section, self).__init__(value, description=description)
|
||||
|
||||
|
||||
class FunctionName(Feature):
|
||||
"""recognized name for statically linked function"""
|
||||
|
||||
def __init__(self, name: str, description=None):
|
||||
# value is function name
|
||||
super(FunctionName, self).__init__(name, description=description)
|
||||
# override the name property set by `capa.features.Feature`
|
||||
# that would be `functionname` (note missing dash)
|
||||
self.name = "function-name"
|
||||
|
||||
299
capa/features/freeze.py
Normal file
299
capa/features/freeze.py
Normal file
@@ -0,0 +1,299 @@
|
||||
"""
|
||||
capa freeze file format: `| capa0000 | + zlib(utf-8(json(...)))`
|
||||
|
||||
json format:
|
||||
|
||||
{
|
||||
'version': 1,
|
||||
'base address': int(base address),
|
||||
'functions': {
|
||||
int(function va): {
|
||||
'basic blocks': {
|
||||
int(basic block va): {
|
||||
'instructions': [instruction va, ...]
|
||||
},
|
||||
...
|
||||
},
|
||||
...
|
||||
},
|
||||
...
|
||||
},
|
||||
'scopes': {
|
||||
'file': [
|
||||
(str(name), [any(arg), ...], int(va), ()),
|
||||
...
|
||||
},
|
||||
'function': [
|
||||
(str(name), [any(arg), ...], int(va), (int(function va), )),
|
||||
...
|
||||
],
|
||||
'basic block': [
|
||||
(str(name), [any(arg), ...], int(va), (int(function va),
|
||||
int(basic block va))),
|
||||
...
|
||||
],
|
||||
'instruction': [
|
||||
(str(name), [any(arg), ...], int(va), (int(function va),
|
||||
int(basic block va),
|
||||
int(instruction va))),
|
||||
...
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and limitations under the License.
|
||||
"""
|
||||
import json
|
||||
import zlib
|
||||
import logging
|
||||
|
||||
import capa.features
|
||||
import capa.features.file
|
||||
import capa.features.insn
|
||||
import capa.features.basicblock
|
||||
import capa.features.extractors
|
||||
from capa.helpers import hex
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def serialize_feature(feature):
|
||||
return feature.freeze_serialize()
|
||||
|
||||
|
||||
KNOWN_FEATURES = {F.__name__: F for F in capa.features.Feature.__subclasses__()}
|
||||
|
||||
|
||||
def deserialize_feature(doc):
|
||||
F = KNOWN_FEATURES[doc[0]]
|
||||
return F.freeze_deserialize(doc[1])
|
||||
|
||||
|
||||
def dumps(extractor):
|
||||
"""
|
||||
serialize the given extractor to a string
|
||||
|
||||
args:
|
||||
extractor: capa.features.extractor.FeatureExtractor:
|
||||
|
||||
returns:
|
||||
str: the serialized features.
|
||||
"""
|
||||
ret = {
|
||||
"version": 1,
|
||||
"base address": extractor.get_base_address(),
|
||||
"functions": {},
|
||||
"scopes": {
|
||||
"file": [],
|
||||
"function": [],
|
||||
"basic block": [],
|
||||
"instruction": [],
|
||||
},
|
||||
}
|
||||
|
||||
for feature, va in extractor.extract_file_features():
|
||||
ret["scopes"]["file"].append(serialize_feature(feature) + (hex(va), ()))
|
||||
|
||||
for f in extractor.get_functions():
|
||||
ret["functions"][hex(f)] = {}
|
||||
|
||||
for feature, va in extractor.extract_function_features(f):
|
||||
ret["scopes"]["function"].append(serialize_feature(feature) + (hex(va), (hex(f),)))
|
||||
|
||||
for bb in extractor.get_basic_blocks(f):
|
||||
ret["functions"][hex(f)][hex(bb)] = []
|
||||
|
||||
for feature, va in extractor.extract_basic_block_features(f, bb):
|
||||
ret["scopes"]["basic block"].append(
|
||||
serialize_feature(feature)
|
||||
+ (
|
||||
hex(va),
|
||||
(
|
||||
hex(f),
|
||||
hex(bb),
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
for insnva, insn in sorted(
|
||||
[(insn.__int__(), insn) for insn in extractor.get_instructions(f, bb)], key=lambda p: p[0]
|
||||
):
|
||||
ret["functions"][hex(f)][hex(bb)].append(hex(insnva))
|
||||
|
||||
for feature, va in extractor.extract_insn_features(f, bb, insn):
|
||||
ret["scopes"]["instruction"].append(
|
||||
serialize_feature(feature)
|
||||
+ (
|
||||
hex(va),
|
||||
(
|
||||
hex(f),
|
||||
hex(bb),
|
||||
hex(insnva),
|
||||
),
|
||||
)
|
||||
)
|
||||
return json.dumps(ret)
|
||||
|
||||
|
||||
def loads(s):
|
||||
"""deserialize a set of features (as a NullFeatureExtractor) from a string."""
|
||||
doc = json.loads(s)
|
||||
|
||||
if doc.get("version") != 1:
|
||||
raise ValueError("unsupported freeze format version: %d" % (doc.get("version")))
|
||||
|
||||
features = {
|
||||
"base address": doc.get("base address"),
|
||||
"file features": [],
|
||||
"functions": {},
|
||||
}
|
||||
|
||||
for fva, function in doc.get("functions", {}).items():
|
||||
fva = int(fva, 0x10)
|
||||
features["functions"][fva] = {
|
||||
"features": [],
|
||||
"basic blocks": {},
|
||||
}
|
||||
|
||||
for bbva, bb in function.items():
|
||||
bbva = int(bbva, 0x10)
|
||||
features["functions"][fva]["basic blocks"][bbva] = {
|
||||
"features": [],
|
||||
"instructions": {},
|
||||
}
|
||||
|
||||
for insnva in bb:
|
||||
insnva = int(insnva, 0x10)
|
||||
features["functions"][fva]["basic blocks"][bbva]["instructions"][insnva] = {
|
||||
"features": [],
|
||||
}
|
||||
|
||||
# in the following blocks, each entry looks like:
|
||||
#
|
||||
# ('MatchedRule', ('foo', ), '0x401000', ('0x401000', ))
|
||||
# ^^^^^^^^^^^^^ ^^^^^^^^^ ^^^^^^^^^^ ^^^^^^^^^^^^^^
|
||||
# feature name args addr func/bb/insn
|
||||
for feature in doc.get("scopes", {}).get("file", []):
|
||||
va, loc = feature[2:]
|
||||
va = int(va, 0x10)
|
||||
feature = deserialize_feature(feature[:2])
|
||||
features["file features"].append((va, feature))
|
||||
|
||||
for feature in doc.get("scopes", {}).get("function", []):
|
||||
# fetch the pair like:
|
||||
#
|
||||
# ('0x401000', ('0x401000', ))
|
||||
# ^^^^^^^^^^ ^^^^^^^^^^^^^^
|
||||
# addr func/bb/insn
|
||||
va, loc = feature[2:]
|
||||
va = int(va, 0x10)
|
||||
loc = [int(lo, 0x10) for lo in loc]
|
||||
|
||||
# decode the feature from the pair like:
|
||||
#
|
||||
# ('MatchedRule', ('foo', ))
|
||||
# ^^^^^^^^^^^^^ ^^^^^^^^^
|
||||
# feature name args
|
||||
feature = deserialize_feature(feature[:2])
|
||||
features["functions"][loc[0]]["features"].append((va, feature))
|
||||
|
||||
for feature in doc.get("scopes", {}).get("basic block", []):
|
||||
va, loc = feature[2:]
|
||||
va = int(va, 0x10)
|
||||
loc = [int(lo, 0x10) for lo in loc]
|
||||
feature = deserialize_feature(feature[:2])
|
||||
features["functions"][loc[0]]["basic blocks"][loc[1]]["features"].append((va, feature))
|
||||
|
||||
for feature in doc.get("scopes", {}).get("instruction", []):
|
||||
va, loc = feature[2:]
|
||||
va = int(va, 0x10)
|
||||
loc = [int(lo, 0x10) for lo in loc]
|
||||
feature = deserialize_feature(feature[:2])
|
||||
features["functions"][loc[0]]["basic blocks"][loc[1]]["instructions"][loc[2]]["features"].append((va, feature))
|
||||
|
||||
return capa.features.extractors.NullFeatureExtractor(features)
|
||||
|
||||
|
||||
MAGIC = "capa0000".encode("ascii")
|
||||
|
||||
|
||||
def dump(extractor):
|
||||
"""serialize the given extractor to a byte array."""
|
||||
return MAGIC + zlib.compress(dumps(extractor).encode("utf-8"))
|
||||
|
||||
|
||||
def is_freeze(buf):
|
||||
return buf[: len(MAGIC)] == MAGIC
|
||||
|
||||
|
||||
def load(buf):
|
||||
"""deserialize a set of features (as a NullFeatureExtractor) from a byte array."""
|
||||
if not is_freeze(buf):
|
||||
raise ValueError("missing magic header")
|
||||
return loads(zlib.decompress(buf[len(MAGIC) :]).decode("utf-8"))
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
import sys
|
||||
import argparse
|
||||
|
||||
import capa.main
|
||||
|
||||
if argv is None:
|
||||
argv = sys.argv[1:]
|
||||
|
||||
formats = [
|
||||
("auto", "(default) detect file type automatically"),
|
||||
("pe", "Windows PE file"),
|
||||
("sc32", "32-bit shellcode"),
|
||||
("sc64", "64-bit shellcode"),
|
||||
]
|
||||
format_help = ", ".join(["%s: %s" % (f[0], f[1]) for f in formats])
|
||||
|
||||
parser = argparse.ArgumentParser(description="save capa features to a file")
|
||||
parser.add_argument("sample", type=str, help="Path to sample to analyze")
|
||||
parser.add_argument("output", type=str, help="Path to output file")
|
||||
parser.add_argument("-v", "--verbose", action="store_true", help="Enable verbose output")
|
||||
parser.add_argument("-q", "--quiet", action="store_true", help="Disable all output but errors")
|
||||
parser.add_argument(
|
||||
"-f", "--format", choices=[f[0] for f in formats], default="auto", help="Select sample format, %s" % format_help
|
||||
)
|
||||
if sys.version_info >= (3, 0):
|
||||
parser.add_argument(
|
||||
"-b",
|
||||
"--backend",
|
||||
type=str,
|
||||
help="select the backend to use",
|
||||
choices=(capa.main.BACKEND_VIV, capa.main.BACKEND_SMDA),
|
||||
default=capa.main.BACKEND_VIV,
|
||||
)
|
||||
args = parser.parse_args(args=argv)
|
||||
|
||||
if args.quiet:
|
||||
logging.basicConfig(level=logging.ERROR)
|
||||
logging.getLogger().setLevel(logging.ERROR)
|
||||
elif args.verbose:
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
else:
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
|
||||
backend = args.backend if sys.version_info > (3, 0) else capa.main.BACKEND_VIV
|
||||
extractor = capa.main.get_extractor(args.sample, args.format, backend)
|
||||
with open(args.output, "wb") as f:
|
||||
f.write(dump(extractor))
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
sys.exit(main())
|
||||
@@ -1,393 +0,0 @@
|
||||
"""
|
||||
capa freeze file format: `| capa0000 | + zlib(utf-8(json(...)))`
|
||||
|
||||
Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and limitations under the License.
|
||||
"""
|
||||
import zlib
|
||||
import logging
|
||||
from enum import Enum
|
||||
from typing import Any, List, Tuple
|
||||
|
||||
import dncil.clr.token
|
||||
from pydantic import Field, BaseModel
|
||||
|
||||
import capa.helpers
|
||||
import capa.version
|
||||
import capa.features.file
|
||||
import capa.features.insn
|
||||
import capa.features.common
|
||||
import capa.features.address
|
||||
import capa.features.basicblock
|
||||
import capa.features.extractors.base_extractor
|
||||
from capa.helpers import assert_never
|
||||
from capa.features.freeze.features import Feature, feature_from_capa
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class HashableModel(BaseModel):
|
||||
class Config:
|
||||
frozen = True
|
||||
|
||||
|
||||
class AddressType(str, Enum):
|
||||
ABSOLUTE = "absolute"
|
||||
RELATIVE = "relative"
|
||||
FILE = "file"
|
||||
DN_TOKEN = "dn token"
|
||||
DN_TOKEN_OFFSET = "dn token offset"
|
||||
NO_ADDRESS = "no address"
|
||||
|
||||
|
||||
class Address(HashableModel):
|
||||
type: AddressType
|
||||
value: Any
|
||||
|
||||
@classmethod
|
||||
def from_capa(cls, a: capa.features.address.Address) -> "Address":
|
||||
if isinstance(a, capa.features.address.AbsoluteVirtualAddress):
|
||||
return cls(type=AddressType.ABSOLUTE, value=int(a))
|
||||
|
||||
elif isinstance(a, capa.features.address.RelativeVirtualAddress):
|
||||
return cls(type=AddressType.RELATIVE, value=int(a))
|
||||
|
||||
elif isinstance(a, capa.features.address.FileOffsetAddress):
|
||||
return cls(type=AddressType.FILE, value=int(a))
|
||||
|
||||
elif isinstance(a, capa.features.address.DNTokenAddress):
|
||||
return cls(type=AddressType.DN_TOKEN, value=a.token.value)
|
||||
|
||||
elif isinstance(a, capa.features.address.DNTokenOffsetAddress):
|
||||
return cls(type=AddressType.DN_TOKEN_OFFSET, value=(a.token.value, a.offset))
|
||||
|
||||
elif a == capa.features.address.NO_ADDRESS or isinstance(a, capa.features.address._NoAddress):
|
||||
return cls(type=AddressType.NO_ADDRESS, value=None)
|
||||
|
||||
elif isinstance(a, capa.features.address.Address) and not issubclass(type(a), capa.features.address.Address):
|
||||
raise ValueError("don't use an Address instance directly")
|
||||
|
||||
elif isinstance(a, capa.features.address.Address):
|
||||
raise ValueError("don't use an Address instance directly")
|
||||
|
||||
else:
|
||||
assert_never(a)
|
||||
|
||||
def to_capa(self) -> capa.features.address.Address:
|
||||
if self.type is AddressType.ABSOLUTE:
|
||||
return capa.features.address.AbsoluteVirtualAddress(self.value)
|
||||
|
||||
elif self.type is AddressType.RELATIVE:
|
||||
return capa.features.address.RelativeVirtualAddress(self.value)
|
||||
|
||||
elif self.type is AddressType.FILE:
|
||||
return capa.features.address.FileOffsetAddress(self.value)
|
||||
|
||||
elif self.type is AddressType.DN_TOKEN:
|
||||
return capa.features.address.DNTokenAddress(dncil.clr.token.Token(self.value))
|
||||
|
||||
elif self.type is AddressType.DN_TOKEN_OFFSET:
|
||||
token, offset = self.value
|
||||
return capa.features.address.DNTokenOffsetAddress(dncil.clr.token.Token(token), offset)
|
||||
|
||||
elif self.type is AddressType.NO_ADDRESS:
|
||||
return capa.features.address.NO_ADDRESS
|
||||
|
||||
else:
|
||||
assert_never(self.type)
|
||||
|
||||
def __lt__(self, other: "Address") -> bool:
|
||||
if self.type != other.type:
|
||||
return self.type < other.type
|
||||
|
||||
if self.type is AddressType.NO_ADDRESS:
|
||||
return True
|
||||
|
||||
else:
|
||||
return self.value < other.value
|
||||
|
||||
|
||||
class GlobalFeature(HashableModel):
|
||||
feature: Feature
|
||||
|
||||
|
||||
class FileFeature(HashableModel):
|
||||
address: Address
|
||||
feature: Feature
|
||||
|
||||
|
||||
class FunctionFeature(HashableModel):
|
||||
"""
|
||||
args:
|
||||
function: the address of the function to which this feature belongs.
|
||||
address: the address at which this feature is found.
|
||||
|
||||
function != address because, e.g., the feature may be found *within* the scope (function).
|
||||
versus right at its starting address.
|
||||
"""
|
||||
|
||||
function: Address
|
||||
address: Address
|
||||
feature: Feature
|
||||
|
||||
|
||||
class BasicBlockFeature(HashableModel):
|
||||
"""
|
||||
args:
|
||||
basic_block: the address of the basic block to which this feature belongs.
|
||||
address: the address at which this feature is found.
|
||||
|
||||
basic_block != address because, e.g., the feature may be found *within* the scope (basic block).
|
||||
versus right at its starting address.
|
||||
"""
|
||||
|
||||
basic_block: Address
|
||||
address: Address
|
||||
feature: Feature
|
||||
|
||||
|
||||
class InstructionFeature(HashableModel):
|
||||
"""
|
||||
args:
|
||||
instruction: the address of the instruction to which this feature belongs.
|
||||
address: the address at which this feature is found.
|
||||
|
||||
instruction != address because, e.g., the feature may be found *within* the scope (basic block),
|
||||
versus right at its starting address.
|
||||
"""
|
||||
|
||||
instruction: Address
|
||||
address: Address
|
||||
feature: Feature
|
||||
|
||||
|
||||
class InstructionFeatures(BaseModel):
|
||||
address: Address
|
||||
features: Tuple[InstructionFeature, ...]
|
||||
|
||||
|
||||
class BasicBlockFeatures(BaseModel):
|
||||
address: Address
|
||||
features: Tuple[BasicBlockFeature, ...]
|
||||
instructions: Tuple[InstructionFeatures, ...]
|
||||
|
||||
|
||||
class FunctionFeatures(BaseModel):
|
||||
address: Address
|
||||
features: Tuple[FunctionFeature, ...]
|
||||
basic_blocks: Tuple[BasicBlockFeatures, ...] = Field(alias="basic block")
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
|
||||
|
||||
class Features(BaseModel):
|
||||
global_: Tuple[GlobalFeature, ...] = Field(alias="global")
|
||||
file: Tuple[FileFeature, ...]
|
||||
functions: Tuple[FunctionFeatures, ...]
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
|
||||
|
||||
class Extractor(BaseModel):
|
||||
name: str
|
||||
version: str = capa.version.__version__
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
|
||||
|
||||
class Freeze(BaseModel):
|
||||
version: int = 2
|
||||
base_address: Address = Field(alias="base address")
|
||||
extractor: Extractor
|
||||
features: Features
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
|
||||
|
||||
def dumps(extractor: capa.features.extractors.base_extractor.FeatureExtractor) -> str:
|
||||
"""
|
||||
serialize the given extractor to a string
|
||||
"""
|
||||
|
||||
global_features: List[GlobalFeature] = []
|
||||
for feature, _ in extractor.extract_global_features():
|
||||
global_features.append(
|
||||
GlobalFeature(
|
||||
feature=feature_from_capa(feature),
|
||||
)
|
||||
)
|
||||
|
||||
file_features: List[FileFeature] = []
|
||||
for feature, address in extractor.extract_file_features():
|
||||
file_features.append(
|
||||
FileFeature(
|
||||
feature=feature_from_capa(feature),
|
||||
address=Address.from_capa(address),
|
||||
)
|
||||
)
|
||||
|
||||
function_features: List[FunctionFeatures] = []
|
||||
for f in extractor.get_functions():
|
||||
faddr = Address.from_capa(f.address)
|
||||
ffeatures = [
|
||||
FunctionFeature(
|
||||
function=faddr,
|
||||
address=Address.from_capa(addr),
|
||||
feature=feature_from_capa(feature),
|
||||
)
|
||||
for feature, addr in extractor.extract_function_features(f)
|
||||
]
|
||||
|
||||
basic_blocks = []
|
||||
for bb in extractor.get_basic_blocks(f):
|
||||
bbaddr = Address.from_capa(bb.address)
|
||||
bbfeatures = [
|
||||
BasicBlockFeature(
|
||||
basic_block=bbaddr,
|
||||
address=Address.from_capa(addr),
|
||||
feature=feature_from_capa(feature),
|
||||
)
|
||||
for feature, addr in extractor.extract_basic_block_features(f, bb)
|
||||
]
|
||||
|
||||
instructions = []
|
||||
for insn in extractor.get_instructions(f, bb):
|
||||
iaddr = Address.from_capa(insn.address)
|
||||
ifeatures = [
|
||||
InstructionFeature(
|
||||
instruction=iaddr,
|
||||
address=Address.from_capa(addr),
|
||||
feature=feature_from_capa(feature),
|
||||
)
|
||||
for feature, addr in extractor.extract_insn_features(f, bb, insn)
|
||||
]
|
||||
|
||||
instructions.append(
|
||||
InstructionFeatures(
|
||||
address=iaddr,
|
||||
features=ifeatures,
|
||||
)
|
||||
)
|
||||
|
||||
basic_blocks.append(
|
||||
BasicBlockFeatures(
|
||||
address=bbaddr,
|
||||
features=bbfeatures,
|
||||
instructions=instructions,
|
||||
)
|
||||
)
|
||||
|
||||
function_features.append(
|
||||
FunctionFeatures(
|
||||
address=faddr,
|
||||
features=ffeatures,
|
||||
basic_blocks=basic_blocks,
|
||||
)
|
||||
)
|
||||
|
||||
features = Features(
|
||||
global_=global_features,
|
||||
file=file_features,
|
||||
functions=function_features,
|
||||
)
|
||||
|
||||
freeze = Freeze(
|
||||
version=2,
|
||||
base_address=Address.from_capa(extractor.get_base_address()),
|
||||
extractor=Extractor(name=extractor.__class__.__name__),
|
||||
features=features,
|
||||
)
|
||||
|
||||
return freeze.json()
|
||||
|
||||
|
||||
def loads(s: str) -> capa.features.extractors.base_extractor.FeatureExtractor:
|
||||
"""deserialize a set of features (as a NullFeatureExtractor) from a string."""
|
||||
import capa.features.extractors.null as null
|
||||
|
||||
freeze = Freeze.parse_raw(s)
|
||||
if freeze.version != 2:
|
||||
raise ValueError("unsupported freeze format version: %d", freeze.version)
|
||||
|
||||
return null.NullFeatureExtractor(
|
||||
base_address=freeze.base_address.to_capa(),
|
||||
global_features=[f.feature.to_capa() for f in freeze.features.global_],
|
||||
file_features=[(f.address.to_capa(), f.feature.to_capa()) for f in freeze.features.file],
|
||||
functions={
|
||||
f.address.to_capa(): null.FunctionFeatures(
|
||||
features=[(fe.address.to_capa(), fe.feature.to_capa()) for fe in f.features],
|
||||
basic_blocks={
|
||||
bb.address.to_capa(): null.BasicBlockFeatures(
|
||||
features=[(fe.address.to_capa(), fe.feature.to_capa()) for fe in bb.features],
|
||||
instructions={
|
||||
i.address.to_capa(): null.InstructionFeatures(
|
||||
features=[(fe.address.to_capa(), fe.feature.to_capa()) for fe in i.features]
|
||||
)
|
||||
for i in bb.instructions
|
||||
},
|
||||
)
|
||||
for bb in f.basic_blocks
|
||||
},
|
||||
)
|
||||
for f in freeze.features.functions
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
MAGIC = "capa0000".encode("ascii")
|
||||
|
||||
|
||||
def dump(extractor: capa.features.extractors.base_extractor.FeatureExtractor) -> bytes:
|
||||
"""serialize the given extractor to a byte array."""
|
||||
return MAGIC + zlib.compress(dumps(extractor).encode("utf-8"))
|
||||
|
||||
|
||||
def is_freeze(buf: bytes) -> bool:
|
||||
return buf[: len(MAGIC)] == MAGIC
|
||||
|
||||
|
||||
def load(buf: bytes) -> capa.features.extractors.base_extractor.FeatureExtractor:
|
||||
"""deserialize a set of features (as a NullFeatureExtractor) from a byte array."""
|
||||
if not is_freeze(buf):
|
||||
raise ValueError("missing magic header")
|
||||
return loads(zlib.decompress(buf[len(MAGIC) :]).decode("utf-8"))
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
import sys
|
||||
import argparse
|
||||
|
||||
import capa.main
|
||||
|
||||
if argv is None:
|
||||
argv = sys.argv[1:]
|
||||
|
||||
parser = argparse.ArgumentParser(description="save capa features to a file")
|
||||
capa.main.install_common_args(parser, {"sample", "format", "backend", "signatures"})
|
||||
parser.add_argument("output", type=str, help="Path to output file")
|
||||
args = parser.parse_args(args=argv)
|
||||
capa.main.handle_common_args(args)
|
||||
|
||||
sigpaths = capa.main.get_signatures(args.signatures)
|
||||
|
||||
extractor = capa.main.get_extractor(args.sample, args.format, args.backend, sigpaths, False)
|
||||
|
||||
with open(args.output, "wb") as f:
|
||||
f.write(dump(extractor))
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
sys.exit(main())
|
||||
@@ -1,332 +0,0 @@
|
||||
import binascii
|
||||
from typing import Union, Optional
|
||||
|
||||
from pydantic import Field, BaseModel
|
||||
|
||||
import capa.features.file
|
||||
import capa.features.insn
|
||||
import capa.features.common
|
||||
import capa.features.basicblock
|
||||
|
||||
|
||||
class FeatureModel(BaseModel):
|
||||
class Config:
|
||||
frozen = True
|
||||
allow_population_by_field_name = True
|
||||
|
||||
def to_capa(self) -> capa.features.common.Feature:
|
||||
if isinstance(self, OSFeature):
|
||||
return capa.features.common.OS(self.os, description=self.description)
|
||||
|
||||
elif isinstance(self, ArchFeature):
|
||||
return capa.features.common.Arch(self.arch, description=self.description)
|
||||
|
||||
elif isinstance(self, FormatFeature):
|
||||
return capa.features.common.Format(self.format, description=self.description)
|
||||
|
||||
elif isinstance(self, MatchFeature):
|
||||
return capa.features.common.MatchedRule(self.match, description=self.description)
|
||||
|
||||
elif isinstance(
|
||||
self,
|
||||
CharacteristicFeature,
|
||||
):
|
||||
return capa.features.common.Characteristic(self.characteristic, description=self.description)
|
||||
|
||||
elif isinstance(self, ExportFeature):
|
||||
return capa.features.file.Export(self.export, description=self.description)
|
||||
|
||||
elif isinstance(self, ImportFeature):
|
||||
return capa.features.file.Import(self.import_, description=self.description)
|
||||
|
||||
elif isinstance(self, SectionFeature):
|
||||
return capa.features.file.Section(self.section, description=self.description)
|
||||
|
||||
elif isinstance(self, FunctionNameFeature):
|
||||
return capa.features.file.FunctionName(self.function_name, description=self.description)
|
||||
|
||||
elif isinstance(self, SubstringFeature):
|
||||
return capa.features.common.Substring(self.substring, description=self.description)
|
||||
|
||||
elif isinstance(self, RegexFeature):
|
||||
return capa.features.common.Regex(self.regex, description=self.description)
|
||||
|
||||
elif isinstance(self, StringFeature):
|
||||
return capa.features.common.String(self.string, description=self.description)
|
||||
|
||||
elif isinstance(self, ClassFeature):
|
||||
return capa.features.common.Class(self.class_, description=self.description)
|
||||
|
||||
elif isinstance(self, NamespaceFeature):
|
||||
return capa.features.common.Namespace(self.namespace, description=self.description)
|
||||
|
||||
elif isinstance(self, BasicBlockFeature):
|
||||
return capa.features.basicblock.BasicBlock(description=self.description)
|
||||
|
||||
elif isinstance(self, APIFeature):
|
||||
return capa.features.insn.API(self.api, description=self.description)
|
||||
|
||||
elif isinstance(self, NumberFeature):
|
||||
return capa.features.insn.Number(self.number, description=self.description)
|
||||
|
||||
elif isinstance(self, BytesFeature):
|
||||
return capa.features.common.Bytes(binascii.unhexlify(self.bytes), description=self.description)
|
||||
|
||||
elif isinstance(self, OffsetFeature):
|
||||
return capa.features.insn.Offset(self.offset, description=self.description)
|
||||
|
||||
elif isinstance(self, MnemonicFeature):
|
||||
return capa.features.insn.Mnemonic(self.mnemonic, description=self.description)
|
||||
|
||||
elif isinstance(self, OperandNumberFeature):
|
||||
return capa.features.insn.OperandNumber(
|
||||
self.index,
|
||||
self.operand_number,
|
||||
description=self.description,
|
||||
)
|
||||
|
||||
elif isinstance(self, OperandOffsetFeature):
|
||||
return capa.features.insn.OperandOffset(
|
||||
self.index,
|
||||
self.operand_offset,
|
||||
description=self.description,
|
||||
)
|
||||
|
||||
else:
|
||||
raise NotImplementedError(f"Feature.to_capa({type(self)}) not implemented")
|
||||
|
||||
|
||||
def feature_from_capa(f: capa.features.common.Feature) -> "Feature":
|
||||
if isinstance(f, capa.features.common.OS):
|
||||
return OSFeature(os=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.Arch):
|
||||
return ArchFeature(arch=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.Format):
|
||||
return FormatFeature(format=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.MatchedRule):
|
||||
return MatchFeature(match=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.Characteristic):
|
||||
return CharacteristicFeature(characteristic=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.file.Export):
|
||||
return ExportFeature(export=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.file.Import):
|
||||
return ImportFeature(import_=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.file.Section):
|
||||
return SectionFeature(section=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.file.FunctionName):
|
||||
return FunctionNameFeature(function_name=f.value, description=f.description)
|
||||
|
||||
# must come before check for String due to inheritance
|
||||
elif isinstance(f, capa.features.common.Substring):
|
||||
return SubstringFeature(substring=f.value, description=f.description)
|
||||
|
||||
# must come before check for String due to inheritance
|
||||
elif isinstance(f, capa.features.common.Regex):
|
||||
return RegexFeature(regex=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.String):
|
||||
return StringFeature(string=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.Class):
|
||||
return ClassFeature(class_=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.Namespace):
|
||||
return NamespaceFeature(namespace=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.basicblock.BasicBlock):
|
||||
return BasicBlockFeature(description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.insn.API):
|
||||
return APIFeature(api=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.insn.Number):
|
||||
return NumberFeature(number=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.common.Bytes):
|
||||
buf = f.value
|
||||
assert isinstance(buf, bytes)
|
||||
return BytesFeature(bytes=binascii.hexlify(buf).decode("ascii"), description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.insn.Offset):
|
||||
return OffsetFeature(offset=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.insn.Mnemonic):
|
||||
return MnemonicFeature(mnemonic=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.insn.OperandNumber):
|
||||
return OperandNumberFeature(index=f.index, operand_number=f.value, description=f.description)
|
||||
|
||||
elif isinstance(f, capa.features.insn.OperandOffset):
|
||||
return OperandOffsetFeature(index=f.index, operand_offset=f.value, description=f.description)
|
||||
|
||||
else:
|
||||
raise NotImplementedError(f"feature_from_capa({type(f)}) not implemented")
|
||||
|
||||
|
||||
class OSFeature(FeatureModel):
|
||||
type: str = "os"
|
||||
os: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class ArchFeature(FeatureModel):
|
||||
type: str = "arch"
|
||||
arch: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class FormatFeature(FeatureModel):
|
||||
type: str = "format"
|
||||
format: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class MatchFeature(FeatureModel):
|
||||
type: str = "match"
|
||||
match: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class CharacteristicFeature(FeatureModel):
|
||||
type: str = "characteristic"
|
||||
characteristic: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class ExportFeature(FeatureModel):
|
||||
type: str = "export"
|
||||
export: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class ImportFeature(FeatureModel):
|
||||
type: str = "import"
|
||||
import_: str = Field(alias="import")
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class SectionFeature(FeatureModel):
|
||||
type: str = "section"
|
||||
section: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class FunctionNameFeature(FeatureModel):
|
||||
type: str = "function name"
|
||||
function_name: str = Field(alias="function name")
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class SubstringFeature(FeatureModel):
|
||||
type: str = "substring"
|
||||
substring: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class RegexFeature(FeatureModel):
|
||||
type: str = "regex"
|
||||
regex: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class StringFeature(FeatureModel):
|
||||
type: str = "string"
|
||||
string: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class ClassFeature(FeatureModel):
|
||||
type: str = "class"
|
||||
class_: str = Field(alias="class")
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class NamespaceFeature(FeatureModel):
|
||||
type: str = "namespace"
|
||||
namespace: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class BasicBlockFeature(FeatureModel):
|
||||
type: str = "basic block"
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class APIFeature(FeatureModel):
|
||||
type: str = "api"
|
||||
api: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class NumberFeature(FeatureModel):
|
||||
type: str = "number"
|
||||
number: Union[int, float]
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class BytesFeature(FeatureModel):
|
||||
type: str = "bytes"
|
||||
bytes: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class OffsetFeature(FeatureModel):
|
||||
type: str = "offset"
|
||||
offset: int
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class MnemonicFeature(FeatureModel):
|
||||
type: str = "mnemonic"
|
||||
mnemonic: str
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class OperandNumberFeature(FeatureModel):
|
||||
type: str = "operand number"
|
||||
index: int
|
||||
operand_number: int = Field(alias="operand number")
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class OperandOffsetFeature(FeatureModel):
|
||||
type: str = "operand offset"
|
||||
index: int
|
||||
operand_offset: int = Field(alias="operand offset")
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
Feature = Union[
|
||||
OSFeature,
|
||||
ArchFeature,
|
||||
FormatFeature,
|
||||
MatchFeature,
|
||||
CharacteristicFeature,
|
||||
ExportFeature,
|
||||
ImportFeature,
|
||||
SectionFeature,
|
||||
FunctionNameFeature,
|
||||
SubstringFeature,
|
||||
RegexFeature,
|
||||
StringFeature,
|
||||
ClassFeature,
|
||||
NamespaceFeature,
|
||||
APIFeature,
|
||||
NumberFeature,
|
||||
BytesFeature,
|
||||
OffsetFeature,
|
||||
MnemonicFeature,
|
||||
OperandNumberFeature,
|
||||
OperandOffsetFeature,
|
||||
# this has to go last because...? pydantic fails to serialize correctly otherwise.
|
||||
# possibly because this feature has no associated value?
|
||||
BasicBlockFeature,
|
||||
]
|
||||
@@ -1,103 +1,40 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
import abc
|
||||
from typing import Union
|
||||
|
||||
from capa.features.common import Feature
|
||||
|
||||
|
||||
def hex(n: int) -> str:
|
||||
"""render the given number using upper case hex, like: 0x123ABC"""
|
||||
if n < 0:
|
||||
return "-0x%X" % (-n)
|
||||
else:
|
||||
return "0x%X" % n
|
||||
from capa.features import Feature
|
||||
|
||||
|
||||
class API(Feature):
|
||||
def __init__(self, name: str, description=None):
|
||||
def __init__(self, name, description=None):
|
||||
# Downcase library name if given
|
||||
if "." in name:
|
||||
modname, _, impname = name.rpartition(".")
|
||||
name = modname.lower() + "." + impname
|
||||
|
||||
super(API, self).__init__(name, description=description)
|
||||
|
||||
|
||||
class Number(Feature):
|
||||
def __init__(self, value: Union[int, float], description=None):
|
||||
super(Number, self).__init__(value, description=description)
|
||||
def __init__(self, value, arch=None, description=None):
|
||||
super(Number, self).__init__(value, arch=arch, description=description)
|
||||
|
||||
def get_value_str(self):
|
||||
if isinstance(self.value, int):
|
||||
return hex(self.value)
|
||||
elif isinstance(self.value, float):
|
||||
return str(self.value)
|
||||
else:
|
||||
raise ValueError("invalid value type")
|
||||
|
||||
|
||||
# max recognized structure size (and therefore, offset size)
|
||||
MAX_STRUCTURE_SIZE = 0x10000
|
||||
return "0x%X" % self.value
|
||||
|
||||
|
||||
class Offset(Feature):
|
||||
def __init__(self, value: int, description=None):
|
||||
super(Offset, self).__init__(value, description=description)
|
||||
def __init__(self, value, arch=None, description=None):
|
||||
super(Offset, self).__init__(value, arch=arch, description=description)
|
||||
|
||||
def get_value_str(self):
|
||||
return hex(self.value)
|
||||
return "0x%X" % self.value
|
||||
|
||||
|
||||
class Mnemonic(Feature):
|
||||
def __init__(self, value: str, description=None):
|
||||
def __init__(self, value, description=None):
|
||||
super(Mnemonic, self).__init__(value, description=description)
|
||||
|
||||
|
||||
# max number of operands to consider for a given instrucion.
|
||||
# since we only support Intel and .NET, we can assume this is 3
|
||||
# which covers cases up to e.g. "vinserti128 ymm0,ymm0,ymm5,1"
|
||||
MAX_OPERAND_COUNT = 4
|
||||
MAX_OPERAND_INDEX = MAX_OPERAND_COUNT - 1
|
||||
|
||||
|
||||
class _Operand(Feature, abc.ABC):
|
||||
# superclass: don't use directly
|
||||
# subclasses should set self.name and provide the value string formatter
|
||||
def __init__(self, index: int, value: int, description=None):
|
||||
super(_Operand, self).__init__(value, description=description)
|
||||
self.index = index
|
||||
|
||||
def __hash__(self):
|
||||
return hash((self.name, self.value))
|
||||
|
||||
def __eq__(self, other):
|
||||
return super().__eq__(other) and self.index == other.index
|
||||
|
||||
|
||||
class OperandNumber(_Operand):
|
||||
# cached names so we don't do extra string formatting every ctor
|
||||
NAMES = ["operand[%d].number" % i for i in range(MAX_OPERAND_COUNT)]
|
||||
|
||||
# operand[i].number: 0x12
|
||||
def __init__(self, index: int, value: int, description=None):
|
||||
super(OperandNumber, self).__init__(index, value, description=description)
|
||||
self.name = self.NAMES[index]
|
||||
|
||||
def get_value_str(self) -> str:
|
||||
assert isinstance(self.value, int)
|
||||
return hex(self.value)
|
||||
|
||||
|
||||
class OperandOffset(_Operand):
|
||||
# cached names so we don't do extra string formatting every ctor
|
||||
NAMES = ["operand[%d].offset" % i for i in range(MAX_OPERAND_COUNT)]
|
||||
|
||||
# operand[i].offset: 0x12
|
||||
def __init__(self, index: int, value: int, description=None):
|
||||
super(OperandOffset, self).__init__(index, value, description=description)
|
||||
self.name = self.NAMES[index]
|
||||
|
||||
def get_value_str(self) -> str:
|
||||
assert isinstance(self.value, int)
|
||||
return hex(self.value)
|
||||
|
||||
116
capa/helpers.py
116
capa/helpers.py
@@ -1,118 +1,36 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import os
|
||||
import logging
|
||||
from typing import NoReturn
|
||||
|
||||
from capa.exceptions import UnsupportedFormatError
|
||||
from capa.features.common import FORMAT_SC32, FORMAT_SC64, FORMAT_UNKNOWN
|
||||
|
||||
EXTENSIONS_SHELLCODE_32 = ("sc32", "raw32")
|
||||
EXTENSIONS_SHELLCODE_64 = ("sc64", "raw64")
|
||||
EXTENSIONS_ELF = "elf_"
|
||||
|
||||
logger = logging.getLogger("capa")
|
||||
|
||||
_hex = hex
|
||||
|
||||
|
||||
def hex(i):
|
||||
return _hex(int(i))
|
||||
# under py2.7, long integers get formatted with a trailing `L`
|
||||
# and this is not pretty. so strip it out.
|
||||
return _hex(oint(i)).rstrip("L")
|
||||
|
||||
|
||||
def get_file_taste(sample_path: str) -> bytes:
|
||||
def oint(i):
|
||||
# there seems to be some trouble with using `int(viv_utils.Function)`
|
||||
# with the black magic we do with binding the `__int__()` routine.
|
||||
# i haven't had a chance to debug this yet (and i have no hotel wifi).
|
||||
# so in the meantime, detect this, and call the method directly.
|
||||
try:
|
||||
return int(i)
|
||||
except TypeError:
|
||||
return i.__int__()
|
||||
|
||||
|
||||
def get_file_taste(sample_path):
|
||||
if not os.path.exists(sample_path):
|
||||
raise IOError("sample path %s does not exist or cannot be accessed" % sample_path)
|
||||
with open(sample_path, "rb") as f:
|
||||
taste = f.read(8)
|
||||
return taste
|
||||
|
||||
|
||||
def is_runtime_ida():
|
||||
try:
|
||||
import idc
|
||||
except ImportError:
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
|
||||
|
||||
def assert_never(value: NoReturn) -> NoReturn:
|
||||
assert False, f"Unhandled value: {value} ({type(value).__name__})"
|
||||
|
||||
|
||||
def get_format_from_extension(sample: str) -> str:
|
||||
if sample.endswith(EXTENSIONS_SHELLCODE_32):
|
||||
return FORMAT_SC32
|
||||
elif sample.endswith(EXTENSIONS_SHELLCODE_64):
|
||||
return FORMAT_SC64
|
||||
return FORMAT_UNKNOWN
|
||||
|
||||
|
||||
def get_auto_format(path: str) -> str:
|
||||
format_ = get_format(path)
|
||||
if format_ == FORMAT_UNKNOWN:
|
||||
format_ = get_format_from_extension(path)
|
||||
if format_ == FORMAT_UNKNOWN:
|
||||
raise UnsupportedFormatError()
|
||||
return format_
|
||||
|
||||
|
||||
def get_format(sample: str) -> str:
|
||||
# imported locally to avoid import cycle
|
||||
from capa.features.extractors.common import extract_format
|
||||
|
||||
with open(sample, "rb") as f:
|
||||
buf = f.read()
|
||||
|
||||
for feature, _ in extract_format(buf):
|
||||
assert isinstance(feature.value, str)
|
||||
return feature.value
|
||||
|
||||
return FORMAT_UNKNOWN
|
||||
|
||||
|
||||
def log_unsupported_format_error():
|
||||
logger.error("-" * 80)
|
||||
logger.error(" Input file does not appear to be a PE or ELF file.")
|
||||
logger.error(" ")
|
||||
logger.error(
|
||||
" capa currently only supports analyzing PE and ELF files (or shellcode, when using --format sc32|sc64)."
|
||||
)
|
||||
logger.error(" If you don't know the input file type, you can try using the `file` utility to guess it.")
|
||||
logger.error("-" * 80)
|
||||
|
||||
|
||||
def log_unsupported_os_error():
|
||||
logger.error("-" * 80)
|
||||
logger.error(" Input file does not appear to target a supported OS.")
|
||||
logger.error(" ")
|
||||
logger.error(
|
||||
" capa currently only supports analyzing executables for some operating systems (including Windows and Linux)."
|
||||
)
|
||||
logger.error("-" * 80)
|
||||
|
||||
|
||||
def log_unsupported_arch_error():
|
||||
logger.error("-" * 80)
|
||||
logger.error(" Input file does not appear to target a supported architecture.")
|
||||
logger.error(" ")
|
||||
logger.error(" capa currently only supports analyzing x86 (32- and 64-bit).")
|
||||
logger.error("-" * 80)
|
||||
|
||||
|
||||
def log_unsupported_runtime_error():
|
||||
logger.error("-" * 80)
|
||||
logger.error(" Unsupported runtime or Python interpreter.")
|
||||
logger.error(" ")
|
||||
logger.error(" capa supports running under Python 3.7 and higher.")
|
||||
logger.error(" ")
|
||||
logger.error(
|
||||
" If you're seeing this message on the command line, please ensure you're running a supported Python version."
|
||||
)
|
||||
logger.error("-" * 80)
|
||||
|
||||
@@ -1,190 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import logging
|
||||
import datetime
|
||||
import contextlib
|
||||
|
||||
import idc
|
||||
import idaapi
|
||||
import idautils
|
||||
import ida_bytes
|
||||
import ida_loader
|
||||
|
||||
import capa
|
||||
import capa.version
|
||||
import capa.features.common
|
||||
|
||||
logger = logging.getLogger("capa")
|
||||
|
||||
# file type as returned by idainfo.file_type
|
||||
SUPPORTED_FILE_TYPES = (
|
||||
idaapi.f_PE,
|
||||
idaapi.f_ELF,
|
||||
idaapi.f_BIN,
|
||||
# idaapi.f_MACHO,
|
||||
)
|
||||
|
||||
# arch type as returned by idainfo.procname
|
||||
SUPPORTED_ARCH_TYPES = ("metapc",)
|
||||
|
||||
|
||||
def inform_user_ida_ui(message):
|
||||
idaapi.info("%s. Please refer to IDA Output window for more information." % message)
|
||||
|
||||
|
||||
def is_supported_ida_version():
|
||||
version = float(idaapi.get_kernel_version())
|
||||
if version < 7.4 or version >= 9:
|
||||
warning_msg = "This plugin does not support your IDA Pro version"
|
||||
logger.warning(warning_msg)
|
||||
logger.warning("Your IDA Pro version is: %s. Supported versions are: IDA >= 7.4 and IDA < 9.0." % version)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def is_supported_file_type():
|
||||
file_info = idaapi.get_inf_structure()
|
||||
if file_info.filetype not in SUPPORTED_FILE_TYPES:
|
||||
logger.error("-" * 80)
|
||||
logger.error(" Input file does not appear to be a supported file type.")
|
||||
logger.error(" ")
|
||||
logger.error(
|
||||
" capa currently only supports analyzing PE, ELF, or binary files containing x86 (32- and 64-bit) shellcode."
|
||||
)
|
||||
logger.error(" If you don't know the input file type, you can try using the `file` utility to guess it.")
|
||||
logger.error("-" * 80)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def is_supported_arch_type():
|
||||
file_info = idaapi.get_inf_structure()
|
||||
if file_info.procname not in SUPPORTED_ARCH_TYPES or not any((file_info.is_32bit(), file_info.is_64bit())):
|
||||
logger.error("-" * 80)
|
||||
logger.error(" Input file does not appear to target a supported architecture.")
|
||||
logger.error(" ")
|
||||
logger.error(" capa currently only supports analyzing x86 (32- and 64-bit).")
|
||||
logger.error("-" * 80)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def get_disasm_line(va):
|
||||
""" """
|
||||
return idc.generate_disasm_line(va, idc.GENDSM_FORCE_CODE)
|
||||
|
||||
|
||||
def is_func_start(ea):
|
||||
"""check if function stat exists at virtual address"""
|
||||
f = idaapi.get_func(ea)
|
||||
return f and f.start_ea == ea
|
||||
|
||||
|
||||
def get_func_start_ea(ea):
|
||||
""" """
|
||||
f = idaapi.get_func(ea)
|
||||
return f if f is None else f.start_ea
|
||||
|
||||
|
||||
def get_file_md5():
|
||||
""" """
|
||||
md5 = idautils.GetInputFileMD5()
|
||||
if not isinstance(md5, str):
|
||||
md5 = capa.features.common.bytes_to_str(md5)
|
||||
return md5
|
||||
|
||||
|
||||
def get_file_sha256():
|
||||
""" """
|
||||
sha256 = idaapi.retrieve_input_file_sha256()
|
||||
if not isinstance(sha256, str):
|
||||
sha256 = capa.features.common.bytes_to_str(sha256)
|
||||
return sha256
|
||||
|
||||
|
||||
def collect_metadata(rules):
|
||||
""" """
|
||||
md5 = get_file_md5()
|
||||
sha256 = get_file_sha256()
|
||||
|
||||
info: idaapi.idainfo = idaapi.get_inf_structure()
|
||||
if info.procname == "metapc" and info.is_64bit():
|
||||
arch = "x86_64"
|
||||
elif info.procname == "metapc" and info.is_32bit():
|
||||
arch = "x86"
|
||||
else:
|
||||
arch = "unknown arch"
|
||||
|
||||
format_name: str = ida_loader.get_file_type_name()
|
||||
if "PE" in format_name:
|
||||
os = "windows"
|
||||
elif "ELF" in format_name:
|
||||
with contextlib.closing(capa.ida.helpers.IDAIO()) as f:
|
||||
os = capa.features.extractors.elf.detect_elf_os(f)
|
||||
else:
|
||||
os = "unknown os"
|
||||
|
||||
return {
|
||||
"timestamp": datetime.datetime.now().isoformat(),
|
||||
"argv": [],
|
||||
"sample": {
|
||||
"md5": md5,
|
||||
"sha1": "", # not easily accessible
|
||||
"sha256": sha256,
|
||||
"path": idaapi.get_input_file_path(),
|
||||
},
|
||||
"analysis": {
|
||||
"format": idaapi.get_file_type_name(),
|
||||
"arch": arch,
|
||||
"os": os,
|
||||
"extractor": "ida",
|
||||
"rules": rules,
|
||||
"base_address": idaapi.get_imagebase(),
|
||||
"layout": {
|
||||
# this is updated after capabilities have been collected.
|
||||
# will look like:
|
||||
#
|
||||
# "functions": { 0x401000: { "matched_basic_blocks": [ 0x401000, 0x401005, ... ] }, ... }
|
||||
},
|
||||
# ignore these for now - not used by IDA plugin.
|
||||
"feature_counts": {
|
||||
"file": {},
|
||||
"functions": {},
|
||||
},
|
||||
"library_functions": {},
|
||||
},
|
||||
"version": capa.version.__version__,
|
||||
}
|
||||
|
||||
|
||||
class IDAIO:
|
||||
"""
|
||||
An object that acts as a file-like object,
|
||||
using bytes from the current IDB workspace.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
super(IDAIO, self).__init__()
|
||||
self.offset = 0
|
||||
|
||||
def seek(self, offset, whence=0):
|
||||
assert whence == 0
|
||||
self.offset = offset
|
||||
|
||||
def read(self, size):
|
||||
ea = ida_loader.get_fileregion_ea(self.offset)
|
||||
if ea == idc.BADADDR:
|
||||
# best guess, such as if file is mapped at address 0x0.
|
||||
ea = self.offset
|
||||
|
||||
logger.debug("reading 0x%x bytes at 0x%x (ea: 0x%x)", size, self.offset, ea)
|
||||
return ida_bytes.get_bytes(ea, size)
|
||||
|
||||
def close(self):
|
||||
return
|
||||
122
capa/ida/helpers/__init__.py
Normal file
122
capa/ida/helpers/__init__.py
Normal file
@@ -0,0 +1,122 @@
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import logging
|
||||
import datetime
|
||||
|
||||
import idc
|
||||
import six
|
||||
import idaapi
|
||||
import idautils
|
||||
|
||||
import capa
|
||||
|
||||
logger = logging.getLogger("capa")
|
||||
|
||||
SUPPORTED_IDA_VERSIONS = [
|
||||
"7.1",
|
||||
"7.2",
|
||||
"7.3",
|
||||
"7.4",
|
||||
"7.5",
|
||||
"7.6",
|
||||
]
|
||||
|
||||
# file type names as returned by idaapi.get_file_type_name()
|
||||
SUPPORTED_FILE_TYPES = [
|
||||
"Portable executable for 80386 (PE)",
|
||||
"Portable executable for AMD64 (PE)",
|
||||
"Binary file", # x86/AMD64 shellcode support
|
||||
]
|
||||
|
||||
|
||||
def inform_user_ida_ui(message):
|
||||
idaapi.info("%s. Please refer to IDA Output window for more information." % message)
|
||||
|
||||
|
||||
def is_supported_ida_version():
|
||||
version = idaapi.get_kernel_version()
|
||||
if version not in SUPPORTED_IDA_VERSIONS:
|
||||
warning_msg = "This plugin does not support your IDA Pro version"
|
||||
logger.warning(warning_msg)
|
||||
logger.warning(
|
||||
"Your IDA Pro version is: %s. Supported versions are: %s." % (version, ", ".join(SUPPORTED_IDA_VERSIONS))
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def is_supported_file_type():
|
||||
file_type = idaapi.get_file_type_name()
|
||||
if file_type not in SUPPORTED_FILE_TYPES:
|
||||
logger.error("-" * 80)
|
||||
logger.error(" Input file does not appear to be a PE file.")
|
||||
logger.error(" ")
|
||||
logger.error(
|
||||
" capa currently only supports analyzing PE files (or binary files containing x86/AMD64 shellcode) with IDA."
|
||||
)
|
||||
logger.error(" If you don't know the input file type, you can try using the `file` utility to guess it.")
|
||||
logger.error("-" * 80)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def get_disasm_line(va):
|
||||
""" """
|
||||
return idc.generate_disasm_line(va, idc.GENDSM_FORCE_CODE)
|
||||
|
||||
|
||||
def is_func_start(ea):
|
||||
""" check if function stat exists at virtual address """
|
||||
f = idaapi.get_func(ea)
|
||||
return f and f.start_ea == ea
|
||||
|
||||
|
||||
def get_func_start_ea(ea):
|
||||
""" """
|
||||
f = idaapi.get_func(ea)
|
||||
return f if f is None else f.start_ea
|
||||
|
||||
|
||||
def get_file_md5():
|
||||
""" """
|
||||
md5 = idautils.GetInputFileMD5()
|
||||
if not isinstance(md5, six.string_types):
|
||||
md5 = capa.features.bytes_to_str(md5)
|
||||
return md5
|
||||
|
||||
|
||||
def get_file_sha256():
|
||||
""" """
|
||||
sha256 = idaapi.retrieve_input_file_sha256()
|
||||
if not isinstance(sha256, six.string_types):
|
||||
sha256 = capa.features.bytes_to_str(sha256)
|
||||
return sha256
|
||||
|
||||
|
||||
def collect_metadata():
|
||||
""" """
|
||||
md5 = get_file_md5()
|
||||
sha256 = get_file_sha256()
|
||||
|
||||
return {
|
||||
"timestamp": datetime.datetime.now().isoformat(),
|
||||
# "argv" is not relevant here
|
||||
"sample": {
|
||||
"md5": md5,
|
||||
"sha1": "", # not easily accessible
|
||||
"sha256": sha256,
|
||||
"path": idaapi.get_input_file_path(),
|
||||
},
|
||||
"analysis": {
|
||||
"format": idaapi.get_file_type_name(),
|
||||
"extractor": "ida",
|
||||
"base_address": idaapi.get_imagebase(),
|
||||
},
|
||||
"version": capa.version.__version__,
|
||||
}
|
||||
@@ -28,33 +28,19 @@ to modify the rule text directly and the `Editor` pane to construct and rearrang
|
||||
|
||||
|
||||
|
||||
For more information on the FLARE team's open-source framework, capa, check out the overview in our first [blog](https://www.mandiant.com/resources/capa-automatically-identify-malware-capabilities).
|
||||
For more information on the FLARE team's open-source framework, capa, check out the overview in our first [blog](https://www.fireeye.com/blog/threat-research/2020/07/capa-automatically-identify-malware-capabilities.html).
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Requirements
|
||||
|
||||
capa explorer supports Python versions >= 3.7.x and the following IDA Pro versions:
|
||||
capa explorer supports Python 2.7 and 3.6+ and the following IDA Pro versions:
|
||||
|
||||
* IDA 7.4
|
||||
* IDA 7.5
|
||||
* IDA 7.6 (caveat below)
|
||||
* IDA 7.7
|
||||
|
||||
capa explorer is however limited to the Python versions supported by your IDA installation (which may not include all Python versions >= 3.7.x). Based on our testing the following matrix shows the Python versions supported
|
||||
by each supported IDA version:
|
||||
|
||||
| | IDA 7.4 | IDA 7.5 | IDA 7.6 |
|
||||
| --- | --- | --- | --- |
|
||||
| Python 3.7.x | Yes | Yes | Yes |
|
||||
| Python 3.8.x | Partial (see below) | Yes | Yes |
|
||||
| Python 3.9.x | No | Partial (see below) | Yes |
|
||||
|
||||
To use capa explorer with IDA 7.4 and Python 3.8.x you must follow the instructions provided by hex-rays [here](https://hex-rays.com/blog/ida-7-4-and-python-3-8/).
|
||||
|
||||
To use capa explorer with IDA 7.5 and Python 3.9.x you must follow the instructions provided by hex-rays [here](https://hex-rays.com/blog/python-3-9-support-for-ida-7-5/).
|
||||
|
||||
If you encounter issues with your specific setup, please open a new [Issue](https://github.com/mandiant/capa/issues).
|
||||
If you encounter issues with your specific setup, please open a new [Issue](https://github.com/fireeye/capa/issues).
|
||||
|
||||
#### IDA 7.6 caveat: IDA 7.6sp1 or patch required
|
||||
|
||||
@@ -75,8 +61,8 @@ Therefore, in order to use capa under IDA 7.6 you need the [Service Pack 1 for I
|
||||
|
||||
capa explorer is limited to the file types supported by capa, which include:
|
||||
|
||||
* Windows x86 (32- and 64-bit) PE and ELF files
|
||||
* Windows x86 (32- and 64-bit) shellcode
|
||||
* Windows 32-bit and 64-bit PE files
|
||||
* Windows 32-bit and 64-bit shellcode
|
||||
|
||||
### Installation
|
||||
|
||||
@@ -86,20 +72,19 @@ You can install capa explorer using the following steps:
|
||||
```
|
||||
$ pip install flare-capa
|
||||
```
|
||||
3. Download the [standard collection of capa rules](https://github.com/mandiant/capa-rules) (capa explorer needs capa rules to analyze a database)
|
||||
4. Copy [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory
|
||||
3. Download the [standard collection of capa rules](https://github.com/fireeye/capa-rules) (capa explorer needs capa rules to analyze a database)
|
||||
4. Copy [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py) to your IDA plugins directory
|
||||
|
||||
### Usage
|
||||
|
||||
1. Open IDA and analyze a supported file type (select the `Manual Load` and `Load Resources` options in IDA for best results)
|
||||
2. Open capa explorer in IDA by navigating to `Edit > Plugins > FLARE capa explorer` or using the keyboard shortcut `Alt+F5`
|
||||
You can also use `ida_loader.load_and_run_plugin("capa_explorer", arg)`. `arg` is a bitflag for which setting the LSB enables automatic analysis. See `capa.ida.plugin.form.Options` for more details.
|
||||
3. Select the `Program Analysis` tab
|
||||
4. Click the `Analyze` button
|
||||
|
||||
When running capa explorer for the first time you are prompted to select a file directory containing capa rules. The plugin conveniently
|
||||
remembers your selection for future runs; you can change this selection and other default settings by clicking `Settings`. We recommend
|
||||
downloading and using the [standard collection of capa rules](https://github.com/mandiant/capa-rules) when getting started with the plugin.
|
||||
downloading and using the [standard collection of capa rules](https://github.com/fireeye/capa-rules) when getting started with the plugin.
|
||||
|
||||
#### Tips for Program Analysis
|
||||
|
||||
@@ -125,15 +110,15 @@ downloading and using the [standard collection of capa rules](https://github.com
|
||||
## Development
|
||||
|
||||
capa explorer is packaged with capa so you will need to install capa locally for development. You can install capa locally by following the steps outlined in `Method 3: Inspecting the capa source code` of the [capa
|
||||
installation guide](https://github.com/mandiant/capa/blob/master/doc/installation.md#method-3-inspecting-the-capa-source-code). Once installed, copy [capa_explorer.py](https://raw.githubusercontent.com/mandiant/capa/master/capa/ida/plugin/capa_explorer.py)
|
||||
installation guide](https://github.com/fireeye/capa/blob/master/doc/installation.md#method-3-inspecting-the-capa-source-code). Once installed, copy [capa_explorer.py](https://raw.githubusercontent.com/fireeye/capa/master/capa/ida/plugin/capa_explorer.py)
|
||||
to your plugins directory to install capa explorer in IDA.
|
||||
|
||||
### Components
|
||||
|
||||
capa explorer consists of two main components:
|
||||
|
||||
* An [feature extractor](https://github.com/mandiant/capa/tree/master/capa/features/extractors/ida) built on top of IDA's binary analysis engine
|
||||
* This component uses IDAPython to extract [capa features](https://github.com/mandiant/capa-rules/blob/master/doc/format.md#extracted-features) from your IDBs such as strings,
|
||||
* An [feature extractor](https://github.com/fireeye/capa/tree/master/capa/features/extractors/ida) built on top of IDA's binary analysis engine
|
||||
* This component uses IDAPython to extract [capa features](https://github.com/fireeye/capa-rules/blob/master/doc/format.md#extracted-features) from your IDBs such as strings,
|
||||
disassembly, and control flow; these extracted features are used by capa to find feature combinations that result in a rule match
|
||||
* An [interactive user interface](https://github.com/mandiant/capa/tree/master/capa/ida/plugin) for displaying and exploring capa rule matches
|
||||
* An [interactive user interface](https://github.com/fireeye/capa/tree/master/capa/ida/plugin) for displaying and exploring capa rule matches
|
||||
* This component integrates the feature extractor and capa, providing an interactive user interface to dissect rule matches found by capa using features extracted directly from your IDBs
|
||||
|
||||
@@ -1,15 +1,17 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import logging
|
||||
|
||||
import idaapi
|
||||
import ida_kernwin
|
||||
|
||||
from capa.ida.helpers import is_supported_file_type, is_supported_ida_version
|
||||
from capa.ida.plugin.form import CapaExplorerForm
|
||||
from capa.ida.plugin.icon import ICON
|
||||
|
||||
@@ -26,8 +28,8 @@ class CapaExplorerPlugin(idaapi.plugin_t):
|
||||
wanted_name = PLUGIN_NAME
|
||||
wanted_hotkey = "ALT-F5"
|
||||
comment = "IDA Pro plugin for the FLARE team's capa tool to identify capabilities in executable files."
|
||||
website = "https://github.com/mandiant/capa"
|
||||
help = "See https://github.com/mandiant/capa/blob/master/doc/usage.md"
|
||||
website = "https://github.com/fireeye/capa"
|
||||
help = "See https://github.com/fireeye/capa/blob/master/doc/usage.md"
|
||||
version = ""
|
||||
flags = 0
|
||||
|
||||
@@ -39,14 +41,10 @@ class CapaExplorerPlugin(idaapi.plugin_t):
|
||||
"""called when IDA is loading the plugin"""
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
|
||||
import capa.ida.helpers
|
||||
|
||||
# do not load plugin if IDA version/file type not supported
|
||||
if not capa.ida.helpers.is_supported_ida_version():
|
||||
if not is_supported_ida_version():
|
||||
return idaapi.PLUGIN_SKIP
|
||||
if not capa.ida.helpers.is_supported_file_type():
|
||||
return idaapi.PLUGIN_SKIP
|
||||
if not capa.ida.helpers.is_supported_arch_type():
|
||||
if not is_supported_file_type():
|
||||
return idaapi.PLUGIN_SKIP
|
||||
return idaapi.PLUGIN_OK
|
||||
|
||||
@@ -55,14 +53,8 @@ class CapaExplorerPlugin(idaapi.plugin_t):
|
||||
pass
|
||||
|
||||
def run(self, arg):
|
||||
"""
|
||||
called when IDA is running the plugin as a script
|
||||
|
||||
args:
|
||||
arg (int): bitflag. Setting LSB enables automatic analysis upon
|
||||
loading. The other bits are currently undefined. See `form.Options`.
|
||||
"""
|
||||
self.form = CapaExplorerForm(self.PLUGIN_NAME, arg)
|
||||
"""called when IDA is running the plugin as a script"""
|
||||
self.form = CapaExplorerForm(self.PLUGIN_NAME)
|
||||
return True
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -8,10 +8,10 @@
|
||||
|
||||
import os
|
||||
import copy
|
||||
import json
|
||||
import logging
|
||||
import itertools
|
||||
import collections
|
||||
from typing import Set, Dict, Optional
|
||||
|
||||
import idaapi
|
||||
import ida_kernwin
|
||||
@@ -20,26 +20,19 @@ from PyQt5 import QtGui, QtCore, QtWidgets
|
||||
|
||||
import capa.main
|
||||
import capa.rules
|
||||
import capa.engine
|
||||
import capa.ida.helpers
|
||||
import capa.render.json
|
||||
import capa.features.common
|
||||
import capa.render.result_document
|
||||
import capa.features.extractors.ida.extractor
|
||||
from capa.engine import FeatureSet
|
||||
from capa.features.common import Feature
|
||||
import capa.render.utils as rutils
|
||||
import capa.features.extractors.ida
|
||||
from capa.ida.plugin.icon import QICON
|
||||
from capa.ida.plugin.view import (
|
||||
CapaExplorerQtreeView,
|
||||
CapaExplorerRulegenEditor,
|
||||
CapaExplorerRulegenPreview,
|
||||
CapaExplorerRulgenEditor,
|
||||
CapaExplorerRulgenPreview,
|
||||
CapaExplorerRulegenFeatures,
|
||||
)
|
||||
from capa.features.address import NO_ADDRESS, Address
|
||||
from capa.ida.plugin.hooks import CapaExplorerIdaHooks
|
||||
from capa.ida.plugin.model import CapaExplorerDataModel
|
||||
from capa.ida.plugin.proxy import CapaExplorerRangeProxyModel, CapaExplorerSearchProxyModel
|
||||
from capa.features.extractors.base_extractor import FunctionHandle
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
settings = ida_settings.IDASettings("capa")
|
||||
@@ -48,16 +41,11 @@ CAPA_SETTINGS_RULE_PATH = "rule_path"
|
||||
CAPA_SETTINGS_RULEGEN_AUTHOR = "rulegen_author"
|
||||
CAPA_SETTINGS_RULEGEN_SCOPE = "rulegen_scope"
|
||||
|
||||
from enum import IntFlag
|
||||
|
||||
|
||||
class Options(IntFlag):
|
||||
DEFAULT = 0
|
||||
ANALYZE = 1 # Runs the analysis when starting the explorer
|
||||
|
||||
|
||||
def write_file(path, data):
|
||||
""" """
|
||||
if os.path.exists(path) and 1 != idaapi.ask_yn(1, "The file already exists. Overwrite?"):
|
||||
return
|
||||
with open(path, "wb") as save_file:
|
||||
save_file.write(data)
|
||||
|
||||
@@ -70,32 +58,32 @@ def trim_function_name(f, max_length=25):
|
||||
return n
|
||||
|
||||
|
||||
def find_func_features(fh: FunctionHandle, extractor):
|
||||
def find_func_features(f, extractor):
|
||||
""" """
|
||||
func_features: Dict[Feature, Set] = collections.defaultdict(set)
|
||||
bb_features: Dict[Address, Dict] = collections.defaultdict(dict)
|
||||
func_features = collections.defaultdict(set)
|
||||
bb_features = collections.defaultdict(dict)
|
||||
|
||||
for (feature, addr) in extractor.extract_function_features(fh):
|
||||
func_features[feature].add(addr)
|
||||
for (feature, ea) in extractor.extract_function_features(f):
|
||||
func_features[feature].add(ea)
|
||||
|
||||
for bbh in extractor.get_basic_blocks(fh):
|
||||
for bb in extractor.get_basic_blocks(f):
|
||||
_bb_features = collections.defaultdict(set)
|
||||
|
||||
for (feature, addr) in extractor.extract_basic_block_features(fh, bbh):
|
||||
_bb_features[feature].add(addr)
|
||||
func_features[feature].add(addr)
|
||||
for (feature, ea) in extractor.extract_basic_block_features(f, bb):
|
||||
_bb_features[feature].add(ea)
|
||||
func_features[feature].add(ea)
|
||||
|
||||
for insn in extractor.get_instructions(fh, bbh):
|
||||
for (feature, addr) in extractor.extract_insn_features(fh, bbh, insn):
|
||||
_bb_features[feature].add(addr)
|
||||
func_features[feature].add(addr)
|
||||
for insn in extractor.get_instructions(f, bb):
|
||||
for (feature, ea) in extractor.extract_insn_features(f, bb, insn):
|
||||
_bb_features[feature].add(ea)
|
||||
func_features[feature].add(ea)
|
||||
|
||||
bb_features[bbh.address] = _bb_features
|
||||
bb_features[capa.helpers.oint(bb)] = _bb_features
|
||||
|
||||
return func_features, bb_features
|
||||
|
||||
|
||||
def find_func_matches(f: FunctionHandle, ruleset, func_features, bb_features):
|
||||
def find_func_matches(f, ruleset, func_features, bb_features):
|
||||
""" """
|
||||
func_matches = collections.defaultdict(list)
|
||||
bb_matches = collections.defaultdict(list)
|
||||
@@ -109,10 +97,10 @@ def find_func_matches(f: FunctionHandle, ruleset, func_features, bb_features):
|
||||
for (name, res) in matches.items():
|
||||
bb_matches[name].extend(res)
|
||||
for (ea, _) in res:
|
||||
func_features[capa.features.common.MatchedRule(name)].add(ea)
|
||||
func_features[capa.features.MatchedRule(name)].add(ea)
|
||||
|
||||
# find rule matches for function, function features include rule matches for basic blocks
|
||||
_, matches = capa.engine.match(ruleset.function_rules, func_features, f.address)
|
||||
_, matches = capa.engine.match(ruleset.function_rules, func_features, capa.helpers.oint(f))
|
||||
for (name, res) in matches.items():
|
||||
func_matches[name].extend(res)
|
||||
|
||||
@@ -121,19 +109,19 @@ def find_func_matches(f: FunctionHandle, ruleset, func_features, bb_features):
|
||||
|
||||
def find_file_features(extractor):
|
||||
""" """
|
||||
file_features = collections.defaultdict(set) # type: FeatureSet
|
||||
for (feature, addr) in extractor.extract_file_features():
|
||||
if addr:
|
||||
file_features[feature].add(addr)
|
||||
file_features = collections.defaultdict(set)
|
||||
for (feature, ea) in extractor.extract_file_features():
|
||||
if ea:
|
||||
file_features[feature].add(ea)
|
||||
else:
|
||||
if feature not in file_features:
|
||||
file_features[feature] = set()
|
||||
return file_features
|
||||
|
||||
|
||||
def find_file_matches(ruleset, file_features: FeatureSet):
|
||||
def find_file_matches(ruleset, file_features):
|
||||
""" """
|
||||
_, matches = capa.engine.match(ruleset.file_rules, file_features, NO_ADDRESS)
|
||||
_, matches = capa.engine.match(ruleset.file_rules, file_features, 0x0)
|
||||
return matches
|
||||
|
||||
|
||||
@@ -167,7 +155,7 @@ class CapaExplorerProgressIndicator(QtCore.QObject):
|
||||
self.progress.emit("extracting features from %s" % text)
|
||||
|
||||
|
||||
class CapaExplorerFeatureExtractor(capa.features.extractors.ida.extractor.IdaFeatureExtractor):
|
||||
class CapaExplorerFeatureExtractor(capa.features.extractors.ida.IdaFeatureExtractor):
|
||||
"""subclass the IdaFeatureExtractor
|
||||
|
||||
track progress during feature extraction, also allow user to cancel feature extraction
|
||||
@@ -177,9 +165,9 @@ class CapaExplorerFeatureExtractor(capa.features.extractors.ida.extractor.IdaFea
|
||||
super(CapaExplorerFeatureExtractor, self).__init__()
|
||||
self.indicator = CapaExplorerProgressIndicator()
|
||||
|
||||
def extract_function_features(self, fh: FunctionHandle):
|
||||
self.indicator.update("function at 0x%X" % fh.inner.start_ea)
|
||||
return super(CapaExplorerFeatureExtractor, self).extract_function_features(fh)
|
||||
def extract_function_features(self, f):
|
||||
self.indicator.update("function at 0x%X" % f.start_ea)
|
||||
return super(CapaExplorerFeatureExtractor, self).extract_function_features(f)
|
||||
|
||||
|
||||
class QLineEditClicked(QtWidgets.QLineEdit):
|
||||
@@ -239,7 +227,7 @@ class CapaSettingsInputDialog(QtWidgets.QDialog):
|
||||
class CapaExplorerForm(idaapi.PluginForm):
|
||||
"""form element for plugin interface"""
|
||||
|
||||
def __init__(self, name, option=Options.DEFAULT):
|
||||
def __init__(self, name):
|
||||
"""initialize form elements"""
|
||||
super(CapaExplorerForm, self).__init__()
|
||||
|
||||
@@ -249,9 +237,8 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
self.parent = None
|
||||
self.ida_hooks = None
|
||||
self.doc: Optional[capa.render.result_document.ResultDocument] = None
|
||||
self.doc = None
|
||||
|
||||
self.rule_paths = None
|
||||
self.rules_cache = None
|
||||
self.ruleset_cache = None
|
||||
|
||||
@@ -280,7 +267,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
self.view_rulegen_editor = None
|
||||
self.view_rulegen_header_label = None
|
||||
self.view_rulegen_search = None
|
||||
self.view_rulegen_limit_features_by_ea = None
|
||||
self.rulegen_current_function = None
|
||||
self.rulegen_bb_features_cache = {}
|
||||
self.rulegen_func_features_cache = {}
|
||||
@@ -289,9 +275,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
self.Show()
|
||||
|
||||
if (option & Options.ANALYZE) == Options.ANALYZE:
|
||||
self.analyze_program()
|
||||
|
||||
def OnCreate(self, form):
|
||||
"""called when plugin form is created
|
||||
|
||||
@@ -471,10 +454,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
label2.setText("Editor")
|
||||
label2.setFont(font)
|
||||
|
||||
self.view_rulegen_limit_features_by_ea = QtWidgets.QCheckBox("Limit features to current dissasembly address")
|
||||
self.view_rulegen_limit_features_by_ea.setChecked(False)
|
||||
self.view_rulegen_limit_features_by_ea.stateChanged.connect(self.slot_checkbox_limit_features_by_ea)
|
||||
|
||||
self.view_rulegen_status_label = QtWidgets.QLabel()
|
||||
self.view_rulegen_status_label.setAlignment(QtCore.Qt.AlignLeft)
|
||||
self.view_rulegen_status_label.setText("")
|
||||
@@ -489,8 +468,8 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
self.view_rulegen_header_label.setText("Features")
|
||||
self.view_rulegen_header_label.setFont(font)
|
||||
|
||||
self.view_rulegen_preview = CapaExplorerRulegenPreview(parent=self.parent)
|
||||
self.view_rulegen_editor = CapaExplorerRulegenEditor(self.view_rulegen_preview, parent=self.parent)
|
||||
self.view_rulegen_preview = CapaExplorerRulgenPreview(parent=self.parent)
|
||||
self.view_rulegen_editor = CapaExplorerRulgenEditor(self.view_rulegen_preview, parent=self.parent)
|
||||
self.view_rulegen_features = CapaExplorerRulegenFeatures(self.view_rulegen_editor, parent=self.parent)
|
||||
|
||||
self.view_rulegen_preview.textChanged.connect(self.slot_rulegen_preview_update)
|
||||
@@ -505,7 +484,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
layout3.addWidget(self.view_rulegen_editor, 65)
|
||||
|
||||
layout2.addWidget(self.view_rulegen_header_label)
|
||||
layout2.addWidget(self.view_rulegen_limit_features_by_ea)
|
||||
layout2.addWidget(self.view_rulegen_search)
|
||||
layout2.addWidget(self.view_rulegen_features)
|
||||
|
||||
@@ -570,10 +548,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
self.limit_results_to_function(idaapi.get_func(ea))
|
||||
self.view_tree.reset_ui()
|
||||
|
||||
def update_rulegen_tree_limit_features_to_selection(self, ea):
|
||||
""" """
|
||||
self.view_rulegen_features.filter_items_by_ea(ea)
|
||||
|
||||
def ida_hook_screen_ea_changed(self, widget, new_ea, old_ea):
|
||||
"""function hook for IDA "screen ea changed" action
|
||||
|
||||
@@ -594,9 +568,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
if not idaapi.get_func(new_ea):
|
||||
return
|
||||
|
||||
if self.view_tabs.currentIndex() == 1 and self.view_rulegen_limit_features_by_ea.isChecked():
|
||||
return self.update_rulegen_tree_limit_features_to_selection(new_ea)
|
||||
|
||||
if idaapi.get_func(new_ea) == idaapi.get_func(old_ea):
|
||||
# user navigated same function - ignore
|
||||
return
|
||||
@@ -622,7 +593,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
def load_capa_rules(self):
|
||||
""" """
|
||||
self.rule_paths = None
|
||||
self.ruleset_cache = None
|
||||
self.rules_cache = None
|
||||
|
||||
@@ -633,7 +603,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
path = self.ask_user_directory()
|
||||
if not path:
|
||||
logger.warning(
|
||||
"You must select a file directory containing capa rules before analysis can be run. The standard collection of capa rules can be downloaded from https://github.com/mandiant/capa-rules."
|
||||
"You must select a file directory containing capa rules before analysis can be run. The standard collection of capa rules can be downloaded from https://github.com/fireeye/capa-rules."
|
||||
)
|
||||
return False
|
||||
settings.user[CAPA_SETTINGS_RULE_PATH] = path
|
||||
@@ -656,12 +626,10 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
rule_paths.append(rule_path)
|
||||
elif os.path.isdir(rule_path):
|
||||
for root, dirs, files in os.walk(rule_path):
|
||||
if ".git" in root:
|
||||
if ".github" in root:
|
||||
# the .github directory contains CI config in capa-rules
|
||||
# this includes some .yml files
|
||||
# these are not rules
|
||||
# additionally, .git has files that are not .yml and generate the warning
|
||||
# skip those too
|
||||
continue
|
||||
for file in files:
|
||||
if not file.endswith(".yml"):
|
||||
@@ -702,24 +670,11 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
)
|
||||
logger.error("Failed to load rules from %s (error: %s).", settings.user[CAPA_SETTINGS_RULE_PATH], e)
|
||||
logger.error(
|
||||
"Make sure your file directory contains properly formatted capa rules. You can download the standard collection of capa rules from https://github.com/mandiant/capa-rules."
|
||||
)
|
||||
logger.error(
|
||||
"Please ensure you're using the rules that correspond to your major version of capa (%s)",
|
||||
capa.version.get_major_version(),
|
||||
)
|
||||
logger.error(
|
||||
"You can check out these rules with the following command:\n %s",
|
||||
capa.version.get_rules_checkout_command(),
|
||||
)
|
||||
logger.error(
|
||||
"Or, for more details, see the rule set documentation here: %s",
|
||||
"https://github.com/mandiant/capa/blob/master/doc/rules.md",
|
||||
"Make sure your file directory contains properly formatted capa rules. You can download the standard collection of capa rules from https://github.com/fireeye/capa-rules."
|
||||
)
|
||||
settings.user[CAPA_SETTINGS_RULE_PATH] = ""
|
||||
return False
|
||||
|
||||
self.rule_paths = rule_paths
|
||||
self.ruleset_cache = ruleset
|
||||
self.rules_cache = rules
|
||||
|
||||
@@ -769,10 +724,9 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
update_wait_box("extracting features")
|
||||
|
||||
try:
|
||||
meta = capa.ida.helpers.collect_metadata(self.rule_paths)
|
||||
meta = capa.ida.helpers.collect_metadata()
|
||||
capabilities, counts = capa.main.find_capabilities(self.ruleset_cache, extractor, disable_progress=True)
|
||||
meta["analysis"].update(counts)
|
||||
meta["analysis"]["layout"] = capa.main.compute_layout(self.ruleset_cache, extractor, capabilities)
|
||||
except UserCancelledError:
|
||||
logger.info("User cancelled analysis.")
|
||||
return False
|
||||
@@ -816,9 +770,9 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
update_wait_box("rendering results")
|
||||
|
||||
try:
|
||||
self.doc = capa.render.result_document.ResultDocument.from_capa(meta, self.ruleset_cache, capabilities)
|
||||
self.doc = capa.render.convert_capabilities_to_result_document(meta, self.ruleset_cache, capabilities)
|
||||
except Exception as e:
|
||||
logger.error("Failed to collect results (error: %s)", e, exc_info=True)
|
||||
logger.error("Failed to render results (error: %s)", e)
|
||||
return False
|
||||
|
||||
try:
|
||||
@@ -827,7 +781,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
"capa rules directory: %s (%d rules)" % (settings.user[CAPA_SETTINGS_RULE_PATH], len(self.rules_cache))
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("Failed to render results (error: %s)", e, exc_info=True)
|
||||
logger.error("Failed to render results (error: %s)", e)
|
||||
return False
|
||||
|
||||
return True
|
||||
@@ -880,7 +834,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
# must use extractor to get function, as capa analysis requires casted object
|
||||
extractor = CapaExplorerFeatureExtractor()
|
||||
except Exception as e:
|
||||
logger.error("Failed to load IDA feature extractor (error: %s)", e)
|
||||
logger.error("Failed to load IDA feature extractor (error: %s)" % e)
|
||||
return False
|
||||
|
||||
if ida_kernwin.user_cancelled():
|
||||
@@ -891,10 +845,10 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
try:
|
||||
f = idaapi.get_func(idaapi.get_screen_ea())
|
||||
if f:
|
||||
fh: FunctionHandle = extractor.get_function(f.start_ea)
|
||||
self.rulegen_current_function = fh
|
||||
f = extractor.get_function(f.start_ea)
|
||||
self.rulegen_current_function = f
|
||||
|
||||
func_features, bb_features = find_func_features(fh, extractor)
|
||||
func_features, bb_features = find_func_features(f, extractor)
|
||||
self.rulegen_func_features_cache = collections.defaultdict(set, copy.copy(func_features))
|
||||
self.rulegen_bb_features_cache = collections.defaultdict(dict, copy.copy(bb_features))
|
||||
|
||||
@@ -905,15 +859,15 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
try:
|
||||
# add function and bb rule matches to function features, for display purposes
|
||||
func_matches, bb_matches = find_func_matches(fh, self.ruleset_cache, func_features, bb_features)
|
||||
for (name, addrs) in itertools.chain(func_matches.items(), bb_matches.items()):
|
||||
func_matches, bb_matches = find_func_matches(f, self.ruleset_cache, func_features, bb_features)
|
||||
for (name, res) in itertools.chain(func_matches.items(), bb_matches.items()):
|
||||
rule = self.ruleset_cache[name]
|
||||
if rule.is_subscope_rule():
|
||||
if rule.meta.get("capa/subscope-rule"):
|
||||
continue
|
||||
for (addr, _) in addrs:
|
||||
func_features[capa.features.common.MatchedRule(name)].add(addr)
|
||||
for (ea, _) in res:
|
||||
func_features[capa.features.MatchedRule(name)].add(ea)
|
||||
except Exception as e:
|
||||
logger.error("Failed to match function/basic block rule scope (error: %s)", e)
|
||||
logger.error("Failed to match function/basic block rule scope (error: %s)" % e)
|
||||
return False
|
||||
else:
|
||||
func_features = {}
|
||||
@@ -921,7 +875,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
logger.info("User cancelled analysis.")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error("Failed to extract function features (error: %s)", e)
|
||||
logger.error("Failed to extract function features (error: %s)" % e)
|
||||
return False
|
||||
|
||||
if ida_kernwin.user_cancelled():
|
||||
@@ -931,7 +885,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
try:
|
||||
file_features = find_file_features(extractor)
|
||||
self.rulegen_file_features_cache = copy.copy(file_features)
|
||||
self.rulegen_file_features_cache = collections.defaultdict(dict, copy.copy(file_features))
|
||||
|
||||
if ida_kernwin.user_cancelled():
|
||||
logger.info("User cancelled analysis.")
|
||||
@@ -940,17 +894,17 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
try:
|
||||
# add file matches to file features, for display purposes
|
||||
for (name, addrs) in find_file_matches(self.ruleset_cache, file_features).items():
|
||||
for (name, res) in find_file_matches(self.ruleset_cache, file_features).items():
|
||||
rule = self.ruleset_cache[name]
|
||||
if rule.is_subscope_rule():
|
||||
if rule.meta.get("capa/subscope-rule"):
|
||||
continue
|
||||
for (addr, _) in addrs:
|
||||
file_features[capa.features.common.MatchedRule(name)].add(addr)
|
||||
for (ea, _) in res:
|
||||
file_features[capa.features.MatchedRule(name)].add(ea)
|
||||
except Exception as e:
|
||||
logger.error("Failed to match file scope rules (error: %s)", e)
|
||||
logger.error("Failed to match file scope rules (error: %s)" % e)
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error("Failed to extract file features (error: %s)", e)
|
||||
logger.error("Failed to extract file features (error: %s)" % e)
|
||||
return False
|
||||
|
||||
if ida_kernwin.user_cancelled():
|
||||
@@ -961,7 +915,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
try:
|
||||
# load preview and feature tree
|
||||
self.view_rulegen_preview.load_preview_meta(
|
||||
fh.address if fh else None,
|
||||
f.start_ea if f else None,
|
||||
settings.user.get(CAPA_SETTINGS_RULEGEN_AUTHOR, "<insert_author>"),
|
||||
settings.user.get(CAPA_SETTINGS_RULEGEN_SCOPE, "function"),
|
||||
)
|
||||
@@ -972,7 +926,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
"capa rules directory: %s (%d rules)" % (settings.user[CAPA_SETTINGS_RULE_PATH], len(self.rules_cache))
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error("Failed to render views (error: %s)", e, exc_info=True)
|
||||
logger.error("Failed to render views (error: %s)" % e)
|
||||
return False
|
||||
|
||||
return True
|
||||
@@ -1013,7 +967,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
self.view_rulegen_editor.reset_view()
|
||||
self.view_rulegen_preview.reset_view()
|
||||
self.view_rulegen_search.clear()
|
||||
self.view_rulegen_limit_features_by_ea.setChecked(False)
|
||||
self.set_rulegen_preview_border_neutral()
|
||||
self.rulegen_current_function = None
|
||||
self.rulegen_func_features_cache = {}
|
||||
@@ -1052,7 +1005,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
def update_rule_status(self, rule_text):
|
||||
""" """
|
||||
if not self.view_rulegen_editor.invisibleRootItem().childCount():
|
||||
if self.view_rulegen_editor.root is None:
|
||||
self.set_rulegen_preview_border_neutral()
|
||||
self.view_rulegen_status_label.clear()
|
||||
return
|
||||
@@ -1170,9 +1123,9 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
idaapi.info("No program analysis to save.")
|
||||
return
|
||||
|
||||
s = self.doc.json().encode("utf-8")
|
||||
s = json.dumps(self.doc, sort_keys=True, cls=capa.render.CapaJsonObjectEncoder).encode("utf-8")
|
||||
|
||||
path = self.ask_user_capa_json_file()
|
||||
path = idaapi.ask_file(True, "*.json", "Choose file to save capa program analysis JSON")
|
||||
if not path:
|
||||
return
|
||||
|
||||
@@ -1205,13 +1158,6 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
|
||||
self.view_tree.reset_ui()
|
||||
|
||||
def slot_checkbox_limit_features_by_ea(self, state):
|
||||
""" """
|
||||
if state == QtCore.Qt.Checked:
|
||||
self.view_rulegen_features.filter_items_by_ea(idaapi.get_screen_ea())
|
||||
else:
|
||||
self.view_rulegen_features.show_all_items()
|
||||
|
||||
def slot_checkbox_show_results_by_function_changed(self, state):
|
||||
"""slot activated if checkbox clicked
|
||||
|
||||
@@ -1255,16 +1201,7 @@ class CapaExplorerForm(idaapi.PluginForm):
|
||||
def ask_user_capa_rule_file(self):
|
||||
""" """
|
||||
return QtWidgets.QFileDialog.getSaveFileName(
|
||||
None,
|
||||
"Please select a location to save capa rule file",
|
||||
settings.user.get(CAPA_SETTINGS_RULE_PATH, ""),
|
||||
"*.yml",
|
||||
)[0]
|
||||
|
||||
def ask_user_capa_json_file(self):
|
||||
""" """
|
||||
return QtWidgets.QFileDialog.getSaveFileName(
|
||||
None, "Please select a location to save capa JSON file", "", "*.json"
|
||||
None, "Please select a capa rule to edit", settings.user.get(CAPA_SETTINGS_RULE_PATH, ""), "*.yml"
|
||||
)[0]
|
||||
|
||||
def set_view_status_label(self, text):
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,15 +6,14 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import sys
|
||||
import codecs
|
||||
from typing import List, Iterator, Optional
|
||||
|
||||
import idc
|
||||
import idaapi
|
||||
from PyQt5 import QtCore
|
||||
|
||||
import capa.ida.helpers
|
||||
from capa.features.address import Address, FileOffsetAddress, AbsoluteVirtualAddress
|
||||
|
||||
|
||||
def info_to_name(display):
|
||||
@@ -28,19 +27,19 @@ def info_to_name(display):
|
||||
return ""
|
||||
|
||||
|
||||
def ea_to_hex(ea):
|
||||
"""convert effective address (ea) to hex for display"""
|
||||
return "%08X" % ea
|
||||
def location_to_hex(location):
|
||||
"""convert location to hex for display"""
|
||||
return "%08X" % location
|
||||
|
||||
|
||||
class CapaExplorerDataItem:
|
||||
class CapaExplorerDataItem(object):
|
||||
"""store data for CapaExplorerDataModel"""
|
||||
|
||||
def __init__(self, parent: "CapaExplorerDataItem", data: List[str], can_check=True):
|
||||
def __init__(self, parent, data, can_check=True):
|
||||
"""initialize item"""
|
||||
self.pred = parent
|
||||
self._data = data
|
||||
self._children: List["CapaExplorerDataItem"] = []
|
||||
self.children = []
|
||||
self._checked = False
|
||||
self._can_check = can_check
|
||||
|
||||
@@ -78,29 +77,29 @@ class CapaExplorerDataItem:
|
||||
"""get item is checked"""
|
||||
return self._checked
|
||||
|
||||
def appendChild(self, item: "CapaExplorerDataItem"):
|
||||
def appendChild(self, item):
|
||||
"""add a new child to specified item
|
||||
|
||||
@param item: CapaExplorerDataItem
|
||||
"""
|
||||
self._children.append(item)
|
||||
self.children.append(item)
|
||||
|
||||
def child(self, row: int) -> "CapaExplorerDataItem":
|
||||
def child(self, row):
|
||||
"""get child row
|
||||
|
||||
@param row: row number
|
||||
"""
|
||||
return self._children[row]
|
||||
return self.children[row]
|
||||
|
||||
def childCount(self) -> int:
|
||||
def childCount(self):
|
||||
"""get child count"""
|
||||
return len(self._children)
|
||||
return len(self.children)
|
||||
|
||||
def columnCount(self) -> int:
|
||||
def columnCount(self):
|
||||
"""get column count"""
|
||||
return len(self._data)
|
||||
|
||||
def data(self, column: int) -> Optional[str]:
|
||||
def data(self, column):
|
||||
"""get data at column
|
||||
|
||||
@param: column number
|
||||
@@ -110,17 +109,17 @@ class CapaExplorerDataItem:
|
||||
except IndexError:
|
||||
return None
|
||||
|
||||
def parent(self) -> "CapaExplorerDataItem":
|
||||
def parent(self):
|
||||
"""get parent"""
|
||||
return self.pred
|
||||
|
||||
def row(self) -> int:
|
||||
def row(self):
|
||||
"""get row location"""
|
||||
if self.pred:
|
||||
return self.pred._children.index(self)
|
||||
return self.pred.children.index(self)
|
||||
return 0
|
||||
|
||||
def setData(self, column: int, value: str):
|
||||
def setData(self, column, value):
|
||||
"""set data in column
|
||||
|
||||
@param column: column number
|
||||
@@ -128,14 +127,14 @@ class CapaExplorerDataItem:
|
||||
"""
|
||||
self._data[column] = value
|
||||
|
||||
def children(self) -> Iterator["CapaExplorerDataItem"]:
|
||||
def children(self):
|
||||
"""yield children"""
|
||||
for child in self._children:
|
||||
for child in self.children:
|
||||
yield child
|
||||
|
||||
def removeChildren(self):
|
||||
"""remove children"""
|
||||
del self._children[:]
|
||||
del self.children[:]
|
||||
|
||||
def __str__(self):
|
||||
"""get string representation of columns
|
||||
@@ -150,7 +149,7 @@ class CapaExplorerDataItem:
|
||||
return self._data[0]
|
||||
|
||||
@property
|
||||
def location(self) -> Optional[int]:
|
||||
def location(self):
|
||||
"""return data stored in location column"""
|
||||
try:
|
||||
# address stored as str, convert to int before return
|
||||
@@ -169,9 +168,7 @@ class CapaExplorerRuleItem(CapaExplorerDataItem):
|
||||
|
||||
fmt = "%s (%d matches)"
|
||||
|
||||
def __init__(
|
||||
self, parent: CapaExplorerDataItem, name: str, namespace: str, count: int, source: str, can_check=True
|
||||
):
|
||||
def __init__(self, parent, name, namespace, count, source, can_check=True):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@@ -193,7 +190,7 @@ class CapaExplorerRuleItem(CapaExplorerDataItem):
|
||||
class CapaExplorerRuleMatchItem(CapaExplorerDataItem):
|
||||
"""store data for rule match"""
|
||||
|
||||
def __init__(self, parent: CapaExplorerDataItem, display: str, source=""):
|
||||
def __init__(self, parent, display, source=""):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@@ -205,7 +202,7 @@ class CapaExplorerRuleMatchItem(CapaExplorerDataItem):
|
||||
|
||||
@property
|
||||
def source(self):
|
||||
"""return rule contents for display"""
|
||||
""" return rule contents for display """
|
||||
return self._source
|
||||
|
||||
|
||||
@@ -214,16 +211,14 @@ class CapaExplorerFunctionItem(CapaExplorerDataItem):
|
||||
|
||||
fmt = "function(%s)"
|
||||
|
||||
def __init__(self, parent: CapaExplorerDataItem, location: Address, can_check=True):
|
||||
def __init__(self, parent, location, can_check=True):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@param location: virtual address of function as seen by IDA
|
||||
"""
|
||||
assert isinstance(location, AbsoluteVirtualAddress)
|
||||
ea = int(location)
|
||||
super(CapaExplorerFunctionItem, self).__init__(
|
||||
parent, [self.fmt % idaapi.get_name(ea), ea_to_hex(ea), ""], can_check
|
||||
parent, [self.fmt % idaapi.get_name(location), location_to_hex(location), ""], can_check
|
||||
)
|
||||
|
||||
@property
|
||||
@@ -249,7 +244,7 @@ class CapaExplorerSubscopeItem(CapaExplorerDataItem):
|
||||
|
||||
fmt = "subscope(%s)"
|
||||
|
||||
def __init__(self, parent: CapaExplorerDataItem, scope):
|
||||
def __init__(self, parent, scope):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@@ -263,23 +258,19 @@ class CapaExplorerBlockItem(CapaExplorerDataItem):
|
||||
|
||||
fmt = "basic block(loc_%08X)"
|
||||
|
||||
def __init__(self, parent: CapaExplorerDataItem, location: Address):
|
||||
def __init__(self, parent, location):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@param location: virtual address of basic block as seen by IDA
|
||||
"""
|
||||
assert isinstance(location, AbsoluteVirtualAddress)
|
||||
ea = int(location)
|
||||
super(CapaExplorerBlockItem, self).__init__(parent, [self.fmt % ea, ea_to_hex(ea), ""])
|
||||
super(CapaExplorerBlockItem, self).__init__(parent, [self.fmt % location, location_to_hex(location), ""])
|
||||
|
||||
|
||||
class CapaExplorerDefaultItem(CapaExplorerDataItem):
|
||||
"""store data for default match e.g. statement (and, or)"""
|
||||
|
||||
def __init__(
|
||||
self, parent: CapaExplorerDataItem, display: str, details: str = "", location: Optional[Address] = None
|
||||
):
|
||||
def __init__(self, parent, display, details="", location=None):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@@ -287,22 +278,14 @@ class CapaExplorerDefaultItem(CapaExplorerDataItem):
|
||||
@param details: text to display in details section of UI
|
||||
@param location: virtual address as seen by IDA
|
||||
"""
|
||||
ea = None
|
||||
if location:
|
||||
assert isinstance(location, AbsoluteVirtualAddress)
|
||||
ea = int(location)
|
||||
|
||||
super(CapaExplorerDefaultItem, self).__init__(
|
||||
parent, [display, ea_to_hex(ea) if ea is not None else "", details]
|
||||
)
|
||||
location = location_to_hex(location) if location else ""
|
||||
super(CapaExplorerDefaultItem, self).__init__(parent, [display, location, details])
|
||||
|
||||
|
||||
class CapaExplorerFeatureItem(CapaExplorerDataItem):
|
||||
"""store data for feature match"""
|
||||
|
||||
def __init__(
|
||||
self, parent: CapaExplorerDataItem, display: str, location: Optional[Address] = None, details: str = ""
|
||||
):
|
||||
def __init__(self, parent, display, location="", details=""):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@@ -310,18 +293,14 @@ class CapaExplorerFeatureItem(CapaExplorerDataItem):
|
||||
@param details: text to display in details section of UI
|
||||
@param location: virtual address as seen by IDA
|
||||
"""
|
||||
if location:
|
||||
assert isinstance(location, (AbsoluteVirtualAddress, FileOffsetAddress))
|
||||
ea = int(location)
|
||||
super(CapaExplorerFeatureItem, self).__init__(parent, [display, ea_to_hex(ea), details])
|
||||
else:
|
||||
super(CapaExplorerFeatureItem, self).__init__(parent, [display, "", details])
|
||||
location = location_to_hex(location) if location else ""
|
||||
super(CapaExplorerFeatureItem, self).__init__(parent, [display, location, details])
|
||||
|
||||
|
||||
class CapaExplorerInstructionViewItem(CapaExplorerFeatureItem):
|
||||
"""store data for instruction match"""
|
||||
|
||||
def __init__(self, parent: CapaExplorerDataItem, display: str, location: Address):
|
||||
def __init__(self, parent, display, location):
|
||||
"""initialize item
|
||||
|
||||
details section shows disassembly view for match
|
||||
@@ -330,17 +309,15 @@ class CapaExplorerInstructionViewItem(CapaExplorerFeatureItem):
|
||||
@param display: text to display in UI
|
||||
@param location: virtual address as seen by IDA
|
||||
"""
|
||||
assert isinstance(location, AbsoluteVirtualAddress)
|
||||
ea = int(location)
|
||||
details = capa.ida.helpers.get_disasm_line(ea)
|
||||
details = capa.ida.helpers.get_disasm_line(location)
|
||||
super(CapaExplorerInstructionViewItem, self).__init__(parent, display, location=location, details=details)
|
||||
self.ida_highlight = idc.get_color(ea, idc.CIC_ITEM)
|
||||
self.ida_highlight = idc.get_color(location, idc.CIC_ITEM)
|
||||
|
||||
|
||||
class CapaExplorerByteViewItem(CapaExplorerFeatureItem):
|
||||
"""store data for byte match"""
|
||||
|
||||
def __init__(self, parent: CapaExplorerDataItem, display: str, location: Address):
|
||||
def __init__(self, parent, display, location):
|
||||
"""initialize item
|
||||
|
||||
details section shows byte preview for match
|
||||
@@ -349,32 +326,30 @@ class CapaExplorerByteViewItem(CapaExplorerFeatureItem):
|
||||
@param display: text to display in UI
|
||||
@param location: virtual address as seen by IDA
|
||||
"""
|
||||
assert isinstance(location, (AbsoluteVirtualAddress, FileOffsetAddress))
|
||||
ea = int(location)
|
||||
byte_snap = idaapi.get_bytes(location, 32)
|
||||
|
||||
byte_snap = idaapi.get_bytes(ea, 32)
|
||||
|
||||
details = ""
|
||||
if byte_snap:
|
||||
byte_snap = codecs.encode(byte_snap, "hex").upper()
|
||||
details = " ".join([byte_snap[i : i + 2].decode() for i in range(0, len(byte_snap), 2)])
|
||||
if sys.version_info >= (3, 0):
|
||||
details = " ".join([byte_snap[i : i + 2].decode() for i in range(0, len(byte_snap), 2)])
|
||||
else:
|
||||
details = " ".join([byte_snap[i : i + 2] for i in range(0, len(byte_snap), 2)])
|
||||
else:
|
||||
details = ""
|
||||
|
||||
super(CapaExplorerByteViewItem, self).__init__(parent, display, location=location, details=details)
|
||||
self.ida_highlight = idc.get_color(ea, idc.CIC_ITEM)
|
||||
self.ida_highlight = idc.get_color(location, idc.CIC_ITEM)
|
||||
|
||||
|
||||
class CapaExplorerStringViewItem(CapaExplorerFeatureItem):
|
||||
"""store data for string match"""
|
||||
|
||||
def __init__(self, parent: CapaExplorerDataItem, display: str, location: Address, value: str):
|
||||
def __init__(self, parent, display, location, value):
|
||||
"""initialize item
|
||||
|
||||
@param parent: parent node
|
||||
@param display: text to display in UI
|
||||
@param location: virtual address as seen by IDA
|
||||
"""
|
||||
assert isinstance(location, (AbsoluteVirtualAddress, FileOffsetAddress))
|
||||
ea = int(location)
|
||||
|
||||
super(CapaExplorerStringViewItem, self).__init__(parent, display, location=location, details=value)
|
||||
self.ida_highlight = idc.get_color(ea, idc.CIC_ITEM)
|
||||
self.ida_highlight = idc.get_color(location, idc.CIC_ITEM)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,20 +6,14 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from typing import Set, Dict, List, Tuple
|
||||
from collections import deque
|
||||
from collections import deque, defaultdict
|
||||
|
||||
import idc
|
||||
import idaapi
|
||||
from PyQt5 import QtGui, QtCore
|
||||
|
||||
import capa.rules
|
||||
import capa.ida.helpers
|
||||
import capa.render.utils as rutils
|
||||
import capa.features.common
|
||||
import capa.features.freeze as frz
|
||||
import capa.render.result_document as rd
|
||||
import capa.features.freeze.features as frzf
|
||||
from capa.ida.plugin.item import (
|
||||
CapaExplorerDataItem,
|
||||
CapaExplorerRuleItem,
|
||||
@@ -33,7 +27,6 @@ from capa.ida.plugin.item import (
|
||||
CapaExplorerStringViewItem,
|
||||
CapaExplorerInstructionViewItem,
|
||||
)
|
||||
from capa.features.address import Address, AbsoluteVirtualAddress
|
||||
|
||||
# default highlight color used in IDA window
|
||||
DEFAULT_HIGHLIGHT = 0xE6C700
|
||||
@@ -347,14 +340,7 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
|
||||
|
||||
return item.childCount()
|
||||
|
||||
def render_capa_doc_statement_node(
|
||||
self,
|
||||
parent: CapaExplorerDataItem,
|
||||
match: rd.Match,
|
||||
statement: rd.Statement,
|
||||
locations: List[Address],
|
||||
doc: rd.ResultDocument,
|
||||
):
|
||||
def render_capa_doc_statement_node(self, parent, statement, locations, doc):
|
||||
"""render capa statement read from doc
|
||||
|
||||
@param parent: parent to which new child is assigned
|
||||
@@ -362,143 +348,126 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
|
||||
@param locations: locations of children (applies to range only?)
|
||||
@param doc: result doc
|
||||
"""
|
||||
|
||||
if isinstance(statement, (rd.AndStatement, rd.OrStatement, rd.OptionalStatement)):
|
||||
display = statement.type
|
||||
if statement.description:
|
||||
display += " (%s)" % statement.description
|
||||
if statement["type"] in ("and", "or", "optional"):
|
||||
display = statement["type"]
|
||||
if statement.get("description"):
|
||||
display += " (%s)" % statement["description"]
|
||||
return CapaExplorerDefaultItem(parent, display)
|
||||
elif isinstance(statement, rd.NotStatement):
|
||||
elif statement["type"] == "not":
|
||||
# TODO: do we display 'not'
|
||||
pass
|
||||
elif isinstance(statement, rd.SomeStatement):
|
||||
display = "%d or more" % statement.count
|
||||
if statement.description:
|
||||
display += " (%s)" % statement.description
|
||||
elif statement["type"] == "some":
|
||||
display = "%d or more" % statement["count"]
|
||||
if statement.get("description"):
|
||||
display += " (%s)" % statement["description"]
|
||||
return CapaExplorerDefaultItem(parent, display)
|
||||
elif isinstance(statement, rd.RangeStatement):
|
||||
elif statement["type"] == "range":
|
||||
# `range` is a weird node, its almost a hybrid of statement + feature.
|
||||
# it is a specific feature repeated multiple times.
|
||||
# there's no additional logic in the feature part, just the existence of a feature.
|
||||
# so, we have to inline some of the feature rendering here.
|
||||
display = "count(%s): " % self.capa_doc_feature_to_display(statement.child)
|
||||
display = "count(%s): " % self.capa_doc_feature_to_display(statement["child"])
|
||||
|
||||
if statement.max == statement.min:
|
||||
display += "%d" % (statement.min)
|
||||
elif statement.min == 0:
|
||||
display += "%d or fewer" % (statement.max)
|
||||
elif statement.max == (1 << 64 - 1):
|
||||
display += "%d or more" % (statement.min)
|
||||
if statement["max"] == statement["min"]:
|
||||
display += "%d" % (statement["min"])
|
||||
elif statement["min"] == 0:
|
||||
display += "%d or fewer" % (statement["max"])
|
||||
elif statement["max"] == (1 << 64 - 1):
|
||||
display += "%d or more" % (statement["min"])
|
||||
else:
|
||||
display += "between %d and %d" % (statement.min, statement.max)
|
||||
display += "between %d and %d" % (statement["min"], statement["max"])
|
||||
|
||||
if statement.description:
|
||||
display += " (%s)" % statement.description
|
||||
if statement.get("description"):
|
||||
display += " (%s)" % statement["description"]
|
||||
|
||||
parent2 = CapaExplorerFeatureItem(parent, display=display)
|
||||
|
||||
for location in locations:
|
||||
# for each location render child node for range statement
|
||||
self.render_capa_doc_feature(parent2, match, statement.child, location, doc)
|
||||
self.render_capa_doc_feature(parent2, statement["child"], location, doc)
|
||||
|
||||
return parent2
|
||||
elif isinstance(statement, rd.SubscopeStatement):
|
||||
display = str(statement.scope)
|
||||
if statement.description:
|
||||
display += " (%s)" % statement.description
|
||||
elif statement["type"] == "subscope":
|
||||
display = statement[statement["type"]]
|
||||
if statement.get("description"):
|
||||
display += " (%s)" % statement["description"]
|
||||
return CapaExplorerSubscopeItem(parent, display)
|
||||
else:
|
||||
raise RuntimeError("unexpected match statement type: " + str(statement))
|
||||
|
||||
def render_capa_doc_match(self, parent: CapaExplorerDataItem, match: rd.Match, doc: rd.ResultDocument):
|
||||
def render_capa_doc_match(self, parent, match, doc):
|
||||
"""render capa match read from doc
|
||||
|
||||
@param parent: parent node to which new child is assigned
|
||||
@param match: match read from doc
|
||||
@param doc: result doc
|
||||
"""
|
||||
if not match.success:
|
||||
if not match["success"]:
|
||||
# TODO: display failed branches at some point? Help with debugging rules?
|
||||
return
|
||||
|
||||
# optional statement with no successful children is empty
|
||||
if isinstance(match.node, rd.StatementNode) and isinstance(match.node.statement, rd.OptionalStatement):
|
||||
if not any(map(lambda m: m.success, match.children)):
|
||||
return
|
||||
if match["node"].get("statement", {}).get("type") == "optional" and not any(
|
||||
map(lambda m: m["success"], match["children"])
|
||||
):
|
||||
return
|
||||
|
||||
if isinstance(match.node, rd.StatementNode):
|
||||
if match["node"]["type"] == "statement":
|
||||
parent2 = self.render_capa_doc_statement_node(
|
||||
parent, match, match.node.statement, [addr.to_capa() for addr in match.locations], doc
|
||||
parent, match["node"]["statement"], match.get("locations", []), doc
|
||||
)
|
||||
elif isinstance(match.node, rd.FeatureNode):
|
||||
elif match["node"]["type"] == "feature":
|
||||
parent2 = self.render_capa_doc_feature_node(
|
||||
parent, match, match.node.feature, [addr.to_capa() for addr in match.locations], doc
|
||||
parent, match["node"]["feature"], match.get("locations", []), doc
|
||||
)
|
||||
else:
|
||||
raise RuntimeError("unexpected node type: " + str(match.node.type))
|
||||
raise RuntimeError("unexpected node type: " + str(match["node"]["type"]))
|
||||
|
||||
for child in match.children:
|
||||
for child in match.get("children", []):
|
||||
self.render_capa_doc_match(parent2, child, doc)
|
||||
|
||||
def render_capa_doc_by_function(self, doc: rd.ResultDocument):
|
||||
def render_capa_doc_by_function(self, doc):
|
||||
""" """
|
||||
matches_by_function: Dict[int, Tuple[CapaExplorerFunctionItem, Set[str]]] = {}
|
||||
matches_by_function = {}
|
||||
for rule in rutils.capability_rules(doc):
|
||||
for location_, _ in rule.matches:
|
||||
location = location_.to_capa()
|
||||
|
||||
if not isinstance(location, AbsoluteVirtualAddress):
|
||||
# only handle matches with a VA
|
||||
continue
|
||||
ea = int(location)
|
||||
|
||||
for ea in rule["matches"].keys():
|
||||
ea = capa.ida.helpers.get_func_start_ea(ea)
|
||||
if ea is None:
|
||||
# file scope, skip rendering in this mode
|
||||
# file scope, skip for rendering in this mode
|
||||
continue
|
||||
if not matches_by_function.get(ea, ()):
|
||||
# new function root
|
||||
matches_by_function[ea] = (
|
||||
CapaExplorerFunctionItem(self.root_node, location, can_check=False),
|
||||
set(),
|
||||
)
|
||||
function_root, match_cache = matches_by_function[ea]
|
||||
if rule.meta.name in match_cache:
|
||||
# rule match already rendered for this function root, skip it
|
||||
continue
|
||||
match_cache.add(rule.meta.name)
|
||||
if None is matches_by_function.get(ea, None):
|
||||
matches_by_function[ea] = CapaExplorerFunctionItem(self.root_node, ea, can_check=False)
|
||||
CapaExplorerRuleItem(
|
||||
function_root,
|
||||
rule.meta.name,
|
||||
rule.meta.namespace or "",
|
||||
len(rule.matches),
|
||||
rule.source,
|
||||
matches_by_function[ea],
|
||||
rule["meta"]["name"],
|
||||
rule["meta"].get("namespace"),
|
||||
len(rule["matches"]),
|
||||
rule["source"],
|
||||
can_check=False,
|
||||
)
|
||||
|
||||
def render_capa_doc_by_program(self, doc: rd.ResultDocument):
|
||||
def render_capa_doc_by_program(self, doc):
|
||||
""" """
|
||||
for rule in rutils.capability_rules(doc):
|
||||
rule_name = rule.meta.name
|
||||
rule_namespace = rule.meta.namespace or ""
|
||||
parent = CapaExplorerRuleItem(self.root_node, rule_name, rule_namespace, len(rule.matches), rule.source)
|
||||
rule_name = rule["meta"]["name"]
|
||||
rule_namespace = rule["meta"].get("namespace")
|
||||
parent = CapaExplorerRuleItem(
|
||||
self.root_node, rule_name, rule_namespace, len(rule["matches"]), rule["source"]
|
||||
)
|
||||
|
||||
for (location_, match) in rule.matches:
|
||||
location = location_.to_capa()
|
||||
|
||||
parent2: CapaExplorerDataItem
|
||||
if rule.meta.scope == capa.rules.FILE_SCOPE:
|
||||
for (location, match) in doc["rules"][rule["meta"]["name"]]["matches"].items():
|
||||
if rule["meta"]["scope"] == capa.rules.FILE_SCOPE:
|
||||
parent2 = parent
|
||||
elif rule.meta.scope == capa.rules.FUNCTION_SCOPE:
|
||||
elif rule["meta"]["scope"] == capa.rules.FUNCTION_SCOPE:
|
||||
parent2 = CapaExplorerFunctionItem(parent, location)
|
||||
elif rule.meta.scope == capa.rules.BASIC_BLOCK_SCOPE:
|
||||
elif rule["meta"]["scope"] == capa.rules.BASIC_BLOCK_SCOPE:
|
||||
parent2 = CapaExplorerBlockItem(parent, location)
|
||||
else:
|
||||
raise RuntimeError("unexpected rule scope: " + str(rule.meta.scope))
|
||||
raise RuntimeError("unexpected rule scope: " + str(rule["meta"]["scope"]))
|
||||
|
||||
self.render_capa_doc_match(parent2, match, doc)
|
||||
|
||||
def render_capa_doc(self, doc: rd.ResultDocument, by_function: bool):
|
||||
def render_capa_doc(self, doc, by_function):
|
||||
"""render capa features specified in doc
|
||||
|
||||
@param doc: capa result doc
|
||||
@@ -514,36 +483,27 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
|
||||
# inform model changes have ended
|
||||
self.endResetModel()
|
||||
|
||||
def capa_doc_feature_to_display(self, feature: frzf.Feature):
|
||||
def capa_doc_feature_to_display(self, feature):
|
||||
"""convert capa doc feature type string to display string for ui
|
||||
|
||||
@param feature: capa feature read from doc
|
||||
"""
|
||||
key = feature.type
|
||||
value = getattr(feature, feature.type)
|
||||
|
||||
key = feature["type"]
|
||||
value = feature[feature["type"]]
|
||||
if value:
|
||||
if isinstance(feature, frzf.StringFeature):
|
||||
value = '"%s"' % capa.features.common.escape_string(value)
|
||||
if feature.description:
|
||||
return "%s(%s = %s)" % (key, value, feature.description)
|
||||
if key == "string":
|
||||
value = '"%s"' % capa.features.escape_string(value)
|
||||
if feature.get("description", ""):
|
||||
return "%s(%s = %s)" % (key, value, feature["description"])
|
||||
else:
|
||||
return "%s(%s)" % (key, value)
|
||||
else:
|
||||
return "%s" % key
|
||||
|
||||
def render_capa_doc_feature_node(
|
||||
self,
|
||||
parent: CapaExplorerDataItem,
|
||||
match: rd.Match,
|
||||
feature: frzf.Feature,
|
||||
locations: List[Address],
|
||||
doc: rd.ResultDocument,
|
||||
):
|
||||
def render_capa_doc_feature_node(self, parent, feature, locations, doc):
|
||||
"""process capa doc feature node
|
||||
|
||||
@param parent: parent node to which child is assigned
|
||||
@param match: match information
|
||||
@param feature: capa doc feature node
|
||||
@param locations: locations identified for feature
|
||||
@param doc: capa doc
|
||||
@@ -554,7 +514,6 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
|
||||
# only one location for feature so no need to nest children
|
||||
parent2 = self.render_capa_doc_feature(
|
||||
parent,
|
||||
match,
|
||||
feature,
|
||||
next(iter(locations)),
|
||||
doc,
|
||||
@@ -565,112 +524,73 @@ class CapaExplorerDataModel(QtCore.QAbstractItemModel):
|
||||
parent2 = CapaExplorerFeatureItem(parent, display)
|
||||
|
||||
for location in sorted(locations):
|
||||
self.render_capa_doc_feature(parent2, match, feature, location, doc)
|
||||
self.render_capa_doc_feature(parent2, feature, location, doc)
|
||||
|
||||
return parent2
|
||||
|
||||
def render_capa_doc_feature(
|
||||
self,
|
||||
parent: CapaExplorerDataItem,
|
||||
match: rd.Match,
|
||||
feature: frzf.Feature,
|
||||
location: Address,
|
||||
doc: rd.ResultDocument,
|
||||
display="-",
|
||||
):
|
||||
def render_capa_doc_feature(self, parent, feature, location, doc, display="-"):
|
||||
"""render capa feature read from doc
|
||||
|
||||
@param parent: parent node to which new child is assigned
|
||||
@param match: match information
|
||||
@param feature: feature read from doc
|
||||
@param doc: capa feature doc
|
||||
@param location: address of feature
|
||||
@param display: text to display in plugin UI
|
||||
"""
|
||||
|
||||
# special handling for characteristic pending type
|
||||
if isinstance(feature, frzf.CharacteristicFeature):
|
||||
characteristic = feature.characteristic
|
||||
if characteristic in ("embedded pe",):
|
||||
if feature["type"] == "characteristic":
|
||||
if feature[feature["type"]] in ("embedded pe",):
|
||||
return CapaExplorerByteViewItem(parent, display, location)
|
||||
|
||||
if characteristic in ("loop", "recursive call", "tight loop"):
|
||||
if feature[feature["type"]] in ("loop", "recursive call", "tight loop"):
|
||||
return CapaExplorerFeatureItem(parent, display=display)
|
||||
|
||||
# default to instruction view for all other characteristics
|
||||
return CapaExplorerInstructionViewItem(parent, display, location)
|
||||
|
||||
elif isinstance(feature, frzf.MatchFeature):
|
||||
if feature["type"] == "match":
|
||||
# display content of rule for all rule matches
|
||||
matched_rule_source = ""
|
||||
return CapaExplorerRuleMatchItem(
|
||||
parent, display, source=doc["rules"].get(feature[feature["type"]], {}).get("source", "")
|
||||
)
|
||||
|
||||
# check if match is a matched rule
|
||||
matched_rule = doc.rules.get(feature.match, None)
|
||||
if matched_rule is not None:
|
||||
matched_rule_source = matched_rule.source
|
||||
if feature["type"] == "regex":
|
||||
return CapaExplorerStringViewItem(
|
||||
parent, display, location, '"%s"' % capa.features.escape_string(feature["match"])
|
||||
)
|
||||
|
||||
return CapaExplorerRuleMatchItem(parent, display, source=matched_rule_source)
|
||||
|
||||
elif isinstance(feature, (frzf.RegexFeature, frzf.SubstringFeature)):
|
||||
for capture, addrs in sorted(match.captures.items()):
|
||||
for addr in addrs:
|
||||
assert isinstance(addr, frz.Address)
|
||||
if location == addr.value:
|
||||
return CapaExplorerStringViewItem(
|
||||
parent, display, location, '"' + capa.features.common.escape_string(capture) + '"'
|
||||
)
|
||||
|
||||
# programming error: the given location should always be found in the regex matches
|
||||
raise ValueError("regex match at location not found")
|
||||
|
||||
elif isinstance(feature, frzf.BasicBlockFeature):
|
||||
if feature["type"] == "basicblock":
|
||||
return CapaExplorerBlockItem(parent, location)
|
||||
|
||||
elif isinstance(
|
||||
feature,
|
||||
(
|
||||
frzf.BytesFeature,
|
||||
frzf.APIFeature,
|
||||
frzf.MnemonicFeature,
|
||||
frzf.NumberFeature,
|
||||
frzf.OffsetFeature,
|
||||
),
|
||||
if feature["type"] in (
|
||||
"bytes",
|
||||
"api",
|
||||
"mnemonic",
|
||||
"number",
|
||||
"offset",
|
||||
"number/x32",
|
||||
"number/x64",
|
||||
"offset/x32",
|
||||
"offset/x64",
|
||||
):
|
||||
# display instruction preview
|
||||
return CapaExplorerInstructionViewItem(parent, display, location)
|
||||
|
||||
elif isinstance(feature, frzf.SectionFeature):
|
||||
if feature["type"] in ("section",):
|
||||
# display byte preview
|
||||
return CapaExplorerByteViewItem(parent, display, location)
|
||||
|
||||
elif isinstance(feature, frzf.StringFeature):
|
||||
if feature["type"] in ("string",):
|
||||
# display string preview
|
||||
return CapaExplorerStringViewItem(
|
||||
parent, display, location, '"%s"' % capa.features.common.escape_string(feature.string)
|
||||
parent, display, location, '"%s"' % capa.features.escape_string(feature[feature["type"]])
|
||||
)
|
||||
|
||||
elif isinstance(
|
||||
feature,
|
||||
(
|
||||
frzf.ImportFeature,
|
||||
frzf.ExportFeature,
|
||||
frzf.FunctionNameFeature,
|
||||
),
|
||||
):
|
||||
if feature["type"] in ("import", "export"):
|
||||
# display no preview
|
||||
return CapaExplorerFeatureItem(parent, location=location, display=display)
|
||||
|
||||
elif isinstance(
|
||||
feature,
|
||||
(
|
||||
frzf.ArchFeature,
|
||||
frzf.OSFeature,
|
||||
frzf.FormatFeature,
|
||||
),
|
||||
):
|
||||
return CapaExplorerFeatureItem(parent, display=display)
|
||||
|
||||
raise RuntimeError("unexpected feature type: " + str(feature.type))
|
||||
raise RuntimeError("unexpected feature type: " + str(feature["type"]))
|
||||
|
||||
def update_function_name(self, old_name, new_name):
|
||||
"""update all instances of old function name with new function name
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
import six
|
||||
from PyQt5 import QtCore
|
||||
from PyQt5.QtCore import Qt
|
||||
|
||||
@@ -207,7 +208,7 @@ class CapaExplorerSearchProxyModel(QtCore.QSortFilterProxyModel):
|
||||
if not data:
|
||||
continue
|
||||
|
||||
if not isinstance(data, str):
|
||||
if not isinstance(data, six.string_types):
|
||||
# sanity check: should already be a string, but double check
|
||||
continue
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -9,16 +9,13 @@ import re
|
||||
from collections import Counter
|
||||
|
||||
import idc
|
||||
import idaapi
|
||||
from PyQt5 import QtGui, QtCore, QtWidgets
|
||||
|
||||
import capa.rules
|
||||
import capa.engine
|
||||
import capa.ida.helpers
|
||||
import capa.features.common
|
||||
import capa.features.basicblock
|
||||
from capa.ida.plugin.item import CapaExplorerFunctionItem
|
||||
from capa.features.address import Address, _NoAddress
|
||||
from capa.ida.plugin.model import CapaExplorerDataModel
|
||||
|
||||
MAX_SECTION_SIZE = 750
|
||||
@@ -28,7 +25,7 @@ COLOR_GREEN_RGB = (79, 121, 66)
|
||||
COLOR_BLUE_RGB = (37, 147, 215)
|
||||
|
||||
|
||||
def calc_indent_from_line(line, prev_level=0):
|
||||
def calc_level_by_indent(line, prev_level=0):
|
||||
""" """
|
||||
if not len(line.strip()):
|
||||
# blank line, which may occur for comments so we simply use the last level
|
||||
@@ -38,13 +35,10 @@ def calc_indent_from_line(line, prev_level=0):
|
||||
# need to adjust two spaces when encountering string description
|
||||
line = line[2:]
|
||||
# calc line level based on preceding whitespace
|
||||
indent = len(line) - len(stripped)
|
||||
|
||||
# round up to nearest even number; helps keep parsing more sane
|
||||
return indent + (indent % 2)
|
||||
return len(line) - len(stripped)
|
||||
|
||||
|
||||
def parse_yaml_line(feature):
|
||||
def parse_feature_for_node(feature):
|
||||
""" """
|
||||
description = ""
|
||||
comment = ""
|
||||
@@ -117,6 +111,30 @@ def parse_node_for_feature(feature, description, comment, depth):
|
||||
return display if display.endswith("\n") else display + "\n"
|
||||
|
||||
|
||||
def yaml_to_nodes(s):
|
||||
level = 0
|
||||
for line in s.splitlines():
|
||||
feature, description, comment = parse_feature_for_node(line.strip())
|
||||
|
||||
o = QtWidgets.QTreeWidgetItem(None)
|
||||
|
||||
# set node attributes
|
||||
setattr(o, "capa_level", calc_level_by_indent(line, level))
|
||||
|
||||
if feature.startswith(("- and:", "- or:", "- not:", "- basic block:", "- optional:")):
|
||||
setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_expression())
|
||||
elif feature.startswith("#"):
|
||||
setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_comment())
|
||||
else:
|
||||
setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_feature())
|
||||
|
||||
# set node text
|
||||
for (i, v) in enumerate((feature, description, comment)):
|
||||
o.setText(i, v)
|
||||
|
||||
yield o
|
||||
|
||||
|
||||
def iterate_tree(o):
|
||||
""" """
|
||||
itr = QtWidgets.QTreeWidgetItemIterator(o)
|
||||
@@ -125,13 +143,6 @@ def iterate_tree(o):
|
||||
itr += 1
|
||||
|
||||
|
||||
def expand_tree(root):
|
||||
""" """
|
||||
for node in iterate_tree(root):
|
||||
if node.childCount() and not node.isExpanded():
|
||||
node.setExpanded(True)
|
||||
|
||||
|
||||
def calc_item_depth(o):
|
||||
""" """
|
||||
depth = 0
|
||||
@@ -166,20 +177,13 @@ def build_context_menu(o, actions):
|
||||
return menu
|
||||
|
||||
|
||||
def resize_columns_to_content(header):
|
||||
""" """
|
||||
header.resizeSections(QtWidgets.QHeaderView.ResizeToContents)
|
||||
if header.sectionSize(0) > MAX_SECTION_SIZE:
|
||||
header.resizeSection(0, MAX_SECTION_SIZE)
|
||||
|
||||
|
||||
class CapaExplorerRulegenPreview(QtWidgets.QTextEdit):
|
||||
class CapaExplorerRulgenPreview(QtWidgets.QTextEdit):
|
||||
|
||||
INDENT = " " * 2
|
||||
|
||||
def __init__(self, parent=None):
|
||||
""" """
|
||||
super(CapaExplorerRulegenPreview, self).__init__(parent)
|
||||
super(CapaExplorerRulgenPreview, self).__init__(parent)
|
||||
|
||||
self.setFont(QtGui.QFont("Courier", weight=QtGui.QFont.Bold))
|
||||
self.setLineWrapMode(QtWidgets.QTextEdit.NoWrap)
|
||||
@@ -197,8 +201,7 @@ class CapaExplorerRulegenPreview(QtWidgets.QTextEdit):
|
||||
" meta:",
|
||||
" name: <insert_name>",
|
||||
" namespace: <insert_namespace>",
|
||||
" authors:",
|
||||
" - %s" % author,
|
||||
" author: %s" % author,
|
||||
" scope: %s" % scope,
|
||||
" references: <insert_references>",
|
||||
" examples:",
|
||||
@@ -284,7 +287,7 @@ class CapaExplorerRulegenPreview(QtWidgets.QTextEdit):
|
||||
self.set_selection(select_start_ppos, select_end_ppos, len(self.toPlainText()))
|
||||
self.verticalScrollBar().setSliderPosition(scroll_ppos)
|
||||
else:
|
||||
super(CapaExplorerRulegenPreview, self).keyPressEvent(e)
|
||||
super(CapaExplorerRulgenPreview, self).keyPressEvent(e)
|
||||
|
||||
def count_previous_lines_from_block(self, block):
|
||||
"""calculate number of lines preceding block"""
|
||||
@@ -304,17 +307,18 @@ class CapaExplorerRulegenPreview(QtWidgets.QTextEdit):
|
||||
self.setTextCursor(cursor)
|
||||
|
||||
|
||||
class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
class CapaExplorerRulgenEditor(QtWidgets.QTreeWidget):
|
||||
|
||||
updated = QtCore.pyqtSignal()
|
||||
|
||||
def __init__(self, preview, parent=None):
|
||||
""" """
|
||||
super(CapaExplorerRulegenEditor, self).__init__(parent)
|
||||
super(CapaExplorerRulgenEditor, self).__init__(parent)
|
||||
|
||||
self.preview = preview
|
||||
|
||||
self.setHeaderLabels(["Feature", "Description", "Comment"])
|
||||
self.header().setSectionResizeMode(QtWidgets.QHeaderView.ResizeToContents)
|
||||
self.header().setStretchLastSection(False)
|
||||
self.setExpandsOnDoubleClick(False)
|
||||
self.setEditTriggers(QtWidgets.QAbstractItemView.NoEditTriggers)
|
||||
@@ -322,10 +326,6 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
self.setSelectionMode(QtWidgets.QAbstractItemView.ExtendedSelection)
|
||||
self.setStyleSheet("QTreeView::item {padding-right: 15 px;padding-bottom: 2 px;}")
|
||||
|
||||
# configure view columns to auto-resize
|
||||
for idx in range(3):
|
||||
self.header().setSectionResizeMode(idx, QtWidgets.QHeaderView.Interactive)
|
||||
|
||||
# enable drag and drop
|
||||
self.setDragEnabled(True)
|
||||
self.setAcceptDrops(True)
|
||||
@@ -335,9 +335,8 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
self.itemChanged.connect(self.slot_item_changed)
|
||||
self.customContextMenuRequested.connect(self.slot_custom_context_menu_requested)
|
||||
self.itemDoubleClicked.connect(self.slot_item_double_clicked)
|
||||
self.expanded.connect(self.slot_resize_columns_to_content)
|
||||
self.collapsed.connect(self.slot_resize_columns_to_content)
|
||||
|
||||
self.root = None
|
||||
self.reset_view()
|
||||
|
||||
self.is_editing = False
|
||||
@@ -374,30 +373,28 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
|
||||
def dragMoveEvent(self, e):
|
||||
""" """
|
||||
super(CapaExplorerRulegenEditor, self).dragMoveEvent(e)
|
||||
super(CapaExplorerRulgenEditor, self).dragMoveEvent(e)
|
||||
|
||||
def dragEventEnter(self, e):
|
||||
""" """
|
||||
super(CapaExplorerRulegenEditor, self).dragEventEnter(e)
|
||||
super(CapaExplorerRulgenEditor, self).dragEventEnter(e)
|
||||
|
||||
def dropEvent(self, e):
|
||||
""" """
|
||||
if not self.indexAt(e.pos()).isValid():
|
||||
return
|
||||
|
||||
super(CapaExplorerRulegenEditor, self).dropEvent(e)
|
||||
super(CapaExplorerRulgenEditor, self).dropEvent(e)
|
||||
|
||||
# self.prune_expressions()
|
||||
self.update_preview()
|
||||
expand_tree(self.invisibleRootItem())
|
||||
self.expandAll()
|
||||
|
||||
def reset_view(self):
|
||||
""" """
|
||||
self.root = None
|
||||
self.clear()
|
||||
|
||||
def slot_resize_columns_to_content(self):
|
||||
""" """
|
||||
resize_columns_to_content(self.header())
|
||||
|
||||
def slot_item_changed(self, item, column):
|
||||
""" """
|
||||
if self.is_editing:
|
||||
@@ -407,21 +404,16 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
def slot_remove_selected(self, action):
|
||||
""" """
|
||||
for o in self.selectedItems():
|
||||
if o.parent() is None:
|
||||
# special handling for top-level items
|
||||
if o == self.root:
|
||||
self.takeTopLevelItem(self.indexOfTopLevelItem(o))
|
||||
self.root = None
|
||||
continue
|
||||
o.parent().removeChild(o)
|
||||
|
||||
def slot_nest_features(self, action):
|
||||
""" """
|
||||
# we don't want to add new features under the invisible root because capa rules should
|
||||
# contain a single top-level node; this may not always be the case so we default to the last
|
||||
# child node that was added to the invisible root
|
||||
top_node = self.invisibleRootItem().child(self.invisibleRootItem().childCount() - 1)
|
||||
|
||||
# create a new parent under top-level node
|
||||
new_parent = self.new_expression_node(top_node, (action.data()[0], ""))
|
||||
# create a new parent under root node, by default; new node added last position in tree
|
||||
new_parent = self.new_expression_node(self.root, (action.data()[0], ""))
|
||||
|
||||
if "basic block" in action.data()[0]:
|
||||
# add default child expression when nesting under basic block
|
||||
@@ -439,7 +431,7 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
""" """
|
||||
expression, o = action.data()
|
||||
if "basic block" in expression and "basic block" not in o.text(
|
||||
CapaExplorerRulegenEditor.get_column_feature_index()
|
||||
CapaExplorerRulgenEditor.get_column_feature_index()
|
||||
):
|
||||
# current expression is "basic block", and not changing to "basic block" expression
|
||||
children = o.takeChildren()
|
||||
@@ -447,7 +439,7 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
for child in children:
|
||||
new_parent.addChild(child)
|
||||
new_parent.setExpanded(True)
|
||||
o.setText(CapaExplorerRulegenEditor.get_column_feature_index(), expression)
|
||||
o.setText(CapaExplorerRulgenEditor.get_column_feature_index(), expression)
|
||||
|
||||
def slot_clear_all(self, action):
|
||||
""" """
|
||||
@@ -458,7 +450,7 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
if not self.indexAt(pos).isValid():
|
||||
# user selected invalid index
|
||||
self.load_custom_context_menu_invalid_index(pos)
|
||||
elif self.itemAt(pos).capa_type == CapaExplorerRulegenEditor.get_node_type_expression():
|
||||
elif self.itemAt(pos).capa_type == CapaExplorerRulgenEditor.get_node_type_expression():
|
||||
# user selected expression node
|
||||
self.load_custom_context_menu_expression(pos)
|
||||
else:
|
||||
@@ -470,8 +462,8 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
def slot_item_double_clicked(self, o, column):
|
||||
""" """
|
||||
if column in (
|
||||
CapaExplorerRulegenEditor.get_column_comment_index(),
|
||||
CapaExplorerRulegenEditor.get_column_description_index(),
|
||||
CapaExplorerRulgenEditor.get_column_comment_index(),
|
||||
CapaExplorerRulgenEditor.get_column_description_index(),
|
||||
):
|
||||
o.setFlags(o.flags() | QtCore.Qt.ItemIsEditable)
|
||||
self.editItem(o, column)
|
||||
@@ -557,7 +549,7 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
font = QtGui.QFont()
|
||||
font.setBold(True)
|
||||
|
||||
o.setFont(CapaExplorerRulegenEditor.get_column_feature_index(), font)
|
||||
o.setFont(CapaExplorerRulgenEditor.get_column_feature_index(), font)
|
||||
|
||||
def style_feature_node(self, o):
|
||||
""" """
|
||||
@@ -568,8 +560,8 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
font.setWeight(QtGui.QFont.Medium)
|
||||
brush.setColor(QtGui.QColor(*COLOR_GREEN_RGB))
|
||||
|
||||
o.setFont(CapaExplorerRulegenEditor.get_column_feature_index(), font)
|
||||
o.setForeground(CapaExplorerRulegenEditor.get_column_feature_index(), brush)
|
||||
o.setFont(CapaExplorerRulgenEditor.get_column_feature_index(), font)
|
||||
o.setForeground(CapaExplorerRulgenEditor.get_column_feature_index(), brush)
|
||||
|
||||
def style_comment_node(self, o):
|
||||
""" """
|
||||
@@ -577,22 +569,22 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
font.setBold(True)
|
||||
font.setFamily("Courier")
|
||||
|
||||
o.setFont(CapaExplorerRulegenEditor.get_column_feature_index(), font)
|
||||
o.setFont(CapaExplorerRulgenEditor.get_column_feature_index(), font)
|
||||
|
||||
def set_expression_node(self, o):
|
||||
""" """
|
||||
setattr(o, "capa_type", CapaExplorerRulegenEditor.get_node_type_expression())
|
||||
setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_expression())
|
||||
self.style_expression_node(o)
|
||||
|
||||
def set_feature_node(self, o):
|
||||
""" """
|
||||
setattr(o, "capa_type", CapaExplorerRulegenEditor.get_node_type_feature())
|
||||
setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_feature())
|
||||
o.setFlags(o.flags() & ~QtCore.Qt.ItemIsDropEnabled)
|
||||
self.style_feature_node(o)
|
||||
|
||||
def set_comment_node(self, o):
|
||||
""" """
|
||||
setattr(o, "capa_type", CapaExplorerRulegenEditor.get_node_type_comment())
|
||||
setattr(o, "capa_type", CapaExplorerRulgenEditor.get_node_type_comment())
|
||||
o.setFlags(o.flags() & ~QtCore.Qt.ItemIsDropEnabled)
|
||||
|
||||
self.style_comment_node(o)
|
||||
@@ -623,144 +615,97 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
|
||||
def update_features(self, features):
|
||||
""" """
|
||||
if not self.invisibleRootItem().childCount():
|
||||
# empty tree; add a default node
|
||||
self.new_expression_node(self.invisibleRootItem(), ("- or:", ""))
|
||||
|
||||
# we don't want to add new features under the invisible root because capa rules should
|
||||
# contain a single top-level node; this may not always be the case so we default to the last
|
||||
# child node that was added to the invisible root
|
||||
top_node = self.invisibleRootItem().child(self.invisibleRootItem().childCount() - 1)
|
||||
if not self.root:
|
||||
# root node does not exist, create default node, set expanded
|
||||
self.root = self.new_expression_node(self, ("- or:", ""))
|
||||
|
||||
# build feature counts
|
||||
counted = list(zip(Counter(features).keys(), Counter(features).values()))
|
||||
|
||||
# single features
|
||||
for (k, v) in filter(lambda t: t[1] == 1, counted):
|
||||
if isinstance(k, (capa.features.common.String,)):
|
||||
value = '"%s"' % capa.features.common.escape_string(k.get_value_str())
|
||||
if isinstance(k, (capa.features.String,)):
|
||||
value = '"%s"' % capa.features.escape_string(k.get_value_str())
|
||||
else:
|
||||
value = k.get_value_str()
|
||||
self.new_feature_node(top_node, ("- %s: %s" % (k.name.lower(), value), ""))
|
||||
self.new_feature_node(self.root, ("- %s: %s" % (k.name.lower(), value), ""))
|
||||
|
||||
# n > 1 features
|
||||
for (k, v) in filter(lambda t: t[1] > 1, counted):
|
||||
if k.value:
|
||||
if isinstance(k, (capa.features.common.String,)):
|
||||
value = '"%s"' % capa.features.common.escape_string(k.get_value_str())
|
||||
if isinstance(k, (capa.features.String,)):
|
||||
value = '"%s"' % capa.features.escape_string(k.get_value_str())
|
||||
else:
|
||||
value = k.get_value_str()
|
||||
display = "- count(%s(%s)): %d" % (k.name.lower(), value, v)
|
||||
else:
|
||||
display = "- count(%s): %d" % (k.name.lower(), v)
|
||||
self.new_feature_node(top_node, (display, ""))
|
||||
self.new_feature_node(self.root, (display, ""))
|
||||
|
||||
self.expandAll()
|
||||
self.update_preview()
|
||||
expand_tree(self.invisibleRootItem())
|
||||
resize_columns_to_content(self.header())
|
||||
|
||||
def make_child_node_from_feature(self, parent, feature):
|
||||
""" """
|
||||
feature, comment, description = feature
|
||||
|
||||
# we need special handling for the "description" tag; meaning we don't add a new node but simply
|
||||
# set the "description" column for the appropriate parent node
|
||||
if feature.startswith("description:"):
|
||||
if not parent:
|
||||
# we shouldn't have description without a parent; do nothing
|
||||
return None
|
||||
|
||||
# we don't add a new node for description; either set description column of parent's last child
|
||||
# or the parent itself
|
||||
if parent.childCount():
|
||||
parent.child(parent.childCount() - 1).setText(1, feature.lstrip("description:").lstrip())
|
||||
else:
|
||||
parent.setText(1, feature.lstrip("description:").lstrip())
|
||||
return None
|
||||
elif feature.startswith("- description:"):
|
||||
if not parent:
|
||||
# we shouldn't have a description without a parent; do nothing
|
||||
return None
|
||||
|
||||
# we don't add a new node for description; set the description column of the parent instead
|
||||
parent.setText(1, feature.lstrip("- description:").lstrip())
|
||||
return None
|
||||
|
||||
node = QtWidgets.QTreeWidgetItem(parent)
|
||||
|
||||
# set node text to data parsed from feature
|
||||
for (idx, text) in enumerate((feature, comment, description)):
|
||||
node.setText(idx, text)
|
||||
|
||||
# we need to set our own type so we can control the GUI accordingly
|
||||
if feature.startswith(("- and:", "- or:", "- not:", "- basic block:", "- optional:")):
|
||||
setattr(node, "capa_type", CapaExplorerRulegenEditor.get_node_type_expression())
|
||||
elif feature.startswith("#"):
|
||||
setattr(node, "capa_type", CapaExplorerRulegenEditor.get_node_type_comment())
|
||||
else:
|
||||
setattr(node, "capa_type", CapaExplorerRulegenEditor.get_node_type_feature())
|
||||
|
||||
# format the node based on its type
|
||||
(self.set_expression_node, self.set_feature_node, self.set_comment_node)[node.capa_type](node)
|
||||
|
||||
parent.addChild(node)
|
||||
|
||||
return node
|
||||
|
||||
def load_features_from_yaml(self, rule_text, update_preview=False):
|
||||
""" """
|
||||
|
||||
def add_node(parent, node):
|
||||
if node.text(0).startswith("description:"):
|
||||
if parent.childCount():
|
||||
parent.child(parent.childCount() - 1).setText(1, node.text(0).lstrip("description:").lstrip())
|
||||
else:
|
||||
parent.setText(1, node.text(0).lstrip("description:").lstrip())
|
||||
elif node.text(0).startswith("- description:"):
|
||||
parent.setText(1, node.text(0).lstrip("- description:").lstrip())
|
||||
else:
|
||||
parent.addChild(node)
|
||||
|
||||
def build(parent, nodes):
|
||||
if nodes:
|
||||
child_lvl = nodes[0].capa_level
|
||||
while nodes:
|
||||
node = nodes.pop(0)
|
||||
if node.capa_level == child_lvl:
|
||||
add_node(parent, node)
|
||||
elif node.capa_level > child_lvl:
|
||||
nodes.insert(0, node)
|
||||
build(parent.child(parent.childCount() - 1), nodes)
|
||||
else:
|
||||
parent = parent.parent() if parent.parent() else parent
|
||||
add_node(parent, node)
|
||||
|
||||
self.reset_view()
|
||||
|
||||
# check for lack of features block
|
||||
if -1 == rule_text.find("features:"):
|
||||
return
|
||||
|
||||
rule_features = rule_text[rule_text.find("features:") + len("features:") :].strip("\n")
|
||||
rule_features = rule_text[rule_text.find("features:") + len("features:") :].strip()
|
||||
rule_nodes = list(yaml_to_nodes(rule_features))
|
||||
|
||||
if not rule_features:
|
||||
# no features; nothing to do
|
||||
# check for lack of nodes
|
||||
if not rule_nodes:
|
||||
return
|
||||
|
||||
# build tree from yaml text using stack-based algorithm to build parent -> child edges
|
||||
stack = [self.invisibleRootItem()]
|
||||
for line in rule_features.splitlines():
|
||||
if not len(line.strip()):
|
||||
continue
|
||||
for o in rule_nodes:
|
||||
(self.set_expression_node, self.set_feature_node, self.set_comment_node)[o.capa_type](o)
|
||||
|
||||
indent = calc_indent_from_line(line)
|
||||
|
||||
# we need to grow our stack to ensure proper parent -> child edges
|
||||
if indent > len(stack):
|
||||
stack.extend([None] * (indent - len(stack)))
|
||||
|
||||
# shave the stack; divide by 2 because even indent, add 1 to avoid shaving root node
|
||||
stack[indent // 2 + 1 :] = []
|
||||
|
||||
# find our parent; should be last node in stack not None
|
||||
parent = None
|
||||
for o in stack[::-1]:
|
||||
if o:
|
||||
parent = o
|
||||
break
|
||||
|
||||
node = self.make_child_node_from_feature(parent, parse_yaml_line(line.strip()))
|
||||
|
||||
# append our new node in case its a parent for another node
|
||||
if node:
|
||||
stack.append(node)
|
||||
self.root = rule_nodes.pop(0)
|
||||
self.addTopLevelItem(self.root)
|
||||
|
||||
if update_preview:
|
||||
self.preview.blockSignals(True)
|
||||
self.preview.setPlainText(rule_text)
|
||||
self.preview.blockSignals(False)
|
||||
|
||||
expand_tree(self.invisibleRootItem())
|
||||
build(self.root, rule_nodes)
|
||||
|
||||
self.expandAll()
|
||||
|
||||
def get_features(self, selected=False, ignore=()):
|
||||
""" """
|
||||
for feature in filter(
|
||||
lambda o: o.capa_type
|
||||
in (CapaExplorerRulegenEditor.get_node_type_feature(), CapaExplorerRulegenEditor.get_node_type_comment()),
|
||||
in (CapaExplorerRulgenEditor.get_node_type_feature(), CapaExplorerRulgenEditor.get_node_type_comment()),
|
||||
tuple(iterate_tree(self)),
|
||||
):
|
||||
if feature in ignore:
|
||||
@@ -772,7 +717,7 @@ class CapaExplorerRulegenEditor(QtWidgets.QTreeWidget):
|
||||
def get_expressions(self, selected=False, ignore=()):
|
||||
""" """
|
||||
for expression in filter(
|
||||
lambda o: o.capa_type == CapaExplorerRulegenEditor.get_node_type_expression(), tuple(iterate_tree(self))
|
||||
lambda o: o.capa_type == CapaExplorerRulgenEditor.get_node_type_expression(), tuple(iterate_tree(self))
|
||||
):
|
||||
if expression in ignore:
|
||||
continue
|
||||
@@ -789,13 +734,10 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
self.parent_items = {}
|
||||
self.editor = editor
|
||||
|
||||
self.setHeaderLabels(["Feature", "Address"])
|
||||
self.setHeaderLabels(["Feature", "Virtual Address"])
|
||||
self.header().setSectionResizeMode(QtWidgets.QHeaderView.ResizeToContents)
|
||||
self.setStyleSheet("QTreeView::item {padding-right: 15 px;padding-bottom: 2 px;}")
|
||||
|
||||
# configure view columns to auto-resize
|
||||
for idx in range(2):
|
||||
self.header().setSectionResizeMode(idx, QtWidgets.QHeaderView.Interactive)
|
||||
|
||||
self.setExpandsOnDoubleClick(False)
|
||||
self.setContextMenuPolicy(QtCore.Qt.CustomContextMenu)
|
||||
self.setSelectionMode(QtWidgets.QAbstractItemView.ExtendedSelection)
|
||||
@@ -803,8 +745,6 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
# connect slots
|
||||
self.itemDoubleClicked.connect(self.slot_item_double_clicked)
|
||||
self.customContextMenuRequested.connect(self.slot_custom_context_menu_requested)
|
||||
self.expanded.connect(self.slot_resize_columns_to_content)
|
||||
self.collapsed.connect(self.slot_resize_columns_to_content)
|
||||
|
||||
self.reset_view()
|
||||
|
||||
@@ -832,24 +772,12 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
""" """
|
||||
self.clear()
|
||||
|
||||
def slot_resize_columns_to_content(self):
|
||||
""" """
|
||||
resize_columns_to_content(self.header())
|
||||
|
||||
def slot_add_selected_features(self, action):
|
||||
""" """
|
||||
selected = [item.data(0, 0x100) for item in self.selectedItems()]
|
||||
if selected:
|
||||
self.editor.update_features(selected)
|
||||
|
||||
def slot_add_n_bytes_feature(self, action):
|
||||
""" """
|
||||
count = idaapi.ask_long(16, f"Enter number of bytes (1-{capa.features.common.MAX_BYTES_FEATURE_SIZE}):")
|
||||
if count and 1 <= count <= capa.features.common.MAX_BYTES_FEATURE_SIZE:
|
||||
item = self.selectedItems()[0].data(0, 0x100)
|
||||
item.value = item.value[:count]
|
||||
self.editor.update_features([item])
|
||||
|
||||
def slot_custom_context_menu_requested(self, pos):
|
||||
""" """
|
||||
actions = []
|
||||
@@ -861,8 +789,6 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
|
||||
if selected_items_count == 1:
|
||||
action_add_features_fmt = "Add feature"
|
||||
if isinstance(self.selectedItems()[0].data(0, 0x100), capa.features.common.Bytes):
|
||||
actions.append(("Add n bytes...", (), self.slot_add_n_bytes_feature))
|
||||
else:
|
||||
action_add_features_fmt = "Add %d features" % selected_items_count
|
||||
|
||||
@@ -892,58 +818,13 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
if data:
|
||||
to_match = data.get_value_str()
|
||||
if not to_match or text.lower() not in to_match.lower():
|
||||
if not o.isHidden():
|
||||
o.setHidden(True)
|
||||
o.setHidden(True)
|
||||
continue
|
||||
if o.isHidden():
|
||||
o.setHidden(False)
|
||||
if o.childCount() and not o.isExpanded():
|
||||
o.setExpanded(True)
|
||||
o.setHidden(False)
|
||||
o.setExpanded(True)
|
||||
else:
|
||||
self.show_all_items()
|
||||
|
||||
def filter_items_by_ea(self, min_ea, max_ea=None):
|
||||
""" """
|
||||
visited = []
|
||||
|
||||
def show_item_and_parents(_o):
|
||||
"""iteratively show and expand an item and its' parents"""
|
||||
while _o:
|
||||
visited.append(_o)
|
||||
if _o.isHidden():
|
||||
_o.setHidden(False)
|
||||
if _o.childCount() and not _o.isExpanded():
|
||||
_o.setExpanded(True)
|
||||
_o = _o.parent()
|
||||
|
||||
for o in iterate_tree(self):
|
||||
if o in visited:
|
||||
# save some cycles, only visit item once
|
||||
continue
|
||||
|
||||
# read ea from "Address" column
|
||||
o_ea = o.text(CapaExplorerRulegenFeatures.get_column_address_index())
|
||||
|
||||
if o_ea == "":
|
||||
# ea may be empty, hide by default
|
||||
if not o.isHidden():
|
||||
o.setHidden(True)
|
||||
continue
|
||||
|
||||
o_ea = int(o_ea, 16)
|
||||
|
||||
if max_ea is not None and min_ea <= o_ea <= max_ea:
|
||||
show_item_and_parents(o)
|
||||
elif o_ea == min_ea:
|
||||
show_item_and_parents(o)
|
||||
else:
|
||||
# made it here, hide by default
|
||||
if not o.isHidden():
|
||||
o.setHidden(True)
|
||||
|
||||
# resize the view for UX
|
||||
resize_columns_to_content(self.header())
|
||||
|
||||
def style_parent_node(self, o):
|
||||
""" """
|
||||
font = QtGui.QFont()
|
||||
@@ -1005,25 +886,23 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
self.parse_features_for_tree(self.new_parent_node(self, ("File Scope",)), file_features)
|
||||
if func_features:
|
||||
self.parse_features_for_tree(self.new_parent_node(self, ("Function/Basic Block Scope",)), func_features)
|
||||
resize_columns_to_content(self.header())
|
||||
|
||||
def parse_features_for_tree(self, parent, features):
|
||||
""" """
|
||||
self.parent_items = {}
|
||||
|
||||
def format_address(e):
|
||||
assert isinstance(e, Address)
|
||||
return "%X" % e if not isinstance(e, _NoAddress) else ""
|
||||
return "%X" % e if e else ""
|
||||
|
||||
def format_feature(feature):
|
||||
""" """
|
||||
name = feature.name.lower()
|
||||
value = feature.get_value_str()
|
||||
if isinstance(feature, (capa.features.common.String,)):
|
||||
value = '"%s"' % capa.features.common.escape_string(value)
|
||||
if isinstance(feature, (capa.features.String,)):
|
||||
value = '"%s"' % capa.features.escape_string(value)
|
||||
return "%s(%s)" % (name, value)
|
||||
|
||||
for (feature, addrs) in sorted(features.items(), key=lambda k: sorted(k[1])):
|
||||
for (feature, eas) in sorted(features.items(), key=lambda k: sorted(k[1])):
|
||||
if isinstance(feature, capa.features.basicblock.BasicBlock):
|
||||
# filter basic blocks for now, we may want to add these back in some time
|
||||
# in the future
|
||||
@@ -1035,7 +914,7 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
|
||||
# level 1
|
||||
if feature not in self.parent_items:
|
||||
if len(addrs) > 1:
|
||||
if len(eas) > 1:
|
||||
self.parent_items[feature] = self.new_parent_node(
|
||||
self.parent_items[type(feature)], (format_feature(feature),), feature=feature
|
||||
)
|
||||
@@ -1045,18 +924,14 @@ class CapaExplorerRulegenFeatures(QtWidgets.QTreeWidget):
|
||||
)
|
||||
|
||||
# level n > 1
|
||||
if len(addrs) > 1:
|
||||
for addr in sorted(addrs):
|
||||
if len(eas) > 1:
|
||||
for ea in sorted(eas):
|
||||
self.new_leaf_node(
|
||||
self.parent_items[feature], (format_feature(feature), format_address(addr)), feature=feature
|
||||
self.parent_items[feature], (format_feature(feature), format_address(ea)), feature=feature
|
||||
)
|
||||
else:
|
||||
if addrs:
|
||||
addr = addrs.pop()
|
||||
else:
|
||||
# some features may not have an address e.g. "format"
|
||||
addr = _NoAddress()
|
||||
for (i, v) in enumerate((format_feature(feature), format_address(addr))):
|
||||
ea = eas.pop()
|
||||
for (i, v) in enumerate((format_feature(feature), format_address(ea))):
|
||||
self.parent_items[feature].setText(i, v)
|
||||
self.parent_items[feature].setData(0, 0x100, feature)
|
||||
|
||||
@@ -1124,7 +999,11 @@ class CapaExplorerQtreeView(QtWidgets.QTreeView):
|
||||
def slot_resize_columns_to_content(self):
|
||||
"""reset view columns to contents"""
|
||||
if self.should_resize_columns:
|
||||
resize_columns_to_content(self.header())
|
||||
self.header().resizeSections(QtWidgets.QHeaderView.ResizeToContents)
|
||||
|
||||
# limit size of first section
|
||||
if self.header().sectionSize(0) > MAX_SECTION_SIZE:
|
||||
self.header().resizeSection(0, MAX_SECTION_SIZE)
|
||||
|
||||
def map_index_to_source_item(self, model_index):
|
||||
"""map proxy model index to source model item
|
||||
|
||||
1055
capa/main.py
1055
capa/main.py
File diff suppressed because it is too large
Load Diff
@@ -1,70 +0,0 @@
|
||||
import logging
|
||||
|
||||
import capa.engine as ceng
|
||||
import capa.features.common
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_node_cost(node):
|
||||
if isinstance(node, (capa.features.common.OS, capa.features.common.Arch, capa.features.common.Format)):
|
||||
# we assume these are the most restrictive features:
|
||||
# authors commonly use them at the start of rules to restrict the category of samples to inspect
|
||||
return 0
|
||||
|
||||
# elif "everything else":
|
||||
# return 1
|
||||
#
|
||||
# this should be all hash-lookup features.
|
||||
# see below.
|
||||
|
||||
elif isinstance(node, (capa.features.common.Substring, capa.features.common.Regex, capa.features.common.Bytes)):
|
||||
# substring and regex features require a full scan of each string
|
||||
# which we anticipate is more expensive then a hash lookup feature (e.g. mnemonic or count).
|
||||
#
|
||||
# TODO: compute the average cost of these feature relative to hash feature
|
||||
# and adjust the factor accordingly.
|
||||
return 2
|
||||
|
||||
elif isinstance(node, (ceng.Not, ceng.Range)):
|
||||
# the cost of these nodes are defined by the complexity of their single child.
|
||||
return 1 + get_node_cost(node.child)
|
||||
|
||||
elif isinstance(node, (ceng.And, ceng.Or, ceng.Some)):
|
||||
# the cost of these nodes is the full cost of their children
|
||||
# as this is the worst-case scenario.
|
||||
return 1 + sum(map(get_node_cost, node.children))
|
||||
|
||||
else:
|
||||
# this should be all hash-lookup features.
|
||||
# we give this a arbitrary weight of 1.
|
||||
# the only thing more "important" than this is checking OS/Arch/Format.
|
||||
return 1
|
||||
|
||||
|
||||
def optimize_statement(statement):
|
||||
# this routine operates in-place
|
||||
|
||||
if isinstance(statement, (ceng.And, ceng.Or, ceng.Some)):
|
||||
# has .children
|
||||
statement.children = sorted(statement.children, key=lambda n: get_node_cost(n))
|
||||
return
|
||||
elif isinstance(statement, (ceng.Not, ceng.Range)):
|
||||
# has .child
|
||||
optimize_statement(statement.child)
|
||||
return
|
||||
else:
|
||||
# appears to be "simple"
|
||||
return
|
||||
|
||||
|
||||
def optimize_rule(rule):
|
||||
# this routine operates in-place
|
||||
optimize_statement(rule.statement)
|
||||
|
||||
|
||||
def optimize_rules(rules):
|
||||
logger.debug("optimizing %d rules", len(rules))
|
||||
for rule in rules:
|
||||
optimize_rule(rule)
|
||||
return rules
|
||||
10
capa/perf.py
10
capa/perf.py
@@ -1,10 +0,0 @@
|
||||
import collections
|
||||
from typing import Dict
|
||||
|
||||
# this structure is unstable and may change before the next major release.
|
||||
counters: Dict[str, int] = collections.Counter()
|
||||
|
||||
|
||||
def reset():
|
||||
global counters
|
||||
counters = collections.Counter()
|
||||
@@ -0,0 +1,266 @@
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import json
|
||||
|
||||
import six
|
||||
|
||||
import capa.rules
|
||||
import capa.engine
|
||||
|
||||
|
||||
def convert_statement_to_result_document(statement):
|
||||
"""
|
||||
"statement": {
|
||||
"type": "or"
|
||||
},
|
||||
|
||||
"statement": {
|
||||
"max": 9223372036854775808,
|
||||
"min": 2,
|
||||
"type": "range"
|
||||
},
|
||||
"""
|
||||
statement_type = statement.name.lower()
|
||||
result = {"type": statement_type}
|
||||
if statement.description:
|
||||
result["description"] = statement.description
|
||||
|
||||
if statement_type == "some" and statement.count == 0:
|
||||
result["type"] = "optional"
|
||||
elif statement_type == "some":
|
||||
result["count"] = statement.count
|
||||
elif statement_type == "range":
|
||||
result["min"] = statement.min
|
||||
result["max"] = statement.max
|
||||
result["child"] = convert_feature_to_result_document(statement.child)
|
||||
elif statement_type == "subscope":
|
||||
result["subscope"] = statement.scope
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def convert_feature_to_result_document(feature):
|
||||
"""
|
||||
"feature": {
|
||||
"number": 6,
|
||||
"type": "number"
|
||||
},
|
||||
|
||||
"feature": {
|
||||
"api": "ws2_32.WSASocket",
|
||||
"type": "api"
|
||||
},
|
||||
|
||||
"feature": {
|
||||
"match": "create TCP socket",
|
||||
"type": "match"
|
||||
},
|
||||
|
||||
"feature": {
|
||||
"characteristic": [
|
||||
"loop",
|
||||
true
|
||||
],
|
||||
"type": "characteristic"
|
||||
},
|
||||
"""
|
||||
result = {"type": feature.name, feature.name: feature.get_value_str()}
|
||||
if feature.description:
|
||||
result["description"] = feature.description
|
||||
if feature.name == "regex":
|
||||
result["match"] = feature.match
|
||||
return result
|
||||
|
||||
|
||||
def convert_node_to_result_document(node):
|
||||
"""
|
||||
"node": {
|
||||
"type": "statement",
|
||||
"statement": { ... }
|
||||
},
|
||||
|
||||
"node": {
|
||||
"type": "feature",
|
||||
"feature": { ... }
|
||||
},
|
||||
"""
|
||||
|
||||
if isinstance(node, capa.engine.Statement):
|
||||
return {
|
||||
"type": "statement",
|
||||
"statement": convert_statement_to_result_document(node),
|
||||
}
|
||||
elif isinstance(node, capa.features.Feature):
|
||||
return {
|
||||
"type": "feature",
|
||||
"feature": convert_feature_to_result_document(node),
|
||||
}
|
||||
else:
|
||||
raise RuntimeError("unexpected match node type")
|
||||
|
||||
|
||||
def convert_match_to_result_document(rules, capabilities, result):
|
||||
"""
|
||||
convert the given Result instance into a common, Python-native data structure.
|
||||
this will become part of the "result document" format that can be emitted to JSON.
|
||||
"""
|
||||
doc = {
|
||||
"success": bool(result.success),
|
||||
"node": convert_node_to_result_document(result.statement),
|
||||
"children": [convert_match_to_result_document(rules, capabilities, child) for child in result.children],
|
||||
}
|
||||
|
||||
# logic expression, like `and`, don't have locations - their children do.
|
||||
# so only add `locations` to feature nodes.
|
||||
if isinstance(result.statement, capa.features.Feature):
|
||||
if bool(result.success):
|
||||
doc["locations"] = result.locations
|
||||
elif isinstance(result.statement, capa.rules.Range):
|
||||
if bool(result.success):
|
||||
doc["locations"] = result.locations
|
||||
|
||||
# if we have a `match` statement, then we're referencing another rule.
|
||||
# this could an external rule (written by a human), or
|
||||
# rule generated to support a subscope (basic block, etc.)
|
||||
# we still want to include the matching logic in this tree.
|
||||
#
|
||||
# so, we need to lookup the other rule results
|
||||
# and then filter those down to the address used here.
|
||||
# finally, splice that logic into this tree.
|
||||
if (
|
||||
doc["node"]["type"] == "feature"
|
||||
and doc["node"]["feature"]["type"] == "match"
|
||||
# only add subtree on success,
|
||||
# because there won't be results for the other rule on failure.
|
||||
and doc["success"]
|
||||
):
|
||||
|
||||
rule_name = doc["node"]["feature"]["match"]
|
||||
rule = rules[rule_name]
|
||||
rule_matches = {address: result for (address, result) in capabilities[rule_name]}
|
||||
|
||||
if rule.meta.get("capa/subscope-rule"):
|
||||
# for a subscope rule, fixup the node to be a scope node, rather than a match feature node.
|
||||
#
|
||||
# e.g. `contain loop/30c4c78e29bf4d54894fc74f664c62e8` -> `basic block`
|
||||
scope = rule.meta["scope"]
|
||||
doc["node"] = {
|
||||
"type": "statement",
|
||||
"statement": {
|
||||
"type": "subscope",
|
||||
"subscope": scope,
|
||||
},
|
||||
}
|
||||
|
||||
for location in doc["locations"]:
|
||||
doc["children"].append(convert_match_to_result_document(rules, capabilities, rule_matches[location]))
|
||||
|
||||
return doc
|
||||
|
||||
|
||||
def convert_capabilities_to_result_document(meta, rules, capabilities):
|
||||
"""
|
||||
convert the given rule set and capabilities result to a common, Python-native data structure.
|
||||
this format can be directly emitted to JSON, or passed to the other `render_*` routines
|
||||
to render as text.
|
||||
|
||||
see examples of substructures in above routines.
|
||||
|
||||
schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"meta": {...},
|
||||
"rules: {
|
||||
$rule-name: {
|
||||
"meta": {...copied from rule.meta...},
|
||||
"matches: {
|
||||
$address: {...match details...},
|
||||
...
|
||||
}
|
||||
},
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Args:
|
||||
meta (Dict[str, Any]):
|
||||
rules (RuleSet):
|
||||
capabilities (Dict[str, List[Tuple[int, Result]]]):
|
||||
"""
|
||||
doc = {
|
||||
"meta": meta,
|
||||
"rules": {},
|
||||
}
|
||||
|
||||
for rule_name, matches in capabilities.items():
|
||||
rule = rules[rule_name]
|
||||
|
||||
if rule.meta.get("capa/subscope-rule"):
|
||||
continue
|
||||
|
||||
doc["rules"][rule_name] = {
|
||||
"meta": dict(rule.meta),
|
||||
"source": rule.definition,
|
||||
"matches": {
|
||||
addr: convert_match_to_result_document(rules, capabilities, match) for (addr, match) in matches
|
||||
},
|
||||
}
|
||||
|
||||
return doc
|
||||
|
||||
|
||||
def render_vverbose(meta, rules, capabilities):
|
||||
# there's an import loop here
|
||||
# if capa.render imports capa.render.vverbose
|
||||
# and capa.render.vverbose import capa.render (implicitly, as a submodule)
|
||||
# so, defer the import until routine is called, breaking the import loop.
|
||||
import capa.render.vverbose
|
||||
|
||||
doc = convert_capabilities_to_result_document(meta, rules, capabilities)
|
||||
return capa.render.vverbose.render_vverbose(doc)
|
||||
|
||||
|
||||
def render_verbose(meta, rules, capabilities):
|
||||
# break import loop
|
||||
import capa.render.verbose
|
||||
|
||||
doc = convert_capabilities_to_result_document(meta, rules, capabilities)
|
||||
return capa.render.verbose.render_verbose(doc)
|
||||
|
||||
|
||||
def render_default(meta, rules, capabilities):
|
||||
# break import loop
|
||||
import capa.render.default
|
||||
import capa.render.verbose
|
||||
|
||||
doc = convert_capabilities_to_result_document(meta, rules, capabilities)
|
||||
return capa.render.default.render_default(doc)
|
||||
|
||||
|
||||
class CapaJsonObjectEncoder(json.JSONEncoder):
|
||||
"""JSON encoder that emits Python sets as sorted lists"""
|
||||
|
||||
def default(self, obj):
|
||||
if isinstance(obj, (list, dict, int, float, bool, type(None))) or isinstance(obj, six.string_types):
|
||||
return json.JSONEncoder.default(self, obj)
|
||||
elif isinstance(obj, set):
|
||||
return list(sorted(obj))
|
||||
else:
|
||||
# probably will TypeError
|
||||
return json.JSONEncoder.default(self, obj)
|
||||
|
||||
|
||||
def render_json(meta, rules, capabilities):
|
||||
return json.dumps(
|
||||
convert_capabilities_to_result_document(meta, rules, capabilities),
|
||||
cls=CapaJsonObjectEncoder,
|
||||
sort_keys=True,
|
||||
)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -8,20 +8,15 @@
|
||||
|
||||
import collections
|
||||
|
||||
import six
|
||||
import tabulate
|
||||
|
||||
import capa.render.utils as rutils
|
||||
import capa.features.freeze as frz
|
||||
import capa.render.result_document as rd
|
||||
import capa.features.freeze.features as frzf
|
||||
from capa.rules import RuleSet
|
||||
from capa.engine import MatchResults
|
||||
from capa.render.utils import StringIO
|
||||
|
||||
tabulate.PRESERVE_WHITESPACE = True
|
||||
|
||||
|
||||
def width(s: str, character_count: int) -> str:
|
||||
def width(s, character_count):
|
||||
"""pad the given string to at least `character_count`"""
|
||||
if len(s) < character_count:
|
||||
return s + " " * (character_count - len(s))
|
||||
@@ -29,49 +24,47 @@ def width(s: str, character_count: int) -> str:
|
||||
return s
|
||||
|
||||
|
||||
def render_meta(doc: rd.ResultDocument, ostream: StringIO):
|
||||
def render_meta(doc, ostream):
|
||||
rows = [
|
||||
(width("md5", 22), width(doc.meta.sample.md5, 82)),
|
||||
("sha1", doc.meta.sample.sha1),
|
||||
("sha256", doc.meta.sample.sha256),
|
||||
("os", doc.meta.analysis.os),
|
||||
("format", doc.meta.analysis.format),
|
||||
("arch", doc.meta.analysis.arch),
|
||||
("path", doc.meta.sample.path),
|
||||
(width("md5", 22), width(doc["meta"]["sample"]["md5"], 82)),
|
||||
("sha1", doc["meta"]["sample"]["sha1"]),
|
||||
("sha256", doc["meta"]["sample"]["sha256"]),
|
||||
("path", doc["meta"]["sample"]["path"]),
|
||||
]
|
||||
|
||||
ostream.write(tabulate.tabulate(rows, tablefmt="psql"))
|
||||
ostream.write("\n")
|
||||
|
||||
|
||||
def find_subrule_matches(doc: rd.ResultDocument):
|
||||
def find_subrule_matches(doc):
|
||||
"""
|
||||
collect the rule names that have been matched as a subrule match.
|
||||
this way we can avoid displaying entries for things that are too specific.
|
||||
"""
|
||||
matches = set([])
|
||||
|
||||
def rec(match: rd.Match):
|
||||
if not match.success:
|
||||
def rec(node):
|
||||
if not node["success"]:
|
||||
# there's probably a bug here for rules that do `not: match: ...`
|
||||
# but we don't have any examples of this yet
|
||||
return
|
||||
|
||||
elif isinstance(match.node, rd.StatementNode):
|
||||
for child in match.children:
|
||||
elif node["node"]["type"] == "statement":
|
||||
for child in node["children"]:
|
||||
rec(child)
|
||||
|
||||
elif isinstance(match.node, rd.FeatureNode) and isinstance(match.node.feature, frzf.MatchFeature):
|
||||
matches.add(match.node.feature.match)
|
||||
elif node["node"]["type"] == "feature":
|
||||
if node["node"]["feature"]["type"] == "match":
|
||||
matches.add(node["node"]["feature"]["match"])
|
||||
|
||||
for rule in rutils.capability_rules(doc):
|
||||
for address, match in rule.matches:
|
||||
rec(match)
|
||||
for node in rule["matches"].values():
|
||||
rec(node)
|
||||
|
||||
return matches
|
||||
|
||||
|
||||
def render_capabilities(doc: rd.ResultDocument, ostream: StringIO):
|
||||
def render_capabilities(doc, ostream):
|
||||
"""
|
||||
example::
|
||||
|
||||
@@ -87,18 +80,18 @@ def render_capabilities(doc: rd.ResultDocument, ostream: StringIO):
|
||||
|
||||
rows = []
|
||||
for rule in rutils.capability_rules(doc):
|
||||
if rule.meta.name in subrule_matches:
|
||||
if rule["meta"]["name"] in subrule_matches:
|
||||
# rules that are also matched by other rules should not get rendered by default.
|
||||
# this cuts down on the amount of output while giving approx the same detail.
|
||||
# see #224
|
||||
continue
|
||||
|
||||
count = len(rule.matches)
|
||||
count = len(rule["matches"])
|
||||
if count == 1:
|
||||
capability = rutils.bold(rule.meta.name)
|
||||
capability = rutils.bold(rule["meta"]["name"])
|
||||
else:
|
||||
capability = "%s (%d matches)" % (rutils.bold(rule.meta.name), count)
|
||||
rows.append((capability, rule.meta.namespace))
|
||||
capability = "%s (%d matches)" % (rutils.bold(rule["meta"]["name"]), count)
|
||||
rows.append((capability, rule["meta"]["namespace"]))
|
||||
|
||||
if rows:
|
||||
ostream.write(
|
||||
@@ -109,7 +102,7 @@ def render_capabilities(doc: rd.ResultDocument, ostream: StringIO):
|
||||
ostream.writeln(rutils.bold("no capabilities found"))
|
||||
|
||||
|
||||
def render_attack(doc: rd.ResultDocument, ostream: StringIO):
|
||||
def render_attack(doc, ostream):
|
||||
"""
|
||||
example::
|
||||
|
||||
@@ -127,17 +120,31 @@ def render_attack(doc: rd.ResultDocument, ostream: StringIO):
|
||||
"""
|
||||
tactics = collections.defaultdict(set)
|
||||
for rule in rutils.capability_rules(doc):
|
||||
for attack in rule.meta.attack:
|
||||
tactics[attack.tactic].add((attack.technique, attack.subtechnique, attack.id))
|
||||
if not rule["meta"].get("att&ck"):
|
||||
continue
|
||||
|
||||
for attack in rule["meta"]["att&ck"]:
|
||||
tactic, _, rest = attack.partition("::")
|
||||
if "::" in rest:
|
||||
technique, _, rest = rest.partition("::")
|
||||
subtechnique, _, id = rest.rpartition(" ")
|
||||
tactics[tactic].add((technique, subtechnique, id))
|
||||
else:
|
||||
technique, _, id = rest.rpartition(" ")
|
||||
tactics[tactic].add((technique, id))
|
||||
|
||||
rows = []
|
||||
for tactic, techniques in sorted(tactics.items()):
|
||||
inner_rows = []
|
||||
for (technique, subtechnique, id) in sorted(techniques):
|
||||
if not subtechnique:
|
||||
for spec in sorted(techniques):
|
||||
if len(spec) == 2:
|
||||
technique, id = spec
|
||||
inner_rows.append("%s %s" % (rutils.bold(technique), id))
|
||||
else:
|
||||
elif len(spec) == 3:
|
||||
technique, subtechnique, id = spec
|
||||
inner_rows.append("%s::%s %s" % (rutils.bold(technique), subtechnique, id))
|
||||
else:
|
||||
raise RuntimeError("unexpected ATT&CK spec format")
|
||||
rows.append(
|
||||
(
|
||||
rutils.bold(tactic.upper()),
|
||||
@@ -154,7 +161,7 @@ def render_attack(doc: rd.ResultDocument, ostream: StringIO):
|
||||
ostream.write("\n")
|
||||
|
||||
|
||||
def render_mbc(doc: rd.ResultDocument, ostream: StringIO):
|
||||
def render_mbc(doc, ostream):
|
||||
"""
|
||||
example::
|
||||
|
||||
@@ -170,17 +177,35 @@ def render_mbc(doc: rd.ResultDocument, ostream: StringIO):
|
||||
"""
|
||||
objectives = collections.defaultdict(set)
|
||||
for rule in rutils.capability_rules(doc):
|
||||
for mbc in rule.meta.mbc:
|
||||
objectives[mbc.objective].add((mbc.behavior, mbc.method, mbc.id))
|
||||
if not rule["meta"].get("mbc"):
|
||||
continue
|
||||
|
||||
mbcs = rule["meta"]["mbc"]
|
||||
if not isinstance(mbcs, list):
|
||||
raise ValueError("invalid rule: MBC mapping is not a list")
|
||||
|
||||
for mbc in mbcs:
|
||||
objective, _, rest = mbc.partition("::")
|
||||
if "::" in rest:
|
||||
behavior, _, rest = rest.partition("::")
|
||||
method, _, id = rest.rpartition(" ")
|
||||
objectives[objective].add((behavior, method, id))
|
||||
else:
|
||||
behavior, _, id = rest.rpartition(" ")
|
||||
objectives[objective].add((behavior, id))
|
||||
|
||||
rows = []
|
||||
for objective, behaviors in sorted(objectives.items()):
|
||||
inner_rows = []
|
||||
for (behavior, method, id) in sorted(behaviors):
|
||||
if not method:
|
||||
inner_rows.append("%s [%s]" % (rutils.bold(behavior), id))
|
||||
for spec in sorted(behaviors):
|
||||
if len(spec) == 2:
|
||||
behavior, id = spec
|
||||
inner_rows.append("%s %s" % (rutils.bold(behavior), id))
|
||||
elif len(spec) == 3:
|
||||
behavior, method, id = spec
|
||||
inner_rows.append("%s::%s %s" % (rutils.bold(behavior), method, id))
|
||||
else:
|
||||
inner_rows.append("%s::%s [%s]" % (rutils.bold(behavior), method, id))
|
||||
raise RuntimeError("unexpected MBC spec format")
|
||||
rows.append(
|
||||
(
|
||||
rutils.bold(objective.upper()),
|
||||
@@ -195,7 +220,7 @@ def render_mbc(doc: rd.ResultDocument, ostream: StringIO):
|
||||
ostream.write("\n")
|
||||
|
||||
|
||||
def render_default(doc: rd.ResultDocument):
|
||||
def render_default(doc):
|
||||
ostream = rutils.StringIO()
|
||||
|
||||
render_meta(doc, ostream)
|
||||
@@ -207,8 +232,3 @@ def render_default(doc: rd.ResultDocument):
|
||||
render_capabilities(doc, ostream)
|
||||
|
||||
return ostream.getvalue()
|
||||
|
||||
|
||||
def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
|
||||
doc = rd.ResultDocument.from_capa(meta, rules, capabilities)
|
||||
return render_default(doc)
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
import capa.render.result_document as rd
|
||||
from capa.rules import RuleSet
|
||||
from capa.engine import MatchResults
|
||||
|
||||
|
||||
def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
|
||||
return rd.ResultDocument.from_capa(meta, rules, capabilities).json(exclude_none=True)
|
||||
@@ -1,557 +0,0 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
import datetime
|
||||
from typing import Any, Dict, Tuple, Union, Optional
|
||||
|
||||
from pydantic import Field, BaseModel
|
||||
|
||||
import capa.rules
|
||||
import capa.engine
|
||||
import capa.features.common
|
||||
import capa.features.freeze as frz
|
||||
import capa.features.address
|
||||
from capa.rules import RuleSet
|
||||
from capa.engine import MatchResults
|
||||
from capa.helpers import assert_never
|
||||
|
||||
|
||||
class FrozenModel(BaseModel):
|
||||
class Config:
|
||||
frozen = True
|
||||
|
||||
|
||||
class Sample(FrozenModel):
|
||||
md5: str
|
||||
sha1: str
|
||||
sha256: str
|
||||
path: str
|
||||
|
||||
|
||||
class BasicBlockLayout(FrozenModel):
|
||||
address: frz.Address
|
||||
|
||||
|
||||
class FunctionLayout(FrozenModel):
|
||||
address: frz.Address
|
||||
matched_basic_blocks: Tuple[BasicBlockLayout, ...]
|
||||
|
||||
|
||||
class Layout(FrozenModel):
|
||||
functions: Tuple[FunctionLayout, ...]
|
||||
|
||||
|
||||
class LibraryFunction(FrozenModel):
|
||||
address: frz.Address
|
||||
name: str
|
||||
|
||||
|
||||
class FunctionFeatureCount(FrozenModel):
|
||||
address: frz.Address
|
||||
count: int
|
||||
|
||||
|
||||
class FeatureCounts(FrozenModel):
|
||||
file: int
|
||||
functions: Tuple[FunctionFeatureCount, ...]
|
||||
|
||||
|
||||
class Analysis(FrozenModel):
|
||||
format: str
|
||||
arch: str
|
||||
os: str
|
||||
extractor: str
|
||||
rules: Tuple[str, ...]
|
||||
base_address: frz.Address
|
||||
layout: Layout
|
||||
feature_counts: FeatureCounts
|
||||
library_functions: Tuple[LibraryFunction, ...]
|
||||
|
||||
|
||||
class Metadata(FrozenModel):
|
||||
timestamp: datetime.datetime
|
||||
version: str
|
||||
argv: Tuple[str, ...]
|
||||
sample: Sample
|
||||
analysis: Analysis
|
||||
|
||||
@classmethod
|
||||
def from_capa(cls, meta: Any) -> "Metadata":
|
||||
return cls(
|
||||
timestamp=meta["timestamp"],
|
||||
version=meta["version"],
|
||||
argv=meta["argv"] if "argv" in meta else None,
|
||||
sample=Sample(
|
||||
md5=meta["sample"]["md5"],
|
||||
sha1=meta["sample"]["sha1"],
|
||||
sha256=meta["sample"]["sha256"],
|
||||
path=meta["sample"]["path"],
|
||||
),
|
||||
analysis=Analysis(
|
||||
format=meta["analysis"]["format"],
|
||||
arch=meta["analysis"]["arch"],
|
||||
os=meta["analysis"]["os"],
|
||||
extractor=meta["analysis"]["extractor"],
|
||||
rules=meta["analysis"]["rules"],
|
||||
base_address=frz.Address.from_capa(meta["analysis"]["base_address"]),
|
||||
layout=Layout(
|
||||
functions=[
|
||||
FunctionLayout(
|
||||
address=frz.Address.from_capa(address),
|
||||
matched_basic_blocks=[
|
||||
BasicBlockLayout(address=frz.Address.from_capa(bb)) for bb in f["matched_basic_blocks"]
|
||||
],
|
||||
)
|
||||
for address, f in meta["analysis"]["layout"]["functions"].items()
|
||||
]
|
||||
),
|
||||
feature_counts=FeatureCounts(
|
||||
file=meta["analysis"]["feature_counts"]["file"],
|
||||
functions=[
|
||||
FunctionFeatureCount(address=frz.Address.from_capa(address), count=count)
|
||||
for address, count in meta["analysis"]["feature_counts"]["functions"].items()
|
||||
],
|
||||
),
|
||||
library_functions=[
|
||||
LibraryFunction(address=frz.Address.from_capa(address), name=name)
|
||||
for address, name in meta["analysis"]["library_functions"].items()
|
||||
],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class StatementModel(FrozenModel):
|
||||
...
|
||||
|
||||
|
||||
class AndStatement(StatementModel):
|
||||
type = "and"
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class OrStatement(StatementModel):
|
||||
type = "or"
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class NotStatement(StatementModel):
|
||||
type = "not"
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class SomeStatement(StatementModel):
|
||||
type = "some"
|
||||
description: Optional[str]
|
||||
count: int
|
||||
|
||||
|
||||
class OptionalStatement(StatementModel):
|
||||
type = "optional"
|
||||
description: Optional[str]
|
||||
|
||||
|
||||
class RangeStatement(StatementModel):
|
||||
type = "range"
|
||||
description: Optional[str]
|
||||
min: int
|
||||
max: int
|
||||
child: frz.Feature
|
||||
|
||||
|
||||
class SubscopeStatement(StatementModel):
|
||||
type = "subscope"
|
||||
description: Optional[str]
|
||||
scope = capa.rules.Scope
|
||||
|
||||
|
||||
Statement = Union[
|
||||
OptionalStatement,
|
||||
AndStatement,
|
||||
OrStatement,
|
||||
NotStatement,
|
||||
SomeStatement,
|
||||
RangeStatement,
|
||||
SubscopeStatement,
|
||||
]
|
||||
|
||||
|
||||
class StatementNode(FrozenModel):
|
||||
type = "statement"
|
||||
statement: Statement
|
||||
|
||||
|
||||
def statement_from_capa(node: capa.engine.Statement) -> Statement:
|
||||
if isinstance(node, capa.engine.And):
|
||||
return AndStatement(description=node.description)
|
||||
|
||||
elif isinstance(node, capa.engine.Or):
|
||||
return OrStatement(description=node.description)
|
||||
|
||||
elif isinstance(node, capa.engine.Not):
|
||||
return NotStatement(description=node.description)
|
||||
|
||||
elif isinstance(node, capa.engine.Some):
|
||||
if node.count == 0:
|
||||
return OptionalStatement(description=node.description)
|
||||
|
||||
else:
|
||||
return SomeStatement(
|
||||
description=node.description,
|
||||
count=node.count,
|
||||
)
|
||||
|
||||
elif isinstance(node, capa.engine.Range):
|
||||
return RangeStatement(
|
||||
description=node.description,
|
||||
min=node.min,
|
||||
max=node.max,
|
||||
child=frz.feature_from_capa(node.child),
|
||||
)
|
||||
|
||||
elif isinstance(node, capa.engine.Subscope):
|
||||
return SubscopeStatement(
|
||||
description=node.description,
|
||||
scope=capa.rules.Scope(node.scope),
|
||||
)
|
||||
|
||||
else:
|
||||
raise NotImplementedError(f"statement_from_capa({type(node)}) not implemented")
|
||||
|
||||
|
||||
class FeatureNode(FrozenModel):
|
||||
type = "feature"
|
||||
feature: frz.Feature
|
||||
|
||||
|
||||
Node = Union[StatementNode, FeatureNode]
|
||||
|
||||
|
||||
def node_from_capa(node: Union[capa.engine.Statement, capa.engine.Feature]) -> Node:
|
||||
if isinstance(node, capa.engine.Statement):
|
||||
return StatementNode(statement=statement_from_capa(node))
|
||||
|
||||
elif isinstance(node, capa.engine.Feature):
|
||||
return FeatureNode(feature=frz.feature_from_capa(node))
|
||||
|
||||
else:
|
||||
assert_never(node)
|
||||
|
||||
|
||||
class Match(BaseModel):
|
||||
"""
|
||||
args:
|
||||
success: did the node match?
|
||||
node: the logic node or feature node.
|
||||
children: any children of the logic node. not relevent for features, can be empty.
|
||||
locations: where the feature matched. not relevant for logic nodes (except range), can be empty.
|
||||
captures: captured values from the string/regex feature, and the locations of those values.
|
||||
"""
|
||||
|
||||
success: bool
|
||||
node: Node
|
||||
children: Tuple["Match", ...]
|
||||
locations: Tuple[frz.Address, ...]
|
||||
captures: Dict[str, Tuple[frz.Address, ...]]
|
||||
|
||||
@classmethod
|
||||
def from_capa(
|
||||
cls,
|
||||
rules: RuleSet,
|
||||
capabilities: MatchResults,
|
||||
result: capa.engine.Result,
|
||||
) -> "Match":
|
||||
success = bool(result)
|
||||
|
||||
node = node_from_capa(result.statement)
|
||||
children = [Match.from_capa(rules, capabilities, child) for child in result.children]
|
||||
|
||||
# logic expression, like `and`, don't have locations - their children do.
|
||||
# so only add `locations` to feature nodes.
|
||||
locations = []
|
||||
if isinstance(node, FeatureNode) and success:
|
||||
locations = list(map(frz.Address.from_capa, result.locations))
|
||||
elif isinstance(node, StatementNode) and isinstance(node.statement, RangeStatement) and success:
|
||||
locations = list(map(frz.Address.from_capa, result.locations))
|
||||
|
||||
captures = {}
|
||||
if isinstance(result.statement, (capa.features.common._MatchedSubstring, capa.features.common._MatchedRegex)):
|
||||
captures = {
|
||||
capture: list(map(frz.Address.from_capa, locs)) for capture, locs in result.statement.matches.items()
|
||||
}
|
||||
|
||||
# if we have a `match` statement, then we're referencing another rule or namespace.
|
||||
# this could an external rule (written by a human), or
|
||||
# rule generated to support a subscope (basic block, etc.)
|
||||
# we still want to include the matching logic in this tree.
|
||||
#
|
||||
# so, we need to lookup the other rule results
|
||||
# and then filter those down to the address used here.
|
||||
# finally, splice that logic into this tree.
|
||||
if (
|
||||
isinstance(node, FeatureNode)
|
||||
and isinstance(node.feature, frz.features.MatchFeature)
|
||||
# only add subtree on success,
|
||||
# because there won't be results for the other rule on failure.
|
||||
and success
|
||||
):
|
||||
name = node.feature.match
|
||||
|
||||
if name in rules:
|
||||
# this is a rule that we're matching
|
||||
#
|
||||
# pull matches from the referenced rule into our tree here.
|
||||
rule_name = name
|
||||
rule = rules[rule_name]
|
||||
rule_matches = {address: result for (address, result) in capabilities[rule_name]}
|
||||
|
||||
if rule.is_subscope_rule():
|
||||
# for a subscope rule, fixup the node to be a scope node, rather than a match feature node.
|
||||
#
|
||||
# e.g. `contain loop/30c4c78e29bf4d54894fc74f664c62e8` -> `basic block`
|
||||
#
|
||||
# note! replace `node`
|
||||
node = StatementNode(
|
||||
statement=SubscopeStatement(
|
||||
scope=rule.meta["scope"],
|
||||
)
|
||||
)
|
||||
|
||||
for location in result.locations:
|
||||
children.append(Match.from_capa(rules, capabilities, rule_matches[location]))
|
||||
else:
|
||||
# this is a namespace that we're matching
|
||||
#
|
||||
# check for all rules in the namespace,
|
||||
# seeing if they matched.
|
||||
# if so, pull their matches into our match tree here.
|
||||
ns_name = name
|
||||
ns_rules = rules.rules_by_namespace[ns_name]
|
||||
|
||||
for rule in ns_rules:
|
||||
if rule.name in capabilities:
|
||||
# the rule matched, so splice results into our tree here.
|
||||
#
|
||||
# note, there's a shortcoming in our result document schema here:
|
||||
# we lose the name of the rule that matched in a namespace.
|
||||
# for example, if we have a statement: `match: runtime/dotnet`
|
||||
# and we get matches, we can say the following:
|
||||
#
|
||||
# match: runtime/dotnet @ 0x0
|
||||
# or:
|
||||
# import: mscoree._CorExeMain @ 0x402000
|
||||
#
|
||||
# however, we lose the fact that it was rule
|
||||
# "compiled to the .NET platform"
|
||||
# that contained this logic and did the match.
|
||||
#
|
||||
# we could introduce an intermediate node here.
|
||||
# this would be a breaking change and require updates to the renderers.
|
||||
# in the meantime, the above might be sufficient.
|
||||
rule_matches = {address: result for (address, result) in capabilities[rule.name]}
|
||||
for location in result.locations:
|
||||
# doc[locations] contains all matches for the given namespace.
|
||||
# for example, the feature might be `match: anti-analysis/packer`
|
||||
# which matches against "generic unpacker" and "UPX".
|
||||
# in this case, doc[locations] contains locations for *both* of thse.
|
||||
#
|
||||
# rule_matches contains the matches for the specific rule.
|
||||
# this is a subset of doc[locations].
|
||||
#
|
||||
# so, grab only the locations for current rule.
|
||||
if location in rule_matches:
|
||||
children.append(Match.from_capa(rules, capabilities, rule_matches[location]))
|
||||
|
||||
return cls(
|
||||
success=success,
|
||||
node=node,
|
||||
children=children,
|
||||
locations=locations,
|
||||
captures=captures,
|
||||
)
|
||||
|
||||
|
||||
def parse_parts_id(s: str):
|
||||
id = ""
|
||||
parts = s.split("::")
|
||||
if len(parts) > 0:
|
||||
last = parts.pop()
|
||||
last, _, id = last.rpartition(" ")
|
||||
id = id.lstrip("[").rstrip("]")
|
||||
parts.append(last)
|
||||
return parts, id
|
||||
|
||||
|
||||
class AttackSpec(FrozenModel):
|
||||
"""
|
||||
given an ATT&CK spec like: `Tactic::Technique::Subtechnique [Identifier]`
|
||||
e.g., `Execution::Command and Scripting Interpreter::Python [T1059.006]`
|
||||
|
||||
args:
|
||||
tactic: like `Tactic` above, perhaps "Execution"
|
||||
technique: like `Technique` above, perhaps "Command and Scripting Interpreter"
|
||||
subtechnique: like `Subtechnique` above, perhaps "Python"
|
||||
id: like `Identifier` above, perhaps "T1059.006"
|
||||
"""
|
||||
|
||||
parts: Tuple[str, ...]
|
||||
tactic: str
|
||||
technique: str
|
||||
subtechnique: str
|
||||
id: str
|
||||
|
||||
@classmethod
|
||||
def from_str(cls, s) -> "AttackSpec":
|
||||
tactic = ""
|
||||
technique = ""
|
||||
subtechnique = ""
|
||||
parts, id = parse_parts_id(s)
|
||||
if len(parts) > 0:
|
||||
tactic = parts[0]
|
||||
if len(parts) > 1:
|
||||
technique = parts[1]
|
||||
if len(parts) > 2:
|
||||
subtechnique = parts[2]
|
||||
|
||||
return cls(
|
||||
parts=parts,
|
||||
tactic=tactic,
|
||||
technique=technique,
|
||||
subtechnique=subtechnique,
|
||||
id=id,
|
||||
)
|
||||
|
||||
|
||||
class MBCSpec(FrozenModel):
|
||||
"""
|
||||
given an MBC spec like: `Objective::Behavior::Method [Identifier]`
|
||||
e.g., `Collection::Input Capture::Mouse Events [E1056.m01]`
|
||||
|
||||
args:
|
||||
objective: like `Objective` above, perhaps "Collection"
|
||||
behavior: like `Behavior` above, perhaps "Input Capture"
|
||||
method: like `Method` above, perhaps "Mouse Events"
|
||||
id: like `Identifier` above, perhaps "E1056.m01"
|
||||
"""
|
||||
|
||||
parts: Tuple[str, ...]
|
||||
objective: str
|
||||
behavior: str
|
||||
method: str
|
||||
id: str
|
||||
|
||||
@classmethod
|
||||
def from_str(cls, s) -> "MBCSpec":
|
||||
objective = ""
|
||||
behavior = ""
|
||||
method = ""
|
||||
parts, id = parse_parts_id(s)
|
||||
if len(parts) > 0:
|
||||
objective = parts[0]
|
||||
if len(parts) > 1:
|
||||
behavior = parts[1]
|
||||
if len(parts) > 2:
|
||||
method = parts[2]
|
||||
|
||||
return cls(
|
||||
parts=parts,
|
||||
objective=objective,
|
||||
behavior=behavior,
|
||||
method=method,
|
||||
id=id,
|
||||
)
|
||||
|
||||
|
||||
class MaecMetadata(FrozenModel):
|
||||
analysis_conclusion: Optional[str] = Field(None, alias="analysis-conclusion")
|
||||
analysis_conclusion_ov: Optional[str] = Field(None, alias="analysis-conclusion-ov")
|
||||
malware_family: Optional[str] = Field(None, alias="malware-family")
|
||||
malware_category: Optional[str] = Field(None, alias="malware-category")
|
||||
malware_category_ov: Optional[str] = Field(None, alias="malware-category-ov")
|
||||
|
||||
class Config:
|
||||
frozen = True
|
||||
allow_population_by_field_name = True
|
||||
|
||||
|
||||
class RuleMetadata(FrozenModel):
|
||||
name: str
|
||||
namespace: Optional[str]
|
||||
authors: Tuple[str, ...]
|
||||
scope: capa.rules.Scope
|
||||
attack: Tuple[AttackSpec, ...] = Field(alias="att&ck")
|
||||
mbc: Tuple[MBCSpec, ...]
|
||||
references: Tuple[str, ...]
|
||||
examples: Tuple[str, ...]
|
||||
description: str
|
||||
|
||||
lib: bool = Field(False, alias="lib")
|
||||
is_subscope_rule: bool = Field(False, alias="capa/subscope")
|
||||
maec: MaecMetadata
|
||||
|
||||
@classmethod
|
||||
def from_capa(cls, rule: capa.rules.Rule) -> "RuleMetadata":
|
||||
return cls(
|
||||
name=rule.meta.get("name"),
|
||||
namespace=rule.meta.get("namespace"),
|
||||
authors=rule.meta.get("authors"),
|
||||
scope=capa.rules.Scope(rule.meta.get("scope")),
|
||||
attack=list(map(AttackSpec.from_str, rule.meta.get("att&ck", []))),
|
||||
mbc=list(map(MBCSpec.from_str, rule.meta.get("mbc", []))),
|
||||
references=rule.meta.get("references", []),
|
||||
examples=rule.meta.get("examples", []),
|
||||
description=rule.meta.get("description", ""),
|
||||
lib=rule.meta.get("lib", False),
|
||||
capa_subscope=rule.meta.get("capa/subscope", False),
|
||||
maec=MaecMetadata(
|
||||
analysis_conclusion=rule.meta.get("maec/analysis-conclusion"),
|
||||
analysis_conclusion_ov=rule.meta.get("maec/analysis-conclusion-ov"),
|
||||
malware_family=rule.meta.get("maec/malware-family"),
|
||||
malware_category=rule.meta.get("maec/malware-category"),
|
||||
malware_category_ov=rule.meta.get("maec/malware-category-ov"),
|
||||
),
|
||||
)
|
||||
|
||||
class Config:
|
||||
frozen = True
|
||||
allow_population_by_field_name = True
|
||||
|
||||
|
||||
class RuleMatches(BaseModel):
|
||||
"""
|
||||
args:
|
||||
meta: the metadata from the rule
|
||||
source: the raw rule text
|
||||
"""
|
||||
|
||||
meta: RuleMetadata
|
||||
source: str
|
||||
matches: Tuple[Tuple[frz.Address, Match], ...]
|
||||
|
||||
|
||||
class ResultDocument(BaseModel):
|
||||
meta: Metadata
|
||||
rules: Dict[str, RuleMatches]
|
||||
|
||||
@classmethod
|
||||
def from_capa(cls, meta, rules: RuleSet, capabilities: MatchResults) -> "ResultDocument":
|
||||
rule_matches: Dict[str, RuleMatches] = {}
|
||||
for rule_name, matches in capabilities.items():
|
||||
rule = rules[rule_name]
|
||||
|
||||
if rule.meta.get("capa/subscope-rule"):
|
||||
continue
|
||||
|
||||
rule_matches[rule_name] = RuleMatches(
|
||||
meta=RuleMetadata.from_capa(rule),
|
||||
source=rule.definition,
|
||||
matches=[
|
||||
(frz.Address.from_capa(addr), Match.from_capa(rules, capabilities, match))
|
||||
for addr, match in matches
|
||||
],
|
||||
)
|
||||
|
||||
return ResultDocument(meta=Metadata.from_capa(meta), rules=rule_matches)
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,25 +6,21 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
import io
|
||||
from typing import Union, Iterator
|
||||
|
||||
import six
|
||||
import termcolor
|
||||
|
||||
import capa.render.result_document as rd
|
||||
|
||||
|
||||
def bold(s: str) -> str:
|
||||
def bold(s):
|
||||
"""draw attention to the given string"""
|
||||
return termcolor.colored(s, "blue")
|
||||
|
||||
|
||||
def bold2(s: str) -> str:
|
||||
def bold2(s):
|
||||
"""draw attention to the given string, within a `bold` section"""
|
||||
return termcolor.colored(s, "green")
|
||||
|
||||
|
||||
def hex(n: int) -> str:
|
||||
def hex(n):
|
||||
"""render the given number using upper case hex, like: 0x123ABC"""
|
||||
if n < 0:
|
||||
return "-0x%X" % (-n)
|
||||
@@ -32,35 +28,28 @@ def hex(n: int) -> str:
|
||||
return "0x%X" % n
|
||||
|
||||
|
||||
def format_parts_id(data: Union[rd.AttackSpec, rd.MBCSpec]):
|
||||
"""
|
||||
format canonical representation of ATT&CK/MBC parts and ID
|
||||
"""
|
||||
return "%s [%s]" % ("::".join(data.parts), data.id)
|
||||
|
||||
|
||||
def capability_rules(doc: rd.ResultDocument) -> Iterator[rd.RuleMatches]:
|
||||
def capability_rules(doc):
|
||||
"""enumerate the rules in (namespace, name) order that are 'capability' rules (not lib/subscope/disposition/etc)."""
|
||||
for (_, _, rule) in sorted(map(lambda rule: (rule.meta.namespace or "", rule.meta.name, rule), doc.rules.values())):
|
||||
if rule.meta.lib:
|
||||
for (_, _, rule) in sorted(
|
||||
map(lambda rule: (rule["meta"].get("namespace", ""), rule["meta"]["name"], rule), doc["rules"].values())
|
||||
):
|
||||
if rule["meta"].get("lib"):
|
||||
continue
|
||||
if rule.meta.is_subscope_rule:
|
||||
if rule["meta"].get("capa/subscope"):
|
||||
continue
|
||||
if rule.meta.maec.analysis_conclusion:
|
||||
if rule["meta"].get("maec/analysis-conclusion"):
|
||||
continue
|
||||
if rule.meta.maec.analysis_conclusion_ov:
|
||||
if rule["meta"].get("maec/analysis-conclusion-ov"):
|
||||
continue
|
||||
if rule.meta.maec.malware_family:
|
||||
if rule["meta"].get("maec/malware-category"):
|
||||
continue
|
||||
if rule.meta.maec.malware_category:
|
||||
continue
|
||||
if rule.meta.maec.malware_category_ov:
|
||||
if rule["meta"].get("maec/malware-category-ov"):
|
||||
continue
|
||||
|
||||
yield rule
|
||||
|
||||
|
||||
class StringIO(io.StringIO):
|
||||
class StringIO(six.StringIO):
|
||||
def writeln(self, s):
|
||||
self.write(s)
|
||||
self.write("\n")
|
||||
|
||||
@@ -3,7 +3,7 @@ example::
|
||||
|
||||
send data
|
||||
namespace communication
|
||||
author william.ballenthin@mandiant.com
|
||||
author william.ballenthin@fireeye.com
|
||||
description all known techniques for sending data to a potential C2 server
|
||||
scope function
|
||||
examples BFB9B5391A13D0AFD787E87AB90F14F5:0x13145D60
|
||||
@@ -14,7 +14,7 @@ example::
|
||||
0x10003415
|
||||
0x10003797
|
||||
|
||||
Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -23,39 +23,12 @@ Unless required by applicable law or agreed to in writing, software distributed
|
||||
See the License for the specific language governing permissions and limitations under the License.
|
||||
"""
|
||||
import tabulate
|
||||
import dnfile.mdtable
|
||||
import dncil.clr.token
|
||||
|
||||
import capa.rules
|
||||
import capa.render.utils as rutils
|
||||
import capa.features.freeze as frz
|
||||
import capa.render.result_document
|
||||
import capa.render.result_document as rd
|
||||
from capa.rules import RuleSet
|
||||
from capa.engine import MatchResults
|
||||
|
||||
|
||||
def format_address(address: frz.Address) -> str:
|
||||
if address.type == frz.AddressType.ABSOLUTE:
|
||||
return rutils.hex(address.value)
|
||||
elif address.type == frz.AddressType.RELATIVE:
|
||||
return f"base address+{rutils.hex(address.value)}"
|
||||
elif address.type == frz.AddressType.FILE:
|
||||
return f"file+{rutils.hex(address.value)}"
|
||||
elif address.type == frz.AddressType.DN_TOKEN:
|
||||
token = dncil.clr.token.Token(address.value)
|
||||
return f"token({rutils.hex(token.value)})"
|
||||
elif address.type == frz.AddressType.DN_TOKEN_OFFSET:
|
||||
token, offset = address.value
|
||||
token = dncil.clr.token.Token(token)
|
||||
return f"token({rutils.hex(token.value)})+{rutils.hex(offset)}"
|
||||
elif address.type == frz.AddressType.NO_ADDRESS:
|
||||
return "global"
|
||||
else:
|
||||
raise ValueError("unexpected address type")
|
||||
|
||||
|
||||
def render_meta(ostream, doc: rd.ResultDocument):
|
||||
def render_meta(ostream, doc):
|
||||
"""
|
||||
like:
|
||||
|
||||
@@ -65,9 +38,7 @@ def render_meta(ostream, doc: rd.ResultDocument):
|
||||
path /tmp/suspicious.dll_
|
||||
timestamp 2020-07-03T10:17:05.796933
|
||||
capa version 0.0.0
|
||||
os windows
|
||||
format pe
|
||||
arch amd64
|
||||
format auto
|
||||
extractor VivisectFeatureExtractor
|
||||
base address 0x10000000
|
||||
rules (embedded rules)
|
||||
@@ -75,31 +46,27 @@ def render_meta(ostream, doc: rd.ResultDocument):
|
||||
total feature count 1918
|
||||
"""
|
||||
rows = [
|
||||
("md5", doc.meta.sample.md5),
|
||||
("sha1", doc.meta.sample.sha1),
|
||||
("sha256", doc.meta.sample.sha256),
|
||||
("path", doc.meta.sample.path),
|
||||
("timestamp", doc.meta.timestamp),
|
||||
("capa version", doc.meta.version),
|
||||
("os", doc.meta.analysis.os),
|
||||
("format", doc.meta.analysis.format),
|
||||
("arch", doc.meta.analysis.arch),
|
||||
("extractor", doc.meta.analysis.extractor),
|
||||
("base address", format_address(doc.meta.analysis.base_address)),
|
||||
("rules", "\n".join(doc.meta.analysis.rules)),
|
||||
("function count", len(doc.meta.analysis.feature_counts.functions)),
|
||||
("library function count", len(doc.meta.analysis.library_functions)),
|
||||
("md5", doc["meta"]["sample"]["md5"]),
|
||||
("sha1", doc["meta"]["sample"]["sha1"]),
|
||||
("sha256", doc["meta"]["sample"]["sha256"]),
|
||||
("path", doc["meta"]["sample"]["path"]),
|
||||
("timestamp", doc["meta"]["timestamp"]),
|
||||
("capa version", doc["meta"]["version"]),
|
||||
("format", doc["meta"]["analysis"]["format"]),
|
||||
("extractor", doc["meta"]["analysis"]["extractor"]),
|
||||
("base address", hex(doc["meta"]["analysis"]["base_address"])),
|
||||
("rules", doc["meta"]["analysis"]["rules"]),
|
||||
("function count", len(doc["meta"]["analysis"]["feature_counts"]["functions"])),
|
||||
(
|
||||
"total feature count",
|
||||
doc.meta.analysis.feature_counts.file
|
||||
+ sum(map(lambda f: f.count, doc.meta.analysis.feature_counts.functions)),
|
||||
doc["meta"]["analysis"]["feature_counts"]["file"]
|
||||
+ sum(doc["meta"]["analysis"]["feature_counts"]["functions"].values()),
|
||||
),
|
||||
]
|
||||
|
||||
ostream.writeln(tabulate.tabulate(rows, tablefmt="plain"))
|
||||
|
||||
|
||||
def render_rules(ostream, doc: rd.ResultDocument):
|
||||
def render_rules(ostream, doc):
|
||||
"""
|
||||
like:
|
||||
|
||||
@@ -112,29 +79,28 @@ def render_rules(ostream, doc: rd.ResultDocument):
|
||||
"""
|
||||
had_match = False
|
||||
for rule in rutils.capability_rules(doc):
|
||||
count = len(rule.matches)
|
||||
count = len(rule["matches"])
|
||||
if count == 1:
|
||||
capability = rutils.bold(rule.meta.name)
|
||||
capability = rutils.bold(rule["meta"]["name"])
|
||||
else:
|
||||
capability = "%s (%d matches)" % (rutils.bold(rule.meta.name), count)
|
||||
capability = "%s (%d matches)" % (rutils.bold(rule["meta"]["name"]), count)
|
||||
|
||||
ostream.writeln(capability)
|
||||
had_match = True
|
||||
|
||||
rows = []
|
||||
for key in ("namespace", "description", "scope"):
|
||||
v = getattr(rule.meta, key)
|
||||
if not v:
|
||||
if key == "name" or key not in rule["meta"]:
|
||||
continue
|
||||
|
||||
v = rule["meta"][key]
|
||||
if isinstance(v, list) and len(v) == 1:
|
||||
v = v[0]
|
||||
|
||||
rows.append((key, v))
|
||||
|
||||
if rule.meta.scope != capa.rules.FILE_SCOPE:
|
||||
locations = list(map(lambda m: m[0], doc.rules[rule.meta.name].matches))
|
||||
rows.append(("matches", "\n".join(map(format_address, locations))))
|
||||
if rule["meta"]["scope"] != capa.rules.FILE_SCOPE:
|
||||
locations = doc["rules"][rule["meta"]["name"]]["matches"].keys()
|
||||
rows.append(("matches", "\n".join(map(rutils.hex, locations))))
|
||||
|
||||
ostream.writeln(tabulate.tabulate(rows, tablefmt="plain"))
|
||||
ostream.write("\n")
|
||||
@@ -143,7 +109,7 @@ def render_rules(ostream, doc: rd.ResultDocument):
|
||||
ostream.writeln(rutils.bold("no capabilities found"))
|
||||
|
||||
|
||||
def render_verbose(doc: rd.ResultDocument):
|
||||
def render_verbose(doc):
|
||||
ostream = rutils.StringIO()
|
||||
|
||||
render_meta(ostream, doc)
|
||||
@@ -153,7 +119,3 @@ def render_verbose(doc: rd.ResultDocument):
|
||||
ostream.write("\n")
|
||||
|
||||
return ostream.getvalue()
|
||||
|
||||
|
||||
def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
|
||||
return render_verbose(rd.ResultDocument.from_capa(meta, rules, capabilities))
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
|
||||
# Copyright (C) 2020 FireEye, Inc. All Rights Reserved.
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at: [package root]/LICENSE.txt
|
||||
@@ -6,182 +6,116 @@
|
||||
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and limitations under the License.
|
||||
|
||||
from typing import Dict, List, Iterable
|
||||
import collections
|
||||
|
||||
import tabulate
|
||||
|
||||
import capa.rules
|
||||
import capa.render.utils as rutils
|
||||
import capa.render.verbose
|
||||
import capa.features.common
|
||||
import capa.features.freeze as frz
|
||||
import capa.features.address
|
||||
import capa.render.result_document as rd
|
||||
import capa.features.freeze.features as frzf
|
||||
from capa.rules import RuleSet
|
||||
from capa.engine import MatchResults
|
||||
|
||||
|
||||
def render_locations(ostream, locations: Iterable[frz.Address]):
|
||||
import capa.render.verbose as v
|
||||
|
||||
def render_locations(ostream, match):
|
||||
# its possible to have an empty locations array here,
|
||||
# such as when we're in MODE_FAILURE and showing the logic
|
||||
# under a `not` statement (which will have no matched locations).
|
||||
locations = list(sorted(locations))
|
||||
|
||||
if len(locations) == 0:
|
||||
return
|
||||
|
||||
ostream.write(" @ ")
|
||||
|
||||
locations = list(sorted(match.get("locations", [])))
|
||||
if len(locations) == 1:
|
||||
ostream.write(v.format_address(locations[0]))
|
||||
|
||||
elif len(locations) > 4:
|
||||
# don't display too many locations, because it becomes very noisy.
|
||||
# probably only the first handful of locations will be useful for inspection.
|
||||
ostream.write(", ".join(map(v.format_address, locations[0:4])))
|
||||
ostream.write(", and %d more..." % (len(locations) - 4))
|
||||
|
||||
ostream.write(" @ ")
|
||||
ostream.write(rutils.hex(locations[0]))
|
||||
elif len(locations) > 1:
|
||||
ostream.write(", ".join(map(v.format_address, locations)))
|
||||
|
||||
else:
|
||||
raise RuntimeError("unreachable")
|
||||
ostream.write(" @ ")
|
||||
if len(locations) > 4:
|
||||
# don't display too many locations, because it becomes very noisy.
|
||||
# probably only the first handful of locations will be useful for inspection.
|
||||
ostream.write(", ".join(map(rutils.hex, locations[0:4])))
|
||||
ostream.write(", and %d more..." % (len(locations) - 4))
|
||||
else:
|
||||
ostream.write(", ".join(map(rutils.hex, locations)))
|
||||
|
||||
|
||||
def render_statement(ostream, match: rd.Match, statement: rd.Statement, indent=0):
|
||||
def render_statement(ostream, match, statement, indent=0):
|
||||
ostream.write(" " * indent)
|
||||
|
||||
if isinstance(statement, rd.SubscopeStatement):
|
||||
# emit `basic block:`
|
||||
# rather than `subscope:`
|
||||
ostream.write(statement.scope)
|
||||
|
||||
if statement["type"] in ("and", "or", "optional", "not", "subscope"):
|
||||
ostream.write(statement["type"])
|
||||
ostream.write(":")
|
||||
if statement.description:
|
||||
ostream.write(" = %s" % statement.description)
|
||||
if statement.get("description"):
|
||||
ostream.write(" = %s" % statement["description"])
|
||||
ostream.writeln("")
|
||||
|
||||
elif isinstance(statement, (rd.AndStatement, rd.OrStatement, rd.OptionalStatement, rd.NotStatement)):
|
||||
# emit `and:` `or:` `optional:` `not:`
|
||||
ostream.write(statement.type)
|
||||
|
||||
ostream.write(":")
|
||||
if statement.description:
|
||||
ostream.write(" = %s" % statement.description)
|
||||
elif statement["type"] == "some":
|
||||
ostream.write("%d or more:" % (statement["count"]))
|
||||
if statement.get("description"):
|
||||
ostream.write(" = %s" % statement["description"])
|
||||
ostream.writeln("")
|
||||
|
||||
elif isinstance(statement, rd.SomeStatement):
|
||||
ostream.write("%d or more:" % (statement.count))
|
||||
|
||||
if statement.description:
|
||||
ostream.write(" = %s" % statement.description)
|
||||
ostream.writeln("")
|
||||
|
||||
elif isinstance(statement, rd.RangeStatement):
|
||||
elif statement["type"] == "range":
|
||||
# `range` is a weird node, its almost a hybrid of statement+feature.
|
||||
# it is a specific feature repeated multiple times.
|
||||
# there's no additional logic in the feature part, just the existence of a feature.
|
||||
# so, we have to inline some of the feature rendering here.
|
||||
|
||||
child = statement.child
|
||||
value = getattr(child, child.type)
|
||||
child = statement["child"]
|
||||
|
||||
if value:
|
||||
if isinstance(child, frzf.StringFeature):
|
||||
value = '"%s"' % capa.features.common.escape_string(value)
|
||||
|
||||
value = rutils.bold2(value)
|
||||
|
||||
if child.description:
|
||||
ostream.write("count(%s(%s = %s)): " % (child.type, value, child.description))
|
||||
if child[child["type"]]:
|
||||
if child["type"] == "string":
|
||||
value = '"%s"' % capa.features.escape_string(child[child["type"]])
|
||||
else:
|
||||
ostream.write("count(%s(%s)): " % (child.type, value))
|
||||
value = child[child["type"]]
|
||||
value = rutils.bold2(value)
|
||||
if child.get("description"):
|
||||
ostream.write("count(%s(%s = %s)): " % (child["type"], value, child["description"]))
|
||||
else:
|
||||
ostream.write("count(%s(%s)): " % (child["type"], value))
|
||||
else:
|
||||
ostream.write("count(%s): " % child.type)
|
||||
ostream.write("count(%s): " % child["type"])
|
||||
|
||||
if statement.max == statement.min:
|
||||
ostream.write("%d" % (statement.min))
|
||||
elif statement.min == 0:
|
||||
ostream.write("%d or fewer" % (statement.max))
|
||||
elif statement.max == (1 << 64 - 1):
|
||||
ostream.write("%d or more" % (statement.min))
|
||||
if statement["max"] == statement["min"]:
|
||||
ostream.write("%d" % (statement["min"]))
|
||||
elif statement["min"] == 0:
|
||||
ostream.write("%d or fewer" % (statement["max"]))
|
||||
elif statement["max"] == (1 << 64 - 1):
|
||||
ostream.write("%d or more" % (statement["min"]))
|
||||
else:
|
||||
ostream.write("between %d and %d" % (statement.min, statement.max))
|
||||
ostream.write("between %d and %d" % (statement["min"], statement["max"]))
|
||||
|
||||
if statement.description:
|
||||
ostream.write(" = %s" % statement.description)
|
||||
render_locations(ostream, match.locations)
|
||||
if statement.get("description"):
|
||||
ostream.write(" = %s" % statement["description"])
|
||||
render_locations(ostream, match)
|
||||
ostream.writeln("")
|
||||
|
||||
else:
|
||||
raise RuntimeError("unexpected match statement type: " + str(statement))
|
||||
|
||||
|
||||
def render_string_value(s: str) -> str:
|
||||
return '"%s"' % capa.features.common.escape_string(s)
|
||||
|
||||
|
||||
def render_feature(ostream, match: rd.Match, feature: frzf.Feature, indent=0):
|
||||
def render_feature(ostream, match, feature, indent=0):
|
||||
ostream.write(" " * indent)
|
||||
|
||||
key = feature.type
|
||||
if isinstance(feature, frzf.ImportFeature):
|
||||
# fixup access to Python reserved name
|
||||
value = feature.import_
|
||||
if isinstance(feature, frzf.ClassFeature):
|
||||
value = feature.class_
|
||||
else:
|
||||
value = getattr(feature, key)
|
||||
key = feature["type"]
|
||||
value = feature[feature["type"]]
|
||||
if key == "regex":
|
||||
key = "string" # render string for regex to mirror the rule source
|
||||
value = feature["match"] # the match provides more information than the value for regex
|
||||
|
||||
if key not in ("regex", "substring"):
|
||||
# like:
|
||||
# number: 10 = SOME_CONSTANT @ 0x401000
|
||||
if key == "string":
|
||||
value = render_string_value(value)
|
||||
if key == "string":
|
||||
value = '"%s"' % capa.features.escape_string(value)
|
||||
|
||||
if key == "number":
|
||||
assert isinstance(value, int)
|
||||
value = hex(value)
|
||||
ostream.write(key)
|
||||
ostream.write(": ")
|
||||
|
||||
ostream.write(key)
|
||||
ostream.write(": ")
|
||||
if value:
|
||||
ostream.write(rutils.bold2(value))
|
||||
|
||||
if value:
|
||||
ostream.write(rutils.bold2(value))
|
||||
if "description" in feature:
|
||||
ostream.write(capa.rules.DESCRIPTION_SEPARATOR)
|
||||
ostream.write(feature["description"])
|
||||
|
||||
if feature.description:
|
||||
ostream.write(capa.rules.DESCRIPTION_SEPARATOR)
|
||||
ostream.write(feature.description)
|
||||
|
||||
if key not in ("os", "arch"):
|
||||
render_locations(ostream, match.locations)
|
||||
ostream.write("\n")
|
||||
else:
|
||||
# like:
|
||||
# regex: /blah/ = SOME_CONSTANT
|
||||
# - "foo blah baz" @ 0x401000
|
||||
# - "aaa blah bbb" @ 0x402000, 0x403400
|
||||
ostream.write(key)
|
||||
ostream.write(": ")
|
||||
ostream.write(value)
|
||||
ostream.write("\n")
|
||||
|
||||
for capture, locations in sorted(match.captures.items()):
|
||||
ostream.write(" " * (indent + 1))
|
||||
ostream.write("- ")
|
||||
ostream.write(rutils.bold2(render_string_value(capture)))
|
||||
render_locations(ostream, locations)
|
||||
ostream.write("\n")
|
||||
render_locations(ostream, match)
|
||||
ostream.write("\n")
|
||||
|
||||
|
||||
def render_node(ostream, match: rd.Match, node: rd.Node, indent=0):
|
||||
if isinstance(node, rd.StatementNode):
|
||||
render_statement(ostream, match, node.statement, indent=indent)
|
||||
elif isinstance(node, rd.FeatureNode):
|
||||
render_feature(ostream, match, node.feature, indent=indent)
|
||||
def render_node(ostream, match, node, indent=0):
|
||||
if node["type"] == "statement":
|
||||
render_statement(ostream, match, node["statement"], indent=indent)
|
||||
elif node["type"] == "feature":
|
||||
render_feature(ostream, match, node["feature"], indent=indent)
|
||||
else:
|
||||
raise RuntimeError("unexpected node type: " + str(node))
|
||||
|
||||
@@ -194,147 +128,97 @@ MODE_SUCCESS = "success"
|
||||
MODE_FAILURE = "failure"
|
||||
|
||||
|
||||
def render_match(ostream, match: rd.Match, indent=0, mode=MODE_SUCCESS):
|
||||
def render_match(ostream, match, indent=0, mode=MODE_SUCCESS):
|
||||
child_mode = mode
|
||||
if mode == MODE_SUCCESS:
|
||||
# display only nodes that evaluated successfully.
|
||||
if not match.success:
|
||||
if not match["success"]:
|
||||
return
|
||||
|
||||
# optional statement with no successful children is empty
|
||||
if isinstance(match.node, rd.StatementNode) and isinstance(match.node.statement, rd.OptionalStatement):
|
||||
if not any(map(lambda m: m.success, match.children)):
|
||||
return
|
||||
|
||||
if match["node"].get("statement", {}).get("type") == "optional" and not any(
|
||||
map(lambda m: m["success"], match["children"])
|
||||
):
|
||||
return
|
||||
# not statement, so invert the child mode to show failed evaluations
|
||||
if isinstance(match.node, rd.StatementNode) and isinstance(match.node.statement, rd.NotStatement):
|
||||
if match["node"].get("statement", {}).get("type") == "not":
|
||||
child_mode = MODE_FAILURE
|
||||
|
||||
elif mode == MODE_FAILURE:
|
||||
# display only nodes that did not evaluate to True
|
||||
if match.success:
|
||||
if match["success"]:
|
||||
return
|
||||
|
||||
# optional statement with successful children is not relevant
|
||||
if isinstance(match.node, rd.StatementNode) and isinstance(match.node.statement, rd.OptionalStatement):
|
||||
if any(map(lambda m: m.success, match.children)):
|
||||
return
|
||||
|
||||
if match["node"].get("statement", {}).get("type") == "optional" and any(
|
||||
map(lambda m: m["success"], match["children"])
|
||||
):
|
||||
return
|
||||
# not statement, so invert the child mode to show successful evaluations
|
||||
if isinstance(match.node, rd.StatementNode) and isinstance(match.node.statement, rd.NotStatement):
|
||||
if match["node"].get("statement", {}).get("type") == "not":
|
||||
child_mode = MODE_SUCCESS
|
||||
else:
|
||||
raise RuntimeError("unexpected mode: " + mode)
|
||||
|
||||
render_node(ostream, match, match.node, indent=indent)
|
||||
render_node(ostream, match, match["node"], indent=indent)
|
||||
|
||||
for child in match.children:
|
||||
for child in match["children"]:
|
||||
render_match(ostream, child, indent=indent + 1, mode=child_mode)
|
||||
|
||||
|
||||
def render_rules(ostream, doc: rd.ResultDocument):
|
||||
def render_rules(ostream, doc):
|
||||
"""
|
||||
like:
|
||||
|
||||
## rules
|
||||
check for OutputDebugString error
|
||||
namespace anti-analysis/anti-debugging/debugger-detection
|
||||
author michael.hunhoff@mandiant.com
|
||||
author michael.hunhoff@fireeye.com
|
||||
scope function
|
||||
mbc Anti-Behavioral Analysis::Detect Debugger::OutputDebugString
|
||||
examples Practical Malware Analysis Lab 16-02.exe_:0x401020
|
||||
function @ 0x10004706
|
||||
and:
|
||||
api: kernel32.SetLastError @ 0x100047C2
|
||||
api: kernel32.GetLastError @ 0x10004A87
|
||||
api: kernel32.OutputDebugString @ 0x10004767, 0x10004787, 0x10004816, 0x10004895
|
||||
"""
|
||||
functions_by_bb: Dict[capa.features.address.Address, capa.features.address.Address] = {}
|
||||
for finfo in doc.meta.analysis.layout.functions:
|
||||
faddress = finfo.address.to_capa()
|
||||
|
||||
for bb in finfo.matched_basic_blocks:
|
||||
bbaddress = bb.address.to_capa()
|
||||
functions_by_bb[bbaddress] = faddress
|
||||
|
||||
had_match = False
|
||||
|
||||
for (_, _, rule) in sorted(map(lambda rule: (rule.meta.namespace or "", rule.meta.name, rule), doc.rules.values())):
|
||||
# default scope hides things like lib rules, malware-category rules, etc.
|
||||
# but in vverbose mode, we really want to show everything.
|
||||
#
|
||||
# still ignore subscope rules because they're stitched into the final document.
|
||||
if rule.meta.is_subscope_rule:
|
||||
continue
|
||||
|
||||
count = len(rule.matches)
|
||||
for rule in rutils.capability_rules(doc):
|
||||
count = len(rule["matches"])
|
||||
if count == 1:
|
||||
capability = rutils.bold(rule.meta.name)
|
||||
capability = rutils.bold(rule["meta"]["name"])
|
||||
else:
|
||||
capability = "%s (%d matches)" % (rutils.bold(rule.meta.name), count)
|
||||
capability = "%s (%d matches)" % (rutils.bold(rule["meta"]["name"]), count)
|
||||
|
||||
ostream.writeln(capability)
|
||||
had_match = True
|
||||
|
||||
rows = []
|
||||
rows.append(("namespace", rule.meta.namespace))
|
||||
for key in capa.rules.META_KEYS:
|
||||
if key == "name" or key not in rule["meta"]:
|
||||
continue
|
||||
|
||||
if rule.meta.maec.analysis_conclusion or rule.meta.maec.analysis_conclusion_ov:
|
||||
rows.append(
|
||||
(
|
||||
"maec/analysis-conclusion",
|
||||
rule.meta.maec.analysis_conclusion or rule.meta.maec.analysis_conclusion_ov,
|
||||
)
|
||||
)
|
||||
|
||||
if rule.meta.maec.malware_family:
|
||||
rows.append(("maec/malware-family", rule.meta.maec.malware_family))
|
||||
|
||||
if rule.meta.maec.malware_category or rule.meta.maec.malware_category:
|
||||
rows.append(
|
||||
("maec/malware-category", rule.meta.maec.malware_category or rule.meta.maec.malware_category_ov)
|
||||
)
|
||||
|
||||
rows.append(("author", ", ".join(rule.meta.authors)))
|
||||
|
||||
rows.append(("scope", rule.meta.scope.value))
|
||||
|
||||
if rule.meta.attack:
|
||||
rows.append(("att&ck", ", ".join([rutils.format_parts_id(v) for v in rule.meta.attack])))
|
||||
|
||||
if rule.meta.mbc:
|
||||
rows.append(("mbc", ", ".join([rutils.format_parts_id(v) for v in rule.meta.mbc])))
|
||||
|
||||
if rule.meta.references:
|
||||
rows.append(("references", ", ".join(rule.meta.references)))
|
||||
|
||||
if rule.meta.description:
|
||||
rows.append(("description", rule.meta.description))
|
||||
v = rule["meta"][key]
|
||||
if isinstance(v, list) and len(v) == 1:
|
||||
v = v[0]
|
||||
elif isinstance(v, list) and len(v) > 1:
|
||||
v = ", ".join(v)
|
||||
rows.append((key, v))
|
||||
|
||||
ostream.writeln(tabulate.tabulate(rows, tablefmt="plain"))
|
||||
|
||||
if rule.meta.scope == capa.rules.FILE_SCOPE:
|
||||
matches = doc.rules[rule.meta.name].matches
|
||||
if rule["meta"]["scope"] == capa.rules.FILE_SCOPE:
|
||||
matches = list(doc["rules"][rule["meta"]["name"]]["matches"].values())
|
||||
if len(matches) != 1:
|
||||
# i think there should only ever be one match per file-scope rule,
|
||||
# because we do the file-scope evaluation a single time.
|
||||
# but i'm not 100% sure if this is/will always be true.
|
||||
# so, lets be explicit about our assumptions and raise an exception if they fail.
|
||||
raise RuntimeError("unexpected file scope match count: %d" % (len(matches)))
|
||||
first_address, first_match = matches[0]
|
||||
render_match(ostream, first_match, indent=0)
|
||||
render_match(ostream, matches[0], indent=0)
|
||||
else:
|
||||
for location, match in sorted(doc.rules[rule.meta.name].matches):
|
||||
ostream.write(rule.meta.scope)
|
||||
for location, match in sorted(doc["rules"][rule["meta"]["name"]]["matches"].items()):
|
||||
ostream.write(rule["meta"]["scope"])
|
||||
ostream.write(" @ ")
|
||||
ostream.write(capa.render.verbose.format_address(location))
|
||||
|
||||
if rule.meta.scope == capa.rules.BASIC_BLOCK_SCOPE:
|
||||
ostream.write(
|
||||
" in function "
|
||||
+ capa.render.verbose.format_address(frz.Address.from_capa(functions_by_bb[location.to_capa()]))
|
||||
)
|
||||
|
||||
ostream.write("\n")
|
||||
ostream.writeln(rutils.hex(location))
|
||||
render_match(ostream, match, indent=1)
|
||||
ostream.write("\n")
|
||||
|
||||
@@ -342,7 +226,7 @@ def render_rules(ostream, doc: rd.ResultDocument):
|
||||
ostream.writeln(rutils.bold("no capabilities found"))
|
||||
|
||||
|
||||
def render_vverbose(doc: rd.ResultDocument):
|
||||
def render_vverbose(doc):
|
||||
ostream = rutils.StringIO()
|
||||
|
||||
capa.render.verbose.render_meta(ostream, doc)
|
||||
@@ -352,7 +236,3 @@ def render_vverbose(doc: rd.ResultDocument):
|
||||
ostream.write("\n")
|
||||
|
||||
return ostream.getvalue()
|
||||
|
||||
|
||||
def render(meta, rules: RuleSet, capabilities: MatchResults) -> str:
|
||||
return render_vverbose(rd.ResultDocument.from_capa(meta, rules, capabilities))
|
||||
|
||||
755
capa/rules.py
755
capa/rules.py
File diff suppressed because it is too large
Load Diff
@@ -1,13 +1 @@
|
||||
__version__ = "4.0.1"
|
||||
|
||||
|
||||
def get_major_version():
|
||||
return int(__version__.partition(".")[0])
|
||||
|
||||
|
||||
def get_rules_branch():
|
||||
return f"v{get_major_version()}"
|
||||
|
||||
|
||||
def get_rules_checkout_command():
|
||||
return f"$ git clone https://github.com/mandiant/capa-rules.git -b {get_rules_branch()} /local/path/to/rules"
|
||||
__version__ = "1.6.3"
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 56 KiB |
@@ -1,8 +1,8 @@
|
||||
# Installation
|
||||
You can install capa in a few different ways. First, if you simply want to use capa, just download the [standalone binary](https://github.com/mandiant/capa/releases). If you want to use capa as a Python library, you can install the package directly from GitHub using `pip`. If you'd like to contribute patches or features to capa, you can work with a local copy of the source code.
|
||||
You can install capa in a few different ways. First, if you simply want to use capa, just download the [standalone binary](https://github.com/fireeye/capa/releases). If you want to use capa as a Python library, you can install the package directly from GitHub using `pip`. If you'd like to contribute patches or features to capa, you can work with a local copy of the source code.
|
||||
|
||||
## Method 1: Standalone installation
|
||||
If you simply want to use capa, use the standalone binaries we host on GitHub: https://github.com/mandiant/capa/releases. These binary executable files contain all the source code, Python interpreter, and associated resources needed to make capa run. This means you can run it without any installation! Just invoke the file using your terminal shell to see the help documentation.
|
||||
If you simply want to use capa, use the standalone binaries we host on GitHub: https://github.com/fireeye/capa/releases. These binary executable files contain all the source code, Python interpreter, and associated resources needed to make capa run. This means you can run it without any installation! Just invoke the file using your terminal shell to see the help documentation.
|
||||
|
||||
We use PyInstaller to create these packages.
|
||||
|
||||
@@ -26,17 +26,7 @@ To install capa as a Python library use `pip` to fetch the `flare-capa` module.
|
||||
|
||||
#### *Note*:
|
||||
This method is appropriate for integrating capa in an existing project.
|
||||
This technique doesn't pull the default rule set, so you should check it out separately from [capa-rules](https://github.com/mandiant/capa-rules/) and pass the directory to the entrypoint using `-r` or set the rules path in the IDA Pro plugin:
|
||||
|
||||
```console
|
||||
$ git clone https://github.com/mandiant/capa-rules.git -b v3 /local/path/to/rules
|
||||
$ capa -r /local/path/to/rules suspicious.exe
|
||||
```
|
||||
|
||||
This technique also doesn't set up the default library identification [signatures](https://github.com/mandiant/capa/tree/master/sigs). You can pass the signature directory using the `-s` argument.
|
||||
For example, to run capa with both a rule path and a signature path:
|
||||
|
||||
capa -r /path/to/capa-rules -s /path/to/capa-sigs suspicious.exe
|
||||
This technique doesn't pull the default rule set, so you should check it out separately from [capa-rules](https://github.com/fireeye/capa-rules/) and pass the directory to the entrypoint using `-r` or set the rules path in the IDA Pro plugin.
|
||||
Alternatively, see Method 3 below.
|
||||
|
||||
### 1. Install capa module
|
||||
@@ -50,16 +40,15 @@ If you'd like to review and modify the capa source code, you'll need to check it
|
||||
|
||||
### 1. Check out source code
|
||||
Next, clone the capa git repository.
|
||||
We use submodules to separate [code](https://github.com/mandiant/capa), [rules](https://github.com/mandiant/capa-rules), and [test data](https://github.com/mandiant/capa-testfiles).
|
||||
We use submodules to separate [code](https://github.com/fireeye/capa), [rules](https://github.com/fireeye/capa-rules), and [test data](https://github.com/fireeye/capa-testfiles).
|
||||
To clone everything use the `--recurse-submodules` option:
|
||||
- CAUTION: The capa testfiles repository contains many malware samples. If you pull down everything using this method, you may want to install to a directory that won't trigger your anti-virus software.
|
||||
- `$ git clone --recurse-submodules https://github.com/mandiant/capa.git /local/path/to/src` (HTTPS)
|
||||
- `$ git clone --recurse-submodules git@github.com:mandiant/capa.git /local/path/to/src` (SSH)
|
||||
- `$ git clone --recurse-submodules https://github.com/fireeye/capa.git /local/path/to/src` (HTTPS)
|
||||
- `$ git clone --recurse-submodules git@github.com:fireeye/capa.git /local/path/to/src` (SSH)
|
||||
|
||||
To only get the source code and our provided rules (common), follow these steps:
|
||||
- clone repository
|
||||
- `$ git clone https://github.com/mandiant/capa.git /local/path/to/src` (HTTPS)
|
||||
- `$ git clone git@github.com:mandiant/capa.git /local/path/to/src` (SSH)
|
||||
- `$ git clone https://github.com/fireeye/capa.git /local/path/to/src` (HTTPS)
|
||||
- `$ git clone git@github.com:fireeye/capa.git /local/path/to/src` (SSH)
|
||||
- `$ cd /local/path/to/src`
|
||||
- `$ git submodule update --init rules`
|
||||
|
||||
@@ -70,57 +59,40 @@ Use `pip` to install the source code in "editable" mode. This means that Python
|
||||
|
||||
You'll find that the `capa.exe` (Windows) or `capa` (Linux/MacOS) executables in your path now invoke the capa binary from this directory.
|
||||
|
||||
#### Development
|
||||
|
||||
##### venv [optional]
|
||||
|
||||
For development, we recommend to use [venv](https://docs.python.org/3/tutorial/venv.html). It allows you to create a virtual environment: a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages. This approach avoids conflicts between the requirements of different applications on your computer. It also ensures that you don't overlook to add a new requirement to `setup.up` using a library already installed on your system.
|
||||
|
||||
To create an environment (in the parent directory, to avoid commiting it by accident or messing with the linters), run:
|
||||
`$ python3 -m venv ../capa-env`
|
||||
|
||||
To activate `capa-env` in Linux or MacOS, run:
|
||||
`$ source ../capa-env/bin/activate`
|
||||
|
||||
To activate `capa-env` in Windows, run:
|
||||
`$ ..\capa-env\Scripts\activate.bat`
|
||||
|
||||
For more details about creating and using virtual environments, check out the [venv documentation](https://docs.python.org/3/tutorial/venv.html).
|
||||
|
||||
##### Install development dependencies
|
||||
|
||||
We use the following tools to ensure consistent code style and formatting:
|
||||
- [black](https://github.com/psf/black) code formatter, with `-l 120`
|
||||
- [isort 5](https://pypi.org/project/isort/) code formatter, with `--profile black --length-sort --line-width 120`
|
||||
- [dos2unix](https://linux.die.net/man/1/dos2unix) for UNIX-style LF newlines
|
||||
- [capafmt](https://github.com/mandiant/capa/blob/master/scripts/capafmt.py) rule formatter
|
||||
- [capafmt](https://github.com/fireeye/capa/blob/master/scripts/capafmt.py) rule formatter
|
||||
|
||||
To install these development dependencies, run:
|
||||
|
||||
`$ pip install -e /local/path/to/src[dev]`
|
||||
|
||||
Note that some development dependencies (including the black code formatter) require Python 3.
|
||||
|
||||
To check the code style, formatting and run the tests you can run the script `scripts/ci.sh`.
|
||||
You can run it with the argument `no_tests` to skip the tests and only run the code style and formatting: `scripts/ci.sh no_tests`
|
||||
|
||||
##### Setup hooks [optional]
|
||||
|
||||
If you plan to contribute to capa, you may want to setup the hooks.
|
||||
Run `scripts/setup-hooks.sh` to set the following hooks up:
|
||||
- The `pre-commit` hook runs checks before every `git commit`.
|
||||
It runs `scripts/ci.sh no_tests` aborting the commit if there are code style or rule linter offenses you need to fix.
|
||||
- The `pre-push` hook runs checks before every `git push`.
|
||||
It runs `scripts/ci.sh` aborting the push if there are code style or rule linter offenses or if the tests fail.
|
||||
This way you can ensure everything is alright before sending a pull request.
|
||||
|
||||
You can skip the checks by using the `--no-verify` git option.
|
||||
|
||||
### 3. Compile binary using PyInstaller
|
||||
We compile capa standalone binaries using PyInstaller. To reproduce the build process check out the source code as described above and follow these steps.
|
||||
|
||||
#### Install PyInstaller:
|
||||
`$ pip install pyinstaller` (Python 3)
|
||||
For Python 2.7: `$ pip install 'pyinstaller==3.*'` (PyInstaller 4 doesn't support Python 2.7)
|
||||
|
||||
For Python 3: `$ pip install 'pyinstaller`
|
||||
|
||||
#### Run Pyinstaller
|
||||
`$ pyinstaller .github/pyinstaller/pyinstaller.spec`
|
||||
|
||||
You can find the compiled binary in the created directory `dist/`.
|
||||
|
||||
### 4. Setup hooks [optional]
|
||||
If you plan to contribute to capa, you may want to setup the hooks.
|
||||
Run `scripts/setup-hooks.sh` to set the following hooks up:
|
||||
- The `pre-commit` hook runs checks before every `git commit`.
|
||||
It runs `scripts/ci.sh no_tests` aborting the commit if there are code style or rule linter offenses you need to fix.
|
||||
You can skip this check by using the `--no-verify` git option.
|
||||
- The `pre-push` hook runs checks before every `git push`.
|
||||
It runs `scripts/ci.sh` aborting the push if there are code style or rule linter offenses or if the tests fail.
|
||||
This way you can ensure everything is alright before sending a pull request.
|
||||
|
||||
@@ -5,7 +5,7 @@ If capa detects that a program may be packed using its rules it warns the user.
|
||||
|
||||
|
||||
# Installers, run-time programs, etc.
|
||||
capa cannot handle installers, run-time programs, or other packaged applications like AutoIt well. This means that the results may be misleading or incomplete.
|
||||
capa cannot handle installers, run-time programs like .NET applications, or other packaged applications like AutoIt well. This means that the results may be misleading or incomplete.
|
||||
|
||||
If capa detects an installer, run-time program, etc. it warns the user.
|
||||
|
||||
@@ -46,6 +46,6 @@ We need more practical use cases and test samples to justify the additional work
|
||||
|
||||
|
||||
# ATT&CK, MAEC, MBC, and other capability tagging
|
||||
capa uses namespaces to group capabilities (see https://github.com/mandiant/capa-rules/tree/master#namespace-organization).
|
||||
capa uses namespaces to group capabilities (see https://github.com/fireeye/capa-rules/tree/master#namespace-organization).
|
||||
|
||||
The `rule.meta` field also supports `att&ck`, `mbc`, and `maec` fields to associate rules with the respective taxonomy (see https://github.com/mandiant/capa-rules/blob/master/doc/format.md#meta-block).
|
||||
The `rule.meta` field also supports `att&ck`, `mbc`, and `maec` fields to associate rules with the respective taxonomy (see https://github.com/fireeye/capa-rules/blob/master/doc/format.md#meta-block).
|
||||
|
||||
@@ -1,16 +1,22 @@
|
||||
# Release checklist
|
||||
|
||||
- [ ] Ensure all [milestoned issues/PRs](https://github.com/mandiant/capa/milestones) are addressed, or reassign to a new milestone.
|
||||
- [ ] Add the `dont merge` label to all PRs that are close to be ready to merge (or merge them if they are ready) in [capa](https://github.com/mandiant/capa/pulls) and [capa-rules](https://github.com/mandiant/capa-rules/pulls).
|
||||
- [ ] Ensure the [CI workflow succeeds in master](https://github.com/mandiant/capa/actions/workflows/tests.yml?query=branch%3Amaster).
|
||||
- [ ] Ensure all [milestoned issues/PRs](https://github.com/fireeye/capa/milestones) are addressed, or reassign to a new milestone.
|
||||
- [ ] Add the `dont merge` label to all PRs that are close to be ready to merge (or merge them if they are ready) in [capa](https://github.com/fireeye/capa/pulls) and [capa-rules](https://github.com/fireeye/capa-rules/pulls).
|
||||
- [ ] Ensure the [CI workflow succeeds in master](https://github.com/fireeye/capa/actions/workflows/tests.yml?query=branch%3Amaster).
|
||||
- [ ] Ensure that `python scripts/lint.py rules/ --thorough` succeeds (only `missing examples` offenses are allowed in the nursery).
|
||||
- [ ] Review changes
|
||||
- capa https://github.com/mandiant/capa/compare/\<last-release\>...master
|
||||
- capa-rules https://github.com/mandiant/capa-rules/compare/\<last-release>\...master
|
||||
- [ ] Update [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md)
|
||||
- capa https://github.com/fireeye/capa/compare/\<last-release\>...master
|
||||
- capa-rules https://github.com/fireeye/capa-rules/compare/\<last-release>\...master
|
||||
- [ ] Update [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md)
|
||||
- Do not forget to add a nice introduction thanking contributors
|
||||
- Remember that we need a major release if we introduce breaking changes
|
||||
- Sections: see template below
|
||||
- Sections
|
||||
- New Features
|
||||
- New Rules
|
||||
- Bug Fixes
|
||||
- Changes
|
||||
- Development
|
||||
- Raw diffs
|
||||
- Update `Raw diffs` links
|
||||
- Create placeholder for `master (unreleased)` section
|
||||
```
|
||||
@@ -18,32 +24,21 @@
|
||||
|
||||
### New Features
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
### New Rules (0)
|
||||
|
||||
-
|
||||
### New Rules
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
### capa explorer IDA Pro plugin
|
||||
### Changes
|
||||
|
||||
### Development
|
||||
|
||||
### Raw diffs
|
||||
- [capa <release>...master](https://github.com/mandiant/capa/compare/<release>...master)
|
||||
- [capa-rules <release>...master](https://github.com/mandiant/capa-rules/compare/<release>...master)
|
||||
- [capa <release>...master](https://github.com/fireeye/capa/compare/<release>...master)
|
||||
- [capa-rules <release>...master](https://github.com/fireeye/capa-rules/compare/<release>...master)
|
||||
```
|
||||
- [ ] Update [capa/version.py](https://github.com/mandiant/capa/blob/master/capa/version.py)
|
||||
- [ ] Create a PR with the updated [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md) and [capa/version.py](https://github.com/mandiant/capa/blob/master/capa/version.py). Copy this checklist in the PR description.
|
||||
- [ ] After PR review, merge the PR and [create the release in GH](https://github.com/mandiant/capa/releases/new) using text from the [CHANGELOG.md](https://github.com/mandiant/capa/blob/master/CHANGELOG.md).
|
||||
- [ ] Verify GH actions [upload artifacts](https://github.com/mandiant/capa/releases), [publish to PyPI](https://pypi.org/project/flare-capa) and [create a tag in capa rules](https://github.com/mandiant/capa-rules/tags) upon completion.
|
||||
- [ ] Manually update capa rules major version rule branch
|
||||
```commandline
|
||||
[capa/rules] $ git pull master
|
||||
[capa/rules] $ git checkout v3 # create if new major version: git checkout -b vX
|
||||
[capa/rules] $ git merge master
|
||||
[capa/rules] $ git push origin v3
|
||||
```
|
||||
- [ ] Update [capa/version.py](https://github.com/fireeye/capa/blob/master/capa/version.py)
|
||||
- [ ] Create a PR with the updated [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md) and [capa/version.py](https://github.com/fireeye/capa/blob/master/capa/version.py). Copy this checklist in the PR description.
|
||||
- [ ] After PR review, merge the PR and [create the release in GH](https://github.com/fireeye/capa/releases/new) using text from the [CHANGELOG.md](https://github.com/fireeye/capa/blob/master/CHANGELOG.md).
|
||||
- [ ] Verify GH actions [upload artifacts](https://github.com/fireeye/capa/releases), [publish to PyPI](https://pypi.org/project/flare-capa) and [create a tag in capa rules](https://github.com/fireeye/capa-rules/tags) upon completion.
|
||||
- [ ] [Spread the word](https://twitter.com)
|
||||
- [ ] Update internal service
|
||||
|
||||
|
||||
72
doc/rules.md
72
doc/rules.md
@@ -1,72 +0,0 @@
|
||||
### rules
|
||||
|
||||
|
||||
capa uses a collection of rules to identify capabilities within a program.
|
||||
The [github.com/mandiant/capa-rules](https://github.com/mandiant/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
|
||||
|
||||
When you download a standalone version of capa, this standard library is embedded within the executable and capa will use these rules by default:
|
||||
|
||||
```console
|
||||
$ capa suspicious.exe
|
||||
```
|
||||
|
||||
However, you may want to modify the rules for a variety of reasons:
|
||||
|
||||
- develop new rules to find behaviors, and/or
|
||||
- tweak existing rules to reduce false positives, and/or
|
||||
- collect a private selection of rules not shared publicly.
|
||||
|
||||
Or, you may want to use capa as a Python library within another application.
|
||||
|
||||
In these scenarios, you must provide the rule set to capa as a directory on your file system. Do this using the `-r`/`--rules` parameter:
|
||||
|
||||
```console
|
||||
$ capa --rules /local/path/to/rules suspicious.exe
|
||||
```
|
||||
|
||||
You can collect the standard set of rules in two ways:
|
||||
|
||||
- [download from the Github releases page](#download-release-archive), or
|
||||
- [clone from Github](#clone-with-git).
|
||||
|
||||
Note that you must use match the rules major version with the capa major version,
|
||||
i.e., use `v1` rules with `v1` of capa.
|
||||
This is so that new versions of capa can update rule syntax, such as by adding new fields and logic.
|
||||
|
||||
Otherwise, using rules with a mismatched version of capa may lead to errors like:
|
||||
|
||||
```
|
||||
$ capa --rules /path/to/mismatched/rules suspicious.exe
|
||||
ERROR:lint:invalid rule: injection.yml: invalid rule: unexpected statement: instruction
|
||||
```
|
||||
|
||||
You can check the version of capa you're currently using like this:
|
||||
|
||||
```console
|
||||
$ capa --version
|
||||
capa 3.0.3
|
||||
```
|
||||
|
||||
#### download release archive
|
||||
|
||||
The releases page is [here](https://github.com/mandiant/capa-rules/tags/).
|
||||
Find the most recent release corresponding to your major version of capa and download the ZIP archive.
|
||||
Here are some quick links:
|
||||
- v1: [v1](https://github.com/mandiant/capa-rules/releases/tag/v1)
|
||||
- v2: [v2](https://github.com/mandiant/capa-rules/releases/tag/v2)
|
||||
- v3: [v3](https://github.com/mandiant/capa-rules/releases/tag/v3)
|
||||
|
||||
#### clone with git
|
||||
|
||||
To fetch with git, clone the appropriate branch like this:
|
||||
|
||||
```console
|
||||
$ git clone https://github.com/mandiant/capa-rules.git -b v3 /local/path/to/rules
|
||||
```
|
||||
|
||||
Note that the branch name (`v3` in the example above) must match the major version of capa you're using.
|
||||
|
||||
- [v1](https://github.com/mandiant/capa-rules/tree/v1): `v1`
|
||||
- [v2](https://github.com/mandiant/capa-rules/tree/v2): `v2`
|
||||
- [v3](https://github.com/mandiant/capa-rules/tree/v3): `v3`
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user