Commit Graph

1214 Commits

Author SHA1 Message Date
William Ballenthin
d3caecc551 pep8 2021-03-04 18:06:06 -07:00
William Ballenthin
004ddb3e66 main: load gzip compressed .pat files 2021-03-04 18:04:46 -07:00
William Ballenthin
20894124e6 tests: test FLIRT matching 2021-03-04 15:50:05 -07:00
William Ballenthin
22c4e3b8c2 viv: cleanup flirt changes 2021-03-04 15:46:14 -07:00
William Ballenthin
c2a4629c62 scripts: add cli arguments to specify signatures 2021-03-04 15:04:33 -07:00
William Ballenthin
c0f4fe6867 merge master 2021-03-04 14:59:17 -07:00
William Ballenthin
f2c95568bd main: add FLIRT signature matching configuration 2021-03-04 14:52:22 -07:00
William Ballenthin
358aab85e7 viv: move FLIRT matching into viv-utils 2021-03-04 14:51:40 -07:00
Ana María Martínez Gómez
829274cd5e Merge pull request #421 from Ana06/viv-py3 2021-03-03 21:40:08 +01:00
Ana Maria Martinez Gomez
c522f5094a Use -j option in test_backend_option
Use `-j` option in `test_backend_option` to check the extractor and that
rules have been extracted. This way we don't need to check if a concrete
rule matches, but only that at least a rule matches.
2021-03-03 18:33:20 +01:00
Ana Maria Martinez Gomez
29b6772721 Test backend option
As `get_extractor` returns only vivisect now, `test_main` is not run for
smda. Test that capa works with all backends. It doesn't test that the
backend is actually called.
2021-03-03 17:36:51 +01:00
Ana Maria Martinez Gomez
695b5b50ab Remove va not None check
Instead of checking if `va` is `None in `get_section()` we should avoid
calling this function with `None`. This have been fixed in the following
PR, so this is not longer needed:
https://github.com/fireeye/capa/pull/442
2021-03-03 17:36:51 +01:00
Ana Maria Martinez Gomez
42af7b2d8b Use default backend instead of None
Set the `backend` variable to the default backend by default instead to
`None`. The `backend` variable is needed in Python 2 as `args.backend`
is only set in Python 3. Although the value of the backend variable is
ignored in Python 2, so that the default value is not used.

Co-authored-by: William Ballenthin <william.ballenthin@fireeye.com>
2021-03-03 17:36:51 +01:00
Ana Maria Martinez Gomez
079a9b5204 Remove backend option from Python 2
Do only provide the backend option in Python 3, as there is only one
backend in Python 2. This way we keep the help text simpler.
2021-03-03 17:36:51 +01:00
Ana Maria Martinez Gomez
e5048fd3ac Add missing va parameter to SegmentationViolation
The `envi.SegmentationViolation()` was missing the `va` required
parameter. This has started failing now, because calling
`vw.getSegment(0x4BA190)` for the `tests/data/mimikatz.exe_` produces
different results in Python 2 and Python 3. It returns `None` in Python
3 while the output in Python 2 is:
`(4939776, 16840, '.data', 'mimikatz')`

I have reported the issue to vivisect:
https://github.com/vivisect/vivisect/issues/370
2021-03-03 17:36:51 +01:00
Ana Maria Martinez Gomez
18eaea95fa Fix TypeError exception in Python3
`va` can be None and this causes Python 3 to raise a TypeError
exception. This is caused by the following breaking change in Python3:
> The ordering comparison operators (<, <=, >=, >) raise a TypeError
> exception when the operands don’t have a meaningful natural ordering.

This didn't failed in the previously tried vivisect version (master from
one week ago and not the release). This may have been caused by a bug in
vivisect that has been fixed.
2021-03-03 17:36:51 +01:00
Ana Maria Martinez Gomez
a4a0a56448 Vivisect 1.0.0 released
Vivisect 1.0.0 (Python 3) has been released, so we do not need to link
to my GitHub branch anymore.

https://pypi.org/project/vivisect
2021-03-03 17:36:50 +01:00
Ana Maria Martinez Gomez
40ed2f39a4 Make backend a required parameter in get_extractor
Make the `backend` argument required in the `get_extractor` internal
routine. Specify a backend in the scripts which call this function. Add
a CLI backend option in capa/features/freeze.py as well.
2021-03-03 17:36:50 +01:00
Ana Maria Martinez Gomez
2859b037aa Use constants for backend option
Use constants instead of string literals for the backend option.
2021-03-03 17:36:50 +01:00
Ana Maria Martinez Gomez
bbb7878e0a Enable tests for vivisect in Python3
Now we support vivisect as backend in Python3. We should test it.
2021-03-03 17:36:50 +01:00
Ana Maria Martinez Gomez
fc438866ec Add option to select the backend in Py3
Now we have two working backends in Python3! Add an option to select
which one to use. With this code, vivisect is the default backend, but
this is really easy to change. We could do some analysis to see if smda
performances better than vivisect once the vivisect implementation.
2021-03-03 17:36:50 +01:00
Ana Maria Martinez Gomez
2da2f498a2 Add script to compare vivisect Python 2 vs 3
Compare the performance of vivisect Python 2 vs 3 by counting the number
of feature of each type extracted for every binary in `tests/data`.
Render the ones that perform bad (under a threshold - 98) and the total
performance. Render also the running time per binary for both Python 2 and 3.

From this result, it seems that vivisect behaves properly with Python3.
2021-03-03 17:36:50 +01:00
Ana Maria Martinez Gomez
29dffffe1b Python3 support for vivisect
Vivisect has moved to Python3. Allow to run vivisect with Python3 in
capa.

I am using the following version of vivisect (which includes fixes for
some bugs I have found and some open PRs in vivisect):
https://github.com/Ana06/vivisect/tree/py-3
2021-03-03 17:36:49 +01:00
Capa Bot
1ecaad5413 Sync capa rules submodule 2021-03-02 15:06:24 +00:00
Willi Ballenthin
cd56d672c0 Merge pull request #442 from fireeye/williballenthin-patch-2
viv: ignore empty branch targets
2021-03-01 08:43:26 -07:00
Willi Ballenthin
68aed3c190 insn: better document when branch va may be none 2021-02-28 23:03:08 -07:00
William Ballenthin
f16ecd837e viv: flirt: add more documentation 2021-02-26 05:02:10 -07:00
Willi Ballenthin
68fcc03d5c viv: ignore empty branch targets
but what does this really mean? why would `getBranches` return `None`?

closes #441
2021-02-25 13:34:59 -07:00
William Ballenthin
bfcae0e754 extractors: viv: match flirt signatures [wip] 2021-02-25 12:21:27 -07:00
William Ballenthin
1b2c8880ee capa: extractors: sketch API extension to support function id 2021-02-25 12:20:29 -07:00
Ana Maria Martinez Gomez
fa7d58d01a Add missing va parameter to SegmentationViolation
The `envi.SegmentationViolation()` was missing the `va` required
parameter. This has started failing now, because calling
`vw.getSegment(0x4BA190)` for the `tests/data/mimikatz.exe_` produces
different results in Python 2 and Python 3. It returns `None` in Python
3 while the output in Python 2 is:
`(4939776, 16840, '.data', 'mimikatz')`

I have reported the issue to vivisect:
https://github.com/vivisect/vivisect/issues/370
2021-02-25 11:20:45 +01:00
Ana Maria Martinez Gomez
ec558f377a Fix TypeError exception in Python3
`va` can be None and this causes Python 3 to raise a TypeError
exception. This is caused by the following breaking change in Python3:
> The ordering comparison operators (<, <=, >=, >) raise a TypeError
> exception when the operands don’t have a meaningful natural ordering.

This didn't failed in the previously tried vivisect version (master from
one week ago and not the release). This may have been caused by a bug in
vivisect that has been fixed.
2021-02-25 10:15:49 +01:00
Ana Maria Martinez Gomez
186eba7197 Vivisect 1.0.0 released
Vivisect 1.0.0 (Python 3) has been released, so we do not need to link
to my GitHub branch anymore.

https://pypi.org/project/vivisect
2021-02-25 10:05:04 +01:00
Ana Maria Martinez Gomez
d28ba3c628 Make backend a required parameter in get_extractor
Make the `backend` argument required in the `get_extractor` internal
routine. Specify a backend in the scripts which call this function. Add
a CLI backend option in capa/features/freeze.py as well.
2021-02-25 10:04:19 +01:00
Ana Maria Martinez Gomez
a026cb84d1 Use constants for backend option
Use constants instead of string literals for the backend option.
2021-02-25 09:35:40 +01:00
Ana Maria Martinez Gomez
3acc3eeabd Enable tests for vivisect in Python3
Now we support vivisect as backend in Python3. We should test it.
2021-02-25 09:35:40 +01:00
Ana Maria Martinez Gomez
a92d2af7f8 Add option to select the backend in Py3
Now we have two working backends in Python3! Add an option to select
which one to use. With this code, vivisect is the default backend, but
this is really easy to change. We could do some analysis to see if smda
performances better than vivisect once the vivisect implementation.
2021-02-25 09:35:40 +01:00
Ana Maria Martinez Gomez
adcb683458 Add script to compare vivisect Python 2 vs 3
Compare the performance of vivisect Python 2 vs 3 by counting the number
of feature of each type extracted for every binary in `tests/data`.
Render the ones that perform bad (under a threshold - 98) and the total
performance. Render also the running time per binary for both Python 2 and 3.

From this result, it seems that vivisect behaves properly with Python3.
2021-02-25 09:35:40 +01:00
Capa Bot
939b29bf60 Sync capa rules submodule 2021-02-24 23:00:34 +00:00
Ana Maria Martinez Gomez
e4925613b3 Python3 support for vivisect
Vivisect has moved to Python3. Allow to run vivisect with Python3 in
capa.

I am using the following version of vivisect (which includes fixes for
some bugs I have found and some open PRs in vivisect):
https://github.com/Ana06/vivisect/tree/py-3
2021-02-24 17:55:39 +01:00
Capa Bot
2f6a6e4628 Sync capa rules submodule 2021-02-24 08:07:52 +00:00
Capa Bot
7938ea34d0 Sync capa rules submodule 2021-02-24 08:06:30 +00:00
Capa Bot
ed94e36f7a Sync capa rules submodule 2021-02-24 00:12:19 +00:00
mike-hunhoff
1c3a8df136 Merge pull request #439 from fireeye/explorer/rulegen-support-file-scope
adding file scope support to rule generator IDA plugin
2021-02-23 11:50:54 -07:00
Michael Hunhoff
9f254b22ee adding file scope support to rule generator IDA plugin 2021-02-23 11:10:34 -07:00
Capa Bot
753f8ce84e Sync capa rules submodule 2021-02-23 17:33:38 +00:00
Capa Bot
acf3b549de Sync capa rules submodule 2021-02-23 15:29:20 +00:00
Capa Bot
669f6dcf98 Sync capa rules submodule 2021-02-23 15:23:19 +00:00
Capa Bot
e4f7c4aab1 Sync capa rules submodule 2021-02-23 15:22:43 +00:00
Moritz
5836d55e21 Merge pull request #438 from fireeye/explorer/show-results-by-function
explorer: adding option to show results by function
2021-02-22 18:23:44 +01:00