Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: YARA-X dumper module #50

Closed
wants to merge 26 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
01d38da
yara-x-dumper init
TommYDeeee Oct 13, 2023
829ca56
add basic output for hardcoded module
TommYDeeee Oct 16, 2023
85ea33a
use all builtin modules for output
TommYDeeee Oct 19, 2023
2222d9f
user defined modules WiP
TommYDeeee Oct 20, 2023
fc5945d
user defined modules
TommYDeeee Oct 23, 2023
67f78d2
multiple serilization formats implemented
TommYDeeee Oct 24, 2023
46b4555
toml,xml added, output can be specified with CLI
TommYDeeee Oct 25, 2023
04911a1
human-readable format WiP
TommYDeeee Oct 30, 2023
fb111a8
human-readable format is similar to YAML
TommYDeeee Oct 31, 2023
7d5a844
colors and comments in human readable format, timestamp support, mach…
TommYDeeee Nov 1, 2023
cd6854b
added colors for fields, module filtering, protobuf map printing
TommYDeeee Nov 2, 2023
c6b38f2
added custom colors and support for skipping 0x00 bytes in the middle…
TommYDeeee Nov 3, 2023
bb4bc0d
code formatting
TommYDeeee Nov 6, 2023
58de81a
added documentation and tests
TommYDeeee Nov 6, 2023
ec8ca96
add module documentation
TommYDeeee Nov 7, 2023
e017dbd
remove unnecessary optional marking
TommYDeeee Nov 7, 2023
4e15ca5
clippy warnings fixed
TommYDeeee Nov 7, 2023
624c525
remove unused dependancies
TommYDeeee Nov 7, 2023
3cdf7ac
yr dump decoupling
TommYDeeee Nov 8, 2023
aa5ad9d
yr dump decoupled
TommYDeeee Nov 8, 2023
5e2844c
remove unused dependancies
TommYDeeee Nov 8, 2023
157bee2
unnecessary formats removed and code refactored
TommYDeeee Nov 9, 2023
ce66466
fix clippy warnings
TommYDeeee Nov 9, 2023
59c6d3c
fix issues with yaml format
TommYDeeee Nov 10, 2023
2209400
make use of new module API
TommYDeeee Nov 10, 2023
188e30a
remove validity check from yara-x and fix clippy warnings
TommYDeeee Nov 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 142 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ rust-version = "1.70.0"
members = [
"yara-x",
"yara-x-cli",
"yara-x-dump",
"yara-x-fmt",
"yara-x-macros",
"yara-x-parser",
Expand Down Expand Up @@ -70,14 +71,15 @@ regex-automata = { git = "https://github.com/plusvic/regex.git", rev="423493d" }
rustc-hash = "1.1.0"
smallvec = "1.10.0"
serde = "1.0.156"
serde_json = "1.0.107"
serde_json = { version = "1.0.108", features = ["preserve_order"] }
thiserror = "1.0.40"
uuid = "1.4.1"
walrus = "0.20.1"
wasmtime = "12.0.2"
yaml-rust = "0.4.5"
yansi = "0.5.1"
yara-x = { path = "yara-x" }
yara-x-dump = { path = "yara-x-dump" }
yara-x-fmt = { path = "yara-x-fmt" }
yara-x-macros = { path = "yara-x-macros" }
yara-x-parser = { path = "yara-x-parser" }
Expand Down
30 changes: 26 additions & 4 deletions docs/Module Developer's Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,9 @@ file, but one describing a module. In fact, you can put any `.proto` file in the
files is describing a YARA module. Only files containing a `yara.module_options`
section will define a module.

Options `name` and `root_message` are required, while `rust_module` is optional.
The `name` option defines the module's name. This is the name that will be used
for importing the module in a YARA rule, in this case our module will be imported
Options `name` and `root_message` are required, while `rust_module` is optional.
The `name` option defines the module's name. This is the name that will be
used for importing the module in a YARA rule, in this case our module will be imported
with `import "text"`. The `root_message` option indicates which is the module's
root structure, it must contain the name of some structure (a.k.a message) defined
in the `.proto` file. In our case the value for `root_message` is `"text.Text"`
Expand All @@ -144,7 +144,29 @@ This is a very simple structure with only two integer fields. Notice that the
numbers after the `=` signs are not the values for those fields, they are
actually field tags (i.e: a unique number identifying each field in a message).
This may be confusing if you are not familiar with protobuf's syntax, so again:
explore the protobuf's [documentation](https://developers.google.com/protocol-buffers).
explore the protobuf's [documentation](https://developers.google.com/protocol-buffers).

One thing that can be done with integer fields is to represent them in some other way.
This optional representation is shown in `yr dump` crate output. This crate provides
two output formats: JSON and YAML. Both can be shown in colored output via `-c|--color` option.
The last mentioned also provides custom representation for integer numbers. Let's say
for some fields it makes sense to show them as hexadecimal numbers. This can be done by
adding `[(yara.field_options).yaml_fmt = "<format>"];` descriptor to the field.
Currently supported formats are: hexadecimal number and human-readable timestamp.
For example:

```
message Macho {
optional uint32 magic = 1 [(yara.field_options).yml_fmt= "x"];
}
```

This will mark magic field as a hexadecimal number and it will be shown as
`magic: 0xfeedfacf` instead of `4277009103`. Other format that is supported right now is
for timestamps. If you want to show some integer field as a timestamp you can do it by
setting `[(yara.field_options).yml_fmt = "t"];` descriptor to the field and
human readable timestamps will be shown in YAML comment after its integer value.


Also notice that we are defining our fields as `optional`. In `proto2` fields
must be either `optional` or `required`, while in `proto3` they are always
Expand Down
7 changes: 6 additions & 1 deletion yara-x-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -36,21 +36,26 @@ logging = ["dep:log", "dep:env_logger"]
[dependencies]
ascii_tree = { workspace = true }
anyhow = { workspace = true }
clap = { workspace = true, features=["cargo"] }
clap = { workspace = true, features=["cargo", "derive"] }
globwalk = { workspace = true }
enable-ansi-support = { workspace = true }
env_logger = { workspace = true , optional = true }
log = { workspace = true, optional = true }
protobuf = { workspace = true }
protobuf-json-mapping = "3.3.0"
serde_json = { workspace = true }
yansi = { workspace = true }
yara-x = { workspace = true }
yara-x-dump = { workspace = true }
yara-x-parser = { workspace = true, features = ["ascii-tree"] }
yara-x-fmt = { workspace = true }
yara-x-proto = { workspace = true }

crossbeam = "0.8.2"
crossterm = "0.27.0"
indent = "0.1.1"
pprof = { version = "0.12.1", features = ["flamegraph"], optional=true }
superconsole = "0.2.0"
wild = "2.1.0"
colored_json = "4.0.0"
strum_macros = "0.25"
Loading
Loading