libfuzzer does not work for some targets / programs

[acidghost]

@EliaGeretto and I were trying to run libfuzzer but some programs (e.g. xmllint and lua) are not instrumented correctly and call the original program's main instead of libfuzzer's. They seem like they are CLI programs and not drivers for fuzzers (i.e. using the libfuzzer interface).

We were wondering if all bugs for a target can be triggered by all drivers for the same target; if not, do you know which bugs can be triggered by which driver?

[EliaGeretto]

If, as suggested here, we skip xmllint, what is the impact on the reachability of bugs?

[adrianherrera]

Yeah, you cannot fuzz those targets (with a main function) with libfuzzer. This is just not how libfuzzer is designed to be used (it's a library fuzzer after all 😃).

This is a good question regarding reachability of bugs. I cannot tell you off the top of my head, but you can probably look at papers that have previously evaluated with magma and compare the bugs reached/triggered across xmllint and xml_fuzzer and get a pretty good idea.

[acidghost]

As far as we understand, you have already checked reachability for the MAGMA paper. From your paper:

Bugs which are not triggered, even after multiple campaigns, are manually inspected to verify path reachability and satisfiability of trigger conditions.

[acidghost]

Could you share the test cases that trigger the bugs with us? We can use them to check which harness triggers each bug, if you do not have that information already.

[adrianherrera]

POCs are here -> https://hexhive.epfl.ch/magma/docs/bugs.html

[acidghost]

Thanks, that helps.

Is there a way to find the association between the bug ID used in the paper (e.g. AAH032, etc.) and the patch ID (e.g. SQL001)?

[acidghost]

I found out that 35eab0e changed the names and made a script to parse it:

#!/usr/bin/env python3
import json
from pathlib import Path
import re
import subprocess


COMMIT_ID = "35eab0ee81000bf7167d780ddefffc51b3975d32"
MAGMA = Path(__file__).parent.parent
FORMATS = ["text", "json", "sed"]


# parse the output of git show <commit_id> and find the files that are renamed
def main(format: str = "text", reverse: bool = False):
    if format not in FORMATS:
        raise ValueError(f"invalid format: {format}")

    p = subprocess.run(
        ["git", "show", COMMIT_ID], cwd=MAGMA, capture_output=True, check=True
    )

    renamed_bugs = {}
    for line in p.stdout.decode().splitlines():
        if m := re.match(r"^diff --git a/(.+) b/(.+)$", line):
            file1, file2 = m.groups()
            if file1 != file2 and file1.endswith(".patch"):
                bug1 = Path(file1).stem
                bug2 = Path(file2).stem
                if format == "json":
                    if reverse:
                        renamed_bugs[bug2] = bug1
                    else:
                        renamed_bugs[bug1] = bug2
                elif format == "sed":
                    if reverse:
                        print(f"s/{bug2}/{bug1}/g")
                    else:
                        print(f"s/{bug1}/{bug2}/g")
                elif reverse:
                    print(bug2, ":", bug1)
                else:
                    print(bug1, ":", bug2)

    if format == "json":
        print(json.dumps(renamed_bugs))


if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(description="Find renamed bugs in MAGMA")
    parser.add_argument(
        "-f",
        "--format",
        help="output format",
        choices=FORMATS,
    )
    parser.add_argument(
        "-r", "--reverse", help="reverse the renaming", action="store_true"
    )
    main(**vars(parser.parse_args()))

[adrianherrera]

That script is great! Thanks for that