Merge pull request #26 from ivg/async-bap

Async bap

.travis.yml

@@ -0,0 +1,6 @@
+sudo: false
+language: python
+python:
+  - "3.5"
+install: pip install tox
+script: tox

CHANGES.md

@@ -0,0 +1,21 @@
+0.2.0
+-----
+* call BAP asynchronously (without blocking IDA)
+* run several instances of BAP in parallel
+* special attribute view (instead of `Alt-T` search)
+* neater comment syntax (attr=value instead of sexp)
+* task manager for primitive job control
+* plugins are now callable from the menu (try `Ctrl-3`)
+* each instance has its own view
+* view selector can switch between views
+* stderr and stdout are properly dumped into the view
+* cross-platform implementation (Docker, Windows should work)
+* more robust type emition
+* new generic ida service integration (for calls to IDA from BAP)
+* added unit tests
+* Travis-CI integration
+* code refactoring: more pythonic, PEP8 compilant, pylint-happy
+
+0.1.0
+-----
+* initial release

README.md

@@ -2,18 +2,35 @@ BAP IDA Python
 ==============
 
 This package provides the necessary IDAPython scripts required for
-interoperatibility between BAP and IDA Pro. It also provides many useful feature additions to IDA, by leveraging power from BAP.
+interoperatibility between BAP and IDA Pro. It also provides many
+useful feature additions to IDA, by leveraging power from BAP.
 
 Features
 --------
 
+BAP-IDA integration package installs several plugins into IDA
+distribution. Some plugins works automatically, and do not require
+user intervention, while others are invoked with keybindings, or via
+the `Edit->Plugins` menu, that can be popped at with the `Ctrl-3`
+bindging.
+
+
 ### Function information augmentation
 
-By just hitting the `Shift+P` key, IDA will call BAP which will use its own analysis (and all the information sources that it knows of) to obtain all the locations where there are functions. This information is then propagated to IDA and used to create functions there automatically. This is especially useful in scenarios where there are a lot of indirect calls etc and BAP (using its different plugins) is able to detect functions in the code which IDA is unable to do so.
+By just hitting the `Shift+P` key, IDA will call BAP which will use
+its own analysis (and all the information sources that it knows of) to
+obtain all the locations where there are functions. This information
+is then propagated to IDA and used to create functions there
+automatically. This is especially useful in scenarios where there are
+a lot of indirect calls etc and BAP (using its different plugins) is
+able to detect functions in the code which IDA is unable to do so.
 
 ### Taint Propagation
 
-By choosing a taint source and hitting either `Ctrl+A` (for tainting register) or `Ctrl+Shift+A` (for tainting pointer), one can easily see how taint propagates through the code, in both disassembly and decompilation views.
+By choosing a taint source and hitting either `Ctrl+A` (for tainting
+an immediate value) or `Ctrl+Shift+A` (for data pointed by a value),
+one can easily see how taint propagates through the code, in both
+disassembly and decompilation views.
 
 #### In Text/Graph View
 ![taint](docs/taint.png)
@@ -23,43 +40,98 @@ By choosing a taint source and hitting either `Ctrl+A` (for tainting register) o
 
 ### BIR Attribute Tagging, with arbitrary BAP plugins
 
-BAP has the ability to tag a lot of possible attributes to instructions. These BIR attributes can be tagged automatically as comments in IDA, by running arbitrary plugins in BAP. Just hit `Ctrl+S`.
+BAP has the ability to tag a lot of possible attributes to
+instructions. These BIR attributes can be tagged automatically as
+comments in IDA, by running arbitrary plugins in BAP. Just hit
+`Ctrl+S`.
+
+Here's an example of output for Saluki showing that a certain malloc
+is unchecked (pointing to a potential vulnerability).
 
-Here's an example of output for Saluki showing that a certain malloc is unchecked (pointing to a potential vulnerability).
+Clearing all BAP comments (without affecting your own personal
+comments in IDA) can be done by pressing `Ctrl+Shift+S`.
 
-Clearing all BAP comments (without affecting your own personal comments in IDA) can be done by pressing `Ctrl+Shift+S`.
+To view all current attributes in the single window hit `Shift-B`.
+You can sort attributes by clicking the columns or you can search
+through them using IDA's extensive search facilities (hit the `Help`
+bottom on the list to get more information). You can jump directly
+to the attribute location by selecting it.
 
 #### In Text/Graph View
 ![bir-attr-saluki](docs/bir-attr-saluki.png)
 
 #### In Pseudocode View
 ![bir-attr-saluki-decompiler](docs/bir-attr-saluki-decompiler.png)
 
-### BAP View
+### BAP Task Manager and Viewer
 
-Sometimes, you just wish to see the BAP output of the command you just ran to generate BIR attributes (or for the taints), and you can do this in IDA by hitting `Ctrl+Alt+Shift+S` to see the command the BAP ran, along with its output. Do note that this also shows bir output from bap.
+Every instance of BAP will have a corresponding view, that will
+accumulate all data written by BAP. The BAP Viewer (`Ctrl-Alt-F5`)
+provides an easy way to switch between multiple BAP Views.
+
+Since you can run multiple instances of BAP asynchronously, it is
+useful to have an ability to view the state of currently running
+processes, and, even, to terminate those who take too much time or
+memory.  The BAP Task Manager (accessible via the `Ctrl-Alt-Shift-F5`
+keybinding, or via the `Ctrl-3` plugin menu) provides such
+functionality.
 
 ![bap-view](docs/bap-view.png)
 
 ### Symbol and Type Information
 
-Whenever possible, `bap-ida-python` passes along the latest symbol and type information from IDA (including changes you might have made manually), so as to aid better and more accurate analysis in BAP. For example, let's say you recognize that a function is a malloc in a stripped binary, by just using IDA's rename feature (Keybinding: `N`), you can inform BAP of this change during the next run of, say, saluki, without needing to do anything extra. It works automagically!
+Whenever possible, `bap-ida-python` passes along the latest symbol and
+type information from IDA (including changes you might have made
+manually), so as to aid better and more accurate analysis in BAP. For
+example, let's say you recognize that a function is a malloc in a
+stripped binary, by just using IDA's rename feature (Keybinding: `N`),
+you can inform BAP of this change during the next run of, say, saluki,
+without needing to do anything extra. It works automagically!
 
 Installation
 ------------
 
-Copy all of the files and directories from the `plugins` directory into `$IDADIR/plugins`.
+Copy all of the files and directories from the `plugins` directory
+into `$IDADIR/plugins`.
 
-The first run of IDA after that will prompt you to provide the path to BAP (along with a default if IDA is able to automatically detect BAP). If you wish to edit the path to BAP manually later, you can edit the file `$IDADIR/cfg/bap.cfg`.
+The first run of IDA after that will prompt you to provide the path to
+BAP (along with a default if IDA is able to automatically detect
+BAP). If you wish to edit the path to BAP manually later, you can edit
+the file `$IDADIR/cfg/bap.cfg`.
 
 #### Opam?
 
-It is usually much easier to install through opam if you have already followed all the installation steps in the [bap repository](https://github.com/BinaryAnalysisPlatform/bap). Just run:
+It is usually much easier to install through opam if you have already
+followed all the installation steps in the
+[bap repository](https://github.com/BinaryAnalysisPlatform/bap). Just
+run:
 
 ```
 opam install bap-ida-python
 ```
 
+Debugging
+---------
+
+The integration package is still in alpha stage, so there are a few
+bugs lurking in the codebase. If you have any issues, then, please,
+enable the debug mode, by typing the following command in the IDA's
+python console:
+
+```python
+BapIda.DEBUG=True
+```
+
+This will increase the verbosity level, so that you can see what commands
+were actually issued to the bap backend. In the debug mode, the temporary
+files will not be removed, so they can be archived and sent to us, for the
+ease of debugging.
+
+
 #### IDA Demo?
 
-You can also use parts of the functionality (i.e. most of everything except for the decompiler outputs, and batch processing from bap) with IDA Free/Demo. However, you would need to install IDAPython. See [here](docs/IDAPython_on_IDADemo.md) for what one of our users reported to work.
+You can also use parts of the functionality (i.e. most of everything
+except for the decompiler outputs, and batch processing from bap) with
+IDA Free/Demo. However, you would need to install IDAPython. See
+[here](docs/IDAPython_on_IDADemo.md) for what one of our users
+reported to work.

plugins/bap/plugins/bap_bir_attr.py

@@ -13,119 +13,106 @@
     Comments in the Text/Graph Views are added using a key-value storage
     with the format (BAP (k1 v1) (k2 v2) ...)
 """
-import idautils
+import idaapi
+import idc
 
+from bap.utils.run import BapIda
 
-class BAP_BIR_Attr(idaapi.plugin_t):
+
+class BapScripter(BapIda):
+
+    def __init__(self, user_args, attrs):
+        super(BapScripter, self).__init__()
+        if attrs:
+            self.action = 'extracting ' + ','.join(attrs)
+        else:
+            self.action = 'running bap ' + user_args
+        self.script = self.tmpfile('py')
+        self.args += user_args.split(' ')
+        self.args += [
+            '--emit-ida-script',
+            '--emit-ida-script-file', self.script.name
+        ]
+        self.args += [
+            '--emit-ida-script-attr='+attr.strip()
+            for attr in attrs
+        ]
+
+
+# perfectly random numbers
+ARGS_HISTORY = 324312
+ATTR_HISTORY = 234345
+
+
+class BapBirAttr(idaapi.plugin_t):
     """
     Plugin to get BIR attributes from arbitrary BAP executions.
 
     Also supports installation of callbacks using install_callback()
     """
+    flags = idaapi.PLUGIN_DRAW
+    comment = "Run BAP "
+    help = "Runs BAP and extracts data from the output"
+    wanted_name = "BAP: Run"
+    wanted_hotkey = "Shift-S"
 
     _callbacks = []
 
-    @classmethod
-    def _do_callbacks(cls):
-        data = {
-            'ea': idc.ScreenEA(),
-        }
-        for callback in cls._callbacks:
-            callback(data)
+    recipes = {}
 
-    @classmethod
-    def run_bap(cls):
+    def _do_callbacks(self, ea):
+        for callback in self._callbacks:
+            callback({'ea': ea})
+
+    def run(self, arg):
         """
         Ask user for BAP args to pass, BIR attributes to print; and run BAP.
 
         Allows users to also use {screen_ea} in the BAP args to get the
         address at the location pointed to by the cursor.
         """
-        import tempfile
-        from bap.utils.run import run_bap_with
-
-        args = {
-            'screen_ea': "0x{:X}".format(idc.ScreenEA()),
-            'ida_script_location': tempfile.mkstemp(suffix='.py',
-                                                    prefix='ida-bap-')[1],
-            'args_from_user': idaapi.askstr(0, '', 'Args to pass to BAP'),
-            'bir_attr': idaapi.askstr(0, 'comment',
-                                      'BIR Attributes (comma separated)')
-        }
-
-        if args['args_from_user'] is None:
-            args['args_from_user'] = ''
-
-        if args['bir_attr'] is not None:
-            for attr in args['bir_attr'].split(','):
-                attr = attr.strip()  # For users who prefer "x, y, z" style
-                args['args_from_user'] += " --emit-ida-script-attr=" + attr
-            args['args_from_user'] += "\
-            --emit-ida-script-file={ida_script_location} \
-            --emit-ida-script \
-            ".format(**args)
-
-        idc.SetStatus(IDA_STATUS_WAITING)
-        idaapi.refresh_idaview_anyway()
-
-        run_bap_with(args['args_from_user'].format(**args))
-
-        idc.SetStatus(IDA_STATUS_READY)
 
-        idaapi.IDAPython_ExecScript(args['ida_script_location'], globals())
+        args_msg = "Arguments that will be passed to `bap'"
 
-        idc.Exec("rm -f \"{ida_script_location}\"".format(**args))  # Cleanup
-
-        idc.Refresh()  # Force the updated information to show up
-
-        cls._do_callbacks()
-
-    @classmethod
-    def clear_bap_comments(cls):
-        """Ask user for confirmation and then clear (BAP ..) comments."""
-        from bap.utils.bap_comment import get_bap_comment
-        from bap.utils.ida import all_valid_ea
-        from idaapi import ASKBTN_CANCEL, ASKBTN_YES
-
-        if idaapi.askyn_c(ASKBTN_CANCEL,
-                          "Delete all (BAP ..) comments?") != ASKBTN_YES:
+        args = idaapi.askstr(ARGS_HISTORY, '--passes=', args_msg)
+        if args is None:
             return
+        attr_msg = "A comma separated list of attributes,\n"
+        attr_msg += "that should be propagated to comments"
+        attr_def = self.recipes.get(args, '')
+        attr = idaapi.askstr(ATTR_HISTORY, attr_def, attr_msg)
 
-        for ea in all_valid_ea():
-            old_comm = idaapi.get_cmt(ea, 0)
-            if old_comm is None:
-                continue
-            _, start_loc, end_loc = get_bap_comment(old_comm)
-            new_comm = old_comm[:start_loc] + old_comm[end_loc:]
-            idaapi.set_cmt(ea, new_comm, 0)
-
-        cls._do_callbacks()
+        if attr is None:
+            return
 
-    flags = idaapi.PLUGIN_FIX
-    comment = "BAP BIR Attr Plugin"
-    help = "BAP BIR Attr Plugin"
-    wanted_name = "BAP BIR Attr Plugin"
-    wanted_hotkey = ""
+        # store a choice of attributes for the given set of arguments
+        # TODO: store recipes in IDA's database
+        self.recipes[args] = attr
+        ea = idc.ScreenEA()
+        attrs = []
+        if attr != '':
+            attrs = attr.split(',')
+        analysis = BapScripter(args, attrs)
+        analysis.on_finish(lambda bap: self.load_script(bap, ea))
+        analysis.run()
+
+    def load_script(self, bap, ea):
+        idc.SetStatus(idc.IDA_STATUS_WORK)
+        idaapi.IDAPython_ExecScript(bap.script.name, globals())
+        self._do_callbacks(ea)
+        idc.Refresh()
+        # do we really need to call this?
+        idaapi.refresh_idaview_anyway()
+        idc.SetStatus(idc.IDA_STATUS_READY)
 
     def init(self):
         """Initialize Plugin."""
-        from bap.utils.ida import add_hotkey
-        add_hotkey("Shift-S", self.run_bap)
-        add_hotkey("Ctrl-Shift-S", self.clear_bap_comments)
         return idaapi.PLUGIN_KEEP
 
     def term(self):
         """Terminate Plugin."""
         pass
 
-    def run(self, arg):
-        """
-        Run Plugin.
-
-        Ignored since keybindings are installed.
-        """
-        pass
-
     @classmethod
     def install_callback(cls, callback_fn):
         """
@@ -136,9 +123,10 @@ def install_callback(cls, callback_fn):
         Dict is guaranteed to get the following keys:
             'ea': The value of EA at point where user propagated taint from.
         """
-        cls._callbacks[ptr_or_reg].append(callback_fn)
+        idc.Message('a callback is installed\n')
+        cls._callbacks.append(callback_fn)
 
 
 def PLUGIN_ENTRY():
     """Install BAP_BIR_Attr upon entry."""
-    return BAP_BIR_Attr()
+    return BapBirAttr()