Merge remote-tracking branch 'fame/master'

.gitignore

@@ -77,3 +77,7 @@ vendor/*
 
 # Agent
 agent/module.py
+
+# Editors
+.vscode
+.mypy_cache

docker/Dockerfile

@@ -0,0 +1,45 @@
+
+FROM debian:stretch
+
+WORKDIR /opt
+
+RUN apt-get update && \
+  apt-get upgrade -y && \
+  DEBIAN_FRONTEND=noninteractive apt-get install -y \
+  git \
+  python-dev \
+  python-pip \
+  screen \
+  p7zip-full \
+  libjpeg-dev \
+  zlib1g-dev \
+  apt-transport-https \
+  ca-certificates \
+  curl \
+  gnupg2 \
+  software-properties-common
+
+RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 \
+   --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
+RUN echo "deb http://repo.mongodb.org/apt/debian stretch/mongodb-org/4.0 main" | tee /etc/apt/sources.list.d/mongodb-org-4.0.list
+
+RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - && \
+    add-apt-repository \
+    "deb [arch=amd64] https://download.docker.com/linux/debian \
+    $(lsb_release -cs) \
+    stable"
+
+RUN apt-get update && apt-get install -y \
+    docker-ce \
+    docker-ce-cli \
+    containerd.io
+
+RUN apt-get update && apt-get install -y \
+    mongodb-org
+
+
+RUN pip install virtualenv && \
+    git clone https://github.com/certsocietegenerale/fame
+
+ENTRYPOINT ["/opt/fame/docker/launch.sh"]
+EXPOSE 4200

docker/README.md

@@ -0,0 +1,23 @@
+# Docker support
+
+This is probably the quickest way to spawn a Fame dev instance.
+
+## Install docker
+
+Follow the [official instructions](https://www.docker.com/community-edition).
+
+## Clone the repo
+
+    $ git clone https://github.com/certsocietegenerale/fame/
+    $ cd fame/docker
+
+## Build docker image
+
+    $ docker build -t famedev:latest .
+
+## Run docker image
+
+To run container based on famedev image with docker inception (bind docker socket and bind fame temp dir).
+So chose a temp directory on your system and use the following command.
+
+    $ docker run -it -v /var/run/docker.sock:/var/run/docker.sock -v <FIXME local temp dir path>:/opt/fame/temp --name famedev -p 4200:4200 famedev:latest

docker/launch.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+cd /opt/fame
+service docker start
+mongod -f /etc/mongod.conf &
+utils/run.sh utils/install.py
+screen -dmS "web"  bash -c "utils/run.sh webserver.py"
+screen -dmS "worker" bash -c "utils/run.sh worker.py"
+/bin/bash

docs/modules.rst

@@ -31,51 +31,6 @@ The best practice is to do the following:
 * Create a python file inside your directory for your module.
 * Inside your python file, create a class that inherits from :class:`fame.core.module.PreloadingModule`, :class:`fame.core.module.ProcessingModule`, :class:`fame.core.module.ReportingModule`, :class:`fame.core.module.ThreatIntelligenceModule` or :class:`fame.core.module.AntivirusModule`.
 
-Writing a Preloading module
-===========================
-
-Preloading modules are used to download a sample file automatically to make it available to FAME.
-
-To define a new Preloading Module just create a Python class that inherits from :class:`fame.core.module.PreloadingModule` and implements :func:`fame.core.module.PreloadingModule.preload`.
-
-These methods should return a boolean indicating if the sample was downloaded successfully. If the return value is ``True``, one thing will happen:
-
-* A tag with the module's name will automatically be added to the analysis.
-
-For example, the module `virustotal_download` takes a hash and download the sample from VirusTotal. If it could not download the sample successfully, it should return ``False`` so that the next preloading module can be scheduled.
-
-Here is the minimal code required for a preloading module::
-
-    from fame.core.module import PreloadingModule
-
-
-    class Dummy(PreloadingModule):
-        # You have to give your module a name, and this name should be unique.
-        name = "dummy"
-
-        # (optional) Describe what your module will do. This will be displayed to users.
-        description = "Does nothing."
-
-        # This method will be called, with the hash of the sample in target
-        def preload(self, target):
-            return True
-
-Scope
------
-
-It may happen that an analyst only has a hash available for analysis. In this case, FAME can download the sample from configured sample sources and trigger an analysis of the sample by its own.
-
-Adding preloading results
--------------------------
-
-Once the module successfully preloaded the sample for FAME, it must add the result to the analysis. As soon as such result is added, FAME schedules the regular analysis based on what was added to the analysis.
-
-You can declare a preloading result by calling :func:`fame.core.module.PreloadingModule.add_preloaded_file`. The function expects a filename and a file-like object with the available data.
-
-Testing Preloading Modules
---------------------------
-
-See :ref:`testing_processing_modules`.
 
 Writing a Processing module
 ===========================
@@ -362,6 +317,45 @@ When it comes to testing your processing modules during development, you have tw
 * The simpler option is to use the :ref:`single_module` utility. This way, you don't need a webserver, a worker or even a MongoDB instance.
 * An ``IsolatedProcessingModule`` can also be tested with the :ref:`single_module` utility. By default, it will execute inside a Virtual Machine (as it should). If you want to test your module without this overhead (if you are already inside the VM for example), you can use the ``-l, --local`` switch.
 
+Writing a Preloading module
+===========================
+
+Preloading modules are used to download a sample file automatically to make it available to FAME.
+
+To define a new Preloading Module just create a Python class that inherits from :class:`fame.core.module.PreloadingModule` and implements :func:`fame.core.module.PreloadingModule.preload`.
+
+If the module was able to successfuly find a sample associated with submitted hash, it should call the `add_preloaded_file` method. If this method is not called, the next preloading module will be scheduled.
+
+For example, the module `virustotal_download` takes a hash and download the sample from VirusTotal.
+
+Here is the minimal code required for a preloading module::
+
+    from fame.core.module import PreloadingModule
+
+
+    class Dummy(PreloadingModule):
+        # You have to give your module a name, and this name should be unique.
+        name = "dummy"
+
+        # (optional) Describe what your module will do. This will be displayed to users.
+        description = "Does nothing."
+
+        # This method will be called, with the hash of the sample in target
+        def preload(self, target):
+            return False
+
+Scope
+-----
+
+It may happen that an analyst only has a hash available for analysis. In this case, FAME can download the sample from configured sample sources and trigger an analysis of the sample by its own.
+
+Adding the preloading result
+----------------------------
+
+Once the module successfully preloaded the sample for FAME, it must add the file to the analysis. Based on what type the file is, FAME then schedules suitable processing modules (if magic mode is enabled).
+
+You can add a preloaded file by calling :func:`fame.core.module.PreloadingModule.add_preloaded_file`. The function expects either a path to a file or a file-like object with the available data (file path has precedence if both are provided).
+
 Common module features
 ======================
 

fame/common/email_utils.py

@@ -68,26 +68,24 @@ class EmailMessage:
     def __init__(self, server, subject):
         self.server = server
 
-        self.msg = MIMEMultipart('alternative')
+        self.msg = MIMEMultipart()
         self.msg['Subject'] = subject
         self.msg['From'] = server.config.from_address
         self.msg['Reply-to'] = server.config.replyto or server.config.from_address
         self.msg['Return-path'] = server.config.replyto or server.config.from_address
         self.msg.preamble = subject
 
-    def add_content(self, text, content_type="text"):
+    def add_content(self, text, content_type="plain"):
         self.msg.attach(MIMEText(text, content_type, "utf-8"))
 
     def add_attachment(self, filepath, filename=None):
         if filename is None:
             filename = os.path.basename(filepath)
 
         with open(filepath, "rb") as f:
-            self.msg.attach(MIMEApplication(
-                f.read(),
-                Content_Disposition='attachment; filename="{}"'.format(filename),
-                Name=filename
-            ))
+            part = MIMEApplication(f.read(), Name=os.path.basename(filepath))
+            part['Content-Disposition'] = 'attachment; filename="{0}"'.format(filename)
+            self.msg.attach(part)
 
     def send(self, to, cc=[], bcc=[]):
         recipients = to + cc + bcc
@@ -134,7 +132,7 @@ def __init__(self, template_path=None):
 
     def new_message(self, subject, body):
         msg = EmailMessage(self, subject)
-        msg.add_content(body, 'text')
+        msg.add_content(body)
 
         return msg