Skip to content

Development Environment

Jump to bottom
Amir M. Mir edited this page Sep 17, 2020 · 4 revisions

On this page, I explain how to set up a debugging environment for the FASTEN repository. At the time of this writing, the debugging env. consists of a Maven crawler and the FASTEN server which generates call graphs.

The guide has the following sections:

Requirements

In order to develop and run the current pipeline of the FASTEN, you need to meet the following requirements:

Installation

In order to test the current pipeline, you need to install the Maven crawler and the FASTEN server which are explained as follows:

Maven crawler

Install the crawler as follows:

pip install mvncrawler

FASTEN server

  1. Clone the FASTEN repository:
git clone https://github.com/fasten-project/fasten && cd fasten
  1. Build the project using Maven:
mvn clean install
  1. Build the Docker image of the FASTEN server:
cd docker && docker build -t fasten-server-dev -f server/Dockerfile .

Datasets

Before using the FASTEN server or developing, you should have some input data. Please check out the README file here for how to load a pre-made dataset.

Metadata Database Setup

In order to use Metadata Plugin which will populate metadata database with records, you need to install PostgreSQL 12+ database with appropriate schema.

  1. Install PostgreSQL 12+ (here are tutorials on how to do it on macOS and on Linux)
  2. Run the SQL file to create the metadata database schema:
cd analyzer/metadata-plugin/src/main/resources
psql -U $username -d $db_name -f metadata_db_init.sql

Replace $username and $db_name with your PostgreSQL's username and database name, respectively.

Now everything should be set up and you can run the FASTEN server as described in the Development.

Usage

This section shows how to generate call graphs using the Maven crawler and the FASTEN server. To do so, follow the below steps:

  1. Run the Maven crawler to extract some Maven coordinates:
mvncrawler --p ./maven/ --q q_items.txt  --t "fasten.mvn.pkg" --c 5 --l 25

The above example extracts 25 Maven coordinates. For more info about usages of the crawler, please check out its README file.

  1. To start the FASTEN server (on macOS), run its Docker image as follows:
docker run -it fasten-server-dev -k host.docker.internal:9092

Check out this page for running the FASTEN server on a Linux system.

You may want to run the following command to see the generated call graphs:

kafkacat -b localhost:9092 -t fasten.opal.cg

Development

If you have made changes to the FASTEN plug-ins and want to see the output of the FASTEN server, follow the below steps:

  1. Build the project again:
mvn clean install
  1. Run the FASTEN server:
cd docker/server
java -jar server-0.0.1-SNAPSHOT-with-dependencies.jar -p ../plugins/

Overriding the default topics of the plug-ins

A FASTEN plug-in may have a default Kafka topic to consume from. However, it is possible to override the default Kafka topic of a plug-in. To do so, you need to pass a key-value pairs of plugin name and topic to the FASTEN server as follows:

java -jar server-0.0.1-SNAPSHOT-with-dependencies.jar -p ../plugins/ -kt OPAL=fasten.opal.cg

Plug-ins with DB connection

The FASTEN server can load plug-ins with a DB connection like metadata DB. However, you need to pass the DB credentials to the CLI of the FASTEN server as shown below:

java -jar server-0.0.1-SNAPSHOT-with-dependencies.jar -p ../plugins/ -d jdbc:postgresql:$DB_NAME -du $DB_USER
  • $DB_NAME: The name of a PostgreSQL database
  • $DB_USER: A username

Database password must be provided through the environmental variable $FASTEN_DBPASS

Running OPAL plug-in remotely

This page describes how to run the OPAL plug-in independently in a remote server (without the FASTEN server). This may be needed to run the plug-in on powerful clusters for generating a very large call graph.

1- First, go to analyzer/javacg-opal. Uncomment the slf4j dependency in the POM file of the OPAL module. Then, build the module as follows:

mvn clean install

2- Go to docker/plugins/ and run the plug-in using the following command:

java  -Xmx64g -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1233 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -jar javacg-opal-0.0.1-SNAPSHOT-with-dependencies.jar -c ai.h2o:sparkling-water-package_2.11:3.26.8-2.4

The above command produces a call graph for the Maven coordinate ai.h2o:sparkling-water-package_2.11:3.26.8-2.4.

3- You can also run VisualVM tool for profiling and monitoring the OPAL plug-in while generating a call graph. In order to do so, follow the below steps:

  • First, set up an SSH tunnel with SOCKS:
ssh -N -v -D 9696 user@hostname

replace user and hostname with your own username and server's address, respectively.

  • In VisualVM, go to Preferences->Network and select Manual Proxy. Create a SOCKS connection at localhost and port 9696
  • To connect the VisualVM with a remote machine, go to File->AddRemoteHost. Enter the hostname of the machine and port 9696 in the advanced settings.
  • By right-clicking on your remote machine, create a JMX connection and enter the JMX port, which is 1233 in this guide.
Clone this wiki locally