DFEE: Interactive DataFlow Execution and Evaluation Kit

This repository provides the code and sample data for our paper "DFEE: Interactive DataFlow Execution and Evaluation Kit" (AAAI 2023 Demo Paper). Please cite the paper if you find the paper and code useful for your work.

@inproceedings{he2023efee
title={DFEE: Interactive DataFlow Execution and Evaluation Kit},
author={He, Han and Feng, Song and Bonadiman, Daniele and Zhang, Yi and Mansour, Saab},
booktitle={AAAI},
year={2023}
}

Installation

Run the following commands under the root folder of this project:

python3.7 -m venv venv
source  venv/bin/activate
pip install -e . --extra-index-url=https://smpypi.z5.web.core.windows.net/

Requirements

The following two opensource libraries are needed for running the code. Please follow the instruction below to clone the repository and apply the provided patch.

Open Pythonic Dataflow Dialogue Framework (OpenDF)

See more about the repo (MIT license.). Most modifications are about API implementations.

git clone https://github.com/telepathylabsai/OpenDF.git
cd OpenDF
git apply ../opendf_changes.patch
mv opendf ../src/.
cd ..

Semantic Parsing with Constrained Language Models

See more details in the repo ( MIT license) . Most modifications are related to issue 13 and supporting V1.

git clone https://github.com/microsoft/semantic_parsing_with_constrained_lm.git
cd semantic_parsing_with_constrained_lm
git apply ../semantic_parsing_with_constrained_lm_changes.patch
mv semantic_parsing_with_constrained_lm ../src/.
cd ..

This codebase has been tested on Deep Learning AMI GPU PyTorch 1.11.0 (Amazon Linux 2) 20220526 and MacOS with Python 3.7. Other OS or Python versions (shipped with Conda) have not been tested and might not work properly.

GUI

To deploy the GUI, you need to install nginx and npm then:

• Copy the config file gui/nginx.conf to /etc/nginx/nginx.conf
• Install and build the GUI package:

cd gui
npm install
npm run build

• Copy dist to /usr/share/nginx/html :

sudo cp -rf gui/* /usr/share/nginx/html

Parser

Before running any other sub-systems, please start the parser sever which is required for other sub-systems. The parser models should be downloaded and unzipped to data/models/trained_models/1.0/.

python src/aws_lex_program_synthesis_calendar/parser_server.py

Demonstration

We provide both CLI and GUI for demonstration.

CLI

Run the following command to start a calendar bot in your terminal:

python src/aws_lex_program_synthesis_calendar/bot_cli.py

GUI

Run the following command to start a calendar bot on the web:

python src/aws_lex_program_synthesis_calendar/bot_server.py

Then, start the UI server:

cd gui
npm install        # first-time installation
npm run dev

If you plan to host it on a file server (e.g., nginx), you can build a distribution version:

npm run build

Then copy the dist folder to your file server.

Benchmarking

The dialogs for benchmarking are in markdown format, e.g., data/benchmark/tiny_samples/0001.md.

To evaluate on a single file:

python src/aws_lex_program_synthesis_calendar/benchmark_cli.py data/benchmark/tiny_samples/0001.md

To evaluate multiple files:

python src/aws_lex_program_synthesis_calendar/benchmark_cli.py data/benchmark/tiny_samples/*.md

Security

See CONTRIBUTING for more information.

License

This library is licensed under the Apache 2.0 License. See the LICENSE file.