Skip to content

Commit

Permalink
Add current limitations, debugging information
Browse files Browse the repository at this point in the history
  • Loading branch information
jak-ing committed Aug 3, 2019
1 parent 816f285 commit 7c1a0ee
Showing 1 changed file with 65 additions and 10 deletions.
75 changes: 65 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,14 @@

The Polyglot Jupyter Kernel is a [kernel for Jupyter notebooks](https://jupyter.readthedocs.io/en/latest/projects/kernels.html) that can handle multiple programming languages. It's running on [GraaLVM](https://www.graalvm.org/).

### Installing
## Installing

Make sure to [install](https://www.graalvm.org/downloads) GraalVM (latest tested version is v19.0.2) and ensure GraalVM's `node` and `npm` are in your `PATH`.
Make sure to [install](https://www.graalvm.org/downloads) GraalVM (latest tested version is v19.0.2) and ensure GraalVM's `node` and `npm` are in your `PATH`:

```sh
export PATH=$GRAALVM_HOME/Contents/Home/jre/languages/js/bin:$PATH
export PATH=$GRAALVM_HOME/Contents/Home/bin:$PATH
```

To install the kernel on Linux follow these steps:
```sh
Expand All @@ -16,7 +21,7 @@ yum install -y python36 python36-devel python36-pip
# 2. install the jupyter pip package:
pip3 install jupyter_core==4.4 jupyter
# 3. install ipolyglot from source:
git clone https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel.git
git clone https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel.git
cd ./pp19-3-jupyter-kernel && npm install .
```

Expand All @@ -30,20 +35,21 @@ brew install pkg-config zeromq
# 3. install the jupyter pip package and its requirements
pip install jupyter_core==4.4 pyzmq jupyter
# 3. install ipolyglot from source:
git clone https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel.git
git clone https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel.git
cd ./pp19-3-jupyter-kernel && npm install . --python=python2.7
```

Proceed by starting a jupyter notebook server (`jupyter notebook`), opening a web-browser on `localhost:8888`, and creating a new GraalNode.js notebook.

### Running IPolyglot with Docker
## Running IPolyglot with Docker

You may also choose to run IPolyglot with docker:

```sh
# 0. download and setup docker
# 1. clone ipolyglot repository
git clone https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel.git
git clone https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel.git
cd pp19-3-jupyter-kernel
# 2. build an ipolyglot docker image
docker build --tag=ipolyglot .
# 3. run it! (choose a different port by changing 8888:<desired port>)
Expand All @@ -52,9 +58,9 @@ docker run -p 8888:8888 ipolyglot

Proceed by opening a web-browser on `localhost:8888`, and creating a new GraalNode.js notebook.

### Using different programming languages
## Using different programming languages

A different language can be used in each notebook cell. Simply specify the desired language as a [magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html) in the first line of the cell:
A different language can be used in each notebook cell. Simply specify the desired language as a [magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html) in the first line of the cell. Available language codes are `python`, `js`, `ruby`, `R`.

```ruby
%polyglot [language code]
Expand All @@ -70,9 +76,11 @@ for example:
my_dict = dict()
```

See the `Demo.ipynb` for a detailed example.

Please be aware that you can use languages that have been [installed to your GraalVM](https://www.graalvm.org/docs/reference-manual/graal-updater/) only.

### Inspecting Polyglot variables
## Inspecting Polyglot variables

The Polyglot Jupyter Kernel shares all available global variables across all notebook cells. It provides an inspector extension to the Jupyter notebook frontend that lists all polyglot variables available.

Expand All @@ -89,6 +97,53 @@ jupyter nbextension enable varInspector/main

Restart your notebook and toggle the variable inspector with the <kbd>🎯</kbd> button in top toolbar.

### Acknowledgments
## Debugging the IPolyglot Kernel

This describes how to debug the IPolyglot Kernel, _not_ how to debug code executed in the code cells.

The `jupyter notebook` command can be invoked like this `DEBUG=true jupyter-notebook --debug` to get debugging information about the messages sent between the kernel and the jupyter notebook.

However, it has often been more helpful to pass debug flags to the GraalNode.js process which actually evaluates the code of Jupyter Notebook cells. This is done in node_modules/nel/lib/nel.js where `Session._args` is defined. There is a commented out line which already includes the necessary flags and can be commented in if need be. Then, a `debugger;` statement can be set anywhere in the NEL code and the Node.js session can be found in [chrome://inspect/#devices](chrome://inspect/#devices) for debugging.

## Current Limitations

### [Output and error handling](https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel/issues/13)

Currently, if `print` / `put` / ... is called in a language other than JavaScript inside a Jupyter Notebook code cell, the print output is not displayed in the frontend of the Jupyter Notebook. Similarly, if an error occurs during execution of the code of a cell, the error might not be properly propagated to the frontend for display to the user.

We've tried a bunch of things to make output and error handling work but have not completely succeeded. Below we want to document two approaches that seem most promising.

#### Approach 1 (more preferable option)

Branch: https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel/tree/feature/output_handling_polyglot_context

This approach creates a `org.graalvm.polyglot.context polyglotContext` in node_modules/nel/lib/server/main.js and passes a `ByteArrayOutputStream polyglotStdout` to it. Whenever `polyglotContext.eval` is called now, print outputs of the executed code is written to `polyglotStdout`. Later on, `polyglotStdout`'s contents can be easily accessed and redirected to the frontend. See short code example here: https://github.com/hpi-swa-lab/pp19/issues/19#issuecomment-514649627

Regarding error handling: If an error occurs during the call to `polyglotContext.eval`, the error is caught and the original error message from the guest language is redirected to the frontend (`initialContext.console.error(codeCellStderr)`).

The issue with this approach is that it doesn't work together at all with our automatic import/export feature: To make the automatic import/export work, we use `Polyglot.eval`, `Polyglot.import` and `Polyglot.export`. However, once we use our custom `polyglotContext.eval` (to capture the stdout properly), the code is evaluated in another context than the context we access using `Polyglot.*`. Therefore, we e.g. can't export variables defined in `polyglotContext.eval` using `Polyglot.import`. According to the [docs](https://www.graalvm.org/sdk/javadoc/org/graalvm/polyglot/Context.html#getPolyglotBindings--), we might be able to work around this by doing `polyglotContext.getPolyglotBindings.import()` / `...export()`. However, calling `polyglotContext.getPolyglotBindings` results in [an error](https://github.com/hpi-swa-lab/pp19/issues/20). If the issue can be solved, this approach seems to be most appropriate and clean.

#### Approach 2 (less preferable option)

Branch: https://github.com/hpi-swa-lab/pp19-3-jupyter-kernel/tree/feature/output_handling

This approach redirects the output from the code evaluation process to a log file and reads from it to send it to the Jupyter Notebook frontend. The solution works, however the lag can be pretty big at times and once you try to restart the kernel from the Jupyter Notebook frontend, the kernel becomes completely unresponsive and you have to restart the entire Jupyter Notebook server process and kill an orphaned/zombie (?) Node.js process manually. We suspected this might be due to a missing close or end call to the file write stream but we checked this and it doesn't seem to be the underlying issue.

We assume that the same approach can be taken to handle errors raised during code cell execution as well but have not tested this yet.

In node_modules/nel/lib/nel.js, a write stream `tmp_stdout` is created which writes into a file 'tmp_stdout.log'. This write stream is specified as the stdout handle for the code evaluation process via `this._config = { stdio: [<stdin>, tmp_stdout, ...] }`.

The code evaluation process itself defines a `stdout_tail` in node_modules/nel/lib/server/main.js which listens to any output redirected to the 'tmp_stdout.log' file and sends it to the Jupyter Notebook frontend.


## Issues we opened in GraalVM during work on this project

- https://github.com/hpi-swa-lab/pp19/issues/19
- https://github.com/hpi-swa-lab/pp19/issues/20
- https://github.com/graalvm/graaljs/issues/173 (closed)
- https://github.com/graalvm/graaljs/issues/184 (closed)
- https://github.com/graalvm/graaljs/issues/185

## Acknowledgments

This projects builds on the tremendous work of [Nicolas Riesco](https://github.com/n-riesco)'s [IJavascript](https://github.com/n-riesco/ijavascript), a Javascript kernel for Jupyter notebooks.

0 comments on commit 7c1a0ee

Please sign in to comment.