Run code line by line

[liu-genomics]

I am trying to develop some scripts in Jupyterlab-sos so I need to run and test code line by line. I have two blocks of code here. The first one is global variable definition, second one is something I need to run in bash.

When I ran line by line. It seems the variables were not recognized. How should I deal with this situation? I used ctrl+shift+enter to run line by line in a jupyterlab-sos browser interface.

[BoPeng]

Did you run the east_snv line in [global] section? I would usually step through the parameter section to get the parameter initialized

[gaow]

@BoPeng I confirm it works using a MWE simlar to @liu-genomics posted where codes live in different cells:

However as you can see the print(_input) statement did not quite work.

[gaow]

Also, see below, line-by-line execution does not seem to work for scripts inside bash action

the first ls does not list the proper input (should be a single file not a number of them). The 2nd echo did not work. I've not used this feature myself to interactively develop codes inside actions in SoS Notebook so not sure if it is user error or something we do not support?

[BoPeng]

The first problem is jupyterlab/jupyterlab#6606 , which might or might not be accepted because its negative impact on indent-less languages such as R. Right now you will have to manually select the entire bash block, or use classic jupyter for which I have implemented 6606.

The _input problem exists because _input is only defined when the workflow is executed with a input: statement so you will have to execute the entire cell or at least include the input: line with your selection.

I could make _input persistent (e.g. after you step through an input statement), but think the case if you are running content with _input in another cell.

[gaow]

I could make _input persistent (e.g. after you step through an input statement), but think the case if you are running content with _input in another cell.

Not sure if I understand this statement. In the context of interactive development we always would expect running the current cell based on content previously executed -- that case I've not seen an issue with persistent _input. Would you elaborate with an example?

The first problem is jupyterlab/jupyterlab#6606 , which might or might not be accepted because its negative impact on indent-less languages such as R

Okay let's wait for the pull request on JupyterLab's end. In the mean time @liu-genomics if you want to do that within scripts you've got to fall back to jupyter-notebook based IDE.

[BoPeng]

Would you elaborate with an example?

The idea is that _input is supposed to be a transient variable only seen inside a step, and can change for each substep. So suppose you step through

input: 'a.txt', 'b.txt', group_by=1

SoS could ignore the line because there is no body of substep, or it could be the last _input, which should be file_target('b.txt').

Currently _input is reset right after you step through the line so that

print(_input)

would fail or print `` for an empty sos_targets (not sure which is the case).

You are right that we usually will not jump between the cells and use _input of one cell in another, but I am not quite comfortable to pass a hidden variable around.

[BoPeng]

You could comment on jupyterlab/jupyterlab#6606 if you have some idea what the behavior should be.

[gaow]

You are right that we usually will not jump between the cells and use _input of one cell in another, but I am not quite comfortable to pass a hidden variable around.

I agree with you that in SoS it does not make sense to access _input in a cell separate from where it is defined. However it seems this constraint has the side effect to line-by-line execution as I've shown above. Removing this contraint would then make this variable accessible by any other cells in the notebook, something not harmful given the context (my opinion) but is uncomfortable (your view). Is
there a middle ground to this?

BTW @liu-genomics when I develop workflow using SoS Notebook I often use a language specific kernel to develop the chunk of code in that language; then after it is confirmed to work I cut-paste those into sos cell putting in the workflow bells and whistles. It is slightly more tedious than what you are trying to do (developing all in SoS cell) but to me it is cleaner in a sense.

[liu-genomics]

Thanks very much Bo and Gao. @gaow Could you please explain a bit more what do you mean by "when I develop workflow using SoS Notebook I often use a language specific kernel to develop the chunk of code in that language". Do you also have variables in this context? For example, in shell script, we could write a test.sh with only one line of code ls $1. If I need to develop such code line-by-line in SoS Notebook, how should I write?

[gaow]

Do you also have variables in this context?

No. I would hard code input data variables etc to a separate language kernel, test the code make sure it works, then reformat that cell to an SoS workflow step -- this is how I usually work with Python and R; and in my view it is quite straightforward. For developing bash scripts I do use variables but I did not realize it does not work with JupyterLab -- last time I did it was long ago using Jupyter Notebook. I'm afraid for that purpose you've got to use Jupyter Notebook until they accepted our PR.

[BoPeng]

Right now in sos you can use parameter args to specify parameters to shell (and other) actions, and SoS Notebook in classic jupyter can pick up the entire shell script. The behavior in JupyterLab is different in that line by line execution will only pick up the first line after the sh line, but hopefully jupyterlab/jupyterlab#6606 will be accepted soon.

[liu-genomics]

Thanks! Hopefully Jupyterlab would accept 6606 soon. Would love to have line-by-line execution with variables.

[BoPeng]

@gaow It is possible to retain _input after processing input: statement, but how about output:?

output: _input.with_suffix('.bak')

might work all right, but

output: whatever, group_by=1

would fail miserably because there is no information on number of substeps. I guess the bottomline is that

output: whatever, group_by=1

would not work in the current implement either so it is not going worse.

[gaow]

Well, at least in my workflows outputs are mostly derived from inputs. I think it is good enough to support this scenario. We did not stress heavily on line-by-line development for SoS Workflows but it is good to see there is a need for it.

[BoPeng]

I have pushed some changes to allow persistent step_input, _input etc. It roughly works but as you can see, there will be a warning message saying _output does not exist if you step through output statement.

I am willing to make the change because it allows easier debugging of actions involving _input and _output, also because it facilitates writing complex input and output statements because users can know what _output would be even if it reflects only one of the substeps.

[BoPeng]

Unfortunately jupyterlab/jupyterlab#6606 was rejected so the behavior will be different between classic Jupyter and JupyterLab unless we change the behavior on the classic jupyter end..

[BoPeng]

I think this also gives us an opportunity to fine tune the behavior. That is to say, we can add a property in our language module and limit the "full block" behavior only to Python and SoS.

[gaow]

@BoPeng is the line-by-line execution feature broken? I cannot seem to get it work now with latest JupyterLab, although Jupyter Notebook works:

when I hit ctrl shift enter nothing happens.

[BoPeng]

That is now a jupyterlab feature. I think I enabled the shortcut but perhaps I did not. Check if run in console from menu works.

[gaow]

Sorry I cannot seem to find that "run in console" in menu ... yes it would be nice if we are consistent with Jupyter Notebook IDE for SoS. I'm on the current pip release of Jupyterlab version 1.1.4

[BoPeng]

It appears to be working, but not at first when the SoS kernel has not been started, which by itself could be a bug.

[benbenz]

@BoPeng is the line-by-line execution feature broken? I cannot seem to get it work now with latest JupyterLab, although Jupyter Notebook works:

when I hit ctrl shift enter nothing happens.

I am not sure this is related the main issue here but I've noticed a very similar issue to the one @gaow is seeing. It is visible at http://sosworkflows.com/

If you start a new SoS console and type anything, nothing will happen. When running locally, I also see the following error:

18:05:08.664 › TypeError: Cannot read property 'model' of null
    at new v (http://localhost:8888/lab/extensions/jupyterlab-sos/static/5.e9a19b6a8ff02c8ed090.js?v=e9a19b6a8ff02c8ed090:1:9450)
    at S.get_info (http://localhost:8888/lab/extensions/jupyterlab-sos/static/5.e9a19b6a8ff02c8ed090.js?v=e9a19b6a8ff02c8ed090:1:11919)
    at http://localhost:8888/lab/extensions/jupyterlab-sos/static/5.e9a19b6a8ff02c8ed090.js?v=e9a19b6a8ff02c8ed090:1:31935
...

I found this solution: create an empty SoS notebook then create the SoS console, and it should work.

This seems to be caused by Manager.currentNotebook being null in jupyterlab-sos labextension.

• line 1007 of index.ts:

      labconsole.promptCellCreated.connect(panel => {
        let info = Manager.manager.get_info(Manager.currentNotebook);
        addLanSelector(panel.promptCell, info);
      });

• into line 200 of manager.ts:

public get_info(notebook: NotebookPanel): NotebookInfo {
    if (!this._info.has(notebook)) {
      console.log("Creating a new notebook info");  
      this._info.set(notebook, new NotebookInfo(notebook));  
    }
    return this._info.get(notebook);  
  }

• into line 44 of manager.ts (with notebook=null):

if (notebook.model.metadata.has("sos"))

[BoPeng]

Thanks for the report. I will have a look.

[benbenz]

Great ! Thanks. And thank you for making such a great extension like SoS.
I just realized the console was probably meant to be associated with a notebook. But, as one can create a standalone console from the launcher, it would be great if it worked autonomously. I tried to modify the code to create a hidden NotebookPanel or separate the formalism betweeen notebooks and consoles (more involved in terms of code) with no luck so far. It's hopefully an easy fix for you :)

[BoPeng]

I pushed a simple fixes and the console appears to be working. However, because the console cannot communicate with the backend to get a list of kernels, it does not have the language dropdown so the console is not fully operational.

I suppose with some more work we can get the language dropdown working for this case but maybe it does not worth the trouble.

[benbenz]

Thanks !
A fully featured console could be interesting down the line: to be able to combine the same feel as the original sub-kernels' software with SoS kernel selection. But this is already fully functional with the magic words so this is probably good for now !