Monitor parallel processes/tasks with prefix in output

[7tupel]

Hi everyone,

I would like to run several processes in parallel from a bb script. All processes produce real time output which I would like to print in real time using std out/err for monitoring similar to this discussion.

To run multiple dependent tasks in parallel I simply use the bb taskrunner. Now the question: Is there a way to inject a marker for each process output at the beginning of each line that is written to out/err by a process?

I.e. i have two tasks, each tasks spawns a process like (shell {:inherit true} "binary-A") and (shell {:inherit true} "binary-B") where each process is long running and writes its monitoring output to stdout. Instead of having both process outputs mixed in the std I would like to get something like

[A] doing some great work
[A] still doing work
[B] I'm skynet, I'm taking over the world!
[A] ....

Ideally with the ability to set a specific color for each marker.

I guess It should be possible by piping the output of one process to another and modifying the output/stream content in the middle. But in my opinion this is quite an overhead and not very elegant. I also tried to use the streaming output example as a starting point but for some reason I'm not getting anything printed to my terminal at all. Still I guess that might be the way to go. Or is there any better way that I'm missing?

[borkdude]

We have an issue for that here: #54

[borkdude]

The approach that I would try is to work with the out stream directly and use a future to read lines and then prefix those lines and print them again.

[borkdude]

@7tupel Here is a demo of how it might work:

(ns proc-demo
  (:require [babashka.process :as process :refer [process]]
            [clojure.java.io :as io]))

(defn output-wrapper [proc prefix]
  (let [stream (:out proc)
        rdr (io/reader stream)
        lines (line-seq rdr)]
    (run! #(println prefix %) lines)))

(let [proc1 (process "ls")
      proc2 (process "ls -la")]
  (future (output-wrapper proc1 "[a]"))
  (future (output-wrapper proc2 "[b]"))
  @proc1
  @proc2)

[7tupel]

Thank you, I'll try it late today.

The project is already public: rum-shadow-boiler The related code can be found in resources/tree/bb/tasks.clj (the template to create a clojurescript shadow-cljs project) and then in the created project in bb/tasks.clj. Probably the easiest to create an example project using the description from the Readme and then work in the new project instead of the template files, otherwise you have to create a new project every time after a change and re-initialize the node modules in that project which takes a moment. When you do that make sure to run bb prepare as first thing in the newly created project (docs are still missing mostly).

[borkdude]

I tried to reproduce it with one of my own shadow-cljs projects:

https://github.com/babashka/nbb/tree/proc-wrapper

Run bb dev to see it in action:

Starting shadow-cljs in watch mode.
Run node lib/nbb_main.js to test nbb
Run bb run-tests to run the tests
[shadow-cljs] No config for build "" found.
[shadow-cljs] [2023-02-07 14:19:27.991 - INFO] duplicate resource runners/karma.conf.js on classpath, using file:/Users/borkdude/.gitlibs/libs/reagent/reagent/315d702436719ea3a9a356773d6a10038ce9ea75/test/runners/karma.conf.js over jar:file:/Users/borkdude/.m2/repository/doo/doo/0.1.11/doo-0.1.11.jar!/runners/karma.conf.js
[shadow-cljs] shadow-cljs - server version: 2.20.16 running at http://localhost:9630
[shadow-cljs] shadow-cljs - nREPL server started on port 59839
[shadow-cljs] shadow-cljs - watching build :modules
[shadow-cljs] [:modules] Configuring build.
[shadow-cljs] [:modules] Compiling ...
[shadow-cljs] [:modules] Build completed. (225 files, 0 compiled, 0 warnings, 3,47s)

I don't see any lines missing compared to regular output.

[7tupel]

It works when I use the unbuffered version you put in the nbb repo. Also works with setting a terminal color on the prefix.

Thanks a lot, this will improve readability greatly when multiple processes run in the same terminal :)

[borkdude]

Cool, thanks for being my guinea pig here. This may form the basis for #54

[7tupel]

No problem :)

You can find (a quick and dirty) version with the coloring in this PR with the code setting the colors in this repo.