jsfbp that doesn't use fibers?

[tlrobinson]

Fibers are neat, but the majority of node.js developers don't and won't use them, and they don't work in browsers at all. It would be nice if we could provide a "classical FBP" system that works using standard JavaScript APIs.

Actually, after reading http://www.jpaulmorrison.com/fbp/noflo.html I'm pretty confused about how JSFBP is considered more "classical" than NoFlo. That page suggests the key difference is that NoFlow is single-threaded (because JavaScript is single-threaded) and thus not suitable for CPU-intensive programs, but then admits JSFBP is also single-threaded. So is the main difference actually the synchronous vs asynchronous style of programming?

Selective reading of ports and backpressure can both be accomplished in an asynchronous style. For example, Node-style Streams have pause and resume methods for handling backpressure. The port could buffer up to a fixed number of IPs then call pause on the stream when the buffer fills up. The port could have a read method that takes a callback (or returns a promise) which gets called when the next IP is available, and calls resume on the stream if the buffer is no longer.

To take advantage of multiple cores you could use something like WebWorkers, though I think a generic system based on a standard IPC mechanism (e.x. ZeroMQ) and serialization protocol (e.x. Protocol Buffers) would be even cooler, allowing you to write components in any language, or even distribute processes across machines.

But anyway, there are several options for sync and async styles in JavaScript:

Asynchronous style:

1. EventEmitters or Streams (see above)
2. raw callbacks: probably Node-style callback(err, result) callbacks. This is my least favorite option, but is still the most popular style in Node.js. I probably wouldn't bother with this.
3. promises: promises are gaining a lot of traction in the JavaScript world lately, but they're still a very asynchronous style

Synchronous style:

1. generators (ES6, can be transpiled to ES5): not available in browsers except via transpiling
2. await (ES7, can be transpiled to ES5): lets you write synchronous style code that awaits on promises
3. some other synchronous-style transpiler: e.x. streamline.js

Also this may be useful relevant reading: https://github.com/kriskowal/gtor

EDIT: task.js is another possibility: http://taskjs.org/

[tlrobinson]

Also, I would argue that this is incorrect:

JSFBP only requires the addition of a small package on top of node.js, node-fibers, which is easy to install. Even though it does not use pure JavaScript, neither does node.js, but the latter is becoming more and more accepted.

(http://www.jpaulmorrison.com/fbp/software.html#JSFBP)

A full Node.js environment could be emulated in the browser (and indeed "browserify" attempts to do this), but node-fibers can't be easily emulated in the browser because it introduces "blocking" semantics (something similar can be implemented once ES6 generators are available, though: https://stackoverflow.com/questions/18293563/can-node-fibers-be-implemented-using-es6-generators)

[alfa256]

Honestly I can't see how an API that looks like JavaFBP and other classical implementations could be made with traditional Javascript. If you can find a way please tell.

This kind of api:

while( (a = read("in")) !== null ){
   send( "out", a.value + 5 )
}

Blocking is not a problem but a feature, and I'm not really convinced that generators can emulate this behavior. I hope it does though, I simply couldn't find a reasonable way.

[tlrobinson]

Here's a simple example (using the Bluebird promise library and ES6 generators, available in iojs and recent versions of node with the "--harmony" flag)

(obviously read and send are fake versions of the real thing, but it should give you an idea of what's possible)

var Promise = require("bluebird");

var messages = [1, 2, 3];
function read(port) {
  return Promise.delay(1000).then(function() {
    if (messages.length > 0) {
      return { value: messages.shift() }
    } else {
      return null;
    }
  });
}

function send(port, value) {
  console.log("sending to port '" + port + "': " + value)
}

add5 = Promise.coroutine(function *() {
  var a;
  while( (a = yield read("in")) !== null ){
    send( "out", a.value + 5 )
  }
});

add5()

[jpaulm]

Hi @tlrobinson, I wrote http://www.jpaulmorrison.com/fbp/noflo.html
before I got the idea of developing JSFBP, so it's not surprising that it
doesn't reflect my thinking about JSFBP. At the time I didn't think it was
possible to build a "classical" FBP system using JS, and NoFlo is certainly
very different from classical FBP. It seems to have a definite niche,
but it's a different one from that of classical FBP.

As you may be aware, I have been working with classical FBP for 40+ years,
and much of that time it was being used to run most of the batch code for a
major bank (using a green thread "classical" FBP implementation). So I
believe I understand it quite well, and I have a lot of faith in it.

Around Christmas, my son suggested I look at return and yield, and, while
researching that, I came across a reference to Marcel Laverdet's
node-fibers in @bjouhier's excellent blog, Bruno's Ramblings -
https://bjouhier.wordpress.com/ . In it he compares a number of JS
approaches to getting rid of "callback hell", and the thing that stood out
to me about node-fibers is that it supports multiple stacks, which in my
experience is key to building a "classical" FBP implementation.

I did not see another approach that was as promising, and in fact, using
node-fibers, it took me no more than a few days to come up with a very
promising proof of concept, which I have been developing since. Your
feedback and @ComFreek's have been very useful as my JavaScript skills are
pretty rudimentary!

I agree that node-fibers is a bit further away from plain JS than nodejs,
say, but to me it still looks like the most promising technology for taking
advantage of JS. We have discussed worker threads, and the consensus is
that the overhead is too high and communication between worker threads too
"narrow-band". I have also been warned to avoid Promises!

If people are willing to add nodejs to basic Javascript, I fail to see why
they would be so unwilling to add one more component - namely
node-fibers. You raise a good point about not being able to browserify
node-fibers, but maybe there are other solutions to getting JSFBP in the
browser - however, I see no immediate urgency to solve this problem. I
will be quite happy if JSFBP can supports servers, providing good
performance and good maintainability. This at least gives users a single
language on both client and server - even if different mindsets!

We have used back pressure, selective receive, and information packets with
defined lifetimes and ownership as sort of a litmus test for classical FBP

• if you can provide these functions with the tools you mention, and decent
  performance and maintainability, I would be very interested in seeing your
  solutions. A number of people have come up with FBP-like approaches over
  the last few decades, and I'm looking forward to seeing how these all work
  out, as we gain a deeper understanding of this technology. It might be
  timely to try to come up with a checklist of success factors, so we can
  measure the different offerings against it. I can tell you one thing,
  though: once you have made the paradigm shift described in my book, you
  can't go back! And I would venture to say that, if you can go back, you
  haven't made the paradigm shift! :-) That's 40+ years of experience
  talking!

Best regards, and many thanks,

Paul M.

On Feb 11, 2015 5:40 PM, "Tom Robinson" [email protected] wrote:

Fibers are neat, but the majority of node.js developers don't and won't
use them, and they don't work in browsers at all. It would be nice if we
could provide a "classical FBP" system that works using standard JavaScript
APIs.

Actually, after reading http://www.jpaulmorrison.com/fbp/noflo.html I'm
pretty confused about how JSFBP is considered more "classical" than NoFlo.
That page suggests the key difference is that NoFlow is single-threaded
(because JavaScript is single-threaded) and thus not suitable for
CPU-intensive programs, but then admits JSFBP is also single-threaded. So
is the main difference actually the synchronous vs asynchronous style of
programming?

Selective reading of ports and backpressure can both be accomplished in an
asynchronous style. For example, Node-style Streams have pause and resume
methods for handling backpressure. The port could buffer up to a fixed
number of IPs then call pause on the stream when the buffer fills up. The
port could have a read method that takes a callback (or returns a
promise) which gets called when the next IP is available, and calls resume
on the stream if the buffer is no longer.

To take advantage of multiple cores you could use something like
WebWorkers, though I think a generic system based on a standard IPC
mechanism (e.x. ZeroMQ) and serialization protocol (e.x. Protocol Buffers)
would be even cooler, allowing you to write components in any language, or
even distribute processes across machines.

But anyway, there are several options for sync and async styles in
JavaScript:

Asynchronous style:

1. EventEmitters or Streams (see above)
2. raw callbacks: probably Node-style callback(err, result) callbacks.
   This is my least favorite option, but is still the most popular style in
   Node.js. I probably wouldn't bother with this.
3. promises: promises are gaining a lot of traction in the JavaScript
   world lately, but they're still a very asynchronous style

Synchronous style:

1. generators (ES6, can be transpiled to ES5): not available in
   browsers except via transpiling
2. await (ES7, can be transpiled to ES5): lets you write synchronous
   style code that awaits on promises
3. some other synchronous-style transpiler: e.x. streamline.js

Also this may be useful relevant reading:
https://github.com/kriskowal/gtor

—
Reply to this email directly or view it on GitHub
#14.

[ComFreek]

That's why I suggested to use an abstraction layer in issue #13. Such an abstraction layer would ideally allow us to switch between different implementations, e.g. one for Node.js and another one for browsers. Another big and related advantage is that the implementation for browsers could only emulate Fibers, so that jsfbp is ready to run in browsers even though not with full performance.

[tlrobinson]

If people are willing to add nodejs to basic Javascript, I fail to see why
they would be so unwilling to add one more component - namely
node-fibers

This is simply not going to happen. AFAIK there has been zero serious discussion of adding fibers to the browser, and even if that started happening it would be many years before they were implemented and widely available.

Compare that to ES6's generators, which are built into current versions of Node.js, in the next version of JavaScript, so likely to start showing up in browsers soon. As you can see from my example above, you can get you something pretty similar to fibers. It's possible there's some reason why you can't implement classical FBP with generators instead of fibers, but I don't know yet (note that Bruno also mentioned generators and preprocessors like Streamline.js as a solution to callback hell)

[alfa256]

Could you try making a prototype that achieves a functionality equivalent
to Collate ?

2015-02-12 13:26 GMT-03:00 Tom Robinson [email protected]:

If people are willing to add nodejs to basic Javascript, I fail to see why
they would be so unwilling to add one more component - namely
node-fibers

This is simply not going to happen. AFAIK there has been zero serious
discussion of adding fibers to the browser, and even if that started
happening it would be many years before they were implemented and widely
available.

Compare that to ES6's generators, which are built into current versions of
Node.js, in the next version of JavaScript, so likely to start showing up
in browsers soon. As you can see from my example above, you can get you
something pretty similar to fibers. It's possible there's some reason why
you can't implement classical FBP with generators instead of fibers, but I
don't know yet (note that Bruno also mentioned generators and preprocessors
like Streamline.js as a solution to callback hell)

—
Reply to this email directly or view it on GitHub
#14 (comment).

[jpaulm]

If you can find a way to simulate multiple stacks in JS, then we should be
able to do something!
On Feb 12, 2015 11:59 AM, "alfa256" [email protected] wrote:

Could you try making a prototype that achieves a functionality equivalent
to Collate ?

2015-02-12 13:26 GMT-03:00 Tom Robinson [email protected]:

If people are willing to add nodejs to basic Javascript, I fail to see
why
they would be so unwilling to add one more component - namely
node-fibers

This is simply not going to happen. AFAIK there has been zero serious
discussion of adding fibers to the browser, and even if that started
happening it would be many years before they were implemented and widely
available.

Compare that to ES6's generators, which are built into current versions
of
Node.js, in the next version of JavaScript, so likely to start showing up
in browsers soon. As you can see from my example above, you can get you
something pretty similar to fibers. It's possible there's some reason why
you can't implement classical FBP with generators instead of fibers, but
I
don't know yet (note that Bruno also mentioned generators and
preprocessors
like Streamline.js as a solution to callback hell)

—
Reply to this email directly or view it on GitHub
#14 (comment).

—
Reply to this email directly or view it on GitHub
#14 (comment).

[tlrobinson]

Can Collate be written in the current version of JSFBP (using fibers)?

[jpaulm]

Absolutely! The structure and even coding would be very similar to https://github.com/jpaulm/jsfbp/blob/master/components/rrmerge.js - you just have to add the "low tag" logic (output the IP with the lowest tag value, and then do a 'receive' from the input array port element that that IP came from).

Historically, Collate also adds open and close brackets to the output stream - this function could be packaged as a separate component, but it uses the same control fields as Collate, so it seemed practical to combine the two functions.

[tlrobinson]

In my quest to understand collate I implemented a version without brackets here: #17

I'll work on proof-of-concept with generators/promises/streams next.

[tlrobinson]

@jpaulm @alfa256 Here's a proof-of-concept using Streams, Promises, and ES6 Generators via Blubird's coroutine method: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400

It may be difficult to understand if you're not very familiar with JavaScript and "promises" or "streams", but the relevant thing to note is I've implemented collate without using Fibers. Note that collate1 uses Generator functions, which will be available in ECMAScript 6, and I believe can be supported in older versions using a "transpiler". collate2 only uses Promises and should run in all browsers today, without any transpiling.

collate1 is a line-by-line port of "collate" that I implemented in JSFBP (https://github.com/jpaulm/jsfbp/blob/master/components/collate.js), the only changes are the addition of the yield statement in front of asynchronous operations (i.e. receive) and the "function *" syntax to make it a "generator function"

collate2 is admittedly less clear because it's written in an asynchronous style using Promises directly, but it proves "collate" can be implemented in JavaScript without fibers or even generators: https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L98-L131

Note that I'm using Streams in "paused mode" (http://nodejs.org/api/stream.html)

Readable streams have two "modes": a flowing mode and a paused mode. When in flowing mode, data is read from the underlying system and provided to your program as fast as possible. In paused mode, you must explicitly call stream.read() to get chunks of data out. Streams start out in paused mode.

I've added a receive method to readable streams which returns a promise. You can use the promise directly (as I do in collate2) or yield it within a Bluebird coroutine if you prefer a more synchronous style (as I do in collate1).

I think basing a FBP system on Streams would be powerful because you can implement a component in whichever style makes the most sense. From the outside it doesn't matter. For example, copier could be implemented as InputPort.openInputPort('IN').pipe(InputPort.openOutputPort('OUT'));

Streams also support "backpressure" so they can communicate to their upstream processes when they do or don't want more data: http://nodejs.org/api/stream.html#stream_event_drain

Also note that I'm using streams in "object mode" http://nodejs.org/api/stream.html#stream_object_mode but normal mode might be interesting in some cases.

[alfa256]

So from what I can see the api changes would be adding a yield to
receive()s ?
If that's the case then It'd be great, also I wonder if the yield could be
even wrapped in the receive() ?
In any case great job, this could be great for the project.

2015-02-17 16:47 GMT-03:00 Tom Robinson [email protected]:

@jpaulm https://github.com/jpaulm @alfa256 https://github.com/alfa256
Here's a proof-of-concept using Streams, Promises, and ES6 Generators via
Blubird's coroutine method:
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400

It may be difficult to understand if you're not very familiar with
JavaScript and "promises" or "streams", but the relevant thing to note is
I've implemented collate without using Fibers. Note that collate1 uses
Generator functions, which will be available in ECMAScript 6, and I believe
can be supported in older versions using a "transpiler". collate2 only
uses Promises and should run in all browsers today, without any transpiling.

collate1
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L62-L101
is a line-by-line port of "collate" that I implemented in JSFBP (
https://github.com/jpaulm/jsfbp/blob/master/components/collate.js), the
only changes are the addition of the yield statement in front of
asynchronous operations (i.e. receive) and the "function _" syntax to
make it a "generator function"
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function_

collate2
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L104-L151
is admittedly less clear because it's written in an asynchronous style
using Promises directly, but it proves "collate" can be implemented in
JavaScript without fibers or even generators:
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L98-L131

Note that I'm using Streams in "paused mode" (
http://nodejs.org/api/stream.html)

Readable streams have two "modes": a flowing mode and a paused mode. When
in flowing mode, data is read from the underlying system and provided to
your program as fast as possible. In paused mode, you must explicitly call
stream.read() to get chunks of data out. Streams start out in paused mode.

I've added a receive
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L32-L48
method to readable streams which returns a promise. You can use the promise
directly (as I do in collate2) or yield it within a Bluebird coroutine if
you prefer a more synchronous style (as I do in collate1).

I think basing a FBP system on Streams would be powerful because you can
implement a component in whichever style makes the most sense. From the
outside it doesn't matter. For example, copier could be implemented as
InputPort.openInputPort('IN').pipe(InputPort.openInputPort('OUT'));

Streams also support "backpressure" so they can communicate to their
upstream processes when they do or don't want more data:
http://nodejs.org/api/stream.html#stream_event_drain

Also note that I'm using streams in "object mode"
http://nodejs.org/api/stream.html#stream_object_mode but normal mode
might be interesting in some cases.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[tlrobinson]

@alfa256 I don't believe yield can be moved inside of receive because it's a special keyword for use in generator functions (function* whatever(){}), but I don't think that's desirable anyway. Ideally the FBP kernel/framework should work without generator functions, but would support them as an extra layer of syntactic sugar in components.

Note that the only code in my proof of concept that's aware of generator functions or yield is the component itself. I think it should stay that way, but it's up for debate.

[brodavi]

Hi! Interested bystander here. I just finished reading the fbp book, and this jsfbp project excites me.

For what it's worth, I would like to second the motion for using streams. Streams have a large "following" (node, gulp, etc) and are relatively well understood even by pedestrian js developers like myself.

I share @tlrobinson's concern that node-fibers are off-putting. Streams, on the other hand, I can rally behind. Plus they are available in the browser today with a choice of packages, and hopefully native support soon.

Of course, there may exist very good reasons against using streams here, but my current (admittedly limited) understanding does not see any.

https://highlandjs.org/
https://github.com/substack/stream-handbook
https://streams.spec.whatwg.org

[ComFreek]

The proof of concepts look _promis_ing!
Note that Node.js 12.0 doesn't support generator functions (unless compiled with --harmony), so I switched to io.js in order to run your sample collate.js. It is essentially a Node.js fork with faster release cycles.

I have finished my changes which separate all Network-related thing from those related to (Fiber-)Runtime.
I would be interested in whether streams or promises can be easily integrated, given the new separated Network/Runtime architecture. A Runtime (stored in core/runtimes/[name]/index.js) is a class which must implement these functions:

• pushToQueue(process): Pushes a process to the Queue
• run(processes, options, callback)
• setCallbackPending(callback)
• getCurrentProc()

Admittedly, the last two methods seem to still heavily rely on Fibers. I will look into their use cases and investigate.

I have read that Promise implementations might tend to create memory leaks (just search for "Promises memory leak" or "Promises garbage collection"). This is especially related to unresolved promises and/or errors. There is a very long discussion titled Promises/A+ implementations can lead to resource leaks in Node.

[tlrobinson]

I've expanded my proof-of-concept into a fully functional prototype: https://github.com/tlrobinson/sbp

Like the PoC, it uses Streams, as well as a combination of Promises, Generator functions, and Bluebird's coroutine (or Q's async) method to achieve a synchronous style similar to existing cFBP (classical FBP) implementations, including jsfbp.

I'd be curious to hear from the experts whether they'd consider this true "classical-FBP", and if not, why not?

[tlrobinson]

@jpaulm @alfa256 @ComFreek @brodavi @kenhkan Any thoughts on this?

[jpaulm]

Collate1 looks interesting - it seems strange that you can't "hide" the
yield, but I guess we can just hold our noses!

What I am curious about is whether you ran a complete test case, e.g.
fbptest01 with only the two changes you mention wrt collate1, and if so how
did it perform relative to the version on JSFBP (
https://github.com/jpaulm/jsfbp/ ).

Collate2 looks much more alien to me - it even seems to have a recursive
loop in it - or did I miscount the brackets?!

@brodavi You said, " I share @tlrobinson's concern that node-fibers are
off-putting" . From a component or network builder's point of view, the
only sign of fibers is one "require" - I don't think that should be
"off-putting". The lack of support for fibers in browsers is apparently
much more of a concern, but this will probably be remedied before long if
and when fibers catches on!

Regards,

Paul M.

@jpaulm https://github.com/jpaulm @alfa256 https://github.com/alfa256
Here's a proof-of-concept using Streams, Promises, and ES6 Generators via
Blubird's coroutine method:
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400

It may be difficult to understand if you're not very familiar with
JavaScript and "promises" or "streams", but the relevant thing to note is
I've implemented collate without using Fibers. Note that collate1 uses
Generator functions, which will be available in ECMAScript 6, and I believe
can be supported in older versions using a "transpiler". collate2 only uses
Promises and should run in all browsers today, without any transpiling.

collate1
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L62-L101
is a line-by-line port of "collate" that I implemented in JSFBP (
https://github.com/jpaulm/jsfbp/blob/master/components/collate.js), the
only changes are the addition of the yield statement in front of
asynchronous operations (i.e. receive) and the "function _" syntax to make
it a "generator function"
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function_

collate2
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L104-L151
is admittedly less clear because it's written in an asynchronous style
using Promises directly, but it proves "collate" can be implemented in
JavaScript without fibers or even generators:
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L98-L131

Note that I'm using Streams in "paused mode" (
http://nodejs.org/api/stream.html)

Readable streams have two "modes": a flowing mode and a paused mode. When
in flowing mode, data is read from the underlying system and provided to
your program as fast as possible. In paused mode, you must explicitly call
stream.read() to get chunks of data out. Streams start out in paused mode.

I've added a receive
https://gist.github.com/tlrobinson/c8407e0e3f31a0da5400#file-collate-js-L32-L48
method to readable streams which returns a promise. You can use the promise
directly (as I do in collate2) or yield it within a Bluebird coroutine if
you prefer a more synchronous style (as I do in collate1).

I think basing a FBP system on Streams would be powerful because you can
implement a component in whichever style makes the most sense. From the
outside it doesn't matter. For example, copier could be implemented as
InputPort.openInputPort('IN').pipe(InputPort.openInputPort('OUT'));

Streams also support "backpressure" so they can communicate to their
upstream processes when they do or don't want more data:
http://nodejs.org/api/stream.html#stream_event_drain

Also note that I'm using streams in "object mode"
http://nodejs.org/api/stream.html#stream_object_mode but normal mode might
be interesting in some cases.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[tlrobinson]

The lack of support for fibers in browsers is apparently
much more of a concern, but this will probably be remedied before long if
and when fibers catches on!

@jpaulm It's highly unlikely fibers will be added to browsers. If you don't believe me try asking a few other JavaScript experts.

[jpaulm]

@tlrobinson And Streams and Promises are better than Fibers... how?! Because some JavaScript guru has blessed them...? Just curious!

[tlrobinson]

@jpaulm The main thing is with fibers you can't tell if a function call is going to block execution, whereas with generators you explicitly yield within a generator function (that's why the yield can't be hidden) Here's a description of the difference between generators and fibers: https://gist.github.com/creationix/6416266

But the important thing is generators are already in ES6 (the next version of JavaScript), and fibers would be mostly redundant, so they're unlikely to ever make it into the JavaScript spec and therefore browsers.

[jpaulm]

I just wrote the following to Comfreek:

send suspends when the connection is full, and receive suspends when the
connection is empty. I don't see this as a leaky abstraction - the
application designer does not even think in these terms - she just sees
processes running asynchronously, with data chunks flowing across
unidirectional connections - with back pressure.

That's why I keep saying it's a paradigm shift!

I don't want to know whether a function call is going to block execution

• it should not affect the logic! Go back and read Dijkstra on sequential
  processes! It's not rocket science!
  On Mar 7, 2015 7:32 PM, "Tom Robinson" [email protected] wrote:

@jpaulm https://github.com/jpaulm The main thing is with fibers you
can't tell if a function call is going to block execution, whereas with
generators you explicitly yield within a generator function (that's why the
yield can't be hidden) Here's a description of the difference between
generators and fibers: https://gist.github.com/creationix/6416266

But the important thing is generators are already in ES6 (the next
version of JavaScript), and fibers would be mostly redundant, so they're
unlikely to ever make it into the JavaScript spec and therefore browsers.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[tlrobinson]

In JavaScript it has always been safe to assume a function will run to completion before another event fires which could change the state that the function is using. Fibers break that assumption, which is unacceptable to a lot of people, including the people in charge of defining future versions of JavaScript.

Generators also break that assumption, but you have to explicitly opt-in by declaring a generator function (with the "function*" syntax) and use the "yield" keyword, so there's no chance of someone else's code deep in some library you're using yielding execution without your knowledge.

BTW this article expands on the gist I linked to in my last comment.

We can argue about which way is better, but it's not going to change the fact that you're unlikely to ever get fibers in the browser.

[jpaulm]

This discussion is interesting, but I feel it should be on the Group, so
it gets a wider audience. Could one of you, @tlrobinson or @ComFreek
start a topic on the Group, maybe with the last few posts. ..?

On Mar 8, 2015 6:01 AM, "Tom Robinson" [email protected] wrote:

In JavaScript it has always been safe to assume a function will run to
completion before another event fires which could change the state that the
function is using. Fibers break that assumption, which is unacceptable to a
lot of people, including the people in charge of defining future versions
of JavaScript.

"Change the state that the function is using."

That's really the whole point of FBP : two functions cannot use the same
"state ". It just seems odd to raise that objection in the context of
FBP! Maybe an education problem?!

[jpaulm]

It seems to me that the appropriate place for much of this discussion is
the JSFBP topic on the Group. Could we move at least the important posts
there? Thanks.

PS I would do it, but it would be very difficult using my little tablet!
:-)
On Mar 8, 2015 8:21 AM, "Paul Morrison" [email protected] wrote:

This discussion is interesting, but I feel it should be on the Group, so
it gets a wider audience. Could one of you, @tlrobinson or @ComFreek
start a topic on the Group, maybe with the last few posts. ..?

On Mar 8, 2015 6:01 AM, "Tom Robinson" [email protected] wrote:

In JavaScript it has always been safe to assume a function will run to
completion before another event fires which could change the state that the
function is using. Fibers break that assumption, which is unacceptable to a
lot of people, including the people in charge of defining future versions
of JavaScript.

"Change the state that the function is using."

That's really the whole point of FBP : two functions cannot use the same
"state ". It just seems odd to raise that objection in the context of
FBP! Maybe an education problem?!

[jpaulm]

Hi ComFreek,

Nice to hear from you - and nice to be back!

I am starting to look at JSFBP's derivatives - your streams-impl, and Tom's sbp (streampunk?). BTW Will you be updating your fork's readme to include the instructions you sent me? I think it would be useful.

What I find so strange is that these (to my way of thinking) contorted forms are said to seem more natural to a JS user (at least you and Tom seem to be saying this) than the nice simple syntax of the current JSFBP version (much improved by you, I admit!). In the former case, the user has to learn and get comfortable with Promises, Streams, etc., plus FBP, while in the case of basic JSFBP they just have to learn FBP!

I also don't see why 'yield' and those asterisks have to be visible - surely making them visible means that it could be meaningful to drop them - what happens if one or both of these are dropped by mistake?

Basically, it seems to come down to the fact that you and Tom don't believe that the JS community will accept node-fibers as readily as they have Promises etc. Surely that's just a matter of building a constituency of people who are more interested in getting the job done than in playing with abstruse concepts which probably don't have a well-defined use case anyway! Marcel @laverdet, are you getting any feedback on this?

ComFreek and Tom, I think I know where you stand on the issue of node-fibers vs. Promises, etc. - Ken, Alex, Ged, @bjouhier, @JeanHuguesRobert, @laverdet, I would be interested in hearing your views. I am double posting this note here, as I don't have email addresses for some of the people I would appreciate feedback from.

FYI The repositories I am looking at are:

https://github.com/jpaulm/jsfbp

https://github.com/ComFreek/jsfbp/tree/streams-impl

https://github.com/tlrobinson/streampunk.js

Best regards to all,

Paul M.

[tlrobinson]

Correct, my position is that it is very unlikely that fibers have a future
in the JavaScript language standard (ECMAScript), and they can't even be
reasonably shimmed into existing versions of JavaScript via "transpiling"
(unlike generators or async/await). Therefore fibers will never work in
browser JavaScript (the largest deployed programming environment, by far),
and should not be used as the foundation of a general purpose programming
framework if there is a suitable alternative.
On Sun, Mar 29, 2015 at 12:01 PM jpaulm [email protected] wrote:

Hi ComFreek,

Nice to hear from you - and nice to be back!

I am starting to look at JSFBP's derivatives - your streams-impl, and
Tom's sbp (streampunk?). BTW Will you be updating your fork's readme to
include the instructions you sent me? I think it would be useful.

What I find so strange is that these (to my way of thinking) contorted
forms are said to seem more natural to a JS user (at least you and Tom
seem to be saying this) than the nice simple syntax of the current JSFBP
version (much improved by you, I admit!). In the former case, the user has
to learn and get comfortable with Promises, Streams, etc., plus FBP,
while in the case of basic JSFBP they just have to learn FBP!

I also don't see why 'yield' and those asterisks have to be visible -
surely making them visible means that it could be meaningful to drop them -
what happens if one or both of these are dropped by mistake?

Basically, it seems to come down to the fact that you and Tom don't
believe that the JS community will accept node-fibers as readily as they
have Promises etc. Surely that's just a matter of building a constituency
of people who are more interested in getting the job done than in playing
with abstruse concepts which probably don't have a well-defined use case
anyway! Marcel @laverdet https://github.com/laverdet, are you getting
any feedback on this?

ComFreek and Tom, I think I know where you stand on the issue of
node-fibers vs. Promises, etc. - Ken, Alex, Ged, @bjouhier
https://github.com/bjouhier, @JeanHuguesRobert
https://github.com/jeanhuguesrobert, @laverdet
https://github.com/laverdet, I would be interested in hearing your
views. I am double posting this note here, as I don't have email addresses
for some of the people I would appreciate feedback from.

FYI The repositories I am looking at are:

https://github.com/jpaulm/jsfbp

https://github.com/ComFreek/jsfbp/tree/streams-impl

https://github.com/tlrobinson/streampunk.js

Best regards to all,

Paul M.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[jonnor]

@ComFreek https://github.com/the-grid/msgflo

[chadrik]

I've also wondered if it would be possible to do this with a FBP wrapper around celery and its canvas primitives.

[audreyt]

node-webworker-threads works on Node 0.10.x and is known to be unstable with 0.12.x and io.js, which I hope to fix in the coming weeks.

It is true that node-webworker-threads cannot share mutable structure across worker threads and with the main thread; the same principle applies to WebWorker in browsers, too.

[jpaulm]

@ComFreek As always, my questions would be: what language(s) and what operating system(s)? Given the existence of languages which don't play together too well, such as Java and C#, sockets would seem a promising approach... JavaScript seems to be able to talk to C[++], so this suggests that it might be interesting to work with Boost... What did you have in mind?

[laverdet]

fwiw I want to echo what @tlrobinson & @bjouhier said about fibers never making it into browsers or a standard. I started working on fibers in early 2011 when early versions of generators were still being discussed for standardization. We all knew yield was coming to JS and that it would likely be single frame, but we didn't know the exact API or syntax. And it definitely wouldn't be ready soon. I wanted yield right then and there and full continuations were the easiest way to build it.

[ComFreek]

@jonnor Thanks for the link, the readme seems very elaborate! Is there any quickstart guide available? I would like to see it in action (e.g. having component A running on my PC and component B on my notebook, both connected to my router).

@jpaulm As it is the very idea of the "FBP library", I would target some a wide range of operating systems and programming languages by employing low-level languages, such as C and C++, and common network protocols.

I'll have a look at celery, thanks for the link @chadrik.

[jonnor]

@ComFreek we're using it production a couple of places, but still have not landed on final API/protocol - so usage docs are not the priority just yet. Will probably tag & announce first stable in a couple of weeks, when we've migrated everything in the production pipeline to it. For now, see: https://github.com/the-grid/msgflo/blob/master/spec/fixtures/participants.coffee https://github.com/jonnor/imgflo-server/blob/master/src/worker.coffee

[jpaulm]

Merci, @laverdet! I think it is becoming obvious that what we really need
is software that takes advantage of multiple cores. AFAIK none of the
planned extensions of JS provide this. As @ComFreek says, it would be
even better if it allows multiple languages and infrastructures to
communicate in a coordinated way. Does msgflo (or any other systems that
people are aware of) address these requirements?
On Apr 8, 2015 1:49 PM, "Marcel Laverdet" [email protected] wrote:

fwiw I want to echo what @tlrobinson https://github.com/tlrobinson &
@bjouhier https://github.com/bjouhier said about fibers never making it
into browsers or a standard. I started working on fibers in early 2011 when
early versions of generators were still being discussed for
standardization. We all knew yield was coming to JS and that it would
likely be single frame, but we didn't know the exact API or syntax. And it
definitely wouldn't be ready soon. I wanted yield right then and there
and full continuations were the easiest way to build it.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[laverdet]

Cluster and workers both take advantage of cores

[jpaulm]

Do you have a link for clusters? Couldn't find anything on Google!
On Apr 9, 2015 3:24 PM, "Marcel Laverdet" [email protected] wrote:

Cluster and workers both take advantage of cores

—
Reply to this email directly or view it on GitHub
#14 (comment).

[laverdet]

First result in Google for "node cluster"

[bjouhier]

@jpaulm I encourage you to also take a look at nodejs/node#1159. This is work in progress on io.js but it looks very promising. A pure JS implementation based on a worker library which is blessed by io.js (and hopefully bundled with it some day soon) could be a winner.

[jpaulm]

Thanks @bjouhier - sounds interesting!
On Apr 9, 2015 4:39 PM, "Bruno Jouhier" [email protected] wrote:

@jpaulm https://github.com/jpaulm I encourage you to also take a look
at nodejs/node#1159 nodejs/node#1159. This is
work in progress on io.js but it looks very promising. A pure JS
implementation based on a worker library which is blessed by io.js (and
hopefully bundled with it some day soon) could be a winner.

—
Reply to this email directly or view it on GitHub
#14 (comment).

[jpaulm]

Here is a quote from http://www.jpaulmorrison.com/fbp/noflo.html :

... basic FBP business functions such as "Collate" require a process to be specific about which port it wants to receive from, and to be able to suspend until data arrives at that port ... A related architectural concept in FBP ... is what we call "back pressure", where an upstream process will be suspended if the connection it feeds into becomes full. Finally, ... the concept of information packet (IP) "lifetimes", by which an IP is tracked from creation to destruction and can only be "owned" by a single process at a time, or be in transit between processes.

How would these be approached in a fully isolated environment such as that described in nodejs/node#1159 ?

[tlrobinson]

The same approaches you would use to distribute FBP processes across OS
processes or hosts, right?
On Sat, Apr 11, 2015 at 1:45 PM jpaulm [email protected] wrote:

Here is a quote from http://www.jpaulmorrison.com/fbp/noflo.html :

... basic FBP business functions such as "Collate" require a process to be
specific about which port it wants to receive from, and to be able to
suspend until data arrives at that port ... A related architectural concept
in FBP ... is what we call "back pressure", where an upstream process will
be suspended if the connection it feeds into becomes full. Finally, ... the
concept of information packet (IP) "lifetimes", by which an IP is tracked
from creation to destruction and can only be "owned" by a single process at
a time, or be in transit between processes.

How would these be approached in a fully isolated environment such as that
described in nodejs/node#1159 nodejs/node#1159 ?

—
Reply to this email directly or view it on GitHub
#14 (comment).

[jpaulm]

I asked first! :-) And no, I don't know how to do this across OS processes or hosts either. IIRC Matt Lai has something he calls the "traffic cop" which can manage things like back pressure, but I'm not sure how that generalizes across multiple OS's.

[jpaulm]

Matt Lai gave a pretty complete description of his approach using Tcl/Tk in his March 19, 2014, post in https://groups.google.com/forum/#!searchin/flow-based-programming/appkata/flow-based-programming/hnst0E6AcUI/6al-uVAKL4oJ .

[bmeck]

Go back and read Dijkstra on sequential
processes! It's not rocket science!

Do you mean Hoare? http://spinroot.com/courses/summer/Papers/hoare_1978.pdf

[jpaulm]

No, Bradley, it is Dijkstra! See http://www.cs.utexas.edu/users/EWD/ewd01xx/EWD123.PDF I can't read it on my tablet but Google found it! I _believe_ this predates Hoare's CSP. Regards, Paul

[jpaulm]

I have just posted a web page about the Concat (Concatenate) component in "classical" FBP and in NoFlo - http://www.jpaulmorrison.com/fbp/concat.html . Comments would be appreciated.

Note: we are not sure if the cited Concat.coffee program is up to date.

[giantelk]

What's the consensus on using Node Fibers? My experience with JavaScript and web app's, or hybrid mobile i.e. Cordova/PhoneGap, React Native... which all use JavaScript is that 70-90% of the app's code is client side. So for me JSFBP needs to run in the browser?

Does NoFlo run in the browser? What do they do differently from JSFBP to make that happen?

[emptist]

Electron utilizes chromium multi process architecture http://dev.chromium.org/developers/design-documents/multi-process-architecture by rpc, can this be helpful to use multiple processes? http://electron.atom.io/blog/2016/07/28/electron-internals-node-integration

[tlrobinson]

Not sure if anyone is still interested in JavaScript implementations of FBP, but I dusted off my streampunk.js project and updated it to use async/await, which is included in ES2017 and supported natively in node.js and most browsers. AFAIK fibers in JavaScript haven't gained much traction in the last few years.

I also implemented the concat component: https://github.com/tlrobinson/streampunk.js/blob/master/components/concat.js As you can see it's about the same length as jsfbp's concat.

streampunk implements selective receive and back pressure: https://github.com/tlrobinson/streampunk.js/blob/master/test/backpressure.js

[ComFreek]

@tlrobinson Thanks for the update. Maybe async generator functions are also of interest for you:

// inside a "async function *myFunc(...) {"
while ((ip = await array[i].receive()) !== null) {
  yield ip;
}

One disadvantage is that you definitely give up your "handle of computation", whereas with the send method, you are free to await it or to perform other actions and then, let's say, collectively await all sent IPs.

Incidentally, I've recently written an AsyncLimitedQueue implementation with async functions and generators in TypeScript, where pushing can block after a certain amount of elements (aka. backpressure): https://github.com/ComFreek/async-playground/blob/21d4f9ec73db732f8fc4dfc54616aeae490cc700/queue/AsyncLimitedQueue.ts#L167

[jpaulm]

Hi Tom, this certainly *does* seem much closer to the spirit of classical FBP! I guess one has to specify "await", but it feels very "classical"! I assume JavaScript still can't handle multiple cores though - do you know if this is likely to change any time soon...? Regards, Paul

…

On Fri, Apr 13, 2018 at 2:43 PM, Tom Robinson ***@***.***> wrote: Not sure if anyone is still interested in JavaScript implementations of FBP, but I dusted off my streampunk.js <https://github.com/tlrobinson/streampunk.js> project and updated it to use async/await, which is included in ES2017 and supported natively in node.js and most browsers. AFAIK fibers haven't gained any traction in the last few years. I also implemented the concat component: https://github.com/tlrobinson/ streampunk.js/blob/master/components/concat.js As you can see it's about the same length as jsfbp's concat. streampunk implements selective receive and back pressure: https://github.com/tlrobinson/streampunk.js/blob/master/ test/backpressure.js — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATGJ8Lah3N90tsCqKafJGNoB1zArqU6ks5toPG5gaJpZM4DfOIf> .

[jpaulm]

I have been reviewing this very interesting correspondence, and I see that there was a long hiatus after @giantelk 's 2015 query (which apparently wasn't answered), until @emptist 's suggestion in 2017, and @tlrobinson 's update a few days ago. I imagine quite a lot must have changed in the JS world since 2015!

Also, since that time I have refined the locking logic in JavaFBP and C#FBP... and it turned out to be a bit more complex than I had thought (I think those two implementations are working now - anyone want to check them?!), so this logic would have to be replicated in a truly asynchronous, multi-core implementation using JS.

@giantelk, I confess I didn't quite understand your query, but my guess is that client side doesn't really need multiple cores, although single-core FBP would improve maintainability, so jsfbp could apply here (or I guess @tlrobinson 's latest, since it seems more viable than jsfbp), but server side would definitely benefit from multi-core support - so why not use JavaFBP or C#FBP... which would in addition provide better performance?! And we wouldn't have to twist JS into a pretzel!

@tlrobinson are you considering providing tutorial material for streampunk...? Or a successor...?

Regards,

Paul

[tlrobinson]

@jpaulm I consider streampunk to be experimental. I don't think I'll have time to make it a real project on my own, but if there's interest I'm happy to collaborate.

The only (standard) way JavaScript can make use of multiple cores is via Web Workers, but memory is typically copied between them, not shared. There's also a proof-of-concept ForkProcess (for node.js only) in streampunk that runs a component in a separate OS process, thus can take advantage of multiple cores (but with the overhead of IPC)

SharedArrayBuffer had potential for multiple JavaScript "threads" sharing memory, but it was disabled recently due to the Spectre attacks.

By the way, it's worth noting the async/await version of streampunk isn't actually very different from the generator function + yield version from 2015. I basically just replaced function* with async function and yield with await (see this commit: tlrobinson/streampunk.js@129c71c#diff-670630be11e62ed09315eb5aa0eea1be).

One other thing I've considered exploring is building streampunk using "Observables" instead of Node.js "Streams". They are similar, but Observables are on track to be part of the ECMAScript language (and already widely used via libraries like RxJS) while Streams are historically a Node.js thing, but are being standardized by WHATWG.

[jpaulm]

@tlrobinson I looked at the WHATWG link you gave and I think I saw "chunks". Do I get the impression that Streams has reinvented Objects?! Or did I misunderstand? I can't imagine why JS dropped Objects - unless JS just followed a different evolutionary path from something like C?

[jpaulm]

@tlrobinson Thanks for the feedback on streampunk, etc. Please keep us posted on how this area of JS evolves, and to what extent it is converging with classical FBP.

Regards, Paul

FYI @laverdet @ComFreek @bjouhier