Gossip Discovery

[utsavdrolia]

I noticed that this is in the TODO list. Any progress on this? If there is a protocol spec for this I could do a PR and work on it.

[blink1073]

This seems like a big job. zgossip is a class in czmq and does not have a counterpart in pyzmq.

The protocol is defined here: https://github.com/zeromq/czmq/blob/master/src/zgossip.xml

[utsavdrolia]

Is there a document for the protocol? Or only the xml?
I know it's a big job, no harm trying :D
On Apr 18, 2015 9:40 PM, "Steven Silvester" [email protected]
wrote:

This seems like a big job. zgossip is a class in czmq and does not have a
counterpart in pyzmq.

The protocol is defined here:
https://github.com/zeromq/czmq/blob/master/src/zgossip.xml

—
Reply to this email directly or view it on GitHub
#65 (comment).

[sphaero]

I'm working on it: https://github.com/sphaero/pyre/tree/gossip
I hope to finish it this month

[maccradar]

any progress on porting gossip to Pyre?

[sphaero]

Most of the code is in place. The clients connect and exchange but the logic of the exchange is still missing (zloop). Only I moved to working on zproject and zproto python support. My use case changed slightly making the need for gossiping less of a priority.

[wesyoung]

@sphaero have you made any [local] updates to your fork/branch ? i'd like to pick up where you left off, gonna sync your fork/branch to a local repo, wasn't sure if you had something stashed you hadn't yet pushed...

[sphaero]

No I don't think so. I got stuck in the engine code. I kind of moved to the zyre python binding and letting zproject generate the state machine in python. Let me know if you need help

[wesyoung]

so are you using gossip through zyre[proper] python bindings?

[sphaero]

I was doing a research in which I needed gossip discovery because it's 'better' for using zyre threads. Although in the end I didn't need it I did get the binding to work properly.
The goal I was aiming for is to have Pyre try to use the Zyre C binding and only if that fails use the native Pyre classes. To accomplish this Pyre needs some adaptation.

[wesyoung]

Ahh. It wasn't clear if pyre was meant to be a pure Python implementation because something didn't mesh well at the c level or not..

I hadn't dug to far into the python bindings of zyre itself but will.. Ty!

[sphaero]

Pyre was written before gossiping was added to Zyre. The gossiping enging is generated using zproject/zproto. There are three options for gossiping support in python:
1: use zyre c binding
2: port gossiping engine to python (most code already in my branch)
3: add python support to the engine generator (zproject/zproto)

[wesyoung]

wanted to update this thread, in case others started poking around. i pushed some initial changes here that gets the branch closer:

https://github.com/zeromq/pyre/compare/master...wesyoung:feat/gossip?expand=1

[this branch merges most of master with the previous work @sphaero started..]

there's still a ways to go (right now it'll connect to the gossip socket, but fails to deliver the key-pairs to existing nodes)

spent a bunch of time reading through the C libs, just getting an idea how the engine works. some initial thoughts (soliciting feedback):

1. the zyre engine (C) will move faster than pyre, but it's not very pythonic (although the python bindings get you a little ways there)
2. doing things more pythonic might make pyre/zyre more accessible to python developers where C is more of a second language to them
3. either within pyre (or zyre?) maybe doing something like pyzmq did where we're sugaring the lower-level generated bindings to make things more "pythonic and accessible" might be a sane direction to head? that way we get the benefit of the lower level binding development, while keeping things python consistent for new(er) users?

at this stage, the engine "almost works" ie: you can start to see the shouts come across the wire, just that new nodes aren't seeing the whole network (something blocking somewhere). but i can def see where native python users may get confused, and there might be a lot of work keeping this up to date as zyre iterates along (not impossible, just finding a balance).

i'm not sure i have strong opinions either way, but am starting to think doing something like pyzmq does (sugar, etc) may be the right direction to take this, keep it accessible to native python users (which means more developers using, improving the core framework) while offloading the engine work to where it belongs?

thoughts? (keep in mind, still getting my feet wet, feel free to make corrections in my assumptions here :)).

[sphaero]

Nice! my thoughts:

1: yes the zyre engine is faster but this is python thing
2/3: I've wanted to make Pyre a drop in replacement for Zyre (binding) This implies Pyre will not be pythonic. An extra layer on top of Pyre would provide the Pythonic approach. I think this is the nicest and most transparent approach. It would also mean the pythonic layer can be used for the Zyre binding.

[wesyoung]

i'm putting this together:

https://github.com/wesyoung/pyzyre

which builds and embed's libzmq.so, libczmq.so and libzyre.so within python (not system wide, auto configures, builds, etc). it's not quite complete, but it wraps everything and puts into place a thin layer for "more pythonic functions" a-top of Zyre (building something similar for czmq).

so by default you have access to zyre.Zyre (_zyre_ctypes) but we can do more magic within pyzyre.Zyre (which could sub-class zyre.Zyre, etc). make things [at the czmq layer] easier to use. by default it installs site-packages/czmq, site-packages/zyre and site-packages/pyzyre.

comments welcome. still a work in progress, dunno where it fits in life, but just the build system alone (re-purposed from pyzmq) helps solve a few problems for me. have an examples dir and pyzyre.utils patch coming soon too.

[sphaero]

Just a quick note; you are aware zyre comes with python bindings included. What would be very welcome is a high level pythonic interface on top of those bindings which would load Pyre if it can't find the zyre libs. Pyre would then just mimic the zyre API exactly.

[wesyoung]

let me know if/when you get a chance to poke around the repo a bit.. following up on our initial thread :)

made a few patches [merged] to zproject [and czmq and zyre] to get things lined up. make it easier to build/install the core libs from pip. not by having to everything by hand to keep up with the zyre engine [or czmq].

i'm not aiming to "load pyre if it fails to find the others" i'm aiming for "it just does magic and builds it" .. sorta like pyzmq does. if people want to continue with a PP version (ie: pyre), that's a-OK. totally understand. but other languages seem to be suffering from the ability of keeping up with the C-engine too, so finishing the build system and making magic happen (ie: pip install pyzre .. and it builds the C libs for you) seems like the next best thing... (imho).

right now, this loads the python [zproject generated] bindings and everything [including the gossip engine] seems to work. next step is factoring in some pythonic methods to get encryption working (and cleaner integration with native pyzmq Sockets, i have some examples just need to clean them up).

[again, not arguing which way is best, just contributing back, if it's useful, great, if not ... it's there if anyone ever needs it]