Support for publish/subscribe systems

[drakkan]

Hi,

from time to time someone ask questions about asynchronous message queue support.

The clear requirement is a way to receive all events generated by SFTPGo without missing anything.

I have other priorities for now, but I would like to group all the suggestions/use cases here and eventually add this support directly inside SFTPGo in future.

We have several events that are suitable for an asyncronous message queue, for example upload, download, delete, rename, but we also have other events that need to receive and process the response synchronously before continuing the internal flow. For example pre-delete action or pre/post-login and other hooks. I think these events must remain synchronous. So we should carefully design new configurations and clearly document which hook applies to which actions to avoid confusion.

I analyzed the available libraries and go-cloud seems the most complete one, it basically support all the pub/sub systems out of there:

• Google Cloud
• Amazon SNS and SQS
• Azure service bus
• RabbitMQ
• NATS
• Kafka

it also seems to have some pitfalls, see for example here.

Do you have experience with this library? Some users, for example @sagikazarmark , seem to prefer watermill, it would be interesting to see why.

thank you

[sagikazarmark]

I'm a huge fan of watermill, because it's a generic (and therefore unbiased) messaging library.

I'm not sure adding support for every possible message queue is a good idea though. It'll significantly increase the binary size.

I've actually grown to like the current webhook system. Initially I wanted to integrate the messaging into the sftpgo binary, but I gave upon it very quickly.

If the concern is resilience then I'd suggest adding a configurable retry with some backoff.

Alternatively/In addition to that adding a watermill HTTP example can also be sufficient: https://watermill.io/pubsubs/http/

I've built a few webhooks with that and it works beautifully.

I can even see the use case for a separate binary implementing a bunch of messaging protocols (via watermill) for those who just want to send everything to a message queue.

If you integrate watermill http on the sending side, then you can utilize features provided by watermill in the receiving end as well (tracing, message ID, etc)

[drakkan]

I like this suggestion, thank you!

I could use something like retryablehttp to make web hooks more resilent and add an example as you suggest. I also plan to make a separate product in the future which receives all these events and can perform some actions automatically, pub/sub support could be implemented within this new project

[sagikazarmark]

retryablehttp is a great tool, I like it.

[drakkan]

retryablehttp is a great tool, I like it.

retryablehttp is now used for notification hooks

[drakkan]

Another interesting possibility to explore is an RPC plugin system like this one. It could be used for other things as well, for example ldap auth, OTP or the external KMS that are currently bundled inside SFTPGo itself