Back end infra ideas

[patrickfitz]

Suffice to say this app has really caught my attention. I've got a long background in infrastructure works physical (eg building an open source cloud environment way back in 2010) through various software and infra projects (including the rollout and management of thousands of Linux workstations at a major European bank).

Bear in mind I've only been using this code for about a week from around the 7th of May. This is quite long too, sorry about that. Even worse is that I haven't really gone though the source code either. My bad.

Is your feature request related to a problem? Please describe.
I started thinking about how to expanding it with a view to making it Kubernetes ready, a richer reporting interface, and allowing other apps to run alongside and interact with the trading bot. At the moment this appears, at first glance, to be quite difficult, because of the dependencies that seem to exist within the different parts of the application.

Describe the solution you'd like
Having a great deal of experience in Django, I like the way that everything revolves around a central settings.py file, that is called early on and loads the various apps, filters, databases, templates and so on. In fact I've used it so long I didn't really get how genius it is. As executable python, it actively builds the apps and settings structures in memory, rather than passively called by a monolithic code base.

So:

1. In today's context I'd split out Coinbase and Binance as separate apps (or "Exchanges" - more could follow)
2. I'd also separate out the different algorithms as backend apps ("Algorithms")
3. Have a different set of apps for "Messaging"; such as Telegram, Signal, Webhooks, MQTT (my personal favourite) etc
4. Yet another for logging.
5. Another for a database connection (for recording trades, settings and values etc, with the default being sqlite)
6. (And so on.)

The benefit of all this is that each area could be worked on separately - for example, some people might want to develop particular types of trading algorithms - whilst others might want to work on authentication (for multi-user login), reporting, alerting and so on. They could all do this independently, as deliver their code as a plug in, without blowing up the entire app stack.

The first step would be to begin to break out the different "Exchanges" as above and determine the common apis to enable all (or most) exchanges. I know this has already been done, but not in a manner where there is a discrete "binance app", for example, that can be plugged in, with a single folder containing everything binance related.

Then create a settings.py that loads the exchanges and their settings. The goal is extensibility - so I write for example, if I write an app called pycryptobot-newcoin that plugs into a newcoin exchange - all a user should need to do is:

pip install pycryptobot-newcoin
then in settings.py:

• add "newcoin" to the exchanges list
• add "newcoin" settings
• restart the app.
  The user then gets all the benefits of newcoin, coinbase, and binance, plus the whatever additional settings (telegram) - when all I've done is enable a single app - and not touched anything else.

I can see this as being supremely useful, and popular. However I can also see how it could against what Michael is trying to achieve though, and fair enough.

Describe alternatives you've considered
Doing it the other way around - making pycryptobot a django app - might be interesting, but the request-response style of web frameworks is completely different to the requirements of real-time, always on, server applets. There could be a lot of code shared by the Django project to contribute to this one though.

Additional context
Take a look at Django - you'll see what I mean.
Thoughts?

[rcarmo]

I have been thinking along the same lines, but smaller (i.e., getting multiple bots to run without infrastructure overhead). Right now I am using Piku to run several, and considering putting together a simple Vue front-end using TradingVue, but honestly think Kubernetes is entirely too complex for this kind of thing, especially since it's quite easy to run multiple bots under some sort of supervision tree.

What is missing to do that is a) proper logging (all those print statements, etc.) and b) some way to monitor them (for which a set of Redis or MQTT topics would be plenty good enough - and could even be logging sinks)

But see #197 for libraries that would allow multiple exchanges to be used. The work is already mostly done in the back-end part, it's just a matter of integrating it and figuring out how to monitor multiple bots.

[patrickfitz]

Yeah I agree that Kubernetes (as is podman or docker) if probably overkill. Python can spawn multiple processes anyway, and each could be its own trading bot, all we need is a framework to manage it. MQTT could provide real time data for the UI using via web sockets - plus management/logging functions. A database to store the setups, trades, and logs.

Essentially: a django-style trading framework, allowing for the separation of concerns and allowing for people to focus on what they are interested in. Doesn't have to be heavy or complex either. Piku looks interesting, and TradingVue as well...

[jovana]

Nice discussion! I like! Same ideas here. And I understand perfectly the approach here. But I think we need to cover two types of use cases.

The case of @patrickfitz is keep in mind a large user base. Probably great a platform where customers can sign up and start trading whiteout technical knowledge.

The case of @rcarmo is to have a environment up and running for one user with basic technical knowledge probably none development knowledge but is able to follow a clear instruction how to run the software on you own setup.

Good examples are sickbears of sabnzb. This is a complex solution but made it easy to run for someone with moderate or almost know technical knowledge.

So, if we bundel our technical knowledge and we create modules (packages) we can move two ways with those packages. Building larger robust environments. But also build a single standalone environment witch can run also on Qnap, raspberry or Synology (eg.).

I love to help and brainstorm this further. Also I'am able to write test and review python (FastAPI), reactjs, javascript, terraform, docker code. My infra knowledge and higher skills are on AWS and linux.

When do we start? 😬🤓