Official docker image need more modules

[tinuva]

I have been following issue 341 and is really excited to be able to use it, however I think it needs some of the prerequisites added to the Dockerfile.

For example the following log entries work on my standalone install:

mqttwarn        | 2018-10-30 11:19:16,856 DEBUG [slack] *** MODULE=services/slack.py: service=slack, target=openhabalarm
mqttwarn        | 2018-10-30 11:19:16,857 ERROR [slack] slacker module missing
mqttwarn        | 2018-10-30 11:19:16,858 WARNING [mqttwarn] Notification of slack for '/openhab/notification/alarm' FAILED or TIMED OUT
mqttwarn        | 2018-10-30 11:19:16,859 DEBUG [mqttwarn] Job queue has 5 items to process
mqttwarn        | 2018-10-30 11:19:16,859 DEBUG [mqttwarn] Processor #0 is handling: 'pushbullet' for david
mqttwarn        | 2018-10-30 11:19:16,864 DEBUG [pushbullet] *** MODULE=services/pushbullet.py: service=pushbullet, target=david
mqttwarn        | 2018-10-30 11:19:16,865 WARNING [pushbullet] pushbullet is not installed
mqttwarn        | 2018-10-30 11:19:16,866 WARNING [mqttwarn] Notification of pushbullet for '/openhab/notification/alarm' FAILED or TIMED OUT

On the normal install I am able to install the dependencies, but the docker image mean I need to build my own for my specific needs unless you could add some of the popular ones at least.

Many thanks in advance!

[jpmens]

We cannot and will not add all prerequisites for all 60+ modules to the Docker image, and "popular" would appear to be in the eyes of the beholder, no?

[rgitzel]

@jpmens Actually, why not add them all? At worst it makes the Docker build time slightly longer, and the image slightly bigger. Any that you don't use yourself just sit there unused. And they don't interfere with your local Python environment.

It might just mean a bit of playing whack-a-mole explicitly listing them all.

[tinuva]

We cannot and will not add all prerequisites for all 60+ modules to the Docker image, and "popular" would appear to be in the eyes of the beholder, no?

Which ones will work currently?

I think email works, but if that is the only one, I am not quite sure what the use of this docker image will be.

Please consider adding all if not most.

[amotl]

Hi there,

as #127 will wrap the dependency management into the canonical setup.py already (at least it will list all possible requirements correlated to each service plugin), i am also leaning towards having an easy way to perform a "full" setup in some way or another in the future. Whether this will happen at build or installation time? Time will tell...

The "full" flavor could be spawned from the pip level as an "extra" requirement, acting as a driver for the pip-in-docker building we will approach eventually after integrating the "develop" branch.

Regarding dockerized setups, we could think about publishing a distinct jpmens/mqttwarn-full image giving all users maximum comfort, no? Otherwise, we would have to think about having popularity shootouts between different service plugins and might end up building jpmens/mqttwarn-slack, jpmens/mqttwarn-xmpp, but not jpmens/mqttwarn-pushover etc. This would explode the matrix eventually.

On the other hand, i am also keen on adding a subcommand to mqttwarn like mqttwarn enable-plugin <plugin>, which could well be executed interactively on behalf of the user using docker run as well.

By combining both tactics, we could get the best out of different worlds, but that's really just my personal two cents.

With kind regards,
Andreas.

P.S.: When looking at a similar scenario in the Debian ecosystem, we find packages like nginx and nginx-common vs. nginx-extras or nginx-full. I believe they made sound decisions differentiating them from each other, so we could dare to take a look.

[rgitzel]

@amotl what is the reason to NOT install all the possible dependencies at once? (Other than taking up disk space.) I'm not a Python dev, so I don't know if it's a compatibility issue or such.

[rgitzel]

@tinuva here's what is currently included: https://github.com/jpmens/mqttwarn/blob/master/Dockerfile#L5

What packages do you install?

[jpmens]

shrugs

I will accept a PR for the Dockerfile to include more services; I don't think we should over-complicate this and I also don't think building jpmens/mqttwarn-*.* is going to do anybody any good. :-)

[rgitzel]

@amotl I'd like to not confuse the use of Docker now with the use of Docker with the dev branch, as they are built differently.

But I do already have a candidate, which in fact is the one I've been running for the past few months: https://github.com/rgitzel/mqttwarn/blob/docker-pip/Dockerfile

[amotl]

what is the reason to NOT install all the possible dependencies at once? (Other than taking up disk space.) I'm not a Python dev, so I don't know if it's a compatibility issue or such.

@rgitzel: Thanks for asking! Not all service plugins have well defined upstream dependencies on PyPI. We were having our times with XMPP for example.

While i definitively want to approach a wash & go "full setup", i am as conservative as @jpmens here until we got everything rock solid. When turning into that direction, we will probably come across having to decide whether we should publish own stable forks of specific service plugin dependencies to PyPI. It's just a story of turtles all the way down and will probably require quite some amount of tedious homework. I don't want to push this that fast but i am confident everything will converge at some time in the future.

mqttwarn-ng from #127 will include the most important subset of dependencies anyway. For example, every service plugin calling out the HTTP APIs using the fine requests package will be covered already.

Don't get me wrong: There should be some convenient way to perform a thing which might be called "full setup", this will also help an expanding #332 when running on CI through tox or similar beasts. Until then, i am totally with @jpmens that building jpmens/mqttwarn-*.* too early is not going to do anybody any good, but will put more burden on our scarce maintenance resources. On the other hand, i will be happy to hear about the outcome of anyone who will give this a try.

That said, i want to add that i really like the growing interest around mqttwarn from all of you. @jpmens and me might just have specific hats on but as you can recognize, we are really cheering the growing developer community around mqttwarn and will do our best to keep this flowing. So: Further suggestions about these topics will always be received with gratitude.

Thanks for your efforts, you know who you are.

[egadgetjnr]

Hey Guys,
Just my 2 cents on this one being somewhat an experienced docker user.

I think going down the path of having ALL the modules installed on an image would be somewhat troublesome as it would increase the image size exponentially.

Also, Using a tag to define the modules would also be difficult if there is more than one module requirement. How would you be able to use Pushbullet and MySQL and hipchat all in the one image?

@amotl is on the money though with having a command that you can run to install them, as you’d be able to use your own dockerfile to specify the modules you want if need be. Failing that, could you use arguments to install the plugins on first run? That way when you use ‘docker run’ you will be able to pass arguments for the module installation at the end of the command.
Both of these approaches are what other developers use for their docker repos and it allows full customisation of different scenarios.

[rgitzel]

I did an experiment to create a Docker image which includes as many dependencies as I could find reading through the README: https://github.com/rgitzel/mqttwarn/blob/docker-dependencies/Dockerfile#L5

With all those dependencies added, the Docker image is 420Mb

With very few dependencies, the "official" image built yesterday is 356Mb

So basically installing all the extra libraries results in no appreciable difference in size.

Then I modified it to use Alpine as the base OS, which reduced the image to 117Mb

That's indeed smaller, but frankly I don't think it's worth the work sorting out all the picky library bits that are missing from Alpine (indeed, the Azure library would be painful). 420Mb isn't anything out of the ordinary, particularly for a server application.

CAVEAT: I haven't tested either of these images beyond running them on the command-line:

$ docker run -t rgitzel/mqttwarn-full
Cannot open configuration at /opt/mqttwarn/conf/mqttwarn.ini: [Errno 2] No such file or directory: '/opt/mqttwarn/conf/mqttwarn.ini'

Because Python is interpreted, the app probably won't fail for a given plugin until it's run. But at least those many dependencies installed without any apparent conflict.

@tinuva could you give one or both of the new images a try? rgitzel/mqttwarn-full-alpine and rgitzel/mqttwarn-full. They hopefully have everything installed that you need.

And if anyone notices any dependency I missed (other than the Mac-dependent ones), let me know and I'll add them.

[tinuva]

@rgitzel tested both and both work for me at least. Which is smtp+pushbullet+slack
The test was simple, start the image with my current configuration and then send a test message to a channel that makes use of all my mqttwarn service providers.

Going to leave it on the Alpine image, which is what I prefer where possible.

My docker-compose snippet:

  mqttwarn:
    container_name: mqttwarn
    image: rgitzel/mqttwarn-full-alpine
    restart: always
    environment:
      MQTTWARNLOG: stream://sys.stderr
    volumes:
      - ${USERDIR}/docker/mqttwarn:/opt/mqttwarn/conf

[psyciknz]

I may be coming late to this. But as someone that a, uses mqttwarn in docker, and b, has a couple of services that for various reasons, wont be incorporated. I would like to see a way of adding custom services to the docker image. To enable this system, I end up building my own docker images....but I also build arm32v7, arm64v8 and amd64 images all at once.

What about a solution where there's a main "services" folder with the pre-built, tested, and selected services. Controlled via the repository. And then via a config option a "user_services" folder, where any .py files in here can be added to the running set.

I believe this method would satisfy the requirement of keeping the installed set of services down, and allowing users to add to extras (from the repository) and/or home built.

[tinuva]

I think @rgitzel 's experiment is doing well. See the most popular mqttwarn container on hub.docker.com....
https://hub.docker.com/search?q=mqttwarn&type=image

That said, I will be perfectly happy with a base mqttwarn image that is able to install additional dependencies on demand.

[rgitzel]

@tinuva Holy smokes!!! That's fantastic. :-) Ironically I don't use my own image, I use the "official" one. Perhaps will need to take mine down and somehow point to the official one... or perhaps a new release of the latter might incentivize people? @jpmens

Anyway... Docker images aren't really meant to be "extendable" (or even "configurable") once they are built; typically you fork and modify an existing Dockerfile and publish the resulting image yourself. (Which isn't hard, especially now that Docker Hub has a builder.)

But thoughts about other ways to accomplish some customization...

• if it's just a matter of including .py files in some path for Python to find (I'm not a Python dev, so I don't know) then it would be easy enough to volume-mount an external folder for that purpose, as part of the image; by default it would be empty
• if modules or other programs need to be installed... you can start the image with an alternate entry point (i.e. bypass the Python command on the last line of the file), namely a shell, where you can do all your extra installation and configuration, and then run Python.

With the latter, there might be a way to stop and restart that particular container, so as not to lose all your changes. Hmm.

The thing to remember is that a Dockerfile is run only on build -- once you have the image, the commands in the Dockerfile have long ago been executed. I still get confused by that sometimes, thinking that the Dockerfile script is executed at runtime...

[amotl]

Dear @tinuva, @rgitzel, @tomhoover, @egadgetjnr, @psyciknz and all Docker lovers,

#127 and #336 have come pretty far that we might think about improving the Dockerization once more.

That said, I will be perfectly happy with a base mqttwarn image that is able to install additional dependencies on demand.

Service plugins can now be installed selectively by using pip, e.g. pip install mqttwarn[xmpp,fbchat].

With kind regards,
Andreas.

[amotl]

Just for the records...

@rgitzel asked on Oct 30, 2018:

What packages do you install?

@tinuva has a nice Dockerfile with some extended dependencies for serving more service plugins out of the box.

@amotl commented on Oct 30, 2018:

Regarding dockerized setups, we could think about publishing a distinct mqttwarn-full image to Docker Hub giving all users maximum comfort, no?

cc @koenvervloesem

[amotl]

Hi again,

I believe we resolved this per #506. mqttwarn-full Docker images [1] will now be available for each upcoming release. [2] has corresponding documentation.

Please let us know about your experiences with those images and feel free to reopen this issue if you feel some details would need more attention.

Keep up the spirit and with kind regards,
Andreas.

[1] https://github.com/users/jpmens/packages/container/package/mqttwarn-full
[2] https://github.com/jpmens/mqttwarn/blob/master/DOCKER.md