appliance: create logsene-ready container

[rgerhards]

This shall create a container that is ready to ship logs to Logsene with limited configuration. This task is neither to simple nor to complex, so it probably is a good starter to understand what we need to do.

Note that this is meant as a development / protoype kind of work. Once this works, we can create the real container. As such, we initially use an Ubuntu 16.04 dev container, as this provides for a quick compile and link cycle.

Steps to do:

• create a basic configuration that ships logs to logsene (see https://github.com/megastef/rsyslog-logsene/blob/master/rsyslog.conf) --> see ./tests/test_logsene.sh in personal dev environment
• create a basic capability to read config vars from file system (or environment) --> provide a way to use env variable for config parameter rsyslog#2416
• create a basic devel container which utilizes these logs
• provide complete listener config (udp, tcp, relp)
• check that spool files correctly work

[rgerhards]

@hakman which certificates do I need to include into the appliance to ensure that https to logsene will work? Any suggestions on this config: https://github.com/rsyslog/rsyslog-docker/blob/master/appliance/alpine/rsyslog.conf.d/log_to_logsene.conf

Would be great if you have a couple of minutes, I would really like to go beta with the container soon, at least in regard to Logsene ;-) Any general comments and suggestions are also appreciated.

[rgerhards]

side-note: we already moved to a "real", alpine-based container.

[radu-gheorghe]

Actually, logging to Logsene via HTTPS should just work (via the certificates coming through the OS). Does it not?

I assume you're talking about the certificates mentioned here? https://github.com/megastef/rsyslog-logsene/blob/master/rsyslog.conf#L14 Those would be for the TLS input of rsyslog. We could support this as well, if we generate them on container start, like @megastef did here: https://github.com/megastef/docker-logsene-kibana/blob/master/run.sh#L18-L19

[radu-gheorghe]

One more thing: the same container could be used to push to any Elasticsearch cluster (though Elasticsearch itself doesn't support HTTPS, and the HTTP port defaults to 9200 - I guess the only missing step is to specify a port variable, and maybe an HTTPS one?).

Other than those comments, I like this a lot. Thanks so much for the effort of putting it all together!

[rgerhards]

Actually, logging to Logsene via HTTPS should just work (via the certificates coming through the OS). Does it not?

I completely missed that ;- ) -- sorry for the noise...

I assume you're talking about the certificates mentioned here? https://github.com/megastef/rsyslog-logsene/blob/master/rsyslog.conf#L14 Those would be for the TLS input of rsyslog. We could support this as well, if we generate them on container start, like @megastef did here: https://github.com/megastef/docker-logsene-kibana/blob/master/run.sh#L18-L19

I'll look into that, but I guess when we use anon mode, we should not need them and do all via D/H key exchange. Probably worth it's own issue.

[rgerhards]

One more thing: the same container could be used to push to any Elasticsearch cluster (though Elasticsearch itself doesn't support HTTPS, and the HTTP port defaults to 9200 - I guess the only missing step is to specify a port variable, and maybe an HTTPS one?).

Yeah, it's just a starter config. I think the container is even more used by folks who do not know rsyslog. So I want to keep them off of the real config and use a meta-config instead. First version here: https://github.com/rsyslog/rsyslog-docker/blob/master/appliance/alpine/internal/container_config -- it probably explains better what I am after than a thousand words ;-)

Rsyslog 8.33 will have a couple of config extension to make this easier to handle. Have a look at the main container rsyslog.conf: https://github.com/rsyslog/rsyslog-docker/blob/master/appliance/alpine/rsyslog.conf -- this is the actual rsyslog.conf, not something we use to generate a file from. So we can keep the container read-only.

Of course, it is possible to override with any config of your liking, but I'd like to keep it focussed. So "generic elasticsearch" is one of the next destinations we need to implement.

Other than those comments, I like this a lot. Thanks so much for the effort of putting it all together!

Thanks for the motivation! Also please keep comments flowing. I am especially interested on feedback on the meta configuration layer.

[rgerhards]

Note: we right now see some warning messages during shutdown. They are ok, but will probably trigger the creation of some issues on the main project (which is a good thing as we now have a reproducible test environment).

[rgerhards]

just for completeness, this first beta is available here: https://hub.docker.com/r/rsyslog/syslog_appliance_alpine/

[megastef]

@rgerhards Hi, is there an ARMHF version of logsene-ready rsyslog container?

[radu-gheorghe]

@megastef I'm only aware of this Alpine container: https://github.com/rsyslog/rsyslog-docker/tree/master/appliance/alpine

I guess that for supporting ARMHF we would "just" need to reuse the same Dockerfile, but with an ARMHF base distro. Ideally Alpine. This one looks OK to me (though I don't know much about ARMHF to begin with): https://hub.docker.com/r/arm32v6/alpine/

[rgerhards]

late to this story ;-)

We would probably need to build rsyslog on armhf. I now have the hardware, but I admit the interest in alpine was so weak (no response for month until this posting) that I have mentally set it to low prio. If there is a larger community interest and (preferably) someone who wants to help maintain it, I would be all ears.

[megastef]

1. Hardware should not be an issue, you can get very cheap ARM cloud servers from Scaleway: https://www.scaleway.com/ - 2,99 EUR per month
2. Alpine Linux - if this is a blocker, we could use Debian/Ubuntu base images.
   It looks that Ubuntu has ARM support: https://hub.docker.com/_/ubuntu

[rgerhards]

As I said, hardware is not the issue. Time and someone willing to maintain it is the issue. I can't do everything ;-)