ES 5.0 changes

[nubunto]

This is just a summary of all the changes I needed to make in order to get a working ES 5.0 cluster working (ish) last week.

• We can no longer pass arguments to ES 5.0 via command line, so all of this code breaks the executor. Not sure how to go by this, but apparently we need to have a default elasticsearch.yml passed to ES, instead of a bunch of command line arguments.

• The vm.max_map_count kernel config needs to be increased on the host, according to their docs. Although not directly related with this project, still worth mentioning in a troubleshooting area if the executor keeps getting killed all the time. They don't mention it on the docs, but this is a bootstrap error.

• There is no easy way to get backwards compatibility regarding ES_HEAP_SIZE vs ES_JAVA_OPTS. We have to cut it out completely for ES > 5.0 and we need it for ES 2.x. Which means that, at configuration time, we need to know what is the version of elasticsearch we're going for, in order to use the correct environment variable.

[frankscholten]

@nubunto Thanks for looking into this!

• Are environment variables supported? Or does ES 5.0 only look at elasticsearch.yml? I guess we have to generate the file inside the Docker image and in the Java executor.

• The vm_max_count could be set before the executor starts.

• Ok. We could support flags for choosing between 2.0 and 5.0. How can and existing Mesos Es cluster using 2.0 be upgraded to an Mesos Es using 5.0? Anything in particular that has to be done in terms of conversion?

[nubunto]

@frankscholten no problem!

• They are, with the only exception being ES_HEAP_SIZE for ES 5.0+. If we set that, then he refuses to start. All other configurations have to go to elasticsearch.yml. I believe I can get in contact with someone from the ES team to confirm this, which I'll do this afternoon.
• That's great! I couldn't find an easy way to do this while poking at the code, but I'm glad it's trivial.
• We need to ditch ES_HEAP_SIZE in favor of ES_JAVA_OPTS if ES 5.0 is detected, but keep it for versions < 2.x. We also need to move all other configurations from command line to elasticsearch.yml (needs confirmation). Unfortunately, since 5.0 is breaking things, is inevitable that we may also break stuff as well, I suppose :(

[albertostratio]

@nubunto Thank you very much for looking into this. We are very interested in having an ES 5.0 working cluster. Are you still working on this? Any progress?

[nubunto]

@albertostratio no problem! 😄

So, I got to a point where I can deploy ES 5.0 and it doesn't fail every second. We can reach it normally. The problem is:

1. I had to remove the part of the code that passes arguments to these instances
2. Backwards compatibility with other ES versions (< 5.0) is not trivial.

I guess I should detach from this, since Java is not my main game. I'll publish a branch with those changes and hopefully you guys can take that forward.
If I can be of any more help, please, let me know.

[nubunto]

opened up PR #597

[jcortejoso]

@nubunto I'm having some problems trying to build thermeon:es-5.0. Can you build it correctly? Apart from some trivial changes as changing new List to new ArrayList, I think we still need to pass the ES configuration that was built by esArguments and passed by args, someway with ES 5.x (elasticsearch.yml and/or jvm.options files).

[davison]

I started looking at this as we want a v5.2.0 cluster on DC/OS. I think that the command line arguments can still be passed to an official elasticsearch docker image, although the format is different; see option "D" in the reference docs. Broadly, cmdline opts are passed with -Esome.option=value

[nubunto]

@jcortejoso I'll give it a look and get back to you

[davison]

I took a fork of the repo and tried to build a local docker image using the gradle scripts (gradle buildDockerImage) but this fails because it seems to have some dependency on a webserver running on localhost:80 where it tries to POST /build?t=mesos/elasticsearch-base. I couldn't find a mesos/elasticsearch-base on dockerhub either which is required as the FROM image in the scheduler's docker file. Can someone let me know what else I need in order to build a local copy of the scheduler from these sources?

[rsingh2411]

Refer #551
The Base image is in the commons folder. You can follow the build instructions in the documentation here:https://github.com/mesos/elasticsearch/blob/master/docs/index.md#how-to-run-on-mac