Remove topic creation from monasca/kafka #138

kornicameister · 2017-08-01T07:05:52Z

Kafka init

Moved kafka topic creation to external image. That allows reusing the image from different locations.

Partially_closes #134

Moved kafka topic creation to external image. That allows reusing the image from different locations. Closes #134

matrixik · 2017-08-07T11:46:40Z

kafka-init/README.md

+
+
+[1]: http://semver.org/
+[2]: https://github.com/monasca/monasca-docker/kafka


This will give 404

matrixik · 2017-08-07T11:48:40Z

kafka-init/README.md

+| `KAFKA_TIMEOUT`               | `60`      | How long to wait for Kafka to become available |
+| `KAFKA_CREATE_TOPICS`         | `unset`   | Topics to create on startup, see below       |
+| `KAFKA_TOPIC_CONFIG`          | `unset`   | Default config args for created topics       |
+| `KAFKA_CREATE_TOPICS_SCRIPT`  | `KAFKA_CREATE_TOPICS_SCRIPT` | Path to script that creates topics |


Should default be /kafka/bin/kafka-topics.sh?

timothyb89 · 2017-08-03T15:59:54Z

kafka-init/README.md

+```
+
+As an example, this is a valid `KAFKA_CREATE_TOPICS` string for [Monasca][8]
+installations as used in the [docker-compose][4] environment:


Looks like these links are broken

Fixed, also removed unused refs from kafka/README.md

timothyb89 · 2017-08-08T17:14:17Z

This seems to work fine in docker-compose. We'll need to bump the version in kafka/build.yml to push a new image, though. I'd say it should at least be a minor version bump (1.0.2 -> 1.1.0). I made some nodes about versioning concerns but I'm not sure if it's worth changing, I'll let you decide 😄

Also, we'll need to coordinate things a bit with monasca-helm - can't bump the kafka version there until we have a kafka-init chart!

timothyb89 · 2017-08-08T17:05:49Z

kafka-init/Dockerfile

+ARG KAFKA_VERSION
+ARG SCALA_VERSION
+
+FROM monasca/kafka:${KAFKA_VERSION:-"0.9.0.1"}-${SCALA_VERSION:-"2.11"}


I'm not sure if it's a big issue or not, but the pr-bot will not be able to parse this FROM tag (and I'm not sure there's a reasonable way to parse it given the the bot's resource model). So this image won't get automatic updates from the base kafka image. We can either live with that (really not too bad) or use a plain FROM monasca/kafka:0.9.0.1-2.11.

BTW, any reason to use ${KAFKA_VERSION:-"0.9.0.1"} rather than just ARG KAFKA_VERSION=0.9.0.1? Does docker not evaluate them properly in FROM or anything like that?

(it might also be good to depend on the explicit container version somehow, e.g. monasca/kafka:${KAFKA_VERSION}-${SCALA_VERSION}-1.0.2)

Replaced that with single ARG. Basically there's no point in having multiple variables as we inherit from the monasca/kafka image here. Should be much simpler now and easier to parse.

timothyb89 · 2017-08-08T17:06:44Z

kafka-init/build.yml

+variants:
+  - tag: 0.0.1
+    aliases:
+        - ':latest'


super nit pick, this line is over-indented

kornicameister · 2017-08-09T06:12:59Z

Upgraded minor version to 1.1.0.
About the Helm...sadly the requirements has changed in a way where we don't want to go with Helm for now. If you really want me to handle that bit, I will do that - it's not in anyone's intention to mess with someones backyard. Just let me know what you think about that..

Changed buliding of kafka-init image to support just single argument as marker of parent monasca/kafka image. Bumped version of monasca/kafka to ```0.9.0.1-2.11-1.1.0``` from ```0.9.0.1-2.11-1.0.2```.

kornicameister · 2017-08-09T07:19:47Z

;-(...seems like it does not build because it cannot find locally build (or kafka-init build happens prior to tagging new monasca/kafka). monasca/kafka:0.9.0.1-2.11-1.1.0.

Are there any changes required in the ci.py or dbuild to pick up such nits and organize the build or should I split the PR into 2 ?

artur-ba · 2017-08-09T11:21:07Z

Tested this in my local env and it works. Topics were created in Kafka and I was able to list them.

timothyb89 · 2017-08-09T16:03:13Z

kafka/build.yml

+      - :latest
+    args:
+      KAFKA_VERSION: "0.9.0.1"
+      SCALA_VERSION: "2.11"
  - tag: 0.9.0.1-2.11-1.0.2


This tag should actually be removed or it will overwrite the existing image (needed in helm for now) with the same content as the new version.

timothyb89 · 2017-08-09T16:21:51Z

re: helm, that's no problem, we'll get around to adding it eventually - this has been on our to-do list for a while now, so you've already saved us a good bit of work 😄

As for the failure, it looks like you're right, kafka-init is being built before the new kafka image. Unfortunately the only way to guarantee they'll build in the correct order is to split the PR.

It would be nice to support some dependency chains but that would require changes to both the CI script and dbuild, since dbuild doesn't guarantee build order (as it may build in parallel). I filed monasca/dbuild#2 to track improvements to this, but I likely won't be able to prioritize it for a while.

kornicameister · 2017-08-10T07:30:41Z

Ok, I've splitted the PR, but unfortunately 0.9.0.1-2.11-1.0.2 is being rebuilt in CI and the Dockerfile meant for 0.9.0.1-2.11-1.1.0 is also used for 0.9.0.1-2.11.1.0.2 hence no topic are actually created inside the CI run.

I expected only 0.9.0.1-2.11-1.1.0 to be build. Is this expected behaviour ?

timothyb89 · 2017-08-10T17:45:04Z

The variant for ...-1.0.2 should be removed entirely from build.yml, it will always try to (re)build (and push) all variants any time build.yml changes (or !push is included in a merge commit).

timothyb89 · 2017-08-10T17:45:29Z

kafka/build.yml

+  - tag: 0.9.0.1-2.11-1.1.0
+    args:
+      KAFKA_VERSION: "0.9.0.1"
+      SCALA_VERSION: "2.11"
  - tag: 0.9.0.1-2.11-1.0.2


This -1.0.2 variant should be removed from build.yml entirely

Ok, I've removed the variant.

@timothyb89 is it like an "official approach"- we should drop older entries in build.yml to disallow overriding older images with changes meant for newer ones ?

@kornicameister I think so, yes. It's less of an issue for images that use date+time stamped tags since they'll always end up unique, but we definitely want to avoid introducing breaking changes by overwriting a tag that looks like it should be unique (like semver)

timothyb89 · 2017-08-10T17:50:06Z

As an FYI to any interested parties (@mhoppal) , we'll probably need to force this PR through since CI should fail. #153 is required to get topic creation working again, so merging this will break master until kafka-init is in.

Remove variant ```-1.0.2``` to disallow overriding it with changes meant for ```1.1.0``` variant.

timothyb89

Alright, looks good, let's try this thing!

timothyb89 · 2017-08-11T16:52:44Z

FYI master will be broken until #153 merges.

Kafka init

c595b10

Moved kafka topic creation to external image. That allows reusing the image from different locations. Closes #134

kornicameister changed the title ~~Kafka init~~ Move topic creation to seperate image Aug 1, 2017

kornicameister requested a review from timothyb89 August 1, 2017 07:49

kornicameister self-assigned this Aug 1, 2017

kornicameister added the enhancement label Aug 1, 2017

Merge branch 'master' into kafka-init

ee48a4d

kornicameister requested a review from mhoppal August 7, 2017 04:46

kornicameister changed the title ~~Move topic creation to seperate image~~ Add kafka-init container Aug 7, 2017

matrixik reviewed Aug 7, 2017

View reviewed changes

timothyb89 reviewed Aug 7, 2017

View reviewed changes

Fixed documentation

723d19b

timothyb89 reviewed Aug 8, 2017

View reviewed changes

Tomasz Trębski and others added 3 commits August 9, 2017 08:14

Upgrade & change version schema

f2a67c1

Changed buliding of kafka-init image to support just single argument as marker of parent monasca/kafka image. Bumped version of monasca/kafka to ```0.9.0.1-2.11-1.1.0``` from ```0.9.0.1-2.11-1.0.2```.

Merge branch 'master' into kafka-init

2d8e2f8

Be explicit about version in Dockerfile

4316bf1

timothyb89 reviewed Aug 9, 2017

View reviewed changes

kornicameister and others added 2 commits August 10, 2017 08:36

Merge branch 'master' into kafka-init

fab9f6e

Split kafka & kafka-init

0d6508a

kornicameister mentioned this pull request Aug 10, 2017

Add kafka-init container #153

Merged

kornicameister changed the title ~~Add kafka-init container~~ Remove topic creation from monasca/kafka Aug 10, 2017

Rollback topic creation variables

f80eabb

timothyb89 reviewed Aug 10, 2017

View reviewed changes

kornicameister mentioned this pull request Aug 10, 2017

Provide versioning of used images #145

Merged

Update build.yml

b96d739

Remove variant ```-1.0.2``` to disallow overriding it with changes meant for ```1.1.0``` variant.

timothyb89 approved these changes Aug 11, 2017

View reviewed changes

Merge branch 'master' into kafka-init

f35ba3f

timothyb89 merged commit 475caf6 into monasca:master Aug 11, 2017

timothyb89 mentioned this pull request Oct 12, 2017

kafka wait for topics? #248

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove topic creation from monasca/kafka #138

Remove topic creation from monasca/kafka #138

kornicameister commented Aug 1, 2017 •

edited

Loading

matrixik Aug 7, 2017

kornicameister Aug 8, 2017

matrixik Aug 7, 2017

kornicameister Aug 8, 2017

timothyb89 Aug 3, 2017

kornicameister Aug 8, 2017

timothyb89 commented Aug 8, 2017

timothyb89 Aug 8, 2017

kornicameister Aug 9, 2017

timothyb89 Aug 8, 2017

kornicameister Aug 9, 2017

kornicameister commented Aug 9, 2017

kornicameister commented Aug 9, 2017

artur-ba commented Aug 9, 2017

timothyb89 Aug 9, 2017

timothyb89 commented Aug 9, 2017

kornicameister commented Aug 10, 2017

timothyb89 commented Aug 10, 2017

timothyb89 Aug 10, 2017

kornicameister Aug 10, 2017

kornicameister Aug 10, 2017

timothyb89 Aug 10, 2017

timothyb89 commented Aug 10, 2017

timothyb89 left a comment

timothyb89 commented Aug 11, 2017



		[1]: http://semver.org/
		[2]: https://github.com/monasca/monasca-docker/kafka

Remove topic creation from monasca/kafka #138

Remove topic creation from monasca/kafka #138

Conversation

kornicameister commented Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothyb89 commented Aug 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kornicameister commented Aug 9, 2017

kornicameister commented Aug 9, 2017

artur-ba commented Aug 9, 2017

Choose a reason for hiding this comment

timothyb89 commented Aug 9, 2017

kornicameister commented Aug 10, 2017

timothyb89 commented Aug 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothyb89 commented Aug 10, 2017

timothyb89 left a comment

Choose a reason for hiding this comment

timothyb89 commented Aug 11, 2017

kornicameister commented Aug 1, 2017 •

edited

Loading