[EAGLE-906] org.apache.storm 1.x with API changes

[jhsenjaliya]

tested on MR Running and MR history app.

[haoch]

@jhsenjaliya building failed, could you please check again?

[haoch]

Have you tested about consuming kafka:0.8.1 with storm-kafka:1.0.2 ?

[jhsenjaliya]

i havent explicitly tested it, but from storm-kafka:1.0.2 docs, it looks compatible with kafka 0.8.1.1 which is used in the example given on that page.
Also eagle repo includes kafka 0.8.2.2, wouldnt it test the compatibility?
is there any other way we can test this?

[haoch]

Just worry about possible compatibility issue as kafka which was refactored a lot recently and here are some tricky dependencies behavior in storm-kafka

[jhsenjaliya]

ok, I m digging further into this, I will try running some apps. pls let me know if you have any other suggestions.

[jhsenjaliya]

@haoch, I tested all of these change with kafka_2.11-0.8.2.2 and it works fine.
I ran hdfs audit log and alert application to test.

[jhsenjaliya]

need to rebase branch-0.5-storm_1.x with master. changing the base to master for time being.

[jhsenjaliya]

i will update this PR with storm 1.1.0 and rebase it after 0.5 release.

[r7raul1984]

@jhsenjaliya In my opition , I don't think it is good time to upgrade storm to 1.1.0. In my experience ,many companies do not use the latest version .If you insist to upgrade storm, I recommend we should support multi storm version (like hive shims moudle) and do more integration test instead of unit test by mock.

[jhsenjaliya]

@r7raul1984 , that is a good point, but this is intended for 0.6 release which will have enough time to take a minor versions with bug fixes if any ( or we can also simply stick to 1.0.x but I would like to compare both 1.0.x and 1.1.x in terms of improvements before change) .

I also agree on exploring a way to support multi version shims way.