elasticsearch 2.x support #11
Conversation
Use the CloudFormation AWS::Logs::SubscriptionFilter resource
The class ```import org.elasticsearch.common.lang3.StringUtils;``` is not used in elasticsearch 2.x, it is instead called directly using ```import org.apache.commons.lang3.StringUtils;```. This PR has a new ```src\main\java\com\amazonaws\services\logs\connectors\elasticsearch\CloudWatchLogsElasticsearch2Document.java``` with the changes, as well as a new ```pom.xml``` working example (working on Centos 6.7).
|
👍 |
|
Hi Dan, Thanks for your writeup. I managed to get the library compiled as per your suggestions but landed with an issue where in the Kinesis client API tries to query DynamoDB for the Lease information and in my case if fails with a HTTP error "Connection Timeout". This to me seems like the library cant connect to DynamoDB. However I am able to create table, etc using AWS CLI on the same EC2 machine. So not sure what I am missing. Did you come across this problem? |
|
@gamefundas I didn't run into this specific problem, can you post a Gist link with the full output? I don't have time to look at it today but I can probably get back to you tomorrow. |
|
Dan, I am getting the following exception. Assuming this has to do with Http proxy on my end as the Lease table is created in DynamoDB but the API fails to connect to it. Verified that I have IAM role/privileges on the table, my AWS CLI works on the table from the EC2 instance. Posted an issue for the kinesis-connector project as I am thinking there is no proxy support at the moment. Whats going on in the library is beyond me. So I am hoping if someone responds back. Please let me know if you recognize the problem. 2016-02-20 02:57:10,741 INFO AmazonHttpClient - Unable to execute HTTP request: connect timed out |
|
@gamefundas are you using sticky sessions for your proxy? I'm not sure if that's the problem but maybe a good place to start, also have you verified you can actually connect through your proxy? I would setup something simple, maybe a Python script that uses boto and see if you can do a simple query to rule out a connectivity issue then go from there, I'm at a loss otherwise :( |
Purpose
Currently there is no elasticsearch 2.x.x support in the
cloudwatch-logs-subscription-consumerlibrary. This PR is a first step forward towards getting support implemented.Status
DEPENDS ON: amazon-archives/amazon-kinesis-connectors#63
WORK IN PROGRESS / NEED HELP
Related Issues and PRs
Background
I was enticed to use the cloudwatch-logs-subscription-consumer by @dvassallo one day and went about trying to get it to work with our already implemented elasticsearch 2.0.0 installation in AWS. By figuring out what had changed between the 1.x and 2.x versions of elasticsearch (great reference here) I was then able to go through the
amazon-kinesis-connectorscode and thecloudwatch-subscription-consumercode and make some changes to support 2.x. I then built a local.jarforamazon-kinesis-connectorsafter making somepom.xmlchanges. Followed by changes tocloudwatch-logs-subscription-consumerthat fixedImmutableSettingsand re-wrote theElasticsearchEmitterfunction to use the methods outlined in the java api doc here.I was then able to make a successful connection to elasticsearch and see records start coming through. There are still a few things to iron out, they are outlined below. But the hope is, people who really need it can start using the configs in the PR and then soon it will be merged and fully functional.
ToDos
Other Notes