Can't authenticate as AWS Elasticsearch Service does not return WWW-Authenticate header

[pushpithaDilhan]

Description
I used an AWS Elasticsearch instance (doesn't support x-pack) with fine grained access control. Used master username and password for basic credentials. I got below exception.

2020-05-31 23:52:57,142 I/O dispatcher 1 WARN Unrecognized token 'Unauthorized': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (org.appenders.log4j2.elasticsearch.hc.ItemSourceContentInputStream); line: 1, column: 13] com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Unauthorized': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (org.appenders.log4j2.elasticsearch.hc.ItemSourceContentInputStream); line: 1, column: 13] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:722) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3556) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2651) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:856) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:753) at com.fasterxml.jackson.databind.ObjectReader._initForReading(ObjectReader.java:357) at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1704) at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1228) at org.appenders.log4j2.elasticsearch.hc.HCHttp$1.deserializeResponse(HCHttp.java:190) at org.appenders.log4j2.elasticsearch.hc.HCHttp$1.deserializeResponse(HCHttp.java:158) at org.appenders.log4j2.elasticsearch.hc.HCResultCallback.completed(HCResultCallback.java:55) at org.appenders.log4j2.elasticsearch.hc.HCResultCallback.completed(HCResultCallback.java:38) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.lang.Thread.run(Thread.java:748)

Can you please give me a solution ?

Configuration
<Elasticsearch name="elasticsearch"> <IndexName indexName="customerdata"/> <JacksonJsonLayout> <PooledItemSourceFactory poolName="itemPool" itemSizeInBytes="1024" initialPoolSize="3000"/> </JacksonJsonLayout> <AsyncBatchDelivery batchSize="1000" deliveryInterval="10000" > <HCHttp serverUris="${env:AWS_ES_URL}"> <Security> <BasicCredentials username="USERNAME" password="PASSWORD" /> </Security> <PooledItemSourceFactory poolName="batchPool" itemSizeInBytes="1024000" initialPoolSize="3"/> </HCHttp> </AsyncBatchDelivery> </Elasticsearch>

Runtime (please complete the following information):

• log4j2-elasticsearch-hc:1.4.1
• ES version 7.4
• JVM openJDK
• OS: Ubuntu

[rfoltyns]

Wow, I expected AWS ES to produce a "nicer response". I'll look into it.

I the meantime - since you've mentioned fine grained access control - make sure that you can cURL following endpoints:

• PUT $AWS_ES_URL/_template/<template_name> - create mappings for your index; you don't use it yet, but it might become handy in a moment
• POST $AWS_ES_URL/_bulk - send bulk documents; apparently you're not authorized to perform this action at the moment

[rfoltyns]

Do you use IAM or internal database? I'd assume internal database since you'd like to supply credentials, but just wanted to make sure before I spend cash to reproduce it :)

[pushpithaDilhan]

Hi @rfoltyns, thanks for looking into this. Unfortunately, this was done in our organizational non-prod ES instance.

I tried executing below cURL command and it worked.
curl -XPOST "ES_URL/testindex/_doc" -H 'Content-Type: application/json' -d '{"data":"test-data"}' -u username:password

Also I tried working with ECE trial version installed in an EC2, in that case it worked because ECE supports x-pack.

[rfoltyns]

Could you also try the cURL below (index document with Bulk API)? Make sure that the index does not exist before running the cURL. Let's verify that the user has following permissions in ES:

• indices:create_index
• indices:write
  at customerdata index (or testindex, or whichever name you choose)

curl -X POST '$AWS_ES_URL/_bulk' \
-H 'Authorization: Basic <user:pass to base64>' \
-H 'Content-Type: application/json' \
--data "
{\"index\":{\"_index\":\"customerindex\",\"_type\":\"_doc\"}}
{\"timestamp\":1591030115187,\"loggerName\":\"test-logger\"}
"

You may also need to create an index template for your index. It will be needed so you can properly create an index pattern in Kibana to make logs visible. You'll need cluster:manage_index_templates permission to do that.

[rfoltyns]

I've found the issue. AWS Elasticsearch Service does not send back the WWW-Authenticate header in the response, so Apache HTTP client cannot find the auth challenge to continue with. At first glance, it looks like a bug in AWS/OpenDistro, but maybe I'm missing something.

In my case, Open Distro config looks correct ("http_authenticator": { "challenge:" true, "type": "basic" }):

GET _opendistro/_security/api/securityconfig
{
    "config": {
        "dynamic": {
            "filtered_alias_mode": "warn",
            "disable_rest_auth": false,
            "disable_intertransport_auth": false,
            "respect_request_indices_options": false,
            "kibana": {
                "multitenancy_enabled": true,
                "server_username": "AmazonESKibanaServerUser",
                "index": ".kibana"
            },
            "http": {
                "anonymous_auth_enabled": false,
                "xff": {
                    "enabled": false,
                    "internalProxies": "10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|192\\.168\\.\\d{1,3}\\.\\d{1,3}|169\\.254\\.\\d{1,3}\\.\\d{1,3}|127\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|172\\.1[6-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.2[0-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.3[0-1]{1}\\.\\d{1,3}\\.\\d{1,3}",
                    "remoteIpHeader": "X-Forwarded-For"
                }
            },
            "authc": {
                "basic_internal_auth_domain": {
                    "http_enabled": true,
                    "transport_enabled": true,
                    "order": 4,
                    "http_authenticator": {
                        "challenge": true,
                        "type": "basic",
                        "config": {}
                    },
                    "authentication_backend": {
                        "type": "intern",
                        "config": {}
                    },
                    "description": "Authenticate via HTTP Basic against internal users database"
                }
            },
            "authz": {},
            "auth_failure_listeners": {},
            "do_not_fail_on_forbidden": false,
            "multi_rolespan_enabled": true,
            "hosts_resolver_mode": "ip-only",
            "do_not_fail_on_forbidden_empty": false
        }
    }
}

However, update is not possible.

PATCH _opendistro/_security/api/securityconfig
[
  {
    "op": "replace", "path": "/config/dynamic/authc/basic_internal_auth_domain/order", "value": 1
  }
]

Resp:
{
    "Message": "Your request: '/_opendistro/_security/api/securityconfig' is not allowed."
}

[pushpithaDilhan]

Hi @rfoltyns,

I executed below command as you said, and it worked. particular record was added to the index with 201 response.

curl -X POST '$AWS_ES_URL/_bulk' \
-H 'Authorization: Basic <user:pass to base64>' \
-H 'Content-Type: application/json' \
--data "
{\"index\":{\"_index\":\"customerindex\",\"_type\":\"_doc\"}}
{\"timestamp\":1591030115187,\"loggerName\":\"test-logger\"}
"

I tried to add a record to an index using below cURL, and it also worked.

curl -X POST 'ES_URL/<index>/_doc' \
-H 'Authorization: Basic <user:pass to base64>' \
-H 'Content-Type: application/json' \
--data "
{
  \"testattrib\":\"testdata\"
}
"

Furthermore I think, there may be some configuration from AWS ES instance to make it work. I am studying this article which explains how Basic auth is handled with fine-grained access control. I'll update if I found anything.

Thanks.

[rfoltyns]

Requests suceeded, that's great! It means that auth works (sort of). Try to remove the Authorization header. AWS ES should return 401 with WWW-Authenticate header (as regular ES with auth setup do). If it doesn't, client has no way to proceed with the request.

Can you raise an issue with AWS Support to address this? I can't do it with my basic subscription, but maybe your company can.

[pushpithaDilhan]

Since this hc-logappender uses Apache:httpcomponents-asyncclient, I implemented below sample java app to send a record to our authenticated ES instance. It was successful with 201 response. (I used httpasyncclient : 4.0.1)

import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.json.JSONObject;

public class App {
    public static void main(String[] args) throws Exception {

        System.setProperty("org.apache.commons.logging.Log","org.apache.commons.logging.impl.SimpleLog");
        System.setProperty("org.apache.commons.logging.simplelog.showdatetime", "true");
        System.setProperty("org.apache.commons.logging.simplelog.log.org.apache.http.wire", "DEBUG");

        CloseableHttpClient client = HttpClients.createDefault();
        HttpPost httpPost = new HttpPost("ES_URL/<index>/_doc");
        httpPost.setHeader("Accept", "application/json");
        httpPost.setHeader("Content-type", "application/json");
        JSONObject entity = new JSONObject();
        entity.append("testkey1", "testvalue1");
        httpPost.setEntity(new StringEntity(entity.toString()));

        UsernamePasswordCredentials creds
                = new UsernamePasswordCredentials("username", "password");
        httpPost.addHeader(new BasicScheme().authenticate(creds, httpPost, null));

        CloseableHttpResponse response = client.execute(httpPost);
        System.out.println("Response: " + response.getStatusLine());
        System.out.println("Shutting down");
        client.close();
    }
}

Here this application uses UsernamePasswordCredentials as auth-provider class.
Then I went through the implementation of hc-logappender plugin and realized, in this plugin you have used BasicCredentialsProvider. So I implemented the same application with BasicCredentialsProvider auth-provider as below.

import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.json.JSONObject;

public class App2 {

    public static void main(String[] args) throws Exception {

        System.setProperty("org.apache.commons.logging.Log","org.apache.commons.logging.impl.SimpleLog");
        System.setProperty("org.apache.commons.logging.simplelog.showdatetime", "true");
        System.setProperty("org.apache.commons.logging.simplelog.log.org.apache.http.wire", "DEBUG");

        BasicCredentialsProvider basicCredentialsProvider = new BasicCredentialsProvider();
        basicCredentialsProvider.setCredentials(
                AuthScope.ANY,
                new UsernamePasswordCredentials("username", "password")
        );
        CloseableHttpClient client = HttpClientBuilder.create()
                .setDefaultCredentialsProvider(basicCredentialsProvider)
                .build();

        HttpPost httpPost = new HttpPost("ES_URL/<index>/_doc");
        httpPost.setHeader("Accept", "application/json");
        httpPost.setHeader("Content-type", "application/json");
        JSONObject entity = new JSONObject();
        entity.append("testkey1", "test-2");
        httpPost.setEntity(new StringEntity(entity.toString()));
        CloseableHttpResponse response = client.execute(httpPost);
        System.out.println("Response: " + response.getStatusLine());
        System.out.println("Shutting down");
        client.close();
    }
}

And this failed with 401 Unauthorized. So I think the issue is with the credential provider class used in log-appender plugin.

[rfoltyns]

Nice one! Notice that it works once you provide the header explicitly (of course! just like the cURL request). Credentials provider is used only during challenges, so everything works as expected.

However, HC module doesn't support explicit auth headers config. IMHO it's not an issue - it's just based on auth challenges. It works perfectly with Elasticsearch with auth. Open Distro should produce this header as configured (according to the docs), but it doesn't - the issue is there.

You can extend HCHttp.createClientProvider(), provide custom HttpClientFactory.createConfiguredClient() method and add headers in custom HCRequestFactory on your own. HCHttp was designed with extensibility in mind. Is that an acceptable solution?

[pushpithaDilhan]

Sure, I'll try that approach and let you know.

[rfoltyns]

I've checked OpenDistro 1.4.0 (7.4.2 equivalent, "latest" AWS ES) and 1.8.0 locally - both are responding with WWW-Authenticate header on lack of Authorization in the request, so the issue is 100% on AWS side.

If only AWS allowed to raise issues with basic subscription..

[pushpithaDilhan]

Sure, I'll raise this issue to AWS.

[pushpithaDilhan]

I raised a support case and got below response.

The FGAC feature does not support Credentials providers at the moment. To use Credentials Providers you have to integrate with Cognito to use SAML or other supported authentication providers. AWS Elasticsearch is not able at the moment to read user credentials and determine their authentication scope directly from Credentials providers. That is why that part is handled by UserPools in Cognito which can read the credentials from the authentication provider and then pass them to IAM which stores the domain's access policy. The "BasicCredentialsProvider" is a simple implementation default CredentialsProvider which is backed by a java.util.HashMap.

Like you mentioned in the case, The 401 (Unauthorized) response message you got requires a response which must include a "WWW-Authenticate" header, but AWS Elasticsearch does not support that yet. AWS Elasticsearch is a bit different from OpenDistro for Elasticsearch. The FGAC feature is enabled just as a plugin not part of the Elasticsearch instance.

As explained here, I will try with integrating Cognito.

[rfoltyns]

must include a "WWW-Authenticate" header, but AWS Elasticsearch does not support that yet

yet gives us some hope here..

Cognito might face similar issues - I think they misread Credentials providers here.

Can you use IAM? I think it can be used for traffic from EC2. Is your instance accessed from outside of AWS?

[pushpithaDilhan]

Yeah, I think they misread it. Cognito is used for SSO. Using an IAM won't work as data is ingested through many sources.