Skip to content

Commit

Permalink
Elasticsearch 8 compatibility - set default mapping type to null (#81)
Browse files Browse the repository at this point in the history 
* HCBatchOperations constructor deprecated
* HCHttp.mappingType deprecated and moved to ElasticsearchBulkApi
* ApiRequestFactory added to allow custom client API builders
* ElasticsearchBulkApi added to handle Elasticsearch builders
* ElasticsearchBulkApiPlugin added to handle Log4j2 config
  • Loading branch information
@rfoltyns
rfoltyns committed Oct 5, 2022
1 parent b036ef0 commit 5a0d3cb
Show file tree
Hide file tree
Showing 32 changed files with 1,147 additions and 257 deletions.
3 changes: 2 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Latest released code (1.5.x) is available [here](https://github.com/rfoltyns/log

Project consists of:
* `log4j2-elasticsearch-core` - skeleton provider for conrete implementations
* `log4j2-elasticsearch-hc` - optimized Apache Async HTTP client compatible with Elasticsearch 2.x, 5.x, 6.x and 7.x clusters
* `log4j2-elasticsearch-hc` - optimized Apache Async HTTP client compatible with Elasticsearch 2.x, 5.x, 6.x, 7.x and 8.x clusters
* `log4j2-elasticsearch-jest` - [Jest HTTP Client](https://github.com/searchbox-io/Jest) compatible with Elasticsearch 2.x, 5.x, 6.x and 7.x clusters
* `log4j2-elasticsearch2-bulkprocessor` - [TCP client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/2.4/java-docs-bulk-processor.html) compatible with 2.x clusters
* `log4j2-elasticsearch5-bulkprocessor` - [TCP client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/java-docs-bulk-processor.html) compatible with 5.x and 6.x clusters
Expand Down Expand Up @@ -39,6 +39,7 @@ Project consists of:
* [Component templates](log4j2-elasticsearch-core#component-templates) configuration
* [Composable index templates](log4j2-elasticsearch-core#composable-index-template) configuration
* [Service Discovery](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc#service-discovery) for HC module
* (1.6) Elasticsearch 8.x support (`null` mapping type)

### Roadmap [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/rfoltyns/log4j2-elasticsearch)

Expand Down
4 changes: 2 additions & 2 deletions ...lasticsearch-core/src/test/java/org/appenders/log4j2/elasticsearch/IndexTemplateTest.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,15 +68,15 @@ public void startsWhenSetupCorrectlyWithNonDefaultApiVersion() {
final IndexTemplate.Builder builder = createTestIndexTemplateBuilder();
builder.withName(TEST_INDEX_TEMPLATE)
.withPath(TEST_PATH)
.withApiVersion(8);
.withApiVersion(7);

// when
final IndexTemplate indexTemplate = builder.build();

// then
assertNotNull(indexTemplate);
assertNotEquals(IndexTemplate.DEFAULT_API_VERSION, indexTemplate.getApiVersion());
assertEquals(8, indexTemplate.getApiVersion());
assertEquals(7, indexTemplate.getApiVersion());
assertEquals(TEST_INDEX_TEMPLATE, indexTemplate.getName());
assertNotNull(indexTemplate.getSource());
assertEquals(IndexTemplate.TYPE_NAME, indexTemplate.getType());
Expand Down
65 changes: 46 additions & 19 deletions log4j2-elasticsearch-hc/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,19 +42,20 @@ It's highly recommended to put this plugin behind `AsyncLogger`. See [log4j2.xml
```

### HCHttp Properties
| Name | Type | Required | Default | Description |
|----------------------------------|-----------|-----------------------------------------------------------------|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| serverUris | Attribute | no (MUST be specified by either `HCHttp` or `ServiceDiscovery`) | None | List of semicolon-separated `http[s]://host:[port]` addresses of Elasticsearch nodes to connect with. |
| connTimeout | Attribute | no | 1000 | Number of milliseconds before ConnectException is thrown while attempting to connect. |
| readTimeout | Attribute | no | 0 | Number of milliseconds before SocketTimeoutException is thrown while waiting for response bytes. |
| maxTotalConnections | Attribute | no | 8 | Number of connections available. |
| ioThreadCount | Attribute | no | No. of available processors | Number of `I/O Dispatcher` threads started by Apache HC `IOReactor` |
| itemSourceFactory | Element | yes | None | `ItemSourceFactory` used to create wrappers for batch requests. `PooledItemSourceFactory` and it's extensions can be used. |
| mappingType | Attribute | no | `_doc` | Name of index mapping type to use in ES cluster. `_doc` is used by default for compatibility with Elasticsearch 7.x. |
| pooledResponseBuffers | Attribute | no | yes | If `true`, pooled `SimpleInputBuffer`s will be used to handle responses. Otherwise, new `SimpleInputBuffer` wil be created for every response. |
| pooledResponseBuffersSizeInBytes | Attribute | no | 1MB (1048756 bytes) | Single response buffer size. |
| auth | Element | no | None | Security config. [Security](#pem-cert-config) |
| serviceDiscovery | Element | no | None | Service discovery config. [ServiceDiscovery](#service-discovery) |
| Name | Type | Required | Default | Description |
|----------------------------------|-----------|-----------------------------------------------------------------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| serverUris | Attribute | no (MUST be specified by either `HCHttp` or `ServiceDiscovery`) | None | List of semicolon-separated `http[s]://host:[port]` addresses of Elasticsearch nodes to connect with. |
| connTimeout | Attribute | no | 1000 | Number of milliseconds before ConnectException is thrown while attempting to connect. |
| readTimeout | Attribute | no | 0 | Number of milliseconds before SocketTimeoutException is thrown while waiting for response bytes. |
| maxTotalConnections | Attribute | no | 8 | Number of connections available. |
| ioThreadCount | Attribute | no | No. of available processors | Number of `I/O Dispatcher` threads started by Apache HC `IOReactor` |
| itemSourceFactory | Element | yes | None | `ItemSourceFactory` used to create wrappers for batch requests. `PooledItemSourceFactory` and it's extensions can be used. |
| mappingType | Attribute | no | `null` since 1.6 | Name of index mapping type to use. Applicable to Elasticsearch <8.x. See [removal of types](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html). <br/> DEPRECATED: As of 1.7, this attribute will be removed. Use [ElasticsearchBulk](#elasticsearchbulk) instead. |
| pooledResponseBuffers | Attribute | no | yes | If `true`, pooled `SimpleInputBuffer`s will be used to handle responses. Otherwise, new `SimpleInputBuffer` wil be created for every response. |
| pooledResponseBuffersSizeInBytes | Attribute | no | 1MB (1048756 bytes) | Single response buffer size. |
| auth | Element | no | None | Security config. [Security](#pem-cert-config) |
| serviceDiscovery | Element | no | None | Service discovery config. [ServiceDiscovery](#service-discovery) |
| clientAPIFactory | Element | no | `ElasticsearchBulk` | Batch API factory. [ElasticsearchBulk](#elasticsearchbulk) |

### Service Discovery

Expand Down Expand Up @@ -123,6 +124,29 @@ Example:

NOTE: Config policies were added for convenience. Recommended configuration should contain `configPolices=none`, `serverUris` configured ONLY for `ServiceDiscovery` and separate `Security` configs if needed.

### Client API Factory

Since 1.6, [ClientAPIFactory](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc/src/main/java/org/appenders/log4j2/elasticsearch/hc/ClientAPIFactory.java) can be configured to further customize the output and runtime capabilities of batches and batch items. [ElasticsearchBulk](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc/src/main/java/org/appenders/log4j2/elasticsearch/hc/ElasticsearchBulkPlugin.java) is used by default.

#### ElasticsearchBulk

Default.

Configures builders and serializers for:
* [BatchRequest](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc/src/main/java/org/appenders/log4j2/elasticsearch/hc/BatchRequest.java) - `/_bulk` request (batch)
* [IndexRequest](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc/src/main/java/org/appenders/log4j2/elasticsearch/hc/IndexRequest.java) - document (batch item)

```xml
<HCHttp>
<ElasticsearchBulk/>
</HCHttp>
```

#### ElasticsearchBulk Properties
| Name | Type | Required | Default | Description |
|-------------|-----------|----------|------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| mappingType | Attribute | no | `null` since 1.6 | Name of index mapping type to use. Applicable to Elasticsearch <8.x. See [removal of types](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html). |

### Programmatic config
See [programmatc config example](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-hc/src/test/java/org/appenders/log4j2/elasticsearch/hc/smoke/SmokeTest.java).

Expand Down Expand Up @@ -167,6 +191,9 @@ Example:
See [index name](../log4j2-elasticsearch-core#index-name) or [index rollover](../log4j2-elasticsearch-core#index-rollover)

### Index template

Since 1.6, this module is compatible with Elasticsearch 8.x by default. Use `apiVersion` for older clusters.

See [index template docs](../log4j2-elasticsearch-core#index-template)

### SSL/TLS
Expand Down Expand Up @@ -200,12 +227,12 @@ Can be configured using `Security` tag:

### Compatibility matrix

| Feature/Version | 2.x | 5.x | 6.x | 7.x |
|------------------|------------|------------|------------|------------|
| IndexTemplate | Yes | Yes | Yes | Yes |
| BasicCredentials | Yes | Yes | Yes | Yes |
| JKS | Yes | Not tested | Not tested | Not tested |
| PEM | Not tested | Yes | Yes | Yes |
| Feature/Version | 2.x | 5.x | 6.x | 7.x | 8.x |
|------------------|------------|------------|------------|------------|------------|
| IndexTemplate | Yes | Yes | Yes | Yes | Yes |
| BasicCredentials | Yes | Yes | Yes | Yes | Yes |
| JKS | Yes | Not tested | Not tested | Not tested | Not tested |
| PEM | Not tested | Yes | Yes | Yes | Yes |

## Pluggable JCTools

Expand Down
2 changes: 1 addition & 1 deletion log4j2-elasticsearch-hc/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<modelVersion>4.0.0</modelVersion>
<artifactId>log4j2-elasticsearch-hc</artifactId>
<name>Log4j2 Elasticsearch Apache HC HTTP client</name>
<description>Log4j2 Appender plugin pushing logs in batches to Elasticsearch (2.x/5.x/6.x/7.x) clusters</description>
<description>Log4j2 Appender plugin pushing logs in batches to Elasticsearch (2.x/5.x/6.x/7.x/8.x) clusters</description>

<dependencies>

Expand Down
Loading

0 comments on commit 5a0d3cb

Please sign in to comment.