Skip to content

Commit

Permalink
Added variable MAX_REQUEST_SIZE to optionally increase the request si…
Browse files Browse the repository at this point in the history
…ze when sending records to Kafka (#116)

* Added MAX_REQUEST_SIZE Kafka producer variable for Metastore events

* Added MAX_REQUEST_SIZE Kafka producer variable for Metastore events

* changelog
  • Loading branch information
javsanbel2 authored Aug 2, 2023
1 parent cb5623f commit dae86aa
Show file tree
Hide file tree
Showing 5 changed files with 16 additions and 3 deletions.
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [3.0.12] - 2023-08-02
### Added
- Added variable `MAX_REQUEST_SIZE` to optionally increase the request size when sending records to Kafka.
- Upgraded `APIARY_EXTENSIONS_VERSION` to `7.3.8` (was `7.3.7`).
- Upgraded `APIARY_GLUESYNC_LISTENER_VERSION` to `7.3.8` (was `7.3.7`).

## [3.0.11] - 2023-07-25
### Added
- Added variable `KAFKA_COMPRESSION_TYPE` to optionally add compression type when sending Metastore events to Kafka through apiary-metastore-listener library.
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
from amazonlinux:2

ENV RANGER_VERSION 2.0.0
ENV APIARY_EXTENSIONS_VERSION 7.3.7
ENV APIARY_GLUESYNC_LISTENER_VERSION 7.3.7
ENV APIARY_EXTENSIONS_VERSION 7.3.8
ENV APIARY_GLUESYNC_LISTENER_VERSION 7.3.8
ENV APIARY_RANGER_PLUGIN_VERSION 5.0.1
ENV APIARY_METASTORE_METRICS_VERSION 4.2.0
ENV APIARY_METASTORE_AUTH_VERSION 4.2.0
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
|INSTANCE_NAME|Yes|Apiary instance name, will be used as prefix on most AWS resources to allow multiple Apiary instance deployments.|
|KAFKA_BOOTSTRAP_SERVERS|No|Kafka Bootstrap Servers to enable Kafka Metastore listener and send Metastore events to Kafka.|
|KAFKA_CLIENT_ID|No|Kafka label you define that names the Kafka producer.|
|KAFKA_COMPRESSION_TYPE|No|Kafka Compression type, if none is specified there is no compression enabled. Values available are gzip, lz4 and snappy.|
|KAFKA_COMPRESSION_TYPE|No (defaults to `1048576`)|The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum uncompressed record batch size.|
|KAFKA_MAX_REQUEST_SIZE|No|Kafka Compression type, if none is specified there is no compression enabled. Values available are gzip, lz4 and snappy.|
|LDAP_BASE|No|LDAP base DN used to search for user groups.|
|LDAP_CA_CERT|Base64 encoded Certificate Authority Bundle to validate LDAP SSL connection.|
|LDAP_SECRET_ARN|No|LDAP bind DN SecretsManager secret ARN.|
Expand Down
5 changes: 5 additions & 0 deletions files/hive-site.xml
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@
<value>none</value>
</property>

<property>
<name>com.expediagroup.apiary.extensions.events.metastore.kafka.messaging.max.request.size</name>
<value>1048576</value>
</property>

<property>
<name>hive.metastore.disallow.incompatible.col.type.changes</name>
<value>true</value>
Expand Down
1 change: 1 addition & 0 deletions files/startup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ if [[ ! -z $KAFKA_BOOTSTRAP_SERVERS ]]; then
[[ -n $KUBERNETES_SERVICE_HOST ]] && export KAFKA_CLIENT_ID="$HOSTNAME"
[[ -n $KAFKA_CLIENT_ID ]] && sed "s/KAFKA_CLIENT_ID/$KAFKA_CLIENT_ID/" -i /etc/hive/conf/hive-site.xml
[[ -n $KAFKA_COMPRESSION_TYPE ]] && update_property.py com.expediagroup.apiary.extensions.events.metastore.kafka.messaging.compression.type "$KAFKA_COMPRESSION_TYPE" /etc/hive/conf/hive-site.xml
[[ -n $KAFKA_MAX_REQUEST_SIZE ]] && update_property.py com.expediagroup.apiary.extensions.events.metastore.kafka.messaging.max.request.size "$KAFKA_MAX_REQUEST_SIZE" /etc/hive/conf/hive-site.xml
fi

APIARY_S3_INVENTORY_SCHEMA=s3_inventory
Expand Down

0 comments on commit dae86aa

Please sign in to comment.