remove atlas metastore listener (#87)

* remove atlas metastore listener

* delete atlas from dockefile and remove atlas-application.properties

* remove everything related to atlas metastore listener

* update changelog

* update changelog

* fix changelog description

* update date in Changelog

CHANGELOG.md

@@ -3,6 +3,12 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [2.0.0] - 2021-05-06
+### Changed
+- Remove Atlas MetaStore listener in favor of internal processes that subscribe to the Kafka HMS event listener and push changes to Ranger.
+
+Note: This release is a *BREAKING* change that removes all support for the Apache Atlas HMS listener.
+
 ## [1.17.1] - 2020-11-18
 ### Changed
 - Enabled ranger audit log summarization.

Dockerfile

@@ -9,7 +9,6 @@ ENV APIARY_GLUESYNC_LISTENER_VERSION 4.2.0
 ENV APIARY_RANGER_PLUGIN_VERSION 5.0.1
 ENV APIARY_METASTORE_METRICS_VERSION 4.2.0
 ENV APIARY_METASTORE_AUTH_VERSION 4.2.0
-ENV ATLAS_VERSION 2.0.0
 ENV KAFKA_VERSION 2.3.1
 ENV COMMONS_CODEC_VERSION 1.12
 ENV GETHOSTNAME4J_VERSION 0.0.3
@@ -40,9 +39,6 @@ wget -q https://search.maven.org/remotecontent?filepath=com/expediagroup/apiary/
 wget -q https://search.maven.org/remotecontent?filepath=com/expediagroup/apiary/apiary-ranger-metastore-plugin/${APIARY_RANGER_PLUGIN_VERSION}/apiary-ranger-metastore-plugin-${APIARY_RANGER_PLUGIN_VERSION}-all.jar -O apiary-ranger-metastore-plugin-${APIARY_RANGER_PLUGIN_VERSION}-all.jar && \
 wget -q https://search.maven.org/remotecontent?filepath=com/expediagroup/apiary/apiary-metastore-metrics/${APIARY_METASTORE_METRICS_VERSION}/apiary-metastore-metrics-${APIARY_METASTORE_METRICS_VERSION}-all.jar -O apiary-metastore-metrics-${APIARY_METASTORE_METRICS_VERSION}-all.jar && \
 wget -q https://search.maven.org/remotecontent?filepath=com/expediagroup/apiary/apiary-metastore-auth/${APIARY_METASTORE_AUTH_VERSION}/apiary-metastore-auth-${APIARY_METASTORE_AUTH_VERSION}.jar -O apiary-metastore-auth-${APIARY_METASTORE_AUTH_VERSION}.jar && \
-wget -q https://search.maven.org/remotecontent?filepath=org/apache/atlas/atlas-notification/${ATLAS_VERSION}/atlas-notification-${ATLAS_VERSION}.jar -O atlas-notification-${ATLAS_VERSION}.jar && \
-wget -q https://search.maven.org/remotecontent?filepath=org/apache/atlas/atlas-intg/${ATLAS_VERSION}/atlas-intg-${ATLAS_VERSION}.jar -O atlas-intg-${ATLAS_VERSION}.jar && \
-wget -q https://search.maven.org/remotecontent?filepath=org/apache/atlas/atlas-common/${ATLAS_VERSION}/atlas-common-${ATLAS_VERSION}.jar -O atlas-common-${ATLAS_VERSION}.jar && \
 wget -q https://search.maven.org/remotecontent?filepath=org/apache/kafka/kafka-clients/${KAFKA_VERSION}/kafka-clients-${KAFKA_VERSION}.jar -O kafka-clients-${KAFKA_VERSION}.jar && \
 wget -q https://search.maven.org/remotecontent?filepath=commons-codec/commons-codec/${COMMONS_CODEC_VERSION}/commons-codec-${COMMONS_CODEC_VERSION}.jar -O commons-codec-${COMMONS_CODEC_VERSION}.jar && \
 wget -q https://search.maven.org/remotecontent?filepath=com/kstruct/gethostname4j/${GETHOSTNAME4J_VERSION}/gethostname4j-${GETHOSTNAME4J_VERSION}.jar -O gethostname4j-${GETHOSTNAME4J_VERSION}.jar && \
@@ -55,22 +51,11 @@ ENV MAVEN_VERSION 3.6.3
 RUN wget -q -O - http://www-us.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz|tar -C /opt -xzf - && \
     ln -sf /opt/apache-maven-${MAVEN_VERSION}/bin/mvn /bin/mvn
 
-COPY files/atlas-${ATLAS_VERSION}-hive-2.3.3.patch /tmp/atlas-${ATLAS_VERSION}-hive-2.3.3.patch
-RUN cd /tmp && \
-    wget -q https://www-us.apache.org/dist/atlas/${ATLAS_VERSION}/apache-atlas-${ATLAS_VERSION}-sources.tar.gz && \
-    tar xfz apache-atlas-${ATLAS_VERSION}-sources.tar.gz && \
-    cd apache-atlas-sources-${ATLAS_VERSION}/ && \
-    patch  -p1 < /tmp/atlas-${ATLAS_VERSION}-hive-2.3.3.patch && \
-    sed -s 's#http://repo1.maven.org#https://repo1.maven.org#' -i pom.xml && \
-    cd addons/hive-bridge && mvn package -Dhive.version=2.3.3 && cp -a target/hive-bridge-${ATLAS_VERSION}.jar /usr/lib/apiary/ && \
-    cd /tmp && rm -rf /root/.m2 && rm -rf /tmp/apache-atlas-sources-${ATLAS_VERSION}/ && rm -f /tmp/apache-atlas-${ATLAS_VERSION}-sources.tar.gz
-
 COPY files/core-site.xml /etc/hadoop/conf/core-site.xml
 COPY files/hive-site.xml /etc/hive/conf/hive-site.xml
 COPY files/hive-log4j2.properties /etc/hive/conf/hive-log4j2.properties
 COPY files/ranger-hive-security.xml /etc/hive/conf/ranger-hive-security.xml
 COPY files/ranger-hive-audit.xml /etc/hive/conf/ranger-hive-audit.xml
-COPY files/atlas-application.properties /etc/hive/conf/atlas-application.properties
 COPY files/jmx-exporter.yaml /etc/hive/conf/jmx-exporter.yaml
 
 

README.md

@@ -9,7 +9,6 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
 |APIARY_S3_INVENTORY_PREFIX|No (defaults to `EntireBucketDaily`)|Prefix used by S3 Inventory when creating data in the inventory bucket.|
 |APIARY_S3_INVENTORY_TABLE_FORMAT|No (defaults to `ORC`)|Format of S3 inventory data - `ORC`, `Parquet`, or `CSV`|
 |APIARY_SYSTEM_SCHEMA|No (defaults to `apiary_system`)|Name for internal system database.|
-|ATLAS_KAFKA_BOOTSTRAP_SERVERS|No|Atlas hive-bridge kafka bootstrap servers.|
 |AWS_REGION|Yes|AWS region to configure various AWS clients.|
 |ENABLE_GLUESYNC|No|Option to turn on GlueSync Hive Metastore listener.|
 |ENABLE_METRICS|No|Option to enable sending Hive Metastore metrics to CloudWatch.|