Skip to content

Commit

Permalink
Enable apiary metastore metrics (#66)
Browse files Browse the repository at this point in the history
* Fix for ECS_TASK_ID

* Use Maven version 3.6.1

* Latest mvn version

*  Fix maven central 501 HTTPS Required error https://support.sonatype.com/hc/en-us/articles/360041287334-Central-501-HTTPS-Required

* Adding changelog

* Updating changelog

* PR Comments

* fix ECS_TASK_ID on k8s

* disable ECS_TASK_ID lookup when running on k8s

* update changelog
  • Loading branch information
spuranda123 authored Feb 5, 2020
1 parent 43e7a58 commit 69fbca5
Show file tree
Hide file tree
Showing 3 changed files with 18 additions and 4 deletions.
12 changes: 10 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [1.6.0] - 2020-02-04

### Added
- Set EKS hostname to ECS_TASK_ID required for enabling metastore metrics.

### Changed
- Update using https for maven central repository as it no longer supports insecure communication over plain HTTP.

## [1.5.1] - 2020-01-10

### Changed
Expand All @@ -12,12 +20,12 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a

### Added
- Atlas cluster name is set to Apiary `ATLAS_CLUSTER_NAME` env variable when using Atlas plugin. If not set, will default to `INSTANCE_NAME` var.

### Changed
- Update Ranger version from to `2.0.0` (was `1.1.0`).
- Update Ranger metastore plugin to `5.0.0` (was `4.2.0`).
- Support Ranger audit-only mode for read-only HMS endpoint when audit destination is SOLR.

## [1.4.0] - 2019-11-18

### Added
Expand Down
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ wget -q https://search.maven.org/remotecontent?filepath=commons-codec/commons-co
wget -q https://search.maven.org/remotecontent?filepath=com/kstruct/gethostname4j/${GETHOSTNAME4J_VERSION}/gethostname4j-${GETHOSTNAME4J_VERSION}.jar -O gethostname4j-${GETHOSTNAME4J_VERSION}.jar && \
wget -q https://search.maven.org/remotecontent?filepath=com/sun/jna/jna/${JNA_VERSION}/jna-${JNA_VERSION}.jar -O jna-${JNA_VERSION}.jar

ENV MAVEN_VERSION 3.5.4
ENV MAVEN_VERSION 3.6.3

RUN wget -q -O - http://www-us.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz|tar -C /opt -xzf - && \
ln -sf /opt/apache-maven-${MAVEN_VERSION}/bin/mvn /bin/mvn
Expand All @@ -57,6 +57,7 @@ RUN cd /tmp && \
tar xfz apache-atlas-${ATLAS_VERSION}-sources.tar.gz && \
cd apache-atlas-sources-${ATLAS_VERSION}/ && \
patch -p1 < /tmp/atlas-${ATLAS_VERSION}-hive-2.3.3.patch && \
sed -s 's#http://repo1.maven.org#https://repo1.maven.org#' -i pom.xml && \
cd addons/hive-bridge && mvn package -Dhive.version=2.3.3 && cp -a target/hive-bridge-${ATLAS_VERSION}.jar /usr/lib/apiary/ && \
cd /tmp && rm -rf /root/.m2 && rm -rf /tmp/apache-atlas-sources-${ATLAS_VERSION}/ && rm -f /tmp/apache-atlas-${ATLAS_VERSION}-sources.tar.gz

Expand Down
7 changes: 6 additions & 1 deletion files/startup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,12 @@ if [[ -n $RANGER_AUDIT_DB_URL ]]; then
fi

if [ ! -z $ENABLE_METRICS ]; then
export ECS_TASK_ID=$(wget -q -O - http://169.254.170.2/v2/metadata|jq -r .TaskARN|awk -F/ '{ print $NF }')
if [ -z $KUBERNETES_SERVICE_HOST ]; then
export ECS_TASK_ID=$(wget -q -O - http://169.254.170.2/v2/metadata|jq -r .TaskARN|awk -F/ '{ print $NF }')
else
# Set ECS_TASK_ID to K8 Hostname for EKS
export ECS_TASK_ID="$HOSTNAME"
fi
export CLOUDWATCH_NAMESPACE="${INSTANCE_NAME}-metastore"
update_property.py hive.metastore.metrics.enabled true /etc/hive/conf/hive-site.xml
fi
Expand Down

0 comments on commit 69fbca5

Please sign in to comment.