Merge pull request #117 from ExpediaGroup/feature/enable_lock_hp

enable hive lock house keeper

CHANGELOG.md

@@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). 
 
+## [3.0.13] - 2023-08-09
+### Added
+- [issue-118](https://github.com/ExpediaGroup/apiary-metastore-docker/issues/118) Added variable `ENABLE_HIVE_LOCK_HOUSE_KEEPER` to support hive lock house keeper. See more details here: apache/iceberg#2301
+
 ## [3.0.12] - 2023-08-02
 ### Added
 - Added variable `MAX_REQUEST_SIZE` to optionally increase the request size when sending records to Kafka.

README.md

@@ -13,6 +13,7 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
 |AWS_WEB_IDENTITY_TOKEN_FILE|No|Path of the AWS Web Identity Token File for IRSA/OIDC AWS authentication.|
 |DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES|No|`true`/`false` value for hive.metastore.disallow.incompatible.col.type.changes, default `true`.|
 |ENABLE_GLUESYNC|No|Option to turn on GlueSync Hive Metastore listener.|
+|ENABLE_HIVE_LOCK_HOUSE_KEEPER|No|Option to turn on Hive Metastore Hive Lock House Keeper.|
 |ENABLE_METRICS|No|Option to enable sending Hive Metastore metrics to CloudWatch.|
 |ENABLE_S3_INVENTORY|No|Option to create Hive tables on top of S3 inventory data if enabled in `apiary-data-lake`. Enabled if value is not null/empty.|
 |ENABLE_S3_LOGS|No|Option to create Hive tables on top of S3 access logs data if enabled in `apiary-data-lake`. Enabled if value is not null/empty.|

files/startup.sh

@@ -189,6 +189,14 @@ if [ ! -z ${AWS_WEB_IDENTITY_TOKEN_FILE} ]; then
     update_property.py fs.s3a.aws.credentials.provider com.amazonaws.auth.WebIdentityTokenCredentialsProvider /etc/hadoop/conf/core-site.xml
 fi
 
+# Enable Hive lock housekeeper
+if [ ! -z ${ENABLE_HIVE_LOCK_HOUSE_KEEPER} ]; then
+    update_property.py hive.txn.manager org.apache.hadoop.hive.ql.lockmgr.DbTxnManager /etc/hive/conf/hive-site.xml
+    update_property.py hive.support.concurrency true /etc/hive/conf/hive-site.xml
+    update_property.py hive.compactor.initiator.on true /etc/hive/conf/hive-site.xml
+    update_property.py hive.compactor.worker.threads 1 /etc/hive/conf/hive-site.xml
+    update_property.py hive.txn.strict.locking.mode false /etc/hive/conf/hive-site.xml 
+fi
 #auto configure heapsize
 if [ ! -z ${ECS_CONTAINER_METADATA_URI} ]; then
     export MEM_LIMIT=$(wget -q -O - ${ECS_CONTAINER_METADATA_URI}/task|jq -r .Limits.Memory)

files/update_property.py

@@ -27,17 +27,24 @@
 
 def write_properties_to_xml(xml_path, property_name='', property_value=''):
 	if(os.path.isfile(xml_path)):
-		try:
+		try:			
 			xml = ET.parse(xml_path)
 		except ExpatError:
-			print "Error while parsing file:"+xml_path
+			print "Error while parsing file:" + xml_path
 			return -1
+		property_exists = False
 		root = xml.getroot()
 		for child in root.findall('property'):
 			name = child.find("name").text.strip()
 			if name == property_name:
+				property_exists = True
 				child.find("value").text = property_value
 				break
+		if not property_exists:
+			new_property = ET.SubElement(root, 'property')
+			ET.SubElement(new_property,'name').text = property_name 	
+			ET.SubElement(new_property,'value').text = property_value 
+
 		xml.write(xml_path)
 		return 0
 	else: