Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added support to use mysql jdbc driver #126

Merged
merged 2 commits into from
Jun 19, 2024
Merged

feat: added support to use mysql jdbc driver #126

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7e0ff73
feat: added support to use mysql jdbc driver
JianLi-Expedia Jun 18, 2024
127fd74
Update CHANGELOG.md
patduin Jun 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [5.0.0] - 2024-06-19
### Changed
- Switch from mariadb driver to default mysql driver. (Override settings to keep using mariadb driver).
### Added
- Added `MYSQL_CONNECTION_DRIVER_NAME` to support use different connection driver, defaults: `com.mysql.jdbc.Driver`.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a major version release? Changing the driver from Maria to MySQL could be a breaking change.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are actually same, but we just trying to make it make better sense, so mariadb is using mariabdb connector jar, mysql is using mysql jar, so I think this is a minor change.

Copy link
Contributor

@patduin patduin Jun 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't someone using MariaDB RDS moving to this version now suddenly using the mysql driver? Maybe a major version makes sense here.

- Added `MYSQL_TYPE` to support use different type of MySQL, defaults: `mysql`.
- Added `mysql-connector-java` to support to use driver `com.mysql.jdbc.Driver`.

## [4.0.1] - 2024-06-03
### Changed
- Upgraded `APIARY_EXTENSIONS_VERSION` to `7.3.9` (was `7.3.8`).
Expand Down
1 change: 1 addition & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ RUN yum -y install java-1.8.0-openjdk \
java-1.8.0-openjdk-devel.x86_64 \
hive-metastore \
mariadb-connector-java \
mysql-connector-java \
mysql \
wget \
zip \
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,13 @@ For more information please refer to the main [Apiary](https://github.com/Expedi
|LDAP_CA_CERT|Base64 encoded Certificate Authority Bundle to validate LDAP SSL connection.|
|LDAP_SECRET_ARN|No| LDAP bind DN SecretsManager secret ARN. |
|LDAP_URL|No| Active Directory URL to enable group mapping in metastore. |
|MYSQL_CONNECTION_DRIVER_NAME|No (defaults to `com.mysql.jdbc.Driver`)| Hive Metastore MySQL database JDBC connection Driver Name. |
|MYSQL_CONNECTION_POOL_SIZE|No (defaults to `10`)| MySQL Connection pool size for Hive Metastore. See [here](https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1181) for more info. |
|MYSQL_DB_HOST|Yes| Hive Metastore MySQL database hostname. |
|MYSQL_DB_NAME|Yes| Hive Metastore MySQL database name. |
|MYSQL_SECRET_ARN|Yes| Hive Metastore MySQL SecretsManager secret ARN. |
|MYSQL_SECRET_USERNAME_KEY|No (defaults to `username`)| Hive Metastore MySQL SecretsManager secret username key. |
|MYSQL_TYPE|No (defaults to `mysql`)| Hive Metastore MySQL database Type (mariadb, mysql). |
|RANGER_AUDIT_DB_URL|No| Ranger audit database JDBC URL. |
|RANGER_AUDIT_SECRET_ARN|No| Ranger audit database secret ARN. |
|RANGER_AUDIT_SOLR_URL|No| Ranger Solr audit URL. |
Expand Down
2 changes: 1 addition & 1 deletion files/hive-site.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.mariadb.jdbc.Driver</value>
<value>com.mysql.jdbc.Driver</value>
</property>

<property>
Expand Down
6 changes: 5 additions & 1 deletion files/startup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ if [[ -n ${HMS_MAX_THREADS} ]]; then
update_property.py hive.metastore.server.max.threads "${HMS_MAX_THREADS}" /etc/hive/conf/hive-site.xml
fi

# configure ConnectionDriverName
if [[ -n ${MYSQL_CONNECTION_DRIVER_NAME} ]]; then
update_property.py javax.jdo.option.ConnectionDriverName "${MYSQL_CONNECTION_DRIVER_NAME}" /etc/hive/conf/hive-site.xml
fi
# config size of MySQL connection pool.
# See https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L1181
# and also make sure 2 * MYSQL_CONNECTION_POOL_SIZE * numHmsContainers is less than max connections for your MySQL instance.
Expand Down Expand Up @@ -212,4 +216,4 @@ fi
[[ -z $HADOOP_HEAPSIZE ]] && export HADOOP_HEAPSIZE=1024

export HADOOP_OPTS="-XshowSettings:vm -Xms${HADOOP_HEAPSIZE}m $EXPORTER_OPTS"
su hive -s/bin/bash -c "/usr/lib/hive/bin/hive --service metastore --hiveconf javax.jdo.option.ConnectionURL=jdbc:mysql://${MYSQL_DB_HOST}:3306/${MYSQL_DB_NAME} --hiveconf javax.jdo.option.ConnectionUserName='${MYSQL_DB_USERNAME}' --hiveconf javax.jdo.option.ConnectionPassword='${MYSQL_DB_PASSWORD}'"
su hive -s/bin/bash -c "/usr/lib/hive/bin/hive --service metastore --hiveconf javax.jdo.option.ConnectionURL=jdbc:${MYSQL_TYPE:-mysql}://${MYSQL_DB_HOST}:3306/${MYSQL_DB_NAME} --hiveconf javax.jdo.option.ConnectionUserName='${MYSQL_DB_USERNAME}' --hiveconf javax.jdo.option.ConnectionPassword='${MYSQL_DB_PASSWORD}'"
Loading