[Feature] Add hbase engine to linkis #4891

CCweixiao · 2023-08-31T01:36:20Z

What is the purpose of the change

Add hbase engine to linkis, The ruby command of hbase-shell can be executed through linkis.

Related issues

Related issues: #2533

Brief change log

Add hbase engine to linkis

Checklist

I have read the Contributing Guidelines on pull requests.
I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
If this is a code change: I have written unit tests to fully verify the new behavior.

peacewong · 2023-09-06T02:28:39Z

ping~ @jackxu2011 @jacktao007

peacewong · 2023-09-07T13:02:58Z

...hbase/src/main/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManager.java

+    String clusterConnUniqueKey = generateUniqueConnectionKey(configuration);
+    LOG.info("Start to get connection for cluster {}.", clusterConnUniqueKey);
+    try {
+      lock.lock();


Is it better to first determine whether it contains the corresponding key and then lock it?

peacewong · 2023-09-07T13:07:52Z

...hbase/src/main/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManager.java

+
+  private void doKerberosLogin(Configuration configuration) {
+    String principal = configuration.get(KERBEROS_PRINCIPAL);
+    String keytab = configuration.get(KERBEROS_KEYTAB_FILE);


Need to adapt Linkis's existing kerberos tools?

I feel it is necessary to optimize it later.

peacewong · 2023-09-07T13:09:31Z

...main/java/org/apache/linkis/manager/engineplugin/hbase/constant/HBaseEngineConnConstant.java

+public class HBaseEngineConnConstant {
+  private HBaseEngineConnConstant() {}
+
+  public static final String LINKIS_PREFIX = "wds.linkis.";


need to remove wds prefix

peacewong · 2023-09-07T13:11:48Z

...c/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellSessionManager.java

+    String sessionId = HBaseConnectionManager.getInstance().generateUniqueConnectionKey(prop);
+    LOG.info("Start to create session {} for cluster.", sessionId);
+    try {
+      lock.lock();


Same as connection and can be optimized

peacewong · 2023-09-07T13:29:23Z

...e/src/test/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManagerTest.java

+    properties.put(HBaseEngineConnConstant.KERBEROS_PRINCIPAL, "[email protected]");
+    properties.put(HBaseEngineConnConstant.KERBEROS_KEYTAB_FILE, "/etc/hbase/conf/hdfs.keytab");
+    // 设置kerberos代理用户
+    properties.put(HBaseEngineConnConstant.KERBEROS_PROXY_USER, "leojie");


leojie should rename to hadoop

casionone · 2023-09-20T14:12:37Z

...-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManager.java

+
+                  while (times < KERBEROS_RE_LOGIN_MAX_RETRY) {
+                    if (runKerberosLogin()) {
+                      LOG.info("Ran kerberos re login command successfully.");


peacewong · 2023-09-21T13:15:22Z

linkis-dist/release-docs/licenses/LICENSE-jaxb-impl.txt

@@ -0,0 +1,759 @@
+COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.1


Is there a GPL LICENSE for this?

peacewong · 2023-09-21T13:16:51Z

linkis-dist/release-docs/licenses/LICENSE-jersey-guice.txt

@@ -0,0 +1,759 @@
+COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.1
+


Is there a GPL LICENSE for this?

peacewong · 2023-09-21T13:30:12Z

...re/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/conf/HBaseConfiguration.scala

+import org.apache.linkis.common.conf.CommonVars
+
+object HBaseConfiguration {
+  val HBASE_CONCURRENT_LIMIT = CommonVars[Int]("wds.linkis.engineconn.hbase.concurrent.limit", 100)


remove wds prefix

peacewong · 2023-09-21T13:33:07Z

...in/scala/org/apache/linkis/manager/engineplugin/hbase/executor/HBaseEngineConnExecutor.scala

+    if (!result.isSuccess) {
+      return ErrorExecuteResponse(result.getResult, result.getThrowable)
+    }
+    engineExecutorContext.appendStdout(result.getResult)


Is it necessary to add a result set？

The execution result of HBase Shell is a bunch of strings. Does it need to be packaged into a result set?

If not table we need to write text result set

ResultSetFactory.TEXT_TYPE?

peacewong · 2023-09-21T13:33:44Z

...in/scala/org/apache/linkis/manager/engineplugin/hbase/executor/HBaseEngineConnExecutor.scala

+
+  override def killTask(taskId: String): Unit = {
+    logger.info(s"Killing hbase query task $taskId")
+    super.killTask(taskId)


need to support kill task

Supports kill task, which will destroy the hbase-shell session. Every time you kill the task and rerun the task, the session will be restarted.

Supports kill task, which will destroy the hbase-shell session. Every time you kill the task and rerun the task, the session will be restarted.

If the task can only be killed by ending the session, I think it is also necessary.

Okay, I'll add this function.

peacewong

LGTM.

Co-authored-by: Longping Jie <[email protected]>

CCweixiao force-pushed the dev-hbase branch 4 times, most recently from f16bc8c to 84b3732 Compare September 5, 2023 15:15

peacewong reviewed Sep 7, 2023

View reviewed changes

CCweixiao force-pushed the dev-hbase branch from 84b3732 to 7e0c253 Compare September 13, 2023 02:10

CCweixiao force-pushed the dev-hbase branch 3 times, most recently from 19895af to 8ea54a4 Compare September 20, 2023 12:53

[Feature] Add hbase engine to linkis

bb4672e

CCweixiao force-pushed the dev-hbase branch from 8ea54a4 to bb4672e Compare September 20, 2023 12:59

casionone reviewed Sep 20, 2023

View reviewed changes

peacewong reviewed Sep 21, 2023

View reviewed changes

Add the function of kill task and add TEXT ResultSet

745699c

CCweixiao force-pushed the dev-hbase branch from 751ef51 to 745699c Compare September 26, 2023 09:45

Merge branch 'master' into dev-hbase

e406f3a

peacewong approved these changes Sep 26, 2023

View reviewed changes

peacewong changed the title ~~[WIP][Feature] Add hbase engine to linkis~~ [Feature] Add hbase engine to linkis Sep 26, 2023

peacewong merged commit 2efef01 into apache:master Sep 26, 2023
12 checks passed

CCweixiao pushed a commit to CCweixiao/linkis-website that referenced this pull request Oct 10, 2023

Add hbase engine docs, apache/linkis#4891

200d76e

peacewong pushed a commit to apache/linkis-website that referenced this pull request Oct 27, 2023

Add hbase engine docs, apache/linkis#4891 (#758)

0d05aa6

Co-authored-by: Longping Jie <[email protected]>

CCweixiao mentioned this pull request Nov 22, 2023

Optimize HBase engine dependencies #4976

Merged

7 tasks

CCweixiao pushed a commit to CCweixiao/linkis-website that referenced this pull request Nov 28, 2023

Fix hbase engine docs, apache/linkis#4891

d13993d

CCweixiao pushed a commit to CCweixiao/linkis-website that referenced this pull request Dec 1, 2023

Fix hbase engine docs, apache/linkis#4891

e02a42b

CCweixiao added a commit to apache/linkis-website that referenced this pull request Dec 1, 2023

Fix hbase engine docs, apache/linkis#4891 (#772)

f79357b

Co-authored-by: Longping Jie <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add hbase engine to linkis #4891

[Feature] Add hbase engine to linkis #4891

CCweixiao commented Aug 31, 2023

peacewong commented Sep 6, 2023

peacewong Sep 7, 2023

peacewong Sep 7, 2023

CCweixiao Sep 20, 2023

peacewong Sep 7, 2023

peacewong Sep 7, 2023

peacewong Sep 7, 2023

casionone Sep 20, 2023

peacewong Sep 21, 2023

peacewong Sep 21, 2023

peacewong Sep 21, 2023

peacewong Sep 21, 2023

CCweixiao Sep 26, 2023

peacewong Sep 26, 2023

CCweixiao Sep 26, 2023

peacewong Sep 21, 2023

CCweixiao Sep 26, 2023

peacewong Sep 26, 2023

CCweixiao Sep 26, 2023

peacewong left a comment

		@@ -0,0 +1,759 @@
		COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.1

[Feature] Add hbase engine to linkis #4891

[Feature] Add hbase engine to linkis #4891

Conversation

CCweixiao commented Aug 31, 2023

What is the purpose of the change

Related issues

Brief change log

Checklist

peacewong commented Sep 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peacewong left a comment

Choose a reason for hiding this comment