Skip to content

Commit

Permalink
Add hbase engine docs, apache/linkis#4891
Browse files Browse the repository at this point in the history
  • Loading branch information
Longping Jie committed Oct 10, 2023
1 parent c5eb9a2 commit 200d76e
Show file tree
Hide file tree
Showing 2 changed files with 205 additions and 0 deletions.
104 changes: 104 additions & 0 deletions docs/engine-usage/hbase.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
title: HBase Engine
sidebar_position: 15
---

# HBase engine usage documentation

This article mainly introduces the installation, use and configuration of the `hbase` engine plugin in `Linkis`.

## 1. Preliminary work

### 1.1 Engine Verification

It is strongly recommended that you check these environment variables for the executing user before executing `hbase` tasks. The specific way is

```
sudo su - ${username}
echo ${JAVA_HOME}
```

## 2. Engine plugin installation

### 2.1 Engine plugin preparation [non-default engine](./overview.md)

Method 2: Compile the engine plug-in separately (requires a `maven` environment)

```
# compile
cd ${linkis_code_dir}/linkis-engineconn-plugins/hbase/
mvn clean install -DskipTests -Dhbase.profile=1.2|1.4|2.2|2.5
# hbase.profile 1.2|1.4|2.2|2.5, The corresponding HBase version is: 1.2.0, 1.4.3, 2.2.6, 2.5.3
# default hbase.profile=2.5, Compile plug-ins for different HBase versions on demand
# The compiled engine plug-in package is located in the following directory
${linkis_code_dir}/linkis-engineconn-plugins/hbase/target/out/
```

[EngineConnPlugin engine plugin installation](../deployment/install-engineconn.md)

### 2.2 Upload and load engine plugins

Upload the engine plug-in package in 2.1 to the engine directory of the server

```bash
${LINKIS_HOME}/lib/linkis-engineplugins
```
The directory structure after uploading is as follows
```
linkis-engineconn-plugins/
├── hbase
│   ├── dist
│   │   └── 2.5.3
│   │   ├── conf
│   │   └── lib
│   └── plugin
│   └── 2.5.3
```
### 2.3 Engine refresh

#### 2.3.1 Restart and refresh
Refresh the engine by restarting the `linkis-cg-linkismanager` service
```bash
cd ${LINKIS_HOME}/sbin
sh linkis-daemon.sh restart cg-linkismanager
```

### 2.3.2 Check if the engine is refreshed successfully
You can check whether the `last_update_time` of this table in the `linkis_engine_conn_plugin_bml_resources` in the database is the time when the refresh is triggered.

```sql
#Login to the linkis database
select * from linkis_cg_engine_conn_plugin_bml_resources;
```


## 3. Use of HBase engine

The `HBase` engine of `Linkis` runs a Jruby environment in the background to execute instructions in hbase-shell,
and is fully compatible with hbase-shell instructions.


### 3.1 Submit tasks through `Linkis-cli`

```shell
sh bin/linkis-cli -engineType hbase-2.5.3 -code "list" \
-codeType shell -runtimeMap linkis.hbase.zookeeper.quorum=zk1,zk2,zk3
```

More `Linkis-Cli` command parameter reference: [`Linkis-Cli` usage](../user-guide/linkiscli-manual.md)

HBase engine supports connection parameters:

| Configuration name | Remarks and default value | Is it necessary |
| --- | --- | --- |
| linkis.hbase.zookeeper.quorum | ZooKeeper address for connecting to HBase, e.g. zk1,zk2,zk3, default value: localhost | Required |
| linkis.hbase.zookeeper.property.clientPort | ZooKeeper port for connecting to HBase, default value: 2181 | Optional, usually the default value |
| linkis.zookeeper.znode.parent | ZooKeeper znode path for connecting to HBase, default value: /hbase | Optional, usually the default value |
| linkis.hbase.rootdir | HDFS root directory for the HBase cluster, default value: /hbase | Optional, usually the default value |
| linkis.hbase.security.authentication | Authentication method for connecting to the HBase cluster: simple or kerberos, default value: simple | Optional, depending on the cluster configuration |
| linkis.hbase.kerberos.principal | HBase Kerberos authentication principal, e.g. [email protected] | Required for Kerberos authentication |
| linkis.hbase.keytab.file | Path to the keytab file required for HBase Kerberos authentication, e.g. /tmp/hbase.keytab | Required for Kerberos authentication |
| linkis.hbase.kerberos.proxy.user | Kerberos proxy user, ensure that the Kerberos authenticated user has the privilege to proxy as a regular user | Optional, can be omitted |
| linkis.hbase.regionserver.kerberos.principal | HBase RegionServer's Kerberos authentication principal, e.g. hbase/_[email protected] (obtain from the cluster configuration file) | Required for Kerberos authentication |
| linkis.hbase.master.kerberos.principal | HBase Master's Kerberos authentication principal, e.g. hbase/_[email protected] (obtain from the cluster configuration file) | Required for Kerberos authentication |
101 changes: 101 additions & 0 deletions i18n/zh-CN/docusaurus-plugin-content-docs/current/engine-usage/hbase.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
title: HBase
sidebar_position: 15
---

# HBase 引擎使用文档

本文主要介绍在 `Linkis` 中，`HBase` 引擎插件的安装、使用和配置。

## 1. 前置工作

### 1.1 引擎验证

强烈建议您在执行 `HBase` 任务之前，检查下执行用户的这些环境变量。具体方式是
```
sudo su - ${username}
echo ${JAVA_HOME}
```

## 2. 引擎插件安装

### 2.1 引擎插件准备（二选一）[非默认引擎](./overview.md)

方式一：单独编译引擎插件（需要有 `maven` 环境）

```
# 编译
cd ${linkis_code_dir}/linkis-engineconn-plugins/hbase/
mvn clean install -DskipTests -Dhbase.profile=1.2|1.4|2.2|2.5
# hbase.profile 支持1.2|1.4|2.2|2.5，对应的HBase版本是1.2.0，1.4.3，2.2.6，2.5.3
# 默认hbase.profile=2.5，按需编译不同HBase版本的插件
# 编译出来的引擎插件包，位于如下目录中
${linkis_code_dir}/linkis-engineconn-plugins/hbase/target/out/
```

[EngineConnPlugin 引擎插件安装](../deployment/install-engineconn.md)

### 2.2 引擎插件的上传和加载

将 2.1 中的引擎插件包上传到服务器的引擎目录下
```bash
${LINKIS_HOME}/lib/linkis-engineplugins
```
上传后目录结构如下所示
```
linkis-engineconn-plugins/
├── hbase
│   ├── dist
│   │   └── 2.5.3
│   │   ├── conf
│   │   └── lib
│   └── plugin
│   └── 2.5.3
```
### 2.3 引擎刷新

#### 2.3.1 重启刷新
通过重启 `linkis-cg-linkismanager` 服务刷新引擎
```bash
cd ${LINKIS_HOME}/sbin
sh linkis-daemon.sh restart cg-linkismanager
```

### 2.3.2 检查引擎是否刷新成功
可以查看数据库中的 `linkis_engine_conn_plugin_bml_resources` 这张表的 `last_update_time` 是否为触发刷新的时间。

```sql
#登陆到linkis的数据库
select * from linkis_cg_engine_conn_plugin_bml_resources;
```


## 3. HBase引擎的使用

`Linkis` 的 `HBase` 引擎是通过后台运行一个Jruby环境来执行hbase-shell中的指令，完全兼容hbase-shell的指令。


### 3.1 通过 `Linkis-cli` 提交任务

```shell
sh bin/linkis-cli -engineType hbase-2.5.3 -code "list" \
-codeType shell -runtimeMap linkis.hbase.zookeeper.quorum=zk1,zk2,zk3
```

更多 `Linkis-Cli` 命令参数参考： [`Linkis-Cli` 使用](../user-guide/linkiscli-manual.md)

HBase引擎，支持的连接参数有：

| 配置名称 | 备注及默认值信息 | 是否必须 |
|-----------------|----------------|----------------------------------------|
| linkis.hbase.zookeeper.quorum | 连接hbase的zk地址，如：zk1,zk2,zk3， 默认值：localhost | 必须 |
| linkis.hbase.zookeeper.property.clientPort | 连接hbase的zk端口，默认值：2181 | 非必须，一般就是默认值 |
| linkis.zookeeper.znode.parent | 连接HBase的zk znode path，默认值：/hbase | 非必须，一般就是默认值 |
| linkis.hbase.rootdir | HBase集群的hdfs根目录，默认值：/hbase | 非必须，一般就是默认值 |
| linkis.hbase.security.authentication | HBase集群连接的认证方式：simple|kerberos，默认值：simple | 非必须，视自己集群情况定 |
| linkis.hbase.kerberos.principal | HBase kerberos认证principal，如：[email protected] | kerberos认证方式下必须 |
| linkis.hbase.keytab.file | HBase kerberos认证所需keytab文件地址，如：/tmp/hbase.keytab | kerberos认证方式下必须 |
| linkis.hbase.kerberos.proxy.user | kerberos代理用户，需保证kerberos认证用户有代理普通用户的权限 | 非必须，可以不用代理用户 |
| linkis.hbase.regionserver.kerberos.principal | 如：hbase/_[email protected]，自行从集群配置文件中获取 | kerberos认证方式下必须 |
| linkis.hbase.master.kerberos.principal | 如：hbase/_[email protected]，自行从集群配置文件中获取 | kerberos认证方式下必须 |

0 comments on commit 200d76e

Please sign in to comment.