-
Notifications
You must be signed in to change notification settings - Fork 112
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
- Loading branch information
1 parent
41f10cf
commit 613b897
Showing
6 changed files
with
131 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
--- | ||
title: ECM takes over EC | ||
sidebar_position: 0.2 | ||
--- | ||
|
||
## 1. Requirement Background | ||
When the ECM restarts, can choose not to kill the engine, and it can take over the existing surviving engine. | ||
Make engine conn manager(ECM) service stateless . | ||
|
||
|
||
## 2. Instructions for use | ||
this feature is enabled by default |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
--- | ||
title: hive engine support concurrent | ||
sidebar_position: 0.2 | ||
--- | ||
|
||
## 1. Requirement Background | ||
hiveEngineConn supports concurrency, reducing the resource consumption of starting hive engine. | ||
|
||
## 2. Instructions for use | ||
First, modify linkis-engineconn.properties file in linkis-engineconn-plugins/hive/src/main/resources directory, | ||
and set linkis.hive.engineconn.concurrent.support to true. | ||
``` | ||
# support parallelism execution | ||
linkis.hive.engineconn.concurrent.support=true | ||
``` | ||
|
||
Second, submit a hive task ,when first task finished ,submit another task. Your could see hive engine has been reused. | ||
|
||
|
||
## 3. Precautions | ||
1, Submit second hive task when first task has been finished. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
--- | ||
title: Extend linkis-storage add support OSS filesystem | ||
sidebar_position: 0.2 | ||
--- | ||
|
||
## 1. Requirement Background | ||
Extend linkis-storage add support OSS filesystem | ||
|
||
## 2. Instructions for use | ||
To store log and resultSet in OSS, add the following configs in conf/linkis-cg-entrance.properties. | ||
``` | ||
#eg: | ||
wds.linkis.entrance.config.log.path=oss://linkis/tmp/ | ||
wds.linkis.resultSet.store.path=oss://linkis/tmp/ | ||
wds.linkis.filesystem.hdfs.root.path=oss://taihao-linkis/tmp/ | ||
wds.linkis.fs.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com | ||
wds.linkis.fs.oss.bucket.name=linkis | ||
wds.linkis.fs.oss.accessKeyId=your accessKeyId | ||
wds.linkis.fs.oss.accessKeySecret=your accessKeySecret | ||
``` | ||
|
||
Add the following configs in engine engineconn plugins conf. Let me use hive conf for example: | ||
modify linkis-engineconn-plugins/hive/src/main/resources/linkis-engineconn.properties and | ||
add the following configs. | ||
``` | ||
#eg: | ||
wds.linkis.fs.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com | ||
wds.linkis.fs.oss.bucket.name=linkis | ||
wds.linkis.fs.oss.accessKeyId=your accessKeyId | ||
wds.linkis.fs.oss.accessKeySecret=your accessKeySecret | ||
``` | ||
|
||
|
||
## 3. Precautions | ||
1, you have an OSS bucket. | ||
2, you have accessKeyId, accessKeySecret to access the above OSS bucket. |
10 changes: 10 additions & 0 deletions
10
i18n/zh-CN/docusaurus-plugin-content-docs/current/feature/ecm-takes-over-ec.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
--- | ||
title: 当ECM重新启动时,可以选择不杀死引擎,而是可以接管现有的存活引擎 | ||
sidebar_position: 0.2 | ||
--- | ||
|
||
## 需求背景 | ||
当ECM重新启动时,可以选择不杀死引擎,而是可以接管现有的存活引擎。使引擎连接管理器(ECM)服务无状态。 | ||
|
||
## 使用说明 | ||
此功能默认已启用。 |
20 changes: 20 additions & 0 deletions
20
...ocusaurus-plugin-content-docs/current/feature/hive-engine-support-concurrent.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
--- | ||
title: hive engine支持并发,支持复用 | ||
sidebar_position: 0.2 | ||
--- | ||
|
||
## 1. 需求背景 | ||
hiveEngineConn支持并发,减少启动hive引擎的资源消耗。 | ||
|
||
## 2. 使用说明 | ||
首先,在linkis-engineconn-plugins/hive/src/main/resources目录下修改linkis-engineconn.properties文件, | ||
并将linkis.hive.engineconn.concurrent.support设置为true。 | ||
``` | ||
# 支持并行执行 | ||
linkis.hive.engineconn.concurrent.support=true | ||
``` | ||
|
||
第二,提交一个hive任务,当第一个任务完成后,再提交另一个任务。您可以看到hive引擎已被重用。 | ||
|
||
## 3. 注意事项 | ||
1、等第一个hive任务执行成功后,再提交第二个hive任务。 |
32 changes: 32 additions & 0 deletions
32
...zh-CN/docusaurus-plugin-content-docs/current/feature/storage-add-support-oss.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
--- | ||
title: 扩展linkis-storage以支持OSS文件系统 | ||
sidebar_position: 0.2 | ||
--- | ||
|
||
## 1. 需求背景 | ||
扩展linkis-storage以支持OSS文件系统。 | ||
|
||
## 2. 使用说明 | ||
为了在OSS中存储日志和resultSet,请在conf/linkis-cg-entrance.properties中添加以下配置。示例: | ||
``` | ||
wds.linkis.entrance.config.log.path=oss://linkis/tmp/ | ||
wds.linkis.resultSet.store.path=oss://linkis/tmp/ | ||
wds.linkis.filesystem.hdfs.root.path=oss://taihao-linkis/tmp/ | ||
wds.linkis.fs.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com | ||
wds.linkis.fs.oss.bucket.name=linkis | ||
wds.linkis.fs.oss.accessKeyId=your accessKeyId | ||
wds.linkis.fs.oss.accessKeySecret=your accessKeySecret | ||
``` | ||
|
||
在engine engineconn插件conf中添加以下配置。以hive conf为例:修改linkis-engineconn-plugins/hive/src/main/resources/linkis-engineconn.properties, | ||
并添加以下配置。示例: | ||
``` | ||
wds.linkis.fs.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com | ||
wds.linkis.fs.oss.bucket.name=linkis | ||
wds.linkis.fs.oss.accessKeyId=your accessKeyId | ||
wds.linkis.fs.oss.accessKeySecret=your accessKeySecret | ||
``` | ||
|
||
## 3. 注意事项 | ||
1、您需要拥有一个OSS存储桶。 | ||
2、您需要accessKeyId和accessKeySecret以访问上述OSS存储桶。 |