Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Entrance support ha and failover task #4282

Merged
merged 66 commits into from
Nov 19, 2023
Merged

[Feature] Entrance support ha and failover task #4282

merged 66 commits into from
Nov 19, 2023

Conversation

guoshupei
Copy link
Collaborator

What is the purpose of the change

Improve the stability of Entrance service

Related issues/PRs

Related issues: #4181

Brief change log

  • The task status and progress can be obtained using taskId;
  • failover task when entrance shutdown;
  • Dynamically adjust the runningjobs parameter.

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
  • If this is a code change: I have written unit tests to fully verify the new behavior.

guoshupei added 30 commits February 28, 2023 15:44
@guoshupei
ServiceInstance add registryTime
4217a3f
@guoshupei
gateway edit and entrance api edit
08fc461
@guoshupei
add method of queryFailoverJobs
17a5931
@guoshupei
entrance ha and failover
7706a60
@guoshupei
Interface development of status,log,progress,kill
00777c6
@guoshupei
failover and kill ec
8c5774c
@guoshupei
failover
497f8b6
@guoshupei
add log
da5b2b7
@guoshupei
entrance taskID
b624a04
@guoshupei
update status limit for failover
cbfbdff
@guoshupei
[bug-fix] failover logic
d7eb302
@guoshupei
add failover log
88017f3
@guoshupei
push log to entrance
e321189
@guoshupei
add entrance log
2dc0c72
@guoshupei
update failover scan interval
fe501c6
@guoshupei
[Bug-fix] gateway choose instance
170c0c9
@guoshupei
batch update instance
c91a6e8
@guoshupei
failover status changed from Running to Cancelled
8c55b18
@guoshupei
entrance refactor failoverExecute
7971904
@guoshupei
change updateOrderFlag
ae9a172
@guoshupei
edit failoverJobServer
a4a41d6
@guoshupei
1.修改获取failover实例列表，不能去除offline lable实例
f075537 
2.移除retry任务时，要更新数据库instance为空
@guoshupei
1.日志信息增加中文描述
db2f0ed
@guoshupei
failover时，任务主动刷日志hdfs
dfb531a
@guoshupei
迁移刷新maxAllowRunningJobs服务，放到consumeManager里，刷新consumer里的group,而不是刷新gr…
8ab841c 
…oupFactory里的
@guoshupei
update name
1cf6321
@guoshupei
add header
789404f
@guoshupei
bug - print failover ids size
2a39fa3
@guoshupei
update failover log
a33ad78
@guoshupei
update log
c172f32
@guoshupei
Update the maximum concurrency of orchestrator from 200 to 1000
2d4f784
@guoshupei
- moved JobInstance from linkis-common to linkis-protocol
8dafb2c 
- add isInitedStr,isRunningStr method and remove uncompleteStatusArray method in SchedulerEventState
@guoshupei
Add description
e624b37
@guoshupei
Merge branch 'dev-1.4.0' of https://github.com/apache/linkis into dev…
a23b73b 
…-1.3.3-feature-ha-4181

� Conflicts:
�	linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java
�	linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/server/DefaultEntranceServer.java
�	linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala
�	linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/JobHistoryHelper.scala
�	linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/dao/JobHistoryMapper.java
@guoshupei
replace constant
94f3ec1
@guoshupei
replace Option.apply to null
acc91db
@guoshupei
Copy link
Collaborator Author

LKIP-27 Linkis entrance 支持高可用和任务故障转移

  • 状态
    【进展情况记录】：
    提议时间:2022/12/18
    讨论时间:
    接受/拒绝时间:
    完成时间:
    【issues】：[Feature][Entrance] Supports HA and task failover #4020
    【email】：
    【发布】：
    【提议人】： 郭书培
  • 背景
    目前Entrance服务是单点的，并且entrance承载的是服务入口功能，如果出现问题，那么将导致整个服务对外不可用，所以需要将entrance做到高可用，并且对已提交任务实现一定程度的接管，尽可能保障任务正常执行，用户无感。
  • 实现的目标
    1.服务高可用
    2.任务故障转移
  • 实现大体思路
    具体参见文档
    Linkis Entrance 高可用实现梳理.pdf

@peacewong peacewong changed the base branch from dev-1.4.0 to master June 20, 2023 09:39
@guoshupei
sql optimize and bug fix
caeeab9
@guoshupei
Merge branch 'master' of https://github.com/apache/linkis into dev-1.…
4571c4c 
…3.3-feature-ha-4181

� Conflicts:
�	linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/service/TaskExecutionServiceImpl.scala
�	linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java
�	linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/server/DefaultEntranceServer.java
�	linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala
�	linkis-dist/package/conf/linkis-cg-entrance.properties
�	linkis-dist/package/conf/linkis-mg-gateway.properties
@guoshupei
merge master
3d681fb
@guoshupei
add comment
bdc3e0e
@guoshupei
add mybatis config
5fa73f2
peacewong
peacewong reviewed Oct 19, 2023
View reviewed changes
...on-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala Outdated Show resolved Hide resolved
...on-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala Outdated Show resolved Hide resolved
...on-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala Show resolved Hide resolved
...-entrance/src/main/scala/org/apache/linkis/entrance/scheduler/EntranceFIFOUserConsumer.scala Show resolved Hide resolved
...nance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/JobHistoryHelper.scala Show resolved Hide resolved
...lic-enhancements/linkis-jobhistory/src/main/resources/mapper/postgresql/JobHistoryMapper.xml Outdated Show resolved Hide resolved
...port/src/main/scala/org/apache/linkis/gateway/ujes/parser/EntranceRequestGatewayParser.scala Show resolved Hide resolved
peacewong
peacewong approved these changes Nov 19, 2023
View reviewed changes
Copy link
Contributor

@peacewong peacewong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@peacewong peacewong merged commit 0453b07 into apache:master Nov 19, 2023
12 checks passed
@guoshupei guoshupei deleted the dev-1.3.3-feature-ha-4181 branch July 11, 2024 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peacewong peacewong peacewong approved these changes

@jackxu2011 jackxu2011 Awaiting requested review from jackxu2011

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@guoshupei @peacewong @jackxu2011