[BugFix] Fix tablet scheduler exception

[sevev]

Why I'm doing:

If TabletScheduler encountered an unhandled exception when scheduling pending tablets, it only removed the task but did not remove the task_ctx. Consequently, subsequent scheduling will ignore these tablets, and clone and other tasks will no longer be triggered, ultimately causing the alter task to stall.

e.g

[OlapTable.renamePartition():1348] rename partition duplicate_complex_table_200_abnormal_base_async_hash_mv in table duplicate_complex_table_200_abnormal_base_async_hash_mv

2024-11-18 04:26:59.256+08:00 ERROR (tablet scheduler|35) [Daemon.run():109] daemon thread got exception. name: tablet scheduler
java.util.ConcurrentModificationException: null
        at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1698) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at com.starrocks.catalog.CatalogRecycleBin.getPhysicalPartition(CatalogRecycleBin.java:249) ~[starrocks-fe.jar:?]
        at com.starrocks.clone.TabletScheduler.checkIfTabletExpired(TabletScheduler.java:558) ~[starrocks-fe.jar:?]
        at com.starrocks.clone.TabletScheduler.checkIfTabletExpired(TabletScheduler.java:539) ~[starrocks-fe.jar:?]
        at com.starrocks.clone.TabletScheduler.getNextTabletCtxBatch(TabletScheduler.java:1799) ~[starrocks-fe.jar:?]
        at com.starrocks.clone.TabletScheduler.schedulePendingTablets(TabletScheduler.java:578) ~[starrocks-fe.jar:?]
        at com.starrocks.clone.TabletScheduler.runAfterCatalogReady(TabletScheduler.java:452) ~[starrocks-fe.jar:?]
        at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72) ~[starrocks-fe.jar:?]
        at com.starrocks.common.util.Daemon.run(Daemon.java:107) ~[starrocks-fe.jar:?]

What I'm doing:

Catch the exception and remove the tablet_ctx.

Fixes https://github.com/StarRocks/StarRocksTest/issues/8829

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.3
  • 3.2
  • 3.1
  • 3.0
  • 2.5

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[FE Incremental Coverage Report]

❌ fail : 3 / 7 (42.86%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/clone/TabletScheduler.java	3	7	42.86%	[1804, 1805, 1806, 1807]

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

@Mergifyio backport branch-3.3

[github-actions]

@Mergifyio backport branch-3.2

[mergify]

backport branch-3.3

✅ Backports have been created

• #53166 [BugFix] Fix tablet scheduler exception (backport #52925) has been created for branch branch-3.3

[mergify]

backport branch-3.2

✅ Backports have been created

• #53167 [BugFix] Fix tablet scheduler exception (backport #52925) has been created for branch branch-3.2

[sevev]

@mergify backport branch-3.4

[mergify]

backport branch-3.4

✅ Backports have been created

• #53169 [BugFix] Fix tablet scheduler exception (backport #52925) has been created for branch branch-3.4