Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support cooldown partition to external table #45536

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

hoffermei
Copy link
Contributor

@hoffermei hoffermei commented May 13, 2024

Why I'm doing:

What I'm doing:

Fixes #51649

support auto cooldown olap table partition to iceberg table, when olap table has been configured with external cooldown attribute

/* set cooldown related attributes */
alter table olap_tbl set('
    'external_cooldown_target'='iceberg.hoffertest.iceberg_external_tbl'
    'external_cooldown_schedule'='START 01:00 END 07:59 EVERY INTERVAL 5 MINUTE',
    'external_cooldown_wait_second'='3600'
);

/* cooldown table */
cooldown table olap_tbl partition;

/* cooldown table force */
cooldown table olap_tbl partition force;

/* cooldown partitions between 2022-04-18 and 2022-04-20 */
cooldown table olap_tbl partition start ('2022-04-18 00:00:00') end ('2022-04-20 00:00:00');

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 3.3
    • 3.2
    • 3.1
    • 3.0
    • 2.5

@hoffermei hoffermei requested review from a team as code owners May 13, 2024 08:46
@wanpengfei-git wanpengfei-git requested a review from a team May 13, 2024 08:47
@wanpengfei-git wanpengfei-git requested a review from a team May 13, 2024 08:47
@hoffermei hoffermei changed the title [WIP] Partition cold down [Feature] Support cooldown partition to external table Oct 8, 2024
@hoffermei hoffermei force-pushed the partition_cold_down branch 4 times, most recently from 81f915e to e56f8e0 Compare October 9, 2024 02:13
@hoffermei hoffermei force-pushed the partition_cold_down branch 2 times, most recently from 5f45f2a to 19a1031 Compare October 9, 2024 03:08
@hoffermei hoffermei force-pushed the partition_cold_down branch 4 times, most recently from d251977 to ddd09de Compare October 9, 2024 06:54
@hoffermei hoffermei closed this Oct 9, 2024
@hoffermei hoffermei reopened this Oct 9, 2024
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Oct 10, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 12, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 12, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 15, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 15, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 17, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 17, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 17, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 17, 2024
hoffermei added a commit to hoffermei/starrocks that referenced this pull request Oct 18, 2024
Copy link

sonarcloud bot commented Oct 18, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
D Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

Copy link

[Java-Extensions Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

[BE Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

[FE Incremental Coverage Report]

pass : 834 / 1039 (80.27%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/sql/ast/AstVisitor.java 0 2 00.00% [247, 251]
🔵 com/starrocks/sql/ast/AstTraverser.java 0 2 00.00% [73, 78]
🔵 com/starrocks/sql/analyzer/AuthorizerStmtVisitor.java 0 2 00.00% [1732, 1737]
🔵 com/starrocks/alter/AlterJobMgr.java 2 6 33.33% [567, 568, 571, 572]
🔵 com/starrocks/scheduler/externalcooldown/ExternalCooldownMgr.java 41 85 48.24% [57, 60, 61, 62, 63, 64, 65, 66, 67, 68, 70, 71, 72, 73, 74, 82, 83, 84, 85, 86, 87, 88, 89, 95, 96, 97, 98, 99, 100, 116, 117, 118, 148, 158, 159, 165, 166, 167, 168, 169, 171, 184, 185, 186]
🔵 com/starrocks/persist/EditLog.java 4 7 57.14% [1093, 1094, 1095]
🔵 com/starrocks/server/LocalMetastore.java 36 56 64.29% [3653, 3654, 3655, 3656, 3657, 3658, 3659, 3662, 3664, 3721, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3882, 3883, 4195]
🔵 com/starrocks/catalog/OlapTable.java 24 34 70.59% [494, 499, 506, 507, 508, 511, 512, 3433, 3458, 3465]
🔵 com/starrocks/catalog/PartitionInfo.java 28 40 70.00% [268, 269, 274, 275, 278, 279, 351, 352, 355, 356, 359, 360]
🔵 com/starrocks/common/util/PropertyAnalyzer.java 38 53 71.70% [1718, 1724, 1728, 1738, 1748, 1749, 1778, 1779, 1780, 1782, 1783, 1784, 1786, 1798, 1815]
🔵 com/starrocks/catalog/TableProperty.java 17 23 73.91% [397, 398, 406, 1087, 1094, 1101]
🔵 com/starrocks/scheduler/PartitionBasedCooldownProcessor.java 75 100 75.00% [68, 71, 76, 80, 83, 88, 91, 97, 107, 108, 109, 110, 111, 112, 113, 114, 154, 155, 156, 157, 165, 166, 167, 168, 198]
🔵 com/starrocks/qe/DDLStmtExecutor.java 29 38 76.32% [429, 445, 448, 453, 454, 455, 456, 458, 459]
🔵 com/starrocks/externalcooldown/ExternalCooldownSchedule.java 48 62 77.42% [50, 54, 57, 67, 68, 73, 74, 76, 85, 113, 129, 134, 141, 144]
🔵 com/starrocks/sql/analyzer/AlterTableClauseAnalyzer.java 22 27 81.48% [336, 344, 349, 358, 367]
🔵 com/starrocks/sql/analyzer/ExternalCooldownAnalyzer.java 18 22 81.82% [34, 35, 46, 62]
🔵 com/starrocks/server/GlobalStateMgr.java 6 7 85.71% [1530]
🔵 com/starrocks/scheduler/externalcooldown/ExternalCooldownMaintenanceJob.java 71 80 88.75% [111, 112, 115, 116, 117, 132, 149, 153, 194]
🔵 com/starrocks/externalcooldown/ExternalCooldownPartitionSelector.java 131 145 90.34% [122, 123, 124, 125, 181, 182, 183, 191, 217, 241, 280, 281, 282, 291]
🔵 com/starrocks/scheduler/TaskBuilder.java 61 65 93.85% [368, 369, 370, 371]
🔵 com/starrocks/sql/analyzer/Analyzer.java 4 4 100.00% []
🔵 com/starrocks/server/OlapTableFactory.java 11 11 100.00% []
🔵 com/starrocks/sql/ast/CreateExternalCooldownStmt.java 24 24 100.00% []
🔵 com/starrocks/scheduler/TaskRunBuilder.java 2 2 100.00% []
🔵 com/starrocks/sql/analyzer/CreateTableLikeAnalyzer.java 1 1 100.00% []
🔵 com/starrocks/sql/ast/CancelExternalCooldownStmt.java 8 8 100.00% []
🔵 com/starrocks/common/proc/PartitionsProcDir.java 15 15 100.00% []
🔵 com/starrocks/scheduler/externalcooldown/ExternalCooldownJobExecutor.java 23 23 100.00% []
🔵 com/starrocks/scheduler/Constants.java 2 2 100.00% []
🔵 com/starrocks/qe/StmtExecutor.java 4 4 100.00% []
🔵 com/starrocks/persist/metablock/SRMetaBlockID.java 1 1 100.00% []
🔵 com/starrocks/alter/AlterJobExecutor.java 13 13 100.00% []
🔵 com/starrocks/sql/ast/CreateTableStmt.java 5 5 100.00% []
🔵 com/starrocks/persist/ModifyPartitionInfo.java 8 8 100.00% []
🔵 com/starrocks/externalcooldown/ExternalCooldownConfig.java 58 58 100.00% []
🔵 com/starrocks/journal/JournalEntity.java 4 4 100.00% []

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation META-REVIEW PROTO-REVIEW
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support cooldown to external lake table
2 participants