Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Hive]To avoid excessive HMS memory usage, when executing AlterTable for a Paimon table containing a large number of fields #4549

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

GangYang-HX
Copy link
Contributor

@GangYang-HX GangYang-HX commented Nov 19, 2024

Purpose

When metastore=hive is set, if the Paimon table has many fields (1600+), the MetaStoreUtils.updateTableStatsSlow() method will be called by default to update the table statistics, which will cause a sharp increase in HMS memory usage.
In fact, Paimon has its own statistics and does not rely on Hive's statistics.

So, an option is provided to control whether to update Hive table statistics to avoid excessive HMS memory usage.

Linked issue: Issue-4507

API and Format

org.apache.paimon.hive.HiveCatalog#alterTableToHms

…tContext, and then set DO_NOT_UPDATE_STATS to prohibit updating hive table statistics.
@xuzifu666
Copy link
Member

Please change your title 【Hive】to [Hive] and make your code style with format, you can execute 'mvn spotless:apply' for it

@GangYang-HX GangYang-HX changed the title 【Hive】To avoid excessive HMS memory usage, when executing AlterTable for a Paimon table containing a large number of fields [Hive]To avoid excessive HMS memory usage, when executing AlterTable for a Paimon table containing a large number of fields Nov 19, 2024
@GangYang-HX
Copy link
Contributor Author

Please change your title 【Hive】to [Hive] and make your code style with format, you can execute 'mvn spotless:apply' for it

Thank you. All the tests can be passed locally, but there are still errors now. It seems that the errors have nothing to do with my changes.

@JingsongLi JingsongLi closed this Nov 20, 2024
@JingsongLi JingsongLi reopened this Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants