Remove non-determinism from test TestPartitionManagement #5553
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
okumin
reviewed
Nov 22, 2024
| @@ -91,13 +91,19 @@ public void tearDown() throws Exception { | |||
| // First drop any databases in catalog | |||
| List<String> databases = client.getAllDatabases(catName); | |||
| for (String db : databases) { | |||
| for (String table : client.getAllTables(catName, db)) { | |||
| client.dropTable(catName, db, table, true, true); | |||
| } | |||
| client.dropDatabase(catName, db, true, false, true); | |||
There was a problem hiding this comment.
Is it possible that the cascade option is not implemented as expected?
https://github.com/apache/hive/blob/rel/release-4.0.1/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java#L1844-L1845
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR fixes the error resulting from the flaky tests: org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
The mentioned test is non-deterministic as it throws error unexpectedly.
Why are the changes needed?
The test fails because the
tearDown()method attempts to delete a record (DBS entry) that is referenced by another table (TBLS), which caused a violation of foreign key constraint ‘TBLS_FK1’.[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hive.metastore.TestPartitionManagement
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 13.15 s <<< FAILURE! -- in org.apache.hadoop.hive.metastore.TestPartitionManagement
[ERROR] org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable -- Time elapsed: 13.15 s <<< ERROR!
MetaException(message:JDODataStoreException: Exception thrown flushing changes to datastore
Root cause: ERROR 23503: DELETE on table 'DBS' caused a violation of foreign key constraint 'TBLS_FK1' for key (2). The statement has been rolled back.)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$drop_database_req_result$drop_database_req_resultStandardScheme.read(ThriftHiveMetastore.java:59587)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$drop_database_req_result$drop_database_req_resultStandardScheme.read(ThriftHiveMetastore.java:59555)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$drop_database_req_result.read(ThriftHiveMetastore.java:59489)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_drop_database_req(ThriftHiveMetastore.java:1545)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.drop_database_req(ThriftHiveMetastore.java:1532)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabaseCascadePerDb(HiveMetaStoreClient.java:1870)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropDatabase(HiveMetaStoreClient.java:1794)
at org.apache.hadoop.hive.metastore.IMetaStoreClient.dropDatabase(IMetaStoreClient.java:1870)
at org.apache.hadoop.hive.metastore.TestPartitionManagement.tearDown(TestPartitionManagement.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)....
Reproduce the test failure
Run the tests with NonDex maven plugin which is used for detecting and debugging wrong assumptions on under-determined Java APIs. The command to reproduce the flaky test failures is:
Does this PR introduce any user-facing change?
No, it just fixes the test.
Is the change a dependency upgrade?
No
Fix
Drop all tables first before dropping the databases.
How was this patch tested?
After adding the code to delete the tables as well, again NonDex was run to ensure the test passes all the times.