[CELEBORN-1711][TEST] Fix flaky test caused by master/worker setup issue #2906

turboFei · 2024-11-12T05:09:31Z

What changes were proposed in this pull request?

retry on BindException when starting master/worker http server
record the used ports and pre-check whether the selected port is used or bounded before binding

Why are the changes needed?

To fix flaky test.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

GA.

turboFei · 2024-11-12T08:36:41Z

[info] org.apache.celeborn.tests.spark.PushDataTimeoutTest *** ABORTED ***
[info]   java.lang.AssertionError: assertion failed
[info]   at scala.Predef$.assert(Predef.scala:208)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.$anonfun$setUpWorkers$7(MiniClusterFeature.scala:234)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.$anonfun$setUpWorkers$7$adapted(MiniClusterFeature.scala:234)
[info]   at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
[info]   at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
[info]   at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
[info]   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
[info]   at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpWorkers(MiniClusterFeature.scala:234)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpWorkers$(MiniClusterFeature.scala:181)
[info]   at org.apache.celeborn.tests.spark.PushDataTimeoutTest.setUpWorkers(PushDataTimeoutTest.scala:34)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpMiniCluster(MiniClusterFeature.scala:254)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpMiniCluster$(MiniClusterFeature.scala:250)
[info]   at org.apache.celeborn.tests.spark.PushDataTimeoutTest.setUpMiniCluster(PushDataTimeoutTest.scala:34)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setupMiniClusterWithRandomPorts(MiniClusterFeature.scala:77)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setupMiniClusterWithRandomPorts$(MiniClusterFeature.scala:49)
[info]   at org.apache.celeborn.tests.spark.PushDataTimeoutTest.setupMiniClusterWithRandomPorts(PushDataTimeoutTest.scala:34)
[info]   at org.apache.celeborn.tests.spark.PushDataTimeoutTest.beforeAll(PushDataTimeoutTest.scala:47)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.celeborn.tests.spark.PushDataTimeoutTest.run(PushDataTimeoutTest.scala:34)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:750)

turboFei · 2024-11-12T08:37:15Z

[info] org.apache.celeborn.tests.spark.SkewJoinSuite *** ABORTED ***
[info]   java.lang.AssertionError: assertion failed
[info]   at scala.Predef$.assert(Predef.scala:208)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.$anonfun$setUpWorkers$7(MiniClusterFeature.scala:234)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.$anonfun$setUpWorkers$7$adapted(MiniClusterFeature.scala:234)
[info]   at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
[info]   at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
[info]   at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
[info]   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
[info]   at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpWorkers(MiniClusterFeature.scala:234)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpWorkers$(MiniClusterFeature.scala:181)
[info]   at org.apache.celeborn.tests.spark.SkewJoinSuite.setUpWorkers(SkewJoinSuite.scala:32)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpMiniCluster(MiniClusterFeature.scala:254)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpMiniCluster$(MiniClusterFeature.scala:250)
[info]   at org.apache.celeborn.tests.spark.SkewJoinSuite.setUpMiniCluster(SkewJoinSuite.scala:32)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setupMiniClusterWithRandomPorts(MiniClusterFeature.scala:77)
[info]   at org.apache.celeborn.service.deploy.MiniClusterFeature.setupMiniClusterWithRandomPorts$(MiniClusterFeature.scala:49)
[info]   at org.apache.celeborn.tests.spark.SkewJoinSuite.setupMiniClusterWithRandomPorts(SkewJoinSuite.scala:32)
[info]   at org.apache.celeborn.tests.spark.SparkTestBase.beforeAll(SparkTestBase.scala:47)
[info]   at org.apache.celeborn.tests.spark.SparkTestBase.beforeAll$(SparkTestBase.scala:45)
[info]   at org.apache.celeborn.tests.spark.SkewJoinSuite.beforeAll(SkewJoinSuite.scala:32)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.celeborn.tests.spark.SkewJoinSuite.run(SkewJoinSuite.scala:32)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)

turboFei · 2024-11-13T00:06:18Z

24/11/12 20:24:33,875 ERROR [ScalaTest-main-running-DiscoverySuite] ReusedExchangeSuite: cannot start all workers after 60000 ms
java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:208)
	at org.apache.celeborn.service.deploy.MiniClusterFeature.$anonfun$setUpWorkers$7(MiniClusterFeature.scala:246)
	at org.apache.celeborn.service.deploy.MiniClusterFeature.$anonfun$setUpWorkers$7$adapted(MiniClusterFeature.scala:246)
	at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
	at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
	at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
	at org.apache.celeborn.service.deploy.MiniClusterFeature.setUpWorkers(MiniClusterFeature.scala:246)

turboFei · 2024-11-13T23:41:48Z

cc @FMX @RexXiong @SteNicholas It is to reduce the flaky test failures.

turboFei · 2024-11-20T19:49:38Z

gentle ping @FMX @RexXiong @SteNicholas

turboFei · 2024-11-21T00:25:36Z

Seems the port conflicts issue happen with high frequency recently.

Hope this patch can help reduce it.

master/src/test/scala/org/apache/celeborn/service/deploy/master/MasterClusterFeature.scala

turboFei · 2024-11-22T04:06:49Z

interesting，so many failures recently

…ception reduce sleep time not needed remove unused used ports nit sleep and then check timeout and config the timeout retest please do not sleep check portBounded more random worker ports retest please synchronized shutdown increase timeout

codecov · 2024-11-23T05:19:53Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 32.52%. Comparing base (aa62549) to head (c92199e).
Report is 29 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2906      +/-   ##
==========================================
+ Coverage   32.07%   32.52%   +0.46%     
==========================================
  Files         331      334       +3     
  Lines       19749    19940     +191     
  Branches     1778     1799      +21     
==========================================
+ Hits         6332     6484     +152     
- Misses      13071    13096      +25     
- Partials      346      360      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

turboFei marked this pull request as draft November 12, 2024 05:22

turboFei changed the title ~~[CELEBORN-1711][TEST] Setup celeborn master with retry on port BindException~~ [WIP][CELEBORN-1711][TEST] Setup celeborn master with retry on port BindException Nov 12, 2024

turboFei force-pushed the retry_master_suite branch from 12e3d19 to 8555887 Compare November 12, 2024 05:58

turboFei changed the title ~~[WIP][CELEBORN-1711][TEST] Setup celeborn master with retry on port BindException~~ [WIP][CELEBORN-1711][TEST] Setup celeborn master/worker http server with retry on port BindException Nov 12, 2024

turboFei force-pushed the retry_master_suite branch from 8555887 to 54fc584 Compare November 12, 2024 06:14

turboFei changed the title ~~[WIP][CELEBORN-1711][TEST] Setup celeborn master/worker http server with retry on port BindException~~ [CELEBORN-1711][TEST] Setup celeborn master/worker http server with retry on port BindException Nov 12, 2024

turboFei force-pushed the retry_master_suite branch from 54fc584 to 9ed757f Compare November 12, 2024 06:43

turboFei mentioned this pull request Nov 12, 2024

[CELEBORN-1711][TEST] Fix '65535' port is invalid #2901

Closed

turboFei force-pushed the retry_master_suite branch from 9ed757f to 2a3b7da Compare November 12, 2024 07:51

turboFei marked this pull request as ready for review November 12, 2024 07:51

turboFei marked this pull request as draft November 12, 2024 09:14

turboFei force-pushed the retry_master_suite branch from 48025b3 to 5b3c7d0 Compare November 12, 2024 17:30

turboFei marked this pull request as ready for review November 12, 2024 17:39

turboFei force-pushed the retry_master_suite branch from dd1e191 to f0ce882 Compare November 12, 2024 18:54

turboFei force-pushed the retry_master_suite branch 3 times, most recently from 7088ead to 03b0282 Compare November 13, 2024 01:46

turboFei requested a review from SteNicholas November 13, 2024 02:47

turboFei changed the title ~~[CELEBORN-1711][TEST] Setup celeborn master/worker http server with retry on port BindException~~ [CELEBORN-1711][TEST] Fix flaky test because of master/worker setup Nov 13, 2024

turboFei changed the title ~~[CELEBORN-1711][TEST] Fix flaky test because of master/worker setup~~ [CELEBORN-1711][TEST] Fix flaky test because of master/worker setup issue Nov 13, 2024

turboFei changed the title ~~[CELEBORN-1711][TEST] Fix flaky test because of master/worker setup issue~~ [CELEBORN-1711][TEST] Fix flaky test caused by master/worker setup issue Nov 13, 2024

turboFei requested review from FMX, cfmcgrady and RexXiong November 20, 2024 19:50

onebox-li reviewed Nov 22, 2024

View reviewed changes

master/src/test/scala/org/apache/celeborn/service/deploy/master/MasterClusterFeature.scala Outdated Show resolved Hide resolved

turboFei requested a review from onebox-li November 22, 2024 03:30

turboFei force-pushed the retry_master_suite branch 2 times, most recently from feb3ac5 to c462899 Compare November 22, 2024 04:35

turboFei force-pushed the retry_master_suite branch from c462899 to c92199e Compare November 22, 2024 05:32

turboFei closed this Nov 23, 2024

turboFei reopened this Nov 23, 2024

onebox-li approved these changes Nov 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CELEBORN-1711][TEST] Fix flaky test caused by master/worker setup issue #2906

[CELEBORN-1711][TEST] Fix flaky test caused by master/worker setup issue #2906

turboFei commented Nov 12, 2024 •

edited

Loading

turboFei commented Nov 12, 2024

turboFei commented Nov 12, 2024

turboFei commented Nov 13, 2024

turboFei commented Nov 13, 2024 •

edited

Loading

turboFei commented Nov 20, 2024

turboFei commented Nov 21, 2024 •

edited

Loading

turboFei commented Nov 22, 2024 •

edited

Loading

codecov bot commented Nov 23, 2024

[CELEBORN-1711][TEST] Fix flaky test caused by master/worker setup issue #2906

Are you sure you want to change the base?

[CELEBORN-1711][TEST] Fix flaky test caused by master/worker setup issue #2906

Conversation

turboFei commented Nov 12, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

turboFei commented Nov 12, 2024

turboFei commented Nov 12, 2024

turboFei commented Nov 13, 2024

turboFei commented Nov 13, 2024 • edited Loading

turboFei commented Nov 20, 2024

turboFei commented Nov 21, 2024 • edited Loading

turboFei commented Nov 22, 2024 • edited Loading

codecov bot commented Nov 23, 2024

Codecov Report

turboFei commented Nov 12, 2024 •

edited

Loading

turboFei commented Nov 13, 2024 •

edited

Loading

turboFei commented Nov 21, 2024 •

edited

Loading

turboFei commented Nov 22, 2024 •

edited

Loading