Slow shutdown with multi-controller clusters

[tombentley]

Describe the bug

When using multiple controllers a KafkaCluster takes 5 minutes to shutdown, every time.

To Reproduce

@ExtendWith(KafkaClusterExtension.class)
class ExampleTest {
    @Test
    public void test(
            @BrokerCluster(numBrokers = 3)
            @KRaftCluster(numControllers = 3)
            KafkaCluster cluster) {
        System.out.println("###### Waiting 10s");
        Thread.sleep(10_000);
        System.out.println("###### End of test");
    }
}

Expected behavior

The cluster would shutdown more quickly.

Logs
logs.txt

[tombentley]

I think this is https://issues.apache.org/jira/browse/KAFKA-14287

[SamBarker]

Should we default to using separate brokers and controllers? Rather than combined mode?

I'd say the extra memory consumption is worth it for faster tests but I don't know that's a reasonable position for the library to take without an opt out?

[tombentley]

I think that might be reasonable. We could either add

• a combinedMode=true attribute on `@KraftCluster
• or a separate @CombinedMode annotation to be used alongside @BrokerCluster and @KraftCluster

The former feels like it should be sufficient.

[SamBarker]

Yeah I think a combinedMode attribute is enough.

I have a loose sense that we might want to consider topology API where users can control things like ID's and racks etc but I don't have any thoughts on what form that would take or if it would really be useful. I sort of see it as being helpful for re-creating topologies in tests but not really clear if there is really value in that.

I'm happy to pick this up but probably wont be for a couple days so if you want a faster fix you would need to pick it up.

[tombentley]

@SamBarker I've started looking at this.