ci: Split Temporal backend suite into three

[Twixes]

Problem

The Temporal suite is taking around 25 minutes to complete (example, not an unusual one). A 25 minute wait kills productivity.

Changes

Splitting the suite into three groups.

[Twixes]

Looks like this didn't have as much of an effect, because the run time is heavily skewed towards a few tests. Group 3 still takes 20 minutes. These tests are the biggest offenders, each taking over 60 seconds:

120.84s call    posthog/temporal/tests/batch_exports/test_snowflake_batch_export_workflow.py::test_snowflake_export_workflow_raises_error_on_copy_fail
105.78s call    posthog/temporal/tests/batch_exports/test_snowflake_batch_export_workflow.py::test_snowflake_export_workflow_raises_error_on_put_fail
61.24s call     posthog/temporal/tests/batch_exports/test_s3_batch_export_workflow.py::test_s3_export_workflow_with_request_timeouts[model0]
60.63s call     posthog/temporal/tests/batch_exports/test_s3_batch_export_workflow.py::test_s3_export_workflow_with_request_timeouts[None]
60.50s call     posthog/temporal/tests/batch_exports/test_s3_batch_export_workflow.py::test_s3_export_workflow_with_request_timeouts[model1]
60.07s call     posthog/temporal/tests/batch_exports/test_s3_batch_export_workflow.py::test_s3_multi_part_upload_raises_retryable_exception

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[neilkakkar]

@Twixes I didn't care too much about the .test_durations file for temporal earlier because there was only one group, but now if you update that, this splitting should be more clever.

Edit: Looks like posthog/temporal/tests/batch_exports/test_snowflake_batch_export_workflow.py::test_snowflake_export_workflow_raises_error_on_copy_fail in that file used to take 20s, so the splitting is old / off now.

[neilkakkar]

this is a good way to do it (vs running locally which can be annoying) -> https://github.com/PostHog/posthog/blob/master/.github/workflows/ci-backend-update-test-timing.yml#L43 -> https://github.com/PostHog/posthog/actions/workflows/ci-backend-update-test-timing.yml

although iirc last time temporal failed for some reason, which is why I split them out in the first place. I think if you just try running temporal tests, might work here.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[Twixes]

@neilkakkar, yup, already updated .test_durations in 09e878b (#25880), only posthog/temporal/**/* ones. So each Temporal group takes ~11 minutes now, rather than one taking ~25.

[neilkakkar]

ah missed that commit, perfect 👍