[Task] Reduce Runtime of Merlin Models Tests With Reorganization and Configuration

[oliverholworthy]

This task is about improving the feedback loop on Merlin Models Pull Requests through the reorganization and configuration of the tests. Refactoring the implementation of the tests themselves is beyond the scope of this issue.

How can we make things run more quickly and learn about failures more quickly through the organizaiton of the tests and configuration of the GitHub Actions Workflows.

Configuration

• Try using a larger runner for the CPU tests. This will enable the tests to run in parallel and reduce the running time of the CPU-only tensorflow tests
  • Requires the NVIDIA-Merlin account to be configured with GitHub Enterprise.
  • Configure one version to run quickly (e.g. latest versions Python 3.11, Tensorflow 2.12) and the others in the test matrix could continue to run in the same smaller runners

Reorganize

• split out the notebook tests on the GPU runner to run in a separate job or workflow from the unit tests
• reorganize the slow running model-based to run after the fast-running unit tests that only test individual layers or functions.