TEST: use Github based GPU instance for CI

[kp992]

Built on top of #181

Use Github Actions GPU instance on CI.

[netlify]

✅ Deploy Preview for incomparable-parfait-2417f8 ready!

Name	Link
🔨 Latest commit	4d75f11
🔍 Latest deploy log	https://app.netlify.com/sites/incomparable-parfait-2417f8/deploys/665ca9bf5294f200088e2aa3
😎 Deploy Preview	https://deploy-preview-183--incomparable-parfait-2417f8.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions]

🚀 Deployed on https://665cac4fc001c38b65e43afe--incomparable-parfait-2417f8.netlify.app

[kp992]

@mmcky Everything looks good now? I can remove the commented cache code.

[mmcky]

@kp992 I still don't have a good undertanding of why we can't just setup anaconda directly on the instance (rather than using a docker container). Any insights into why the kernel keeps dying in that context?

[kp992]

Thanks @mmcky, I will try to setup it and check.

[kp992]

@mmcky I waited for about 18 mins but seems like all the machines are busy. This can be a tricky problem to solve if we would depend on GA actions job. I cancelled it for now as I didn't want it to keep it running because that I am experimenting and runtime may go high.

[mmcky]

@kp992 thanks for trying some things out however 2ff9298 still uses a docker container which is not required. If you take a look at most of our github actions workflows then you can see we don't use a docker container in most of them. That is what I am proposed as the docker download takes a long time.

[kp992]

Thanks @mmcky, I'm also trying to find that approach. I was verifying that we somehow need to install cuda. One of the approach is through docker as seen. I will try with conda.

[kp992]

Hmm, its not working. The best way I think is to use the docker that you have made (existing main branch CI) and cache it. Since we are not updating the image frequently, that should be easier to manage. How does that sound?
Ref: https://github.com/marketplace/actions/build-docker-images-using-cache

[mmcky]

Hmm, its not working. The best way I think is to use the docker that you have made (existing main branch CI) and cache it. Since we are not updating the image frequently, that should be easier to manage. How does that sound? Ref: https://github.com/marketplace/actions/build-docker-images-using-cache

@kp992 can we build the docker image on GitHub actions and then cache it from another repo?

I build the image https://github.com/quantecon/lecture-python.docker and then sync it to docker hub. But if we can save it as an asset in QuantEcon GitHub then we can pull it locally which should be quicker.

[kp992]

I am trying to do the same, but, for a single repo. For example check the last two commits of this PR which have Github actions run: https://github.com/QuantEcon/lecture-jax/actions/runs/9258083875/job/25555094247 (first run) and https://github.com/QuantEcon/lecture-jax/actions/runs/9287258102/job/25555816581(second run). First run took about 13m and the second one about 11m so we are saving 2 mins. I think we can also optimize the Anaconda setup by including it in the docker itself which you have done in https://github.com/quantecon/lecture-python.docker. I will try it and see the results.

[kp992]

https://github.com/QuantEcon/lecture-jax/actions/runs/9339691234/job/25704378502 took about 34 mins to load our docker which is very high compared to pulling nvidia docker and setting up conda differently which took about 13 mins.

[kp992]

@mmcky How does this commit 9ee6f21 look? Should we go ahead with nvidia cuda docker (cache enables) and Anaconda setup separately?

[mmcky]

https://github.com/QuantEcon/lecture-jax/actions/runs/9339691234/job/25704378502 took about 34 mins to load our docker which is very high compared to pulling nvidia docker and setting up conda differently which took about 13 mins.

@kp992 I suspect the nvidia docker is cached as a I think they are similar GB sizes. I was wantign to see if we can cache our docker container as a GitHub asset to improve on this. Do you know if this is possible -- we build that container on github and then send it to dockerhub but a local asset might be a better way to go if there is a local registry.

[kp992]

Hmm, I am not aware of anything like that. I guess Github provides about 10 GB of caching where we can use our pulled docker container to be saved in .zip file and pull it whenever we run it next time. Since, docker won't be updated very frequently, it is maintainable too.

[mmcky]

@kp992 rather than managing cache objects ourselves (as they usually expire after 1 month of inactivity) I was thinking of using the github registry for the docker container. Is this feasible?

https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry

[mmcky]

@kp992 I think this can now be closed in favour of #181