Run CI on self-hosted machine

[kunxian-xia]

Closes #451.

[matthiasgoergens]

Looks like the runners ain't picking this up, yet?

EDIT: Oh, looks like it's running, but failing (in a way that doesn't give use the X on GitHub.)

I suggest we consider running the CI inside a docker container on the self-hosted machine. Then we can put the config for that container inside of the repository. (Instead of manually setting up the self-hosted machine with the config only known outside of the repository.) @kunxian-xia

[matthiasgoergens]

Looking at the logs, it looks like the integration test actually succeeded, but didn't manage to tell github about it?

[matthiasgoergens]

Oh, @kunxian-xia reports that GitHub actions is having problems.

[matthiasgoergens]

Please add the Dockerfile (or similar) and scripts necessary to change the container we are running on.

Right now, it works by 'magic', but a reader of the repository would have no clue how to eg change to Ubuntu 24 or anything like that.

[kunxian-xia]

You can find the Dockerfile and scripts here.

[matthiasgoergens]

You can find the Dockerfile and scripts here.

Please add them to the repository. Thanks.

Please add enough context so that someone else can take what's in the repository and upgrade our container, if necessary.

Adding the scripts to a random gist that's not under the scroll-tech organisation and that you can only find if you go through old PR review comments (and not from the repository source) is pretty much useless. There's also not enough context with just these two scripts to enable anyone to actually do anything with them.

We need this in the repository, so that you can go on holiday, or take a sick day, or get a promotion to CTO or even just go to bed and sleep, and still someone else can figure out how everything works and make changes.

[matthiasgoergens]

Thanks for adding the two files.

Please add some context in the README that describes what someone needs to do to get changes to these files to be reflected in the running CI. I assume it's not automatic like with changes to eg lints.yml?

[kunxian-xia]

Please add some context in the README that describes what someone needs to do to get changes to these files to be reflected in the running CI. I assume it's not automatic like with changes to eg lints.yml?

@matthiasgoergens Added in b64ab0f.

[kunxian-xia]

The test scheme::tests::test_rw_lk_expression_combination is hanging for like 30 mins.

[matthiasgoergens]

How does doing this manually compare to using the official https://github.com/actions/runner/pkgs/container/actions-runner?

[matthiasgoergens]

Please see comments.

[kunxian-xia]

How does doing this manually compare to using the official https://github.com/actions/runner/pkgs/container/actions-runner?

I don't know how the official container work.

[matthiasgoergens]

Let's just use the official image, instead of cobbling together our own. Less hassle that way.

[kunxian-xia]

The default CI runner provided by GitHub is not powerful enough to run the fibonacci example in #368.

We can address it in future PR.

[naure]

Have you tried --max-steps=10000 ?

Or any other number of steps. That makes it as fast as you want. You can extrapolate the timings to the full 11_538_921 steps (e.g. multiply by 1153.8921).

[matthiasgoergens]

Please use the official image, instead of copy and paste.

[matthiasgoergens]

Please see my outstanding comments.

Please don't do even more copy and paste, and please provide some comments for how someone would go and fix the runner or make any changes.

(Btw, as far as I can tell our VPN is still insecure. Has the Ceno server been fixed? I can't access it via VPN without endangering my scroll credentials.)

[kunxian-xia]

First of all, I don't know how to use the official github image. If you know how to use that to spin up CI machine, feel free to make a PR.
Secondly, the Dockerfile and start.sh does work for our case, why not just use it?

[matthiasgoergens]

Have you tried learning how to use the official image? Have you hit any blockers I can try to help you with? You are a smart fellow, you can do it!

I would love to set up a self hosted runner and create a new PR (or take over this one), but I can't access the VPN without endangering my username and password. (And you should probably not access the VPN either, until it's either fixed or Sina tells you it's ok.) I reported the problem a while ago, and recently followed up again.

Please have a look at my other comments, too. The PR does not give enough context for someone to fix or upgrade the runner. We already discussed this.

Let’s try it first!

[kunxian-xia]

Please refer to https://www.notion.so/scrollzkp/Ceno-s-self-hosted-CI-runner-1437792d22af800eba16e4b971e82619?pvs=4 for more detailed instructions on setting up our private CI runner.

[matthiasgoergens]

Please refer to https://www.notion.so/scrollzkp/Ceno-s-self-hosted-CI-runner-1437792d22af800eba16e4b971e82619?pvs=4 for more detailed instructions on setting up our private CI runner.

Please put that where someone can find it. Put it in the README for example. No one looks in old reviews when their hair is on fire. The instruction on notion aren't particularly useful, either: I still have no clue how to access the server, and it's mostly just copy and pasted what's already in the README here. (I left some comments on notion.)

Having said that, sadly this is better than nothing.. Why do you need to make this so difficult? Are there are any constraints I'm missing? Please educate me, I'm really at a loss.