Testing create-wsl-image.yml

[soehms]

This issue continues the discussion started in a sage-devel thread on Using Sage on Windows regarding the approach suggested by @tobiasdiez using a custom WSL distribution. An according implementation of this can be found here in .github/workflows/create-wsl-image.yml. Tests of it are documented in this issue.

[soehms]

Testreport on commit 7d49318.

The test was on a VM running Windows 10 Home.

Again I was unable to paste the script into the Powershell terminal. This point worked better in my previous test last Saturday. But maybe this is a problem caused by the virtual machine. When this happens, the paste process is very slow, it seems to go line by line and at the end the code appears in reverse order in the terminal (see attached file paste.log).

So I saved the code from the editor under a different name sagemath-10.4-wsl-installer-test.ps1 and ran that instead. Now the installation of WSL was done. The tar-file was downloaded before rebooting the system. The output can be found in install_7d49318.log.

After rebooting the VM, I ran sagemath-10.4-wsl-installer-test.ps1 again. This gave me working Sage icons (see install_7d49318_part2.log).

Before installation, the VM had 19.4 GB of used disk space, after installation it took up about 35 GB. I think there should be potential to reduce that. In your post on sage-devel (in the Using Sage on Windows thread), you said:

My installer installs a minimized Sage (no build facility, no optional package, no local docs), basically the same sage that you get from the sage-binder-env (https://github.com/sagemath/sage-binder-env).

I guess you mean ghcr.io/sagemath/sage-binder-env:10.4? This is more than 3/5 larger than the according image on Docker Hub:

sebastian@SebaThinkPad:~$ docker images
REPOSITORY                         TAG          IMAGE ID       CREATED         SIZE
ghcr.io/sagemath/sage-binder-env   10.4         548c4e2559de   21 hours ago    8.37GB
sagemath/sagemath                  10.5.beta6   c952169891ff   2 weeks ago     5.14GB
sagemath/sagemath                  10.4         8d657a42f33a   3 months ago    5.21GB
computop/sage                      10.0         ae32949dc28d   16 months ago   9.39GB

Another suggestions: I think it would be worth considering user messages that make it clear that a reboot is required and the script needs to be called again afterwards. I did something similar regarding the installation of Docker Desktop (see section 2.1 of the documentation).

[kwankyu]

Thanks for testing.

The test was on a VM running Windows 10 Home.

I used Windows 11 Pro for testing. WSL is WSL2. I have no machine Windows 10 with WSL enabled...

Again I was unable to paste the script into the Powershell terminal. This point worked better in my previous test last Saturday. But maybe this is a problem caused by the virtual machine. When this happens, the paste process is very slow, it seems to go line by line and at the end the code appears in reverse order in the terminal (see attached file paste.log).

This is very strange. On my side, I had a different set of strange things happening. Then I learned that powershell works with UTF16 output encoding by default. Thus some of the strange things were cleared up after I set [Console]::OutputEncoding = [System.Text.Encoding]::UTF8. The current .ps1 installer starts with setting the UTF8 encoding. Would you try again?

You can download the installer, and then right click it and "Run with PowerShell" to avoid copy and paste.

[tobiasdiez]

Usually, people recommend to download the installer script and then execute it. This circumvents the issues with the slow pasting process. Eg. chocolatey recommends on https://chocolatey.org/install#individual:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

[kwankyu]

I provided a .exe installer created by ps2exe from the .ps1 file. But because Windows falsely detects a virus from the .exe file and because I don't see a big merit of the .exe installer, I removed it.

[soehms]

I used Windows 11 Pro for testing. WSL is WSL2. I have no machine Windows 10 with WSL enabled...

My host system is also Windows 11 Pro. For testing, I use Hyper-V Manager, for which I set up Windows 10 and 11 Home using ISO images. If you have worked with Virtual-Box before, this is very similar to handle. However, do not use Virtual-Box (neither on Linux nor on Windows)! This is not supposed to expose the CPU virtualization sufficiently to the guest sytem for WSL.

You can download the installer, and then right click it and "Run with PowerShell" to avoid copy and paste.

I'll do that later on (I don't carry a Windows machine with me).

[soehms]

Usually, people recommend to download the installer script and then execute it. This circumvents the issues with the slow pasting process. Eg. chocolatey recommends on https://chocolatey.org/install#individual:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

Does this change the registry?

[kwankyu]

Another suggestions: I think it would be worth considering user messages that make it clear that a reboot is required ...

Installing the WSL image does not require a reboot.

Installing WSL may need a reboot. Do you mean this?

[kwankyu]

For me,

REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
ghcr.io/sagemath/sage-binder-env   10.4      169f8a0e546e   12 hours ago   5.91GB
sagemath/sagemath                  10.4      438901ea598d   3 months ago   3.45GB

Anyway, this is striking to me. How can sagemath/sagemath be so small?

[kwankyu]

The sage directories have similar sizes

(ghcr.io/sagemath/sage-binder-env)
du -sh sage
3.5G	sage

(sagemath/sagemath)
du -sh sage
3.2G	sage

The difference seems to be in system packages.

[soehms]

Another suggestions: I think it would be worth considering user messages that make it clear that a reboot is required ...

Installing the WSL image does not require a reboot.

Installing WSL may need a reboot. Do you mean this?

Exactly!

[soehms]

For me,

REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
ghcr.io/sagemath/sage-binder-env   10.4      169f8a0e546e   12 hours ago   5.91GB
sagemath/sagemath                  10.4      438901ea598d   3 months ago   3.45GB

I guess you obtained this on Linux. On Linux Mint I get a similar result:

REPOSITORY                  TAG          IMAGE ID       CREATED         SIZE
sagemath/sagemath           10.4.beta6   40971036aa1a   5 months ago    3.13GB

Yes, that's weird! I was wondering why the image size of sagemath/sagemath increased so much in a short period of time. But the larger sizes seem to be due to Docker on Windows. But even when I run docker images in a WSL Ubuntu Bash terminal, I get the same sizes as with Powershell. I'll see later if the size changes in WSL Ubuntu when I stop (or uninstall) Docker Desktop.

Anyway, this is striking to me. How can sagemath/sagemath be so small?

AFAIK the essential part for this in docker/Dockerfile is in this code-line:

COPY --chown=sage:sage --from=make-release $SAGE_ROOT/ $SAGE_ROOT/

But I'm not sure. Maybe, @saraedum knows it?

[kwankyu]

For me,

REPOSITORY                         TAG       IMAGE ID       CREATED        SIZE
ghcr.io/sagemath/sage-binder-env   10.4      169f8a0e546e   12 hours ago   5.91GB
sagemath/sagemath                  10.4      438901ea598d   3 months ago   3.45GB

I guess you obtained this on Linux.

I obtained that on Mac.

It seems that the local size of a docker image may fluctuate in time since an image consists of layers, which may change(?) for some reasons. I less care about this.

As I shown above, the sage-binder-env image has much larger /usr directory (specifically /usr/lib directory) than the sagemath image. I wonder if (and how) the sage in sage-binder-env is more powerful than the sage in sagemath by the extra size. Otherwise I would squeeze the sage-binder-env image more to make it smaller :-) I have no idea now.

[kwankyu]

Another suggestions: I think it would be worth considering user messages that make it clear that a reboot is required and the script needs to be called again afterwards.

Basically the installer assumes a functional WSL installation. (It runs wsl --install --no-distritbution at the beginning, but this is rather for checking WSL installation) So I think such a user message is not relevant. Instead I added WSL requirement to the Release description to guide the user in that regard.

[kwankyu]

I tested the installer on a virgin (no WSL install before :-) computer. Now I agree with you about the need to reboot. I added a relevant message to the installer. Thanks.

[soehms]

You can download the installer, and then right click it and "Run with PowerShell" to avoid copy and paste.

Yes, that worked better!

[soehms]

It seems that the local size of a docker image may fluctuate in time since an image consists of layers, which may change(?) for some reasons. I less care about this.

On another Windows machine where I also installed Docker using apt in WSL and Docker Desktop, the size is shown correctly at 3.45 GB (in both Bash and Powershell). So the incorrect size seems to be an issue on that particular machine. I'm sure I had Docker running in WSL before installing Docker Desktop on the machine which shows the correct sizes. Maybe it was the other way around on the other machine, so my guess at the moment is that this is the reason.

[kwankyu]

OK. The official size of sagemath/sagemath is 3.45GB.

The sage-binder-env image was using the ubuntu-focal-standard docker image, built in a workflow run like https://github.com/sagemath/sage/actions/runs/11304611125/job/31443198982 (the actual run log was removed)

On the other hand, from its dockerfile, it seems that the sagemath/sagemath image is a custom build close to the ubuntu-jammy-minimal docker image.

The "minimal" build minimizes install of system packages while "standard" build installs many system packages. I guess this made the size differences between the sage-binder-env image and the sagemath/sagemath image.

I just uploaded a patch for sage-binder-env to use the ubuntu-jammy-minimal docker image. Hopefully this will reduce its size close to the sagemath/sagemath image. Let's see.

[kwankyu]

Hmm. The current records:

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED             SIZE
ghcr.io/sagemath/sage-binder-env   10.4      c630becdab28   About an hour ago   4.68GB
sagemath/sagemath                  10.4      438901ea598d   3 months ago        3.45GB

[tobiasdiez]

Usually, people recommend to download the installer script and then execute it. This circumvents the issues with the slow pasting process. Eg. chocolatey recommends on https://chocolatey.org/install#individual:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

Does this change the registry?

I don't think so. Because of -Scope Process it only affects the current Pwsh session (https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.security/set-executionpolicy?view=powershell-7.4#example-6-set-the-execution-policy-for-the-current-powershell-session).

Concerning the docker size discussion: why do we use the binder image and not just the classical sagemath image?

[kwankyu]

Concerning the docker size discussion: why do we use the binder image and not just the classical sagemath image?

In this repo, that was a natural choice to me. If you will, you may duplicate the code into the sagemath/sage repo, using the sagemath image.

[soehms]

I don't think so. Because of -Scope Process it only affects the current Pwsh session

Yes! Sorry I missed that!

[soehms]

Concerning the docker size discussion: why do we use the binder image and not just the classical sagemath image?

In this repo, that was a natural choice to me. If you will, you may duplicate the code into the sagemath/sage repo, using the sagemath image.

There is an archived repo sagemath/sage-windows for the old Cygwin installers. Should we create something similar for the both new installation methods (the one here with WSL import and the one I created with Docker Desktop)? We could rename the old one to sage-windows-cygwin and use the original name.

The advantage of the approach here is that it leads more directly to a working Sage. But additional documentation of the installer with screenshots would be helpful in this case, too. I would keep such details out of the Sage repository and just reference them in the installation instructions.

As developers, we are no longer aware of how difficulties with a small technical step can cause frustration and make people give up. In addition, we should make clear how the software changes the users system (where possible) and show the steps to uninstall it. The more transparency we provide, the more people will try it.

As I said, the advantage of the Docker Desktop approach is the flexibility in deciding which Docker images to use. The choice of these images can easily be extended, for example to include images that provide certain optional packages, as already visible in the computop/sage repository. Maybe we can also add images from the ghcr.io registry.

So I think both methods should be available for Windows users.

[kwankyu]

That is a very good idea. I agree with all your points.

I will ask sagemath-admins for a renewed sagemath/sage-windows repo.

[kwankyu]

Meanwhile I got

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED          SIZE
ghcr.io/sagemath/sage-binder-env   10.4      97f7739f6ae4   35 minutes ago   4.28GB
sagemath/sagemath                  10.4      438901ea598d   3 months ago     3.45GB

However, we should use the smaller image for the renewed sagemath/sage-windows repo.

[kwankyu]

How to completely remove:

PS C:\Users\kwankyu> wsl --unregister SageMath-10.4
Unregistering.
The operation completed successfully.
PS C:\Users\kwankyu> rm -recurse .\AppData\Local\SageMath
PS C:\Users\kwankyu>

and delete the desktop shortcuts.

[kwankyu]

Now I got

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED         SIZE
ghcr.io/sagemath/sage-binder-env   10.4      2c8c9539b69a   8 minutes ago   3GB
sagemath/sagemath                  10.4      438901ea598d   3 months ago    3.45GB

and added uninstaller.

[soehms]

Now I got

$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED         SIZE
ghcr.io/sagemath/sage-binder-env   10.4      2c8c9539b69a   8 minutes ago   3GB
sagemath/sagemath                  10.4      438901ea598d   3 months ago    3.45GB

and added uninstaller.

Great! I'll take a look at it next week. I guess most of it is achieved by replacing ghcr.io/sagemath/sage/sage-ubuntu-jammy-minimal-with-system-packages:latest with ubuntu:jammy in the FROM statement. In docker/Dockerfile the starting layer run-time-dependencies adds some build tools to ubuntu:jammy:

# Sage needs the fortran libraries at run-time because we do not build gfortran
# with Sage but use the system's.
# We need gcc/g++ and libstdc++-10-dev to allow compilation of cython at run-time from the notebook.
# We also install sudo for the sage user, see below.
RUN apt-get -qq update \
    && apt-get -qq install -y --no-install-recommends gfortran gcc g++ libstdc++-10-dev sudo openssl \
    && apt-get -qq clean \
    && rm -r /var/lib/apt/lists/*

Don't you need that any more?

[kwankyu]

Basically both images aim to be a minimal build of sage that is just runnable. The sagemath image achieves that by building sage first and minimizing it while the sage-binder-env does it by taking an already built image and minimizing it.

Both use similar tactics for minimizing. I borrowed some tactics from docker/Dockerfile: removing artifacts like .c .cpp files produced by compiling .pyx files cut the last several hundred MBs; removing __pycache__ directories did another.

I think both images contain almost the same sage and the same base ubuntu image. A major difference is that the sage in the sagemath image is originally a non-editable install and the sage in the sage-binder-env is originally an editable install. Hence perhaps just some differences in organizing files at most.

At the moment, I regard the "Run SageMath on WSL" in this repo as a release of the sage-binder-env image for local use on Windows. For sagemath/sage-windows repo, you may duplicate the code but based on the sagemath image. I think we may keep both (at least for some time). The sagemath/sage-windows repo and the sagemath image have an "official" flavor :-)

I asked sagemath-admins to give the admin role of sagemath/sage-windows to you. I wonder who is the current admin/maintainer.

[kwankyu]

Ah I forgot to mention that the sage-binder-env uses the system install of jupyterlab, and hence the latest.

[tobiasdiez]

The advantage of the approach here is that it leads more directly to a working Sage. But additional documentation of the installer with screenshots would be helpful in this case, too. I would keep such details out of the Sage repository and just reference them in the installation instructions.

I don't have a strong opinion on this, but feeling wise I would favor it if the install instructions (including screenshots) are included in the main sage docs. Otherwise it looks like windows is a second-class citizen. Similarly, I think building the wsl image should be done on each release and thus is best included in the main repo as well. Maybe then the powershell script can also live there?

As developers, we are no longer aware of how difficulties with a small technical step can cause frustration and make people give up. In addition, we should make clear how the software changes the users system (where possible) and show the steps to uninstall it. The more transparency we provide, the more people will try it.

Totally agree!

As I said, the advantage of the Docker Desktop approach is the flexibility in deciding which Docker images to use. The choice of these images can easily be extended, for example to include images that provide certain optional packages, as already visible in the computop/sage repository. Maybe we can also add images from the ghcr.io registry.

I haven't looked at the scripts in detail, but can we merge both approaches? In the combined script, the user will first be asked if he wants to use WSL or Docker. I think it would also make sense to provide the option to install different WSL images in the future (the only requirement for this is of course that these images have to be made available somewhere first.)

[kwankyu]

... I would favor it if the install instructions (including screenshots) are included in the main sage docs. Otherwise it looks like windows is a second-class citizen. Similarly, I think building the wsl image should be done on each release and thus is best included in the main repo as well. Maybe then the powershell script can also live there?

Release of sagemath/sage is a release of source code. The source code does not work on Windows. In that sense, Windows is a second-class citizen. We don't need to hide the reality.

All downstream packagings of sage, even for linux and macOS, are done outside of sagemath/sage. For windows, sagemath/sage-windows would be an ideal place.

Things would be simpler technically when we separate concerns.

[soehms]

Similarly, I think building the wsl image should be done on each release and thus is best included in the main repo as well. Maybe then the powershell script can also live there?

Which place do you suggest? At the moment we don't have Powershell scripts in the source tree:

$ find . -name *.ps1
./local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/mesonbuild/scripts/cmd_or_ps.ps1
./local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/virtualenv/activation/powershell/activate.ps1

[soehms]

Release of sagemath/sage is a release of source code.

Furthermore, there is no real dependency on the version of sagemath/sage. In the Docker approach there is actually none and in the other case it is auto-generated.

[tobiasdiez]

... I would favor it if the install instructions (including screenshots) are included in the main sage docs. Otherwise it looks like windows is a second-class citizen. Similarly, I think building the wsl image should be done on each release and thus is best included in the main repo as well. Maybe then the powershell script can also live there?

Release of sagemath/sage is a release of source code.

That's not quite true. Every release of sage publishes over 50 linux and macos images at https://github.com/orgs/sagemath/packages, so are many binary wheels.

Similarly, I think building the wsl image should be done on each release and thus is best included in the main repo as well. Maybe then the powershell script can also live there?

Which place do you suggest? At the moment we don't have Powershell scripts in the source tree:

I suggest to build the wsl tar image at https://github.com/sagemath/sage/blob/develop/.github/workflows/docker_hub.yml (or in a new workflow that is triggered by it). Not sure if there is a canonical place for the powershell script but it can just be in the root (or under docker). Or it goes in it's own repo as you have suggested.

[soehms]

Both use similar tactics for minimizing. I borrowed some tactics from docker/Dockerfile: removing artifacts like .c .cpp files produced by compiling .pyx files cut the last several hundred MBs; removing __pycache__ directories did another.

It seems that .cpp file are not removed in sage-binder-env:

root@f6f3e8f91c12:/sage# ls -ltr src/sage/matrix/matrix_modn_dense_float*
-rw-r--r-- 1 root root    4939 Jul 20 05:02 src/sage/matrix/matrix_modn_dense_float.pyx
-rw-r--r-- 1 root root     241 Jul 20 05:02 src/sage/matrix/matrix_modn_dense_float.pxd

-rw-r--r-- 1 root root 1676043 Jul 20 05:28 src/sage/matrix/matrix_modn_dense_float.cpp

-rwxr-xr-x 1 root root  807400 Oct 26 13:56 src/sage/matrix/matrix_modn_dense_float.cpython-312-x86_64-linux-gnu.so

The sed command looks different in the Makefile (sed 's/\(.*\)[.]pyx$$/\1.c \1.cpp/') and more reasonable to me. The group instruction is missing in your version. Thus, you might save 100 MB in addition.

I think both images contain almost the same sage and the same base ubuntu image. A major difference is that the sage in the sagemath image is originally a non-editable install and the sage in the sage-binder-env is originally an editable install. Hence perhaps just some differences in organizing files at most.

There is one difference that I intended to address with my question "Don't you need that anymore?": In sagemath/sagemath you have a working gcc, gfortran ... in its runtime environment, which is not the case with ghcr.io/sagemath/sage-binder-env. But I don't know if this is still necessary? I don't have much experience with Jupyter notebooks. Is "to allow compilation of cython at run-time from the notebook" (as commented in the Dockerfile) still a point?

If I move the build tools from runtime-dependencies to build-time-dependencies,

sage$ git diff
diff --git a/docker/Dockerfile b/docker/Dockerfile
index a97664b2292..c67b2215816 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -90,7 +90,8 @@ RUN ln -s /usr/bin/sage /usr/bin/sagemath
 # We need gcc/g++ and libstdc++-10-dev to allow compilation of cython at run-time from the notebook.
 # We also install sudo for the sage user, see below.
 RUN apt-get -qq update \
-    && apt-get -qq install -y --no-install-recommends gfortran gcc g++ libstdc++-10-dev sudo openssl \
+    && apt-get -qq install -y --no-install-recommends sudo \
     && apt-get -qq clean \
     && rm -r /var/lib/apt/lists/*
 # Sage refuses to build as root, so we add a "sage" user that can sudo without a password.
@@ -111,7 +112,7 @@ WORKDIR $HOME
 FROM run-time-dependencies AS build-time-dependencies
 # Install the build time dependencies & git & rdfind
 RUN sudo apt-get -qq update \
-    && sudo apt-get -qq install -y wget build-essential automake m4 dpkg-dev python3 libssl-dev git rdfind \
+    && sudo apt-get -qq install -y wget build-essential automake m4 dpkg-dev python3 libssl-dev git rdfind gfortran gcc g++ libstdc++-10-dev openssl \
     && sudo apt-get -qq clean \
     && sudo rm -r /var/lib/apt/lists/*

the image size goes down to 3.18 GB

[kwankyu]

... The sed command looks different in the Makefile (sed 's/\(.*\)[.]pyx$$/\1.c \1.cpp/') and more reasonable to me. The group instruction is missing in your version. Thus, you might save 100 MB in addition.

You are right. Thanks. With the fix, the sage-binder-env image becomes 2.87 GB.

There is one difference that I intended to address with my question "Don't you need that anymore?": In sagemath/sagemath you have a working gcc, gfortran ... in its runtime environment, which is not the case with ghcr.io/sagemath/sage-binder-env.

Right..

But I don't know if this is still necessary? I don't have much experience with Jupyter notebooks. Is "to allow compilation of cython at run-time from the notebook" (as commented in the Dockerfile) still a point?

Actually I was not aware of the issue. I think the cython compilation functionality is a non-trivial feature of sage. To recover the functionality to the sage-binder-env image, I had to restore

FROM ghcr.io/sagemath/sage/sage-ubuntu-jammy-minimal-with-system-packages

Then the size of the image becomes 3.27 GB. For 400 MB, I would rather buy it :-)

[kwankyu]

Release of sagemath/sage is a release of source code.

Furthermore, there is no real dependency on the version of sagemath/sage. In the Docker approach there is actually none and in the other case it is auto-generated.

The sagemath/sage repo is a crowded place, containing too much information meaningful only to developers.

The sagemath/sage-windows will be relatively tranquil. We may put screenshots right into the README of the repo, immediately visible to the visitors.