Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finalize core22 transition (5.0-edge) #518

Conversation

simondeziel
Copy link
Member

@simondeziel simondeziel commented Jul 16, 2024

This PR contains all the commits I deemed appropriate to cherry-pick from latest-edge right before it moved from core22 to core24 (so at commit e6c25ef).

Here's what was not cherry-picked:

  • apparmor unconfined changes
  • uefi variable edition
  • edk2 commit dadf771 edk2: Cleanup build flags and switch to gcc5 as that caused issue with secure-boot cert injection
  • edk2 refreshed part definition/override-build section
  • CSM support
  • minio support
  • nasm man page patch refresh (since nasm version was not bumped)
  • nvme support
  • qemu change to Ubuntu/LP source
  • apparmor part removal
  • zfs-0.6, zfs-0.7 and zfs-2.0 part removal
  • parts versions updates
  • ceph removal from armhf
  • lxc-to-lxd removal
  • lxd-benchmark removal
  • lxd-migrate removal
  • nano part removal
  • vim part removal
  • shiftfs removal
  • upgrade-bridge script removal
  • XDG_* variable removal
  • anything from bots
  • LXD_OVMF_PATH replacement by LXD_QEMU_FW_PATH as stable-5.0 still uses the former
  • /var/lib/lxd/unix.socket handling
  • LXD-UI remains disabled
  • microceph plug
  • microovn plug
  • license change
  • LXD builtin documentation assets

I also tweaked the zfs-0-6 part to no longer go a git cherry-pick + full clone but instead use (another) .patch file.

@simondeziel simondeziel changed the title Finalize core22 transition take2 5.0 edge Finalize core22 transition take2 (5.0-edge) Jul 16, 2024
@simondeziel simondeziel force-pushed the finalize-core22-transition-take2-5.0-edge branch from 21aa874 to 3ac2415 Compare July 16, 2024 00:59
@simondeziel simondeziel marked this pull request as ready for review July 16, 2024 01:55
@simondeziel simondeziel requested a review from tomponline July 16, 2024 01:55
@simondeziel simondeziel changed the title Finalize core22 transition take2 (5.0-edge) Finalize core22 transition (5.0-edge) Jul 16, 2024
@tomponline
Copy link
Member

@simondeziel please can you update the PR description to help me understand the methodology you used here to select the commits - was it primarily just doing a diff between 5.21-edge and 5.0-edge and then looking for the associated commit?

What I'm trying to figure out is what to look for in terms of risk areas/large changes. Thanks

@simondeziel simondeziel marked this pull request as draft July 16, 2024 14:13
@simondeziel simondeziel force-pushed the finalize-core22-transition-take2-5.0-edge branch 4 times, most recently from bb70b57 to 2220cbf Compare July 17, 2024 19:10
@simondeziel
Copy link
Member Author

Locally built snap:

Generated snap metadata                                                                                                                  
'--enable-manifest' is deprecated, and will be removed in core24.                                                                        
Version has been set to '0+git.2220cbf4'                                                                                                 
Generated snap manifest                                                                                                                  
Lint warnings:                                                                                                                           
- library: libapparmor.so.1: unused library 'lib/libapparmor.so.1.17.1'. (https://snapcraft.io/docs/linters-library)                     
- library: libnvidia-container-go.so.1: unused library 'lib/libnvidia-container-go.so.1.14.1'. (https://snapcraft.io/docs/linters-library)                                                                                                                                         
- library: libatomic.so.1: unused library 'lib/x86_64-linux-gnu/libatomic.so.1.2.0'. (https://snapcraft.io/docs/linters-library)         
- library: libjson-glib-1.0.so.0: unused library 'lib/x86_64-linux-gnu/libjson-glib-1.0.so.0.600.6'. (https://snapcraft.io/docs/linters-library)                                                                                                                                   
- library: libusbredirhost.so.1: unused library 'lib/x86_64-linux-gnu/libusbredirhost.so.1.0.2'. (https://snapcraft.io/docs/linters-library)                                                                                                                                       
- library: libnvpair.so.1: unused library 'zfs-0.6/lib/libnvpair.so.1.0.1'. (https://snapcraft.io/docs/linters-library)                  
- library: libuutil.so.1: unused library 'zfs-0.6/lib/libuutil.so.1.0.1'. (https://snapcraft.io/docs/linters-library)                    
- library: libzfs.so.2: unused library 'zfs-0.6/lib/libzfs.so.2.0.0'. (https://snapcraft.io/docs/linters-library)                        
- library: libzfs_core.so.1: unused library 'zfs-0.6/lib/libzfs_core.so.1.0.0'. (https://snapcraft.io/docs/linters-library)              
- library: libzpool.so.2: unused library 'zfs-0.6/lib/libzpool.so.2.0.0'. (https://snapcraft.io/docs/linters-library)                    
- library: libnvpair.so.1: unused library 'zfs-0.8/lib/libnvpair.so.1.0.1'. (https://snapcraft.io/docs/linters-library)                  
- library: libuutil.so.1: unused library 'zfs-0.8/lib/libuutil.so.1.0.1'. (https://snapcraft.io/docs/linters-library)                    
- library: libzfs.so.2: unused library 'zfs-0.8/lib/libzfs.so.2.0.0'. (https://snapcraft.io/docs/linters-library)                        
- library: libzfs_core.so.1: unused library 'zfs-0.8/lib/libzfs_core.so.1.0.0'. (https://snapcraft.io/docs/linters-library)              
- library: libzpool.so.2: unused library 'zfs-0.8/lib/libzpool.so.2.0.0'. (https://snapcraft.io/docs/linters-library)                    
- library: libzfsbootenv.so.1: unused library 'zfs-2.0/lib/libzfsbootenv.so.1.0.0'. (https://snapcraft.io/docs/linters-library)          
- library: libzpool.so.4: unused library 'zfs-2.0/lib/libzpool.so.4.0.0'. (https://snapcraft.io/docs/linters-library)                    
- library: libnvpair.so.3: unused library 'zfs-2.1/lib/libnvpair.so.3.0.0'. (https://snapcraft.io/docs/linters-library)                  
- library: libuutil.so.3: unused library 'zfs-2.1/lib/libuutil.so.3.0.0'. (https://snapcraft.io/docs/linters-library)                    
- library: libzfs.so.4: unused library 'zfs-2.1/lib/libzfs.so.4.1.0'. (https://snapcraft.io/docs/linters-library)                        
- library: libzfs_core.so.3: unused library 'zfs-2.1/lib/libzfs_core.so.3.0.0'. (https://snapcraft.io/docs/linters-library)              
- library: libzfsbootenv.so.1: unused library 'zfs-2.1/lib/libzfsbootenv.so.1.0.0'. (https://snapcraft.io/docs/linters-library)          
- library: libzpool.so.5: unused library 'zfs-2.1/lib/libzpool.so.5.0.0'. (https://snapcraft.io/docs/linters-library)                    
- library: libnvpair.so.3: unused library 'zfs-2.2/lib/libnvpair.so.3.0.0'. (https://snapcraft.io/docs/linters-library)                  
- library: libuutil.so.3: unused library 'zfs-2.2/lib/libuutil.so.3.0.0'. (https://snapcraft.io/docs/linters-library)                    
- library: libzfs.so.4: unused library 'zfs-2.2/lib/libzfs.so.4.1.0'. (https://snapcraft.io/docs/linters-library)                        
- library: libzfs_core.so.3: unused library 'zfs-2.2/lib/libzfs_core.so.3.0.0'. (https://snapcraft.io/docs/linters-library)              
- library: libzfsbootenv.so.1: unused library 'zfs-2.2/lib/libzfsbootenv.so.1.0.0'. (https://snapcraft.io/docs/linters-library)          
- library: libzpool.so.5: unused library 'zfs-2.2/lib/libzpool.so.5.0.0'. (https://snapcraft.io/docs/linters-library)                    
Version has been set to '0+git.2220cbf4'                                                                                                 
Created snap package lxd_0+git.2220cbf4_amd64.snap 

@simondeziel
Copy link
Member Author

This snap was tested in local VMs using 22.04 and 24.04. A subset of the lxd-ci suite was ran in each. Minimal manual tests were also done in a 20.04 VM.

@simondeziel simondeziel marked this pull request as ready for review July 18, 2024 00:39
@tomponline
Copy link
Member

ceph removal from armhf

@simondeziel i dont think we can do this, this was supported in 5.0 and is still available in core22 right?

@tomponline
Copy link
Member

lxc-to-lxd removal
lxd-benchmark removal
lxd-migrate removal

These need to stay in - we are just changing the base and bringing in updates, not taking away functionality.

@tomponline
Copy link
Member

nano part removal
vim part removal
shiftfs removal
upgrade-bridge script removal

These need to stay in, as it is a late LTS point release so support/behaviours should remain as stable as possible.

@simondeziel
Copy link
Member Author

simondeziel commented Jul 18, 2024

@tomponline yep, was easy to miss but the list in the PR description is for thing I did not cherry-pick :)

@tomponline
Copy link
Member

Apologies I misread that you had not cherry-picked that list :)

Copy link
Member

@tomponline tomponline left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you add the docs to the snap because of the UI support?

snapcraft.yaml Outdated Show resolved Hide resolved
snapcraft.yaml Outdated Show resolved Hide resolved
snapcraft.yaml Outdated Show resolved Hide resolved
snapcraft.yaml Outdated Show resolved Hide resolved
lxd-migrate/lxd.go Outdated Show resolved Hide resolved
snapcraft.yaml Outdated Show resolved Hide resolved
simondeziel and others added 16 commits September 17, 2024 19:56
Signed-off-by: Simon Deziel <[email protected]>
(cherry picked from commit cda2184)
Signed-off-by: Max Asnaashari <[email protected]>
(cherry picked from commit dd708ad)
Signed-off-by: Simon Deziel <[email protected]>
Signed-off-by: Simon Deziel <[email protected]>
(cherry picked from commit 7b8f29e)
…files

Those files are not world-readable and LXD should already have done the proper
changes to have `unshare` work anyway.

Signed-off-by: Simon Deziel <[email protected]>
(cherry picked from commit be91e34)
…nfined files

Those files are not world-readable and LXD should already have done the proper
changes to have `unshare` work anyway.

Signed-off-by: Simon Deziel <[email protected]>
(cherry picked from commit e387b3a)
It's a fix for a build failure on s390:
Installing build-packages
Cannot find package listed in 'build-packages': acpica-tools
Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240621-073250.087743.log'
Build failed

Reason is that, in Ubuntu 24.04 acpica-tools package is not available for s390:
https://packages.ubuntu.com/noble/acpica-tools
compare with:
https://packages.ubuntu.com/jammy/acpica-tools

Let's workaround this by preventing build packages installation
for any architectures other than amd64/arm64. It's safe, because
we only build edk2 for these two architectures.

Signed-off-by: Alexander Mikhalitsyn <[email protected]>
(cherry picked from commit a0e011e)
Signed-off-by: Simon Deziel <[email protected]>
snapcraft syntax a little bit unobvious, and it turned out that:
- on amd64,arm64:
means AND and not OR. Which is obviously wrong. Let's workaround this.

Fixes: a0e011e ("snapcraft: install edk2 build dependencies only when needed")

Signed-off-by: Alexander Mikhalitsyn <[email protected]>
(cherry picked from commit b57a131)
Signed-off-by: Simon Deziel <[email protected]>
Only for core24.

Signed-off-by: Alexander Mikhalitsyn <[email protected]>
(cherry picked from commit 5910450)
Signed-off-by: Simon Deziel <[email protected]>
Signed-off-by: Simon Deziel <[email protected]>
(cherry picked from commit 526ed11)
Signed-off-by: Simon Deziel <[email protected]>
Signed-off-by: Thomas Parrott <[email protected]>
(cherry picked from commit 30a24f3)
Signed-off-by: Simon Deziel <[email protected]>
snapcraft deduplication is not applied to files from packages listed in
stage-packages.  To avoid duplicating lz4's .so, make it explicit that we don't
want to prime it and instead relying on the one provided in the base snap.

Signed-off-by: Simon Deziel <[email protected]>
(cherry picked from commit 84c3f5c)
Let's ship newer version of AppArmor with support of new features.
Only needed for core22. But we will use with core24 based series for consistency.

Also bumped apparmor to v4.0.2.

Signed-off-by: Alexander Mikhalitsyn <[email protected]>
(cherry picked from commit 8591ed4)
Signed-off-by: Thomas Parrott <[email protected]>
Signed-off-by: Simon Deziel <[email protected]>
@simondeziel simondeziel force-pushed the finalize-core22-transition-take2-5.0-edge branch from 0f0540b to 25ec456 Compare September 17, 2024 23:58
@simondeziel simondeziel marked this pull request as draft September 18, 2024 04:20
@simondeziel
Copy link
Member Author

The nvidia-container part fails to build:

2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.001 :: + '[' x86_64 '!=' x86_64 ']'
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.001 :: + set -ex
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.001 :: + git config user.email [email protected]
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.003 :: + git config user.name 'LXD snap builder'
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.004 :: + patch -p1
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.009 :: patching file Makefile
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.010 :: Hunk #2 succeeded at 165 (offset -3 lines).
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.010 :: + patch -p1
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.011 :: patching file src/driver.c
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.011 :: Hunk #1 succeeded at 123 (offset 1 line).
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.011 :: + mkdir -p /root/parts/nvidia-container/install/bin/
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.014 :: + cp /root/project/snapcraft/wrappers/nvidia-container-cli /root/parts/nvidia-container/install/bin/
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.028 :: + set +ex
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.600 Executing PosixPath('/root/parts/nvidia-container/run/build.sh')
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.602 :: + make -j6
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.630 :: Makefile:106: *** Invalid major version.  Stop.
2024-09-18 01:13:24.222 :: 2024-09-18 01:13:23.631 :: error:

Signed-off-by: Alexander Mikhalitsyn <[email protected]>
(cherry picked from commit e6c25ef)
@simondeziel
Copy link
Member Author

The nvidia-container build issue seems to be because upstream had a problem with their release tagging. Both v1.16.1 and v1.16.0 point to the exact same commit NVIDIA/libnvidia-container@4c2494f

Using `source-commit` causes the build to fail:

```
Executing PosixPath('/root/parts/nvidia-container/run/build.sh')
:: + make -j6
:: Makefile:106: *** Invalid major version.  Stop.
```

Using `source-tag` makes it misteriously work. This oddity was reproduced with
v1.14.6, v1.16.0 and v1.16.1

Signed-off-by: Simon Deziel <[email protected]>
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:base",
"schedule:earlyMondays"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomponline
Copy link
Member

@simondeziel does this include the fix for the snap preseed config?

@tomponline
Copy link
Member

@simondeziel is this ready to merge?

@simondeziel
Copy link
Member Author

@simondeziel does this include the fix for the snap preseed config?

I did not include the breaking commit nor the one that reverted it. Bottom-line is the preseed/FIRSTRUN detection should work.

@simondeziel is this ready to merge?

I finally got around to testing the resulting snap:

root@v2:~# lxd init --auto
root@v2:~# lxc launch images:alpine/edge a1
Creating a1
Starting a1                               
root@v2:~# lxc launch images:alpine/edge av1 --vm -c security.secureboot=false
Creating av1
Starting av1                                
root@v2:~# lxc ls
+------+---------+----------------------+-----------------------------------------------+-----------------+-----------+
| NAME |  STATE  |         IPV4         |                     IPV6                      |      TYPE       | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+-----------------+-----------+
| a1   | RUNNING | 10.172.22.219 (eth0) | fd42:7205:c1dc:a1e7:216:3eff:fe71:eafe (eth0) | CONTAINER       | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------------+-----------+
| av1  | RUNNING | 10.172.22.26 (eth0)  | fd42:7205:c1dc:a1e7:216:3eff:fe8a:c961 (eth0) | VIRTUAL-MACHINE | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------------+-----------+
root@v2:~# ll /snap/lxd/
total 8
drwxr-xr-x  4 root root 4096 Sep 20 20:41 ./
drwxr-xr-x  6 root root 4096 Sep 20 20:40 ../
drwxr-xr-x 21 root root  291 Sep 17 15:19 30317/
lrwxrwxrwx  1 root root    2 Sep 20 20:41 current -> x1/
drwxr-xr-x 19 root root  267 Sep 18 21:21 x1/

@simondeziel simondeziel marked this pull request as ready for review September 20, 2024 20:55
@tomponline tomponline merged commit 909ac01 into canonical:5.0-edge Sep 23, 2024
6 checks passed
@simondeziel simondeziel deleted the finalize-core22-transition-take2-5.0-edge branch September 23, 2024 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants