Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nexusmods-app: 0.4.1 -> 0.6.1 #331150

Merged
merged 2 commits into from
Sep 29, 2024
Merged

Conversation

MattSturgeon
Copy link
Contributor

@MattSturgeon MattSturgeon commented Jul 30, 2024

Notes for end-users

Caution

Before upgrading to a new version of nexusmods-app you will need a fresh start.
If you have configuration leftover from an old version, the updated app will crash.

Tip

You can reset all modded games and wipe your nexusmods-app config using:

NexusMods.App uninstall-app
  • This should be done before updating the app.
  • Be careful: There's no way to recover a wiped config!

Tip

You may also need to remove old desktop entries to avoid other issues:

rm ~/.local/share/applications/com.nexusmods.app.desktop

Description of changes

This PR updates to the latest version, 0.6.1.

This is WIP and still has a few issues to resolve (help needed): Most issues are now resolved, just needs some cleanup and polish:

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

cc @l0b0 @corngood @NickCao @GGG-KILLER @erri120

Closes #318647
Closes #329592
Related #321405
Related #317852


Add a 👍 reaction to pull requests you find important.

pkgs/by-name/ne/nexusmods-app/deps.nix Outdated Show resolved Hide resolved
pkgs/by-name/ne/nexusmods-app/package.nix Outdated Show resolved Hide resolved
pkgs/by-name/ne/nexusmods-app/update.bash Outdated Show resolved Hide resolved
pkgs/by-name/ne/nexusmods-app/package.nix Outdated Show resolved Hide resolved
pkgs/by-name/ne/nexusmods-app/package.nix Outdated Show resolved Hide resolved
pkgs/by-name/ne/nexusmods-app/package.nix Outdated Show resolved Hide resolved
pkgs/by-name/ne/nexusmods-app/package.nix Outdated Show resolved Hide resolved
pkgs/by-name/ne/nexusmods-app/package.nix Outdated Show resolved Hide resolved
@MattSturgeon
Copy link
Contributor Author

MattSturgeon commented Jul 31, 2024

The tests pass if I keep the patch that replaces the bundled 7zz with ours.

This implies that maybe NEXUSMODS_APP_USE_SYSTEM_EXTRACTOR isn't being checked by the tests and they're still looking for the bundled 7zz instead of the one on the PATH.

Either that, or I'm not setting the NEXUSMODS_APP_USE_SYSTEM_EXTRACTOR build constant correctly.

Or perhaps the wrapper PATH isn't used when running the tests?

I still can't start the GUI, but the CLI commands seem to work. --help at least prints some output.

I haven't looked into patching/copying the .desktop files yet.

@MattSturgeon
Copy link
Contributor Author

MattSturgeon commented Aug 21, 2024

I'm able to build, and I can run CLI commands such as --help. However I cannot launch the GUI.

I get no errors on stdout/stderr, however logs are printed to ~/.local/state/NexusMods.App/Logs/nexusmods.app.main.current.log:

00:00:00.651 [FATAL] (NexusMods.App.Startup) Exception crashed the application!|Avalonia.Markup.Xaml.XamlLoadException:
  No precompiled XAML found for NexusMods.App.App, make sure to specify x:Class and include your XAML file as AvaloniaResource
   at Avalonia.Markup.Xaml.AvaloniaXamlLoader.Load(Object obj)
   at NexusMods.App.App.Initialize() in /build/source/src/NexusMods.App/App.axaml.cs:line 37
   at Avalonia.AppBuilder.SetupUnsafe()
   at Avalonia.AppBuilder.Setup()
   at Avalonia.AppBuilder.SetupWithLifetime(IApplicationLifetime lifetime)
   at Avalonia.ClassicDesktopStyleApplicationLifetimeExtensions.StartWithClassicDesktopLifetime(AppBuilder builder, String[] args, Action`1 lifetimeBuilder)
   at NexusMods.App.Startup.Main(IServiceProvider provider, String[] args) in /build/source/src/NexusMods.App/Startup.cs:line 56
00:00:00.652 [DEBUG] (Microsoft.Extensions.Hosting.Internal.Host) Hosting stopping
00:00:00.652 [INFO] (Microsoft.Hosting.Lifetime) Application is shutting down...
00:00:00.654 [INFO] (NexusMods.SingleProcess.CliServer) TCP listener was cancelled, stopping
00:00:00.657 [DEBUG] (Microsoft.Extensions.Hosting.Internal.Host) Hosting stopped

Full log: nexusmods.app.main.current.log

There's other errors in the log relating to XDG stuff, however the big issue seems to be GUI resources being missing:

No precompiled XAML found for NexusMods.App.App, make sure to specify x:Class and include your XAML file as AvaloniaResource

I guess it could relate to not using the full sln here, however 0.4.1 seems to work fine even with that change...

# If the whole solution is published, there seems to be a race condition where
# it will sometimes publish the wrong version of a dependent assembly, for
# example: Microsoft.Extensions.Hosting.dll 6.0.0 instead of 8.0.0.
# https://learn.microsoft.com/en-us/dotnet/core/compatibility/sdk/7.0/solution-level-output-no-longer-valid
# TODO: do something about this in buildDotnetModule
projectFile = "src/NexusMods.App/NexusMods.App.csproj";
testProjectFile = "NexusMods.App.sln";

cc @corngood @erri120 @Sewer56

@Sewer56
Copy link

Sewer56 commented Aug 21, 2024

I got some time to kill. I'm not sure about the rest yet, but I can at least try fix up NEXUSMODS_APP_USE_SYSTEM_EXTRACTOR being borked in upstream.

@Sewer56
Copy link

Sewer56 commented Aug 21, 2024

Nexus-Mods/NexusMods.App#1919

This should fix the 7z issue upstream.
As for the rest, I'd have to look. It's very strange; not something I've experienced before.

[
# From https://github.com/Nexus-Mods/NexusMods.App/blob/v0.5.3/src/NexusMods.App/app.pupnet.conf#L38
"--property:Version=${version}"
"--property:TieredCompilation=true"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll also want the ReadyToRun flag here; it improves startup times, especially in something as JIT heavy at startup as NMA.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We removed ReadyToRun because we didn't have crossgen2 support in nix before, I believe after the changes by @corngood that might've been solved though.

Copy link
Contributor Author

@MattSturgeon MattSturgeon Aug 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 0205410. I'll leave this thread unresolved until we can identify if it a) works with dotnet build (I can only find it documented for dotnet publish) and b) doesn't introduce issues related to crossgen2.

See #233422

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note @GGG-KILLER previously advised against this (#318647 (comment)):

This makes it so that the code is pre-compiled to the target runtime, given that we don't add the framework's RID unless it's a self-contained build, I don't recommend setting this.

...but the nexus folks claim it has some performance benefits.

I'm not sure what is meant by "framework's RID" so I'm not in a position to make an educated decision.

@Sewer56
Copy link

Sewer56 commented Aug 21, 2024

I think, per your own comment Nexus-Mods/NexusMods.App#1866 (comment) adding ICU might be recommended here; if it's not transitively available already.

@Sewer56
Copy link

Sewer56 commented Aug 21, 2024

There's other errors in the log relating to XDG stuff, however the big issue seems to be GUI resources being missing:

No precompiled XAML found for NexusMods.App.App, make sure to specify x:Class and include your XAML file as AvaloniaResource

I guess it could relate to not using the full sln here, however 0.4.1 seems to work fine even with that change...

As for the cause of the issue; I'm not really sure myself.

Normally this sort of thing can happen if you build an Avalonia .NET package with trimming (Or NativeAOT, which requires trimming). If some assembly isn't correctly marked for trimming, or excluded from the 'trimmer', this sort of thing can happen.
However, I'm pretty sure no trimming is going on here; instead some build step from Avalonia might not just be kicking in.

I think instead of looking at runtime errors, it may be better to grab a log of the build time, i.e. when the following code gets executed

. Maybe there's something telling in the build log.

dotnetBuildFlags =
let
# From https://nexus-mods.github.io/NexusMods.App/developers/Contributing/#for-package-maintainers
# TODO: add DefineConstants support to buildDotnetModule
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel like we need to specially add support for this when we already have a general build flag input.

From my experience it's rare to see any programs using DefineConstant to configure how they're built, usually they use MSBuild properties which is the idiomatic way to enable and disable things in the .NET world.

Copy link
Contributor Author

@MattSturgeon MattSturgeon Aug 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

usually they use MSBuild properties which is the idiomatic way to enable and disable things in the .NET world.

The upstream project is still in a somewhat "experimental" state, so perhaps they'd be willing to migrate to a more idiomatic method then.

cc @Sewer56 @erri120

I'm somewhat unfamiliar with dotnet, so this is a useful insight. I'll have to go read some docs 😁

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're trying to avoid build constants in the actual App as much as possible.
They're only really being used when it's desirable for package maintainers to have slightly different default runtime behaviour. Everything else is configurable at runtime.

For instance, for AppImage & Package Manager builds we add an extra message telling users to update using the System, rather than via the App. For manual installs we don't show update dialogs on the other hand.

It is possible to assign a MSBuild property other than DefineConstants here, or something like AdditionalDefineConstants that would just append to the existing constants. However at the end of the day you'll always be making some minor change. Be it adding a few characters to the build command, dropping a file, or replacing an enum value in the source to set some default behaviour at compile time.

Copy link
Contributor

@GGG-KILLER GGG-KILLER Aug 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant that, usually, what projects do is have some msbuild property that then defines the constant, something like this in the .csproj:

  <PropertyGroup Condition="$(UseSystemExtractor) == '1'">
    <DefineConstants>$(DefineConstants);NEXUSMODS_APP_USE_SYSTEM_EXTRACTOR</DefineConstants>
  </PropertyGroup>

and then when building you'd do dotnet build -c Release -p:UseSystemExtractor=1.

So I don't know if it's worth adding DefineConstants support in buildDotnetModulefor something that only one app will use most likely

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I've done it myself this way before too. I'd be happy to provide a constant like this upstream if the others there wouldn't mind.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's a need to change this now unless if there's an actual need for it, as it's been done like this already it'll just generate CI churn and problems for other packagers.

I'm just speaking on behalf of the .NET team in NixOS with relation to adding a builtin option for doing this, there's nothing inherently wrong with the way this has been done upstream, it's just not conventional so I think there's other things we can direct our effort towards.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's a need to change this now unless if there's an actual need for it, as it's been done like this already it'll just generate CI churn and problems for other packagers.

I think it's still early enough in nexusmods-app's life that it shouldn't really be an issue for other downstream packagers.

Some of the issues @Sewer56 encountered while investigating Nexus-Mods/NexusMods.App#1919 sound like they stemmed from us explicitly passing DefineConstants on the CLI. Long-term it may be better to have custom property groups, if so.

It's certainly easier to pass many "-p:{}" flags instead of joining a list of constants together using "%3B" (escaped ";")...

But I don't want to over-blow the issue; we can work with whichever implementation upstream prefers 😀

[
# From https://github.com/Nexus-Mods/NexusMods.App/blob/v0.5.3/src/NexusMods.App/app.pupnet.conf#L38
"--property:Version=${version}"
"--property:TieredCompilation=true"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We removed ReadyToRun because we didn't have crossgen2 support in nix before, I believe after the changes by @corngood that might've been solved though.

@GGG-KILLER
Copy link
Contributor

GGG-KILLER commented Aug 22, 2024

@Sewer56 I'm looking at the build logs currently, and it seems like it's processing the XAML files:

/build/source/src/NexusMods.App.UI/Controls/Spine/Buttons/Download/SpineDownloadButtonView.axaml.cs(18,13): warning CS0618: 'ReactiveUiExtensions.BindToUi<TValue, TTarget, TTValue>(IObservable<TValue>, TTarget?, Expression<Func<TTarget, TTValue?>>, object?, IBindingTypeConverter?)' is obsolete: 'This should not be used anymore. See UI Coding Conventions for more details.' [/build/source/src/NexusMods.App.UI/NexusMods.App.UI.csproj]
/build/source/src/NexusMods.App.UI/Controls/Spine/Buttons/Download/SpineDownloadButtonView.axaml.cs(31,13): warning CS0618: 'ReactiveUiExtensions.BindToUi<TValue, TTarget, TTValue>(IObservable<TValue>, TTarget?, Expression<Func<TTarget, TTValue?>>, object?, IBindingTypeConverter?)' is obsolete: 'This should not be used anymore. See UI Coding Conventions for more details.' [/build/source/src/NexusMods.App.UI/NexusMods.App.UI.csproj]
/build/source/src/NexusMods.App.UI/Controls/Spine/Buttons/Download/SpineDownloadButtonView.axaml.cs(37,13): warning CS0618: 'ReactiveUiExtensions.BindToUi<TValue, TTarget, TTValue>(IObservable<TValue>, TTarget?, Expression<Func<TTarget, TTValue?>>, object?, IBindingTypeConverter?)' is obsolete: 'This should not be used anymore. See UI Coding Conventions for more details.' [/build/source/src/NexusMods.App.UI/NexusMods.App.UI.csproj]
/build/source/src/NexusMods.App.UI/Controls/Spine/Buttons/Download/SpineDownloadButtonView.axaml.cs(42,13): warning CS0618: 'ReactiveUiExtensions.BindToUi<TValue, TTarget, TTValue>(IObservable<TValue>, TTarget?, Expression<Func<TTarget, TTValue?>>, object?, IBindingTypeConverter?)' is obsolete: 'This should not be used anymore. See UI Coding Conventions for more details.' [/build/source/src/NexusMods.App.UI/NexusMods.App.UI.csproj]

I've also found the following line in the build logs but it seems unrelated:

  dotnet "/nix/store/2919w4fqxh6yr201p30z2q2y12g4iliv-nexusmods-app-0.5.3-nuget-deps/share/nuget/packages/flatsharp.compiler/7.6.0/tools/net8.0/FlatSharp.Compiler.dll" --input "Serializers/DiskStateTreeSchema.fbs" --includes "" --output obj/Release/net8.0/ --nullable-warnings true

This is the full log if anyone would like to take a look at it: https://gist.github.com/GGG-KILLER/1cd7370246aa02b6bc27eefc694543d0

I also don't get any errors when running the program but the UI still doesn't show up:
image


runtimeInputs = [
_7zz
desktop-file-utils
];

runtimeDeps = [
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replying to #331150 (comment) by @Sewer56:

I think, per your own comment Nexus-Mods/NexusMods.App#1866 (comment) adding ICU might be recommended here; if it's not transitively available already.

It looks like ICU is made available at runtime on the LD_LIBRARY_PATH by buildDotnetModule:

makeWrapperArgs = args.makeWrapperArgs or [ ] ++ [
"--prefix"
"LD_LIBRARY_PATH"
":"
"${dotnet-sdk.icu}/lib"
];

Doesn't look like it's made available at build time though (e.g. via buildInputs or nativeBuildInputs), but that might be done elsewhere 🤔

@MattSturgeon
Copy link
Contributor Author

MattSturgeon commented Sep 13, 2024

For users updating there's a few things that need to be done:

  • Reset all modded games to a vanilla state
    • e.g. wipe steam library and reinstall
    • or "purge" in the pre-upgrade version of nexusmods-app
    • or NexusMods.App uninstall-app (before updating)
  • Wipe app config
    • e.g. NexusMods.App uninstall-app
    • or rm -rf ~/.local/{share,state}/NexusMods.App
  • Remove any dynamically installed desktop entries
    • e.g. rm ~/.local/share/applications/com.nexusmods.app.desktop
    • NexusMods.App uninstall-app does not do this 🙁
Details

I tested whether uninstall-app will remove dynamically installed desktop entries:

$ touch ~/.local/share/applications/com.nexusmods.app.desktop

$ NexusMods.App uninstall-app
Deleted file: /home/matt/.local/state/NexusMods.App/Logs/nexusmods.app.main.current.logDeleted file: /home/matt/.local/state/NexusMods.App/Logs/nexusmods.app.slim.current.logDeleted directory: /home/matt/.local/share/NexusMods.App/DataModel/MnemonicDB.rocksdbDeleted directory: /home/matt/.local/state/NexusMods.App/LogsDeleted directory: /home/matt/.local/share/NexusMods.App/ConfigsDeleted directory: /home/matt/.local/share/NexusMods.App/DataModelApplication uninstalled successfully⏎

$ ls ~/.local/share/applications/com.nexusmods.app.desktop
/home/matt/.local/share/applications/com.nexusmods.app.desktop

Its unfortunate that this all needs to be done, and it's made more awkward by the fact that most users won't know the app will get updated when they bump their system config, let alone that these steps are required.

Note

The help text for uninstall-app is: Uninstall the application and revert games to their original state
I believe it'd have to be run using the app version that generated the app config.


I think we need to accept that the app is still "unstable" and new versions have breaking changes. 😓

Does this mean we should be retaining old versions we've packaged to allow users to pin a version they use (pkgs.nexusmods-app-0.4.1 & pkgs.nexusmods-app-0.6.0)? Or should we simply ask users to deal with all this themselves, until things stabilize upstream? Perhaps add some user-guides to the nixos wiki...

@GGG-KILLER
Copy link
Contributor

Does this mean we should be retaining old versions we've packaged to allow users to pin a version they use (pkgs.nexusmods-app-0.4.1 & pkgs.nexusmods-app-0.6.0)? Or should we simply ask users to deal with all this themselves, until things stabilize upstream? Perhaps add some user-guides to the nixos wiki...

Having fixed versions is unnecessary imo, if people want to fix versions they can override the package to use older versions or "vendor" the older version into their config.

@MattSturgeon
Copy link
Contributor Author

Otherwise the best option is to just document that these manual migration steps are necessary as upstream haven't added anything to handle these version migrations on their own.

Where should this kinda thing be documented? Is meta.longDescription appropriate for this? Presumably wiki articles are also a good place for less formal documentation 🤔

@corngood
Copy link
Contributor

corngood commented Sep 17, 2024

Apologies for conflicting this. This is what I was thinking about how to deal with it:

#336824 (comment)

@GGG-KILLER
Copy link
Contributor

GGG-KILLER commented Sep 17, 2024

Where should this kinda thing be documented? Is meta.longDescription appropriate for this? Presumably wiki articles are also a good place for less formal documentation 🤔

I honestly have no idea, maybe someone else with more knowledge can tell you.

I'd go with the NixOS Wiki personally because I don't usually read longDescription (since it's folded by default in search.nixos.org), it's easier to find when google searching and we might end up with migration steps for every single version if these aren't specific to only this version.

@MattSturgeon
Copy link
Contributor Author

MattSturgeon commented Sep 24, 2024

Given the app is still unstable/experimental, would it be reasonable to add a lib.warn to the package eval, informing users of some caveats, quoting or referencing https://nexus-mods.github.io/NexusMods.App/users/faq/#why-do-i-have-to-uninstall-everything-to-update-the-app and https://nexus-mods.github.io/NexusMods.App/users/Uninstall/ ?

EDIT: I've opened #344251 to tackle this.

@corngood
Copy link
Contributor

@MattSturgeon you've done a lot of work on this PR, and I want to make sure it's not being held up waiting on feedback/review. Just let me know if you feel like it's ready, or if you'd like help with anything.

@MattSturgeon MattSturgeon marked this pull request as ready for review September 24, 2024 19:55
MattSturgeon added a commit to MattSturgeon/nixpkgs that referenced this pull request Sep 25, 2024
Define new constants:
- `NEXUSMODS_APP_USE_SYSTEM_EXTRACTOR`
- `INSTALLATION_METHOD_PACKAGE_MANAGER`

Also:
- Stop setting `APPIMAGE`
  We now have `INSTALLATION_METHOD_PACKAGE_MANAGER`
- Set `ReadyToRun`
  See NixOS#331150 (comment)
- Install desktop entry, appstream, & icons
  See https://nexus-mods.github.io/NexusMods.App/developers/Contributing/#for-package-maintainers
Define new constants:
- `NEXUSMODS_APP_USE_SYSTEM_EXTRACTOR`
- `INSTALLATION_METHOD_PACKAGE_MANAGER`

Also:
- Stop setting `APPIMAGE`
  We now have `INSTALLATION_METHOD_PACKAGE_MANAGER`
- Set `ReadyToRun`
  See NixOS#331150 (comment)
- Install desktop entry, appstream, & icons
  See https://nexus-mods.github.io/NexusMods.App/developers/Contributing/#for-package-maintainers
@MattSturgeon MattSturgeon changed the title nexusmods-app: 0.4.1 -> 0.6.0 nexusmods-app: 0.4.1 -> 0.6.1 Sep 25, 2024
@corngood
Copy link
Contributor

@SuperSandro2000 this seems to be blocked on your previous change request, but I think it's all been addressed.

@SuperSandro2000 SuperSandro2000 merged commit 6eaaa1d into NixOS:master Sep 29, 2024
37 checks passed
@MattSturgeon MattSturgeon deleted the nexus-0.5.3 branch September 29, 2024 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants