Distribution on flathub #1267

Ryex · 2022-11-29T07:38:54Z

Ryex
Nov 29, 2022

Hello again.

I've recently been looking into making a flatpak package. I've for the most part succeeded in that it can build and run successfully but there are some intergration kinks to work out, mainly the generation of a proper metainfo.xml and the appid standards for flatpaks (appid=io.novelwriter.novelWriter in flatpak land).

The main goal of having a flatpak build would be distribution on flathub (or another repo/"appstore")

the Flathub application submission process would be be completely outside the current setup.py/pypi/github workflow even if a working flatpak manifest was provided in this repo just due to how it works. details on that can be found here
https://github.com/flathub/flathub/wiki/App-Submission

would flathub submission and maintenance be something you would be willing to undertake assuming the initial setup was taken care of?

In theory, the flathup distribution can be done as a third party (ie I, or another could be a maintainer) but I think it would be preferable if it was under the auspices of the core project.

vkbo · 2022-11-29T10:14:15Z

vkbo
Nov 29, 2022
Maintainer

I've looked into Flatpak and other similar services in the past. I'm not opposed to it, I'm just concerned about an ever increasing list of things to do during a release. I would like to limit the options per platform to a handful of manageable packages.

Currently there is:

PyPi for all platforms for people who are primarily command line users (easy to maintain).
Launchpad + deb package (built together) for Ubuntu, Debian and Mint (usually easy to do, although sometimes fail)
AppImage for other Linux distros (I do this in a Ubuntu VM, so includes additional effort, but is ok)
Minimal packages for Windows, Linux and Mac (trivial as they're only zip files)
Installer for Windows (needs a VM, but is now built from the minimal package, so not too much effort)

As you can see, Mac is the one clearly lacking, where the only options are the zip file or PyPi.

Linux seems to be well covered though, so I'm wondering what Flatpak would provide, or alternatively, replace. I'm anyway going to maintain Launchpad/.deb as it is the environment I use myself.

4 replies

Ryex Nov 29, 2022
Author

The main advantage of Flatpak would be the same as an Appimage, it runs on every linux distro and contains all it's own dependencies. The advantages are that Flatpak are much more widely supported and used, the app is sandboxed, and the app would automatically appear in all the various linux app-stores (ie. KDE's Discover and GNOME's Software etc., most of them now support flathub on almost every distro). Final advantage is it becomes really easy to update the app. There are additionally some linux distros that only use flatpaks, I personally don;t use them but I know they exist.

All that said, your correct, it's additional burden when there are existing solutions.

Is there any particular reason your not using github workflows/runners to build and upload some or all of your release artifacts? I know it's a bit of work to get it all setup but it would completely remove the need to run your own VM. I'd be willing to do some work there if it would be desirable. It would make adding an MacOS release (something else I've been looking into) much easier because one of those workflows can be run under a macOS runner.

it's also in theory possible to trigger automatic flathub updates when you make a github release by integrating https://github.com/flathub/flatpak-external-data-checker. But worst case the update process for flathub after it's submitted is the creating and merging of a pull request on the flathub managed repo which updates the manifest.

vkbo Nov 29, 2022
Maintainer

Running my own VMs is not an issue, I need them to test novelWriter for compatibility anyway, and I also use them for other things. I have a bunch of them.

The main reason to not use github to build the packages has in part been security (I need to sign packages with my sign key and prefer to do that in a controlled environment) and that the build process still needs some manual verification before I upload them. Mostly, it's been fairly easy to do the releases as the setup.py script does almost everything. I have bash scripts that do the job in the correct order. It is the special cases that add work, like AppImage and Windows installer, and I expect, Flatpak.

The MacOS release is a good candidate for a github workflow though. I do have a macOS VM, but I don't really know my way around a Mac, and how to build packages for it. Any assistance there would be much appreciated. The current system isn't really working very well.

As for Flatpak, if the main job is getting it set up, and releasing is fairly easy, then we could give it a try.

Ryex Nov 29, 2022
Author

ok well, good news is I've completed a working flatpak manifest that would be fairly simple to work with and update

flatpak/io.novelwriter.novelWriter.yml

app-id: io.novelwriter.novelWriter
runtime: org.kde.Platform
runtime-version: '5.15-22.08'
sdk: org.kde.Sdk
base: com.riverbankcomputing.PyQt.BaseApp
base-version: '5.15-22.08'
cleanup-commands:
  - /app/cleanup-BaseApp.sh
command: novelwriter
build-options:
  env:
    - BASEAPP_REMOVE_WEBENGINE=1
finish-args:
  # X11 + XShm access
  - --share=ipc
  - --socket=x11
  # Wayland access
  - --socket=wayland
  # Needs to talk to the network:
  - --share=network
  # Needs to save files locally
  #- --filesystem=xdg-documents
  #- --filesystem=home
  - --filesystem=host
  # OpenGL rendering
  - --device=dri
modules:
  - pypi-deps.json
  - name: enchant
    buildsystem: autotools
    sources:
      - type: archive
        url: https://github.com/AbiWord/enchant/releases/download/v2.3.3/enchant-2.3.3.tar.gz
        sha256: 3da12103f11cf49c3cf2fd2ce3017575c5321a489e5b9bfa81dd91ec413f3891
  - name: novelWriter
    buildsystem: simple
    build-commands:
      - python3 setup.py install --prefix=${FLATPAK_DEST} --root=/
      - install -Dm644 setup/data/novelwriter.desktop -T ${FLATPAK_DEST}/share/applications/${FLATPAK_ID}.desktop
      - sed -i 's/Icon=novelwriter/Icon=io.novelwriter.novelWriter/' ${FLATPAK_DEST}/share/applications/${FLATPAK_ID}.desktop
      - for size in 16x16 24x24 32x32 48x48 64x64 128x128 256x256; do 
        install -Dm644 -T setup/data/hicolor/$size/apps/novelwriter.png ${FLATPAK_DEST}/share/icons/hicolor/$size/apps/${FLATPAK_ID}.png;
        install -Dm644 -T setup/data/hicolor/$size/mimetypes/application-x-novelwriter-project.png ${FLATPAK_DEST}/share/icons/hicolor/$size/mimetypes/application-x-novelwriter-project.png;
        done
      - install -Dm644 -T setup/data/hicolor/scalable/apps/novelwriter.svg ${FLATPAK_DEST}/share/icons/hicolor/scalable/apps/${FLATPAK_ID}.svg;
      - install -Dm644 -T setup/data/hicolor/scalable/mimetypes/application-x-novelwriter-project.svg ${FLATPAK_DEST}/share/icons/hicolor/scalable/mimetypes/application-x-novelwriter-project.svg;
      - install -Dm644 -T setup/data/novelwriter.appdata.xml ${FLATPAK_DEST}/share/metainfo/${FLATPAK_ID}.metainfo.xml
      - sed -i 's/novelwriter.desktop/io.novelwriter.novelWriter.desktop/' ${FLATPAK_DEST}/share/metainfo/${FLATPAK_ID}.metainfo.xml
    sources:
      - type: dir
        path: ../
      # - type: archive
      #   path: ../dist_minimal/novelwriter-1.7b1-minimal-linux.zip
      #   sha256: d4ff96280d5c86c2b14d7ed3b9bc3ced0fc401530afd1eeaa5aabc63d10ceba6
      #   strip-components: 0

flatpak/pypi-deps.json

{
    "name": "pypi-deps",
    "buildsystem": "simple",
    "build-commands": [],
    "modules": [
        {
            "name": "python3-lxml",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"lxml>=4.2.0\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/70/bb/7a2c7b4f8f434aa1ee801704bf08f1e53d7b5feba3d5313ab17003477808/lxml-4.9.1.tar.gz",
                    "sha256": "fe749b052bb7233fe5d072fcb549221a8cb1a16725c47c37e42b0b9cb3ff2c3f"
                }
            ]
        },
        {
            "name": "python3-pyenchant",
            "buildsystem": "simple",
            "build-commands": [
                "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"pyenchant>=3.0.0\" --no-build-isolation"
            ],
            "sources": [
                {
                    "type": "file",
                    "url": "https://files.pythonhosted.org/packages/54/4c/a741dddab6ad96f257d90cb4d23067ffadac526c9cab3a99ca6ce3c05477/pyenchant-3.2.2-py3-none-any.whl",
                    "sha256": "5facc821ece957208a81423af7d6ec7810dad29697cb0d77aae81e4e11c8e5a6"
                }
            ]
        }
    ]
}

and as you can see there at the bottom (though I commented it out currently) it can build off of the minimal-zip artifact from github releases provided there are two extra files provided with it.

MANIFEST.in to ensure novelwriter/assets get copied with the install command
an updated appdata.xml located in setup/data that I can make a pull request for

setup/data/novelwriter.appdata.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- Copyright Copyright 2018–2022, Veronica Berglyd Olsen <[email protected]> -->
<component type="desktop-application">
    <id>io.novelwriter.novelWriter</id>
    <metadata_license>CC0-1.0</metadata_license>
    <project_license>GPL-3.0</project_license>
    <name>novelWriter</name>
    <summary>A markdown-like text editor for planning and writing novels</summary>
    <description>
        <p>novelWriter is a plain text editor designed for writing novels assembled from
        many smaller text documents. It uses a minimal formatting syntax inspired by
        Markdown, and adds a meta data syntax for comments, synopsis, and
        cross-referencing. It's designed to be a simple text editor that allows for
        easy organisation of text and notes, using human readable text files as
        storage for robustness.</p>
    </description>
    <launchable type="desktop-id">novelwriter.desktop</launchable>
    <url type="homepage">https://novelwriter.io/</url>
    <screenshots>
        <screenshot type="source">
            <image>https://novelwriter.io/images/screenshot-multi.png</image>
        </screenshot>
        <screenshot type="source">
            <image>https://novelwriter.io/images/f-nw-build-light.png</image>
        </screenshot>
        <screenshot type="source">
            <image>https://novelwriter.io/images/f-nw-editor-dark.png</image>
        </screenshot>
        <screenshot type="source">
            <image>https://novelwriter.io/images/f-nw-editor-light.png</image>
        </screenshot>
        <screenshot type="source">
            <image>https://novelwriter.io/images/f-nw-novelview-dark.png</image>
        </screenshot>
        <screenshot type="source">
            <image>https://novelwriter.io/images/f-nw-novelview-light.png</image>
        </screenshot>
        <screenshot type="source">
            <image>https://novelwriter.io/images/f-nw-outline-dark.png</image>
        </screenshot>
        <screenshot type="source">
            <image>https://novelwriter.io/images/f-nw-outline-light.png</image>
        </screenshot>
    </screenshots>
    <provides>
        <id>novelwriter.desktop</id>
    </provides>
    <content_rating type="oars-1.0" />
    <releases>
        <release version="v2.0.1" date="2022-11-29">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v2.0.1</url>
        </release>
        <release version="v2.0" date="2022-11-28">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v2.0</url>
        </release>
        <release version="v1.6.6" date="2022-10-25">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.6.6</url>
        </release>
        <release version="v1.6.5" date="2022-10-13">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.6.5</url>
        </release>
        <release version="v1.6.4" date="2022-09-29">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.6.4</url>
        </release>
        <release version="v1.6.3" date="2022-08-18">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.6.3</url>
        </release>
        <release version="v1.6.2" date="2022-03-20">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.6.2</url>
        </release>
        <release version="v1.6.1" date="2022-03-16">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.6.1</url>
        </release>
        <release version="v1.6" date="2022-02-20">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.6</url>
        </release>
        <release version="v1.5.5" date="2022-01-05">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.5.5</url>
        </release>
        <release version="v1.5.4" date="2022-01-04">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.5.4</url>
        </release>
        <release version="v1.5.3" date="2021-12-31">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.5.3</url>
        </release>
        <release version="v1.5.2" date="2021-12-12">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.5.2</url>
        </release>
        <release version="v1.5.1" date="2021-10-23">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.5.1</url>
        </release>
        <release version="v1.5" date="2021-09-19">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.5</url>
        </release>
        <release version="v1.4.2" date="2021-08-30">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.4.2</url>
        </release>
        <release version="v1.4.1" date="2021-07-27">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.4.1</url>
        </release>
        <release version="v1.4" date="2021-07-27">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.4</url>
        </release>
        <release version="v1.3.3" date="2021-06-13">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.3.3</url>
        </release>
        <release version="v1.3.2" date="2021-05-30">
            <url>https://github.com/vkbo/novelWriter/releases/tag/v1.3.2</url>
        </release>
    </releases>
</component>

when submitted to flathub that sources section of the yml would look like

    sources:
      - type: archive
        url: https://github.com/vkbo/novelWriter/releases/download/v2.0.1/novelwriter-2.0.1-minimal-linux.zip
        sha256: e0c9e02d9e0589f4fe268413e82a7033791ac1e1856fe4e8ab9ffa99d81a577c
        strip-components: 0

so making an updated flathub release would be as simple as making a PR over there to update the archive link and checksum in the manifest.

Additionally you would want to add a new </release> tag to the appdata.xml before tagging the release.

if you ever bump the version requirements for the requirements.txt deps or just want to update them them in the flatpak build the pypi-deps.json can be updated with a simple script:

flatpak/generate-pypi-deps-json.sh

#! /bin/sh

cp ../requirements.txt .
sed -e '/pyqt5/s/^/#/g' -i requirements.txt
curl -LO https://github.com/flatpak/flatpak-builder-tools/raw/master/pip/flatpak-pip-generator
python3 flatpak-pip-generator \
    --requirements-file=requirements.txt \
    --runtime=com.riverbankcomputing.PyQt.BaseApp//5.15-22.08 \
    -o pypi-deps
rm requirements.txt
rm flatpak-pip-generator

Would you like me to make a PR to add these files to the project?
after than I can mostly walk you though submitting to Flathub if your satisfied (The maintainers over they may ask for some refinements before it's accepted which I'll be glad to help out with) .

vkbo Nov 30, 2022
Maintainer

Sure, make a PR and let's give this a go.

On the point about the minimal package, I am considering dropping them from the releases, but tweak them so they can be used as the basis for different packaging pathways. It would make the setup.py script a lot simpler too.

Currently, the windows installer file is built directly of the windows minimal zip file. Feel free to tweak the linux one to work with Flatpak, and ideally also AppImage? Perhaps it would be more useful to add general purpose functions that generate a folder with the needed files for a given platform. The minimal zip packages can then also be built from that.

Anyway, I can help with that as long as it's clear what each packaging tool needs.

Also, I think it would be wise to not link directly to the screenshots used by the website. There's always a risk I will update them or make changes to the live site that will break the Flatpak info. Perhaps it is better to add a folder for Flathub with a copy of the images. I can do that, or you can make a PR to https://github.com/vkbo/novelWriter-website (the site is not updated from the repo, but I will do it manually).

johnblommers · 2022-11-29T23:28:48Z

johnblommers
Nov 29, 2022

I'm a huge fan of using the flatpak method to install and update applications.

I hope this approach takes traction with novelWriter.

That said, the AppImage approach is also good because that format lends to integration with the OS (via a `desktop file IIRC) as well as automatic updates.

1 reply

vkbo Nov 30, 2022
Maintainer

Yeah, I think it's worth trying out. Thanks for the input!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distribution on flathub #1267

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Distribution on flathub #1267

Ryex Nov 29, 2022

Replies: 2 comments · 5 replies

vkbo Nov 29, 2022 Maintainer

Ryex Nov 29, 2022 Author

vkbo Nov 29, 2022 Maintainer

Ryex Nov 29, 2022 Author

vkbo Nov 30, 2022 Maintainer

johnblommers Nov 29, 2022

vkbo Nov 30, 2022 Maintainer

Ryex
Nov 29, 2022

Replies: 2 comments 5 replies

vkbo
Nov 29, 2022
Maintainer

Ryex Nov 29, 2022
Author

vkbo Nov 29, 2022
Maintainer

Ryex Nov 29, 2022
Author

vkbo Nov 30, 2022
Maintainer

johnblommers
Nov 29, 2022

vkbo Nov 30, 2022
Maintainer