Build a bare-bones version of appimagetool

[probonopd]

Once #86 is done, we may want to also build a bare-bones version of the tool that does only one thing, turn AppDirs into AppImages (like the old appimagetool in https://github.com/AppImage/AppImageKit did). This might make it more viable for other projects (e.g., electron-builder, linuxdeployqt, appimage-builder,...) to use it under the hood.

That one might possibly be called appimagetool and might be made command-line compatible to the old appimagetool from AppImageKit, which at that point might be deprecated.

While I don't have a need for this myself (I will use the do-it-all-automagic full blown tool), I'd welcome PRs that would build that bare-bones version.

Only requirement:
Things don't get more complicated for me. Especially, no multi-repository setups.

cc @TheAssassin @srevinsaju

[srevinsaju]

The src/appimagetool/appimagetool.go has become quite long and complex. Would it be nice to split appimagetool.go into smaller go files which would help to achieve this? and we would not need any extra repository setup, all the files in the same repository, same folder.

Points to ponder

• will we distribute this appimagetool in the Releases page
• should we make this tool accessible for end users, or only for developers?

[probonopd]

The src/appimagetool/appimagetool.go has become quite long and complex. Would it be nice to split appimagetool.go into smaller go files which would help to achieve this?

Somehow I find one long, linear-written (you can read it from top to bottom, like a page in a book) file much easier to understand than a directory of files where stuff is calling each other. May just be a personal preference, but I like one long program that executes linearly from top to bottom. (I am always looking for the simplest possible solution.) Should also help newcomers to understand what is going on.

I think the main objective is not to make code (in my view) more complicated by splitting it up into multiple files, but to have a separate binary that just does the basic task of converting an AppDir into an AppImage. Others may think differently, but as I said above, my requirement is that things don't get more complicated for me. This is my hobby project and it's supposed to stay fun and simple.

The more I think about it:

Maybe we should indeed just copy

https://github.com/probonopd/go-appimage/blob/master/src/appimagetool/appimagetool.go#L437-L510

and put it into its own repository for those who want a low-level tool. By doing this, src/appimagetool/appimagetool.go in this repository could stay linear, unchanged. Those who would need a low-level tool could use that other repository.

[srevinsaju]

The src/appimagetool/appimagetool.go has become quite long and complex. Would it be nice to split appimagetool.go into smaller go files which would help to achieve this?

Somehow I find one long, linear-written (you can read it from top do bottom, like a page in a book) file much easier to understand than a directory of files where stuff is calling each other. May just be a personal preference, but I like one long program that executes linarly from top to bottom. (I am always looking for the simplest possible solution.) Should also help newcomers to understand what is going on.

I used to follow that same idea, 2 years ago. 😄, but later, splitting in separate files helped me to

• increase code readability
• increase code quality
• reduce code duplication
• use better variable names

I think the main objective is not to make code (in my view) more complicated by splitting it up into multiple files, but to have a separate binary that just does the basic task of converting an AppDir into an AppImage. Others may think differently, but as I said above, my requirement is that things don't get more complicated for me. This is my hobby project and it's supposed to stay fun and simple.

The more I think about it:

Maybe we should indeed just copy

https://github.com/probonopd/go-appimage/blob/master/src/appimagetool/appimagetool.go#L437-L510

and put it into its own repository for those who want a low-level tool. By doing this, src/appimagetool/appimagetool.go in this repository could stay linear, unchanged.

Hmm. ok...

[probonopd]

I think it comes down to how actively involved one is with a codebase. For me, it can happen that i won't look at something for a year, or longer. Then one file with linear code is much easier to pick up (again) than a set of files where you have no clue about the logic flow. For me, at least (this may be personal preference).

[srevinsaju]

SourceTrail is the right tool for you 😄

[probonopd]

There's even an AppImage of it: https://github.com/CoatiSoftware/Sourcetrail/releases

But that's my point. One linear file, readable from top to bottom, saves one having to spend extra time with tools like this.

[srevinsaju]

Ok, my viewpoint is different, I'm not sure if its right

From https://www.quora.com/Why-do-you-want-to-split-your-program-code-into-multiple-files

Basically, a single file is easy to write. Multiple files are much easy to read, maintain and manage. For software development, the later is much more important. Even if you are working alone, future-you does not know everything that past-you has done.

I have never understood how go-appimage works so far, nor did I understand AppImageKit, but I understood how libappimage works, a well documented code. I tried for 2 months. Right now, all I did was a few git greps to approximate the variable names you would have used and by searching for the output from the terminal. And then, from that line, I go back to see why and how the error was created, but it may work differently for others I guess.
I don't mind keeping it linear 😄

[TheAssassin]

Things don't get more complicated for me. Especially, no multi-repository setups.

In that case, we will have to continue maintaining a proper, minimal appimagetool upstream. This repository is not "core" in any way, and if you insist on your point, I'd rather continue maintaining a real appimagetool, not this monster in a programming language I don't like. However, a future appimagetool will likely take advantage of real, modern C++, not pure unsafe C. Maybe even just Rust.

[srevinsaju]

Things don't get more complicated for me. Especially, no multi-repository setups.

In that case, we will have to continue maintaining a proper, minimal appimagetool upstream. This repository is not "core" in any way, and if you insist on your point, I'd rather continue maintaining a real appimagetool, not this monster in a programming language I don't like. However, a future appimagetool will likely take advantage of real, modern C++, not pure unsafe C. Maybe even just Rust.

I would seriously like it, if its in Rust 😄 🚀

[TheAssassin]

I consider

Only requirement:
Things don't get more complicated for me. Especially, no multi-repository setups.

quite selfish. Everyone already agreed to learn Go for this tool, so we already made a big tradeoff.

Go's build system integrates Git very well anyway, so reusing code from a separate repository isn't even hard at all.

I don't understand why @probonopd can't move at least a little. We all work on this in our free time. But as said, if these restrictions remain, I think we will maintain our own tool. It's easier that way, and we can choose a language freely and democratically.

Also, it's not like the Go code in here was very good yet. You'd probably benefit from our time, too, as the first thing we'd have to do is refactor, refactor, refactor.

To sum up: no, thanks, but we'll rather do our thing then, if you insist on these conditions.

---

Somehow I find one long, linear-written (you can read it from top do bottom, like a page in a book) file much easier to understand than a directory of files where stuff is calling each other.

That's contradictory to

Should also help newcomers to understand what is going on.

Code re-use, DRY, separation of concerns, abstraction, being able to unit-test, ... are not there to make your life more complicated. No, they help make things simple and understandable. This huge one-file concept is a bad practice for various reasons. I've seen these programs, and they're hard to navigate and understand. The main reason to create linuxdeploy for instance was because I had to spend hours fixing things in linuxdeployqt, where duplicate code was the norm, not the exception.

much easier to pick up (again) than a set of files where you have no clue about the logic flow. For me, at least (this may be personal preference).

I highly disagree. For you it might be easier, because you created the mess in the first place. Onboarding new people on such codebases is pretty much impossible. Any software developer is immediately put off by such messy, huge files. In fact, I had to convince myself quite hard before touching code like linuxdeployqt in the first place. And it took me many hours to be able to navigate properly in there. It's not just the lack of structure, but also the lack of documentation (which is also easier to add if you split up your program following some best practices...).

There's a reason we implement abstractions, multiple layer architectures etc. One could describe it as "divide and conquer". You, as the developer of component A, don't need to understand how a function from component B works internally. You read its description, and know what it should do. You can rely on the developer of that function to program it correctly. Ideally, it's also unit-tested.
If not, you can isolate the issue, write a minimal example and break down the issue without needing to simulate entire program runs. Ideally you write a new test case that triggers the buggy behavior (and thus fails), then fix the issue, and make sure the same issue won't affect the program again in the future.
Design by contract, design patters, they're used everywhere. But unfortunately not in most AppImage project software...

Single-file applications in the style which you seem to like so much are also impossible to test. The poor software architecture at most allows for system tests, i.e., you need to always simulate a complete run, without being able to test single units (hence, unit tests). That's just a waste of computational time.

Your arguments don't convince me, and I highly doubt they'll convince any other experienced programmer. I know, many produce bad code themselves, but they can recognize and appreciate properly written programs. @srevinsaju just said it: compare appimagetool (a huge mess) to e.g., libappimage (which you consider "complex" because it's not all dumped in one file). libappimage wins, by far.

I would seriously like it, if its in Rust smile rocket

@srevinsaju great you'd like a Rust implementation. I thought, if we start writing e.g., the runtime in Rust, for a (admittedly, not too large) tool like appimagetool, it also seems pretty suitable. The last time I wrote serious Rust was a couple years ago, but I'd be interested in getting into it again.
I personally know how to program safe C and C++, but hey, why not use a language for tools as widespread as the runtime that prevents memory safety issues language wise, right?
We also don't have to work with FUSE or squashfs a lot, we just call external tools mostly.

Perhaps @probonopd will then also use that new appimagetool. Because redundancy makes no sense. But we'll see how it turns out. I don't mind @probonopd spending time on this. But I won't.

[probonopd]

Your arguments don't convince me, and I highly doubt they'll convince any other experienced programmer.

Maybe "drive-by", occasional programmers will relate more to the linear, top-to-bottom coding style. This is what I meant by "personal preference". I find it hardly astounding that someone with a CS degree coding on a daily basis using fancy IDEs and tools has some different positions and priorities than an occasional hobby programmer who doesn't touch this code for months in a row. No offense. We should try to understand that there are multiple ways of looking at things. But please let me have me this one place that can stay easy.

libappimage I do not understand. It is too complicated for me. I have never touched it for this reason.

My thinking:
Do A -> then do B -> then do C -> Done.
(Like a cooking recipe.)

C++ tinking:
B* is a special instance of B with some Y mixed in. C implements B and depends on A.
Since A is a component it must also be usable not only for B and C but also D, E, and F. E only wants parts of my code, so they want me to split my code so that they can just use the parts they want. And L wants to distribute it, so it needs an interface. With this comes an expectation that my code suddenly should also address edge cases around L.
(Like a construction kit.)

See what I mean?

[TheAssassin]

Do you complain the same way about Qt and all libraries you use? Do you always copy all their code and paste it into huge files? I highly doubt that. No, I am sure you benefit from the good documentation and the many classes which are properly separated by their concerns.

I'm not saying libappimage was great or on par with Qt. There surely could be some tutorials, some improved documentation. But instead of working on such non-code issues (which, really, anyone could do, one should avoid having "someone with a CS degree" spend their time on this, and rather go improve the code), in AppImage, stuff is constantly just rewritten, duplicated etc.

My thinking:
Do A -> then do B -> then do C -> Done.

C++ tinking:
B* is a special instance of B with some Y mixed in. C implements B and depends on A.
Since A is a component it must also be usable not only for B and C but also D, E, and F. And L wants to distribute it, so it needs an interface.

No, I don't see it. That's Java stuff. C++ doesn't even force you to use classes. You can follow a merely functional programming style (although it doesn't make sense).
This "example" just shows your intention is to make up some unnecessarily complicated setup. In fact, you just made more complex through ~8 edits. That's not a fair discussion in any way. It just shows you're not interested in a discussion.

By the way, you constantly have misconceptions about modern C++. You always mix your imagination of C++ with what you know from Qt (which is C++98, so, more than 20 years old) and C (you don't need pointers a lot in C++). C++ is a flexible language. You can build complex code there as you can in Go. In the end it's the programmer who defines it, not the language (except for Java maybe...).

As said, please feel free to have this one place that you consider easy (while everyone else considers it a mess). But then let us have a properly written program in the other place. I am not interested in a discussion any more either.

[probonopd]

That's what I am proposing: Feel free to copy-paste the lines of code you like and ignore the ones you don't like.

(Probably that will come down to those or maybe fewer than those, or even nothing at all. I will not be offended.)

[probonopd]

So, no need for a bare-bones version of appimagetool within this repository as it will be done elsewhere.

[srevinsaju]

seriously like it, if its in Rust smile rocket

@srevinsaju great you'd like a Rust implementation. I thought, if we start writing e.g., the runtime in Rust, for a (admittedly, not too large) tool like appimagetool, it also seems pretty suitable. The last time I wrote serious Rust was a couple years ago, but I'd be interested in getting into it again.

@TheAssassin I have very less Rust experience too, but I am interested to learn a new thing, rather than revolving around the same old programming languages I learnt. I learnt basic go in two days, I just learn when I really need it, I guess most programmers don't learn all languages, but manually adjust their style to other languages, as they need it. And Rust would be a big add-on!

I personally know how to program safe C and C++, but hey, why not use a language for tools as widespread as the runtime that prevents memory safety issues language wise, right?

Yes, exactly. I hope Rust would make the runtime and the new appimagetool more contributor friendly.

[probonopd]

Yes, exactly. I hope Rust would make the runtime and the new appimagetool more contributor friendly.

Well, according to https://www.tiobe.com/tiobe-index/ Rust is rather niche, hence not many developers will know it. I agree that Go can be learned very quickly, I haven't looked into Rust yet so I don't know if it is true for Rust as well.

Rust's strength seems to be safety, which would be especially desirable for the runtime. But then, as long as the runtime uses squashfuse and libfuse, the main part of it won't be Rust anyway (unless all of that is rewritten in Rust as well, which I'd imagine to be quite an undertaking).

[srevinsaju]

Yes, go as a programming language is easier to learn, as compared to Rust, but I am afraid of #87 (comment) and #87 (comment)

If you are willing to change linearly written appimagetool.go to smaller script files, it would be great. I would still send small patches whenever I notice a bug, but we cannot "revolutionize" this tool, without, as what @TheAssassin has told, divide-and-conqeur. Go is very easy. If the software you are writing, if you want it to be a community led project, Go is the way to go 😄 , but, for that, we need to heavy refactor the code. Its okay for me to refactor the code, as it would help me to learn better about the AppImage concept.

I don't mind it being written in any language, as long as it is useful and configurable, and simple ofc. I would be glad, if the future appimagetool is written in either Go, or Rust, so I can find some ground to contribute, and as long as the project does not die.

[TheAssassin]

As said, the complexity is more related to the programmer, not the language. I would also be open to writing the tools in Python...

[srevinsaju]

Yes, I agree with @TheAssassin

simply put in

Go + Linearly written code = Learning Rust

[probonopd]

I would also be open to writing the tools in Python...

The dowside of using Python is that you don't get this nice static binary for all CPUs and OSes "for free" that Go gives you without having to install anything special. And since you can never know what users have installed on their Python environment, things tend to break more. This is why I like Go. Once compiled, the user cannot change/mess up anything. (Says someone who likes coding in Python.)

[probonopd]

configurable, and simple ofc

Those are directly contradicting each other imho.

[srevinsaju]

The current appimagetool on AppImageKit is simple enough and configurable right?

[probonopd]

Let's say it's a "low-level" tool. You basically always need other tools in addition to it. Most people who think they know better and just use bash and the current appimagetool on AppImageKit end up with AppImages that are broken.

This is why I wanted to make a tool that always gives the user a working AppImage including update-ability etc. (or else, it's the tool's fault only).

@TheAssassin thinks that people read manuals, and understand everything correctly, and then indeed follow everything as written. I think that's wishful thinking most of the time.

[srevinsaju]

Yes, I agree @probonopd, most of the devs don't read documentation, assuming they are smart, and fall into trouble, and come back to look for them.

But consider, I stopped using GitHub, and I am going to setup a custom Gitea instance on my private server. Or even invent.kde.org for example. How would I configure go-appimage without rewriting the some part of its code?

[probonopd]

Yes, I agree @probonopd, most of the devs don't read documentation, assuming they are smart, and fall into trouble, and come back to look for them.

The sad thing is that badly made AppImage usually tend to work on the developer machine. It's only when they hit our automated test which tries to run on the oldest still-supported Ubuntu that those AppImages fail. I want them to fail immediately, when the developer is trying to build an AppImage for the first time. Increases chances of the AppImage getting fixed right from the start. See what I mean?

But consider, I stopped using GitHub, and I am going to setup a custom Gitea instance on my private server. Or even invent.kde.org for example. How would I configure go-appimage without rewriting the some part of its code?

GitLab support is highly desirable. Gitea... well, if someone cares to write a PR i'd gladly take it. Totally exotic systems: I am not interested in supporting them with this tool, because I cannot control the end-to-end quality of the result.

I'd rather just support a handful (e.g., Travis CI, GitHub Actions, GitLab CI) and have them work perfectly well with zero configuration.

(The "low-level tool" may have other priorities.)

[TheAssassin]

Python distribution is a solved problem. Everyone knows how to use pip, and we can also produce AppImages. So I don't consider that a big downside.

@TheAssassin thinks that people read manuals, and understand everything correctly, and then indeed follow everything as written.

No. I am convinced that you need to provide both proper docs and hint people. Automagic guessification doesn't help. It just binds everything to commercial infrastructure.

Don't put it like I'm a pure optimist. I consider myself a realist. And I consider my approach more realistic, as I focus on the long term, not the short term success.

The dowside of using Python is that you don't get this nice static binary for all CPUs and OSes

Static binaries also have lots of flaws. You still think people will just take care of updating all components regularly etc. It's not happening, though.

And arguably, Go dependencies are not as well established as all the libraries we use in other languages like C++ or Python. I don't see why any Go or Rust library downloaded with the built-in package managers are to be trusted. With Python for instance, one can rely on a few libraries in very widespread use (e.g., requests, Flask, ...).

configurable, and simple ofc

Those are directly contradicting each other imho.

Nonsense IMO. Simple by default, powerful when needed is the mantra to follow. You're not facing random users, you're facing developers. And developers must be re-educated to care about what they publish. You always cite low-quality AppImages. The solution can't be to try to fix issues for people. Might work, but as @srevinsaju just said, if it does not, having to rewrite the tool is not a great solution. So why not just offer the option right away? Doesn't mean a tool can't be simple by default.

I still believe (and I get feedback that it is what (good) developers also want) that what you need to do is explain what people are doing wrong and showing them how to do better. That way, they will take the accountability and responsibility for their own products. If you do not properly communicate issues, you're in the end who is going to be responsible. But I want to reduce my own workload and responsibility, and the only good way is to hand it off to the users of my tools.

Remember e.g., the linuxdeployqt libc mess: I would always prefer showing the pros and cons of supporting old distros and suggest how to do better instead of paternalizing users through ideologic errors. I provide a linter that correctly warns if something isn't compatible with the oldest still-supported LTS of Ubuntu, but I don't require them specify parameters with likely offending wordings. There's a reason people would attack you for implementing this weird error, whereas people thank me for being realistic when I explain the matters and help them find solutions how to implement backwards compatibility.

Off topic: this entire "what should be the baseline for support" discussion is anyway quite polarizing. For instance, with less than 6 months to xenial's EOL, I don't think requiring new projects to support it is viable. One can suggest it, but I can understand anyone who wants to e.g., use C++17 nowadays and therefore will only support bionic.

[probonopd]

change linearly written appimagetool.go to smaller script files

If you look at https://github.com/probonopd/go-appimage/tree/master/internal/helpers then you will see that I am actually doing this to some extent. For me it is OK to factor out helper functions that deal with certain aspects, as long as the main program's logic flow is still very clearly readable from top to bottom.

If you want an example where this is not the case: https://github.com/probonopd/linuxdeployqt/. I took the code from the existing macdeployqt tool and changed it until it did something for Linux. To this date I have trouble understanding the program flow. This was the reason for me to no longer invest into linuxdeployqt and start here from scratch. I pledged to myself, I want a clear linear program flow that everyone will immediately understand, and not this pseudo-OO spaghetti code.

[srevinsaju]

Python distribution is a solved problem. Everyone knows how to use pip, and we can also produce AppImages. So I don't consider that a big downside.

@TheAssassin thinks that people read manuals, and understand everything correctly, and then indeed follow everything as written.

No. I am convinced that you need to provide both proper docs and hint people. Automagic guessification doesn't help. It just binds everything to commercial infrastructure.

Agreed

Don't put it like I'm a pure optimist. I consider myself a realist. And I consider my approach more realistic, as I focus on the long term, not the short term success.

The downside of using Python is that you don't get this nice static binary for all CPUs and OSes

Static binaries also have lots of flaws. You still think people will just take care of updating all components regularly etc. It's not happening, though.

Yes, python3 interpreter provided by the distribution is pretty much very safe than a binary built from Go, Rust or even C, C++, becaue its an interpreted language and not a binary.

And arguably, Go dependencies are not as well established as all the libraries we use in other languages like C++ or Python. I don't see why any Go or Rust library downloaded with the built-in package managers are to be trusted. With Python for instance, one can rely on a few libraries in very widespread use (e.g., requests, Flask, ...).

configurable, and simple ofc

Those are directly contradicting each other imho.

Nonsense IMO. Simple by default, powerful when needed is the mantra to follow. You're not facing random users, you're facing developers. And developers must be re-educated to care about what they publish. You always cite low-quality AppImages. The solution can't be to try to fix issues for people. Might work, but as @srevinsaju just said, if it does not, having to rewrite the tool is not a great solution. So why not just offer the option right away? Doesn't mean a tool can't be simple by default.

😃

I still believe (and I get feedback that it is what (good) developers also want) that what you need to do is explain what people are doing wrong and showing them how to do better. That way, they will take the accountability and responsibility for their own products. If you do not properly communicate issues, you're in the end who is going to be responsible. But I want to reduce my own workload and responsibility, and the only good way is to hand it off to the users of my tools.

I agree here. @TheAssassin helped me understand how zsync works, and how its being used in appimagetool, also helped me to write proper update information, and told me to wrap my build around appimagelint to find out some unnoticed errors. Had not @TheAssassin explained it to me, I would have never understood why / how zsync information is used by appimagetool.

I could later download and reuse Ubuntu 20.10 beta builds without downloading the entire ISO. I indeed learnt a lot.

Remember e.g., the linuxdeployqt libc mess: I would always prefer showing the pros and cons of supporting old distros and suggest how to do better instead of paternalizing users through ideologic errors. I provide a linter that correctly warns if something isn't compatible with the oldest still-supported LTS of Ubuntu, but I don't require them specify parameters with likely offending wordings. There's a reason people would attack you for implementing this weird error, whereas people thank me for being realistic when I explain the matters and help them find solutions how to implement backwards compatibility.

Off topic: this entire "what should be the baseline for support" discussion is anyway quite polarizing. For instance, with less than 6 months to xenial's EOL, I don't think requiring new projects to support it is viable. One can suggest it, but I can understand anyone who wants to e.g., use C++17 nowadays and therefore will only support bionic.

change linearly written appimagetool.go to smaller script files

If you look at https://github.com/probonopd/go-appimage/tree/master/internal/helpers then you will see that I am actually doing this to some extent. For me it is OK to factor out helper functions that deal with certain aspects, as long as the main program's logic flow is still very clearly readable from top to bottom.

ok, Go is a functional programming language, so everything in main() function should be meaningful, provided they are given good variable names and functions. I hope its okay to replace the

go-appimage/src/appimagetool/appimagetool.go

Lines 48 to 56 in 921dfeb

	// Do not exit if ioutil.ReadFile("/proc/1/cgroup") fails. This happens, e.g., on FreeBSD
	if strings.HasPrefix(string(res), "/lxc") || strings.HasPrefix(string(res), "/docker") || helpers.Exists("/.dockerenv") == true {
	log.Println("Running inside Docker. Please make sure that the environment variables from Travis CI")
	log.Println("available inside Docker if you are running on Travis CI.")
	log.Println("This can be achieved by using something along the lines of 'docker run --env-file <(env)'.")
	log.Println("Please see https://github.com/docker/cli/issues/2210.")
	}
	}

with something like checkRunningWithinDockerContainer() function? Then following the similar pattern, if allowed, would be nice, and helpful for new programmers. Thanks 😄

If you want an example where this is not the case: https://github.com/probonopd/linuxdeployqt/. I took the code from the existing macdeployqt tool and changed it until it did something for Linux. To this date I have trouble understanding the program flow. This was the reason for me to no longer invest into linuxdeployqt and start here from scratch. I pledged to myself, I want a clear linear program flow that everyone will immediately understand, and not this pseudo-OO spaghetti code.

I looked at the source code of linuxdeployqt.

500 lines + 1300 lines

You were indeed a fan of long files right? 😄

[probonopd]

Yes, replacing Lines 48 to 56 in 921dfeb with one function that has a speaking name is perfectly acceptable to me.

[probonopd]

Here is an example why the automatic tool should name the file, and upload the file, in one go, without offering a way to rename things:

1000ch/quail#5 (comment)