One-liner bootstrap

[jhgorse]

Pursuant to a ssh-copy-id like solution for setting up a new account.

My bootstrap:

wget https://raw.githubusercontent.com/RichiH/vcsh/master/vcsh
chmod +x vcsh
./vcsh clone https://github.com/jhgorse/bash.git bash # Perhaps a recursive clone some day
./vcsh run bash submodule init
./vcsh run bash submodule update

Proposed:

sh "$(curl -fsSL https://raw.githubusercontent.com/RichiH/vcsh/master/vcsh)" clone --recursive https://github.com/jhgorse/bash.git bash

or perhaps a separate boostrap script that adds a ~/bin or ~/.bin directory, vcsh, chmod+x, then calls the trailing bootstrap args upon setup completion.

[RichiH]

Personally, I don't like to cURL into a shell, but if you want your own personal bootstrap, wouldn't it make sense to just have a shell script you can cURL?

curl https://my.dom/script.sh | sh

or the like.

[jhgorse]

@RichiH what is your preferred manner of bootstrapping a new environment?

[alerque]

I think I'm going to chime in strong agreement with Richard on this one. I don't think public projects should endorse the sh <(curl <something>)" paradigm. It's just so dangerous on so many levels... If you're going to do that kind of thing you should be responsible for both ends of the stick. Ironically my own bootstrap method involves something like this, but at least I am responsible for my own destiny. If I make a bum turn...

Keep in mind that there are two parts here: you need a system with vcsh itself, and you need your own personal dotfiles repository. We don't want to meddle much with the latter, and the former is highly system dependent. Getting vcsh on a system should be done through that system's package manger. If any bootstrapping is done it should really be done tailored to your system, so again I think hosting your own preferred jump starter kit is probably better than anything this project can provide upstream.

If you have ideas for how this could be improved without compromising ⓐ security best practices and ⓑ flexibility please do comment and we'll reconsider.

[jhgorse]

So what then is the purpose of vcsh if it is not to improve the bootstrapping experience of a new shell?

Each flavor of unix that I come across is just compatible enough to use the core settings of my environment, but never all of them in exactly the way they are stored. There is always editing necessary, so vcsh is less useful in this need to have a hierarchical tree of shell configurations based on the unix flavor, version, and date for automation.

And as far as security best practices go, operability is preferred over the security of effective nonexistence. Forcing people to type passwords to confirm identity and "are you really sure you want to do something useful and write data to your computer" is getting to a point where it gets in the way of using the technology. Does that make it better? I think it helps the security folks feel important, though the paradigm mostly adds dependency and breaks software. None of these systems can work without an element of good faith. But that is okay. Onward to the next thing. Thank you for the note.

[alerque]

So what then is the purpose of vcsh if it is not to improve the bootstrapping experience of a new shell?

Fair question.

Surely bootstrapping new shells is part of the deal for sure, but a couple things come to mind, at least for my uses cases:

1. I only bootstrap a new system a few times a year. On the other hand I use vcsh every single day to save, tweak, and sync configs between machines. Getting off the ground with a new system is a much smaller part of my usage than the every day grind of modifying various configs and expecting other systems to keep pace.

2. Bootstrapping a new system is a lot less about vcsh itself and much more about my personal dotfiles (some of which are private!). VCSH isn't really the kind of tool that you need to bootstrap on a system—it isn't a package manager for example. There is a pretty good use case for Homebrew using a bootstrap script to get itself setup. VCSH on the other hand can just be installed via your system package manager.

Once you have VCSH available, the thing you need to do on a new system is clone your repositories. This is a very personal arrangement. Do you have one or many? Are they hosted on a public Git server or a private one? Do you use any hooks? Are there secrets involved in fetching them? Do they contain any secrets that need unlocking? And so on and so forth...I'm sure out respective use cases don't look similar at all.

Given those points, what exactly would you be "bootstraping"? What would you expect VCSH to do that would make your use case easier.

Each flavor of unix that I come across is just compatible enough to use the core settings of my environment, but never all of them in exactly the way they are stored. There is always editing necessary, so vcsh is less useful in this need to have a hierarchical tree of shell configurations based on the unix flavor, version, and date for automation.

At least as I'm reading it, this is a completely different issue. What to include or not include and how to organize dotfiles in such a way that they can be used across systems—when to include code (say in a shell rc file) for conditionally enabling configs, when to use separate repositories for systems with something in common, when to branch a set of configs for a system with minor differences, when to use hooks to adapt a common repository for use in differing contexts—all of these things are bits that can be discussed and there are many approaches that work in VCSH. But none of them are really relevant to the initial question as I understand it.

And as far as security best practices go, operability is preferred over the security of effective nonexistence. Forcing people to type passwords to confirm identity and "are you really sure you want to do something useful and write data to your computer" is getting to a point where it gets in the way of using the technology. Does that make it better? I think it helps the security folks feel important, though the paradigm mostly adds dependency and breaks software. None of these systems can work without an element of good faith. But that is okay. Onward to the next thing. Thank you for the note.

Honestly I don't think this is a fair critique. I don't think your beef has anything to do with anything we are doing or not doing for security reasons. If you want to run a one liner to run VCSH "hot" off the main branch there is nothing stopping you from doing so:

$ sh <(curl -sL https://raw.githubusercontent.com/RichiH/vcsh/main/vcsh) --help

I guess by point is whether you install it using a system package manager, the traditional make install, or eschew the install and just run it from a blind download is your prerogative, but we don't intend to promote the latter for various reasons (security being only one of them). Having gotten the executable from wherever you want, you have to tools to write your own bootstrap script that fetches your dotfiles from wherever they are.

What more would you have this project do?

[RichiH]

I have thought a lot about this topic over the last year or three in the context of https://github.com/prometheus/prometheus

I personally wouldn't want to cURL install anything; at the same time, it's a common way to do it.
Is it on me to prescribe how others use my software? Not really.
Is it on me to give guidance on something commonly done, but insecure? Arguably yes.

My gut is to have a oneliner for the people who want it with a Big Fat Warning about the risks of doing so, and linking to the raw view of the script what will be executed so everyone can look at it for themselves.

Enabling use cases while teaching is arguably better long-term than preventing it and having people bake their own.

[jhgorse]

@alerque Thank you for the write up. Your use of vcsh to synchronize across systems helps me see some of the more immediate utility for vcsh across my shells. And yes, I may have ranted a bit with the security issue and you may not have deserved (all of) it. =)

I do agree that the long-term pedantic approach is best. Show how to do the useful things with some convention and let people learn through experience.

Cheers,
Joe

[RichiH]

As per https://github.com/RichiH/vcsh/issues/73 having one "this is POSIX, relies on $PATH, and should Just Work" version seems like a defensible position to me.

[nferch]

I am commenting here in reference to the changes made in #288 which have impacted my usage of this tool.

Previously I had a bootstrap script similar to the OP. It was simple to bake it into image building process or deploy using config management, or install manually, with the only dependencies being git, curl/wget, and the other things you'd find in slim/minimal installations.

This minimized the amount of complexity and work to bootstrap my shell environment, in a manner that it'd "just work" on systems that had minimal installations, or were broken in some shape or form.

With the introduction of autotools things are more complicated: I can no longer simply download the script from https://raw.githubusercontent.com/RichiH/vcsh/master/vcsh or scp it around, in fact I'd have to ensure there are dependent tools installed, tools that might not be present in a system that isn't used for development.

So I'd love for some simple and fast means to install vcsh. The portability, simplicity and lack of dependencies are truly one of the reasons vcsh is unique and powerful.

Happy to explain further, share my bootstrap method, and discuss solutions and tradeoffs.

[alerque]

Thanks for commenting on this issue too.

I'll be prioritizing some kind of bootstrap method (i.e. something that resolves this issue) before the next release tag. For now you can still download the script directly from the last release tag, you just can't fetch it directly from the main branch that way (note this project renamed master→main just before I started massaging things, so your bootstrap scripts would need to have changed that anyway, but please change them to a tagged version instead of main since that will continue to have changes).

With the introduction of autotools things are more complicated: I can no longer simply download the script from master/vcsh (raw) or scp it around

True on not being able to download it raw from the current Git HEAD. However just a note you can scp it from any place it is already installed to any similar system.

, in fact I'd have to ensure there are dependent tools installed, tools that might not be present in a system that isn't used for development.

Autoconf itself should be the only truly new dependency. The only thing besides that we're using now in the script that we were not before is awk, but as that's a pretty standard base thing to have anywhere Git, sed, grep, etc. are present.

So I'd love for some simple and fast means to install vcsh. The portability, simplicity and lack of dependencies are truly one of the reasons vcsh is unique and powerful.

I'm not discounting that some slim systems may not want autoconf at all and we will look into some sort of bootstrap edition — I'm just pointing out that the generated script after build is actually still something you can just copy/scp to a target system and use.

Happy to explain further, share my bootstrap method, and discuss solutions and tradeoffs.

What do you think about having a generated generic/bootstrap edition script attached to releases along with the source tarball? This would allow distro packages and any systems with special constraints to build from the source tarball as usual, but also leave a single-script artifact in plain sight that is compiled without paths (i.e. calls git not whatever was detected on the host at build time such as /usr/bin/git). This could even be fetched using GitHub's latest keyword without knowing a version tag.

Do you have a better idea where to publish such a script?

[nferch]

Thanks for commenting on this issue too.

I'll be prioritizing some kind of bootstrap method (i.e. something that resolves this issue) before the next release tag. For now you can still download the script directly from the last release tag, you just can't fetch it directly from the main branch that way (note this project renamed master_→_main just before I started massaging things, so your bootstrap scripts would need to have changed that anyway, but please change them to a tagged version instead of main since that will continue to have changes).

Yep, that's my band-aid fix, which I'd expect to be workable for the foreseeable future, as that version of vcsh satisfies all of my personal needs. Just wanted to chime in here to see if others had similar concerns and see if I could contribute to a solution.

With the introduction of autotools things are more complicated: I can no longer simply download the script from master/vcsh (raw) or scp it around

True on not being able to download it raw from the current Git HEAD. However just a note you can scp it from any place it is already installed to any similar system.

How similar would they need to be?

, in fact I'd have to ensure there are dependent tools installed, tools that might not be present in a system that isn't used for development.

Autoconf itself should be the only truly new dependency. The only thing besides that we're using now in the script that we were not before is awk, but as that's a pretty standard base thing to have anywhere Git, sed, grep, etc. are present.

Yeah, awk is pretty ubiquitous, although there can be some differences between awk/mawk/gawk and the like.

Autoconf is a different kettle of fish, it's rarely included in base installations, it tends to be included in things like Debian/Ubuntu 'build-essential' and the like.

So I'd love for some simple and fast means to install vcsh. The portability, simplicity and lack of dependencies are truly one of the reasons vcsh is unique and powerful.

I'm not discounting that some slim systems may not want autoconf at all and we will look into some sort of bootstrap edition — I'm just pointing out that the generated script after build is actually still something you can just copy/scp to a target system and use.

I think the "bootstrap edition" idea has some legs. Would the generated script be universal or have variances depending on the target? How much benefit we get from a build that produces a varying output, compared to the benefit of having a single universal script. The version of vcsh I'm currently running is only 674 lines long, curious how much buildtime configuration is needed that couldn't be determined at runtime.

Happy to explain further, share my bootstrap method, and discuss solutions and tradeoffs.

What do you think about having a generated generic/bootstrap edition script attached to releases along with the source tarball? This would allow distro packages and any systems with special constraints to build from the source tarball as usual, but also leave a single-script artifact in plain sight that is compiled without paths (i.e. calls git not whatever was detected on the host at build time such as /usr/bin/git). This could even be fetched using GitHub's latest keyword without knowing a version tag.

Do you have a better idea where to publish such a script?

Unpacking a release tarball that contained a generated generic script sounds good, as it wouldn't introduce any more dependencies other than tar.

As I mentioned before I'm curious what the difference would be functionally between the generic script and the generated script, I suspect there's some ideas and plans for enhancements that would benefit from that, I imagine my usage of vcsh is pretty basic. What's the benefit of having fully qualified pathnames for things like git compared to resolving them using $PATH?

[alerque]

Yep, that's my band-aid fix, which I'd expect to be workable for the foreseeable future, as that version of vcsh satisfies all of my personal needs. Just wanted to chime in here to see if others had similar concerns and see if I could contribute to a solution.

That's also why we tagged a safe harbor release before doing anything that might be breaking. Frankly the usage where people just assume that a curl of a raw script from a HEAD commit is never going to break is not a good thing. People that want their scripts to not break should at least download from a tag instead of a branch tip!

One example of why I'm glad this broke this way is that we're considering an overhaul of the alias system. This might break some things for people to, but I'd rather they be forced to start using tagged commits now, then later be forced to upgrade to that change explicitly via a version bump rather than having their assumption that the HEAD commit will always do exactly the same thing have some weird edge case with their aliases that wreaks havoc.

How similar would they need to be?

Right now, similar enough that the paths to comm, cmp, git, grep, and sed are all the same and compatible versions.

Yeah, awk is pretty ubiquitous, although there can be some differences between awk/mawk/gawk and the like.

Just a PS to my last comment, awk is currently only used during build, not run time, although we've discussed it being a better option for some run time things so that might change soon.

I think the "bootstrap edition" idea has some legs. Would the generated script be universal or have variances depending on the target?

See above, except that the generated version we would have to post online would have to rely on $PATH instead of hard coded paths.

How much benefit we get from a build that produces a varying output, compared to the benefit of having a single universal script. The version of vcsh I'm currently running is only 674 lines long, curious how much buildtime configuration is needed that couldn't be determined at runtime.

Runtime dependency detection has several drawbacks. Speed and how hard it is to setup are not really huge factors, just a side effect really. The biggest difference is not for people that want to run a bootstrap version like you do, the big difference comes for distro packagers. Having build time configuration allows for a lot more robust packages, ones that don't break even if the user-space tooling changes. As one example, one usage (that I hate but became quite common recently) is replacing Git with some wrapper tool (Hub or others). This is typically done in user space, but having system tooling that relies on runtime dependency detection can get messed up by these sort of things. Other cases like my personal one, sometimes my user space path includes an unstable/devel git version I'm testing. I don't necessarily want vcsh to switch to that, I want it to use my system's stable default Git install.

There are a couple things this enable such as being able to install more than one version, rename it to something else, etc. All of that needs to be build-time config.

Supporting BSD was also something that was seriously broken before that this fixes.

TL;DR; This is better for distros that package vcsh and peopse that use distro packages, maybe not better for people that just want to scp vcsh hostname:/usr/bin, but a bootstrap edition should solve the latter case too.

Unpacking a release tarball that contained a generated generic script sounds good, as it wouldn't introduce any more dependencies other than tar.

I wasn't thinking of even having it in the tarball, that would make it harder to inspect before running (something you should always to before curl | bash!), I was thinking of just having it as a bare script file attached to the releases.

[alerque]

One more thing: My comment above about the advantages of a build time configure step for the script itself were only about the script itself, but I should note that's only a small part of the picture. Even if some people's usage is just as a bare script, that's not all there is. There is a man page (which is a bug-bear to build), test fixtures, autocomplete setups for many shells, sample hook scripts, etc. All these things need installing, and that end of things was a mess before. It wasn't getting autocompletions where they needed to be on many distros, some distros had a hard time getting man pages, and a host of other ailments. Having build time configs fixes this and will make distro packaging beyond the initial targeted Debian much easier.

I'm still not saying we can't come up with a bare script version for people that want it, but just pointing out that fixing packaging is a big deal and the recent changes do that quite well. Hence we're not just weighing benefits of the final form of the script.

[alerque]

@nferch and @jhgorse Would you guys be able to have a look at my PR in #299 and especially the simulated release on my fork that has a standalone script attached? Since I don't actually have a use case for this and you do it would be good to know if what I worked up actually meets your needs or not.

[alerque]

@nferch et all, we went ahead and merged the PR that adds features to facilitate deployment as a standalone script, but please do chine in if there is still feedback. See especially this mock-up release on my fork for how we expect it to look and this comment for how curl connects ti it. A section has also been added at the end of doc/INSTALL.md with my ideas on how this might be used.

Does this actually match your expectations and use cases? If not please open a new issue with details and we'll look into what we can adjust.

[nferch]

@alerque looks good! tested https://github.com/alerque/vcsh/releases/tag/v1.9.0 by updating my script to curl/wget the new location per #299 (comment), worked on my Ubuntu, FreeBSD and Darwin environments.

[alerque]

To anybody who was subscribed to this issue and cares: the v2 series has officially been released, but I've noticed the download counter for the stand-alone script on the beta tag is still going up — even faster than on any of the stable release tags since.

That suggests somebody in this cover (or perhaps @arndtc) has hard coded the beta tag. It might be time to bump that to a stable tag and/or consider the latest keyword which actually works now (since it is tied to stable releases, the beta was marked as pre-release so the latest keyword was unavailable).

[nferch]

Is there a stable or perhaps v2 tag to track? Otherwise I'd be inclined to lock the latest v2 release which isn't optimal either.

[alerque]

@nferch The latest stable release is available using GitHub's latest keyword. We will mark betas and anything else as pre-release (as in this beta was marked)

Note to directly access the download using that keyword you have to reverse the path segment where the tag would be with the 'download' segment:

• tagged url: https://github.com/RichiH/vcsh/releases/download/v2.0.2/vcsh-standalone.sh
• latest url: https://github.com/RichiH/vcsh/releases/latest/download/vcsh-standalone.sh

No there is not a v2 or other series tag yet. If it would really be useful we could consider them (ala the way GitHub Actions handles versioning) but I'm not a huge fan really. Sticking to tagged versions you have hand selected isn't that bad an idea as long as you periodically check for updates.

[nferch]

Just tested the tagged URL and it worked as expected, as in perfectly well.

Hoping this will allow me to set it and forget it-- which doesn't stand in the way of the appreciation and admiration for this project and its contributors, it truly is unique and awesome.