Migrate the project to a proper language

[apflieger]

Needs

• Better testability. For instance, I would like to mesure test coverage.
• Get some abstractions because bash is just a mess.
• Something easier to debug and develop

constrains

The biggest constraint is that people are currently using git-octopus on various OS : macOS, ubuntu, Windows... If we want to use a compiled language, would it be easy to distribute binaries ?

I'm considering python3 with pygit2 for git stuff.

Don't hesitate to give opinions :)

[dbaeli]

+1

In which langages are implémented such extensions ?

Dimitri BAELI

Le 18 nov. 2016 à 18:11, Arnaud Pflieger [email protected] a écrit :

Needs

Better testability. For instance, I would like to mesure test coverage.
Get some abstractions because bash is just a mess.
Something easier to debug and develop
constrains

The biggest constraint is that people are currently using git-octopus on various OS : macOS, ubuntu, Windows... If we want to use a compiled language, would it be easy to distribute binaries ?

I'm considering python3 with pygit2 for git stuff.

Don't hesitate to give opinions :)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[apflieger]

gitflow is in bash and has no tests.

In git/contrib there's bash, python and pearl.

[apflieger]

I looked on homebrew packages prefixed by git-

• git-archive-all : unique python file directly copied to /usr/local/bin via make
• git-cal : unique pearl script directly copied to /usr/local/bin via make
• git-credential-manager : java program. Declare a brew dependency on java. The formula installs a jar that is an artifact of github release. Puting artifact here is manual I guess but it's interesting.
• git-crypt : cpp. installed the same way as git-octopus after compilation
• git extra : bash scripts copied to /usr/local/bin via make
• git-fixup : bash script copied to /usr/local/bin via make
• git-fresh : bash script copied to /usr/local/bin via custom script
• git-ftp : bash script copied to /usr/local/bin via make
• git-gerrit : bash script installed the same way as git-octops
• git-imerge : single python script copied via make
• git-lfs : written in go. Built locally and installed by homebrew
• git-plus : multiple python scripts using shared piece of code
• git-series : written in rust. Runs a local 'cargo' build and install the binaries via homebrew
• git-tracker : build and install standalone ruby package with rake

I skiped a lot of bash projects that are all using the same installation steps

Most of them are written in bash and are standalone files. They are simply copied as binaries. So are python or pearl scripts. A few of them are compiled in go, cpp...

[dbaeli]

Go is interesting (devops tooling, lightweight), would it fit the dev need ?
Python is obvious if not too big or complex for installing this on a CI agent (compatibility version for ex).

Which of those have tests ? :-)

Dimitri BAELI

Le 21 nov. 2016 à 15:25, Arnaud Pflieger [email protected] a écrit :

I looked on homebrew packages prefixed by git-

git-archive-all : unique python file directly copied to /usr/local/bin via make
git-cal : unique pearl script directly copied to /usr/local/bin via make
git-credential-manager : java program. Declare a brew dependency on java. The formula installs a jar that is an artifact of github release. Puting artifact here is manual I guess but it's interesting.
git-crypt : cpp. installed the same way as git-octopus after compilation
git extra : bash scripts copied to /usr/local/bin via make
git-fixup : bash script copied to /usr/local/bin via make
git-fresh : bash script copied to /usr/local/bin via custom script
git-ftp : bash script copied to /usr/local/bin via make
git-gerrit : bash script installed the same way as git-octops
git-imerge : single python script copied via make
git-lfs : written in go. Built in bin within the project and installed in bin by homebrew
git-plus : multiple python scripts using shared piece of code
git-series : written in rust. Runs a local 'cargo' build and install the binaries via homebrew
git_tracker : build and install standalone ruby package with rake
Most of them are written in bash and are standalone files. They are simply copied as binaries. So are python or pearl scripts. A few of them are compiled in go, cpp...

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[apflieger]

I spent some time looking at Go actually. A very interesting thing is that the compiler allow to build binaries for all systems, see. So the distribution of binaries would be quite easy.
It seems to me that the downside of go currently is dependency management but since git-octopus would use only a few (maybe just one), we could deal with it

Both go and Python can be easily tested.

I have doubt on python now, packaging seems not that great

[lesfurets-admin]

Go for go ! ☺

Dimitri BAELI
IT Associate Director

[cid:[email protected]]https://www.lesfurets.com/
12-14 rue Médéric
75017 Paris
+33 1 56 02 36 50
LesFurets.comhttps://www.lesfurets.com/
Facebookhttps://www.facebook.com/hervelesfurets - Twitterhttps://twitter.com/lesfurets

From: Arnaud Pflieger [email protected]
Reply-To: lesfurets/git-octopus [email protected]
Date: Tuesday, November 22, 2016 at 10:41 AM
To: lesfurets/git-octopus [email protected]
Subject: Re: [lesfurets/git-octopus] Migrate the project to a proper language (#25)

I spent some time looking at Go actually. A very interesting thing is that the compiler allow to build binaries for all systems, seehttps://golang.org/doc/install/source#go14. So the distribution of binaries would be quite easy.
It seems to me that the downside of go currently is dependency managementhttps://github.com/golang/go/wiki/PackageManagementTools but since git-octopus would use only a few (maybe just one), we could deal with it

Both go and Python can be easily tested.

I have doubt on python now, packaging seems not that great

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/25#issuecomment-262192809, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACyNWd9DRJtSg5FgOn-_-Z4nQ3w0hWKCks5rArjBgaJpZM4K2tjH.

[dubreuia]

+1 for Go

[durzo]

+1 for leaving as bash.

We use git-octopus in our CI environment and bash means our docker image can remain around 100MB

[apflieger]

Is it a matter of size ? A compiled binary shouldn't be bigger than current scripts (I guess)

[durzo]

size and dependencies. I wouldn't want to install python just for octopus and binaries need compiling unless you will ship precompiled binaries, in which case what about dynamically linked libraries? We are using Alpine Linux docker image for size and its certainly not compatible with redhat/ubuntu precompiled dynamic binaries.

having said that, the current bash implementation depends on perl because it uses shasum and some perl regex logic.. it would be nice to see that move to sha1sum which is provided by coreutils and the search and replace done by sed instead of perl if you were staying with bash.

[lesfurets-admin]

Is Go a blocker in your case ? Dimitri Baeli

…

________________________________ From: durzo <[email protected]> Sent: Thursday, November 24, 2016 2:35:09 AM To: lesfurets/git-octopus Cc: LesFurets.com; Comment Subject: Re: [lesfurets/git-octopus] Migrate the project to a proper language (#25) size and dependencies. I wouldn't want to install python just for octopus and binaries need compiling unless you will ship precompiled binaries, in which case what about dynamically linked libraries? We are using Alpine Linux docker image for size and its certainly not compatible with redhat/ubuntu precompiled dynamic binaries. having said that, the current bash implementation depends on perl because it uses shasum and some perl regex logic.. it would be nice to see that move to sha1sum which is provided by coreutils and the search and replace done by sed instead of perl if you were staying with bash. - You are receiving this because you commented. Reply to this email directly, view it on GitHub<#25 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACyNWUJ3PDqKyi1LzluzZ7q4odhgF6FWks5rBOnNgaJpZM4K2tjH>.

[apflieger]

I agree that dependency on python is problem, especially the version.
Go seems to do the job. In terms of dependencies, the only one that could be interesting is libgit2 for accessing git primitive instead of running git commands but that's not on the table for a first step.

The plan would be to have precompiled binaries. Here's what go claims to be capable of

OS	architecture
android	arm
darwin	386
darwin	amd64
darwin	arm
darwin	arm64
dragonfly	amd64
freebsd	386
freebsd	amd64
freebsd	arm
linux	386
linux	amd64
linux	arm
linux	arm64
linux	ppc64
linux	ppc64le
linux	mips64
linux	mips64le
netbsd	386
netbsd	amd64
netbsd	arm
openbsd	386
openbsd	amd64
openbsd	arm
plan9	386
plan9	amd64
solaris	amd64
windows	386
windows	amd64

We would provide all of them. So do git-lfs https://github.com/git-lfs/git-lfs/releases
Alpine Linux should be able to run linux binaries I guess.

Talking about perl dependency, here's when I introduced it ad4932d There was an incompatibility of sed between ubuntu and macos if I remember well. MacOS allowed only -e "blblbl" whereas ubuntu -e"blblbl" (with no space). That's ridiculous I know, If you have a better solution ;D
Same for shasum, I don't have sha1sum on macos, so shasum looked better for interop

[durzo]

Go binaries should be fine, its just the versions of dynamically linked libraries that I worry about, but it should be OK - happy to give it a try!

I dont use mac so wasn't aware of those missing binaries, makes sense now. The only thing you can do in that case is have an OS detection function first that sets paths to the required binaries for each platform.

see https://gist.github.com/durzo/892c4f81b255c1f24daba92922224db0 for an example

[apflieger]

I hope we can build binaries without any dependencies. Go standard library has regex for sure. For sha1 sums computations, we should be able to embed the code (so do git itself)

[apflieger]

Actually, go binaries are not that lightweight. I compiled a Hello World, 1.5 MB. git-lfs is 3 MB.

We would have something around 2 MB maybe.

[durzo]

if it removes the dependency on perl 2 MB is fantastic!

[apflieger]

Actually 2MB per binary. We have 2, git-octopus and git-conflict

[hvihvi]

+1 for Go as well, it seems to match perfectly the listed constraints.

[apflieger]

I'll might be there tomorrow https://www.meetup.com/gdg-paris/events/235668597/

[apflieger]

Update: I released a first beta version written in go https://github.com/lesfurets/git-octopus/releases/tag/v2.0-beta.1

[danoliv]

Hi, any news about the 2.0 release?

[apflieger]

I still miss the conflict resolution feature. I'm not actively working on it :/
I have a question for you all:
Until know, git-octopus and git-conflict are 2 separate commands based on 2 scripts. This is quite ugly to do in go because it would require 2 different binaries. What do you think about having a single binary with subcommands ?

[dbaeli]

You mean : git-octopus conflict git-octopus merge pattern That's good for me Dimitri BAELI Le 29 mai 2017 à 19:56, Arnaud Pflieger <[email protected]> a écrit : I still miss the conflict resolution feature. I'm not actively working on it :/ I have a question for you all: Until know, git-octopus and git-conflict are 2 separate commands based on 2 scripts. This is quite ugly to do in go because it would require 2 different binaries. What do you think about having a single binary with subcommands ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[apflieger]

Yes.
Actually I consider to have a seperate binary that will not integrate with git directly (not named git-octopus) and we could provide small shell scripts that offer backward compatibility
Given a binary somebinary that would be in the $PATH

git-octopus

#!/bin/bash
somebinary merge "$@"

and git-conflict

#!/bin/bash
somebinary conflict "$@"

Sounds good.

The ambition is to grow somebinary to a bigger project. This is what I talked about to some of you on my https://github.com/apflieger/tie prototype (which has no documentation and is quite obscure to understand, I admit that)

[hvihvi]

As a first step, migrating to cobra (https://github.com/spf13/cobra) would probably help in writing a more consistent CLI.

[apflieger]

yes, that's what i'm using here https://github.com/apflieger/tie. I use libgit2/git2go as well.
That's the plan if I merge this project in git-octopus