POC: alternative for libgit for serverless to fetch git-sourced functions

[kwiatekus]

Description

Prove that we can stop using libgit2 in favor of an alternative library for git-fetching.

• Function CRD contract must be preserved
• Library which allows fetching changes from a basedir (not entire repo) should be preferred
• we must support github and azure devops
• we could support other git providers like bitbucket, gitea, gitlab, etc
• check if the library supports webhooks from git providers (i.e github webhooks )

Reasons
git2go is dead - find alternative
libgit2/libgit2#6849

Related
Part of
#1315

[pPrecel]

I checked the go-git library and It's promising I can say. My example application:

package main

import (
	"fmt"
	"math/rand"
	"os"

	"github.com/go-git/go-git/v5"
	"github.com/go-git/go-git/v5/config"
	"github.com/go-git/go-git/v5/plumbing/protocol/packp/capability"
	"github.com/go-git/go-git/v5/plumbing/transport"
	"github.com/go-git/go-git/v5/plumbing/transport/http"
	"github.com/go-git/go-git/v5/plumbing/transport/ssh"
	"github.com/go-git/go-git/v5/storage/memory"
	crypto_ssh "golang.org/x/crypto/ssh"
)

var (
	SSHAddress    = "..." // shh address from the repo view
	HTTPAddress   = "..." // simple http address from the repo view
	BasicUsername = "..." // username or mail
	BasicPassword = "..." // accesstoken/apppassword (or password as alternative)
	SSHPrivateKey = []byte(`...`) // ssh private key
)

func main() {
	fmt.Println("create auth obj")
	auth, url := chooseAuthAndURL(false)

	// required by Azure Devops (works with Github, Gitlab, Bitbucket)
	// https://github.com/go-git/go-git/blob/master/_examples/azure_devops/main.go#L21-L36
	transport.UnsupportedCapabilities = []capability.Capability{
		capability.ThinPack,
	}

	// test: list (in-memory) remotes
	remote := git.NewRemote(memory.NewStorage(), &config.RemoteConfig{
		Name: "origin",
		URLs: []string{url},
	})

	fmt.Println("list remotes")
	rfs, err := remote.List(&git.ListOptions{
		Auth: auth,
	})
	failOnErr(err)

	fmt.Println("printing rfs")
	for _, rf := range rfs {
		fmt.Printf("Hash: %s\n\tName: %s\n\tType: %s\n\tTarget: %s\n", rf.Hash().String(), rf.Name().Short(), rf.Type().String(), rf.Target())
		fmt.Printf("\tIsTag: %v\n\tIsBranch: %v\n\tIsRemote: %v\n", rf.Name().IsTag(), rf.Name().IsBranch(), rf.Name().IsRemote())
	}

	tmpDirPath := fmt.Sprintf("%s%d", os.TempDir(), rand.Int())
	defer os.RemoveAll(tmpDirPath)

	// test: clone
	fmt.Println("clone", tmpDirPath)
	_, cloneErr := git.PlainClone(tmpDirPath, false, &git.CloneOptions{
		URL:   url,
		Auth:  auth,
		Depth: 0,
	})
	failOnErr(cloneErr)
}

func chooseAuthAndURL(use_ssh bool) (transport.AuthMethod, string) {
	if use_ssh {
		auth, err := ssh.NewPublicKeys("git", SSHPrivateKey, "")
		failOnErr(err)

		// set callback to func that always returns nil while checking known hosts
		// this disables known hosts validation
		auth.HostKeyCallback = crypto_ssh.InsecureIgnoreHostKey()

		return auth, SSHAddress
	}

	// access token
	return &http.BasicAuth{
		Username: BasicUsername,
		Password: BasicPassword,
	}, HTTPAddress
}

func failOnErr(err error) {
	if err != nil {
		fmt.Println(err.Error())
		os.Exit(1)
	}
}

The example application does a few things. First, create an auth structure that allows to connect to the private registries. I've tested such connection for all providers:

• Azure - SSH key and PersonalAccessToken
• GitHub - SSH key and PersonalAccessToken
• Gitlab - SSH key and PersonalAccessToken
• Bitbucket - SSH key and App Password

Everything works fine and without any problems.

The second step is to fetch all remotes to the in-memory storage that simulated fetching all the latest commits for all remotes ( functionality of the serverless-controller ), and the third step is cloning the repository to the tmp dir ( functionality of the building job ). All steps are based on the removed logic.

Requirements:

• First we need the line below to enable using any SSH host because in other cases, the library will return an error message stating that the given host is unknown.

  auth.HostKeyCallback = crypto_ssh.InsecureIgnoreHostKey()
  

• Then we have to configure the library transport layer to use ThinPack capabilities that are required to connect with the Azure DevOps. The good news is that this line works with other Git providers like Github, Gitlab, and Bitbucket.

  transport.UnsupportedCapabilities = []capability.Capability{
    	  capability.ThinPack,
      }