Grant opportunity

[cloudhead]

Hey there,

As discussed briefly with @Byron, the Radicle Foundation would be interested in helping fund the development of Gitoxide, with the hopes that Gitoxide could replace our usage of libgit2 down the line. Let's discuss the specifics here! There is another thread, though perhaps out-of-date, started by @FintanH here: #51, that could inspire this discussion.

What I'd like to figure out is:

• What is the scope of the work that could be funded by Radicle
• What are the terms and timeline
• Who would be carrying out this work

Let's discuss!

[Byron]

Hi Alexis,

it's good to hear from you here to talk about this exciting prospect. Gitoxide would definitely love to become part of Radicle and help getting there faster. As a matter of fact, the client connection code already knows Radicle, despite not being able to handle it yet or call handler programs that are configured elsewhere.

The current task forecast is split into two parts. The first is all about generating packs and at least to some extend getting a server going. It is a direct consequence of @joshtriplett sponsorship and it's what I am (trying) to focus on.

The other line of work was there originally and deals with all it takes to successfully clone a non-bare project. Nobody is working on this as far as I know.

Continuing with 1) on the agenda, I suggest for someone familiar with Radicle's transitioning plan or needs to have a look at the current tasks forecast and see where there is intersection or where there are missing pieces. From there one might be able to construct a new tasks list which includes Radicle related priorities and work. It's certainly fine to add an entirely new train of work and we find overlaps together.

As work is no fun if it doesn't get done, it might be worth looking into who can contribute or point 3) on the agenda. Me aside for the moment, @edward-shen comes to mind and I invite him to chime in as he sees fit. @avoidscorn as the contributor of the commitgraph crate should certainly be mentioned as well along with the cordial invitation to chime in as needed.

As I myself am already set on a task I am much less available to focus on anything else, but will always take time to facilitate this project and help everyone involved to get desirable outcomes.

I hope to reply here once a day and am looking forward to hearing your thoughts and findings.

[kim]

Hi, Radicle core developer here. To start off, let me be that guy and dial down the expectations a bit: there are no immediate plans to move away from libgit2, and I doubt there will be any for a long time to come. I think we all agree that a git implementation in Rust would be a great addition to the ecosystem, and if the Radicle Foundation can help pushing gitoxide -- both as an implementation and as a community -- forward towards that goal, that would be great. Let's be real, however: there's a lot of work to be done until then. Alright, with that out of the way, I can elaborate a bit on why I wouldn't depend on gitoxide any time soon, perhaps that can help framing the scope. For Radicle, we basically have two kinds of uses for a git library: 1. Offline, read-mostly access to a normal git repo That's for things like browsing the current tree, the history, refs, diffs, etc. Maybe occasionally create a commit or a tag through a GUI. In short, anything you'd expect from a non-CLI git client. 2. The peer-to-peer node ([radicle-link]) For 1. I think the only real nuisance of libgit2 are its C idiosyncracies (which are, for a good part, git's idiosyncracies). I haven't exhaustively reviewed gitoxide for the feature set we would need there, maybe it's on-par, maybe it's not there yet. Either way, I see no pressure to move away from libgit2. Naturally, I care more about 2. There, the single biggest pain point has been to find ways to safely use blocking code in an async/await context (and it's not pretty). However, that's more a problem of async/await than of libgit2 in particular. It is certainly the case that there are inefficiencies due to not being able to share repository pointers between threads, and it's not always obvious without reviewing the implementation which concurrent operations on a repository require external synchronisation. This is an area where gitoxide could shine -- but I'd like to point out that concurrent access to a repository needs to deal with concurrent **processes**, some of which may be from a different implementation, so library level concurrency primitives are only a fraction of the problem. The second thing I'm worried about is the lack of protocol-v2 support in libgit2. We only really need the "client" (that is, fetch) half, there's no issue with running `git upload-pack` on the other side. There is potentially some code in gitoxide which we could reuse, but a. we'll need to deal with the async transport again, and b. we actually need more control over the sequence of commands to run than what I can see is exposed. Another big thing is the refdb. We're dealing with a (potentially large) "monorepo", which will eventually accumulate A LOT of refs. There are large projects which have outgrown packed-refs, and to address that the git community has settled on introducing [reftable] as a scalable backend. For better or for worse: implementing reftable comes at a considerable complexity cost, yet it has all the properties I'd want (it's in many ways a "real" database). The point is, the refdb needs to integrate with the rest of git, and as long as the repository is not _exclusively_ accessed through a single implementation of git, every implementation needs to do what git does. No, it's not in `next` yet, but I believe @hanwen is actively working on it, and the latest version of the series is only a few days old. I would expect that it is _much_ easier to port it over to libgit2, than to implement a Rust version. It is probably the case that we can also benefit from bitmap and multi-pack indexes, but I'm not too concerned about that at the moment. Fourth, the elephant: SHA-256. I don't yet understand all the implications of this, but I'm very sure that it will be quite a mess as the plan is to support both SHA-1 and SHA-256 for quite some time. Again, every implementation will have to do what git does, and the hash algorithm basically touches everything. Aside: until we can start using SHA-256, we rely on the implementation to use "Hardened SHA-1". I'm not sure the `sha1` crate implements that. Lastly, security. Obviously, the major implementations (git, libgit2, JGit) receive more scrutiny, and CVEs tend to consider them. git doesn't have a full-disclosure policy, so that's something to consider. So yeah, looking at the tasks forecast, I'm not sure there's an awful lot of overlap at the moment, unfortunately. I'll definitely try to get some protocol-v2 going sometime soon-ish, and see if anything comes out of that which would be worth contributing back. I'd love to hear your thoughts on the reftable and hash algorithm topics. [radicle-link]: https://github.com/radicle-dev/radicle-link [reftable]: https://github.com/google/reftable

[Byron]

Hi Kim,

sorry for the late reply! Yesterday I took a first look and the liberty of adjusting the desired capabilities based on the pain points/requirements that you mentioned. Please don't take the absence of a point a silent neglect but as me not getting it or having missed a point. So if there is some checkboxes missing, please let me know or even better, send a PR to add them. Gitoxide wants to be the right choice for Radicle or anyone who wants to build great tools on top of it, so we will be as accommodating as possible.

[…] that they don’t seem to match with what gitoxide provides today nor in the near future.

I see quite a lot of overlap and am confident that we will figure out a path forward here, just to throw that in :).

I think what I’d like to see is a team of maintainers committed to the long term and involved in the git community.

Would we have this conversation just two weeks ago, the maintainers would have been only me and quite a high bus-factor co-efficient. With the arrival of @edward-shen we are on our way to become healthier, even though we have a way to go to become balanced and now each others code equally well.

Even though I am aware that this is nothing you asked for, and I am not saying this out of defence, I may gladly state that I am far from done with gitoxide, and I need it to become complete and stay for my own sake. Having maintained GitPython for more than a decade I know that Gitoxide is the logical next step I am now able to take as the time, the tools, the environment and support system are finally right.

My negligence of 'the git community' probably means I am not actively part of it, but I am confident this can change once gitoxide reached a level of feature completeness that allows it to ask: "what's next?", and propose or implement improvements that may one day even be adopted by the canonical implementation.

If money can help, then let’s make it.

To me it's a first to be paid for open source work and I consider myself very lucky for that. This luck can hopefully be shared, for instance using the GitHub sponsorship system or external platforms to motivate people to join in who otherwise couldn't. But truth to be told, I have no idea how to do that correctly and am happy for any suggestions.

My intuition is the following:

• be transparent both with time spent as well as project income
• be transparent with who else is paid and how much
• use money to pay those who have already contributed on their own time to help them keep doing that (but I don't know if I can serve as 'trust' to distribute funds using the sponsors system)
• allow maintainers to get personally sponsored on the project to build their own income stream

Thanks to generous and helpful @joshtriplett I will soon be able to get a first taste of this and hope the experience makes me wiser than what my probably flawed intuition tells me.

About the hashing algorithm

Everything in the codebase is prepared and adjusting this enum should be all to add support (clearly famous last words but nothing I would be afraid of at all).

However, I don't know if the sha1 crate supports strengthened forms of it, but I feel confident that the lack of it could be addressed with them (especially if there is funding to share…)

Regarding reftable

I took a look at the spec and wasn't terribly frightened, but again, that might be out of ignorance. Maybe I feel that way because dealing with the git protocol was a major pain and yet it was possible to get it done even though it is less well defined/specified (especially V1). Also I am glad @edward-shen is willing to take on the challenge, and I have absolutely no doubts he can deliver (and hope he will allow me to share my thoughts during different stages of the development :D).

Protocol V2

The client implementation is considered 'done' enough to act as a client with a lot of control over what's happening. There is the fetch::Delegate which can be used to control server interaction in a protocol agnostic way but geared towards fitting V2 perfectly.
I recommend trying it out based on the existing implementations and see what it can do already. Change requests are welcome, I don't think there is anything that can't be done here (or that we would be unwilling to do as long as it's adhering to the protocol).

Working with the local repository

Here Gitoxide is certainly most lacking right now, and I'd agree it's best to focus on eliminating the pain points Radicle has first. Working with a local repo is probably quite alright currently making feature parity the less valuable goal here.

Conclusions

To sketch a path forward I would love to empty checkboxes for missing features in the crate-status file in the probably case I missed something, the more complete the better.

And while you try implementing a client using gitoxide, I would be happy to provide support with any questions you have up to having a face-to-face call if that gets you started more quickly. Funnily enough it's what I offer anyone who is serious enough because our work here is that important to me. And yes, it's fun, too :D.

Furthermore I would love to hear your thoughts about running funded open source projects and how to do it, simple solutions are preferred and I don't mind being the BDFL for now with the goal of working towards something more democratic.

Have a good weekend everyone ☀️.

[happybeing]

Great to see this discussion. Although my own project (a decentralised Git Portal ) is not using gitoxide yet due to a need to use things that are complete and minimise the code I have to write, I see gitoxide as preferable and necessary in the long run, so I remain interested.

Although I'm reviewing my own project's future I want to highlight another funding option for gitoxide. Work on gitoxide could be funded as part of Git Portal or any other client project that is deemed important to be among the early applications on Safe Network.

Following a generous donation, MaidSafe and the Safe Network community have begun administering a brand new find of about $2m, see Announcing the BambooGarden Fund.

I'm confident that my Git Portal would qualify for this, but personally am not interested in a funded development (neither to be funded myself nor in charge of funding), so it would need to be managed by someone else or to be a different Safe related project requiring gitoxide functionality. Rust is a very popular choice for Safe/MaidSafe so even though it's not an easy step to obtain funding this way (as I'm not going to do it!), I wanted to flag it up in case it might be tapped for the benefit of gitoxide.

TL;DR: a way to fund gitoxide would be for someone to create an application for Safe Network (eg take over Git Portal or create something similar), and apply to the BambooGarden committee for help with the app, and gitoxide as a component.

I may resume work on Git Portal, or maybe not. But if I do it would be at a much lower level of activity. It's a big project for one person, which is why I'm not sure it's worth me continuing.

[Byron]

Thanks a lot for sharing! I took a closer look at the Safe Network, their mission, and of course Git Portal and believe to have a decent understanding now.

That said, porting Git Portal's backend to Rust + gitoxide to have a chance to receive funds seems absolutely worth it. For gitoxide this means to add WASM support to run in the browser, and of course a whole lot of additional work to have Git Portal use it. Even though that seems like a lot of extra work I have no problem to try to take in an application that uses gitoxide - it's an explicit goal to be suitable for applications that are similar to GitHub but bring something new to the table. Besides that, I am personally interested in decentrialized tech but couldn't even dabble in it yet.

Of course by then I hope there are more people who think similarly as there certainly isn't enough time to do all of that at least as it stands right now 😅.

TL;DR: Thanks for sharing, so good to know, I am definitely interested to give that a shot when WASM support should receive extra funding and an incentive to implement it.

[happybeing]

Maybe we should have a separate thread, but just want to mention that they are keen on supporting WASM for the API in preference to lots language bindings, and would be diving into WASI if it had networking support. I know the project, its vision, founder, the dev team well (online) having been involved daily since 2014 and active with code since a year 2015. I also did some work with Tim Berners-Lee to try and connect Solid using Safe Network as the backend, and made a proof-of-concept demo for this.

If you think there's a way to move forward with this don't hesitate to begin a conversation and I'll be happy to update you with Git Portal, my situation and try to answer any questions related to Safe Network etc.

[Byron]

Agreed! All this is very exciting, and I think to do this option justice it's best to start the conversation when there is a little more time for me to spend on it. Even though I don't know when that will be, I am sure it will happen. Thanks again for chiming in!

[kim]

Thanks for the responses.

I'll rephrase a bit: gitoxide does not currently fit the needs of the Radicle project.

There are two ways to think about changing that:

1. The gitoxide project continues at its own pace and with its own roadmap. It may receive funding from the Radicle Foundation and/or others, in the hopes that it will become a full-featured git library implementation in the future (the benchmark being JGit). Radicle projects would re-evaluate if it is a fit at some point, or if filling in the gaps would be within their respective scopes.
2. radicle-link becomes an "early adopter" of gitoxide, and takes it as a liability in addition to libgit2 and git proper. In this scenario, the roadmap would be adjusted to the needs of radicle-link, and individuals would commit on doing that work. In exchange, they'd receive funding, feedback, testing, and contributions. If that goes well, Radicle remains committed to the project, and deploys resources to eventually meet the goal outlined in 1.

I'm not at this point interested in 1., I don't know if @cloudhead is.

If 2. is an option, our requirements are as follows (in this order):

1. A "sans I/O" formulation of the git protocol-v2

   Goals:

   • Can be used with any I/O paradigm/runtime (incl. sideband)
   • Allows to execute commands in any order, and read the response
   • Allows to inspect/modify state on application level
   • Customisable packfile negotiation (also as pure functions)

   Non-Goals:

   • Degradation to v1
   • High-level APIs
   • packfile building / indexing

2. reftable support

   • Rust port of the C reference implementation as inlined into git.git
   • Extensive testing, both for correctness and compatibility
   • Integration as a backend for the standard refs API, including transactions + symrefs
   • In case the feature doesn't land in git.git master soon, tooling for
     • introspection
     • maintenance (compaction)
     • migrating an existing repository

3. Anything that might be missing or incomplete of the following:

   • packfile building / indexing / compaction (does not require async I/O, but wouldn't hurt either). Must be atomic / respect file locking conventions.
   • advanced indexing (bitmap, multi-pack)
   • basics: read/write any type of git object, revwalks
   • efficient reachability checks
   • full read/write support for config, atomic on-disk updates, transparent reload when modified by another process
   • "Hardened SHA-1"
   • fine-grained control over the lifetime of memory-resident structures (mmapped or otherwise)

4. SHA256

   This is actually not very complete in git.git, when compared to what's outlined in Documentation/technical/hash-function-transition.txt. With the features in 1.-3., we should however no longer need to rely on git(1) at all. The task would thus be to implement this specification, most likely in close collaboration with the upstream efforts.

Please let me know if this sounds interesting.

Thanks,
Kim

[Byron]

Thanks for taking the time to formulate ways forward.

To me 1) is like coming back to this discussion at a later point in time to re-evaluate the situation. Of course, by then more of Radicle's code will rely on Git2 which doesn't make using gitoxide that evolved without it in mind any easier.

Thus 2) is what I want to work towards to, knowing that getting there will take time and effort for everyone involved. To show my commitment a little more, I made my involvement with gitoxide public. It's marked as part-time as this is the best I can do right now, hoping it will get closer to full time in the coming months.

There is always a risk that I will do other jobs next to it if they pay better, but gitoxide will always get some of my time as I want it realized as much (or maybe even more) than you do just because I think Rust is best suited to power technology that most devs on the planet use every day.

With that in mind, I am happy to juggle potentially different interests coming from sponsorships, but that also means that the course might not be as direct as it otherwise could be. For now that doesn't seem to be a problem though.

For instance, with 2.1) I suggest us having a guided review session of what's already there (if you didn't check the client-code already) and we see what's missing to truly fit the needs. That would probably go both ways as I can do a better job if I see how you are currently doing things on protocol level and learn how it can be improved.

If the risks mentioned above are acceptable we could find a way to go into a form of 'trial' mode in the hopes the relationship can grow gradually from there. From the beginning of the next week I should be able to spend much more time with gitoxide than I currently do, which is 2h on a good day :D.

Cheers,
Sebastian

PS: For brevity I refrain from commenting on the other technical requirements. There shouldn't be a problem with any of them, even though I believe there will be plenty of questions from my side once we get there :) .

[kim]

by then more of Radicle's code will rely on Git2 which doesn't make using gitoxide that evolved without it in mind any easier.

Not necessarily: Radicle is envisioned as being composed of many applications, some of which will need “just git” functionality. radicle-link specifically, however, needs only a small fraction of the git surface, but access to plumbing functionality targeted towards “git on the server”-like usage patterns. A big part of that is async/await. It’s okay to spawn_blocking until you need to re-enter the runtime from blocking code (because of network access) — and then it ripples through other parts of the system. As I don’t see the ecosystem moving away from async/await (much the opposite), anything which requires us to jump through such hoops again is not a net improvement. This is in conflict with early design decisions in gitoxide — if we’d try to fix our issues ourselves, that would be unlikely to be upstreamable, and vice versa. That’s the dilemma.

There is always a risk that I will do other jobs

Sorry, I had wrongly assumed that more specifics had been communicated before this thread was started. As you can guess from my laundry list, it's not going to get done by buying someone a coffee. On the contrary, there is enough stuff to keep someone busy full-time for a while, and that’s not something out of the question. Full-time can have different forms, and we can discuss what would suit. Whatever works will, however, have an impact on the direction of this project, or be its own effort. That’s how I see it 🤷 I agree that it might be best to let this sit for a while, and discuss specifics outside of this forum. If people following this want to get in touch, the email on my profile should work. Thanks

[Byron]

Even though I intend to respect the desire to let this sit until some clarifications were made on alternative channels, I hope to address a technical aspect here nonetheless.

This is in conflict with early design decisions in gitoxide — if we’d try to fix our issues ourselves, that would be unlikely to be upstreamable, and vice versa. That’s the dilemma.

Gitoxide tried to make a decisions based on early knowledge and experience with async/await and that might as well be wrong. The finding was that most of git is CPU bound, and thus far only the client side is the only one implemented and I saw now reason to go async for the single connection I thought these would have.

Servers generating packs would also be CPU bound, but it's easy to see them juggling a lot of connections and thus it certainly makes sense to go async for that one nonetheless. Whenever possible CPU bound code implements Iterators and checks for abort requests, so wrapping them into a well-behaved future should be quite possible. The blocking crate for example makes wrapping iterators very easy.

Whatever works will, however, have an impact on the direction of this project, or be its own effort. That’s how I see it 🤷

I think it makes a lot of sense to keep efforts to get complete (as in JGit) Rust implementation of Git focussed on a single implementation to reach that goal faster, and I am happy to do what it takes to make that happen. Ultimately it's not about what I think is right because I don't have all the answers, but to find good solutions to technical problems collaboratively. Ideally these will be useful to plenty of applications without much delay, for a round-house-kick win for everyone.

[kim]

Servers generating packs would also be CPU bound

That's fair yes. I still think the protocol and transport crates would require some substantial rework. I don't know what the best way is to deal with refdb accesses, which we interleave with network calls. Technically, these need to be spawned, but that seems like a context switch nightmare. We have to live with this I guess, but you see where this is going?

[Byron]

If the transport and protocol crates would be easiest to work with by being async, then so be it. Right now they are fully separated from CPU bound operations, which users of these crate would still have to deal with and bridging them into the async world.

The latter I consider solved as long as…

1. …the blocking tasks don't run in the executor thread
2. …can be cancelled by dropping their future

1. is easy to do with the likes of spawn_blocking or the blocking crate. The latter makes 2) easy for iterators which (I think) become Streams, thus being abortable after each next() call is done. 'Easy' it is only if that iterator is used within an async context.

This won't work for long-running operations that take iterators as inputs, as these would be spawned_blocking and then run to completion no matter what. The missing link here seems to be wiring up these input iterators with the spawned future so that dropping the future stops the iterator (or even makes it fail in some way). As an alternative to that, I can imagine upgrading the Progress trait to allow checking for cancellation.

Even though it's not 'clean' in the sense that everything is async, I think it can never be in the moment computation is done based on what's coming in from or going out to the network - CPU's will be busy and I think spawn-blocking + cancellation within the blocking code is a decent way of doing that.

I hope this spreads some optimism :).