Figure out a good way to rate packages

[bluddy]

I'd like to have some scale to measure the quality of packages.

1. how well maintained they are:

a. Well maintained
b. Average maintenance (issues pile up, but it's ok)
c. Not maintained, issues pile up, bugs appear over time.

2. How mature the library is. Can it carry out its purpose, or does it have many holes.

a. Mature (lacking support for needed features, serious bugs etc)
b. Immature (lacking some features, buggy)
c. Unusuable (too many bugs, lacking too many features)

3. How up to date the library is with modern standards and needs. A mature library may be good enough for what it does, but it may not support the current needs of users in the domain, or may not use good paradigms. As an easy example, the Arg module of the stdlib is mature, but it uses outdated paradigms, including global references.

a. Modern (using all modern standards and maybe even pushing the envelope)
b. Somewhat outdated (doesn't fully keep up with standards)
c. Antiquated (too outdated to be of use in a modern context)

What do you guys think? Can we summarize these 3 axes in some convenient way? Unfortunately each of these starts with M (maintenance/maturity/modernity), but maybe we can express these with other words.

[loxs]

Looks good to me, except with the Mature's explanation, which seems wrong to me.

[bluddy]

OK here's another attempt at coming up with relevant dimensions. I deliberately tried to distinguish each section with a different letter, so we could have the 'BME' scale.

1. (B)ugs:

• A. Well maintained (no bugs or issues)
• B. Average maintenance (issues pile up but concern minor bugs)
• C. Buggy, with serious maintenance issues (issues pile up and haven't been touched, some concerning serious bugs)
• F. Unmaintained: the library is unusable due to maintenance issues.

2. (M)aturity: does the library cover the features it sets out to provide; is it usable? is it used seriously by many users?

• A+. Mature feature-wise and used by a large number of users.
• A. Mature, but not used heavily in production. No missing features though.
• B. Immature: features may be missing.
• F. Unusable: too many features are not present or not functional.

3. Meets user (E)xpectations: does the library meet users' expectations? Does it implement modern standards? Is it easy to use? Is it well documented? Does it perform well? Does it use modern software engineering paradigms? (This is like the "miscellaneous" category).

• A: Meets all expectations, including documentation, design, the latest standards etc.
• B: Meets main expectations, but may be lacking documentation or a little unintuitive.
• C: Has serious issues with regard to user expectations. May use old paradigms and standards or perform badly.
• F: Unusable due to not meeting user expectations at all.

[kalouantonis]

I have some issues with the definitions we're using.

• Well maintained

Does having no bugs or issues mean that the project is well maintained? This could also indicate that the project is dead or has no users. I think there are very few popular projects that don't have bugs or issues.

I'd also like to note that for more esoteric languages like OCaml there are quite a few projects that haven't had changes in a year or more, but that doesn't mean they don't suit their purpose.

• Maturity

Specifically you mention a "large number of users". How do we check that and what actually constitutes "large" in this context? Is it dependent on the number of stars, forks or watchers? OPAM seems to have basic statistics, so maybe that could be a metric?

awesome-scala uses stars as the main metric for popularity, even making the most stared projects bold. I'm personally not fond of this approach.

[pmetzger]

I think "Maturity" is too un-obvious a word. I'd suggest as categories:

1. Is the library complete (that is, is it finished and ready for real users.)
2. Is the library well designed (that is, is the chosen API clean and usable, i.e. something people would feel happy using.)
3. Is the library well documented (can you figure out how to use it only reading the docs without needing to read the code?)
4. Is the library on a modern and public source control system? Libraries that are on github pass, those that have no obvious public source code control location fail, in between is in between.
5. Is it well maintained (that is, are bug reports rapidly and responsively paid attention to, and do releases happen at appropriate intervals. Note for a very boring library this might not be very often at all unless bugs are reported.)
6. Only for libraries that do things like providing OCaml interfaces to C libraries or provide code to implement things that deal with externally maintained standards (like HTML), is the library up to date (in that it interfaces to a recent version of the library, implements a recent version of the standard, etc.)

[bluddy]

Ooh. I like these, @pmetzger. Much better than mine.

We can fold source control into maintenance: I'm happy taking the position that nowadays, open source software that's not on public source control is also not properly maintainable.

As for the docs, it'd be nice to note if docs are available online. Perhaps that gets the highest grade.

So that gives us 5 dimensions I'm pretty happy with. How do we summarize these in a way that allows us to avoid writing an article per project? Perry, do you want to add a rubric just as I did? The way I see it right now, we can annotate the projects, and add a comment or 2 if there are specific issues worth mentioning.

[pmetzger]

Let met think about it for a bit. (And if I forget to think about it, poke me.)

[mars0i]

I like the direction this is going a lot. Just one thought. Would it add too much complication to allow, in some cases, to rate different, "dimensions" of a package differently?

I keep coming back to the ocaml_hdf5 example. Is it complete? Well, as a wrapper for all of the functions in the HDF5 API, it is, afaik. Then there's another part that's an attempt to provide high-level convenience functions built out of the API functions. That is not complete, and maybe it's buggy. Would be possible in some cases to say something like

ocaml_hdf5 - HDF5 API: Complete, ...
ocaml_hdf5 - high-level convenience functions: Incomplete, buggy, ...

This could get out of hand. Giving a package multiple ratings like this should only be used when there's a sufficiently good reason.

[bluddy]

I think that's fair. We can go into more detail in these kinds of cases, just as you did here.

[ghost]

I would lean more towards having opam do this automatically, though not as accurate as manually deriving it. I think npm metrics are a good start, even if crude.