Designing a checklist for success for plugin developers

[neuromusic]

The napari hub team is working to support community plugin quality standards to help developers communicate their plugin’s capabilities. This effort is inspired by similar efforts, such as Github’s community standards checklist, Kaggle’s Usability Rating, and JOSS’s Reviewer Checklist.

We’ve compiled a provisional checklist from interviews with users and developers, covering tasks that plugin developers should take (and we can objectively determine success) to improve the documentation, usability, and reliability of their plugin.

Once we get feedback on the provisional list via our survey, and finalize it, we plan to open PRs to suggest improvements to plugins where we can. Then, in the coming months, we also plan to incorporate the score into the Plugin Preview Page so plugin developers can have more feedback on where they can improve (see timeline below). Eventually, we may consider adding a badge or score to the napari hub interface to help users find plugins that meet a minimal standard.

You can learn more about our objectives and desired outcome below.

Why are we doing this?

This work is grounded in the goal that plugin developers have the resources and feedback necessary to improve the quality of their plugin. Through interviews, we found that developers find the current documentation difficult to write and that the hub has an opportunity to help developers communicate their plugin’s capabilities in a way that biologists easily understand and evaluate. The assistance should make management and support of plugins easier, not add inefficient work to the developer’s load.

Desired Outcome: A Checklist for Success

Our checklist will be based on assertions about individual plugins that indicate that the developer has done critical activities necessary to document their plugin and ensure that it works well for users. These assertions will then be weighted based on relative value to end users to come up with a composite score representing the “completeness” of the plugin developer's effort and the plugin’s readiness for end users.

Once a plugin developer has completed the items in this checklist, a plugin developers should feel confident that:

• new users will be able to quickly understand what the plugin does and whether it is relevant to their needs (documentation)
• new users will be able to learn how to use the plugin on their own, try it out, and see how well it suits their needs (documentation)
• new users will be unlikely to run into technical barriers using the plugin (usability, reliability)
• if users face difficulty with the plugin, it will be clear where they can get help and support (user support)

We believe every item in the checklist should be objective, controllable, user-centric, and independent:

1. Objective: Every assertion must be unambiguously assessable programmatically
   1. Good Example: “A plugin must be installable into the latest version of the napari desktop application”
   2. Bad Example: “A plugin must clearly state what the plugin does and who it is for.”
2. Controllable: Every assertion must be within the locus of control of the plugin developer
   1. Good Example: “A plugin must provide users with a logo.”
   2. Bad Example: “A plugin must have at least 1,000 downloads.”
3. User-centric: Every assertion must provide direct value to end users of the plugin
   1. Good Example: “A plugin must indicate to users where to go for additional help”
   2. Bad Example: “A plugin must conform to PEP8”
4. Independent: Every assertion must be independently valuable & verifiable
   1. Good Example: “A plugin must have an OSI-Approved license”
   2. Bad Example: “A plugin must have EDAM categories if it has a Widget”

Timeline

• Phase 0 (now): Align on the optimal checklist via survey and github discussions
• Phase 1 (May): Help improve existing plugins by reviewing and opening PRs to improve completeness, where possible
• Phase 2 (no earlier than June 1): Give developers feedback their plugin’s completeness score though the Plugin Preview Page and/or API (score will not be surfaced publicly on hub at this time)
• Phase 3: (no earlier than October 1, 2022) Surface plugin completeness scores through a “badge” or similar public-facing indicator on the napari hub

How We’ll Use This Discussion Page

We will keep you up to date on our progress throughout the timeline, and use this as a space to converse on the work.

Any questions or feedback? Let us know below!

[haesleinhuepf]

Hi @neuromusic , hi all,

very cool initiative! I'm wondering if there might be also space for highlighting with which napari-versions a plugin is compatible with. I could imagine that plugins outdate, tests fail when napari and/or its dependencies are updated (see related issue) and at some point plugins don't work anymore with recent napari. I think this could be mentioned on the hub in a way that users can know what's compatible with their napari-version, but also in a way that developers don't feel too much under pressure. On Java/Fiji island, a technology in this context is called the melting pot, it tests if plugins break when dependencies are updated. Just FYI