Proposed Initiative: "Maintainer Experience" of OpenSSF #169
Comments
|
Thank you for taking the time to not only report the problems you are seeing but also make concrete suggestions on how we might go at addressing them. This is very valuable! We do have one WG that is maintainers oriented: Securing Software Repositories https://github.com/ossf/wg-securing-software-repos Have you looked into this? I'm curious to know whether you didn't run into it (which wouldn't surprise me, there is a lot going on and it's not easy to know what's out there) or if you know about it but didn't think it fits the bill. If so, why? Thanks again! |
|
@lehors - Securing Software Repositories is definitely worth looking at. However, that WG focuses on "maintainers of software repositories, software registries, and tools", e.g., PyPI/pip, npm, RubyGems/bundler, system software registries / {apt-get,dnf}. So that WG does not directly apply to the maintainers of most OSS projects. Of course, most widely-used OSS projects end up in some repo, so a typically OSS maintainer would want them to work well, but that's not the same thing. The best practices WG generates a number of guides & materials specifically for maintainers, and I think that's the closest match at this moment... but it often focuses on work we're doing, not "here are finished products you can use". There are a lot of interesting proposals here! A "landing page" for maintainers, in particular, sounds like a great idea to me. We could point to our guides, the education materials, sigstore, etc. |
|
I think that the “maintainer/developer” persona is one of the core foci of our stalwart band of security-friends. I love the idea of a landing page, and materials dedicated to helping kickstart or assist a developer in their personal security journey. I agree with David, the BEST WG feels like a *slightly* better fit, and we even can augment some of our targets for the EDU.SIG to address getting materials composed/collated/presented for this group.
Cheers,
CRob
Director of Security Communications
Intel Product Assurance and Security
From: David A. Wheeler ***@***.***>
Sent: Tuesday, May 30, 2023 9:47 AM
To: ossf/tac ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [ossf/tac] Proposed Initiative: "Maintainer Experience" of OpenSSF (Issue #169)
@lehors<https://github.com/lehors> - Securing Software Repositories<https://github.com/ossf/wg-securing-software-repos> is definitely worth looking at. However, that WG focuses on "maintainers of software repositories, software registries, and tools", e.g., PyPI/pip, npm, RubyGems/bundler, system software registries / {apt-get,dnf}. So that WG does not on the maintainers of most OSS projects. Of course, most widely-used OSS projects end up in some repo, so a typically OSS maintainer would want them to work well, but that's not the same thing.
The best practices WG<https://github.com/ossf/wg-best-practices-os-developers> generates a number of guides & materials specifically for maintainers, and I think that's the closest match at this moment... but it often focuses on work we're doing, not "here are finished products you can use".
There are a lot of interesting proposals here! A "landing page" for maintainers, in particular, sounds like a great idea to me. We could point to our guides, the education materials, sigstore, etc.
—
Reply to this email directly, view it on GitHub<#169 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQRFDLD37V47WIYKFUEZMGLXIX26FANCNFSM6AAAAAAYS7XDVU>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>
|
|
Right, echoing some of the other comments here:
Does this distinction make sense? |
|
Also, thank you so much for taking the time / effort to read and consider this, and also for your patience as I muddle my way through an unfamiliar process. :) |
As chair of this WG, +1, absolutely right! |
|
We are VERY happy to have you participating to make our efforts more accessible for everyone! Thank YOU, Angie!!
|
|
Ok, I can see how the Best Practices WG might be a better fit. This WG has primarily been oriented towards developing material like guides though. I wonder what's better: focus that WG more towards the maintainers needs as described by @webchick or create a new WG dedicated to just that. But for that matter I agree with the idea of adding a dedicated landing page idea and in fact it will help people find their way to whichever WG we decide is where they should go. :-) |
|
I think the persona (to issue #165) with the problem @webchick describes is a bit different than the Best Practices WG. As I understand it, Best Practices WG is about content development for the 100k+ developers writing code that's open source, and not knowing they should do something like set up GitHub Branch Protection from the get-go. This seems to be about what and how the OpenSSF helps an "OSS Core Maintainer." For example, sigstore was set up for Kubernetes because lead participants of the relevant k8s SIG were interested and kicked it off. That kind of difference makes me think this is a new effort (which ties into similar "maintainer/contributor experience" problems the @ossf/governance-committee has been discussing, so I think we've got some kismet here...) |
|
As additional endorsement on the need for this, when the OpenSSF Maintainer Workshop was held (invite only, small audience and also whose intent was to learn and capture these sentiments), the need articulated here, the observations and experience summarized are all key points that arose in those discussions. Unfortunately (like many things) the volunteers who worked to execute the event where very limited in their time and couldn't commit to provide a comprehensive analysis and write up from all the notes we captured (even though we really wanted to) - but we did capture notes that would potentially be beneficial as an initial gauge on interactions and engagement with maintainers. @SecurityCRob and @bobcallaway were both excellent Moderators in those discussions. Those notes still exist but will need scrubbed a bit as the event was under Chatham House Rules. Once that is done it could be provided publicly. If you were part of that effort, please let me know if anything i stated is misrepresented as my participation was time-limited and i may not have the follow-on actions correct. |
|
As a maintainer of open source that has been looking at the OpenSSF for quite some time now, I will support what @webchick said. Most of the "best practices" presented by the OpenSSF are, quite directly, a no-go. At this point, I have written off all that comes out of the OpenSSF and the wider "securing supply chain" ecosystem as "the classic infosec that never matters for us or tries to help us" and I am not alone, most of the maintainers I know have made this analysis too. I want to be clear. I am not saying that because I want to take down the OpenSSF. I say that to give you real feedback from the ground that what @webchick is doing is being really nice to tell you that, in its current shape and operation, the OpenSSF did not manage to talk to "weekend warriors" maintainers and did not manage to set roots in these communities. Her recommendations and the work on personas are definitely a path to getting better results there. Note in particular that k8s is definitely not representative of the "weekend warrior" persona as a group of maintainers and contributors 😄 They do a great job, just really different persona. |
|
I'll loop Katherine Druckman in today during the Governance committee meeting. She is helping lead our new DevRel committee that seems to be an excellent place to address @webchick 's observations and concerns. BEST WG will be glad to collaborate on assisting to advocate for the developer/maintainer perspective, and we'll seek to get other groups to participate too. |
SecurityCRob
added
documentation
|
I also come to OpenSSF from a maintainer perspective. I'd be happy to participate in follow-up work. |
|
All: Here's cut at a "Developer landing page": Suggestions welcome. We could link to it from the top-level openssf.org page if we think it's useful. This page does not resolve this issue, but I think it's a step on the path. |
|
FYI, there's a best practices WG issue to vote to post the "developer landing page" on the main OpenSSF website: ossf/wg-best-practices-os-developers#195 |
Agreed, this is a great start. Could we also link to s2c2f? Or perhaps we want a separate "for organisations" page that would link to s2c2f? 🤔 |
|
related to #84 |
To be fair the fuzzing branch is actually useful. They actually talk to actual maintainers, try to cover even unusual use cases (if it's doable) and so on. Unfortunately snapshot fuzzing (or any other custom stuff tailored to specific projects) is unlikely to ever be supported there but it isn't the end of the world if people fuzzing projects have their own clusters. That being said that branch existed long before OpenSSF was created so it's probably safe to say that it isn't representative of OpenSSF in general.
Agreed. That WG group appears to operate on the assumption that everyone is uneducated and has no clue what they are doing and that sort of attitude isn't exactly helpful if the idea is to win the "hearts and minds" of maintainers. I'm not sure the proposed solutions can work as long as the "consumer" persona is much more important to OpenSSF but it's good to know that it's at least trying to move in the right direction.
It would be great if it was possible to download things like https://openssf.org/soss-vision-brief/ without filling out those forms. |
|
I think the WGs that work on Scorecard and the Badge don't entirely agree on their purpose and meaning, and I personally don't agree at all with the consensus, yet I use these almost every day. What isn't useful is the idea of an objective quantity of security, because the accuracy is too low to make a big difference. What is very useful is guiding my roadmap as a security TPM for FOSS projects. I treat the criteria as checklists. By running the metrics I figure out what I do and don't need to work on. For example, this month I am cleaning up CodeQL alerts as a result of working through https://bestpractices.coreinfrastructure.org/en/projects/6358#analysis and getting a failing score on the static_analysis_fixed metric. What I would want from the WGs is to reframe their outputs as a guide for security-related workflows. For example there could be a Kanban template with the first swim lane prefilled with cards corresponding to metrics. I have something similar for OpenChain, which has a comparable role for getting an OSPO organized. |
Agreed. I think the problem is that they were never designed to be integrated into maintainers' workflows. Up until recently scorecard just dumped json and ignored maintainers' feedback completely. The human-friendly UI was added eventually but it took a weird amount of time and at least one critical project planning to throw scorecard out. The scorecard action is useless in terms of actually preventing issues and so on. (In theory this discussion can be moved to the corresponding bug trackers but I don't think it's going to be addressed. They no longer auto-close issues though so at least it can be kept there) |
|
@jkjell and I have taken over the reins of the Security Toolbelt SIG in the Best Practices WG, with ongoing support from @SecurityCRob, who helped jumpstart the SIG. While we are not representative of the "Maintainer experience" we are interested to have our work contribute to and enable the maintainer experience within the broader OSSF community. Perhaps the links below, in conjunction with efforts of the DevRel community can help this conversation thread? While we are still in the process of refining a roadmap and contributions, we have done quite a bit of work to date on the following topics, some of which support the maintainer experience:
|
|
Thank you for the links. As far as I can remember I took a look Security Toolbelt SIG and DevRel community some time ago. Security Toolbelt SIG looked too abstract to me and the DevRel community seems to have focused on conference talks. I'll try to take a closer look later.
Off the top of my head things like scorecard and allstar are reactive and can't be used to prevent, say, dangerous workflows that can compromise repositories from being merged. For example I'm not sure why people have to spend their time on looking for things that can be caught and remediated automatically when PRs are opened. It can't detect dangerous workflows where data flows have to be analyzed either and so on and so forth. All those things have been reported and ignored because it's much more involved to make those things work than to, say, generate markdown checklists. I'll take a look at the personas, use cases and so on once I've gotten round to it. Thanks again for the links! Edit: I'm sorry but I can't see how "Ursula the Upstream Maintainer" can affect anything. She is unlikely to attend OpenSSF meetings (and that's kind of sad because based on my observations all the decisions are made there so she is effectively excluded from the conversation regarding what sort of stuff is dumped onto her thanks to a bunch of automated tools). Generally I'm not sure how the following objective is going to be met
given that even critical projects are currently ignored. To judge from https://docs.google.com/document/d/1hO6NuSiNr_7PO1QTYsB6qzcS8pAFW7p_6JT2y0XL5Nk/edit#heading=h.z9vrmtiy2usx the DevRel community should
but without attending a lot of meetings it's a one-way street anyway. If that sort of dynamics can be changed somehow maybe it can work. Either way it's a lot to process so it might take a while. |
|
Ok so i am going to try something slightly different here. I am going to explain what it is to be a maintainer first. This is ofc limited to my experience and knowledge of talking with others, subjective, anecdotal, etc. YMMV. Then i will try to show based on this why the OSSF stuff does not help me. Persona: Diana, the weekend warrior.I maintain a couple of small packages and contribute new medium size but impactful features to my underlying ecosystem. Think a compiler optimisation for floats that takes a few months of work and extremely niche knowledge to get right. This is a really common and critical profile. I am in a loose network of other niche people doing the same in my ecosystem. We usually get something like 2h per month to spend on FOSS. Sometimes up to 4h, sometimes less. 1 to 2h per month ate dedicated to simply updating base dependencies. This is when it is just a few patches or minor versions. Sometimes up to 4h are taken for this. Sometimes a big major version in an important dependency happens and it takes us 10h to fix, over a quarter. This means that nearly all our time is spent handling dependency stuff. Basic stuff. Not security emergencies, or anything like that. Releasing a new version that just is kept up to date basically. I do not have more time to give. Life is what it is, i have family, a job, friends, etc. My tests are shit. I know it. I keep wanting to make them better. Fuzzing? I could not find the time to fix the bugs found anyway. Reading security oriented material? When? Most of my time is spent fighting my build and dependency management tools. This is a constant hell, they break all the time in new obscure ways. I, by luck, keeps finding an incantation that makes them produce something that works. It uses non of the security stuff. I even go out of my way with my build file to deacticate all sandboxing, because it keeps breaking my builds. I suppose there are ways around it, but they would need me to understand how these things work and also to adapt them to my need. I do not have the time. The basic tools breaks already too much and eat my time. Also wait. Tons of user reports!! But i only did a minor version update for that dependency which had an innocuous changelog?!?? Wait they messed up their versionning. This was API breaking change but they had no idea. I would have the same mistake. It is not obvious. Aaaargh. Well i guess i will have to tell everyone to use the old version for the next 6 months while we spend our 6h per quarter all fixing it. Hope no emergency security patch come through during that time. At least the yubikey i have used for the past 7 years is used for my packages now... Oh no. Too old. Not supported. I guess i need to find money for a replacement now ConclusionThere. You have it. Now, you want better security? Help me get some of the time fighting the tools back. Help me be able to install and build my software without needing shell hacks and root access to the whole machine and the internet. Help me just standardise the system root certificate store so i can access it. All these things that will give me the time to even think about what the OSSF talks about. Edit note: there are some projects looking at this. Not from the security crowd, with highly limited funding. Rust as a whole is one, with tools there like semver-check but also just cargo. And other ecosystems too. On the other hand, autotools has no maintainers for a decade. I understand the data you look at. The research you look at. The way you see the world. I understand why you think you are helping. And you are! At the margin. But this is not the margins we need anymore. Stop looking at surveys. Sit down, live the life of the weekend warrior. Literally. Listen to them in person. Do qualitative work. And remember. Most weekend warrior are not on surveys. Not at conferences. Because their employers do not support them and they do not have the time and money to go there. I will be, probably, at FOSDEM this year. Probably. In the EUC DevRoom. I say probably. It is in 2 weeks and idk how i will pay for travel or lodging there yet. Plus it will take all my FOSS time budget for the year. Or i cut other important stuff in my life for it. And pay the prize. To hear again and again people touting how they are helping with all kind of work that does nothing for me. But happy to talk more there! |
|
Thank you @DianaOlympos for taking the time to put your thoughts into this thread. As always, they are very insightful, and I appreciate it. I will take your persona and merge it into our existing work within the toolbelt - https://github.com/ossf/toolbelt/tree/main/personas as your real-world perspective adds a lot of colour and value there. I will not be at FOSDEM, but I think several others within the Foundation would be, let me ask around and see and perhaps we could arrange a short talk between you and them. I'd be interested to hear more about which tools you and other maintainers commonly use and think through how we can influence things to help make them more reliable and invisible to your work. I completely understand the frustration of wrestling with systems/access/misbehaving tools/etc. The TAC is discussing this issue today (23Jan), and ideally we'll be able to get some folks to collaborate more actively on making this a priority for us. |
|
FYI, a few changes in the best practices badge project that I think might be relevant:
|
|
That's nice but personally my badge experience would be greatly improved if those badges were actually vetted and maintained. It's hard to get past dead links from 2017 and try to figure out why projects with, say, no fuzz targets anywhere (or links to their downstream fuzzing infrastructure or anything) meet the "routinely fuzzed" criterion with unit tests. I think they are great in terms of raising awareness but the "Consumers of the badge can quickly assess which FLOSS projects are following best practices" part is questionable. Either way I was kind of hoping that DianaOlympos's comment (which is so well put that I don't even know what to say) could shift the discussion from badges and checklists. |
I've merged a first pass at your persona feedback @DianaOlympos . Can you please see if this meets what you were describing: https://github.com/ossf/toolbelt/blob/main/personas/README.md#name-diana-the-weekend-warrior The Toolbelt team has recently agreed that this Weekend Warrior persona will be one of the key focuses of their deliverables in the coming months (to streamline, simplify, and help integrate tasks, data, etc for this persona. |
|
Will have a look later, if i have said nothing in a few days, feel free to ping again please |
|
Today on the DevRel / Community chat, @Cheukting created a #maintainer-experience channel. :D The idea is for this to be where we direct folks we meet at conferences / meetups / etc. who have an interest in being consulted / included in initiatives from a "user experience" testing POV. |
|
Also, I finally had time to catch up on this discussion, and +1000 to @DianaOlympos's extraordinarily accurate description of the vast majority of open source maintainers' experiences. If I could pick a single "North Star" metric for the OpenSSF to focus on, it would be: Time to Critical Project Maintainer Value == as infinitesimally tiny as possibleGiven sufficient focus on that, not only would Weekend Warriors stand a fighting chance of learning about and adopting some of the great work that's gone into OpenSSF's various tools and resources, but there would also be beneficial "halo" effects of improving things for all other Open Source Maintainers, as well as the learners just dipping their toes into Open Source for the first time. In practice, this means things like:
I've been super impressed here btw with both open source maintainers' willingness to share "real world" feedback with the OpenSSF, and also with the OpenSSF's willingness to engage with some tough comments and provide additional context. We're all ultimately on the same "side" here and most of the comments here have really shown that. Great job, everyone. 🎉 |
For various reasons some people (myself included) can't use Slack (and most of the communication channels OpenSSF uses) but I'm glad that folks are going to be met there by people who can open thoughtful issues like this one. It would be great if the DevRel team could convince TIs/WGs to use asynchronous means of communication like issues on GitHub and commit message a bit more. Some projects like OSV are great at posting updates but some projects refuse for whatever reason even when asked explicitly and move discussions to meetings.
I agree it would be great but I think it would be nice if TIs/WGs did some sort of research first. One example would be https://github.com/ossf/sbom-everywhere/pull/27/files#diff-84d5c6f22596cd94bba61aea654d191da533d7016ef2682e2805ddcbdfefc397R11. I don't know if that plan went anywhere but at least it shows that that WG is aware that there are a lot of different ecosystems, a lot of different ways to distribute software and is interested in actually doing things before giving out untested guidance. They also talk to people who have been doing those things long before 2021 so they know where it doesn't make any sense to shift those downstream responsibilities to upstream open-source projects for example. |
|
FYI #330 to make getting/staying involved in TIs easier. |
|
@webchick: now that #330 to make getting/staying involved in TIs easier. is open, should we make sure these comments are carried forward in that issue and bring this one to a close. There has been really great discussion here. If we keep it open, what work remains to help close it out in your opinion? |
|
/vote |
|
Gitvote was added as a tool to test for stream lining the TI Funding process. Community members can show their support by also voting, however only the "TAC" GH Group's votes will count. The current passing threshold is 70% and the committee is the TAG GH group. All these parameters can by fine tuned or changed here |
Vote statusSo far Summary
Binding votes (0)
|
|
@riaankleinhans I don't think we need a vote on this issue. I have greatly appreciated the dialog here. @SecurityCRob, now that you are the Chief Architect, and I know there are persona conversations related to this issue on your plate, how should we proceed to close out this issue? Is an acceptable solution to work through your role and partnership with TAC/DevRel Community? We can close out this issue, and use it as a reference for future dialog. @webchick with you being an initial contributor, how do you feel about this issue disposition? |
|
I don't think anything has substantially changed. OpenSSF is still focused on consumers and that xz response didn't help to win hearts and minds of maintainers either. The things discussed here were either archived or are on their way to target companies:
and the list goes on. On the bright side https://www.sovereigntechfund.de/ seems to participate in some meetings so maybe they can help by bringing their experience. |
|
/close-vote |
|
I put a link to this issue in a comment for MVSR v3 discussion. |
(I recently attended Open Source Summit in Vancouver, and this issue is based on a convo with Anne Bertucio from Google, who directed me to post an issue here. Apologies in advance if there is a better place / better way to go about sharing this feedback!)
Context: I'm a long-time core maintainer and general cat herder of the Drupal open source project, a member of their security team, and a general security enthusiast since the alt.2600 days. I sort of accidentally learned about OpenSSF's existence last year (and also accidentally learned we are deemed a "critical project"), and since then have spent a few weekends digging around the website and reading a whole bunch. I've attended a couple of town halls, and attended the morning of OpenSSF Day a couple of weeks back. I would rate my knowledge about open source maintainership 9/10, my OpenSSF knowledge maybe 3/10.
So with that in mind, gulp, here goes. :)
Problem Statement
Right now, OpenSSF's activities are presented in a way that is congruent with other Linux Foundation projects: there's a strong focus on the how the OpenSSF is structured, how the governance works, how it is run day to day, what are the recent achievements, and so on. All of that makes total sense, through the lens of being a member-driven Linux Foundation project, and wanting to demonstrate your value to prospective members.
However, by and large, open source maintainers:
Objectives
Taken together, it seems like in order for OpenSSF to broadly win the "hearts and minds" of maintainers, there needs to be focus on (at least) the following things:
(This effort could dovetail nicely with the work going on in #165)
Proposed Solutions
There are many ways to go about this, and I'm sure you would know better than me what makes sense :), but here are some initial thoughts:
"Maintainer Experience" Working Group
Create an OpenSSF Working Group specifically around maintainers and their needs (similar to the End Users group, but for the producers of open source).
This could also manifest as something similar to the TAC but a committee made up of maintainer representatives. (Just watch the time commitment.)
Dedicated "Landing Page" Specifically Aimed At Maintainers
For example, some sort of "call to action" on the OpenSSF website "Maintainer? Click here!" and/or maybe repurpose https://github.com/openssf (which currently appears empty) for this purpose?
The contents of this page would be geared toward maintainers and their pain points, rather than oriented around how OpenSSF thinks of them. (I can assure you that maintainers don't care whether something is a working group or a project ;)).
For example:
Oh, speaking of which...
Dedicated Slack Channel (+Mailing List?) Aimed At Maintainer Relations
Upon joining OpenSSF Slack I was blown away by the calibre of people there. :O However, if I was entering that space as a maintainer, I would quickly reach the conclusion that this is not a space for me, because the vast majority of conversation in there is about promotion/enablement of OpenSSF itself, the Working Groups' day to day toils, etc.
So some sort of dedicated channel(s) where maintainers can come in, introduce themselves, ask questions related to OpenSSF's activities, get answers from someone, meet and network with other security-conscious open source maintainers. Maybe also a corresponding "Maintainer Office Hours" meeting folks could join, and periodic "What is openSSF for maintainers" onboarding sessions (could re-use these at events).
Forge Partnerships With Open Source Security Teams
Open source security projects' teams are overwhelmed and exhausted. Having partnership from an organization like OpenSSF would be a huge shot in the arm, and an excellent, pragmatic way to demonstrate OpenSSF's value in a way maintainers can fully understand.
What could this look like? Maybe a set of "tiger teams" grouped by programming language who prioritize projects with massive ecosystems, helping them to implement things like sigstore, teach best practices like CVEs, etc.
Go Where Maintainers Are
In each of the critical projects (once again, probably starting within those with massive ecosystems, such as Python), figure out where those maintainers congregate in "real life" (PyCon), and go there. Even better: co-present with the security maintainers from the previous step; now you have automatic credibility within that community, and a reason for people to show up to your talk.
Assume Zero Knowledge In Community Events
In Town Halls, OpenSSF Day, etc. always assume someone in the audience is hearing about OpenSSF for the first time (if you're doing lots of outreach, they probably are!). The first time you use terms like "SBOM" define for people what that is (and/or have a handy glossary page that's distributed ahead of time). When someone gives a status update on Sigstore, give 30 seconds of context as to what that is and why it's important.
Without this, the risk once again is giving off the strong vibe that OpenSSF is for "insiders" and not for maintainers themselves.
Thoughts?
Haha, sorry for the novel, apparently there was a lot to say. ;) Does the problem statement ring true to you? Is there already an initiative underway in OpenSSF that maintainers interested in smoothing these sorts of partnerships could join and participate in? Is this far out of what OpenSSF sees as their mission and better for some other group to tackle?
Very curious of your thoughts!
The text was updated successfully, but these errors were encountered: