Considerations for Open Source Software Evaluation

[KingBain]

Some of the folks in my group have been looking at how to Evaluate Open source software and have identified some areas of concern . Its a bit much for me to break apart so I wanted to just upload the doucment. I asked the document author for permission and he gave it.

I hope some of it is useful or can create spin off discussions

Introduction

When evaluating open source software, as with commercial software, there should be two primary considerations: functionality and total cost of ownership. Together the overall value of the software can be evaluated. However the value of open source software may come in a different form than commercial software. The following discussion considers the key issues for evaluating open source software vs. commercial software. It has been assembled from various online blogs and forums.

Considerations

The first consideration is that open source software tends to have narrowly focused functionally. There is an expectation users will integrate and manage multiple products together to create a solution. A large open source solution commonly requires one or more core products and 3rd-party plugins. The work of integration, support, and management tends to fall more on the user than with commercial software. This may not be a concern if in-house expertise is valued as part of the solution.

Second, access to support needs to be considered. The quality of open source updates, documentation, and forums will differ by product from excellent to non-existent. However most open source products do not provide a contractual means by which a specific question can be answered or update obtained in a timely manner. This tends to be a greater issue early in a solution’s lifecycle therefore significant lab testing can help offset concerns in this area. Some companies offer commercial support for open source software however this rarely covers all parts of an integrated solution.

The final major consideration is the benefits of open source software. The most obvious relate to acquisition and deployment. While open source software will have an attached license, they generally allow free acquisition and deployment without restriction. Open source software tends to provide ease of integration through open APIs, data exchange tools, and open storage formats. Also, open source software allows full access to the source code for ease of inspection and modification. For these reasons open source software tends to allow flexibly to scale and change the solution to meet future needs and reduce product lock-in.

Evaluation

When evaluating open source software the availability of information presents a barrier. Unlike commercial software, most open source projects do not have a sales and marketing team creating easy to consume information. This poses a challenge and may require searching the internet for secondary sources, reading technical documentation, and lab testing.

In general, open source functionally should be evaluated in the same manner as other software. Beyond functionality, the key questions are: can the solution be managed, is support available, and do the benefits lead to best value? While these questions apply to commercial software, they have increased relevance for open source and can be harder to answer. The questions below provide a roadmap to answering these larger questions for open source software.

1. What open source product(s) are needed to meet the requirements?
2. Integrating a large number open source products may meet requirements but cause other problems. Can the same requirements be met with a less complex solution?
3. Many open source products assume an industry standard environment. Can the requirements be changed to match industry standards and allow open source products to work with less integration or modification?
4. Can integration be accomplished with open APIs or a well defined plug-in architecture?
5. Is data stored in an easy to manipulate (import/export/inspection/migration) format?
6. Are the APIs and storage formats used by other products and do they use open standards?
7. What are the license restrictions?
8. Are the products normally used in the proposed manner?
9. How large is the user base and is it growing or shrinking?
10. Are there other users, similar to you, using similar solutions?
11. Who is the developer and are they likely to continue with the project? E.g single developer, research group, small company, tech giant, consortiums, …
12. Is one group the only significant contributor?
13. Is one group clearly limiting features? For example, is there one company that is the primary contributor and also offers a commercial “enterprise” version? If so, are they doing so in a logical manner and what would the long term impact be?
14. Does the project have rules for contributing that are supportive of community contributions?
15. Are proposed enhancements from 3rd-partys (pull requests) left unresolved?
16. Does the project follow a documented release management process?
17. Are all parts of the solution released and tested together?
18. Does a recent Long Term Support (LTS) release exist? Is so, what is the planned support duration?
19. Has the project been forked? If so, are the forks taking away from the main project or contributing to it?
20. Are bugs public? Is so, are bugs left unresolved for long periods?
21. Are security related bugs resolved quickly with easy to deploy patches?
22. Is documentation complete and up to date?
23. Are there bugs related to documentation and are they resolved in a timely manner?
24. Are their secondary sources of documentation, tutorials, etc.?
25. Is there a dedicated and active community support forum? What is the tone of the forum? Are forums limited in any way such that you may be forced to buy support or an enterprise version?
26. Is commercial support available? Is so, is it available from more then one source? Is commercial support available from someone other than the primary contributor?
27. Can the product(s) be tested in a lab environment?
28. Do we already have in-house expertise or can it be developed? What will the cost be?

[ShadeWyrm]

Hi! I'm curious if you feel the focus on Technical Evaluation and Technical Options Analysis should be made a part of the 'Standards'.

I have concerns that a focus on technical aspects would do more to damage when the focus is on enablement. Could you let me know if you feel this level of detail is required for what will be considered 'Mandatory' for Departments or if you'd be in agreement for this material to help establish our Playbook (Guidance) which will appear in the open on GCCollab?

https://wiki.gccollab.ca/index.php?title=GoC_Open_Source_Playbook

[KingBain]

I think some of these could be used to add sections to

Procurement of Open Source
and
Use of Open Source Software

If they're too low level then I think it could be put into the playbook, but why move them there. The procurement section looks like it needs content.

The author(not me) high lights how these could be used as a roadmap. Maybe they could be group as principles for easier consumption ?

the OSS standard has a bit of sprawl to it... let me play with the points. I think step one for me was to dump the document in github :D