Testing Framework Enhancements

[ugo94490]

Hi everyone,

Recently we had several ideas for improvements with @ThibFrgsGmz and we wanted to post them here to get some feedback from the community.

Testing Framework

After spending some time using cFS, we thought that using an open source unit testing framework might be a good idea to get a more complete and accurate testing environment.

We already have some open-source frameworks in mind:

• Unity Testing (especally for embedded software and quoted in the book: Test-Driven Development for Embedded C)
• Google Test

Using an open-source framework would then allow the cFS development team not to take care of the update and fix bugs coming from the test framework.

This part would be done automatically by another team / active community like for Google Test for example.

We would be delighted to hear your opinions on these various improvements and to discuss with you in order to improve them or to add others

[astrogeco]

Have you looked at our UTAssert?

[mbenson1]

I liked UTAssert. Yes it was a simple framework but the real benefit was that the CFS included almost all the hooks and stubs needed for application unit testing. With any test framework you use, you're still going to have to manually write a lot of setup code or stubs for functions not under test. Expensive tools like LDRA help automate and decrease the time to write those stubs. Usually, the default stubs included with CFS were sufficient for most testing but when they weren't, you could easily substitute it with your own stub at runtime without forking the code. So rather than spending a lot of time writing stubs for code not under test, you spent most of your time focused on the code under test. And since UTAssert was so thin with little abstraction, it was unobtrusive when stepping through code in a debugger. You could reuse a lot of those hooks and stubs for other frameworks. Unfortunately, all of the hooks and stubs were deleted somewhere around version 6.7.

[skliper]

Really the stubs/handlers just moved so that they are now owned by the API provider, such that the hooks/stubs can now be kept up to date with API updates without crossing repo boundaries. We've also added an auto-stub generation tool, the capability to both provide default "handler" logic or completely replace it, while still maintaining the "hook" support for legacy handlers. We've also added support for non-default return type handling within stubs. It really has improved consistency and provided for much more test flexibility (the test can own any unique handling, the auto-generated stubs really do provide very nice default behavior), and it's worth taking a look at the updates! It also means the UtAssert tool/repo is extremely stable (we've only added a couple helper macros recently) vs having to frequently adjust it based on "external" API updates.

[mbenson1]

Can you point me to any documentation? I've used the new unit test system before and didn't have any problems until unit tests failed, causing me to start using the debugger. With all the macros and runtime abstraction in the new unit test system, its very difficult to follow the flow of execution.

Is the entire API usable now? The last time I checked an official release, the working UtAssert hooks and stubs were deleted before the new one was fully completed.

Also, are all the product line applications updated to the new unit test system and officially released yet? The last time I checked a release, none of the applications had been updated to the new unit test system. CFE unit tests built and ran, but none of the product line application unit tests would build.

[skliper]

Can you point me to any documentation?

Right now the best documentation is in the API headers - https://github.com/nasa/osal/tree/main/ut_assert/inc. I did just add an issue to process those with doxygen to create an actual document at nasa/osal#1123. If you benefit from learning by example OSAL, cFE and sample_app all utilize UtAssert now in various ways (both with stubs for coverage testing and in functional testing w/o stubs).

Some coverage links (compiled w/ stubs) - https://github.com/nasa/osal/tree/main/src/unit-test-coverage, https://github.com/nasa/cFE/tree/main/modules/es/ut-coverage (each cFE module "owns" it's own coverage test), https://github.com/nasa/sample_app/tree/main/unit-test/coveragetest, https://github.com/nasa/PSP/tree/main/unit-test-coverage (this is new-ish, PSP coverage support!)

Some functional links (compiled against the real stack/libraries) - https://github.com/nasa/osal/tree/main/src/unit-tests, https://github.com/nasa/osal/tree/main/src/tests, https://github.com/nasa/cFE/tree/main/modules/cfe_testcase (actually ends up building a cFE app that functionally checks cFE APIs)

I've used the new unit test system before and didn't have any problems until unit tests failed, causing me to start using the debugger. With all the macros and runtime abstraction in the new unit test system, its very difficult to follow the flow of execution.

It is more complex than the original unit test system, but really it's just assert logic and stub logic in the end. Asserts end up being very trivial, with quite a few helper macros to support consistent reporting to make automated test coverage verification easier (we actually compare the code API documentation against the test report to ensure full "black box" coverage with the tests).

Is the entire API usable now?

The entire concept has changed. UtAssert does not own the stubs, it just provides the logic supporting stubs. The API owners should be supplying the stubs (and if they want they can also include default handlers), but if they don't we also provide a tool to automatically generate stubs at: https://github.com/nasa/osal/blob/main/ut_assert/scripts/generate_stubs.pl. So if an API owner doesn't provide them, you can generate them on the test side using the API headers. The autogenerated stub logic gives you return override capabilities (for any return type, a big improvement!), call counts, and the ability to add a "handler"... really just a function pointer to test-owned logic that can do whatever you need the test to do for that particular API (has access to all the parameters, control of the return, can share globals w/ the test code, etc). Some API owners provide default handlers, but those can be overridden if the "special sauce" doesn't meet your test needs (no need to force the API owner to change to meet your testing requirements, a big improvement!)

Really this means you don't have to wait for every API you use to provide you stubs/handlers that support your test. Nor are all those stubs/handlers owned by a single repo (which would historically "break" whenever any API throughout the entire ecosystem changed, or when someone wanted different behavior it had to be negotiated across the entire user base... no longer issues and a HUGE reduction in technical debt).

Examples of provided stubs/handlers: https://github.com/nasa/osal/tree/main/src/ut-stubs, https://github.com/nasa/cFE/tree/main/modules/core_api/ut-stubs, https://github.com/nasa/PSP/tree/main/ut-stubs

Also, are all the product line applications updated to the new unit test system and officially released yet?

Nope! We don't own everything in the cFS ecosystem, so in the wild you'll see a varying level of support provided out of the box. This is something we can't change, which is why we provide a way to autogenerate the basic stubs and provide handlers from the context of the test. The framework (cFE/OSAL/PSP) utilizes it all throughout, and was updated with the autogenerated stubs in the Caelum development cycle (not released "officially" yet). A significant set of the GSFC apps are very close to getting released that all use the current unit-test system and are compatible w/ Caelum (and will provide autogenerated stubs).

I'm sure you can follow the thread here - the point is to NOT have to rely on the API owners to provide support (or incur delays based on external unit test dependencies.) You now can easily control and support what you need for your test (autogenerated except for whatever special sauce you need for your test, doesn't get much easier than that.)

[mbenson1]

Then who owns the CFS applications originally developed by NASA and being used on Orion and Gateway? NASA no longer owns them? Are they just being abandoned now?

…

On Fri, Jul 30, 2021 at 8:20 AM Jake Hageman ***@***.***> wrote: Can you point me to any documentation? Right now the best documentation is in the API headers - https://github.com/nasa/osal/tree/main/ut_assert/inc. I did just add an issue to process those with doxygen to create an actual document at nasa/osal#1123 <nasa/osal#1123>. If you benefit from learning by example OSAL, cFE and sample_app all utilize UtAssert now in various ways (both with stubs for coverage testing and in functional testing w/o stubs). Some coverage links (compiled w/ stubs) - https://github.com/nasa/osal/tree/main/src/unit-test-coverage, https://github.com/nasa/cFE/tree/main/modules/es/ut-coverage (each cFE module "owns" it's own coverage test), https://github.com/nasa/sample_app/tree/main/unit-test/coveragetest, https://github.com/nasa/PSP/tree/main/unit-test-coverage (this is new-ish, PSP coverage support!) Some functional links (compiled against the real stack/libraries) - https://github.com/nasa/osal/tree/main/src/unit-tests, https://github.com/nasa/osal/tree/main/src/tests, https://github.com/nasa/cFE/tree/main/modules/cfe_testcase (actually ends up building a cFE app that functionally checks cFE APIs) I've used the new unit test system before and didn't have any problems until unit tests failed, causing me to start using the debugger. With all the macros and runtime abstraction in the new unit test system, its very difficult to follow the flow of execution. It is more complex than the original unit test system, but really it's just assert logic and stub logic in the end. Asserts end up being very trivial, with quite a few helper macros to support consistent reporting to make automated test coverage verification easier (we actually compare the code API documentation against the test report to ensure full "black box" coverage with the tests). Is the entire API usable now? The entire concept has changed. UtAssert does not own the stubs, it just provides the logic supporting stubs. The API owners should be supplying the stubs (and if they want they can also include default handlers), but if they don't we also provide a tool to automatically generate stubs at: https://github.com/nasa/osal/blob/main/ut_assert/scripts/generate_stubs.pl. So if an API owner doesn't provide them, you can generate them on the test side using the API headers. The autogenerated stub logic gives you return override capabilities (for any return type, a big improvement!), call counts, and the ability to add a "handler"... really just a function pointer to test-owned logic that can do whatever you need the test to do for that particular API (has access to all the parameters, control of the return, can share globals w/ the test code, etc). Some API owners provide default handlers, but those can be overridden if the "special sauce" doesn't meet your test needs (no need to force the API owner to change to meet your testing requirements, a big improvement!) Really this means you don't have to wait for every API you use to provide you stubs/handlers that support your test. Nor are all those stubs/handlers owned by a single repo (which would historically "break" whenever any API throughout the entire ecosystem changed, or when someone wanted different behavior it had to be negotiated across the entire user base... no longer issues and a HUGE reduction in technical debt). Examples of provided stubs/handlers: https://github.com/nasa/osal/tree/main/src/ut-stubs, https://github.com/nasa/cFE/tree/main/modules/core_api/ut-stubs, https://github.com/nasa/PSP/tree/main/ut-stubs Also, are all the product line applications updated to the new unit test system and officially released yet? Nope! We don't own everything in the cFS ecosystem, so in the wild you'll see a varying level of support provided out of the box. This is something we can't change, which is why we provide a way to autogenerate the basic stubs and provide handlers from the context of the test. The framework (cFE/OSAL/PSP) utilizes it all throughout, and was updated with the autogenerated stubs in the Caelum development cycle (not released "officially" yet). A significant set of the GSFC apps are very close to getting released that all use the current unit-test system and are compatible w/ Caelum (and will provide autogenerated stubs). I'm sure you can follow the thread here - the point is to NOT have to rely on the API owners to provide support (or incur delays based on external unit test dependencies.) You now can easily control and support what you need for your test (autogenerated except for whatever special sauce you need for your test, doesn't get much easier than that.) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#308 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHW7LDVRN5FWQIHBPNIYC5DT2KRKVANCNFSM5AVKQRXA> .

-- *Mathew Benson* CEO | Chief Engineer Windhover Labs, LLC 832-640-4018 www.windhoverlabs.com

[skliper]

There is no single "owner" for all the open source cFS apps (aka maintenance group and/or funding source), there is no restriction that they all need to be owned by NASA nor is there a requirement for every app to be updated and released open source with every framework release. It's an open source ecosystem. You'll need to check in with whoever is (or isn't) maintaining whatever app you are interested in to see if it is "abandoned" or not.

At my level (cFS framework) we work within the constraints of stakeholder funding in support of specific missions, if you are interested in a higher level picture as to how NASA will (or won't) support the open source cFS ecosystem in general that's above my pay grade. I can only support/release/maintain what I have the resources to cover, and open source is "best effort". As I'm sure you know, within the context of a real mission (Orion/Gateway/whatever) the process is much more rigorous with firm delivery schedules and formal/funded maintenance plans, but that rigor isn't 1 to 1 with the open source releases. Not every app used by real missions can or will be released open source, nor do all the funded maintenance activities translate/apply to open source releases. Really, "you get what you pay for" so your expectations related to open source are a bit out of sync with the current business/management reality.

[skliper]

That said, if you are interested in more detailed information related to Orion/Gateway (what is being supported/updated/maintained and who's doing it), it's probably more appropriate to work through those official channels than to discuss in an open source forum :)

[mbenson1]

I'm only asking because there are a lot of projects that use the NASA developed applications, not just NASA projects. I would have expected that support would continue if the applications are still being used internally, but it doesn't sound like they are. I don't know of anybody that uses CFS without any of the product line applications. Losing support for applications like SCH is significant. Thanks for the info.

…

On Fri, Jul 30, 2021 at 10:13 AM Jake Hageman ***@***.***> wrote: That said, if you are interested in more detailed information related to Orion/Gateway (what is being supported/updated/maintained and who's doing it), it's probably more appropriate to work through those official channels than to discuss in an open source forum :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#308 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHW7LDQNCH5FL5O6ZAJSG5DT2K6QFANCNFSM5AVKQRXA> .

-- *Mathew Benson* CEO | Chief Engineer Windhover Labs, LLC 832-640-4018 www.windhoverlabs.com

[astrogeco]

Just to add some more context, we are NOT dropping support for all the apps. Like @skliper said, many of the "GSFC Apps" are actively maintained and we hope to release Caelum compatibility updates soon. That being said, everything is best effort and just because an app is being used in a NASA project does not automatically mean that it will be released or maintained at the open source level. I agree that we could be more transparent on this. We are happy to have more thorough discussions: reach out to the program team at [email protected]

[skliper]

"Losing support" really isn't the correct term since there never was/is a guarantee of open source support from NASA. As a user of the open source, it's really important to take that into account. We released Apache 2.0, best effort, w/ all that entails (no fitness for use). There's elements released open source that may never get another official release. We release open source to benefit the community, not because there is an maintenance contract or funding supporting open source under which you can assume there's any additional long-term "support". Do we plan on continuing to release updates open source for elements where it makes sense and it's possible for us to do so? Sure, but if you need something more formal you need more than open source discussions to get that sort of agreement/guarantee in place.

[mbenson1]

Sorry. I thought it was public knowledge that CFS is being used on Orion and Gateway, given its on the NASA public website. https://www.nasa.gov/feature/goddard/2021/core-flight-software-chosen-for-lunar-gateway https://ntrs.nasa.gov/api/citations/20190000011/downloads/20190000011.pdf On Fri, Jul 30, 2021 at 10:20 AM Mathew Benson ***@***.***> wrote:

…

I'm only asking because there are a lot of projects that use the NASA developed applications, not just NASA projects. I would have expected that support would continue if the applications are still being used internally, but it doesn't sound like they are. I don't know of anybody that uses CFS without any of the product line applications. Losing support for applications like SCH is significant. Thanks for the info. On Fri, Jul 30, 2021 at 10:13 AM Jake Hageman ***@***.***> wrote: > That said, if you are interested in more detailed information related to > Orion/Gateway (what is being supported/updated/maintained and who's doing > it), it's probably more appropriate to work through those official channels > than to discuss in an open source forum :) > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#308 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHW7LDQNCH5FL5O6ZAJSG5DT2K6QFANCNFSM5AVKQRXA> > . > -- *Mathew Benson* CEO | Chief Engineer Windhover Labs, LLC 832-640-4018 www.windhoverlabs.com

-- *Mathew Benson* CEO | Chief Engineer Windhover Labs, LLC 832-640-4018 www.windhoverlabs.com

[skliper]

Use of cFS certainly is public knowledge, but version details, maintenance plans, funding sources, etc is beyond what's really appropriate for me to detail in an open source forum and really doesn't apply in this context (nor do I even have all that info). I'm not saying they aren't fair questions, they just aren't mine to answer. Also the relation of those activities to open source releases isn't always straight forward, so better to get that information directly from the projects themselves based on their plans/requirements/etc. I have no idea if Orion plans to open source any app updates based on any of their maintenance activities or what version of the framework they plan to maintain compatibility with. I seem to recall they diverged from open source long ago, so there may be no applicability/compatibility with open source at all.

[mbenson1]

In response to the OP, which version of CFS are you using? I have a 6.7 branch on our CFS repo with the original UtAssert and hooks and stubs reintegrated and updated to the CFE 6.7 API. I have maybe about of the applications and unit tests updated to build and pass with the updated CFE API. Its temporarily shelved, but you're welcome to it. I don't know anything about Unity Testing and very little about Google Test. I would be interested to know if any of those hooks or stubs could be used with any other test framework.