Jv integration tests for runtime config #1899

JoukoVirtanen · 2024-10-18T18:20:16Z

Description

Integration tests for runtime configuration. The tests create various versions of the runtime configuration file and check that the configuration introspection endpoint returns the correct expected configuration.

Checklist

Investigated and inspected CI test results
Updated documentation accordingly

Automated testing

Added unit tests
Added integration tests
Added regression tests

If any of these don't apply, please comment below.

Testing Performed

TODO(replace-me)
Use this space to explain how you tested your PR, or, if you didn't test it, why you did not do so. (Valid reasons include "CI is sufficient" or "No testable changes")
In addition to reviewing your code, reviewers must also review your testing instructions, and make sure they are sufficient.

For more details, ref the Confluence page about this section.

JoukoVirtanen · 2024-11-06T17:57:59Z

collector/lib/ConfigLoader.cpp

+  } catch (const YAML::Exception& e) {
+    CLOG(ERROR) << "Failed to parse the configuration file: " << config_file << ". Error: " << e.what();
+  } catch (const std::exception& e) {
+    CLOG(ERROR) << "An unknown error occured while loading the configuration file: " << config_file << ". Error: " << e.what();


Molter73

I'm less than half way through, I'll finish the review tomorrow.

collector/lib/ConfigLoader.cpp

Molter73 · 2024-11-06T18:05:56Z

collector/test/ConfigLoaderTest.cpp

@@ -67,12 +67,20 @@ TEST(CollectorConfigTest, TestYamlConfigToConfigInvalid) {
 }

 TEST(CollectorConfigTest, TestYamlConfigToConfigEmpty) {


Please update this test name.

Molter73 · 2024-11-06T18:07:30Z

integration-tests/pkg/collector/collector_docker.go

@@ -41,6 +41,7 @@ func NewDockerCollectorManager(e executor.Executor, name string) *DockerCollecto
 		"/host/etc:ro":              "/etc",
 		"/host/usr/lib:ro":          "/usr/lib",
 		"/host/sys/kernel/debug:ro": "/sys/kernel/debug",
+		"/etc/stackrox:ro":          "/tmp",


Consider creating a separate directory for this like /tmp/collector-test and use that instead of the full /tmp directory.

Molter73 · 2024-11-06T18:16:26Z

integration-tests/pkg/collector/introspection.go

+func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body := QueryConfig(t, collectorIP)
+		var response types.RuntimeConfig
+		err := json.Unmarshal(body, &response)
+		assert.NoError(t, err)
+
+		return response.Networking.ExternalIps.Enable == enable
+	})
+}
+
+func AssertNoRuntimeConfig(t *testing.T, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body := QueryConfig(t, collectorIP)
+		return strings.TrimSpace(string(body)) == "{}"
+	})
+}
+
+// TODO: This should be in its own package
+func AssertRepeated(t *testing.T, condition func() bool) {
+	tick := time.NewTicker(1 * time.Second)
+	timer := time.After(3 * time.Minute)
+
+	for {
+		select {
+		case <-tick.C:
+			if condition() {
+				// Condition has been met
+				return
+			}
+
+		case <-timer:
+			// TODO: This message should be passed in rather than hard coded here
+			t.Log("Timeout reached: Runtime configuration was not updated")
+			t.FailNow()
+		}
+	}
+}


We probably want to move all assertions to a separate pkg/assert or similar.

Molter73 · 2024-11-06T18:19:02Z

integration-tests/pkg/collector/introspection.go

+func QueryConfig(t *testing.T, collectorIP string) []byte {
+	log.Info("Querying: /state/config")
+	body, err := IntrospectionQuery(collectorIP, "/state/config")
+	assert.NoError(t, err)
+	log.Info("Response: %q", body)
+	return body
+}


I'm not 100% sold this is something we need, from a test we could just do:

body, err := IntrospectionQuery(s.Collector().IP(), "/state/config") s.Require().NoError(err)

This is simple enough and you can add the logging if you deem it necessary.

JoukoVirtanen · 2024-11-07T02:26:12Z

integration-tests/pkg/mock_sensor/expect_conn.go

@@ -33,7 +33,7 @@ loop:
 		case <-timer:
 			// we know they don't match at this point, but by using
 			// ElementsMatch we get much better logging about the differences
-			return assert.ElementsMatch(t, expected, s.Connections(containerID), "timed out waiting for networks")
+			return assert.ElementsMatch(t, expected, s.Connections(containerID), "timed out waiting for network connections")


The changes in this file were made here https://github.com/stackrox/collector/pull/1902/files

JoukoVirtanen · 2024-11-07T02:26:25Z

integration-tests/pkg/mock_sensor/server.go

@@ -32,7 +32,6 @@ const (
 // us to use any comparable type as the key)
 type ProcessMap map[types.ProcessInfo]interface{}
 type LineageMap map[types.ProcessLineage]interface{}
-type ConnMap map[types.NetworkInfo]interface{}


The changes in this file were made here https://github.com/stackrox/collector/pull/1902/files

Molter73 · 2024-11-07T12:17:08Z

integration-tests/pkg/assert/assert.go

+func AssertExternalIps(t *testing.T, enable bool, collectorIP string) {
+	AssertRepeated(t, func() bool {
+		body, err := collector.IntrospectionQuery(collectorIP, "/state/config")


We don't need to do this here, but the IntrospectionQuery method needs to be part of a collector manager object, it would change this to be a lot simpler/clearer:

func AssertExternalIps(t *testing.T, enable bool, collector *CollectorManager) { AssertRepeated(t, func() bool { body, err := collector.IntrospectionQuery("/state/config")

I think it can be done in another PR. Maybe we should have an Introspection struct that could be a member of a Collector interface. The introspection object would get the IP address from the collector object.

integration-tests/pkg/types/runtime_config.go

integration-tests/suites/runtime_config_file.go

Molter73

Thanks for splitting the tests up, as silly as it may sound, it's a lot easier to understand what is going on now.

The comments I left now are mostly on moving some comments around to leave the code blocks on their own. This should help reduce clutter a bit more, since the description of what will be done is provided up front and the code is left as clean as possible.

integration-tests/suites/runtime_config_file.go

Molter73 · 2024-11-11T16:56:03Z

integration-tests/suites/runtime_config_file.go

+	// The runtime config file was deleted before starting collector so there should not be any config
+	assert.AssertNoRuntimeConfig(s.T(), collectorIP)
+	// Since there is no config the default is used, which means external IPs is disabled and we should
+	// expect a normalized connection


Suggested change

// The runtime config file was deleted before starting collector so there should not be any config

assert.AssertNoRuntimeConfig(s.T(), collectorIP)

// Since there is no config the default is used, which means external IPs is disabled and we should

// expect a normalized connection

// The runtime config file was deleted before starting collector.

// Default configuration is external IPs disabled.

// We expect normalized connections.

assert.AssertNoRuntimeConfig(s.T(), collectorIP)

This same change can be applied to the other 2 tests.

This was not applied to all tests.

It has now been applied to all tests.

integration-tests/suites/runtime_config_file.go

JoukoVirtanen · 2024-11-12T23:15:17Z

CI results

Run 1
Many Konflux integration tests failed. rhcos failed with

=== RUN   TestProcfsScraperDisableFeatureFlag/TestProcfsScraper
    expect_conn.go:181: 
        	Error Trace:	/tests/pkg/mock_sensor/expect_conn.go:181
        	            				/tests/suites/procfs_scraper.go:72
        	Error:      	elements differ
        	            	
        	            	extra elements in list A:
        	            	([]interface ***) (len=1) ***
        	            	 (types.EndpointInfo) ***
        	            	  Protocol: (string) (len=15) "L4_PROTOCOL_TCP",
        	            	  Address: (types.ListenAddress) ***
        	            	   AddressData: (string) (len=4) "\x00\x00\x00\x00",
        	            	   Port: (int) 80,
        	            	   IpNetwork: (string) (len=5) "\x00\x00\x00\x00 "
        	            	  ***,
        	            	  CloseTimestamp: (string) (len=5) "<nil>",
        	            	  Originator: (types.ProcessOriginator) ***
        	            	   ProcessName: (string) "",
        	            	   ProcessExecFilePath: (string) "",
        	            	   ProcessArgs: (string) ""
        	            	  ***
        	            	 ***
        	            	***
        	            	
        	            	
        	            	listA:
        	            	([]types.EndpointInfo) (len=1) ***
        	            	 (types.EndpointInfo) ***
        	            	  Protocol: (string) (len=15) "L4_PROTOCOL_TCP",
        	            	  Address: (types.ListenAddress) ***
        	            	   AddressData: (string) (len=4) "\x00\x00\x00\x00",
        	            	   Port: (int) 80,
        	            	   IpNetwork: (string) (len=5) "\x00\x00\x00\x00 "
        	            	  ***,
        	            	  CloseTimestamp: (string) (len=5) "<nil>",
        	            	  Originator: (types.ProcessOriginator) ***
        	            	   ProcessName: (string) "",
        	            	   ProcessExecFilePath: (string) "",
        	            	   ProcessArgs: (string) ""
        	            	  ***
        	            	 ***
        	            	***
        	            	
        	            	
        	            	listB:
        	            	([]types.EndpointInfo) ***
        	            	***
        	Test:       	TestProcfsScraperDisableFeatureFlag/TestProcfsScraper
        	Messages:   	timed out waiting for endpoints

The other konflux integration tests failed with either problems pulling the image. Both konflux and non-konflux cos integration tests failed with verifier errors.

Run 2:
Similar errors as above. No procfs test error. Collector also fails for fedora coreos.

Run 3:
rhcos had the following error

=== RUN   TestUdpNetworkFlow/TestUdpNetorkflow/sendto_recvmmsg
    udp_networkflow.go:274: 
        	Error Trace:	/tests/suites/udp_networkflow.go:274
        	            				/tests/suites/udp_networkflow.go:244
        	            				/tests/suites/udp_networkflow.go:105
        	            				/tests/suites/udp_networkflow.go:96
        	            				/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
        	Error:      	Received unexpected error:
        	            	Error response from daemon: container create: creating container storage: the container name "udp-server" is already in use by 0b88d4d870e77d5fed05bd5bad8a1dd1e5391d4e229d7ffb3d92a5bd57d0febf. You have to remove that container to be able to reuse that name: that name is already in use
        	            	create udp-server

I did not see a previous failure that caused this error. There were other failures unrelated to the changes in the PR.

Molter73

Changes look ok, but #1902 still has comments on it that haven't been addressed, if your intent is to merge the changes from that PR with this one, please close it so I can move my comments to this PR.

Stringy · 2024-11-13T10:36:01Z

integration-tests/pkg/assert/assert.go

+	tick := time.NewTicker(1 * time.Second)
+	timer := time.After(3 * time.Minute)


Can these timeouts be configurable via some parameters?

JoukoVirtanen · 2024-11-14T04:02:00Z

Changes look ok, but #1902 still has comments on it that haven't been addressed, if your intent is to merge the changes from that PR with this one, please close it so I can move my comments to this PR.

I have addressed the PR review comments in the other PR.

e8bb38d Starting work 23ee1b0 Working with config introspection endpoint. No checks for connections yet 1b919ee Moved code for introspection endpoint from mock sensor to own directory 91ee35e Testing with an invalid config ae8bf37 Creates a container with an external connection and adds tests for it. Also made changes so that external connections can be checked for 34247e2 Mock server no longer maintains just a map of connections seen, but all connections edf7ca3 Should be able to handle cidr blocks and non-cidr blocks 4b227bd Able to handle the case when there is only port data dbed0a1 Brought in changes to server.go to handle external ips fd24fe8 Apply suggestions from code review 3eb5e48 Cleanup 13f8539 Able to control if connections should be ordered or not. Also added comments f9ea3fc Listening for new connection events instead of using a ticker 6d06416 Using ExpectSameElementsConnections instead of ExpectExactConnections

…ape interval

Co-authored-by: Mauro Ezequiel Moltrasio <[email protected]>

Co-authored-by: Giles Hutton <[email protected]>

openshift-ci bot added the do-not-merge/work-in-progress label Oct 18, 2024

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 316fecb to ae8bf37 Compare October 20, 2024 18:12

JoukoVirtanen mentioned this pull request Oct 22, 2024

ROX-26711: Collector mock server should handle external ips #1901

Merged

5 tasks

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 4b6cf76 to 4b227bd Compare October 22, 2024 20:43

JoukoVirtanen mentioned this pull request Oct 23, 2024

ROX-26726: Collector integration tests should record all connections #1902

Open

5 tasks

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 6d06416 to 075a4df Compare November 1, 2024 18:59

JoukoVirtanen commented Nov 6, 2024

View reviewed changes

Molter73 reviewed Nov 6, 2024

View reviewed changes

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from b446ed0 to 347114d Compare November 6, 2024 19:21

JoukoVirtanen changed the base branch from master to jv-ROX-26726-collector-integration-tests-should-record-all-connections November 7, 2024 02:18

JoukoVirtanen changed the base branch from jv-ROX-26726-collector-integration-tests-should-record-all-connections to master November 7, 2024 02:23

JoukoVirtanen commented Nov 7, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 7, 2024 03:30

Molter73 reviewed Nov 7, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 11, 2024 04:05

Molter73 reviewed Nov 11, 2024

View reviewed changes

JoukoVirtanen requested a review from Molter73 November 12, 2024 02:28

JoukoVirtanen marked this pull request as ready for review November 12, 2024 23:16

JoukoVirtanen requested a review from a team as a code owner November 12, 2024 23:16

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 12, 2024

Molter73 reviewed Nov 13, 2024

View reviewed changes

Stringy reviewed Nov 13, 2024

View reviewed changes

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from 8420864 to 9b4f72e Compare November 14, 2024 01:54

JoukoVirtanen requested review from Molter73 and Stringy November 14, 2024 04:13

JoukoVirtanen added 2 commits November 15, 2024 16:26

X-Smart-Branch-Parent: master

b1cdc47

JoukoVirtanen and others added 20 commits November 15, 2024 16:26

More error handling for reading the configuration. Using a volume mount

dade64c

Fixed lint error

95c001b

The runtime config file is only modified on the host machine

ec61352

There is now only one source code file for runtime config introspection

0f0be47

Minor fixes for k8s

a9f5c3c

Added comments and sleeps when config does not change so we get a scr…

e410b11

…ape interval

Increased the afterglow period

9cdf634

Update collector/lib/ConfigLoader.cpp

7dfd521

Co-authored-by: Mauro Ezequiel Moltrasio <[email protected]>

Renamed unit test. Integration tests use a dedicated directory

7f1fa8a

Created a new assert package

a30b703

Apply suggestions from code review

29f768f

Co-authored-by: Mauro Ezequiel Moltrasio <[email protected]>

Split test into smaller tests

a144acd

Apply suggestions from code review

4862733

Co-authored-by: Mauro Ezequiel Moltrasio <[email protected]>

Removed unneeded sort

0ae580b

Using slices.ContainsFunc in HasConnection

087c326

Made timings in AssertRepeated configurable

b70f822

Always sort the connections when comparing

c724053

Not using assertMismatch

b01f751

Updated comments

678fc86

Apply suggestions from code review

f092de4

Co-authored-by: Giles Hutton <[email protected]>

JoukoVirtanen force-pushed the jv-integration-tests-for-runtime-config branch from d734a7c to f092de4 Compare November 16, 2024 00:27

JoukoVirtanen added 3 commits November 15, 2024 18:18

Sorting connections inside Connections instead of SortedConnections

11fd614

Created ElementsMatchFunc in assert pkg

30a7ad8

Using spew for logging

766747d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jv integration tests for runtime config #1899

Jv integration tests for runtime config #1899

JoukoVirtanen commented Oct 18, 2024

JoukoVirtanen Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 left a comment

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 6, 2024

JoukoVirtanen Nov 7, 2024

JoukoVirtanen Nov 7, 2024

JoukoVirtanen Nov 7, 2024

Molter73 Nov 7, 2024

JoukoVirtanen Nov 7, 2024

Molter73 left a comment

Molter73 Nov 11, 2024

Molter73 Nov 11, 2024

Molter73 Nov 13, 2024

JoukoVirtanen Nov 14, 2024

JoukoVirtanen commented Nov 12, 2024 •

edited

Loading

Molter73 left a comment

Stringy Nov 13, 2024

JoukoVirtanen Nov 14, 2024

JoukoVirtanen commented Nov 14, 2024

		@@ -67,12 +67,20 @@ TEST(CollectorConfigTest, TestYamlConfigToConfigInvalid) {
		}

		TEST(CollectorConfigTest, TestYamlConfigToConfigEmpty) {

		tick := time.NewTicker(1 * time.Second)
		timer := time.After(3 * time.Minute)

Jv integration tests for runtime config #1899

Are you sure you want to change the base?

Jv integration tests for runtime config #1899

Conversation

JoukoVirtanen commented Oct 18, 2024

Description

Checklist

Testing Performed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoukoVirtanen commented Nov 12, 2024 • edited Loading

Molter73 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoukoVirtanen commented Nov 14, 2024

JoukoVirtanen commented Nov 12, 2024 •

edited

Loading