Osquerybeat: Fix data_stream configuration, enforce the default values used before 8.6.0

[aleksmaus]

What does this PR do?

Fixes the values for the data_stream: type anddataset.

The 8.6.0 has wrong values set after switching to using the V2 libbeat management.CreateInputsFromStreams implementation that is inserting the policy processor:

  processors:
  - add_fields:
      fields:
        dataset: generic
        namespace: default
        type: osquery
      target: data_stream

resulting in these data_stream properties.

"data_stream": {
            "namespace": "default",
            "type": "osquery",
            "dataset": "generic"
},

This PR sets the expected default values on the datastream before the configuration transformation takes place thus setting the values back to what it was before 8.6.0:

"data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "osquery_manager.result"
},

These fields are mapped as the constant_keywords and should never be changed.

Why is it important?

Fixes the breakage with the Osquery Logstash integration because the logstash sends the data based on the data_stream "hint", resulting in the output going to osquery-generic-default instead of logs-osquery_manager.result-default

Also the defect makes 8.6.0 osquery results incompatible with previous releases results documents data. And since the fields are constant, in the mixed agent versions environment or upon upgrade this can cause the issues indexing the osquery results documents.
Resulting in the errors such as:

[constant_keyword] field [data_stream.type] only accepts values that are equal to the value defined in the mappings [logs], but got [osquery]

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes OSQuery results are not viewable in Kibana when upgrading to the 8.6.0 Agent running the OSQuery Manager integration #34250

Screenshots

Verified the document data_stream is set correctly when sending data to elasticsearch output

Verified the document data_stream is set correctly when sending data to logstash output

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-13T16:05:26.522+0000

• Duration: 44 min 54 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1632
Skipped	0
Total	1632

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[cmacknz]

This works, but if the dataset or type ever becomes configurable it will break. That's probably unlikely though.

[cmacknz]

I can't comment directly on this, but this block below can be removed:

beats/x-pack/osquerybeat/cmd/root.go

Lines 89 to 91 in b5246c4

	for iter := range modules {
	modules[iter]["type"] = "log"
	}

That is trying to set an input type, which I don't think matters for osquerybeat. It is just adding an arbitrary type: log into the generated config that osquerybeat will be ignoring.

[aleksmaus]

yeah, don't remember the reason for this. can clean up later, separately from this more urgent fix

[cmacknz]

I think it is a copy/paste error from one of the other Beats, agree this can be removed separately.

[cmacknz]

You should probably have a changelog entry for this one.

[cmacknz]

The packaging failure is unrelated, caused by elastic/golang-crossbuild#232