fluent source with log_namespace puts all fields to namespace

[jiping-s]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When the fluent source is used with log_namespace enabled, all fields are moved into that namespace, instead of only the source's additional fields. This behavior is inconsistent with http_server and other sources.

Configuration

sources:
    in_fluent:
        type: fluent
        address: 0.0.0.0:24224
        log_namespace: true
transforms:
  tf_mark_vector:
    type: remap
    inputs: [in_fluent]
    source: |
      .fluent = %fluent
sinks:
  out_stdout:
    type: console
    inputs: [tf_mark_vector]
    encoding:
      codec: json
      timestamp_format: rfc3339

Version

0.38.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Thanks @jiping-s . I was able to reproduce this bug. The issue is that record shouldn't be inserted as metadata, but instead should be used as the object itself. I think the bug is around here:

vector/src/sources/fluent/mod.rs

Lines 620 to 628 in 8325300

	for (key, value) in record.into_iter() {
	let value: Value = value.into();
	log_namespace.insert_source_metadata(
	FluentConfig::NAME,
	&mut log,
	Some(LegacyKey::Overwrite(path!(key.as_str()))),
	path!("record", key.as_str()),
	value,
	);

As a workaround until the bug is fixed you can do:

. = %record