-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
loki_out: add stuctured_metadata_map_keys #9530
base: master
Are you sure you want to change the base?
loki_out: add stuctured_metadata_map_keys #9530
Conversation
7f1db76
to
a9cdcad
Compare
a9cdcad
to
e9fee7d
Compare
* Adds stuctured_metadata_map_keys config to dynamically populate stuctured_metadata from a map Signed-off-by: Greg Eales <[email protected]>
Signed-off-by: Greg Eales <[email protected]>
dc24840
to
1f9301d
Compare
Signed-off-by: Greg Eales <[email protected]>
pack_label_key(mp_pck, (char*) accessed_map_kv.key.via.str.ptr, | ||
accessed_map_kv.key.via.str.size); | ||
/* | ||
* Does this need optimising? For example, to handle |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can anyone answer this question? Should I optimise this somehow?
@@ -78,6 +78,7 @@ struct flb_loki { | |||
struct mk_list *labels; | |||
struct mk_list *label_keys; | |||
struct mk_list *structured_metadata; | |||
struct mk_list *structured_metadata_map_keys; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this config property name okay structured_metadata_map_keys
?
@@ -83,7 +83,7 @@ void flb_test_basic() | |||
in_ffd = flb_input(ctx, (char *) "lib", NULL); | |||
flb_input_set(ctx, in_ffd, "tag", "test", NULL); | |||
|
|||
/* Elasticsearch output */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am assuming that all these should have said Loki and not Elasticsearch, and that it is okay to chnage them in this PR
"{\"sub_key1\": \"sub_value1\", \"sub_key2\": false}}, \"map2\": {\"key4\": " \ | ||
"\"value1\", \"key5\": false}, \"map3\": {\"key1\": \"map3_value1\", \"key2\": " \ | ||
"\"map3_value2\"}}]" | ||
void flb_test_structured_metadata_map_params(char *remove_keys, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are quite a few different variations on this test, so I have created this function which takes various parameters, and then called it for each actual test (below)
{"structured_metadata_map_and_explicit", | ||
flb_test_structured_metadata_map_and_explicit}, | ||
{"structured_metadata_map_single_missing_map", | ||
flb_test_structured_metadata_map_single_missing_map}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any test cases missing that I should add?
&ctx->structured_metadata_map_keys_list); | ||
} | ||
/* | ||
* explicit structured_metadata entries override |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this works as expected.
The intention is, if we try to add an entry to the structured_metadata with the same key twice (first from the new structured_metadata_map_keys
, then from the existing structured_metadata
), then the second entry should "win" and overrwite the first.
I am assuming msgpack works this way. But maybe I actually need to do some kind of explicit check, and only add a new entry where there is not already one for a given key?
* separate item. Non-string map values are serialised to JSON, as Loki requires | ||
* all values to be strings. | ||
*/ | ||
static void pack_maps(struct flb_loki *ctx, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This essentially tries to follow the flow of the proceeding pack_kv
function.
Do you need a more detailed spec of what exactly this is supposed to be doing?
It should implement the proposal descibed in the FR #9463
I added a PR for the docs here fluent/fluent-bit-docs#1527 |
Addresses #9463
Enter
[N/A]
in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-test
label to test for all targets (requires maintainer to do).Documentation
fluent/fluent-bit-docs#1527
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.