enhancement(http-server): add response_body_key for setting http response body

[frankh]

backwards compatible change to add a response_body_key option for the http-server source.

If specified, returns the json encoding of the given key in the event as the response body

fixes #21013

[bits-bot]

All committers have signed the CLA.

[frankh]

@jszwedko any interest in this PR?

we want this to be able to return a uuid to the client to be able to track an event, feels like a reasonably common use-case

[jszwedko]

Thanks for opening this PR @frankh !

Thinking about it a bit more, I think it'd be more flexible to let users specify a VRL program to generate the response. I think what you have here is pretty specific in that it pulls the field from the first event and then returns it as JSON.

I'm imagining, instead, that users could provide a VRL program that receives as input the entire series of input events and then can craft a response. For example:

ids = map_values!(.) -> |event| { event.id }

json_encode!({ ids: ids })

This would return [<list of ids from events>] as the response.

You could still return just the first field like you are doing here with something like:

json_encode!(.[0].id)

Granted, this is also a medium-sized change to Vector's UX. I don't think we have any precedent for it so I'd like to get some input from other maintainers. Before I do that, I'm curious if my suggestion would meet your need?

We could enable even more response customization by letting users return an object from the VRL program that includes things like Content-Type like:

ids = map_values!(.) -> |event| { event.id }

{
  headers: {
    "Content-Type": "application/json",
  },
  body: json_encode!({ ids: ids }),
  status_code: 200
}

(this is all VRL pseudocode)