Add benchmark stream command

[aspacca]

Similarly to benchmark rally command we want to generate schema-b documents for a given integrations.
Instead of creating a rally track out of them we will stream them, according to a configurable rate, directly to an ES cluster, using bulk requets

[ruflin]

It would be nice, if you can run the command multiple times in parallel, something like:

# Inside nginx package
./elastic-package benchmark stream --benchmark access-logs-benchmark -v
./elastic-package benchmark stream --benchmark error-logs-benchmark -v
# Inside apache package
./elastic-package benchmark stream --benchmark access-logs-benchmark -v

For a developer working on the UI or testing the UI, this makes it super easy to have multiple docs mixed together.

[aspacca]

It would be nice, if you can run the command multiple times in parallel, something like:

open different terminal windows would be enough, or were you thinking about a solution inside the command itself?

[ruflin]

Different terminal windows is good enough. I mostly want to make sure they do not get into the way each other (some global state or similar).

[aspacca]

some global state or similar

no global state: we won't even save the data to a temporary file so there should not be race condition on that

[ruflin]

The discussions in elastic/integrations#8431 (comment) triggered an additional thought for the the stream command. This should not block any progress on what we have discussed so far but let me share it here. The stream command will allow devs / users to see real time data coming in. Unfortunately, the data is only from the time they started the command. What if we could backfill data quickly for the last X minutes on startup initially. Following example for system metrics:

• User starts the stream command and has set fillup: -15min (much better name needed)
• The stream command starts, and fills up the data for the last 15min as fast as possible
• From now on, it ships the configured events every 10 seconds
• Ideally, the two would happen in parallel so real time data is seen immediately and the fill up would happen in the background

Like this, the dev does not have to wait 15min to see data for 15min, but has historical data to build a visualisation / test a query and also has new data constantly coming in.

[ruflin]

To make the stream command work without requiring a local source copy of the package, there needs a place where raw packages are available. Left a comment here: elastic/package-spec#351 (comment)

[ruflin]

We had recently a discussion, that the stream command is already useful without support for the --package... flag, meaning it has to be run with the source checked out.