[Discuss] Support for bulk upload API

[majagrubic]

As you may know, the shared-ux team is working on sample data re-architecture, and we'd like to leverage EPR for that. (Previous discussion here). Right now, sample data is a (gzipped) json file with Elasticsearch documents that are then bulk-uploaded to ES when a user requests an install.
If we were to add a new integration that would contain this sample data, what is the best way to make the bulk upload work?

cc @joshdover

[majagrubic]

As discussed in the call today, we would also need to add support for a new package type that would be "asset only" or something similar. Let me know if you want me to open a new issue to discuss that, or can we use this issue to coordinate.

[jsoriano]

I have added this issue to the implementation plan in #346. I think we can continue there.

[jen-huang]

My initial thinking is that we can do something very similar to what sample data does today - perhaps we can even re-purpose the existing code.

That assumes that we are ok with storing archive files on the registry. @jsoriano the other discussion threads are a bit long - do you think that is ok?

This means that the package-spec will need to define enough information for Fleet to know that filexyz.gzip contains ES documents and should be handled as such.

@kpollich, WDYT?

[jsoriano]

That assumes that we are ok with storing archive files on the registry. @jsoriano the other discussion threads are a bit long - do you think that is ok?

This means that the package-spec will need to define enough information for Fleet to know that filexyz.gzip contains ES documents and should be handled as such.

Yes, if we distribute these files through the registry, their format must be defined in the package-spec. This would be aligned with the implementation plan in the description of #346, that summarizes the discussion threads.