Documentation API OSQuery Packs Policy shards feature - partial deployment

[111andre111]

Description

There has been brought up some a new feature called partial deployment - policy shards in 8.6.0 for OSQuery Packs execution.
elastic/kibana#143948
It seems that as of today we have this documentation for scheduling OSQuery Packs:
https://www.elastic.co/guide/en/security/8.9/alerts-run-osquery.html
https://www.elastic.co/guide/en/kibana/8.9/osquery.html#osquery-schedule-status

However it looks that we don't have in out api documentation however any mention about the shards field:
https://www.elastic.co/guide/en/kibana/8.9/osquery-manager-packs-api-create.html#osquery-manager-packs-api-create-example
The example mentioned there ends up with an error [request body.shards]: expected value of type [object] but got [undefined]

So after adding "shards": {}, to the body the request works.
Can we please add the particular shards field to the API docs? I suppose we might need it as well on some others like update OSQuery packs and so on.

Acceptance Test Criteria

I propose to backport this documentation with shards in API appropriately as it's atm mandatory field in earlier versions. So even if that would be changed it should be obvious from the docs in case it's not mandatory any more.

Notes

• Add all appropriate labels to the issue, especially the version number label.
• Be sure to add any necessary screenshots, code text or console commands for clarity.
• Include any conditions or caveats that may affect customers.

[natasha-moore-elastic]

Thanks for raising this, @111andre111 !

@tomsonpl, could you confirm that we should add a shards object to this API? If so, could you help answer/provide the following:

• Which APIs do we need to add this object to?
• How far back do we need to backport this change?
• Could you provide a definition for the shards object that we can add to request body schema, including the definitions for any child parameters?
• Could you provide updated code examples?

Thanks in advance!

[tomsonpl]

Hey, I've just double checked the code, and I believe that we do not really need this as REQUIRED field. I think that we might just make it optional in the schema, and that could do fix the issue. However, some docs should be added anyway. I'll prepare the fix and doc answers for tomorrow.
@kevinlog @caitlinbetz does this sound ok to you?
@111andre111 FYI

[tomsonpl]

However... I am afraid we cannot backport this change further that 8.10.1 ... so that would make the API inconsistent, and this functionality goes back to 8.6.0.

[tomsonpl]

Also, the shards field is optional for updating pack, but required for creating pack. I just haven't seen this in time ;/

[tomsonpl]

elastic/kibana#166178

[tomsonpl]

@natasha-moore-elastic Hey, sorry I know I promised to prepare this for today, but got pulled away by other high priority stuff. I'll try to prepare the answers asap

[tomsonpl]

Hey @natasha-moore-elastic , sorry for the delay, I hope this answers your questions :)

I merged a PR that makes the 'shards' field optional for 8.10.1 and 8.11.0 onwards, so these version of docs can have shards field as optional. However previous version starting with 8.6.0 unfortunately should have this field as required, but only in Create Pack (because it was already optional in Update Pack).

• Which APIs do we need to add this object to?
  We have to adjust two APIs: Create Pack , Update Pack

• How far back do we need to backport this change?
  This functionality was introduced in 8.6

• Could you provide a definition for the shards object that we can add to request body schema, including the definitions for any child parameters?
  Shards: An object with shard configuration for policies included in the pack.
  Shard: Restrict this query to a percentage (1-100) of target hosts

• Could you provide updated code examples?
  This is the code from current create pack api docs. The difference is in 'shards' , and I added new policy to 'policy_ids'

  "name": "my_pack",
  "description": "My pack",
  "enabled": true,
  "policy_ids": [
    "my_policy_id", 
    "fleet-server-policy"
  ],
  "shards": {
     "my_policy_id": 35, 
     "fleet-server-policy": 58,
   },
  "queries": {
    "my_query": {
      "query": "SELECT * FROM listening_ports;",
      "interval": 60,
      "ecs_mapping": {
        "client.port": {
          "field": "port"
        },
        "tags": {
          "value": ["tag1", "tag2"]
        }
      }
    }
  }
}

[nastasha-solomon]

@tomsonpl thanks for being so helpful on this! Can you also share the PR that makes the shards field optional for 8.10.1+ and make sure it's labeled for release notes? I'll need to document this change in the 8.10.1 release notes.

[tomsonpl]

Hey, no problem, I caused the issue in the first place :P
#3822 (comment)
What label should it have? currently it's set up to skip.

[natasha-moore-elastic]

Updates for 8.6.0–8.10.0 are in elastic/kibana#166363.
Updates for 8.10.1 and 8.11.0 onwards to be made after 8.10.1 feature freeze.

[nastasha-solomon]

@tomsonpl size and impact-wise, the change seems more like an enhancement than a new feature. I'll defer to you for the final decision though.