Connector end-to-end tests

[pmuellr]

Kibana's connectors communicate with external systems. To ensure they function correctly, we need a suite of tests where Kibana's connectors communicate with external systems and the results are properly validated. The existing Kibana functional tests are mocking the external systems, so they are not sufficient for ensuring that the connectors continue to behave correctly.

Watcher actions are tested end-to-end using secrets that are stored in vault. Therefore, we should follow a similar approach here. However, we will need to take into consideration that not all Kibana developers will have access to vault, but we want to ensure these tests are always ran as part of CI on a continuous basis.

The qaf-tests's rac.yml does have some of these connectors configured for use by alerting-rules; however, there are no tests to ensure that the connectors are communicating with the external system correctly.

Full list of connectors, their owner, whether the connector is setup in the qaf-tests, and the status of their end-to-end tests ran by CI.

Connector	Owner	Connector setup in qaf-tests	CI E2E Tests
Email	@elastic/response-ops	Yes	No
Slack	@elastic/response-ops	Yes	No
Slack API	@elastic/response-ops	No	No
PagerDuty	@elastic/response-ops	Yes	No
Service Now - ITSM	@elastic/response-ops	No	No
Service Now - SecOps	@elastic/response-ops	No	No
Service Now - ITOM	@elastic/response-ops	Yes	No
Webhook	@elastic/response-ops	No	No
IBM Resilient	@elastic/response-ops	Yes	No
Jira	@elastic/response-ops	Yes	No
Microsoft Teams	@elastic/response-ops	No	No
Swimlane	@elastic/response-ops	Yes	No
xMatters	@elastic/response-ops	Yes	No
Webhook - Case Management	@elastic/response-ops	No	No
OpsGenie	@elastic/response-ops	No	No
Torq	Security Solution	No	No
Tines	Security Solution	No	No
OpenAI	@elastic/obs-knowledge-team	No	No
Amazon Bedrock	@elastic/security-generative-ai	No	No
D3 Security	Security Solution	No	No
SentinelOne	@elastic/security-defend-workflows	No	No

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

I chatted with @tylersmalley on this topic. I'll leave these notes for whenever we pick this up.

It sounds like supporting this isn't too complex from an operations perspective. We need to find a way to connect credentials in the vault to our connectors and create a pipeline to periodically run the tests (nightly, on PRs when code in folder X changes, etc.).

[LeeDr]

We have some tests in a private integration-test repo that are semi-automated. For example, a watch that emails PDF and PNGs of a dashboard and we (QA team) visually check those emails.

[pmuellr]

Ya, I think we probably have an option of doing this via FT somehow (that doesn't run all the time), or the private integration test repo. We can defer on picking one of those right now, and decide on which is best once we start working on the issue.

[EricDavisX]

I may look into this at some point, based on prioritization among Response Ops tasks. I'd like to know about the private repo if you can ping me @LeeDr (not urgent). I suspect the elastic/machine-learning-qa repo may be a potential home for the framework needs as well, the MLR-QA team is starting to build out some Rules/Alerts/Cases/Connectors data. I'll note this issue as part of my info gathering.

[EricDavisX]

As I get into it more, the ML-QA repo is very accommodating to this type of test need. And of note, the jest tests are fairly good:
https://github.com/elastic/kibana/blob/main/x-pack/plugins/actions/server/builtin_action_types/email.test.ts

The functional side tests with 'live' connectors would catch if the 3rd party api's fail, and would be needed if we add new versions or a new connector overall.

FYI - Docs for connectors are here. I'm following up with if we have private instances of some of them we can/should re-use. https://www.elastic.co/guide/en/kibana/current/action-types.html

[EricDavisX]

I had a quick meeting with RespOps engineers to get initial info on what we have setup that we can re-use, i'll find the right place and will store off the connectivity for more of us to see.

[EricDavisX]

I updated the description with a tentative plan and list of connectors. @dolaru particularly fyi. I've posted to the ResponseOps team to see if any takers want to work with this me. I can prioritize it later, time frame TBD. for now I'll ty to work it little by little when I need to de-focus from my primary work.

[EricDavisX]

FYI - this was prioritized for the MLR-QA Team's Q3 work, and we expect to have updates on 2 fronts soon:

• many more connectors in the ml-qa framework's env-bootstrap execution, and those hooked up to a live rule
• a small test in the ml-qa framework that accesses the 3rd party service, we're likely to start with email as an easy test and then will expand out from there.