Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(investigation): add usage collector #197659

Merged
merged 7 commits into from
Nov 12, 2024
Merged

feat(investigation): add usage collector #197659

merged 7 commits into from
Nov 12, 2024

Conversation

kdelemme
Copy link
Contributor

@kdelemme kdelemme commented Oct 24, 2024 •
edited by kibanamachine
Loading

Resolves #191644

🌸 Summary

This PR collects usage of the investigation app when enabled.
The data we collect is:

  • total investigation
  • count of investigation by status
  • count of investigation by origin type (alert or blank)
  • avg/p90/p95/min/max number of items per investigation
  • avg/p90/p95/min/max number of notes per investigation
{
    total: 0,
    by_status: {
      triage: 0,
      active: 0,
      mitigated: 0,
      resolved: 0,
      cancelled: 0,
    },
    by_origin: {
      alert: 0,
      blank: 0,
    },
    items: {
      avg: 0,
      p90: 0,
      p95: 0,
      max: 0,
      min: 0,
    },
    notes: {
      avg: 0,
      p90: 0,
      p95: 0,
      max: 0,
      min: 0,
    },
  };

@kdelemme kdelemme added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:obs-ux-management Observability Management User Experience Team v8.17.0 labels Oct 24, 2024
@kdelemme kdelemme self-assigned this Oct 24, 2024
@kdelemme kdelemme marked this pull request as ready for review October 24, 2024 14:20
@kdelemme kdelemme requested review from a team as code owners October 24, 2024 14:20
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

@botelastic botelastic bot added the ci:project-deploy-observability Create an Observability project label Oct 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@kdelemme kdelemme force-pushed the feat/add-investigation-collector-usage branch from 3b77d02 to 215979c Compare October 24, 2024 15:19
@kdelemme kdelemme force-pushed the feat/add-investigation-collector-usage branch from f89344c to 494da04 Compare October 24, 2024 17:31
TinaHeiligers
TinaHeiligers approved these changes Oct 24, 2024
View reviewed changes
Copy link
Contributor

@TinaHeiligers TinaHeiligers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for telemetry and added dependency

maryam-saeidi
maryam-saeidi reviewed Oct 30, 2024
View reviewed changes
Copy link
Member

@maryam-saeidi maryam-saeidi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM!

@kdelemme I haven't worked with snapshot telemetry, how can I test it locally?

shahzad31
shahzad31 reviewed Nov 12, 2024
View reviewed changes
x-pack/plugins/observability_solution/investigate_app/server/lib/collectors/fetcher.ts Outdated
Comment on lines 14 to 17
const finder = context.soClient.createPointInTimeFinder<StoredInvestigation>({
type: SO_INVESTIGATION_TYPE,
perPage: 1000,
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could be potentially very memory heavy task, if we let's say store lot of data as part of investigation, i imagine storing images, screenshots, lens embeddables attributes JSON, LLM output, all of that data is stored in items right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's possible indeed - I can reduce the batch size to something smaller, like 10?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think the status part could be fetched as an aggregation, the length of notes and items arrays is tricky to get since that isn't mapped.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah let's reduce the perPage for now. i can't think of anything else. since we can't use fields as well. If we really want to optimise, we can save length of items and notes as separate fields and get those. that way items and notes we won't have to fetch. but i guess it might be overkill.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed with 89bc326

shahzad31
shahzad31 approved these changes Nov 12, 2024
View reviewed changes
Copy link
Contributor

@shahzad31 shahzad31 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM other than memory concern !!

image

@elasticmachine
Copy link
Contributor

elasticmachine commented Nov 12, 2024 •
edited
Loading

💚 Build Succeeded

  • Buildkite Build
  • Commit: bb40c04
  • Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-197659-bb40c049f5e8

Metrics [docs]

✅ unchanged

History

cc @kdelemme

@kdelemme kdelemme merged commit 5b9908e into elastic:main Nov 12, 2024
26 checks passed
@kdelemme kdelemme deleted the feat/add-investigation-collector-usage branch November 12, 2024 18:56
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11803859030

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 12, 2024
@kdelemme
feat(investigation): add usage collector (elastic#197659)
083a1a6 
(cherry picked from commit 5b9908e)
@kibanamachine kibanamachine mentioned this pull request Nov 12, 2024
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Nov 12, 2024
@kibanamachine @kdelemme
[8.x] feat(investigation): add usage collector (#197659) (#199866)
f0fdec2 
# Backport

This will backport the following commits from `main` to `8.x`:
- [feat(investigation): add usage collector
(#197659)](#197659)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Kevin
Delemme","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-12T18:56:01Z","message":"feat(investigation):
add usage collector
(#197659)","sha":"5b9908ee622506c35873d4d8ae74a21e9b46d6e4","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","ci:project-deploy-observability","Team:obs-ux-management","v8.17.0"],"title":"feat(investigation):
add usage
collector","number":197659,"url":"https://github.com/elastic/kibana/pull/197659","mergeCommit":{"message":"feat(investigation):
add usage collector
(#197659)","sha":"5b9908ee622506c35873d4d8ae74a21e9b46d6e4"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197659","number":197659,"mergeCommit":{"message":"feat(investigation):
add usage collector
(#197659)","sha":"5b9908ee622506c35873d4d8ae74a21e9b46d6e4"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Kevin Delemme <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Nov 18, 2024
@kdelemme @CAWilson94
feat(investigation): add usage collector (elastic#197659)
565955e
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Nov 18, 2024
@kdelemme @CAWilson94
feat(investigation): add usage collector (elastic#197659)
36fdd7d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maryam-saeidi maryam-saeidi maryam-saeidi left review comments

@TinaHeiligers TinaHeiligers TinaHeiligers approved these changes

@shahzad31 shahzad31 shahzad31 approved these changes

Assignees

@kdelemme kdelemme

Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) ci:project-deploy-observability Create an Observability project release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-management Observability Management User Experience Team v8.17.0 v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[RCA] Initial telemetry collection setup for investigations (register usage collector, snapshot telemetry)
6 participants
@kdelemme @elasticmachine @kibanamachine @shahzad31 @TinaHeiligers @maryam-saeidi