DDSaaS: StreamNative: Crawler Integration v1.0.0 #18782

narendranandaniya-crest · 2024-10-07T12:07:37Z

What does this PR do?

This is an initial release PR of StreamNative integration including all the required assets.

Motivation

Crawler code for this integration has been committed in its respective repo
Pipeline created for this integration are available in our sandbox and would be shared separately with the required teams.
Samples for the pipeline review would also be shared separately with the required teams.
Since during the standard attribute remapping we are not preserving the source attributes as per suggested best practices, it would result in filters using these standard attributes populating the values of other integrations as well as per current Datadog behavior.

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Changelog entries must be created for modifications to shipped code
Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

…avnariyanicrest/integrations-core into streamnative-assets-v1.0.0

apiazza-dd · 2024-11-03T14:50:34Z

streamnative/README.md

+
+[StreamNative][1] provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services.
+
+The StreamNative integration collects below types of metrics:


Can we please get a link to the types of metrics collected?

apiazza-dd · 2024-11-03T14:50:52Z

streamnative/README.md

+
+### Configuration
+
+StreamNative integration requires a StreamNative account and its Client Id, Client Secret, Organization ID and Instance Name. Below are the steps to fetch these details from StreamNative console:


Id > ID for consistency please. Would also be great to see these formatted as code with ``. Thanks!

apiazza-dd · 2024-11-03T14:51:50Z

streamnative/README.md

+
+#### Steps to retrieve Organization ID, Instance Name, Client Id & Client Secret
+
+1. Login to the [StreamNative Cloud Console Account][2] 


Please add a . at the end of #1 for consistency. Thanks!

apiazza-dd · 2024-11-03T14:52:53Z

streamnative/README.md

+
+1. Login to the [StreamNative Cloud Console Account][2] 
+2. Go to the profile icon and select to the **Organizations** option.
+3. Go into the **Organization** from which data needs to be collected in the Datadog.


the Datadog > Datadog

apiazza-dd · 2024-11-03T14:53:25Z

streamnative/README.md

+1. Login to the [StreamNative Cloud Console Account][2] 
+2. Go to the profile icon and select to the **Organizations** option.
+3. Go into the **Organization** from which data needs to be collected in the Datadog.
+4. Copy and save the **Organization ID** to configure in the Integration.


the Integration > the Streamnative Integration Tile

apiazza-dd · 2024-11-03T14:53:36Z

streamnative/README.md

+2. Go to the profile icon and select to the **Organizations** option.
+3. Go into the **Organization** from which data needs to be collected in the Datadog.
+4. Copy and save the **Organization ID** to configure in the Integration.
+5. From the **Select an Instance** section, Copy and save the **Instance Name** to configure in the Integration.


Copy > copy

apiazza-dd · 2024-11-03T14:53:46Z

streamnative/README.md

+3. Go into the **Organization** from which data needs to be collected in the Datadog.
+4. Copy and save the **Organization ID** to configure in the Integration.
+5. From the **Select an Instance** section, Copy and save the **Instance Name** to configure in the Integration.
+6. Go to the profile icon and select the **Accounts & Accesses** tab.


Go to > "click"

apiazza-dd · 2024-11-08T14:56:05Z

Thanks for all of the updates, we're running this by our docs team and PM.

streamnative/README.md

streamnative/assets/dashboards/streamnative_pulsar_resource.json

streamnative/assets/dashboards/streamnative_sink_connector.json

streamnative/assets/dashboards/streamnative_source_connector.json

jhgilbert

Apologies, I missed these capitalization tweaks the first time around.

streamnative/README.md

dpoint01 · 2024-11-13T17:05:19Z

streamnative/README.md

+
+The StreamNative integration collects and forwards the following metrics to Datadog.
+
+1. Health


@apiazza-dd is this where we should have the table view with all the metric names + descriptions for the public docs?

The Metrics section on the public docs should reflect a table view of all the metrics collected that show up on the "Data collected" tab on the tile (from the metadata.csv)

Please see comment above

streamnative/README.md

apiazza-dd · 2024-11-13T18:36:44Z

@narendranandaniya-crest Can you please take this PR out of draft? We have reviews from all stakeholders, so once they're addressed, we're happy to approve and merge! Thanks!

ashah-crest · 2024-11-14T09:41:47Z

Hi @apiazza-dd , normally we follow this path where once the without Assets PR is merged we will move the Assets PR in Ready for Review State.
As we have without Assets PR in open state, we have kept this Assets PR in draft state. But considering we have the required reviews and approvals, we can move the Assets PR out of draft for merging and close the without Assets PR since everything related to non assets PR is already covered in here. CC: @dj0well

jhgilbert · 2024-11-14T16:36:05Z

This looks good, thanks for all of the updates! I'll need to verify the docs build once the PR is completely finished and ready to ship, just mention me here and I'll be happy to run the final check and approve it.

apiazza-dd · 2024-11-14T17:55:20Z

streamnative/README.md

+
+[StreamNative][1] provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services.
+
+The StreamNative integration collects the below types of [metrics][2]:


"collects the below" > Collects the following types of metrics

apiazza-dd · 2024-11-14T17:56:26Z

streamnative/README.md

+5. Click **Download OAuth2 Key** to get the **Client ID** and **Client Secret**.
+
+
+#### Add StreamNative credentials


Let's add more clarity here. What do you think of the following:

1. Click the profile icon and select **Organizations**. 2. Choose the **Organization** for which data needs to be collected. 3. From the **Select an Instance** dropdown, note the **Instance Name**. #### Get the `Client ID` and `Client Secret`: 1. Click the profile icon and navigate to the **Accounts & Accesses** tab. 2. Find the Service Account with **Admin** permissions set to **Enabled**. - If no Service Account exists, select **New -> Service Account** to create one, and make sure to enable the **Super Admin** option. 3. On the right side of the chosen Service Account, click the `...` button. 4. Select **Download OAuth2 Key** to obtain the **Client ID** and **Client Secret**. ### Setting Up the Integration Tile After gathering the necessary details, add the following Stream Native credentials to the integration tile - **Organization ID** - **Instance Name** - **Client ID** - **Client Secret**

Hi @apiazza-dd, we have updated the first half including the steps to get the configuration details.
Regarding the second part from "Setting Up the Integration Tile", earlier on the similar lines we have had an discussion around this over slack with @dpoint01 ( ref ). Can you please have a look. thanks!

@ashah-crest Thanks, I don't have access to that slack environment, but if @dpoint01 approved, lgtm.

michaelcretzman · 2024-11-19T17:55:54Z

streamnative/README.md

+
+[StreamNative][1] provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services.
+
+The StreamNative integration Collects the following types of [metrics][2]:


Suggested change

The StreamNative integration Collects the following types of [metrics][2]:

The StreamNative integration collects the following types of [metrics][2]:

michaelcretzman · 2024-11-19T17:57:22Z

streamnative/README.md

+
+#### Get StreamNative credentials
+
+##### Log in to the [StreamNative Cloud Console Account][3].


Suggested change

##### Log in to the [StreamNative Cloud Console Account][3].

Log into the [StreamNative Cloud Console Account][3].

narendranandaniya-crest added 2 commits October 1, 2024 17:51

DDSaaS: StreamNative: Crawler integration without Assets

7137dad

initial commit

f34e5c9

datadog-assets bot added docs/review-requested ecosystems/review-requested product/review-requested labels Oct 7, 2024

datadog-agent-integrations-bot bot added documentation changelog/no-changelog dev/testing dev/tooling qa/skip-qa Automatically skip this PR for the next QA labels Oct 7, 2024

narendranandaniya-crest added 13 commits October 7, 2024 18:00

Sorted metadata.csv file

462c200

updated image path

c8f5029

sorted metdata file

2fcd238

updated metadata file

0a338ca

fixed metadata failure

19eb166

Merge branch 'master' into streamnative-assets-v1.0.0

f405191

updated manifest file

9e4ec3a

Merge branch 'streamnative-assets-v1.0.0' of https://github.com/bharg…

8da2014

…avnariyanicrest/integrations-core into streamnative-assets-v1.0.0

updated readme steps

6f70e58

added metric check

001b23e

updated manifest

16f0cfb

updated readme

0d4c180

Merge branch 'master' into streamnative-assets-v1.0.0

587acd1

apiazza-dd reviewed Nov 3, 2024

View reviewed changes

updated readme & monitor names

7666e7a

jhgilbert reviewed Nov 11, 2024

View reviewed changes

updated asset files as per review

64fbe4e

jhgilbert reviewed Nov 13, 2024

View reviewed changes

streamnative/README.md Outdated Show resolved Hide resolved

streamnative/README.md Outdated Show resolved Hide resolved

streamnative/README.md Outdated Show resolved Hide resolved

dpoint01 reviewed Nov 13, 2024

View reviewed changes

apiazza-dd reviewed Nov 13, 2024

View reviewed changes

streamnative/README.md Show resolved Hide resolved

updated readme as per review

e39ece2

apiazza-dd reviewed Nov 14, 2024

View reviewed changes

narendranandaniya-crest marked this pull request as ready for review November 15, 2024 05:57

narendranandaniya-crest requested review from a team as code owners November 15, 2024 05:57

datadog-agent-integrations-bot bot added team/agent-integrations team/documentation labels Nov 15, 2024

narendranandaniya-crest added 2 commits November 15, 2024 13:08

Merge branch 'master' into streamnative-assets-v1.0.0

5b321b7

updated readme

462056e

michaelcretzman requested changes Nov 19, 2024

View reviewed changes

datadog-assets bot added docs/requested-changes and removed docs/review-requested labels Nov 19, 2024

steveny91 previously approved these changes Nov 19, 2024

View reviewed changes

updated readme file

82ef7d7

narendranandaniya-crest dismissed steveny91’s stale review via 82ef7d7 November 20, 2024 12:45

Merge branch 'master' into streamnative-assets-v1.0.0

435e804

michaelcretzman approved these changes Nov 20, 2024

View reviewed changes

datadog-assets bot added docs/approved and removed docs/requested-changes labels Nov 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DDSaaS: StreamNative: Crawler Integration v1.0.0 #18782

DDSaaS: StreamNative: Crawler Integration v1.0.0 #18782

narendranandaniya-crest commented Oct 7, 2024

apiazza-dd Nov 3, 2024

narendranandaniya-crest Nov 7, 2024

apiazza-dd Nov 3, 2024 •

edited

Loading

narendranandaniya-crest Nov 7, 2024

apiazza-dd Nov 3, 2024

apiazza-dd Nov 3, 2024

apiazza-dd Nov 3, 2024

apiazza-dd Nov 3, 2024

apiazza-dd Nov 3, 2024 •

edited

Loading

narendranandaniya-crest Nov 7, 2024

apiazza-dd commented Nov 8, 2024

jhgilbert left a comment

dpoint01 Nov 13, 2024

dpoint01 Nov 13, 2024

apiazza-dd Nov 13, 2024

apiazza-dd commented Nov 13, 2024

ashah-crest commented Nov 14, 2024

jhgilbert commented Nov 14, 2024

apiazza-dd Nov 14, 2024

narendranandaniya-crest Nov 15, 2024

apiazza-dd Nov 14, 2024 •

edited

Loading

ashah-crest Nov 15, 2024 •

edited

Loading

apiazza-dd Nov 19, 2024

michaelcretzman Nov 19, 2024

narendranandaniya-crest Nov 20, 2024

michaelcretzman Nov 19, 2024

narendranandaniya-crest Nov 20, 2024


		[StreamNative][1] provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services.

		The StreamNative integration collects below types of metrics:


		### Configuration

		StreamNative integration requires a StreamNative account and its Client Id, Client Secret, Organization ID and Instance Name. Below are the steps to fetch these details from StreamNative console:


		#### Steps to retrieve Organization ID, Instance Name, Client Id & Client Secret

		1. Login to the [StreamNative Cloud Console Account][2]


		The StreamNative integration collects and forwards the following metrics to Datadog.

		1. Health


		[StreamNative][1] provides an enterprise-grade messaging and event streaming platform built on Apache Pulsar. It offers scalable, real-time data streaming solutions with features like multi-tenancy, geo-replication, and seamless integration with cloud services.

		The StreamNative integration collects the below types of [metrics][2]:

		5. Click Download OAuth2 Key to get the Client ID and Client Secret.


		#### Add StreamNative credentials


		#### Get StreamNative credentials

		##### Log in to the [StreamNative Cloud Console Account][3].

	##### Log in to the [StreamNative Cloud Console Account][3].

	Log into the [StreamNative Cloud Console Account][3].

DDSaaS: StreamNative: Crawler Integration v1.0.0 #18782

Are you sure you want to change the base?

DDSaaS: StreamNative: Crawler Integration v1.0.0 #18782

Conversation

narendranandaniya-crest commented Oct 7, 2024

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apiazza-dd Nov 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apiazza-dd Nov 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apiazza-dd commented Nov 8, 2024

jhgilbert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apiazza-dd commented Nov 13, 2024

ashah-crest commented Nov 14, 2024

jhgilbert commented Nov 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apiazza-dd Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

ashah-crest Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apiazza-dd Nov 3, 2024 •

edited

Loading

apiazza-dd Nov 3, 2024 •

edited

Loading

apiazza-dd Nov 14, 2024 •

edited

Loading

ashah-crest Nov 15, 2024 •

edited

Loading