Add redpanda connect secret manager to console #1527

andresaristizabal · 2024-11-22T00:05:44Z

This PR introduces a new tab for Redpanda Connect

Features:

Create secrets
Edit secrets
Delete secrets
Update secrets
Secret autocomplete on pipeline RPCN editor
Choose which connect tab is showing as default based on query param defaultTab

Pending:

Handle BYOC, Serverless and Hosted cluster
Avoid duplications auto complete secrets in the pipeline editor -- disable autocomplete
Disable buttons in the "Update Secret" form while an update request is in progress.

Dedicated integration cluster

Screen.Recording.2024-11-21.at.8.43.30.PM.mov

Serverless integration cluster

Screen.Recording.2024-11-26.at.8.41.10.AM.mov

Before:

Enterprice self-hosted

Screen.Recording.2024-11-25.at.7.42.10.PM.mov

CLAassistant · 2024-11-22T00:05:50Z

All committers have signed the CLA.

malinskibeniamin · 2024-11-26T10:00:33Z

frontend/src/components/pages/connect/Overview.tsx

+const getDefaultView = (defaultView: string): { initialTab: ConnectView, redpandaConnectTab: ConnectView } => {
+
+    const showPipelines = Features.pipelinesApi
+    const showKafkaTab = {initialTab: ConnectView.kafkaConnect, redpandaConnectTab: ConnectView.RedpandaConnect}
+    const showRedpandaConnectTab = {initialTab: ConnectView.RedpandaConnect, redpandaConnectTab: ConnectView.RedpandaConnect}


Nothing to say about this change but overall we desperately need a linter 😓 I will try to put something together when I have spare 5 mins

malinskibeniamin · 2024-11-26T10:00:53Z

frontend/src/components/pages/connect/Overview.tsx

+        case 'redpanda-connect':
+            return showRedpandaConnectTab;
+        case 'redpanda-connect-secret':
+            return {initialTab: ConnectView.RedpandaConnect, redpandaConnectTab: ConnectView.RedpandaConncetSecret};


RedpandaConncetSecret -> RedpandaConnectSecret

malinskibeniamin · 2024-11-26T10:01:38Z

frontend/src/components/pages/connect/Overview.tsx

+import { useLocation } from 'react-router-dom';
+
+enum ConnectView {
+    kafkaConnect = 'kafka-connect',


Let's make it uppercase to match the other values
kafkaConnect -> KafkaConnect

malinskibeniamin · 2024-11-26T10:02:51Z

frontend/src/components/pages/connect/Overview.tsx

+    const {search}= useLocation();
+    const searchParams = new URLSearchParams(search);
+    const defaultTab = searchParams.get('defaultTab') || '';
+    return <KafkaConnectOverview defaultView={defaultTab} {...props}/>


What happens if defaultTab is ''? I think we need to return to tabConnectView.RedpandaConnect by default?

it should default from getDefaultView()

malinskibeniamin · 2024-11-26T10:04:23Z

frontend/src/components/pages/rp-connect/Pipelines.Create.tsx

+                    startLineNumber: position.lineNumber,
+                    endLineNumber: position.lineNumber,


It's always going to start and end on the same line, is that intentional?

yes, it is intencional, but I disabled autocomplete, because it needs more testing and there isn't time now

remove this function for now, I need to improve the autocomplete

malinskibeniamin · 2024-11-26T10:04:44Z

frontend/src/components/pages/rp-connect/Pipelines.Create.tsx

+                const last_chars = model.getValueInRange({startLineNumber: position.lineNumber, startColumn: 0, endLineNumber: position.lineNumber, endColumn: position.column});
+                const words = last_chars.replace('\t', '').replace('\{', '').split(' ');
+                const active_typing = words[words.length - 1];
+                const empty = {suggestions: []}


Maybe we can continue using camelCase

malinskibeniamin · 2024-11-26T10:07:05Z

frontend/src/components/pages/rp-connect/secrets/Secrets.Create.tsx

+            <PageContent>
+                <ToastContainer/>
+                <Flex flexDirection="column" gap={5}>
+                    <FormField label="Secret name" isInvalid={alreadyExists} errorText="Secret name is already in use">


Is that always going to be the reason for the error? What if some invalid character is used? Wouldn't it show wrong error text?

malinskibeniamin · 2024-11-26T10:08:16Z

frontend/src/components/pages/rp-connect/secrets/Secrets.List.tsx

+                if (String(err).includes('404')) {
+                    // Hacky special handling for OSS version, it is possible for the /endpoints request to not complete in time for this to render
+                    // so in this case there would be an error shown because we were too fast (with rendering, or the req was too slow)
+                    // We don't want to show an error in that case
+                    return;
+                }


I think maybe it sounds worse than it is, but thanks for explaining 😬

I copy and past this handle.

malinskibeniamin · 2024-11-26T10:08:41Z

frontend/src/components/pages/rp-connect/secrets/Secrets.List.tsx

+        return (
+            <PageContent>
+                <Section>
+                    <ToastContainer/>


Do we need ToastContainer copied everywhere? There's no global provider?

Again, I use pipeline as model, we can improve it later, when we have all console context

malinskibeniamin · 2024-11-26T10:10:21Z

frontend/src/components/pages/rp-connect/secrets/Secrets.Update.tsx

+
+    initPage(p: PageInitHelper) {
+        p.title = 'Update Secret';
+        p.addBreadcrumb('Redpanda Connect Secret Manager', '/rp-connect/secret/update');


Note: I think in the future we should have a global place with all the routes to ensure we can easily change name/forbid accessing it across the app if we needed to. Just in the router file there should be a place to get all the route paths.

I don't know what was the reason for this change, we can discuss it later

Should it be secret or secrets?

it should be secrets, I fixing it

malinskibeniamin · 2024-11-26T10:11:13Z

frontend/src/components/pages/rp-connect/secrets/Secrets.Update.tsx

+                            <Input
+                                placeholder="Enter a secret name..."
+                                data-testid="secretId"
+                                pattern="[a-zA-Z0-9_\-]+"


Maybe we can reuse this regex for secrets, actually I see this pattern is different
^[A-Z][A-Z0-9_]*$ vs [a-zA-Z0-9_\-]+, which one should we use?

I updated this based on proto

malinskibeniamin · 2024-11-26T10:11:47Z

frontend/src/state/ui.ts

@@ -234,6 +234,10 @@ const defaultUiSettings = {
        quickSearch: ''
    },

+    rpncSecretList: {


I think you meant
rpcnSecretList -> rpncSecretList

jvorcak · 2024-11-27T12:34:23Z

frontend/src/components/pages/rp-connect/secrets/Secrets.Create.tsx

+
+
+@observer
+class RpConnectSecretCreate extends PageComponent {


When we try to load these new pages and RP-connect is not enabled, we get infinite loading, this applies for both new URLs that have been added to the router. Can we please show something that prompts the user to configure it?

when it happens, rpcn-secret continue working:

https://redpandadata.slack.com/archives/C074X8HD68L/p1732568383289989?thread_ts=1732294471.397329&cid=C074X8HD68L

frontend/src/components/pages/rp-connect/secrets/Secrets.Create.tsx

frontend/src/components/pages/rp-connect/secrets/Secrets.List.tsx

frontend/src/components/pages/connect/Overview.tsx

bojand · 2024-11-27T14:40:10Z

Maybe it doesn't have to be part of this PR, but it's related, is there any way we can add the well known predefined contextual env vars to auto completion. The env vars added in https://github.com/redpanda-data/cloudv2/pull/18566.
Defined:
https://github.com/redpanda-data/cloudv2/blob/main/apps/redpanda-connect-api/config/config.go#L229
https://github.com/redpanda-data/cloudv2/blob/main/apps/redpanda-connect-api/config/config.go#L382-L391

andresaristizabal · 2024-11-27T14:46:36Z

Maybe it doesn't have to be part of this PR, but it's related, is there any way we can add the well known predefined contextual env vars to auto completion. The env vars added in redpanda-data/cloudv2#18566. Defined: https://github.com/redpanda-data/cloudv2/blob/main/apps/redpanda-connect-api/config/config.go#L229 https://github.com/redpanda-data/cloudv2/blob/main/apps/redpanda-connect-api/config/config.go#L382-L391

Good idea, we need to check how we can add it on autocomplete, I disabled autocomplete because I want to spend more time on it

andresaristizabal added 9 commits November 21, 2024 18:35

console: add rpcn create secret page

841b35a

console: add rpcn list secret page

7cabae5

console: add rpcn update secret page

bf4f64f

console: add redpanda connect grpc client

dcc1289

console: add redpanda connect secret routes

2cabe88

console: use same design buttons as last secret manager design

d87484c

console: add secret tab in redpanda connect page

6bbb7c3

console: add autocomplete secrets to create pipeline

390fddf

console: add autocomplete secrets to edit pipeline

b8b41cc

andresaristizabal force-pushed the feature/rpcn-secrets branch from 359c7df to 1e0317a Compare November 22, 2024 01:38

andresaristizabal changed the title ~~Add redpanda connect secret manager to console~~ WIP: Add redpanda connect secret manager to console Nov 22, 2024

andresaristizabal marked this pull request as draft November 22, 2024 01:41

andresaristizabal requested review from jvorcak, malinskibeniamin, bojand and sago2k8 November 25, 2024 14:18

malinskibeniamin reviewed Nov 26, 2024

View reviewed changes

console: add query parameter to select the default Connect tab to open.

8e90601

andresaristizabal added 5 commits November 26, 2024 10:50

rp-connect: hide search field if there are no secret or pipeline

837d6e5

rp-connect: add password input for secret values.

deaac9a

connect: add copy button to secret list

94e4bc6

pipeline-editor: disable secret autocomplete

d03be2e

rpcn-secret: fix typo

b329bba

andresaristizabal marked this pull request as ready for review November 26, 2024 16:20

andresaristizabal force-pushed the feature/rpcn-secrets branch from 1e0317a to b329bba Compare November 26, 2024 16:21

andresaristizabal changed the title ~~WIP: Add redpanda connect secret manager to console~~ Add redpanda connect secret manager to console Nov 26, 2024

andresaristizabal requested a review from malinskibeniamin November 26, 2024 17:21

jvorcak reviewed Nov 27, 2024

View reviewed changes

andresaristizabal added 3 commits November 27, 2024 13:26

rpcn-secret: use util base64 encode fn

b7f33ca

rpcn-secret: use correct style for button

5421d5a

rpcn-secret: change secrets on url instead of secret

05837ee

andresaristizabal requested review from jvorcak and weeco November 27, 2024 19:34

malinskibeniamin approved these changes Nov 27, 2024

View reviewed changes

jvorcak approved these changes Nov 27, 2024

View reviewed changes

rpcn-secret: add manual validation for secret name

a30244f

andresaristizabal merged commit 0d0f7f3 into master Nov 28, 2024
6 checks passed

andresaristizabal deleted the feature/rpcn-secrets branch November 28, 2024 12:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add redpanda connect secret manager to console #1527

Add redpanda connect secret manager to console #1527

andresaristizabal commented Nov 22, 2024 •

edited

Loading

CLAassistant commented Nov 22, 2024 •

edited

Loading

malinskibeniamin Nov 26, 2024

malinskibeniamin Nov 26, 2024

andresaristizabal Nov 26, 2024

malinskibeniamin Nov 26, 2024

andresaristizabal Nov 26, 2024

malinskibeniamin Nov 26, 2024 •

edited

Loading

andresaristizabal Nov 26, 2024

malinskibeniamin Nov 26, 2024

andresaristizabal Nov 26, 2024

andresaristizabal Nov 26, 2024

malinskibeniamin Nov 26, 2024

malinskibeniamin Nov 26, 2024

malinskibeniamin Nov 26, 2024

andresaristizabal Nov 26, 2024

malinskibeniamin Nov 26, 2024 •

edited

Loading

andresaristizabal Nov 26, 2024

malinskibeniamin Nov 26, 2024

andresaristizabal Nov 26, 2024

weeco Nov 26, 2024

andresaristizabal Nov 27, 2024

malinskibeniamin Nov 26, 2024

andresaristizabal Nov 26, 2024

malinskibeniamin Nov 26, 2024

andresaristizabal Nov 26, 2024

jvorcak Nov 27, 2024

andresaristizabal Nov 27, 2024

bojand commented Nov 27, 2024

andresaristizabal commented Nov 27, 2024

		startLineNumber: position.lineNumber,
		endLineNumber: position.lineNumber,



		@observer
		class RpConnectSecretCreate extends PageComponent {

Add redpanda connect secret manager to console #1527

Add redpanda connect secret manager to console #1527

Conversation

andresaristizabal commented Nov 22, 2024 • edited Loading

Dedicated integration cluster

Serverless integration cluster

Before:

Enterprice self-hosted

CLAassistant commented Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malinskibeniamin Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malinskibeniamin Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bojand commented Nov 27, 2024

andresaristizabal commented Nov 27, 2024

andresaristizabal commented Nov 22, 2024 •

edited

Loading

CLAassistant commented Nov 22, 2024 •

edited

Loading

malinskibeniamin Nov 26, 2024 •

edited

Loading

malinskibeniamin Nov 26, 2024 •

edited

Loading