KOGITO-9971: [Guides] Dataindex deployment use cases with operator (#519

) * KOGITO-9971: [Guides] Dataindex deployment use cases with operator * KOGITO-9808:Add Data Index persistence addon documentation * Adding reference to examples to be able to explore the implied files * Typo fix * Updated references to examples after moving the files to operator use cases * removed dataindex_platform references
apache · Dec 12, 2023 · ea6e22d · ea6e22d
1 parent 74489e9
commit ea6e22d
Show file tree

Hide file tree

Showing 11 changed files with 964 additions and 41 deletions.
diff --git a/serverlessworkflow/antora.yml b/serverlessworkflow/antora.yml
@@ -65,6 +65,7 @@ asciidoc:
     sonataflow_devmode_imagename: quay.io/kiegroup/kogito-swf-devmode
     kogito_examples_repository_url: https://github.com/apache/incubator-kie-kogito-examples
     kogito_sw_examples_url: https://github.com/apache/incubator-kie-kogito-examples/tree/main/serverless-workflow-examples
+    kogito_sw_operator_examples_url: https://github.com/apache/incubator-kie-kogito-examples/tree/main/serverless-operator-examples
     kogito_examples_url: https://github.com/apache/incubator-kie-kogito-examples.git
     kogito_apps_url: https://github.com/apache/incubator-kie-kogito-apps/tree/main
     quarkus_cli_url: https://quarkus.io/guides/cli-tooling

diff --git a/serverlessworkflow/modules/ROOT/assets/images/data-index/data-index-addon.drawio b/serverlessworkflow/modules/ROOT/assets/images/data-index/data-index-addon.drawio
diff --git a/serverlessworkflow/modules/ROOT/assets/images/data-index/data-index-addon.png b/serverlessworkflow/modules/ROOT/assets/images/data-index/data-index-addon.png
diff --git a/serverlessworkflow/modules/ROOT/nav.adoc b/serverlessworkflow/modules/ROOT/nav.adoc
@@ -110,7 +110,10 @@
 * Data Index
 ** xref:data-index/data-index-core-concepts.adoc[Core concepts]
 ** xref:data-index/data-index-service.adoc[Standalone service]
-** xref:data-index/data-index-quarkus-extension.adoc[Quarkus Extension]
+** xref:data-index/data-index-quarkus-extension.adoc[Quarkus Extensions]
+** Operator
+*** xref:data-index/data-index-usecase-singleton.adoc[]
+*** xref:data-index/data-index-usecase-multi.adoc[]
 //** Quarkus Extensions TODO: https://issues.redhat.com/browse/KOGITO-9463
 * Use Cases
 ** xref:use-cases/orchestration-based-saga-pattern.adoc[Saga Orchestration]

diff --git a/...rkflow/modules/ROOT/pages/data-index/common/_dataindex_deployment_operator.adoc b/...rkflow/modules/ROOT/pages/data-index/common/_dataindex_deployment_operator.adoc
@@ -0,0 +1,220 @@
+
+link:{flow_examples_operator_url}/tree/main/infra/dataindex[Here] you can find the infrastructure kustomization required to deploy {data_index_ref} service and a postgresql database explained in this use case.
+
+Thas folder contains four files:
+
+* kustomization.yaml
+* 01-postgres.yaml
+* 02-dataindex.yaml
+* application.properties
+
+.`kustomization.yaml` resources that deploy {data_index_ref} deployment with persistence to a postgresql database
+[source,yaml,subs="attributes+"]
+----
+resources:
+- 01-postgres.yaml <1>
+- 02-dataindex.yaml <2>
+
+secretGenerator:
+  - name: postgres-secrets
+    literals:
+      - POSTGRES_USER=sonataflow
+      - POSTGRES_PASSWORD=sonataflow
+      - POSTGRES_DB=sonataflow
+      - PGDATA=/var/lib/postgresql/data/mydata
+
+configMapGenerator:
+  - name: dataindex-properties
+    files:
+      - application.properties
+----
+<1> Postgres database deployment
+<2> {data_index_ref} deployment
+
+.`01_postgres.yaml` that deploys Postgresql database
+[source,yaml,subs="attributes+"]
+----
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  labels:
+    app.kubernetes.io/name: postgres
+  name: postgres-pvc
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app.kubernetes.io/name: postgres
+  name: postgres
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: postgres
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: postgres
+    spec:
+      containers:
+        - name: postgres
+          image: postgres:13.2-alpine
+          imagePullPolicy: 'IfNotPresent'
+          ports:
+            - containerPort: 5432
+          volumeMounts:
+            - name: storage
+              mountPath: /var/lib/postgresql/data
+          envFrom:
+            - secretRef:
+                name: postgres-secrets
+          readinessProbe:
+            exec:
+              command: ["pg_isready"]
+            initialDelaySeconds: 15
+            timeoutSeconds: 2
+          livenessProbe:
+            exec:
+              command: ["pg_isready"]
+            initialDelaySeconds: 15
+            timeoutSeconds: 2
+          resources:
+            limits:
+              memory: "256Mi"
+              cpu: "500m"
+      volumes:
+      - name: storage
+        persistentVolumeClaim:
+          claimName: postgres-pvc
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app.kubernetes.io/name: postgres
+  name: postgres
+spec:
+  selector:
+    app.kubernetes.io/name: postgres
+  ports:
+    - port: 5432
+----
+
+.`02-dataindex.yaml` that deploys {data_index_ref} with persistence to the previous defined postgresql database
+[source,yaml,subs="attributes+"]
+----
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app.kubernetes.io/name: data-index-service-postgresql
+  name: data-index-service-postgresql
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: data-index-service-postgresql
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: data-index-service-postgresql
+    spec:
+      containers:
+        - name: data-index-service-postgresql
+          image: quay.io/kiegroup/kogito-data-index-postgresql:latest
+          imagePullPolicy: Always
+          resources:
+            limits:
+              memory: "256Mi"
+              cpu: "500m"
+          ports:
+            - containerPort: 8080
+              name: http
+              protocol: TCP
+          env:
+            - name: KOGITO_DATA_INDEX_QUARKUS_PROFILE
+              value: http-events-support
+            - name: KUBERNETES_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+            - name: QUARKUS_DATASOURCE_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  key: POSTGRES_USER
+                  name: postgres-secrets
+            - name: QUARKUS_DATASOURCE_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  key: POSTGRES_PASSWORD
+                  name: postgres-secrets
+          volumeMounts:
+            - name: application-config
+              mountPath: "/home/kogito/config"
+          livenessProbe:
+            failureThreshold: 3
+            httpGet:
+              path: /q/health/live
+              port: 8080
+              scheme: HTTP
+            initialDelaySeconds: 0
+            periodSeconds: 30
+            successThreshold: 1
+            timeoutSeconds: 10
+          readinessProbe:
+            failureThreshold: 3
+            httpGet:
+              path: /q/health/ready
+              port: 8080
+              scheme: HTTP
+            initialDelaySeconds: 0
+            periodSeconds: 30
+            successThreshold: 1
+            timeoutSeconds: 10
+      volumes:
+        - name: application-config
+          configMap:
+            name: dataindex-properties
+      initContainers:
+        - name: init-postgres
+          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
+          imagePullPolicy: IfNotPresent
+          command: ['sh', '-c', 'until (echo 1 > /dev/tcp/postgres.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local/5432) >/dev/null 2>&1; do echo "Waiting for postgres server"; sleep 3; done;']
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app.kubernetes.io/name: data-index-service-postgresql
+  name: data-index-service-postgresql
+spec:
+  ports:
+    - name: http
+      port: 80
+      targetPort: 8080
+  selector:
+    app.kubernetes.io/name: data-index-service-postgresql
+  type: NodePort
+----
+.`application.properties` referenced by `kustomization.yaml`
+[source,properties]
+----
+quarkus.http.port=8080
+quarkus.http.cors=true
+quarkus.http.cors.origins=/.*/
+
+quarkus.datasource.jdbc.url=jdbc:postgresql://postgres:5432/sonataflow?currentSchema=data-index-service
+quarkus.hibernate-orm.database.generation=update
+quarkus.flyway.migrate-at-start=true
+
+# Disable kafka client health check since the quarkus-http connector is being used instead.
+quarkus.smallrye-health.check."io.quarkus.kafka.client.health.KafkaHealthCheck".enabled=false
+----
diff --git a/serverlessworkflow/modules/ROOT/pages/data-index/common/_prerequisites.adoc b/serverlessworkflow/modules/ROOT/pages/data-index/common/_prerequisites.adoc
@@ -0,0 +1,28 @@
+
+.Prerequisites
+* Minikube installed with `registry` addon enabled
+* `kubectl` {kubectl_prereq}
+* SonataFlow operator installed if workflows are deployed. To install the operator you can see xref:cloud/operator/install-serverless-operator.adoc[].
+
+[NOTE]
+====
+We recommend that you start Minikube with the following parameters, note that the `registry` addon must be enabled.
+
+[source,shell]
+----
+minikube start --cpus 4 --memory 10240 --addons registry --addons metrics-server --insecure-registry "10.0.0.0/24" --insecure-registry "localhost:5000"
+----
+
+To verify that the registry addon was property added you can execute this command:
+
+[source,shell]
+----
+minikube addons list | grep registry
+----
+
+----
+| registry                    | minikube | enabled ✅   | Google                         |
+| registry-aliases            | minikube | disabled     | 3rd party (unknown)            |
+| registry-creds              | minikube | disabled     | 3rd party (UPMC Enterprises)   |
+----
+====
diff --git a/serverlessworkflow/modules/ROOT/pages/data-index/common/_querying_dataindex.adoc b/serverlessworkflow/modules/ROOT/pages/data-index/common/_querying_dataindex.adoc
@@ -0,0 +1,105 @@
+[[querying-dataindex-minikube]]
+== Querying Data Index service on Minikube
+
+You can use the public Data Index endpoint to play around with the GraphiQL interface.
+
+.Procedure
+This procedure apply to all use cases with that deploys the Data Index Service.
+
+* Get the Data Index Url:
+[source,shell]
+----
+minikube service data-index-service-postgresql --url -n my_usecase
+----
+
+* Open the GrahiqlUI
+
+Using the url returned, open a browser window in the following url http://192.168.49.2:32409/graphiql/,
+
+[NOTE]
+====
+that IP and port will be different in your installation, and don't forget to add the last slash "/" to the url, otherwise the GraphiqlUI won't be opened.
+====
+
+
+To see the process instances information you can execute this query:
+
+[source,shell]
+----
+{
+  ProcessInstances {
+    id,
+    processId,
+    processName,
+    variables,
+    state,
+    endpoint,
+    serviceUrl,
+    start,
+    end
+  }
+}
+----
+
+The results should be something like:
+
+[source]
+----
+{
+  "data": {
+    "ProcessInstances": [
+      {
+        "id": "3ed8bf63-85c9-425d-9099-49bfb63608cb",
+        "processId": "greeting",
+        "processName": "workflow",
+        "variables": "{\"workflowdata\":{\"name\":\"John\",\"greeting\":\"Hello from JSON Workflow, \",\"language\":\"English\"}}",
+        "state": "COMPLETED",
+        "endpoint": "/greeting",
+        "serviceUrl": "http://greeting",
+        "start": "2023-09-13T06:59:24.319Z",
+        "end": "2023-09-13T06:59:24.400Z"
+      }
+    ]
+  }
+}
+----
+
+To see the jobs instances information, if any, you can execute this query:
+
+[source]
+----
+{
+  Jobs {
+    id,
+    processId,
+    processInstanceId,
+    status,
+    expirationTime,
+    retries,
+    endpoint,
+    callbackEndpoint
+  }
+}
+----
+
+The results should be something like:
+
+[source]
+----
+{
+  "data": {
+    "Jobs": [
+      {
+        "id": "55c7aadb-3dff-4b97-af8e-cc45014b1c0d",
+        "processId": "callbackstatetimeouts",
+        "processInstanceId": "299886b7-2b78-4965-a701-16783c4162d8",
+        "status": "EXECUTED",
+        "expirationTime": null,
+        "retries": 0,
+        "endpoint": "http://jobs-service-postgresql/jobs",
+        "callbackEndpoint": "http://callbackstatetimeouts:80/management/jobs/callbackstatetimeouts/instances/299886b7-2b78-4965-a701-16783c4162d8/timers/-1"
+      }
+    ]
+  }
+}
+----