From 87d388e3e326f40b5ff9a1de0bf4ad7d979b4d5a Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Tue, 6 Feb 2024 17:44:31 +0100 Subject: [PATCH 1/8] [KOGITO-9982] Add persistence to SonataFlow CRD Signed-off-by: Jordi Gil --- .../modules/ROOT/pages/cloud/index.adoc | 8 ++ .../cloud/operator/using-persistence.adoc | 107 ++++++++++++++++++ 2 files changed, 115 insertions(+) create mode 100644 serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/index.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/index.adoc index f9502a1c9..59b944cbe 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/index.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/index.adoc @@ -109,6 +109,14 @@ xref:cloud/operator/service-discovery.adoc[] Learn how to configure and use the {operator_name} Kubernetes service discovery -- +[.card] +-- +[.card-title] +xref:cloud/operator/using-persistence.adoc[] +[.card-description] +Learn how to define the workflow `Persistence` field to allow the workflow to store its context +-- + [.card] -- [.card-title] diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc new file mode 100644 index 000000000..eb74423cb --- /dev/null +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -0,0 +1,107 @@ += Building a Custom Development Image +:compat-mode!: +// Metadata: +:description: Using persistence in the workflow instance to store its context +:keywords: sonataflow, workflow, serverless, operator, kubernetes, persistence + + +This document describes how to configure a SonataFlow instance to use persistence to store the flow's context in a relational database. + +== Configuring the SonataFlow CR to use persistence + +Kubernetes's pods are stateless by definition. In some scenarios this can be a challenge for workloads that require to maintain the status of +the application regardless of the pod's lifecycle. In the case of SonataFlow, the context of the workflow is lost when the pod restarts. +If your workflow requires to recover from such scenarios, you need to make these additions to your Workflow CR: +Use the `persistence` field in the `SonataFlow` workflow spec to define the database service located in the same cluster. +There are 2 ways to accomplish this: + +* Using the Platform CR's defined persistence +When the Platform CR is deployed with its persistence spec populated it enables workflows to leverage on its configuration to populate the persistence +properties in the workflows. + +[source,yaml,subs="attributes+"] +---- +apiVersion: sonataflow.org/v1alpha08 +kind: SonataFlowPlatform +metadata: + name: sonataflow-platform +spec: + persistence: + postgresql: + secretRef: + name: postgres-secrets + userKey: POSTGRES_USER + passwordKey: POSTGRES_PASSWORD + serviceRef: + name: postgres + port: 5432 + databaseName: sonataflow + databaseSchema: shared + build: + config: + strategyOptions: + KanikoBuildCacheEnabled: "true" +--- + +The values of `POSTGRES_USER` and `POSTGRES_PASSWORD` are the keys in the secret that contains the credentials to connect to the postgreSQL instance. +The SonataFlow Workflow CR +is defined with its `Persistence` field defined as empty. + +[source,yaml,subs="attributes+"] +apiVersion: sonataflow.org/v1alpha08 +kind: SonataFlow +metadata: + name: callbackstatetimeouts + annotations: + sonataflow.org/description: Callback State Timeouts Example k8s + sonataflow.org/version: 0.0.1 +spec: + persistence: {} + ... +--- + +This configuration signals the operator that the workflow requires persistence and that it expects its configuration to be populated accordingly. +The operator will add the revelant JDBC properties in the `application.properties` +generated and as part of the pod´s environment so that it can connect to the persistence service defined in the `Platform` CR. + +Note that currently only PostgreSQL is the only persistence supported. + +* Using the custom defined persistence in the SonataFlow CR + +Alternatively you can define a dedicated configuration in the SonataFlow CR instance using the same schema format found in the Platform CRD: + +[source,yaml,subs="attributes+"] +apiVersion: sonataflow.org/v1alpha08 +kind: SonataFlow +metadata: + name: callbackstatetimeouts + annotations: + sonataflow.org/description: Callback State Timeouts + sonataflow.org/version: 0.0.1 +spec: + persistence: + postgresql: + secretRef: + name: postgres-secrets + userKey: POSTGRES_USER + passwordKey: POSTGRES_PASSWORD + serviceRef: + name: postgres + port: 5432 + databaseName: sonataflow + databaseSchema: callbackstatetimeouts + ... +--- + +Like in the Platform CR case, the values of the `POSTGRES_USER` and `POSTGRES_PASSWORD` are the keys in the secret that contain the credentials to connect to +the postgreSQL instace. + +== Conclusion +You can enable SQL persistence in your workflows by configuring each `SonataFlow` CR instance. And when the `SonataFlowPlatform` CR contains the persistence field configured, +the operator uses this information to configure those `SonataFlow` CRs that request persistence. When both the `Platform CR` and the `SonataFlow CR` contain persistence +configuration, the operator will use the `Persistence` values from the `SonataFlow` CR. + +== Additional resources +* xref:cloud/operator/developing-workflows.adoc[] + +include::../../../pages/_common-content/report-issue.adoc[] \ No newline at end of file From 22fe5ab26a776a2e1daa9ee833866fe61cd24bb6 Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Wed, 7 Feb 2024 11:55:35 +0100 Subject: [PATCH 2/8] Change header title Signed-off-by: Jordi Gil --- .../modules/ROOT/pages/cloud/operator/using-persistence.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc index eb74423cb..13e0a735d 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -1,4 +1,4 @@ -= Building a Custom Development Image += Using persistence in the SonataFlow Workflow CR :compat-mode!: // Metadata: :description: Using persistence in the workflow instance to store its context From e6eb8c0f4082353ee3a00e0e73e3213f03114c74 Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Thu, 29 Feb 2024 14:52:33 -0500 Subject: [PATCH 3/8] Changes based on feedback Signed-off-by: Jordi Gil --- .../pages/cloud/operator/using-persistence.adoc | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc index 13e0a735d..3346c289d 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -9,18 +9,18 @@ This document describes how to configure a SonataFlow instance to use persistenc == Configuring the SonataFlow CR to use persistence -Kubernetes's pods are stateless by definition. In some scenarios this can be a challenge for workloads that require to maintain the status of +Kubernetes's pods are stateless by definition. In some scenarios, this can be a challenge for workloads that require maintaining the status of the application regardless of the pod's lifecycle. In the case of SonataFlow, the context of the workflow is lost when the pod restarts. -If your workflow requires to recover from such scenarios, you need to make these additions to your Workflow CR: +If your workflow requires recovery from such scenarios, you must to make these additions to your workflow CR: Use the `persistence` field in the `SonataFlow` workflow spec to define the database service located in the same cluster. There are 2 ways to accomplish this: * Using the Platform CR's defined persistence -When the Platform CR is deployed with its persistence spec populated it enables workflows to leverage on its configuration to populate the persistence +When the Platform CR is deployed with its persistence spec populated it enables workflows to leverage its configuration to populate the persistence properties in the workflows. [source,yaml,subs="attributes+"] ----- +--- apiVersion: sonataflow.org/v1alpha08 kind: SonataFlowPlatform metadata: @@ -61,14 +61,14 @@ spec: --- This configuration signals the operator that the workflow requires persistence and that it expects its configuration to be populated accordingly. -The operator will add the revelant JDBC properties in the `application.properties` +The operator will add the relevant JDBC properties in the `application.properties` generated and as part of the pod´s environment so that it can connect to the persistence service defined in the `Platform` CR. Note that currently only PostgreSQL is the only persistence supported. * Using the custom defined persistence in the SonataFlow CR -Alternatively you can define a dedicated configuration in the SonataFlow CR instance using the same schema format found in the Platform CRD: +Alternatively, you can define a dedicated configuration in the SonataFlow CR instance using the same schema format found in the Platform CRD: [source,yaml,subs="attributes+"] apiVersion: sonataflow.org/v1alpha08 @@ -93,8 +93,8 @@ spec: ... --- -Like in the Platform CR case, the values of the `POSTGRES_USER` and `POSTGRES_PASSWORD` are the keys in the secret that contain the credentials to connect to -the postgreSQL instace. +Like in the Platform CR case, the values of the `POSTGRES_USER` and `POSTGRES_PASSWORD` are the secret keys in the secret that contain the credentials to connect to +the PostgreSQL instace. == Conclusion You can enable SQL persistence in your workflows by configuring each `SonataFlow` CR instance. And when the `SonataFlowPlatform` CR contains the persistence field configured, From b8d6d3c9d2c02e028841cefe9e9a09b95411ba00 Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Mon, 11 Mar 2024 10:04:17 -0400 Subject: [PATCH 4/8] Update serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc Co-authored-by: Ricardo Zanini <1538000+ricardozanini@users.noreply.github.com> --- .../modules/ROOT/pages/cloud/operator/using-persistence.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc index 3346c289d..48aa1a2cc 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -10,7 +10,7 @@ This document describes how to configure a SonataFlow instance to use persistenc == Configuring the SonataFlow CR to use persistence Kubernetes's pods are stateless by definition. In some scenarios, this can be a challenge for workloads that require maintaining the status of -the application regardless of the pod's lifecycle. In the case of SonataFlow, the context of the workflow is lost when the pod restarts. +the application regardless of the pod's lifecycle. In the case of {product_name}, the context of the workflow is lost when the pod restarts. If your workflow requires recovery from such scenarios, you must to make these additions to your workflow CR: Use the `persistence` field in the `SonataFlow` workflow spec to define the database service located in the same cluster. There are 2 ways to accomplish this: From 39534e23e1b0dbb2098e5ee66659fef3fbf158ad Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Mon, 11 Mar 2024 10:04:29 -0400 Subject: [PATCH 5/8] Update serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc Co-authored-by: Ricardo Zanini <1538000+ricardozanini@users.noreply.github.com> --- .../modules/ROOT/pages/cloud/operator/using-persistence.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc index 48aa1a2cc..fb511dc37 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -43,7 +43,7 @@ spec: KanikoBuildCacheEnabled: "true" --- -The values of `POSTGRES_USER` and `POSTGRES_PASSWORD` are the keys in the secret that contains the credentials to connect to the postgreSQL instance. +The values of `POSTGRES_USER` and `POSTGRES_PASSWORD` are the keys in the [Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/) that contains the credentials to connect to the postgreSQL instance. The SonataFlow Workflow CR is defined with its `Persistence` field defined as empty. From 4347c687bbe88fb776b4513b4e361f97ed4d80d2 Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Mon, 11 Mar 2024 10:04:39 -0400 Subject: [PATCH 6/8] Update serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc Co-authored-by: Ricardo Zanini <1538000+ricardozanini@users.noreply.github.com> --- .../modules/ROOT/pages/cloud/operator/using-persistence.adoc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc index fb511dc37..a14e2c988 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -64,7 +64,10 @@ This configuration signals the operator that the workflow requires persistence a The operator will add the relevant JDBC properties in the `application.properties` generated and as part of the pod´s environment so that it can connect to the persistence service defined in the `Platform` CR. -Note that currently only PostgreSQL is the only persistence supported. +[NOTE] +==== +Currently, PostgreSQL is the only persistence supported. +==== * Using the custom defined persistence in the SonataFlow CR From fe550e9d75b4f3688385dc8e14bbca00a7f88357 Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Mon, 11 Mar 2024 10:04:47 -0400 Subject: [PATCH 7/8] Update serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc Co-authored-by: Ricardo Zanini <1538000+ricardozanini@users.noreply.github.com> --- .../modules/ROOT/pages/cloud/operator/using-persistence.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc index a14e2c988..5cb1eab14 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -69,7 +69,7 @@ generated and as part of the pod´s environment so that it can connect to the pe Currently, PostgreSQL is the only persistence supported. ==== -* Using the custom defined persistence in the SonataFlow CR +* Using the custom defined persistence in the `SonataFlow` CR Alternatively, you can define a dedicated configuration in the SonataFlow CR instance using the same schema format found in the Platform CRD: From e750f8e872fbf914aaca9c8dc3689299be7b9d6c Mon Sep 17 00:00:00 2001 From: Jordi Gil Date: Mon, 11 Mar 2024 10:04:54 -0400 Subject: [PATCH 8/8] Update serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc Co-authored-by: Ricardo Zanini <1538000+ricardozanini@users.noreply.github.com> --- .../modules/ROOT/pages/cloud/operator/using-persistence.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc index 5cb1eab14..1c5323f65 100644 --- a/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc +++ b/serverlessworkflow/modules/ROOT/pages/cloud/operator/using-persistence.adoc @@ -71,7 +71,7 @@ Currently, PostgreSQL is the only persistence supported. * Using the custom defined persistence in the `SonataFlow` CR -Alternatively, you can define a dedicated configuration in the SonataFlow CR instance using the same schema format found in the Platform CRD: +Alternatively, you can define a dedicated configuration in the `SonataFlow` CR instance using the same schema format found in the Platform CRD: [source,yaml,subs="attributes+"] apiVersion: sonataflow.org/v1alpha08