diff --git a/docs/guides/README.md b/docs/guides/README.md index e7f0486867..065bc32b01 100644 --- a/docs/guides/README.md +++ b/docs/guides/README.md @@ -16,7 +16,7 @@ aliases: # Guides Guides to show you how to perform tasks with KubeDB: -- [Druid](/docs/guides/kafka/README.md). Shows how to manage Druid using KubeDB. +- [Druid](/docs/guides/druid/README.md). Shows how to manage Druid using KubeDB. - [Elasticsearch](/docs/guides/elasticsearch/README.md). Shows how to manage Elasticsearch & OpenSearch using KubeDB. - [Kafka](/docs/guides/kafka/README.md). Shows how to manage Kafka using KubeDB. - [FerretDB](/docs/guides/ferretdb/README.md). Shows how to manage FerretDB using KubeDB. diff --git a/docs/guides/druid/README.md b/docs/guides/druid/README.md index 3117d47cb3..e1f9c98d61 100644 --- a/docs/guides/druid/README.md +++ b/docs/guides/druid/README.md @@ -15,23 +15,35 @@ aliases: > New to KubeDB? Please start [here](/docs/README.md). +## Overview + +Apache Druid is a real-time analytics database designed for fast slice-and-dice analytics ("OLAP" queries) on large data sets. Druid is most often used as a database for powering use cases where real-time ingest, fast query performance, and high uptime are important. As such, Druid is commonly used for powering GUIs of analytical applications, or as a backend for highly-concurrent APIs that need fast aggregations. Druid works best with event-oriented data. + ## Supported Druid Features -| Features | Availability | -|----------------------------------------------------------------------------|:------------:| -| Clustering | ✓ | -| Druid Dependency Management (MySQL, PostgreSQL and ZooKeeper) | ✓ | -| Authentication & Authorization | ✓ | -| Custom Configuration | ✓ | -| Backup/Recovery: Instant, Scheduled ( [KubeStash](https://kubestash.com/)) | ✓ | -| Monitoring with Prometheus & Grafana | ✓ | -| Builtin Prometheus Discovery | ✓ | -| Using Prometheus operator | ✓ | -| Externally manageable Auth Secret | ✓ | -| Reconfigurable Health Checker | ✓ | -| Persistent volume | ✓ | -| Dashboard ( Druid Web Console ) | ✓ | +| Features | Availability | +|------------------------------------------------------------------------------------|:-----:| +| Clustering | ✓ | +| Druid Dependency Management (MySQL, PostgreSQL and ZooKeeper) | ✓ | +| Authentication & Authorization | ✓ | +| Custom Configuration | ✓ | +| Backup/Recovery: Instant, Scheduled ( [KubeStash](https://kubestash.com/)) | ✓ | +| Monitoring with Prometheus & Grafana | ✓ | +| Builtin Prometheus Discovery | ✓ | +| Using Prometheus operator | ✓ | +| Externally manageable Auth Secret | ✓ | +| Reconfigurable Health Checker | ✓ | +| Persistent volume | ✓ | +| Dashboard ( Druid Web Console ) | ✓ | +| Automated Version Update | ✓ | +| Automatic Vertical Scaling | ✓ | +| Automated Horizontal Scaling | ✓ | +| Automated db-configure Reconfiguration | ✓ | +| TLS: Add, Remove, Update, Rotate ( [Cert Manager](https://cert-manager.io/docs/) ) | ✓ | +| Automated Reprovision | ✓ | +| Automated Volume Expansion | ✓ | +| Autoscaling (vertically) | ✓ | ## Supported Druid Versions @@ -39,7 +51,7 @@ KubeDB supports The following Druid versions. - `28.0.1` - `30.0.1` -> The listed DruidVersions are tested and provided as a part of the installation process (ie. catalog chart), but you are open to create your own [DruidVersion](/docs/guides/druid/concepts/catalog.md) object with your custom Druid image. +> The listed DruidVersions are tested and provided as a part of the installation process (ie. catalog chart), but you are open to create your own [DruidVersion](/docs/guides/druid/concepts/druidversion.md) object with your custom Druid image. ## Lifecycle of Druid Object @@ -52,14 +64,12 @@ ref : https://cacoo.com/diagrams/bbB63L6KRIbPLl95/9A5B0

## User Guide -- [Quickstart Druid](/docs/guides/druid/quickstart/overview/index.md) with KubeDB Operator. - -[//]: # (- Druid Clustering supported by KubeDB) - -[//]: # ( - [Topology Clustering](/docs/guides/druid/clustering/topology-cluster/index.md)) - -[//]: # (- Use [kubedb cli](/docs/guides/druid/cli/cli.md) to manage databases like kubectl for Kubernetes.) - +- [Quickstart Druid](/docs/guides/druid/quickstart/guide/index.md) with KubeDB Operator. +- [Druid Clustering](/docs/guides/druid/clustering/overview/index.md) with KubeDB Operator. +- [Backup & Restore](/docs/guides/druid/backup/overview/index.md) Druid databases using KubeStash. +- Start [Druid with Custom Config](/docs/guides/druid/configuration/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). +- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md). - Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). - -[//]: # (- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).) \ No newline at end of file +- Detail concepts of [DruidVersion object](/docs/guides/druid/concepts/druidversion.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). \ No newline at end of file diff --git a/docs/guides/druid/autoscaler/_index.md b/docs/guides/druid/autoscaler/_index.md new file mode 100644 index 0000000000..a39f2bfba3 --- /dev/null +++ b/docs/guides/druid/autoscaler/_index.md @@ -0,0 +1,10 @@ +--- +title: Autoscaling +menu: + docs_{{ .version }}: + identifier: guides-druid-autoscaler + name: Autoscaling + parent: guides-druid + weight: 100 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/druid/autoscaler/compute/_index.md b/docs/guides/druid/autoscaler/compute/_index.md new file mode 100644 index 0000000000..c2c1eea280 --- /dev/null +++ b/docs/guides/druid/autoscaler/compute/_index.md @@ -0,0 +1,10 @@ +--- +title: Compute Autoscaling +menu: + docs_{{ .version }}: + identifier: guides-druid-autoscaler-compute + name: Compute Autoscaling + parent: guides-druid-autoscaler + weight: 46 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/autoscaler/compute/guide.md b/docs/guides/druid/autoscaler/compute/guide.md new file mode 100644 index 0000000000..b0810f67b2 --- /dev/null +++ b/docs/guides/druid/autoscaler/compute/guide.md @@ -0,0 +1,864 @@ +--- +title: Druid Topology Autoscaling +menu: + docs_{{ .version }}: + identifier: guides-druid-autoscaler-compute-guide + name: Druid Compute Autoscaling + parent: guides-druid-autoscaler-compute + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Autoscaling the Compute Resource of a Druid Topology Cluster + +This guide will show you how to use `KubeDB` to autoscale compute resources i.e. cpu and memory of a Druid topology cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. + +- Install `KubeDB` Provisioner, Ops-manager and Autoscaler operator in your cluster following the steps [here](/docs/setup/README.md). + +- Install `Metrics Server` from [here](https://github.com/kubernetes-sigs/metrics-server#installation) + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidAutoscaler](/docs/guides/druid/concepts/druidautoscaler.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Compute Resource Autoscaling Overview](/docs/guides/druid/autoscaler/compute/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/examples/druid](/docs/examples/druid) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +## Autoscaling of Topology Cluster + +Here, we are going to deploy a `Druid` Topology Cluster using a supported version by `KubeDB` operator. Then we are going to apply `DruidAutoscaler` to set up autoscaling. + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/autoscaler/compute/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +Now, we are going to deploy a `Druid` combined cluster with version `28.0.1`. + +### Deploy Druid Cluster + +In this section, we are going to deploy a Druid Topology cluster with version `28.0.1`. Then, in the next section we will set up autoscaling for this database using `DruidAutoscaler` CRD. Below is the YAML of the `Druid` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: WipeOut +``` + +Let's create the `Druid` CRO we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/autoscaler/compute/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait until `druid-cluster` has status `Ready`. i.e, + +```bash +$ kubectl get kf -n demo -w +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 0s +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 24s +. +. +druid-cluster kubedb.com/v1alpha2 28.0.1 Ready 118s +``` + +## Druid Topology Autoscaler + +Let's check the Druid resources for coordinators and historicals, + +```bash +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.coordinators.podTemplate.spec.containers[].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} + +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.historicals.podTemplate.spec.containers[].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} +``` + +Let's check the coordinators and historicals Pod containers resources, + +```bash +$ kubectl get pod -n demo druid-cluster-coordinators-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} + +$ kubectl get pod -n demo druid-cluster-historicals-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} +``` + +You can see from the above outputs that the resources for coordinators and historicals are same as the one we have assigned while deploying the druid. + +We are now ready to apply the `DruidAutoscaler` CRO to set up autoscaling for these coordinators and historicals nodes. + +### Compute Resource Autoscaling + +Here, we are going to set up compute resource autoscaling using a DruidAutoscaler Object. + +#### Create DruidAutoscaler Object + +In order to set up compute resource autoscaling for this topology cluster, we have to create a `DruidAutoscaler` CRO with our desired configuration. Below is the YAML of the `DruidAutoscaler` object that we are going to create, + +```yaml +apiVersion: autoscaling.kubedb.com/v1alpha1 +kind: DruidAutoscaler +metadata: + name: druid-autoscaler + namespace: demo +spec: + databaseRef: + name: druid-quickstart + compute: + coordinators: + trigger: "On" + podLifeTimeThreshold: 1m + minAllowed: + cpu: 600m + memory: 2Gi + maxAllowed: + cpu: 1000m + memory: 5Gi + resourceDiffPercentage: 20 + controlledResources: ["cpu", "memory"] + historicals: + trigger: "On" + podLifeTimeThreshold: 1m + minAllowed: + cpu: 600m + memory: 2Gi + maxAllowed: + cpu: 1000m + memory: 5Gi + resourceDiffPercentage: 20 + controlledResources: [ "cpu", "memory"] +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing compute resource scaling operation on `druid-cluster` cluster. +- `spec.compute.coordinators.trigger` specifies that compute autoscaling is enabled for this node. +- `spec.compute.coordinators.podLifeTimeThreshold` specifies the minimum lifetime for at least one of the pod to initiate a vertical scaling. +- `spec.compute.coordinators.resourceDiffPercentage` specifies the minimum resource difference in percentage. The default is 10%. If the difference between current & recommended resource is less than ResourceDiffPercentage, Autoscaler Operator will ignore the updating. +- `spec.compute.coordinators.minAllowed` specifies the minimum allowed resources for the cluster. +- `spec.compute.coordinators.maxAllowed` specifies the maximum allowed resources for the cluster. +- `spec.compute.coordinators.controlledResources` specifies the resources that are controlled by the autoscaler. +- `spec.compute.coordinators.containerControlledValues` specifies which resource values should be controlled. The default is "RequestsAndLimits". +- `spec.compute.historicals` can be configured the same way shown above. +- `spec.opsRequestOptions` contains the options to pass to the created OpsRequest. It has 2 fields. + - `timeout` specifies the timeout for the OpsRequest. + - `apply` specifies when the OpsRequest should be applied. The default is "IfReady". + +> **Note:** You can also configure autoscaling configurations for all other nodes as well. You can apply autoscaler for each node in separate YAML or combinedly in one a YAML as shown above. + +Let's create the `DruidAutoscaler` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/autoscaler/compute/yamls/druid-autoscaler.yaml +druidautoscaler.autoscaling.kubedb.com/druid-autoscaler created +``` + +#### Verify Autoscaling is set up successfully + +Let's check that the `druidautoscaler` resource is created successfully, + +```bash +$ kubectl describe druidautoscaler druid-autoscaler -n demo + kubectl describe druidautoscaler druid-autoscaler -n demo +Name: druid-autoscaler +Namespace: demo +Labels: +Annotations: +API Version: autoscaling.kubedb.com/v1alpha1 +Kind: DruidAutoscaler +Metadata: + Creation Timestamp: 2024-10-24T10:04:22Z + Generation: 1 + Managed Fields: + API Version: autoscaling.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:compute: + .: + f:coordinators: + .: + f:controlledResources: + f:maxAllowed: + .: + f:cpu: + f:memory: + f:minAllowed: + .: + f:cpu: + f:memory: + f:podLifeTimeThreshold: + f:resourceDiffPercentage: + f:trigger: + f:historicals: + .: + f:controlledResources: + f:maxAllowed: + .: + f:cpu: + f:memory: + f:minAllowed: + .: + f:cpu: + f:memory: + f:podLifeTimeThreshold: + f:resourceDiffPercentage: + f:trigger: + f:databaseRef: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-24T10:04:22Z + API Version: autoscaling.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:ownerReferences: + .: + k:{"uid":"c2a5c29d-3589-49d8-bc18-585b9c05bf8d"}: + Manager: kubedb-autoscaler + Operation: Update + Time: 2024-10-24T10:04:22Z + API Version: autoscaling.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:checkpoints: + f:conditions: + f:vpas: + Manager: kubedb-autoscaler + Operation: Update + Subresource: status + Time: 2024-10-24T10:16:20Z + Owner References: + API Version: kubedb.com/v1alpha2 + Block Owner Deletion: true + Controller: true + Kind: Druid + Name: druid-cluster + UID: c2a5c29d-3589-49d8-bc18-585b9c05bf8d + Resource Version: 274969 + UID: 069fbdd7-87ad-4fd7-acc7-9753fa188312 +Spec: + Compute: + Coordinators: + Controlled Resources: + cpu + memory + Max Allowed: + Cpu: 1000m + Memory: 5Gi + Min Allowed: + Cpu: 600m + Memory: 2Gi + Pod Life Time Threshold: 1m + Resource Diff Percentage: 20 + Trigger: On + Historicals: + Controlled Resources: + cpu + memory + Max Allowed: + Cpu: 1000m + Memory: 5Gi + Min Allowed: + Cpu: 600m + Memory: 2Gi + Pod Life Time Threshold: 1m + Resource Diff Percentage: 20 + Trigger: On + Database Ref: + Name: druid-cluster +Status: + Checkpoints: + Cpu Histogram: + Bucket Weights: + Index: 0 + Weight: 10000 + Index: 5 + Weight: 490 + Reference Timestamp: 2024-10-24T10:05:00Z + Total Weight: 2.871430450948392 + First Sample Start: 2024-10-24T10:05:07Z + Last Sample Start: 2024-10-24T10:16:03Z + Last Update Time: 2024-10-24T10:16:20Z + Memory Histogram: + Bucket Weights: + Index: 25 + Weight: 3648 + Index: 29 + Weight: 10000 + Reference Timestamp: 2024-10-24T10:10:00Z + Total Weight: 3.3099198846728424 + Ref: + Container Name: druid + Vpa Object Name: druid-cluster-historicals + Total Samples Count: 12 + Version: v3 + Cpu Histogram: + Bucket Weights: + Index: 0 + Weight: 3040 + Index: 1 + Weight: 10000 + Index: 2 + Weight: 3278 + Index: 14 + Weight: 1299 + Reference Timestamp: 2024-10-24T10:10:00Z + Total Weight: 1.0092715955023177 + First Sample Start: 2024-10-24T10:04:53Z + Last Sample Start: 2024-10-24T10:14:03Z + Last Update Time: 2024-10-24T10:14:20Z + Memory Histogram: + Bucket Weights: + Index: 24 + Weight: 10000 + Index: 27 + Weight: 8706 + Reference Timestamp: 2024-10-24T10:10:00Z + Total Weight: 3.204567438391289 + Ref: + Container Name: druid + Vpa Object Name: druid-cluster-coordinators + Total Samples Count: 10 + Version: v3 + Conditions: + Last Transition Time: 2024-10-24T10:07:19Z + Message: Successfully created druidOpsRequest demo/drops-druid-cluster-coordinators-g02xtu + Observed Generation: 1 + Reason: CreateOpsRequest + Status: True + Type: CreateOpsRequest + Vpas: + Conditions: + Last Transition Time: 2024-10-24T10:05:19Z + Status: True + Type: RecommendationProvided + Recommendation: + Container Recommendations: + Container Name: druid + Lower Bound: + Cpu: 600m + Memory: 2Gi + Target: + Cpu: 600m + Memory: 2Gi + Uncapped Target: + Cpu: 100m + Memory: 764046746 + Upper Bound: + Cpu: 1 + Memory: 5Gi + Vpa Name: druid-cluster-historicals + Conditions: + Last Transition Time: 2024-10-24T10:06:19Z + Status: True + Type: RecommendationProvided + Recommendation: + Container Recommendations: + Container Name: druid + Lower Bound: + Cpu: 600m + Memory: 2Gi + Target: + Cpu: 600m + Memory: 2Gi + Uncapped Target: + Cpu: 100m + Memory: 671629701 + Upper Bound: + Cpu: 1 + Memory: 5Gi + Vpa Name: druid-cluster-coordinators +Events: +``` +So, the `druidautoscaler` resource is created successfully. + +you can see in the `Status.VPAs.Recommendation` section, that recommendation has been generated for our database. Our autoscaler operator continuously watches the recommendation generated and creates an `druidopsrequest` based on the recommendations, if the database pods resources are needed to scaled up or down. + +Let's watch the `druidopsrequest` in the demo namespace to see if any `druidopsrequest` object is created. After some time you'll see that a `druidopsrequest` will be created based on the recommendation. + +```bash +$ watch kubectl get druidopsrequest -n demo +Every 2.0s: kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +drops-druid-cluster-coordinators-g02xtu VerticalScaling Progressing 8m +drops-druid-cluster-historicals-g3oqje VerticalScaling Progressing 8m + +``` +Progressing +Let's wait for the ops request to become successful. + +```bash +$ kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +drops-druid-cluster-coordinators-g02xtu VerticalScaling Successful 12m +drops-druid-cluster-historicals-g3oqje VerticalScaling Successful 13m +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to scale the cluster. + +```bash +$ kubectl describe druidopsrequests -n demo drops-druid-cluster-coordinators-f6qbth +Name: drops-druid-cluster-coordinators-g02xtu +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-cluster + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-24T10:07:19Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: + f:app.kubernetes.io/component: + f:app.kubernetes.io/instance: + f:app.kubernetes.io/managed-by: + f:app.kubernetes.io/name: + f:ownerReferences: + .: + k:{"uid":"069fbdd7-87ad-4fd7-acc7-9753fa188312"}: + f:spec: + .: + f:apply: + f:databaseRef: + f:type: + f:verticalScaling: + .: + f:coordinators: + .: + f:resources: + .: + f:limits: + .: + f:memory: + f:requests: + .: + f:cpu: + f:memory: + Manager: kubedb-autoscaler + Operation: Update + Time: 2024-10-24T10:07:19Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-24T10:07:43Z + Owner References: + API Version: autoscaling.kubedb.com/v1alpha1 + Block Owner Deletion: true + Controller: true + Kind: DruidAutoscaler + Name: druid-autoscaler + UID: 069fbdd7-87ad-4fd7-acc7-9753fa188312 + Resource Version: 273990 + UID: d14d964b-f4ae-4570-a296-38e91c802473 +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Type: VerticalScaling + Vertical Scaling: + Coordinators: + Resources: + Limits: + Memory: 2Gi + Requests: + Cpu: 600m + Memory: 2Gi +Status: + Conditions: + Last Transition Time: 2024-10-24T10:07:19Z + Message: Druid ops-request has started to vertically scale the Druid nodes + Observed Generation: 1 + Reason: VerticalScaling + Status: True + Type: VerticalScaling + Last Transition Time: 2024-10-24T10:07:28Z + Message: Successfully updated PetSets Resources + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-24T10:07:43Z + Message: Successfully Restarted Pods With Resources + Observed Generation: 1 + Reason: RestartPods + Status: True + Type: RestartPods + Last Transition Time: 2024-10-24T10:07:33Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-24T10:07:33Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-24T10:07:38Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-24T10:07:43Z + Message: Successfully completed the vertical scaling for RabbitMQ + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 12m KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-druid-cluster-coordinators-g02xtu + Normal Starting 12m KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 12m KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-coordinators-g02xtu + Normal UpdatePetSets 12m KubeDB Ops-manager Operator Successfully updated PetSets Resources + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 12m KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 12m KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 12m KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Normal RestartPods 12m KubeDB Ops-manager Operator Successfully Restarted Pods With Resources + Normal Starting 12m KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 12m KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-coordinators-g02xtu +``` + +Let's describe the other `DruidOpsRequest` created for scaling of historicals. + +```bash +$ kubectl describe druidopsrequests -n demo drops-druid-cluster-historicals-g3oqje +Name: drops-druid-cluster-historicals-g3oqje +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-cluster + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-24T10:06:19Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: + f:app.kubernetes.io/component: + f:app.kubernetes.io/instance: + f:app.kubernetes.io/managed-by: + f:app.kubernetes.io/name: + f:ownerReferences: + .: + k:{"uid":"069fbdd7-87ad-4fd7-acc7-9753fa188312"}: + f:spec: + .: + f:apply: + f:databaseRef: + f:type: + f:verticalScaling: + .: + f:historicals: + .: + f:resources: + .: + f:limits: + .: + f:memory: + f:requests: + .: + f:cpu: + f:memory: + Manager: kubedb-autoscaler + Operation: Update + Time: 2024-10-24T10:06:19Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-24T10:06:37Z + Owner References: + API Version: autoscaling.kubedb.com/v1alpha1 + Block Owner Deletion: true + Controller: true + Kind: DruidAutoscaler + Name: druid-autoscaler + UID: 069fbdd7-87ad-4fd7-acc7-9753fa188312 + Resource Version: 273770 + UID: fc13624c-42d4-4b03-9448-80f451b1a888 +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Type: VerticalScaling + Vertical Scaling: + Historicals: + Resources: + Limits: + Memory: 2Gi + Requests: + Cpu: 600m + Memory: 2Gi +Status: + Conditions: + Last Transition Time: 2024-10-24T10:06:19Z + Message: Druid ops-request has started to vertically scale the Druid nodes + Observed Generation: 1 + Reason: VerticalScaling + Status: True + Type: VerticalScaling + Last Transition Time: 2024-10-24T10:06:22Z + Message: Successfully updated PetSets Resources + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-24T10:06:37Z + Message: Successfully Restarted Pods With Resources + Observed Generation: 1 + Reason: RestartPods + Status: True + Type: RestartPods + Last Transition Time: 2024-10-24T10:06:27Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-24T10:06:27Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-24T10:06:32Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-historicals-0 + Last Transition Time: 2024-10-24T10:06:37Z + Message: Successfully completed the vertical scaling for RabbitMQ + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 16m KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-druid-cluster-historicals-g3oqje + Normal Starting 16m KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 16m KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-historicals-g3oqje + Normal UpdatePetSets 16m KubeDB Ops-manager Operator Successfully updated PetSets Resources + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 16m KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 16m KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 16m KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Normal RestartPods 16m KubeDB Ops-manager Operator Successfully Restarted Pods With Resources + Normal Starting 16m KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 16m KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-historicals-g3oqje + +``` + +Now, we are going to verify from the Pod, and the Druid yaml whether the resources of the coordinators and historicals node has updated to meet up the desired state, Let's check, + +```bash +$ kubectl get pod -n demo druid-cluster-coordinators-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "memory": "1536Mi" + }, + "requests": { + "cpu": "600m", + "memory": "1536Mi" + } +} + +$ kubectl get pod -n demo druid-cluster-historicals-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "memory": "2Gi" + }, + "requests": { + "cpu": "600m", + "memory": "2Gi" + } +} + +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.coordinators.podTemplate.spec.containers[].resources' +{ + "limits": { + "memory": "1536Mi" + }, + "requests": { + "cpu": "600m", + "memory": "1536Mi" + } +} + +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.historicals.podTemplate.spec.containers[].resources' +{ + "limits": { + "memory": "2Gi" + }, + "requests": { + "cpu": "600m", + "memory": "2Gi" + } +} +``` + +The above output verifies that we have successfully auto scaled the resources of the Druid topology cluster for coordinators and historicals. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete druidopsrequest -n demo drops-druid-cluster-coordinators-g02xtu drops-druid-cluster-historicals-g3oqje +kubectl delete druidautoscaler -n demo druid-autoscaler +kubectl delete dr -n demo druid-cluster +kubectl delete ns demo +``` +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/autoscaler/compute/images/compute-autoscaling.png b/docs/guides/druid/autoscaler/compute/images/compute-autoscaling.png new file mode 100644 index 0000000000..9406e2b2e9 Binary files /dev/null and b/docs/guides/druid/autoscaler/compute/images/compute-autoscaling.png differ diff --git a/docs/guides/druid/autoscaler/compute/overview.md b/docs/guides/druid/autoscaler/compute/overview.md new file mode 100644 index 0000000000..457bbfe62c --- /dev/null +++ b/docs/guides/druid/autoscaler/compute/overview.md @@ -0,0 +1,55 @@ +--- +title: Druid Compute Autoscaling Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-autoscaler-compute-overview + name: Overview + parent: guides-druid-autoscaler-compute + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Compute Resource Autoscaling + +This guide will give an overview on how KubeDB Autoscaler operator autoscales the database compute resources i.e. cpu and memory using `druidautoscaler` crd. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidAutoscaler](/docs/guides/druid/concepts/druidautoscaler.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + +## How Compute Autoscaling Works + +The following diagram shows how KubeDB Autoscaler operator autoscales the resources of `Druid` database components. Open the image in a new tab to see the enlarged version. + +
+  Volume Expansion process of Druid +
Fig: Compute Auto Scaling process of Druid
+
+ +The Auto Scaling process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource Object (CRO). + +2. `KubeDB` Provisioner operator watches the `Druid` CRO. + +3. When the operator finds a `Druid` CRO, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Then, in order to set up autoscaling of the various components (ie. Coordinators, Overlords, Historicals, MiddleManagers, Brokers, Routers) of the `Druid` cluster the user creates a `DruidAutoscaler` CRO with desired configuration. + +5. `KubeDB` Autoscaler operator watches the `DruidAutoscaler` CRO. + +6. `KubeDB` Autoscaler operator generates recommendation using the modified version of kubernetes [official recommender](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/pkg/recommender) for different components of the database, as specified in the `DruidAutoscaler` CRO. + +7. If the generated recommendation doesn't match the current resources of the database, then `KubeDB` Autoscaler operator creates a `DruidOpsRequest` CRO to scale the database to match the recommendation generated. + +8. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CRO. + +9. Then the `KubeDB` Ops-manager operator will scale the database component vertically as specified on the `DruidOpsRequest` CRO. + +In the next docs, we are going to show a step-by-step guide on Autoscaling of various Druid database components using `DruidAutoscaler` CRD. diff --git a/docs/guides/druid/autoscaler/compute/yamls/deep-storage-config.yaml b/docs/guides/druid/autoscaler/compute/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/autoscaler/compute/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/autoscaler/compute/yamls/druid-autoscaler.yaml b/docs/guides/druid/autoscaler/compute/yamls/druid-autoscaler.yaml new file mode 100644 index 0000000000..ef5d96ec78 --- /dev/null +++ b/docs/guides/druid/autoscaler/compute/yamls/druid-autoscaler.yaml @@ -0,0 +1,31 @@ +apiVersion: autoscaling.kubedb.com/v1alpha1 +kind: DruidAutoscaler +metadata: + name: druid-storage-autoscaler + namespace: demo +spec: + databaseRef: + name: druid-cluster + compute: + coordinators: + trigger: "On" + podLifeTimeThreshold: 1m + minAllowed: + cpu: 600m + memory: 2Gi + maxAllowed: + cpu: 1000m + memory: 5Gi + resourceDiffPercentage: 20 + controlledResources: ["cpu", "memory"] + historicals: + trigger: "On" + podLifeTimeThreshold: 1m + minAllowed: + cpu: 600m + memory: 2Gi + maxAllowed: + cpu: 1000m + memory: 5Gi + resourceDiffPercentage: 20 + controlledResources: [ "cpu", "memory"] diff --git a/docs/guides/druid/autoscaler/compute/yamls/druid-cluster.yaml b/docs/guides/druid/autoscaler/compute/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..ffac2b300b --- /dev/null +++ b/docs/guides/druid/autoscaler/compute/yamls/druid-cluster.yaml @@ -0,0 +1,31 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + historicals: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + middleManagers: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + routers: + replicas: 1 + deletionPolicy: Delete diff --git a/docs/guides/druid/autoscaler/storage/_index.md b/docs/guides/druid/autoscaler/storage/_index.md new file mode 100644 index 0000000000..e197ec8429 --- /dev/null +++ b/docs/guides/druid/autoscaler/storage/_index.md @@ -0,0 +1,10 @@ +--- +title: Storage Autoscaling +menu: + docs_{{ .version }}: + identifier: guides-druid-autoscaler-storage + name: Storage Autoscaling + parent: guides-druid-autoscaler + weight: 46 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/autoscaler/storage/guide.md b/docs/guides/druid/autoscaler/storage/guide.md new file mode 100644 index 0000000000..02b3571625 --- /dev/null +++ b/docs/guides/druid/autoscaler/storage/guide.md @@ -0,0 +1,896 @@ +--- +title: Druid Topology Autoscaling +menu: + docs_{{ .version }}: + identifier: guides-druid-autoscaler-storage-guide + name: Druid Storage Autoscaling + parent: guides-druid-autoscaler-storage + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Storage Autoscaling of a Druid Topology Cluster + +This guide will show you how to use `KubeDB` to autoscale the storage of a Druid Topology cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. + +- Install `KubeDB` Provisioner, Ops-manager and Autoscaler operator in your cluster following the steps [here](/docs/setup/README.md). + +- Install `Metrics Server` from [here](https://github.com/kubernetes-sigs/metrics-server#installation) + +- Install Prometheus from [here](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) + +- You must have a `StorageClass` that supports volume expansion. + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidAutoscaler](/docs/guides/druid/concepts/druidautoscaler.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Storage Autoscaling Overview](/docs/guides/druid/autoscaler/storage/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/examples/druid](/docs/examples/druid) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +## Storage Autoscaling of Topology Cluster + +At first verify that your cluster has a storage class, that supports volume expansion. Let's check, + +```bash +$ kubectl get storageclass +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 28h +longhorn (default) driver.longhorn.io Delete Immediate true 28h +longhorn-static driver.longhorn.io Delete Immediate true 28h +``` + +We can see from the output the `longhorn` storage class has `ALLOWVOLUMEEXPANSION` field as true. So, this storage class supports volume expansion. We can use it. + +Now, we are going to deploy a `Druid` topology using a supported version by `KubeDB` operator. Then we are going to apply `DruidAutoscaler` to set up autoscaling. + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/autoscaler/storage/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +### Deploy Druid Cluster + +In this section, we are going to deploy a Druid topology cluster with monitoring enabled and with version `28.0.1`. Then, in the next section we will set up autoscaling for this cluster using `DruidAutoscaler` CRD. Below is the YAML of the `Druid` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + historicals: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + middleManagers: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + routers: + replicas: 1 + deletionPolicy: Delete + monitor: + agent: prometheus.io/operator + prometheus: + serviceMonitor: + labels: + release: prometheus + interval: 10s +``` + +Let's create the `Druid` CRO we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/autoscaler/storage/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait until `druid-cluster` has status `Ready`. i.e, + +```bash +$ kubectl get dr -n demo -w +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 0s +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 24s +. +. +druid-cluster kubedb.com/v1alpha2 28.0.1 Ready 2m20s +``` + +Let's check volume size from petset, and from the persistent volume, + +```bash +$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"1Gi" +$ kubectl get petset -n demo druid-cluster-middleManagers -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"1Gi" +$ kubectl get pv -n demo +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE +pvc-2c0ef2aa-0438-4d75-9cb2-c12a176bae6a 1Gi RWO Delete Bound demo/druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 longhorn 95s +pvc-5f4cea5f-e0c8-4339-b67c-9cb8b02ba49d 1Gi RWO Delete Bound demo/druid-cluster-segment-cache-druid-cluster-historicals-0 longhorn 96s +``` + +You can see the petset for both historicals and middleManagers has 1GB storage, and the capacity of all the persistent volume is also 1GB. + +We are now ready to apply the `DruidAutoscaler` CRO to set up storage autoscaling for this cluster(historicals and middleManagers). + +### Storage Autoscaling + +Here, we are going to set up storage autoscaling using a DruidAutoscaler Object. + +#### Create DruidAutoscaler Object + +In order to set up vertical autoscaling for this topology cluster, we have to create a `DruidAutoscaler` CRO with our desired configuration. Below is the YAML of the `DruidAutoscaler` object that we are going to create, + +```yaml +apiVersion: autoscaling.kubedb.com/v1alpha1 +kind: DruidAutoscaler +metadata: + name: druid-storage-autoscaler + namespace: demo +spec: + databaseRef: + name: druid-cluster + storage: + historicals: + expansionMode: "Offline" + trigger: "On" + usageThreshold: 60 + scalingThreshold: 100 + middleManagers: + expansionMode: "Offline" + trigger: "On" + usageThreshold: 60 + scalingThreshold: 100 +``` + +Here, + +- `spec.clusterRef.name` specifies that we are performing vertical scaling operation on `druid-cluster` cluster. +- `spec.storage.historicals.trigger/spec.storage.middleManagers.trigger` specifies that storage autoscaling is enabled for historicals and middleManagers of topology cluster. +- `spec.storage.historicals.usageThreshold/spec.storage.middleManagers.usageThreshold` specifies storage usage threshold, if storage usage exceeds `60%` then storage autoscaling will be triggered. +- `spec.storage.historicals.scalingThreshold/spec.storage.historicals.scalingThreshold` specifies the scaling threshold. Storage will be scaled to `100%` of the current amount. +- It has another field `spec.storage.historicals.expansionMode/spec.storage.middleManagers.expansionMode` to set the opsRequest volumeExpansionMode, which support two values: `Online` & `Offline`. Default value is `Online`. + +Let's create the `DruidAutoscaler` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/autoscaler/storage/yamls/druid-storage-autoscaler.yaml +druidautoscaler.autoscaling.kubedb.com/druid-storage-autoscaler created +``` + +#### Storage Autoscaling is set up successfully + +Let's check that the `druidautoscaler` resource is created successfully, + +```bash +$ kubectl get druidautoscaler -n demo +NAME AGE +druid-storage-autoscaler 34s + +$ kubectl describe druidautoscaler -n demo druid-storage-autoscaler +Name: druid-storage-autoscaler +Namespace: demo +Labels: +Annotations: +API Version: autoscaling.kubedb.com/v1alpha1 +Kind: DruidAutoscaler +Metadata: + Creation Timestamp: 2024-10-25T09:52:37Z + Generation: 1 + Managed Fields: + API Version: autoscaling.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:databaseRef: + f:storage: + .: + f:historicals: + .: + f:expansionMode: + f:scalingThreshold: + f:trigger: + f:usageThreshold: + f:middleManagers: + .: + f:expansionMode: + f:scalingThreshold: + f:trigger: + f:usageThreshold: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-25T09:52:37Z + API Version: autoscaling.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:ownerReferences: + .: + k:{"uid":"712730e8-41ef-4700-b184-825b30ecbc8c"}: + Manager: kubedb-autoscaler + Operation: Update + Time: 2024-10-25T09:52:37Z + Owner References: + API Version: kubedb.com/v1alpha2 + Block Owner Deletion: true + Controller: true + Kind: Druid + Name: druid-cluster + UID: 712730e8-41ef-4700-b184-825b30ecbc8c + Resource Version: 226662 + UID: 57cbd906-a9b7-4649-bfe0-304840bb60c1 +Spec: + Database Ref: + Name: druid-cluster + Ops Request Options: + Apply: IfReady + Storage: + Historicals: + Expansion Mode: Offline + Scaling Rules: + Applies Upto: + Threshold: 100pc + Scaling Threshold: 100 + Trigger: On + Usage Threshold: 60 + Middle Managers: + Expansion Mode: Offline + Scaling Rules: + Applies Upto: + Threshold: 100pc + Scaling Threshold: 100 + Trigger: On + Usage Threshold: 60 +Events: +``` +So, the `druidautoscaler` resource is created successfully. + +Now, for this demo, we are going to manually fill up the persistent volume to exceed the `usageThreshold` using `dd` command to see if storage autoscaling is working or not. + +We are autoscaling volume for both historicals and middleManagers. So we need to fill up the persistent volume for both historicals and middleManagers. + +1. Lets exec into the historicals pod and fill the cluster volume using the following commands: + +```bash +$ kubectl exec -it -n demo druid-cluster-historicals-0 -- bash +bash-5.1$ df -h /druid/data/segments +Filesystem Size Used Available Use% Mounted on +/dev/longhorn/pvc-d4ef15ef-b1af-4a1f-ad25-ad9bc990a2fb 973.4M 92.0K 957.3M 0% /druid/data/segment + +bash-5.1$ dd if=/dev/zero of=/druid/data/segments/file.img bs=600M count=1 +1+0 records in +1+0 records out +629145600 bytes (600.0MB) copied, 46.709228 seconds, 12.8MB/s + +bash-5.1$ df -h /druid/data/segments +Filesystem Size Used Available Use% Mounted on +/dev/longhorn/pvc-d4ef15ef-b1af-4a1f-ad25-ad9bc990a2fb 973.4M 600.1M 357.3M 63% /druid/data/segments +``` + +2. Let's exec into the middleManagers pod and fill the cluster volume using the following commands: + +```bash +$ kubectl exec -it -n demo druid-cluster-middleManagers-0 -- bash +druid@druid-cluster-middleManagers-0:~$ df -h /var/druid/task +Filesystem Size Used Available Use% Mounted on +/dev/longhorn/pvc-2c0ef2aa-0438-4d75-9cb2-c12a176bae6a 973.4M 24.0K 957.4M 0% /var/druid/task +druid@druid-cluster-middleManagers-0:~$ dd if=/dev/zero of=/var/druid/task/file.img bs=600M count=1 +1+0 records in +1+0 records out +629145600 bytes (629 MB, 600 MiB) copied, 3.39618 s, 185 MB/s +druid@druid-cluster-middleManagers-0:~$ df -h /var/druid/task +Filesystem Size Used Available Use% Mounted on +/dev/longhorn/pvc-2c0ef2aa-0438-4d75-9cb2-c12a176bae6a 973.4M 600.0M 357.4M 63% /var/druid/task +``` + +So, from the above output we can see that the storage usage is 63% for both nodes, which exceeded the `usageThreshold` 60%. + +There will be two `DruidOpsRequest` created for both historicals and middleManagers to expand the volume of the cluster for both nodes. +Let's watch the `druidopsrequest` in the demo namespace to see if any `druidopsrequest` object is created. After some time you'll see that a `druidopsrequest` of type `VolumeExpansion` will be created based on the `scalingThreshold`. + +```bash +$ watch kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +druidopsrequest.ops.kubedb.com/drops-druid-cluster-gq9huj VolumeExpansion Progressing 46s +druidopsrequest.ops.kubedb.com/drops-druid-cluster-kbw4fd VolumeExpansion Successful 4m46s +``` + +Once ops request has succeeded. Let's wait for the other one to become successful. + +```bash +$ kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +druidopsrequest.ops.kubedb.com/drops-druid-cluster-gq9huj VolumeExpansion Successful 3m18s +druidopsrequest.ops.kubedb.com/drops-druid-cluster-kbw4fd VolumeExpansion Successful 7m18s +``` + +We can see from the above output that the both `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` one by one we will get an overview of the steps that were followed to expand the volume of the cluster. + +```bash +$ kubectl describe druidopsrequest -n demo drops-druid-cluster-kbw4fd +Name: drops-druid-cluster-kbw4fd +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-cluster + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-25T09:57:14Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: + f:app.kubernetes.io/component: + f:app.kubernetes.io/instance: + f:app.kubernetes.io/managed-by: + f:app.kubernetes.io/name: + f:ownerReferences: + .: + k:{"uid":"57cbd906-a9b7-4649-bfe0-304840bb60c1"}: + f:spec: + .: + f:apply: + f:databaseRef: + f:type: + f:volumeExpansion: + .: + f:historicals: + f:mode: + Manager: kubedb-autoscaler + Operation: Update + Time: 2024-10-25T09:57:14Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-25T10:00:20Z + Owner References: + API Version: autoscaling.kubedb.com/v1alpha1 + Block Owner Deletion: true + Controller: true + Kind: DruidAutoscaler + Name: druid-storage-autoscaler + UID: 57cbd906-a9b7-4649-bfe0-304840bb60c1 + Resource Version: 228016 + UID: 1fa750bb-2db3-4684-a7cf-1b3047bc07af +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Type: VolumeExpansion + Volume Expansion: + Historicals: 2041405440 + Mode: Offline +Status: + Conditions: + Last Transition Time: 2024-10-25T09:57:14Z + Message: Druid ops-request has started to expand volume of druid nodes. + Observed Generation: 1 + Reason: VolumeExpansion + Status: True + Type: VolumeExpansion + Last Transition Time: 2024-10-25T09:57:22Z + Message: get pet set; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPetSet + Last Transition Time: 2024-10-25T09:57:22Z + Message: is pet set deleted; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsPetSetDeleted + Last Transition Time: 2024-10-25T09:57:32Z + Message: successfully deleted the petSets with orphan propagation policy + Observed Generation: 1 + Reason: OrphanPetSetPods + Status: True + Type: OrphanPetSetPods + Last Transition Time: 2024-10-25T09:57:37Z + Message: get pod; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPod + Last Transition Time: 2024-10-25T09:57:37Z + Message: is ops req patched; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsOpsReqPatched + Last Transition Time: 2024-10-25T09:57:37Z + Message: create pod; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CreatePod + Last Transition Time: 2024-10-25T09:57:42Z + Message: get pvc; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPvc + Last Transition Time: 2024-10-25T09:57:42Z + Message: is pvc patched; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsPvcPatched + Last Transition Time: 2024-10-25T09:59:27Z + Message: compare storage; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CompareStorage + Last Transition Time: 2024-10-25T09:59:27Z + Message: create; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: Create + Last Transition Time: 2024-10-25T09:59:35Z + Message: is druid running; ConditionStatus:False + Observed Generation: 1 + Status: False + Type: IsDruidRunning + Last Transition Time: 2024-10-25T09:59:57Z + Message: successfully updated historicals node PVC sizes + Observed Generation: 1 + Reason: UpdateHistoricalsNodePVCs + Status: True + Type: UpdateHistoricalsNodePVCs + Last Transition Time: 2024-10-25T10:00:15Z + Message: successfully reconciled the Druid resources + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-25T10:00:20Z + Message: PetSet is recreated + Observed Generation: 1 + Reason: ReadyPetSets + Status: True + Type: ReadyPetSets + Last Transition Time: 2024-10-25T10:00:20Z + Message: Successfully completed volumeExpansion for Druid + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 8m29s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-druid-cluster-kbw4fd + Normal Starting 8m29s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 8m29s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-kbw4fd + Warning get pet set; ConditionStatus:True 8m21s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Warning is pet set deleted; ConditionStatus:True 8m21s KubeDB Ops-manager Operator is pet set deleted; ConditionStatus:True + Warning get pet set; ConditionStatus:True 8m16s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Normal OrphanPetSetPods 8m11s KubeDB Ops-manager Operator successfully deleted the petSets with orphan propagation policy + Warning get pod; ConditionStatus:True 8m6s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 8m6s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning create pod; ConditionStatus:True 8m6s KubeDB Ops-manager Operator create pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m1s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 8m1s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning is pvc patched; ConditionStatus:True 8m1s KubeDB Ops-manager Operator is pvc patched; ConditionStatus:True + Warning compare storage; ConditionStatus:False 8m1s KubeDB Ops-manager Operator compare storage; ConditionStatus:False + Warning get pod; ConditionStatus:True 7m56s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m56s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m51s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m51s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m46s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m46s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m41s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m41s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m36s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m36s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m31s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m31s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m26s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m26s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m21s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m21s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m16s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m16s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m11s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m11s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m6s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m6s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 7m1s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 7m1s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m56s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m56s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m51s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m51s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m46s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m46s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m41s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m41s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m36s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m36s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m31s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m31s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m26s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m26s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m21s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m21s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m16s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 6m16s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning compare storage; ConditionStatus:True 6m16s KubeDB Ops-manager Operator compare storage; ConditionStatus:True + Warning create; ConditionStatus:True 6m16s KubeDB Ops-manager Operator create; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 6m16s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m11s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is druid running; ConditionStatus:False 6m8s KubeDB Ops-manager Operator is druid running; ConditionStatus:False + Warning get pod; ConditionStatus:True 6m6s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 6m1s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 5m56s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 5m51s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Normal UpdateHistoricalsNodePVCs 5m46s KubeDB Ops-manager Operator successfully updated historicals node PVC sizes + Normal UpdatePetSets 5m28s KubeDB Ops-manager Operator successfully reconciled the Druid resources + Warning get pet set; ConditionStatus:True 5m23s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Normal ReadyPetSets 5m23s KubeDB Ops-manager Operator PetSet is recreated + Normal Starting 5m23s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 5m23s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-kbw4fd + Normal UpdatePetSets 5m18s KubeDB Ops-manager Operator successfully reconciled the Druid resources + Normal UpdatePetSets 5m8s KubeDB Ops-manager Operator successfully reconciled the Druid resources + Normal UpdatePetSets 4m57s KubeDB Ops-manager Operator successfully reconciled the Druid resources +``` + +```bash +$ kubectl describe druidopsrequest -n demo drops-druid-cluster-gq9huj +Name: drops-druid-cluster-gq9huj +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-cluster + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-25T10:01:14Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: + f:app.kubernetes.io/component: + f:app.kubernetes.io/instance: + f:app.kubernetes.io/managed-by: + f:app.kubernetes.io/name: + f:ownerReferences: + .: + k:{"uid":"57cbd906-a9b7-4649-bfe0-304840bb60c1"}: + f:spec: + .: + f:apply: + f:databaseRef: + f:type: + f:volumeExpansion: + .: + f:middleManagers: + f:mode: + Manager: kubedb-autoscaler + Operation: Update + Time: 2024-10-25T10:01:14Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-25T10:04:12Z + Owner References: + API Version: autoscaling.kubedb.com/v1alpha1 + Block Owner Deletion: true + Controller: true + Kind: DruidAutoscaler + Name: druid-storage-autoscaler + UID: 57cbd906-a9b7-4649-bfe0-304840bb60c1 + Resource Version: 228783 + UID: 3b97380c-e867-467f-b366-4b50c7cd7d6d +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Type: VolumeExpansion + Volume Expansion: + Middle Managers: 2041405440 + Mode: Offline +Status: + Conditions: + Last Transition Time: 2024-10-25T10:01:14Z + Message: Druid ops-request has started to expand volume of druid nodes. + Observed Generation: 1 + Reason: VolumeExpansion + Status: True + Type: VolumeExpansion + Last Transition Time: 2024-10-25T10:01:22Z + Message: get pet set; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPetSet + Last Transition Time: 2024-10-25T10:01:22Z + Message: is pet set deleted; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsPetSetDeleted + Last Transition Time: 2024-10-25T10:01:32Z + Message: successfully deleted the petSets with orphan propagation policy + Observed Generation: 1 + Reason: OrphanPetSetPods + Status: True + Type: OrphanPetSetPods + Last Transition Time: 2024-10-25T10:01:37Z + Message: get pod; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPod + Last Transition Time: 2024-10-25T10:01:37Z + Message: is ops req patched; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsOpsReqPatched + Last Transition Time: 2024-10-25T10:01:37Z + Message: create pod; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CreatePod + Last Transition Time: 2024-10-25T10:01:42Z + Message: get pvc; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPvc + Last Transition Time: 2024-10-25T10:01:42Z + Message: is pvc patched; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsPvcPatched + Last Transition Time: 2024-10-25T10:03:32Z + Message: compare storage; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CompareStorage + Last Transition Time: 2024-10-25T10:03:32Z + Message: create; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: Create + Last Transition Time: 2024-10-25T10:03:40Z + Message: is druid running; ConditionStatus:False + Observed Generation: 1 + Status: False + Type: IsDruidRunning + Last Transition Time: 2024-10-25T10:03:52Z + Message: successfully updated middleManagers node PVC sizes + Observed Generation: 1 + Reason: UpdateMiddleManagersNodePVCs + Status: True + Type: UpdateMiddleManagersNodePVCs + Last Transition Time: 2024-10-25T10:04:07Z + Message: successfully reconciled the Druid resources + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-25T10:04:12Z + Message: PetSet is recreated + Observed Generation: 1 + Reason: ReadyPetSets + Status: True + Type: ReadyPetSets + Last Transition Time: 2024-10-25T10:04:12Z + Message: Successfully completed volumeExpansion for Druid + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 5m33s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-druid-cluster-gq9huj + Normal Starting 5m33s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 5m33s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-gq9huj + Warning get pet set; ConditionStatus:True 5m25s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Warning is pet set deleted; ConditionStatus:True 5m25s KubeDB Ops-manager Operator is pet set deleted; ConditionStatus:True + Warning get pet set; ConditionStatus:True 5m20s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Normal OrphanPetSetPods 5m15s KubeDB Ops-manager Operator successfully deleted the petSets with orphan propagation policy + Warning get pod; ConditionStatus:True 5m10s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 5m10s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning create pod; ConditionStatus:True 5m10s KubeDB Ops-manager Operator create pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 5m5s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 5m5s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning is pvc patched; ConditionStatus:True 5m5s KubeDB Ops-manager Operator is pvc patched; ConditionStatus:True + Warning compare storage; ConditionStatus:False 5m5s KubeDB Ops-manager Operator compare storage; ConditionStatus:False + Warning get pod; ConditionStatus:True 5m KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 5m KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m55s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m55s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m50s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m50s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m45s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m45s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m40s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m40s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m35s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m35s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m30s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m30s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m25s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m25s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m20s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m20s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m15s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m15s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m10s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m10s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m5s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m5s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 4m KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 4m KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m55s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m55s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m50s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m50s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m45s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m45s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m40s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m40s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m35s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m35s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m30s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m30s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m25s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m25s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m20s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m20s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m15s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 3m15s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning compare storage; ConditionStatus:True 3m15s KubeDB Ops-manager Operator compare storage; ConditionStatus:True + Warning create; ConditionStatus:True 3m15s KubeDB Ops-manager Operator create; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 3m15s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m10s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is druid running; ConditionStatus:False 3m7s KubeDB Ops-manager Operator is druid running; ConditionStatus:False + Warning get pod; ConditionStatus:True 3m5s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 3m KubeDB Ops-manager Operator get pod; ConditionStatus:True + Normal UpdateMiddleManagersNodePVCs 2m55s KubeDB Ops-manager Operator successfully updated middleManagers node PVC sizes + Normal UpdatePetSets 2m40s KubeDB Ops-manager Operator successfully reconciled the Druid resources + Warning get pet set; ConditionStatus:True 2m35s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Normal ReadyPetSets 2m35s KubeDB Ops-manager Operator PetSet is recreated + Normal Starting 2m35s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 2m35s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-druid-cluster-gq9huj +``` + +Now, we are going to verify from the `Petset`, and the `Persistent Volume` whether the volume of the topology cluster has expanded to meet the desired state, Let's check, + +```bash +$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"2041405440" +$ kubectl get petset -n demo druid-cluster-middleManagers -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"2041405440" +$ kubectl get pv -n demo +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE +pvc-2c0ef2aa-0438-4d75-9cb2-c12a176bae6a 1948Mi RWO Delete Bound demo/druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 longhorn 19m +pvc-5f4cea5f-e0c8-4339-b67c-9cb8b02ba49d 1948Mi RWO Delete Bound demo/druid-cluster-segment-cache-druid-cluster-historicals-0 longhorn 19m +``` + +The above output verifies that we have successfully autoscaled the volume of the Druid topology cluster for both historicals and middleManagers. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete druidopsrequests -n demo drops-druid-cluster-gq9huj drops-druid-cluster-kbw4fd +kubectl delete druidutoscaler -n demo druid-storage-autoscaler +kubectl delete dr -n demo druid-cluster +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/autoscaler/storage/images/storage-autoscaling.png b/docs/guides/druid/autoscaler/storage/images/storage-autoscaling.png new file mode 100644 index 0000000000..a48b564e1c Binary files /dev/null and b/docs/guides/druid/autoscaler/storage/images/storage-autoscaling.png differ diff --git a/docs/guides/druid/autoscaler/storage/overview.md b/docs/guides/druid/autoscaler/storage/overview.md new file mode 100644 index 0000000000..607e246032 --- /dev/null +++ b/docs/guides/druid/autoscaler/storage/overview.md @@ -0,0 +1,55 @@ +--- +title: Druid Storage Autoscaling Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-autoscaler-storage-overview + name: Overview + parent: guides-druid-autoscaler-storage + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Vertical Autoscaling + +This guide will give an overview on how KubeDB Autoscaler operator autoscales the database storage using `druidautoscaler` crd. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidAutoscaler](/docs/guides/druid/concepts/druidautoscaler.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + +## How Storage Autoscaling Works + +The following diagram shows how KubeDB Autoscaler operator autoscales the resources of `Druid` cluster components. Open the image in a new tab to see the enlarged version. + +
+  Storage Auto Scaling process of Druid +
Fig: Storage Auto Scaling process of Druid
+
+ +The Auto Scaling process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource (CR). + +2. `KubeDB` Provisioner operator watches the `Druid` CR. + +3. When the operator finds a `Druid` CR, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Each PetSet creates a Persistent Volume according to the Volume Claim Template provided in the petset configuration. + +5. Then, in order to set up storage autoscaling of the druid data nodes (i.e. Historicals, MiddleManagers) of the `Druid` cluster, the user creates a `DruidAutoscaler` CRO with desired configuration. + +6. `KubeDB` Autoscaler operator watches the `DruidAutoscaler` CRO. + +7. `KubeDB` Autoscaler operator continuously watches persistent volumes of the clusters to check if it exceeds the specified usage threshold. If the usage exceeds the specified usage threshold, then `KubeDB` Autoscaler operator creates a `DruidOpsRequest` to expand the storage of the database. + +8. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CRO. + +9. Then the `KubeDB` Ops-manager operator will expand the storage of the cluster component as specified on the `DruidOpsRequest` CRO. + +In the next docs, we are going to show a step-by-step guide on Autoscaling storage of various Druid cluster components using `DruidAutoscaler` CRD. diff --git a/docs/guides/druid/autoscaler/storage/yamls/deep-storage-config.yaml b/docs/guides/druid/autoscaler/storage/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/autoscaler/storage/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/autoscaler/storage/yamls/druid-cluster.yaml b/docs/guides/druid/autoscaler/storage/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..5415590a2b --- /dev/null +++ b/docs/guides/druid/autoscaler/storage/yamls/druid-cluster.yaml @@ -0,0 +1,40 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + historicals: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + middleManagers: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + routers: + replicas: 1 + deletionPolicy: Delete + monitor: + agent: prometheus.io/operator + prometheus: + serviceMonitor: + labels: + release: prometheus + interval: 10s diff --git a/docs/guides/druid/autoscaler/storage/yamls/druid-storage-autoscaler.yaml b/docs/guides/druid/autoscaler/storage/yamls/druid-storage-autoscaler.yaml new file mode 100644 index 0000000000..d1a2f5c438 --- /dev/null +++ b/docs/guides/druid/autoscaler/storage/yamls/druid-storage-autoscaler.yaml @@ -0,0 +1,19 @@ +apiVersion: autoscaling.kubedb.com/v1alpha1 +kind: DruidAutoscaler +metadata: + name: druid-storage-autoscaler + namespace: demo +spec: + databaseRef: + name: druid-cluster + storage: + historicals: + expansionMode: "Offline" + trigger: "On" + usageThreshold: 60 + scalingThreshold: 100 + middleManagers: + expansionMode: "Offline" + trigger: "On" + usageThreshold: 60 + scalingThreshold: 100 diff --git a/docs/guides/druid/backup/_index.md b/docs/guides/druid/backup/_index.md index bb47dcc106..31146d6c14 100644 --- a/docs/guides/druid/backup/_index.md +++ b/docs/guides/druid/backup/_index.md @@ -5,6 +5,6 @@ menu: identifier: guides-druid-backup name: Backup & Restore parent: guides-druid - weight: 40 + weight: 50 menu_name: docs_{{ .version }} --- \ No newline at end of file diff --git a/docs/guides/druid/backup/application-level/index.md b/docs/guides/druid/backup/application-level/index.md index 4c6394ca81..627c865406 100644 --- a/docs/guides/druid/backup/application-level/index.md +++ b/docs/guides/druid/backup/application-level/index.md @@ -20,7 +20,7 @@ This guide will give you how you can take application-level backup and restore y ## Before You Begin - At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using `Minikube` or `Kind`. -- Install `KubeDB` in your cluster following the steps [here](/docs/setup/README.md). +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency with helm command. - Install `KubeStash` in your cluster following the steps [here](https://kubestash.com/docs/latest/setup/install/kubestash). - Install KubeStash `kubectl` plugin following the steps [here](https://kubestash.com/docs/latest/setup/install/kubectl-plugin/). - If you are not familiar with how KubeStash backup and restore Druid databases, please check the following guide [here](/docs/guides/druid/backup/overview/index.md). @@ -52,7 +52,6 @@ This section will demonstrate how to take application-level backup of a `Druid` ## Deploy Sample Druid Database - **Create External Dependency (Deep Storage):** One of the external dependency of Druid is deep storage where the segments are stored. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. diff --git a/docs/guides/druid/backup/auto-backup/index.md b/docs/guides/druid/backup/auto-backup/index.md index 3c16e8bf33..d2ba2b5bbc 100644 --- a/docs/guides/druid/backup/auto-backup/index.md +++ b/docs/guides/druid/backup/auto-backup/index.md @@ -20,7 +20,7 @@ In this tutorial, we are going to show how you can configure a backup blueprint ## Before You Begin - At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using `Minikube` or `Kind`. -- Install `KubeDB` in your cluster following the steps [here](/docs/setup/README.md). +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency with helm command. - Install `KubeStash` in your cluster following the steps [here](https://kubestash.com/docs/latest/setup/install/kubestash). - Install KubeStash `kubectl` plugin following the steps [here](https://kubestash.com/docs/latest/setup/install/kubectl-plugin/). - If you are not familiar with how KubeStash backup and restore Druid databases, please check the following guide [here](/docs/guides/druid/backup/overview/index.md). diff --git a/docs/guides/druid/backup/cross-ns-dependencies/index.md b/docs/guides/druid/backup/cross-ns-dependencies/index.md index a3f200ec6f..9cbb5a066e 100644 --- a/docs/guides/druid/backup/cross-ns-dependencies/index.md +++ b/docs/guides/druid/backup/cross-ns-dependencies/index.md @@ -22,7 +22,7 @@ This guide will give you how you can take [Application Level Backup](https://git ## Before You Begin - At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using `Minikube` or `Kind`. -- Install `KubeDB` in your cluster following the steps [here](/docs/setup/README.md). +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency with helm command. - Install `KubeStash` in your cluster following the steps [here](https://kubestash.com/docs/latest/setup/install/kubestash). - Install KubeStash `kubectl` plugin following the steps [here](https://kubestash.com/docs/latest/setup/install/kubectl-plugin/). - If you are not familiar with how KubeStash backup and restore Druid databases, please check the following guide [here](/docs/guides/druid/backup/overview/index.md). diff --git a/docs/guides/druid/backup/logical/index.md b/docs/guides/druid/backup/logical/index.md index dbce094ab8..e6e1d06447 100644 --- a/docs/guides/druid/backup/logical/index.md +++ b/docs/guides/druid/backup/logical/index.md @@ -20,7 +20,7 @@ This guide will give you how you can take backup and restore your `Druid` databa ## Before You Begin - At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using `Minikube` or `Kind`. -- Install `KubeDB` in your cluster following the steps [here](/docs/setup/README.md). +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency with helm command. - Install `KubeStash` in your cluster following the steps [here](https://kubestash.com/docs/latest/setup/install/kubestash). - Install KubeStash `kubectl` plugin following the steps [here](https://kubestash.com/docs/latest/setup/install/kubectl-plugin/). - If you are not familiar with how KubeStash backup and restore Druid databases, please check the following guide [here](/docs/guides/druid/backup/overview/index.md). diff --git a/docs/guides/druid/backup/overview/index.md b/docs/guides/druid/backup/overview/index.md index c8adef63e6..723754834f 100644 --- a/docs/guides/druid/backup/overview/index.md +++ b/docs/guides/druid/backup/overview/index.md @@ -2,7 +2,7 @@ title: Backup & Restore Druid Overview menu: docs_{{ .version }}: - identifier: guides-druid-backup-overview + identifier: guides-druid-backup-guide name: Overview parent: guides-druid-backup weight: 10 diff --git a/docs/guides/druid/clustering/_index.md b/docs/guides/druid/clustering/_index.md new file mode 100644 index 0000000000..20b929a8a2 --- /dev/null +++ b/docs/guides/druid/clustering/_index.md @@ -0,0 +1,10 @@ +--- +title: Druid Clustering +menu: + docs_{{ .version }}: + identifier: guides-druid-clustering + name: Clustering + parent: guides-druid + weight: 30 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/clustering/guide/index.md b/docs/guides/druid/clustering/guide/index.md new file mode 100644 index 0000000000..b062348a72 --- /dev/null +++ b/docs/guides/druid/clustering/guide/index.md @@ -0,0 +1,925 @@ +--- +title: Druid Topology Cluster Guide +menu: + docs_{{ .version }}: + identifier: guides-druid-clustering-guide + name: Deploy Druid Cluster + parent: guides-druid-clustering + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# KubeDB - Druid Cluster + +This tutorial will show you how to use KubeDB to provision a Druid Cluster. + +## Before You Begin + +Before proceeding: + +- Read [druid topology cluster overview](/docs/guides/druid/clustering/overview/index.md) to get a basic idea about the design and architecture of Druid. + +- You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency with helm command. + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. Run the following command to prepare your cluster for this tutorial: + + ```bash + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: The yaml files used in this tutorial are stored in [docs/guides/druid/clustering/topology-cluster-guide/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/druid/clustering/topology-cluster-guide/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/backup/application-level/examples/deep-storage-config.yaml +secret/deep-storage-config created +``` + +## Deploy Druid Cluster + +The following is an example `Druid` object which creates a Druid cluster of six nodes (coordinators, overlords, brokers, routers, historicals and middleManager). Each with one replica. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/clustering/guide/yamls/druid-with-monitoring.yaml +druid.kubedb.com/druid-cluster created +``` + +KubeDB operator watches for `Druid` objects using Kubernetes API. When a `Druid` object is created, KubeDB operator will create new PetSets and Services with the matching Druid object name. KubeDB operator will also create a governing service for the PetSet with the name `-pods`. + +```bash +$ kubectl describe druid -n demo druid-cluster +Name: druid-cluster +Namespace: demo +Labels: +Annotations: +API Version: kubedb.com/v1alpha2 +Kind: Druid +Metadata: + Creation Timestamp: 2024-10-21T06:01:32Z + Finalizers: + kubedb.com/druid + Generation: 1 + Managed Fields: + API Version: kubedb.com/v1alpha2 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:finalizers: + .: + v:"kubedb.com/druid": + Manager: druid-operator + Operation: Update + Time: 2024-10-21T06:01:32Z + API Version: kubedb.com/v1alpha2 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:deepStorage: + .: + f:configSecret: + f:type: + f:deletionPolicy: + f:healthChecker: + .: + f:failureThreshold: + f:periodSeconds: + f:timeoutSeconds: + f:topology: + .: + f:routers: + .: + f:replicas: + f:version: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-21T06:01:32Z + API Version: kubedb.com/v1alpha2 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:phase: + Manager: druid-operator + Operation: Update + Subresource: status + Time: 2024-10-21T06:04:29Z + Resource Version: 52093 + UID: a2e12db2-6694-419f-ad07-2c906df5b611 +Spec: + Auth Secret: + Name: druid-cluster-admin-cred + Deep Storage: + Config Secret: + Name: deep-storage-config + Type: s3 + Deletion Policy: Delete + Health Checker: + Failure Threshold: 3 + Period Seconds: 30 + Timeout Seconds: 10 + Metadata Storage: + Create Tables: true + Linked DB: druid + Name: druid-cluster-mysql-metadata + Namespace: demo + Type: MySQL + Version: 8.0.35 + Topology: + Brokers: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Coordinators: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Historicals: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Storage: + Access Modes: + ReadWriteOnce + Resources: + Requests: + Storage: 1Gi + Storage Type: Durable + Middle Managers: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 2560Mi + Requests: + Cpu: 500m + Memory: 2560Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Storage: + Access Modes: + ReadWriteOnce + Resources: + Requests: + Storage: 1Gi + Storage Type: Durable + Routers: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Version: 28.0.1 + Zookeeper Ref: + Name: druid-cluster-zk + Namespace: demo + Version: 3.7.2 +Status: + Conditions: + Last Transition Time: 2024-10-21T06:01:32Z + Message: The KubeDB operator has started the provisioning of Druid: demo/druid-cluster + Observed Generation: 1 + Reason: DatabaseProvisioningStartedSuccessfully + Status: True + Type: ProvisioningStarted + Phase: Provisioning +Events: + +$ kubectl get petset -n demo +NAME AGE +druid-cluster-brokers 13m +druid-cluster-coordinators 13m +druid-cluster-historicals 13m +druid-cluster-middlemanagers 13m +druid-cluster-mysql-metadata 14m +druid-cluster-routers 13m +druid-cluster-zk 14m + +$ kubectl get pvc -n demo -l app.kubernetes.io/name=druids.kubedb.com +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 Bound pvc-d288b621-d281-4004-995d-7a25bb4149de 1Gi RWO standard 14m +druid-cluster-segment-cache-druid-cluster-historicals-0 Bound pvc-ccca6be2-658a-46af-a270-de1c6a041af7 1Gi RWO standard 14m + + +$ kubectl get pv -n demo +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE +pvc-4f8538f6-a6ce-4233-b533-8566852f5b98 1Gi RWO Delete Bound demo/druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 standard 4m39s +pvc-8823d3ad-d614-4172-89ac-c2284a17f502 1Gi RWO Delete Bound demo/druid-cluster-segment-cache-druid-cluster-historicals-0 standard 4m35s + +$ kubectl get service -n demo +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +druid-cluster-brokers ClusterIP 10.96.186.168 8082/TCP 17m +druid-cluster-coordinators ClusterIP 10.96.122.235 8081/TCP 17m +druid-cluster-mysql-metadata ClusterIP 10.96.109.2 3306/TCP 18m +druid-cluster-mysql-metadata-pods ClusterIP None 3306/TCP 18m +druid-cluster-mysql-metadata-standby ClusterIP 10.96.97.152 3306/TCP 18m +druid-cluster-pods ClusterIP None 8081/TCP,8090/TCP,8083/TCP,8091/TCP,8082/TCP,8888/TCP 17m +druid-cluster-routers ClusterIP 10.96.138.237 8888/TCP 17m +druid-cluster-zk ClusterIP 10.96.148.251 2181/TCP 18m +druid-cluster-zk-admin-server ClusterIP 10.96.2.106 8080/TCP 18m +druid-cluster-zk-pods ClusterIP None 2181/TCP,2888/TCP,3888/TCP 18m +``` + +KubeDB operator sets the `status.phase` to `Ready` once the database is successfully created. Run the following command to see the modified `Druid` object: + +```bash +$ kubectl describe druid -n demo druid-cluster +Name: druid-cluster +Namespace: demo +Labels: +Annotations: +API Version: kubedb.com/v1alpha2 +Kind: Druid +Metadata: + Creation Timestamp: 2024-10-21T06:01:32Z + Finalizers: + kubedb.com/druid + Generation: 1 + Managed Fields: + API Version: kubedb.com/v1alpha2 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:finalizers: + .: + v:"kubedb.com/druid": + Manager: druid-operator + Operation: Update + Time: 2024-10-21T06:01:32Z + API Version: kubedb.com/v1alpha2 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:deepStorage: + .: + f:configSecret: + f:type: + f:deletionPolicy: + f:healthChecker: + .: + f:failureThreshold: + f:periodSeconds: + f:timeoutSeconds: + f:topology: + .: + f:routers: + .: + f:replicas: + f:version: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-21T06:01:32Z + API Version: kubedb.com/v1alpha2 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:phase: + Manager: druid-operator + Operation: Update + Subresource: status + Time: 2024-10-21T06:04:29Z + Resource Version: 52093 + UID: a2e12db2-6694-419f-ad07-2c906df5b611 +Spec: + Auth Secret: + Name: druid-cluster-admin-cred + Deep Storage: + Config Secret: + Name: deep-storage-config + Type: s3 + Deletion Policy: Delete + Health Checker: + Failure Threshold: 3 + Period Seconds: 30 + Timeout Seconds: 10 + Metadata Storage: + Create Tables: true + Linked DB: druid + Name: druid-cluster-mysql-metadata + Namespace: demo + Type: MySQL + Version: 8.0.35 + Topology: + Brokers: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Coordinators: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Historicals: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Storage: + Access Modes: + ReadWriteOnce + Resources: + Requests: + Storage: 1Gi + Storage Type: Durable + Middle Managers: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 2560Mi + Requests: + Cpu: 500m + Memory: 2560Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Storage: + Access Modes: + ReadWriteOnce + Resources: + Requests: + Storage: 1Gi + Storage Type: Durable + Routers: + Pod Template: + Controller: + Metadata: + Spec: + Containers: + Name: druid + Resources: + Limits: + Memory: 1Gi + Requests: + Cpu: 500m + Memory: 1Gi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Init Containers: + Name: init-druid + Resources: + Limits: + Memory: 512Mi + Requests: + Cpu: 200m + Memory: 512Mi + Security Context: + Allow Privilege Escalation: false + Capabilities: + Drop: + ALL + Run As Non Root: true + Run As User: 1000 + Seccomp Profile: + Type: RuntimeDefault + Pod Placement Policy: + Name: default + Security Context: + Fs Group: 1000 + Replicas: 1 + Version: 28.0.1 + Zookeeper Ref: + Name: druid-cluster-zk + Namespace: demo + Version: 3.7.2 +Status: + Conditions: + Last Transition Time: 2024-10-21T06:01:32Z + Message: The KubeDB operator has started the provisioning of Druid: demo/druid-cluster + Observed Generation: 1 + Reason: DatabaseProvisioningStartedSuccessfully + Status: True + Type: ProvisioningStarted + Last Transition Time: 2024-10-21T06:03:03Z + Message: Database dependency is ready + Observed Generation: 1 + Reason: DatabaseDependencyReady + Status: True + Type: DatabaseDependencyReady + Last Transition Time: 2024-10-21T06:03:34Z + Message: All desired replicas are ready. + Observed Generation: 1 + Reason: AllReplicasReady + Status: True + Type: ReplicaReady + Last Transition Time: 2024-10-21T06:04:04Z + Message: The Druid: demo/druid-cluster is accepting client requests and nodes formed a cluster + Observed Generation: 1 + Reason: DatabaseAcceptingConnectionRequest + Status: True + Type: AcceptingConnection + Last Transition Time: 2024-10-21T06:04:29Z + Message: The Druid: demo/druid-cluster is ready. + Observed Generation: 1 + Reason: ReadinessCheckSucceeded + Status: True + Type: Ready + Last Transition Time: 2024-10-21T06:04:29Z + Message: The Druid: demo/druid-cluster is successfully provisioned. + Observed Generation: 1 + Reason: DatabaseSuccessfullyProvisioned + Status: True + Type: Provisioned + Phase: Ready +Events: +``` + + +## Connect with Druid Database +We will use [port forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to connect with our routers of the Druid database. Then we will use `curl` to send `HTTP` requests to check cluster health to verify that our Druid database is working well. It is also possible to use `External-IP` to access druid nodes if you make `service` type of that node as `LoadBalancer`. + +### Check the Service Health + +Let's port-forward the port `8888` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-routers 8888 +Forwarding from 127.0.0.1:8888 -> 8888 +Forwarding from [::1]:8888 -> 8888 +``` + +Now, the Druid cluster is accessible at `localhost:8888`. Let's check the [Service Health](https://druid.apache.org/docs/latest/api-reference/service-status-api/#get-service-health) of Routers of the Druid database. + +```bash +$ curl "http://localhost:8888/status/health" +true +``` +From the retrieved health information above, we can see that our Druid cluster’s status is `true`, indicating that the service can receive API calls and is healthy. In the same way it possible to check the health of other druid nodes by port-forwarding the appropriate services. + +### Access the web console + +We can also access the [web console](https://druid.apache.org/docs/latest/operations/web-console) of Druid database from any browser by port-forwarding the routers in the same way shown in the aforementioned step or directly using the `External-IP` if the router service type is `LoadBalancer`. + +Now hit the `http://localhost:8888` from any browser, and you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  lifecycle +

+ +You can use this web console for loading data, managing datasources and tasks, and viewing server status and segment information. You can also run SQL and native Druid queries in the console. + +## Cleaning up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +$ kubectl patch -n demo druid druid-cluster -p '{"spec":{"deletionPolicy":"WipeOut"}}' --type="merge" +kafka.kubedb.com/druid-cluster patched + +$ kubectl delete dr druid-cluster -n demo +druid.kubedb.com "druid-cluster" deleted + +$ kubectl delete namespace demo +namespace "demo" deleted +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Detail concepts of [DruidDBVersion object](/docs/guides/druid/concepts/druidversion.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/clustering/guide/yamls/deep-storage-config.yaml b/docs/guides/druid/clustering/guide/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/clustering/guide/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/clustering/guide/yamls/druid-cluster.yaml b/docs/guides/druid/clustering/guide/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..7a89d0dc91 --- /dev/null +++ b/docs/guides/druid/clustering/guide/yamls/druid-cluster.yaml @@ -0,0 +1,15 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete diff --git a/docs/guides/druid/clustering/overview/images/druid-architecture.svg b/docs/guides/druid/clustering/overview/images/druid-architecture.svg new file mode 100644 index 0000000000..3f86a412cf --- /dev/null +++ b/docs/guides/druid/clustering/overview/images/druid-architecture.svg @@ -0,0 +1,19 @@ + + diff --git a/docs/guides/druid/clustering/overview/index.md b/docs/guides/druid/clustering/overview/index.md new file mode 100644 index 0000000000..24893bcce0 --- /dev/null +++ b/docs/guides/druid/clustering/overview/index.md @@ -0,0 +1,115 @@ +--- +title: Druid Topology Cluster Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-clustering-overview + name: Druid Clustering Overview + parent: guides-druid-clustering + weight: 15 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Architecture + +Druid has a distributed architecture that is designed to be cloud-friendly and easy to operate. You can configure and scale services independently for maximum flexibility over cluster operations. This design includes enhanced fault tolerance: an outage of one component does not immediately affect other components. + +The following diagram shows the services that make up the Druid architecture, their typical arrangement across servers, and how queries and data flow through this architecture. + +![Druid Architecture](/docs/guides/druid/clustering/overview/images/druid-architecture.svg) + +Image ref: [Druid Official Documentation](https://druid.apache.org/assets/images/druid-architecture-7db1cd79d2d70b2e5ccc73b6bebfcaa4.svg) + + +## Druid services + +Druid has several types of services: + +- **Coordinator** manages data availability on the cluster. +- **Overlord** controls the assignment of data ingestion workloads. +- **Broker** handles queries from external clients. +- **Router** routes requests to Brokers, Coordinators, and Overlords. +- **Historical** stores queryable data. +- **MiddleManager** and Peon ingest data. +- **Indexer** serves an alternative to the MiddleManager + Peon task execution system. + +## Druid servers + +You can deploy Druid services according to your preferences. For ease of deployment, we recommend organizing them into three server types: **Master**, **Query**, and **Data**. + +### Master server +A Master server manages data ingestion and availability. It is responsible for starting new ingestion jobs and coordinating availability of data on the Data server. + +Master servers divide operations between **Coordinator** and **Overlord** services. + +#### Coordinator service +[Coordinator](https://druid.apache.org/docs/latest/design/coordinator/) services watch over the Historical services on the Data servers. They are responsible for assigning segments to specific servers, and for ensuring segments are well-balanced across Historicals. + +### Overlord service +[Overlord](https://druid.apache.org/docs/latest/design/overlord/) services watch over the MiddleManager services on the Data servers and are the controllers of data ingestion into Druid. They are responsible for assigning ingestion tasks to MiddleManagers and for coordinating segment publishing. + +### Query server +A Query server provides the endpoints that users and client applications interact with, routing queries to Data servers or other Query servers (and optionally proxied Master server requests). + +Query servers divide operations between Broker and Router services. + +#### Broker service +[Broker](https://druid.apache.org/docs/latest/design/broker/) services receive queries from external clients and forward those queries to Data servers. When Brokers receive results from those subqueries, they merge those results and return them to the caller. Typically, you query Brokers rather than querying Historical or MiddleManager services on Data servers directly. + +#### Router service +[Router](https://druid.apache.org/docs/latest/design/router/) services provide a unified API gateway in front of Brokers, Overlords, and Coordinators. + +The Router service also runs the web console, a UI for loading data, managing datasources and tasks, and viewing server status and segment information. + +### Data server +A Data server executes ingestion jobs and stores queryable data. + +Data servers divide operations between Historical and MiddleManager services. + +#### Historical service +[Historical](https://druid.apache.org/docs/latest/design/historical/) services handle storage and querying on historical data, including any streaming data that has been in the system long enough to be committed. Historical services download segments from deep storage and respond to queries about these segments. They don't accept writes. + +#### MiddleManager service +[MiddleManager](https://druid.apache.org/docs/latest/design/middlemanager) services handle ingestion of new data into the cluster. They are responsible for reading from external data sources and publishing new Druid segments. + +## External dependencies +In addition to its built-in service types, Druid also has three external dependencies. These are intended to be able to leverage existing infrastructure, where present. + +### Deep storage +Druid uses deep storage to store any data that has been ingested into the system. Deep storage is shared file storage accessible by every Druid server. In a clustered deployment, this is typically a distributed object store like S3 or HDFS, or a network mounted filesystem. In a single-server deployment, this is typically local disk. + +Druid uses deep storage for the following purposes: + +- To store all the data you ingest. Segments that get loaded onto Historical services for low latency queries are also kept in deep storage for backup purposes. Additionally, segments that are only in deep storage can be used for queries from deep storage. +- As a way to transfer data in the background between Druid services. Druid stores data in files called segments. + +Historical services cache data segments on local disk and serve queries from that cache as well as from an in-memory cache. Segments on disk for Historical services provide the low latency querying performance Druid is known for. + +You can also query directly from deep storage. When you query segments that exist only in deep storage, you trade some performance for the ability to query more of your data without necessarily having to scale your Historical services. + +When determining sizing for your storage, keep the following in mind: + +- Deep storage needs to be able to hold all the data that you ingest into Druid. +- On disk storage for Historical services need to be able to accommodate the data you want to load onto them to run queries. The data on Historical services should be data you access frequently and need to run low latency queries for. + +- Deep storage is an important part of Druid's elastic, fault-tolerant design. Druid bootstraps from deep storage even if every single data server is lost and re-provisioned. + +For more details, please see the [Deep storage](https://druid.apache.org/docs/latest/design/deep-storage/) page. + +### Metadata storage +The metadata storage holds various shared system metadata such as segment usage information and task information. In a clustered deployment, this is typically a traditional RDBMS like PostgreSQL or MySQL. In a single-server deployment, it is typically a locally-stored Apache Derby database. + +For more details, please see the [Metadata storage](https://druid.apache.org/docs/latest/design/metadata-storage/) page. + +### ZooKeeper +Used for internal service discovery, coordination, and leader election. + +For more details, please see the [ZooKeeper](https://druid.apache.org/docs/latest/design/zookeeper/) page. + + +## Next Steps + +- [Deploy Druid Cluster](/docs/guides/druid/clustering/overview/index.md) using KubeDB. +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md) diff --git a/docs/guides/druid/concepts/_index.md b/docs/guides/druid/concepts/_index.md index 67c3be7748..a5c61e7e6b 100755 --- a/docs/guides/druid/concepts/_index.md +++ b/docs/guides/druid/concepts/_index.md @@ -7,4 +7,4 @@ menu: parent: guides-druid weight: 20 menu_name: docs_{{ .version }} ---- +--- \ No newline at end of file diff --git a/docs/guides/druid/concepts/appbinding.md b/docs/guides/druid/concepts/appbinding.md index 60f9b5bb4d..02c4750799 100644 --- a/docs/guides/druid/concepts/appbinding.md +++ b/docs/guides/druid/concepts/appbinding.md @@ -22,218 +22,127 @@ If you deploy a database using [KubeDB](https://kubedb.com/docs/latest/welcome/) KubeDB uses [Stash](https://appscode.com/products/stash/) to perform backup/recovery of databases. Stash needs to know how to connect with a target database and the credentials necessary to access it. This is done via an `AppBinding`. -[//]: # (## AppBinding CRD Specification) +## AppBinding CRD Specification + +Like any official Kubernetes resource, an `AppBinding` has `TypeMeta`, `ObjectMeta` and `Spec` sections. However, unlike other Kubernetes resources, it does not have a `Status` section. + +An `AppBinding` object created by `KubeDB` for Druid database is shown below, -[//]: # () -[//]: # (Like any official Kubernetes resource, an `AppBinding` has `TypeMeta`, `ObjectMeta` and `Spec` sections. However, unlike other Kubernetes resources, it does not have a `Status` section.) +```yaml +apiVersion: appcatalog.appscode.com/v1alpha1 +kind: AppBinding +metadata: + annotations: + kubectl.kubernetes.io/last-applied-configuration: | + {"apiVersion":"kubedb.com/v1alpha2","kind":"Druid","metadata":{"annotations":{},"name":"druid-quickstart","namespace":"demo"},"spec":{"deepStorage":{"configSecret":{"name":"deep-storage-config"},"type":"s3"},"topology":{"routers":{"replicas":1}},"version":"28.0.1"}} + creationTimestamp: "2024-10-16T13:28:40Z" + generation: 1 + labels: + app.kubernetes.io/component: database + app.kubernetes.io/instance: druid-quickstart + app.kubernetes.io/managed-by: kubedb.com + app.kubernetes.io/name: druids.kubedb.com + name: druid-quickstart + namespace: demo + ownerReferences: + - apiVersion: kubedb.com/v1alpha2 + blockOwnerDeletion: true + controller: true + kind: Druid + name: druid-quickstart + uid: 06dc7c5f-65ad-4310-a203-b18c0d33d662 + resourceVersion: "45154" + uid: 58861709-99f9-4c78-8cf9-b5dc6534102e +spec: + appRef: + apiGroup: kubedb.com + kind: Druid + name: druid-quickstart + namespace: demo + clientConfig: + caBundle: dGhpcyBpcyBub3QgYSBjZXJ0 + service: + name: druid-quickstart-pods + port: 8888 + scheme: http + url: http://druid-quickstart-coordinators-0.druid-quickstart-pods.demo.svc.cluster.local:8081,http://druid-quickstart-overlords-0.druid-quickstart-pods.demo.svc.cluster.local:8090,http://druid-quickstart-middlemanagers-0.druid-quickstart-pods.demo.svc.cluster.local:8091,http://druid-quickstart-historicals-0.druid-quickstart-pods.demo.svc.cluster.local:8083,http://druid-quickstart-brokers-0.druid-quickstart-pods.demo.svc.cluster.local:8082,http://druid-quickstart-routers-0.druid-quickstart-pods.demo.svc.cluster.local:8888 + secret: + name: druid-quickstart-admin-cred + tlsSecret: + name: druid-client-cert + type: kubedb.com/druid + version: 28.0.1 +``` +Here, we are going to describe the sections of an `AppBinding` crd. -[//]: # () -[//]: # (An `AppBinding` object created by `KubeDB` for PostgreSQL database is shown below,) +### AppBinding `Spec` -[//]: # () -[//]: # (```yaml) +An `AppBinding` object has the following fields in the `spec` section: -[//]: # (apiVersion: appcatalog.appscode.com/v1alpha1) +#### spec.type -[//]: # (kind: AppBinding) +`spec.type` is an optional field that indicates the type of the app that this `AppBinding` is pointing to. -[//]: # (metadata:) + + -[//]: # ( app.kubernetes.io/instance: quick-postgres) +#### spec.secret -[//]: # ( app.kubernetes.io/managed-by: kubedb.com) +`spec.secret` specifies the name of the secret which contains the credentials that are required to access the database. This secret must be in the same namespace as the `AppBinding`. -[//]: # ( app.kubernetes.io/name: postgres) +This secret must contain the following keys for Druid: -[//]: # ( app.kubernetes.io/version: "10.2"-v2) +| Key | Usage | +| ---------- |------------------------------------------------| +| `username` | Username of the target Druid instance. | +| `password` | Password for the user specified by `username`. | -[//]: # ( app.kubernetes.io/name: postgreses.kubedb.com) -[//]: # ( app.kubernetes.io/instance: quick-postgres) +#### spec.appRef +appRef refers to the underlying application. It has 4 fields named `apiGroup`, `kind`, `name` & `namespace`. -[//]: # (spec:) +#### spec.clientConfig -[//]: # ( type: kubedb.com/postgres) +`spec.clientConfig` defines how to communicate with the target database. You can use either a URL or a Kubernetes service to connect with the database. You don't have to specify both of them. -[//]: # ( secret:) +You can configure following fields in `spec.clientConfig` section: -[//]: # ( name: quick-postgres-auth) +- **spec.clientConfig.url** -[//]: # ( clientConfig:) + `spec.clientConfig.url` gives the location of the database, in standard URL form (i.e. `[scheme://]host:port/[path]`). This is particularly useful when the target database is running outside the Kubernetes cluster. If your database is running inside the cluster, use `spec.clientConfig.service` section instead. -[//]: # ( service:) +> Note that, attempting to use a user or basic auth (e.g. `user:password@host:port`) is not allowed. Stash will insert them automatically from the respective secret. Fragments ("#...") and query parameters ("?...") are not allowed either. -[//]: # ( name: quick-postgres) +- **spec.clientConfig.service** -[//]: # ( path: /) + If you are running the database inside the Kubernetes cluster, you can use Kubernetes service to connect with the database. You have to specify the following fields in `spec.clientConfig.service` section if you manually create an `AppBinding` object. -[//]: # ( port: 5432) + - **name :** `name` indicates the name of the service that connects with the target database. + - **scheme :** `scheme` specifies the scheme (i.e. http, https) to use to connect with the database. + - **port :** `port` specifies the port where the target database is running. -[//]: # ( query: sslmode=disable) +- **spec.clientConfig.insecureSkipTLSVerify** -[//]: # ( scheme: postgresql) + `spec.clientConfig.insecureSkipTLSVerify` is used to disable TLS certificate verification while connecting with the database. We strongly discourage to disable TLS verification during backup. You should provide the respective CA bundle through `spec.clientConfig.caBundle` field instead. -[//]: # ( secretTransforms:) +- **spec.clientConfig.caBundle** -[//]: # ( - renameKey:) + `spec.clientConfig.caBundle` is a PEM encoded CA bundle which will be used to validate the serving certificate of the database. -[//]: # ( from: POSTGRES_USER) +## Next Steps -[//]: # ( to: username) - -[//]: # ( - renameKey:) - -[//]: # ( from: POSTGRES_PASSWORD) - -[//]: # ( to: password) - -[//]: # ( version: "10.2") - -[//]: # (```) - -[//]: # () -[//]: # (Here, we are going to describe the sections of an `AppBinding` crd.) - -[//]: # () -[//]: # (### AppBinding `Spec`) - -[//]: # () -[//]: # (An `AppBinding` object has the following fields in the `spec` section:) - -[//]: # () -[//]: # (#### spec.type) - -[//]: # () -[//]: # (`spec.type` is an optional field that indicates the type of the app that this `AppBinding` is pointing to. Stash uses this field to resolve the values of `TARGET_APP_TYPE`, `TARGET_APP_GROUP` and `TARGET_APP_RESOURCE` variables of [BackupBlueprint](https://appscode.com/products/stash/latest/concepts/crds/backupblueprint/) object.) - -[//]: # () -[//]: # (This field follows the following format: `/`. The above AppBinding is pointing to a `postgres` resource under `kubedb.com` group.) - -[//]: # () -[//]: # (Here, the variables are parsed as follows:) - -[//]: # () -[//]: # (| Variable | Usage |) - -[//]: # (| --------------------- | --------------------------------------------------------------------------------------------------------------------------------- |) - -[//]: # (| `TARGET_APP_GROUP` | Represents the application group where the respective app belongs (i.e: `kubedb.com`). |) - -[//]: # (| `TARGET_APP_RESOURCE` | Represents the resource under that application group that this appbinding represents (i.e: `postgres`). |) - -[//]: # (| `TARGET_APP_TYPE` | Represents the complete type of the application. It's simply `TARGET_APP_GROUP/TARGET_APP_RESOURCE` (i.e: `kubedb.com/postgres`). |) - -[//]: # () -[//]: # (#### spec.secret) - -[//]: # () -[//]: # (`spec.secret` specifies the name of the secret which contains the credentials that are required to access the database. This secret must be in the same namespace as the `AppBinding`.) - -[//]: # () -[//]: # (This secret must contain the following keys:) - -[//]: # () -[//]: # (PostgreSQL :) - -[//]: # () -[//]: # (| Key | Usage |) - -[//]: # (| ------------------- | --------------------------------------------------- |) - -[//]: # (| `POSTGRES_USER` | Username of the target database. |) - -[//]: # (| `POSTGRES_PASSWORD` | Password for the user specified by `POSTGRES_USER`. |) - -[//]: # () -[//]: # (MySQL :) - -[//]: # () -[//]: # (| Key | Usage |) - -[//]: # (| ---------- | ---------------------------------------------- |) - -[//]: # (| `username` | Username of the target database. |) - -[//]: # (| `password` | Password for the user specified by `username`. |) - -[//]: # () -[//]: # (MongoDB :) - -[//]: # () -[//]: # (| Key | Usage |) - -[//]: # (| ---------- | ---------------------------------------------- |) - -[//]: # (| `username` | Username of the target database. |) - -[//]: # (| `password` | Password for the user specified by `username`. |) - -[//]: # () -[//]: # (Elasticsearch:) - -[//]: # () -[//]: # (| Key | Usage |) - -[//]: # (| ---------------- | ----------------------- |) - -[//]: # (| `ADMIN_USERNAME` | Admin username |) - -[//]: # (| `ADMIN_PASSWORD` | Password for admin user |) - -[//]: # () -[//]: # (#### spec.clientConfig) - -[//]: # () -[//]: # (`spec.clientConfig` defines how to communicate with the target database. You can use either an URL or a Kubernetes service to connect with the database. You don't have to specify both of them.) - -[//]: # () -[//]: # (You can configure following fields in `spec.clientConfig` section:) - -[//]: # () -[//]: # (- **spec.clientConfig.url**) - -[//]: # () -[//]: # ( `spec.clientConfig.url` gives the location of the database, in standard URL form (i.e. `[scheme://]host:port/[path]`). This is particularly useful when the target database is running outside of the Kubernetes cluster. If your database is running inside the cluster, use `spec.clientConfig.service` section instead.) - -[//]: # () -[//]: # ( > Note that, attempting to use a user or basic auth (e.g. `user:password@host:port`) is not allowed. Stash will insert them automatically from the respective secret. Fragments ("#...") and query parameters ("?...") are not allowed either.) - -[//]: # () -[//]: # (- **spec.clientConfig.service**) - -[//]: # () -[//]: # ( If you are running the database inside the Kubernetes cluster, you can use Kubernetes service to connect with the database. You have to specify the following fields in `spec.clientConfig.service` section if you manually create an `AppBinding` object.) - -[//]: # () -[//]: # ( - **name :** `name` indicates the name of the service that connects with the target database.) - -[//]: # ( - **scheme :** `scheme` specifies the scheme (i.e. http, https) to use to connect with the database.) - -[//]: # ( - **port :** `port` specifies the port where the target database is running.) - -[//]: # () -[//]: # (- **spec.clientConfig.insecureSkipTLSVerify**) - -[//]: # () -[//]: # ( `spec.clientConfig.insecureSkipTLSVerify` is used to disable TLS certificate verification while connecting with the database. We strongly discourage to disable TLS verification during backup. You should provide the respective CA bundle through `spec.clientConfig.caBundle` field instead.) - -[//]: # () -[//]: # (- **spec.clientConfig.caBundle**) - -[//]: # () -[//]: # ( `spec.clientConfig.caBundle` is a PEM encoded CA bundle which will be used to validate the serving certificate of the database.) - -[//]: # (## Next Steps) - -[//]: # () -[//]: # (- Learn how to use KubeDB to manage various databases [here](/docs/guides/README.md).) - -[//]: # (- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).) +- Learn how to use KubeDB to manage various databases [here](/docs/guides/README.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/concepts/catalog.md b/docs/guides/druid/concepts/catalog.md deleted file mode 100644 index 57ef475dc8..0000000000 --- a/docs/guides/druid/concepts/catalog.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -title: DruidVersion CRD -menu: - docs_{{ .version }}: - identifier: guides-druid-concepts-catalog - name: DruidVersion - parent: guides-druid-concepts - weight: 15 -menu_name: docs_{{ .version }} -section_menu_id: guides ---- - -> New to KubeDB? Please start [here](/docs/README.md). - -# DruidVersion - -[//]: # (## What is DruidVersion) - -[//]: # () -[//]: # (`DruidVersion` is a Kubernetes `Custom Resource Definitions` (CRD). It provides a declarative configuration to specify the docker images to be used for [PgBouncer](https://pgbouncer.github.io/) server deployed with KubeDB in a Kubernetes native way.) - -[//]: # () -[//]: # (When you install KubeDB, a `DruidVersion` custom resource will be created automatically for every supported PgBouncer release versions. You have to specify the name of `DruidVersion` crd in `spec.version` field of [PgBouncer](/docs/guides/pgbouncer/concepts/pgbouncer.md) crd. Then, KubeDB will use the docker images specified in the `DruidVersion` crd to create your expected PgBouncer instance.) - -[//]: # () -[//]: # (Using a separate crd for specifying respective docker image names allow us to modify the images independent of KubeDB operator. This will also allow the users to use a custom PgBouncer image for their server. For more details about how to use custom image with PgBouncer in KubeDB, please visit [here](/docs/guides/pgbouncer/custom-versions/setup.md).) - -[//]: # (## DruidVersion Specification) - -[//]: # () -[//]: # (As with all other Kubernetes objects, a DruidVersion needs `apiVersion`, `kind`, and `metadata` fields. It also needs a `.spec` section.) - -[//]: # () -[//]: # (```yaml) - -[//]: # (apiVersion: catalog.kubedb.com/v1alpha1) - -[//]: # (kind: DruidVersion) - -[//]: # (metadata:) - -[//]: # ( name: "1.17.0") - -[//]: # ( labels:) - -[//]: # ( app: kubedb) - -[//]: # (spec:) - -[//]: # ( deprecated: false) - -[//]: # ( version: "1.17.0") - -[//]: # ( pgBouncer:) - -[//]: # ( image: "${KUBEDB_CATALOG_REGISTRY}/pgbouncer:1.17.0") - -[//]: # ( exporter:) - -[//]: # ( image: "${KUBEDB_CATALOG_REGISTRY}/pgbouncer_exporter:v0.1.1") - -[//]: # (```) - -[//]: # () -[//]: # (### metadata.name) - -[//]: # () -[//]: # (`metadata.name` is a required field that specifies the name of the `DruidVersion` crd. You have to specify this name in `spec.version` field of [PgBouncer](/docs/guides/pgbouncer/concepts/pgbouncer.md) crd.) - -[//]: # () -[//]: # (We follow this convention for naming DruidVersion crd:) - -[//]: # () -[//]: # (- Name format: `{Original pgbouncer image version}-{modification tag}`) - -[//]: # () -[//]: # (We plan to modify original PgBouncer docker images to support additional features. Re-tagging the image with v1, v2 etc. modification tag helps separating newer iterations from the older ones. An image with higher modification tag will have more features than the images with lower modification tag. Hence, it is recommended to use DruidVersion crd with highest modification tag to take advantage of the latest features.) - -[//]: # () -[//]: # (### spec.version) - -[//]: # () -[//]: # (`spec.version` is a required field that specifies the original version of PgBouncer that has been used to build the docker image specified in `spec.server.image` field.) - -[//]: # () -[//]: # (### spec.deprecated) - -[//]: # () -[//]: # (`spec.deprecated` is an optional field that specifies whether the docker images specified here is supported by the current KubeDB operator.) - -[//]: # () -[//]: # (The default value of this field is `false`. If `spec.deprecated` is set `true`, KubeDB operator will not create the server and other respective resources for this version.) - -[//]: # () -[//]: # (### spec.pgBouncer.image) - -[//]: # () -[//]: # (`spec.pgBouncer.image` is a required field that specifies the docker image which will be used to create Petset by KubeDB operator to create expected PgBouncer server.) - -[//]: # () -[//]: # (### spec.exporter.image) - -[//]: # () -[//]: # (`spec.exporter.image` is a required field that specifies the image which will be used to export Prometheus metrics.) - -[//]: # (## Next Steps) - -[//]: # () -[//]: # (- Learn about PgBouncer crd [here](/docs/guides/pgbouncer/concepts/catalog.md).) - -[//]: # (- Deploy your first PgBouncer server with KubeDB by following the guide [here](/docs/guides/pgbouncer/quickstart/quickstart.md).) \ No newline at end of file diff --git a/docs/guides/druid/concepts/druid.md b/docs/guides/druid/concepts/druid.md index 41824e7cfe..5d3ad66d29 100644 --- a/docs/guides/druid/concepts/druid.md +++ b/docs/guides/druid/concepts/druid.md @@ -12,357 +12,518 @@ section_menu_id: guides > New to KubeDB? Please start [here](/docs/README.md). - # Druid -[//]: # () -[//]: # (## What is PgBouncer) - -[//]: # () -[//]: # (`PgBouncer` is a Kubernetes `Custom Resource Definitions` (CRD). It provides declarative configuration for [PgBouncer](https://www.pgbouncer.github.io/) in a Kubernetes native way. You only need to describe the desired configurations in a `PgBouncer` object, and the KubeDB operator will create Kubernetes resources in the desired state for you.) - -[//]: # () -[//]: # (## PgBouncer Spec) - -[//]: # () -[//]: # (Like any official Kubernetes resource, a `PgBouncer` object has `TypeMeta`, `ObjectMeta`, `Spec` and `Status` sections.) - -[//]: # () -[//]: # (Below is an example PgBouncer object.) - -[//]: # () -[//]: # (```yaml) - -[//]: # (apiVersion: kubedb.com/v1alpha2) - -[//]: # (kind: PgBouncer) - -[//]: # (metadata:) - -[//]: # ( name: pgbouncer-server) - -[//]: # ( namespace: demo) - -[//]: # (spec:) - -[//]: # ( version: "1.18.0") - -[//]: # ( replicas: 2) - -[//]: # ( databases:) - -[//]: # ( - alias: "postgres") - -[//]: # ( databaseName: "postgres") - -[//]: # ( databaseRef:) - -[//]: # ( name: "quick-postgres") - -[//]: # ( namespace: demo) - -[//]: # ( connectionPool:) - -[//]: # ( maxClientConnections: 20) - -[//]: # ( reservePoolSize: 5) - -[//]: # ( monitor:) - -[//]: # ( agent: prometheus.io/operator) - -[//]: # ( prometheus:) - -[//]: # ( serviceMonitor:) - -[//]: # ( labels:) - -[//]: # ( release: prometheus) - -[//]: # ( interval: 10s) - -[//]: # (```) - -[//]: # () -[//]: # (### spec.version) - -[//]: # () -[//]: # (`spec.version` is a required field that specifies the name of the [PgBouncerVersion](/docs/guides/pgbouncer/concepts/catalog.md) crd where the docker images are specified. Currently, when you install KubeDB, it creates the following `PgBouncerVersion` resources,) - -[//]: # () -[//]: # (- `1.18.0`) - -[//]: # () -[//]: # (### spec.replicas) - -[//]: # () -[//]: # (`spec.replicas` specifies the total number of available pgbouncer server nodes for each crd. KubeDB uses `PodDisruptionBudget` to ensure that majority of the replicas are available during [voluntary disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#voluntary-and-involuntary-disruptions).) - -[//]: # () -[//]: # (### spec.databases) - -[//]: # () -[//]: # (`spec.databases` specifies an array of postgres databases that pgbouncer should add to its connection pool. It contains three `required` fields and two `optional` fields for each database connection.) - -[//]: # () -[//]: # (- `spec.databases.alias`: specifies an alias for the target database located in a postgres server specified by an appbinding.) - -[//]: # (- `spec.databases.databaseName`: specifies the name of the target database.) - -[//]: # (- `spec.databases.databaseRef`: specifies the name and namespace of the AppBinding that contains the path to a PostgreSQL server where the target database can be found.) - -[//]: # () -[//]: # (ConnectionPool is used to configure pgbouncer connection-pool. All the fields here are accompanied by default values and can be left unspecified if no customisation is required by the user.) - -[//]: # () -[//]: # (- `spec.connectionPool.port`: specifies the port on which pgbouncer should listen to connect with clients. The default is 5432.) - -[//]: # () -[//]: # (- `spec.connectionPool.poolMode`: specifies the value of pool_mode. Specifies when a server connection can be reused by other clients.) - -[//]: # () -[//]: # ( - session) - -[//]: # () -[//]: # ( Server is released back to pool after client disconnects. Default.) - -[//]: # () -[//]: # ( - transaction) - -[//]: # () -[//]: # ( Server is released back to pool after transaction finishes.) +## What is Druid + +`Druid` is a Kubernetes `Custom Resource Definitions` (CRD). It provides declarative configuration for [Druid](https://druid.apache.org/) in a Kubernetes native way. You only need to describe the desired database configuration in a `Druid`object, and the KubeDB operator will create Kubernetes objects in the desired state for you. + +## Druid Spec + +As with all other Kubernetes objects, a Druid needs `apiVersion`, `kind`, and `metadata` fields. It also needs a `.spec` section. Below is an example Druid object. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid + namespace: demo +spec: + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + metadataStorage: + type: PostgreSQL + name: pg-demo + namespace: demo + externallyManaged: true + zookeeperRef: + name: zk-demo + namespace: demo + externallyManaged: true + authSecret: + name: druid-admin-cred + configSecret: + name: druid-custom-config + enableSSL: true + healthChecker: + failureThreshold: 3 + periodSeconds: 20 + timeoutSeconds: 10 + keystoreCredSecret: + name: druid-keystore-cred + deletionPolicy: DoNotTerminate + tls: + certificates: + - alias: server + secretName: druid-server-cert + - alias: client + secretName: druid-client-cert + issuerRef: + apiGroup: cert-manager.io + kind: Issuer + name: druid-ca-issuer + topology: + coordinators: + podTemplate: + spec: + containers: + - name: druid + resources: + requests: + cpu: 500m + memory: 1024Mi + limits: + cpu: 700m + memory: 2Gi + overlords: + podTemplate: + spec: + containers: + - name: druid + resources: + requests: + cpu: 500m + memory: 1024Mi + limits: + cpu: 700m + memory: 2Gi + brokers: + podTemplate: + spec: + containers: + - name: druid + resources: + requests: + cpu: 500m + memory: 1024Mi + limits: + cpu: 700m + memory: 2Gi + routers: + podTemplate: + spec: + containers: + - name: druid + resources: + requests: + cpu: 500m + memory: 1024Mi + limits: + cpu: 700m + memory: 2Gi + middleManagers: + podTemplate: + spec: + containers: + - name: druid + resources: + requests: + cpu: 500m + memory: 1024Mi + limits: + cpu: 700m + memory: 2Gi + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: standard + historicals: + podTemplate: + spec: + containers: + - name: druid + resources: + requests: + cpu: 500m + memory: 1024Mi + limits: + cpu: 700m + memory: 2Gi + storageType: Durable + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: standard + monitor: + agent: prometheus.io/operator + prometheus: + exporter: + port: 56790 + serviceMonitor: + labels: + release: prometheus + interval: 10s + version: 30.0.0 +``` + +### spec.version + +`spec.version` is a required field specifying the name of the [DruidVersion](/docs/guides/druid/concepts/druidversion.md) crd where the docker images are specified. Currently, when you install KubeDB, it creates the following `Druid` resources, + +- `28.0.1` +- `30.0.0` + +### spec.replicas + +`spec.replicas` the number of members in Druid replicaset. + +If `spec.topology` is set, then `spec.replicas` needs to be empty. Instead use `spec.topology.controller.replicas` and `spec.topology.broker.replicas`. You need to set both of them for topology clustering. + +KubeDB uses `PodDisruptionBudget` to ensure that majority of these replicas are available during [voluntary disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#voluntary-and-involuntary-disruptions) so that quorum is maintained. + +### spec.authSecret + +`spec.authSecret` is an optional field that points to a Secret used to hold credentials for `druid` admin user. If not set, KubeDB operator creates a new Secret `{druid-object-name}-auth` for storing the password for `admin` user for each Druid object. + +We can use this field in 3 mode. +1. Using an external secret. In this case, You need to create an auth secret first with required fields, then specify the secret name when creating the Druid object using `spec.authSecret.name` & set `spec.authSecret.externallyManaged` to true. +```yaml +authSecret: + name: + externallyManaged: true +``` + +2. Specifying the secret name only. In this case, You need to specify the secret name when creating the Druid object using `spec.authSecret.name`. `externallyManaged` is by default false. +```yaml +authSecret: + name: +``` + +3. Let KubeDB do everything for you. In this case, no work for you. + +AuthSecret contains a `user` key and a `password` key which contains the `username` and `password` respectively for Druid `admin` user. + +Example: + +```bash +$ kubectl create secret generic druid-auth -n demo \ +--from-literal=username=jhon-doe \ +--from-literal=password=6q8u_2jMOW-OOZXk +secret "druid-auth" created +``` -[//]: # () -[//]: # ( - statement) +```yaml +apiVersion: v1 +data: + password: NnE4dV8yak1PVy1PT1pYaw== + username: amhvbi1kb2U= +kind: Secret +metadata: + name: druid-auth + namespace: demo +type: Opaque +``` -[//]: # () -[//]: # ( Server is released back to pool after query finishes. Long transactions spanning multiple statements are disallowed in this mode.) +Secrets provided by users are not managed by KubeDB, and therefore, won't be modified or garbage collected by the KubeDB operator (version 0.13.0 and higher). -[//]: # () -[//]: # (- `spec.connectionPool.maxClientConnections`: specifies the value of max_client_conn. When increased then the file descriptor limits should also be increased. Note that actual number of file descriptors used is more than max_client_conn. Theoretical maximum used is:) +### spec.configSecret -[//]: # () -[//]: # ( ```bash) +`spec.configSecret` is an optional field that points to a Secret used to hold custom Druid configuration. If not set, KubeDB operator will use default configuration for Druid. -[//]: # ( max_client_conn + (max pool_size * total databases * total users)) +### spec.topology -[//]: # ( ```) +`spec.topology` represents the topology configuration for Druid cluster in KRaft mode. -[//]: # () -[//]: # ( if each user connects under its own username to server. If a database user is specified in connect string (all users connect under same username), the theoretical maximum is:) +When `spec.topology` is set, the following fields needs to be empty, otherwise validating webhook will throw error. -[//]: # () -[//]: # ( ```bash) +- `spec.replicas` +- `spec.podTemplate` +- `spec.storage` -[//]: # ( max_client_conn + (max pool_size * total databases)) +#### spec.topology.coordinators -[//]: # ( ```) +`coordinators` represents configuration for coordinators node of Druid. It is a mandatory node. So, if not mentioned in the `YAML`, this node will be initialized by `KubeDB` operator. -[//]: # () -[//]: # ( The theoretical maximum should be never reached, unless somebody deliberately crafts special load for it. Still, it means you should set the number of file descriptors to a safely high number.) +Available configurable fields: -[//]: # () -[//]: # ( Search for `ulimit` in your favorite shell man page. Note: `ulimit` does not apply in a Windows environment.) +- `topology.coordinators`: + - `replicas` (`: "1"`) - is an `optional` field to specify the number of nodes (ie. pods ) that act as the dedicated Druid `coordinators` pods. Defaults to `1`. + - `suffix` (`: "coordinators"`) - is an `optional` field that is added as the suffix of the coordinators PetSet name. Defaults to `coordinators`. + - `resources` (`: "cpu: 500m, memory: 1Gi" `) - is an `optional` field that specifies how much computational resources to request or to limit for each of the `coordinators` pods. -[//]: # () -[//]: # ( Default: 100) +#### spec.topology.overlords -[//]: # () -[//]: # (- `spec.connectionPool.defaultPoolSize`: specifies the value of default_pool_size. Used to determine how many server connections to allow per user/database pair. Can be overridden in the per-database configuration.) +`overlords` represents configuration for overlords node of Druid. It is an optional node. So, it is only going to be deployed by the `KubeDB` operator if explicitly mentioned in the `YAML`. Otherwise, `coordinators` node will act as `overlords`. -[//]: # () -[//]: # ( Default: 20) +Available configurable fields: -[//]: # () -[//]: # (- `spec.connectionPool.minPoolSize`: specifies the value of min_pool_size. PgBouncer adds more server connections to pool if below this number. Improves behavior when usual load comes suddenly back after period of total inactivity.) +- `topology.overlords`: + - `replicas` (`: "1"`) - is an `optional` field to specify the number of nodes (ie. pods ) that act as the dedicated Druid `overlords` pods. Defaults to `1`. + - `suffix` (`: "overlords"`) - is an `optional` field that is added as the suffix of the overlords PetSet name. Defaults to `overlords`. + - `resources` (`: "cpu: 500m, memory: 1Gi" `) - is an `optional` field that specifies how much computational resources to request or to limit for each of the `overlords` pods. -[//]: # () -[//]: # ( Default: 0 (disabled)) +#### spec.topology.brokers -[//]: # () -[//]: # (- `spec.connectionPool.reservePoolSize`: specifies the value of reserve_pool_size. Used to determine how many additional connections to allow to a pool. 0 disables.) +`brokers` represents configuration for brokers node of Druid. It is a mandatory node. So, if not mentioned in the `YAML`, this node will be initialized by `KubeDB` operator. -[//]: # () -[//]: # ( Default: 0 (disabled)) +Available configurable fields: -[//]: # () -[//]: # (- `spec.connectionPool.reservePoolTimeout`: specifies the value of reserve_pool_timeout. If a client has not been serviced in this many seconds, pgbouncer enables use of additional connections from reserve pool. 0 disables.) +- `topology.brokers`: + - `replicas` (`: "1"`) - is an `optional` field to specify the number of nodes (ie. pods ) that act as the dedicated Druid `brokers` pods. Defaults to `1`. + - `suffix` (`: "brokers"`) - is an `optional` field that is added as the suffix of the brokers PetSet name. Defaults to `brokers`. + - `resources` (`: "cpu: 500m, memory: 1Gi" `) - is an `optional` field that specifies how much computational resources to request or to limit for each of the `brokers` pods. -[//]: # () -[//]: # ( Default: 5.0) +#### spec.topology.routers -[//]: # () -[//]: # (- `spec.connectionPool.maxDbConnections`: specifies the value of max_db_connections. PgBouncer does not allow more than this many connections per-database (regardless of pool - i.e. user). It should be noted that when you hit the limit, closing a client connection to one pool will not immediately allow a server connection to be established for another pool, because the server connection for the first pool is still open. Once the server connection closes (due to idle timeout), a new server connection will immediately be opened for the waiting pool.) +`routers` represents configuration for routers node of Druid. It is an optional node. So, it is only going to be deployed by the `KubeDB` operator if explicitly mentioned in the `YAML`. Otherwise, `coordinators` node will act as `routers`. -[//]: # () -[//]: # ( Default: unlimited) +Available configurable fields: -[//]: # () -[//]: # (- `spec.connectionPool.maxUserConnections`: specifies the value of max_user_connections. PgBouncer does not allow more than this many connections per-user (regardless of pool - i.e. user). It should be noted that when you hit the limit, closing a client connection to one pool will not immediately allow a server connection to be established for another pool, because the server connection for the first pool is still open. Once the server connection closes (due to idle timeout), a new server connection will immediately be opened for the waiting pool.) +- `topology.routers`: + - `replicas` (`: "1"`) - is an `optional` field to specify the number of nodes (ie. pods ) that act as the dedicated Druid `routers` pods. Defaults to `1`. + - `suffix` (`: "routers"`) - is an `optional` field that is added as the suffix of the routers PetSet name. Defaults to `routers`. + - `resources` (`: "cpu: 500m, memory: 1Gi" `) - is an `optional` field that specifies how much computational resources to request or to limit for each of the `routers` pods. -[//]: # ( Default: unlimited) +#### spec.topology.historicals -[//]: # () -[//]: # (- `spec.connectionPool.statsPeriod`: sets how often the averages shown in various `SHOW` commands are updated and how often aggregated statistics are written to the log.) +`historicals` represents configuration for historicals node of Druid. It is a mandatory node. So, if not mentioned in the `YAML`, this node will be initialized by `KubeDB` operator. -[//]: # ( Default: 60) +Available configurable fields: -[//]: # () -[//]: # (- `spec.connectionPool.authType`: specifies how to authenticate users. PgBouncer supports several authentication methods including pam, md5, scram-sha-256, trust , or any. However hba, and cert are not supported.) +- `topology.historicals`: + - `replicas` (`: "1"`) - is an `optional` field to specify the number of nodes (ie. pods ) that act as the dedicated Druid `historicals` pods. Defaults to `1`. + - `suffix` (`: "historicals"`) - is an `optional` field that is added as the suffix of the controller PetSet name. Defaults to `historicals`. + - `storage` is a `required` field that specifies how much storage to claim for each of the `historicals` pods. + - `resources` (`: "cpu: 500m, memory: 1Gi" `) - is an `optional` field that specifies how much computational resources to request or to limit for each of the `historicals` pods. -[//]: # () -[//]: # (- `spec.connectionPool.IgnoreStartupParameters`: specifies comma-separated startup parameters that pgbouncer knows are handled by admin and it can ignore them.) +#### spec.topology.middleManagers -[//]: # () -[//]: # (### spec.monitor) +`middleManagers` represents configuration for middleManagers node of Druid. It is a mandatory node. So, if not mentioned in the `YAML`, this node will be initialized by `KubeDB` operator. -[//]: # () -[//]: # (PgBouncer managed by KubeDB can be monitored with builtin-Prometheus and Prometheus operator out-of-the-box. To learn more,) +Available configurable fields: -[//]: # () -[//]: # (- [Monitor PgBouncer with builtin Prometheus](/docs/guides/pgbouncer/monitoring/using-builtin-prometheus.md)) +- `topology.middleManagers`: + - `replicas` (`: "1"`) - is an `optional` field to specify the number of nodes (ie. pods ) that act as the dedicated Druid `middleManagers` pods. Defaults to `1`. + - `suffix` (`: "middleManagers"`) - is an `optional` field that is added as the suffix of the controller PetSet name. Defaults to `middleManagers`. + - `storage` is a `required` field that specifies how much storage to claim for each of the `middleManagers` pods. + - `resources` (`: "cpu: 500m, memory: 1Gi" `) - is an `optional` field that specifies how much computational resources to request or to limit for each of the `middleManagers` pods. -[//]: # (- [Monitor PgBouncer with Prometheus operator](/docs/guides/pgbouncer/monitoring/using-prometheus-operator.md)) -[//]: # () -[//]: # (### spec.podTemplate) +### spec.enableSSL -[//]: # () -[//]: # (KubeDB allows providing a template for pgbouncer pods through `spec.podTemplate`. KubeDB operator will pass the information provided in `spec.podTemplate` to the PetSet created for PgBouncer server) +`spec.enableSSL` is an `optional` field that specifies whether to enable TLS to HTTP layer. The default value of this field is `false`. -[//]: # () -[//]: # (KubeDB accept following fields to set in `spec.podTemplate:`) +```yaml +spec: + enableSSL: true +``` -[//]: # () -[//]: # (- metadata) +### spec.tls -[//]: # ( - annotations (pod's annotation)) +`spec.tls` specifies the TLS/SSL configurations. The KubeDB operator supports TLS management by using the [cert-manager](https://cert-manager.io/). Currently, the operator only supports the `PKCS#8` encoded certificates. -[//]: # (- controller) +```yaml +spec: + tls: + issuerRef: + apiGroup: "cert-manager.io" + kind: Issuer + name: druid-issuer + certificates: + - alias: server + privateKey: + encoding: PKCS8 + secretName: druid-client-cert + subject: + organizations: + - kubedb + - alias: http + privateKey: + encoding: PKCS8 + secretName: druid-server-cert + subject: + organizations: + - kubedb +``` -[//]: # ( - annotations (petset's annotation)) +The `spec.tls` contains the following fields: -[//]: # (- spec:) +- `tls.issuerRef` - is an `optional` field that references to the `Issuer` or `ClusterIssuer` custom resource object of [cert-manager](https://cert-manager.io/docs/concepts/issuer/). It is used to generate the necessary certificate secrets for Druid. If the `issuerRef` is not specified, the operator creates a self-signed CA and also creates necessary certificate (valid: 365 days) secrets using that CA. + - `apiGroup` - is the group name of the resource that is being referenced. Currently, the only supported value is `cert-manager.io`. + - `kind` - is the type of resource that is being referenced. The supported values are `Issuer` and `ClusterIssuer`. + - `name` - is the name of the resource ( `Issuer` or `ClusterIssuer` ) that is being referenced. -[//]: # ( - env) +- `tls.certificates` - is an `optional` field that specifies a list of certificate configurations used to configure the certificates. It has the following fields: + - `alias` - represents the identifier of the certificate. It has the following possible value: + - `server` - is used for the server certificate configuration. + - `client` - is used for the client certificate configuration. -[//]: # ( - resources) + - `secretName` - ( `string` | `"-alias-cert"` ) - specifies the k8s secret name that holds the certificates. -[//]: # ( - initContainers) + - `subject` - specifies an `X.509` distinguished name (DN). It has the following configurable fields: + - `organizations` ( `[]string` | `nil` ) - is a list of organization names. + - `organizationalUnits` ( `[]string` | `nil` ) - is a list of organization unit names. + - `countries` ( `[]string` | `nil` ) - is a list of country names (ie. Country Codes). + - `localities` ( `[]string` | `nil` ) - is a list of locality names. + - `provinces` ( `[]string` | `nil` ) - is a list of province names. + - `streetAddresses` ( `[]string` | `nil` ) - is a list of street addresses. + - `postalCodes` ( `[]string` | `nil` ) - is a list of postal codes. + - `serialNumber` ( `string` | `""` ) is a serial number. -[//]: # ( - imagePullSecrets) + For more details, visit [here](https://golang.org/pkg/crypto/x509/pkix/#Name). -[//]: # ( - affinity) + - `duration` ( `string` | `""` ) - is the period during which the certificate is valid. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as `"300m"`, `"1.5h"` or `"20h45m"`. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". + - `renewBefore` ( `string` | `""` ) - is a specifiable time before expiration duration. + - `dnsNames` ( `[]string` | `nil` ) - is a list of subject alt names. + - `ipAddresses` ( `[]string` | `nil` ) - is a list of IP addresses. + - `uris` ( `[]string` | `nil` ) - is a list of URI Subject Alternative Names. + - `emailAddresses` ( `[]string` | `nil` ) - is a list of email Subject Alternative Names. -[//]: # ( - tolerations) -[//]: # ( - priorityClassName) +### spec..storageType -[//]: # ( - priority) +`spec.storageType` is an optional field that specifies the type of storage to use for database. It can be either `Durable` or `Ephemeral`. The default value of this field is `Durable`. If `Ephemeral` is used then KubeDB will create Druid cluster using [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume. -[//]: # ( - lifecycle) +### spec..storage -[//]: # () -[//]: # (Usage of some fields in `spec.podTemplate` is described below,) +If you set `spec..storageType:` to `Durable`, then `spec..storage` is a required field that specifies the StorageClass of PVCs dynamically allocated to store data for the database. This storage spec will be passed to the PetSet created by KubeDB operator to run database pods. You can specify any StorageClass available in your cluster with appropriate resource requests. -[//]: # () -[//]: # (#### spec.podTemplate.spec.env) +- `spec..storage.storageClassName` is the name of the StorageClass used to provision PVCs. PVCs don’t necessarily have to request a class. A PVC with its storageClassName set equal to "" is always interpreted to be requesting a PV with no class, so it can only be bound to PVs with no class (no annotation or one set equal to ""). A PVC with no storageClassName is not quite the same and is treated differently by the cluster depending on whether the DefaultStorageClass admission plugin is turned on. +- `spec..storage.accessModes` uses the same conventions as Kubernetes PVCs when requesting storage with specific access modes. +- `spec..storage.resources` can be used to request specific quantities of storage. This follows the same resource model used by PVCs. -[//]: # () -[//]: # (`spec.podTemplate.spec.env` is an optional field that specifies the environment variables to pass to the PgBouncer docker image. To know about supported environment variables, please visit [here](https://hub.docker.com/kubedb/pgbouncer/).) +To learn how to configure `spec..storage`, please visit the links below: -[//]: # () -[//]: # (Also, note that KubeDB does not allow updates to the environment variables as updating them does not have any effect once the server is created. If you try to update environment variables, KubeDB operator will reject the request with following error,) +- https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims -[//]: # () -[//]: # (```ini) +### spec.monitor -[//]: # (Error from server (BadRequest): error when applying patch:) +Druid managed by KubeDB can be monitored with Prometheus operator out-of-the-box. To learn more, +- [Monitor Apache Druid with Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md) +- [Monitor Apache Druid with Built-in Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md) -[//]: # (...) +### spec..podTemplate -[//]: # (for: "./pgbouncer.yaml": admission webhook "pgbouncer.validators.kubedb.com" denied the request: precondition failed for:) +KubeDB allows providing a template for database pod through `spec..podTemplate`. KubeDB operator will pass the information provided in `spec..podTemplate` to the PetSet created for Druid cluster. -[//]: # (...) +KubeDB accept following fields to set in `spec..podTemplate:` -[//]: # (At least one of the following was changed:) +- metadata: + - annotations (pod's annotation) + - labels (pod's labels) +- controller: + - annotations (petset's annotation) + - labels (petset's labels) +- spec: + - containers + - volumes + - podPlacementPolicy + - initContainers + - containers + - imagePullSecrets + - nodeSelector + - affinity + - serviceAccountName + - schedulerName + - tolerations + - priorityClassName + - priority + - securityContext + - livenessProbe + - readinessProbe + - lifecycle -[//]: # ( apiVersion) +You can check out the full list [here](https://github.com/kmodules/offshoot-api/blob/master/api/v2/types.go#L26C1-L279C1). +Uses of some field of `spec..podTemplate` is described below, -[//]: # ( kind) +#### spec..podTemplate.spec.tolerations -[//]: # ( name) +The `spec.podTemplate.spec.tolerations` is an optional field. This can be used to specify the pod's tolerations. -[//]: # ( namespace) +#### spec..podTemplate.spec.volumes -[//]: # ( spec.podTemplate.spec.nodeSelector) +The `spec..podTemplate..volumes` is an optional field. This can be used to provide the list of volumes that can be mounted by containers belonging to the pod. -[//]: # (```) +#### spec..podTemplate.spec.podPlacementPolicy -[//]: # () -[//]: # (#### spec.podTemplate.spec.imagePullSecrets) +`spec..podTemplate.spec.podPlacementPolicy` is an optional field. This can be used to provide the reference of the podPlacementPolicy. This will be used by our Petset controller to place the db pods throughout the region, zone & nodes according to the policy. It utilizes kubernetes affinity & podTopologySpreadContraints feature to do so. -[//]: # () -[//]: # (`spec.podTemplate.spec.imagePullSecrets` is an optional field that points to secrets to be used for pulling docker image if you are using a private docker registry. For more details on how to use private docker registry, please visit [here](/docs/guides/pgbouncer/private-registry/using-private-registry.md).) +#### spec..podTemplate.spec.nodeSelector -[//]: # () -[//]: # (#### spec.podTemplate.spec.nodeSelector) +`spec..podTemplate.spec.nodeSelector` is an optional field that specifies a map of key-value pairs. For the pod to be eligible to run on a node, the node must have each of the indicated key-value pairs as labels (it can have additional labels as well). To learn more, see [here](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) . -[//]: # () -[//]: # (`spec.podTemplate.spec.nodeSelector` is an optional field that specifies a map of key-value pairs. For the pod to be eligible to run on a node, the node must have each of the indicated key-value pairs as labels (it can have additional labels as well). To learn more, see [here](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) .) +### spec.serviceTemplates -[//]: # () -[//]: # (#### spec.podTemplate.spec.resources) +You can also provide template for the services created by KubeDB operator for Druid cluster through `spec.serviceTemplates`. This will allow you to set the type and other properties of the services. -[//]: # () -[//]: # (`spec.podTemplate.spec.resources` is an optional field. This can be used to request compute resources required by the database pods. To learn more, visit [here](http://kubernetes.io/docs/user-guide/compute-resources/).) +KubeDB allows following fields to set in `spec.serviceTemplates`: +- `alias` represents the identifier of the service. It has the following possible value: + - `stats` for is used for the `exporter` service identification. -[//]: # () -[//]: # (### spec.serviceTemplate) +Druid comes with four services for `coordinators`, `overlords`, `routers` and `brokers`. There are two options for providing serviceTemplates: + - To provide `serviceTemplates` for a specific service, the `serviceTemplates.ports.port` should be equal to the port of that service and `serviceTemplate` will be used for that particular service only. + - However, to provide a common `serviceTemplates`, `serviceTemplates.ports.port` should be empty. -[//]: # () -[//]: # (KubeDB creates a service for each PgBouncer instance. The service has the same name as the `pgbouncer.name` and points to pgbouncer pods.) +- metadata: + - labels + - annotations +- spec: + - type + - ports + - clusterIP + - externalIPs + - loadBalancerIP + - loadBalancerSourceRanges + - externalTrafficPolicy + - healthCheckNodePort + - sessionAffinityConfig -[//]: # () -[//]: # (You can provide template for this service using `spec.serviceTemplate`. This will allow you to set the type and other properties of the service. If `spec.serviceTemplate` is not provided, KubeDB will create a service of type `ClusterIP` with minimal settings.) +See [here](https://github.com/kmodules/offshoot-api/blob/kubernetes-1.21.1/api/v1/types.go#L237) to understand these fields in detail. -[//]: # () -[//]: # (KubeDB allows the following fields to set in `spec.serviceTemplate`:) -[//]: # () -[//]: # (- metadata:) +#### spec..podTemplate.spec.containers -[//]: # ( - annotations) +The `spec..podTemplate.spec.containers` can be used to provide the list containers and their configurations for to the database pod. some of the fields are described below, -[//]: # (- spec:) +##### spec..podTemplate.spec.containers[].name +The `spec..podTemplate.spec.containers[].name` field used to specify the name of the container specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL). Cannot be updated. -[//]: # ( - type) +##### spec..podTemplate.spec.containers[].args +`spec..podTemplate.spec.containers[].args` is an optional field. This can be used to provide additional arguments to database installation. -[//]: # ( - ports) +##### spec..podTemplate.spec.containers[].env -[//]: # ( - clusterIP) +`spec..podTemplate.spec.containers[].env` is an optional field that specifies the environment variables to pass to the Redis containers. -[//]: # ( - externalIPs) +##### spec..podTemplate.spec.containers[].resources -[//]: # ( - loadBalancerIP) +`spec..podTemplate.spec.containers[].resources` is an optional field. This can be used to request compute resources required by containers of the database pods. To learn more, visit [here](http://kubernetes.io/docs/user-guide/compute-resources/). -[//]: # ( - loadBalancerSourceRanges) +### spec.deletionPolicy -[//]: # ( - externalTrafficPolicy) +`deletionPolicy` gives flexibility whether to `nullify`(reject) the delete operation of `Druid` crd or which resources KubeDB should keep or delete when you delete `Druid` crd. KubeDB provides following four deletion policies: -[//]: # ( - healthCheckNodePort) +- DoNotTerminate +- WipeOut +- Halt +- Delete -[//]: # ( - sessionAffinityConfig) +When `deletionPolicy` is `DoNotTerminate`, KubeDB takes advantage of `ValidationWebhook` feature in Kubernetes 1.9.0 or later clusters to implement `DoNotTerminate` feature. If admission webhook is enabled, `DoNotTerminate` prevents users from deleting the database as long as the `spec.deletionPolicy` is set to `DoNotTerminate`. -[//]: # () -[//]: # (See [here](https://github.com/kmodules/offshoot-api/blob/kubernetes-1.16.3/api/v1/types.go#L163) to understand these fields in detail.) +## spec.healthChecker +It defines the attributes for the health checker. +- `spec.healthChecker.periodSeconds` specifies how often to perform the health check. +- `spec.healthChecker.timeoutSeconds` specifies the number of seconds after which the probe times out. +- `spec.healthChecker.failureThreshold` specifies minimum consecutive failures for the healthChecker to be considered failed. +- `spec.healthChecker.disableWriteCheck` specifies whether to disable the writeCheck or not. -[//]: # () -[//]: # (## Next Steps) +Know details about KubeDB Health checking from this [blog post](https://appscode.com/blog/post/kubedb-health-checker/). -[//]: # () -[//]: # (- Learn how to use KubeDB to run a PostgreSQL database [here](/docs/guides/postgres/README.md).) +## Next Steps -[//]: # (- Learn how to how to get started with PgBouncer [here](/docs/guides/pgbouncer/quickstart/quickstart.md).) +- Learn how to use KubeDB to run Apache Druid cluster [here](/docs/guides/druid/README.md). +- Deploy [dedicated topology cluster](/docs/guides/druid/clustering/guide/index.md) for Apache Druid +- Monitor your Druid cluster with KubeDB using [`out-of-the-box` Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). +- Detail concepts of [DruidVersion object](/docs/guides/druid/concepts/druidversion.md). -[//]: # (- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md).) +[//]: # (- Learn to use KubeDB managed Druid objects using [CLIs](/docs/guides/druid/cli/cli.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/concepts/druidautoscaler.md b/docs/guides/druid/concepts/druidautoscaler.md new file mode 100644 index 0000000000..b22504ce2f --- /dev/null +++ b/docs/guides/druid/concepts/druidautoscaler.md @@ -0,0 +1,132 @@ +--- +title: DruidAutoscaler CRD +menu: + docs_{{ .version }}: + identifier: guides-druid-concepts-druidautoscaler + name: DruidAutoscaler + parent: guides-druid-concepts + weight: 50 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# DruidAutoscaler + +## What is DruidAutoscaler + +`DruidAutoscaler` is a Kubernetes `Custom Resource Definitions` (CRD). It provides a declarative configuration for autoscaling [Druid](https://druid.apache.org/) compute resources and storage of database components in a Kubernetes native way. + +## DruidAutoscaler CRD Specifications + +Like any official Kubernetes resource, a `DruidAutoscaler` has `TypeMeta`, `ObjectMeta`, `Spec` and `Status` sections. + +Here, some sample `DruidAutoscaler` CROs for autoscaling different components of database is given below: + +**Sample `DruidAutoscaler` for `druid` cluster:** + +```yaml +apiVersion: autoscaling.kubedb.com/v1alpha1 +kind: DruidAutoscaler +metadata: + name: dr-autoscaler + namespace: demo +spec: + databaseRef: + name: druid-prod + opsRequestOptions: + timeout: 3m + apply: IfReady + compute: + coordinators: + trigger: "On" + podLifeTimeThreshold: 24h + minAllowed: + cpu: 200m + memory: 300Mi + maxAllowed: + cpu: 1 + memory: 1Gi + controlledResources: ["cpu", "memory"] + containerControlledValues: "RequestsAndLimits" + resourceDiffPercentage: 10 + brokers: + trigger: "On" + podLifeTimeThreshold: 24h + minAllowed: + cpu: 200m + memory: 300Mi + maxAllowed: + cpu: 1 + memory: 1Gi + controlledResources: ["cpu", "memory"] + containerControlledValues: "RequestsAndLimits" + resourceDiffPercentage: 10 + storage: + historicals: + expansionMode: "Online" + trigger: "On" + usageThreshold: 60 + scalingThreshold: 50 + middleManagers: + expansionMode: "Online" + trigger: "On" + usageThreshold: 60 + scalingThreshold: 50 +``` + +Here, we are going to describe the various sections of a `DruidAutoscaler` crd. + +A `DruidAutoscaler` object has the following fields in the `spec` section. + +### spec.databaseRef + +`spec.databaseRef` is a required field that point to the [Druid](/docs/guides/druid/concepts/druid.md) object for which the autoscaling will be performed. This field consists of the following sub-field: + +- **spec.databaseRef.name :** specifies the name of the [Druid](/docs/guides/druid/concepts/druid.md) object. + +### spec.opsRequestOptions +These are the options to pass in the internally created opsRequest CRO. `opsRequestOptions` has two fields. + +### spec.compute + +`spec.compute` specifies the autoscaling configuration for the compute resources i.e. cpu and memory of the database components. This field consists of the following sub-field: + +- `spec.compute.coordinators` indicates the desired compute autoscaling configuration for coordinators of a topology Druid database. +- `spec.compute.overlords` indicates the desired compute autoscaling configuration for overlords of a topology Druid database. +- `spec.compute.brokers` indicates the desired compute autoscaling configuration for brokers of a topology Druid database. +- `spec.compute.routers` indicates the desired compute autoscaling configuration for routers of a topology Druid database. +- `spec.compute.historicals` indicates the desired compute autoscaling configuration for historicals of a topology Druid database. +- `spec.compute.middleManagers` indicates the desired compute autoscaling configuration for middleManagers of a topology Druid database. + + +All of them has the following sub-fields: + +- `trigger` indicates if compute autoscaling is enabled for this component of the database. If "On" then compute autoscaling is enabled. If "Off" then compute autoscaling is disabled. +- `minAllowed` specifies the minimal amount of resources that will be recommended, default is no minimum. +- `maxAllowed` specifies the maximum amount of resources that will be recommended, default is no maximum. +- `controlledResources` specifies which type of compute resources (cpu and memory) are allowed for autoscaling. Allowed values are "cpu" and "memory". +- `containerControlledValues` specifies which resource values should be controlled. Allowed values are "RequestsAndLimits" and "RequestsOnly". +- `resourceDiffPercentage` specifies the minimum resource difference between recommended value and the current value in percentage. If the difference percentage is greater than this value than autoscaling will be triggered. +- `podLifeTimeThreshold` specifies the minimum pod lifetime of at least one of the pods before triggering autoscaling. + +There are two more fields, those are only specifiable for the percona variant inMemory databases. +- `inMemoryStorage.UsageThresholdPercentage` If db uses more than usageThresholdPercentage of the total memory, memoryStorage should be increased. +- `inMemoryStorage.ScalingFactorPercentage` If db uses more than usageThresholdPercentage of the total memory, memoryStorage should be increased by this given scaling percentage. + +### spec.storage + +`spec.storage` specifies the autoscaling configuration for the storage resources of the database components. This field consists of the following sub-field: + +- `spec.storage.historicals` indicates the desired storage autoscaling configuration for historicals of a topology Druid cluster. +- `spec.storage.middleManagers` indicates the desired storage autoscaling configuration for middleManagers of a topology Druid cluster. + +> `spec.storage` is only supported for druid data nodes i.e. `historicals` and `middleManagers` as they are the only nodes containing volumes. + +All of them has the following sub-fields: + +- `trigger` indicates if storage autoscaling is enabled for this component of the database. If "On" then storage autoscaling is enabled. If "Off" then storage autoscaling is disabled. +- `usageThreshold` indicates usage percentage threshold, if the current storage usage exceeds then storage autoscaling will be triggered. +- `scalingThreshold` indicates the percentage of the current storage that will be scaled. +- `expansionMode` indicates the volume expansion mode. diff --git a/docs/guides/druid/concepts/druidopsrequest.md b/docs/guides/druid/concepts/druidopsrequest.md new file mode 100644 index 0000000000..6a846814f2 --- /dev/null +++ b/docs/guides/druid/concepts/druidopsrequest.md @@ -0,0 +1,473 @@ +--- +title: DruidOpsRequest CRD +menu: + docs_{{ .version }}: + identifier: guides-druid-concepts-druidopsrequest + name: DruidOpsRequest + parent: guides-druid-concepts + weight: 40 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + + +> New to KubeDB? Please start [here](/docs/README.md). + +# DruidOpsRequest + +## What is DruidOpsRequest + +`DruidOpsRequest` is a Kubernetes `Custom Resource Definitions` (CRD). It provides a declarative configuration for [Druid](https://druid.apache.org/) administrative operations like database version updating, horizontal scaling, vertical scaling etc. in a Kubernetes native way. + +## DruidOpsRequest CRD Specifications + +Like any official Kubernetes resource, a `DruidOpsRequest` has `TypeMeta`, `ObjectMeta`, `Spec` and `Status` sections. + +Here, some sample `DruidOpsRequest` CRs for different administrative operations is given below: + +**Sample `DruidOpsRequest` for updating database:** + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: update-version + namespace: demo +spec: + type: UpdateVersion + databaseRef: + name: druid-prod + updateVersion: + targetVersion: 30.0.1 +status: + conditions: + - lastTransitionTime: "2024-07-25T18:22:38Z" + message: Successfully completed the modification process + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +**Sample `DruidOpsRequest` Objects for Horizontal Scaling of different component of the database:** + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-hscale-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: druid-prod + horizontalScaling: + topology: + coordinators: 2 + historicals: 2 +status: + conditions: + - lastTransitionTime: "2024-07-25T18:22:38Z" + message: Successfully completed the modification process + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +**Sample `DruidOpsRequest` Objects for Vertical Scaling of different component of the database:** + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-vscale + namespace: demo +spec: + type: VerticalScaling + databaseRef: + name: druid-prod + verticalScaling: + coordinators: + resources: + requests: + memory: "1.5Gi" + cpu: "0.7" + limits: + memory: "2Gi" + cpu: "1" + historicals: + resources: + requests: + memory: "1.5Gi" + cpu: "0.7" + limits: + memory: "2Gi" + cpu: "1" +status: + conditions: + - lastTransitionTime: "2024-07-25T18:22:38Z" + message: Successfully completed the modification process + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +**Sample `DruidOpsRequest` Objects for Reconfiguring different druid mode:** + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-reconfiugre + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: druid-prod + configuration: + applyConfig: + middleManager.properties: | + druid.worker.capacity=5 +status: + conditions: + - lastTransitionTime: "2024-07-25T18:22:38Z" + message: Successfully completed the modification process + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-reconfiugre + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: druid-prod + configuration: + configSecret: + name: new-configsecret +status: + conditions: + - lastTransitionTime: "2024-07-25T18:22:38Z" + message: Successfully completed the modification process + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +**Sample `DruidOpsRequest` Objects for Volume Expansion of different database components:** + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-volume-exp + namespace: demo +spec: + type: VolumeExpansion + databaseRef: + name: druid-prod + volumeExpansion: + mode: "Online" + historicals: 2Gi + middleManagers: 2Gi +status: + conditions: + - lastTransitionTime: "2024-07-25T18:22:38Z" + message: Successfully completed the modification process + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +**Sample `DruidOpsRequest` Objects for Reconfiguring TLS of the database:** + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-add-tls + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-prod + tls: + issuerRef: + name: dr-issuer + kind: Issuer + apiGroup: "cert-manager.io" + certificates: + - alias: client + emailAddresses: + - abc@appscode.com +``` + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-rotate + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-dev + tls: + rotateCertificates: true +``` + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-change-issuer + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-prod + tls: + issuerRef: + name: dr-new-issuer + kind: Issuer + apiGroup: "cert-manager.io" +``` + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-remove + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-prod + tls: + remove: true +``` + +Here, we are going to describe the various sections of a `DruidOpsRequest` crd. + +A `DruidOpsRequest` object has the following fields in the `spec` section. + +### spec.databaseRef + +`spec.databaseRef` is a required field that point to the [Druid](/docs/guides/druid/concepts/druid.md) object for which the administrative operations will be performed. This field consists of the following sub-field: + +- **spec.databaseRef.name :** specifies the name of the [Druid](/docs/guides/druid/concepts/druid.md) object. + +### spec.type + +`spec.type` specifies the kind of operation that will be applied to the database. Currently, the following types of operations are allowed in `DruidOpsRequest`. + +- `UpdateVersion` +- `HorizontalScaling` +- `VerticalScaling` +- `VolumeExpansion` +- `Reconfigure` +- `ReconfigureTLS` +- `Restart` + +> You can perform only one type of operation on a single `DruidOpsRequest` CR. For example, if you want to update your database and scale up its replica then you have to create two separate `DruidOpsRequest`. At first, you have to create a `DruidOpsRequest` for updating. Once it is completed, then you can create another `DruidOpsRequest` for scaling. + +### spec.updateVersion + +If you want to update you Druid version, you have to specify the `spec.updateVersion` section that specifies the desired version information. This field consists of the following sub-field: + +- `spec.updateVersion.targetVersion` refers to a [DruidVersion](/docs/guides/druid/concepts/druidversion.md) CR that contains the Druid version information where you want to update. + +> You can only update between Druid versions. KubeDB does not support downgrade for Druid. + +### spec.horizontalScaling + +If you want to scale-up or scale-down your Druid cluster or different components of it, you have to specify `spec.horizontalScaling` section. This field consists of the following sub-field: + +- `spec.horizontalScaling.topology` indicates the configuration of topology nodes for Druid topology cluster after scaling. This field consists of the following sub-field: + - `spec.horizontalScaling.topoloy.coordinators` indicates the desired number of coordinators nodes for Druid topology cluster after scaling. + - `spec.horizontalScaling.topology.overlords` indicates the desired number of overlords nodes for Druid topology cluster after scaling. + - `spec.horizontalScaling.topology.brokers` indicates the desired number of brokers nodes for Druid topology cluster after scaling. + - `spec.horizontalScaling.topology.routers` indicates the desired number of routers nodes for Druid topology cluster after scaling. + - `spec.horizontalScaling.topology.historicals` indicates the desired number of historicals nodes for Druid topology cluster after scaling. + - `spec.horizontalScaling.topology.middleManagers` indicates the desired number of middleManagers nodes for Druid topology cluster after scaling. + +### spec.verticalScaling + +`spec.verticalScaling` is a required field specifying the information of `Druid` resources like `cpu`, `memory` etc that will be scaled. This field consists of the following sub-fields: +- `spec.verticalScaling.coordinators` indicates the desired resources for coordinators of Druid topology cluster after scaling. +- `spec.verticalScaling.overlords` indicates the desired resources for overlords of Druid topology cluster after scaling. +- `spec.verticalScaling.brokers` indicates the desired resources for brokers of Druid topology cluster after scaling. +- `spec.verticalScaling.routers` indicates the desired resources for routers of Druid topology cluster after scaling. +- `spec.verticalScaling.historicals` indicates the desired resources for historicals of Druid topology cluster after scaling. +- `spec.verticalScaling.middleManagers` indicates the desired resources for middleManagers of Druid topology cluster after scaling. + +All of them has the below structure: + +```yaml +requests: + memory: "200Mi" + cpu: "0.1" +limits: + memory: "300Mi" + cpu: "0.2" +``` + +Here, when you specify the resource request, the scheduler uses this information to decide which node to place the container of the Pod on and when you specify a resource limit for the container, the `kubelet` enforces those limits so that the running container is not allowed to use more of that resource than the limit you set. You can found more details from [here](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). + +### spec.volumeExpansion + +> To use the volume expansion feature the storage class must support volume expansion + +If you want to expand the volume of your Druid cluster or different components of it, you have to specify `spec.volumeExpansion` section. This field consists of the following sub-field: + +- `spec.mode` specifies the volume expansion mode. Supported values are `Online` & `Offline`. The default is `Online`. +- `spec.volumeExpansion.historicals` indicates the desired size for the persistent volume for historicals of a Druid topology cluster. +- `spec.volumeExpansion.middleManagers` indicates the desired size for the persistent volume for middleManagers of a Druid topology cluster. + +> It is only possible to expand the data servers ie. `historicals` and `middleManagers` as they only comes with persistent volumes. + +All of them refer to [Quantity](https://v1-22.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#quantity-resource-core) types of Kubernetes. + +Example usage of this field is given below: + +```yaml +spec: + volumeExpansion: + node: "2Gi" +``` + +This will expand the volume size of all the combined nodes to 2 GB. + +### spec.configuration + +If you want to reconfigure your Running Druid cluster or different components of it with new custom configuration, you have to specify `spec.configuration` section. This field consists of the following sub-field: + +- `spec.configuration.configSecret` points to a secret in the same namespace of a Druid resource, which contains the new custom configurations. If there are any configSecret set before in the database, this secret will replace it. The value of the field `spec.stringData` of the secret like below: +```yaml +common.runtime.properties: | + druid.storage.archiveBucket="my-druid-archive-bucket" +middleManagers.properties: | + druid.worker.capacity=5 +``` +> Similarly, it is possible to provide configs for `coordinators`, `overlords`, `brokers`, `routers` and `historicals` through `coordinators.properties`, `overlords.properties`, `brokers.properties`, `routers.properties` and `historicals.properties` respectively. + +- `applyConfig` contains the new custom config as a string which will be merged with the previous configuration. + +- `applyConfig` is a map where key supports 3 values, namely `server.properties`, `broker.properties`, `controller.properties`. And value represents the corresponding configurations. + +```yaml + applyConfig: + common.runtime.properties: | + druid.storage.archiveBucket="my-druid-archive-bucket" + middleManagers.properties: | + druid.worker.capacity=5 +``` + +- `removeCustomConfig` is a boolean field. Specify this field to true if you want to remove all the custom configuration from the deployed druid cluster. + +### spec.tls + +If you want to reconfigure the TLS configuration of your Druid i.e. add TLS, remove TLS, update issuer/cluster issuer or Certificates and rotate the certificates, you have to specify `spec.tls` section. This field consists of the following sub-field: + +- `spec.tls.issuerRef` specifies the issuer name, kind and api group. +- `spec.tls.certificates` specifies the certificates. You can learn more about this field from [here](/docs/guides/druid/concepts/druid.md#spectls). +- `spec.tls.rotateCertificates` specifies that we want to rotate the certificate of this druid. +- `spec.tls.remove` specifies that we want to remove tls from this druid. + +### spec.timeout +As we internally retry the ops request steps multiple times, This `timeout` field helps the users to specify the timeout for those steps of the ops request (in second). +If a step doesn't finish within the specified timeout, the ops request will result in failure. + +### spec.apply +This field controls the execution of obsRequest depending on the database state. It has two supported values: `Always` & `IfReady`. +Use IfReady, if you want to process the opsRequest only when the database is Ready. And use Always, if you want to process the execution of opsReq irrespective of the Database state. + +### DruidOpsRequest `Status` + +`.status` describes the current state and progress of a `DruidOpsRequest` operation. It has the following fields: + +### status.phase + +`status.phase` indicates the overall phase of the operation for this `DruidOpsRequest`. It can have the following three values: + +| Phase | Meaning | +|-------------|----------------------------------------------------------------------------------| +| Successful | KubeDB has successfully performed the operation requested in the DruidOpsRequest | +| Progressing | KubeDB has started the execution of the applied DruidOpsRequest | +| Failed | KubeDB has failed the operation requested in the DruidOpsRequest | +| Denied | KubeDB has denied the operation requested in the DruidOpsRequest | +| Skipped | KubeDB has skipped the operation requested in the DruidOpsRequest | + +Important: Ops-manager Operator can skip an opsRequest, only if its execution has not been started yet & there is a newer opsRequest applied in the cluster. `spec.type` has to be same as the skipped one, in this case. + +### status.observedGeneration + +`status.observedGeneration` shows the most recent generation observed by the `DruidOpsRequest` controller. + +### status.conditions + +`status.conditions` is an array that specifies the conditions of different steps of `DruidOpsRequest` processing. Each condition entry has the following fields: + +- `types` specifies the type of the condition. DruidOpsRequest has the following types of conditions: + +| Type | Meaning | +|-------------------------------|---------------------------------------------------------------------------| +| `Progressing` | Specifies that the operation is now in the progressing state | +| `Successful` | Specifies such a state that the operation on the database was successful. | +| `HaltDatabase` | Specifies such a state that the database is halted by the operator | +| `ResumeDatabase` | Specifies such a state that the database is resumed by the operator | +| `Failed` | Specifies such a state that the operation on the database failed. | +| `StartingBalancer` | Specifies such a state that the balancer has successfully started | +| `StoppingBalancer` | Specifies such a state that the balancer has successfully stopped | +| `UpdateShardImage` | Specifies such a state that the Shard Images has been updated | +| `UpdateReplicaSetImage` | Specifies such a state that the Replicaset Image has been updated | +| `UpdateConfigServerImage` | Specifies such a state that the ConfigServer Image has been updated | +| `UpdateMongosImage` | Specifies such a state that the Mongos Image has been updated | +| `UpdatePetSetResources` | Specifies such a state that the Petset resources has been updated | +| `UpdateShardResources` | Specifies such a state that the Shard resources has been updated | +| `UpdateReplicaSetResources` | Specifies such a state that the Replicaset resources has been updated | +| `UpdateConfigServerResources` | Specifies such a state that the ConfigServer resources has been updated | +| `UpdateMongosResources` | Specifies such a state that the Mongos resources has been updated | +| `ScaleDownReplicaSet` | Specifies such a state that the scale down operation of replicaset | +| `ScaleUpReplicaSet` | Specifies such a state that the scale up operation of replicaset | +| `ScaleUpShardReplicas` | Specifies such a state that the scale up operation of shard replicas | +| `ScaleDownShardReplicas` | Specifies such a state that the scale down operation of shard replicas | +| `ScaleDownConfigServer` | Specifies such a state that the scale down operation of config server | +| `ScaleUpConfigServer` | Specifies such a state that the scale up operation of config server | +| `ScaleMongos` | Specifies such a state that the scale down operation of replicaset | +| `VolumeExpansion` | Specifies such a state that the volume expansion operaton of the database | +| `ReconfigureReplicaset` | Specifies such a state that the reconfiguration of replicaset nodes | +| `ReconfigureMongos` | Specifies such a state that the reconfiguration of mongos nodes | +| `ReconfigureShard` | Specifies such a state that the reconfiguration of shard nodes | +| `ReconfigureConfigServer` | Specifies such a state that the reconfiguration of config server nodes | + +- The `status` field is a string, with possible values `True`, `False`, and `Unknown`. + - `status` will be `True` if the current transition succeeded. + - `status` will be `False` if the current transition failed. + - `status` will be `Unknown` if the current transition was denied. +- The `message` field is a human-readable message indicating details about the condition. +- The `reason` field is a unique, one-word, CamelCase reason for the condition's last transition. +- The `lastTransitionTime` field provides a timestamp for when the operation last transitioned from one state to another. +- The `observedGeneration` shows the most recent condition transition generation observed by the controller. diff --git a/docs/guides/druid/concepts/druidversion.md b/docs/guides/druid/concepts/druidversion.md new file mode 100644 index 0000000000..74bd7e76f2 --- /dev/null +++ b/docs/guides/druid/concepts/druidversion.md @@ -0,0 +1,102 @@ +--- +title: DruidVersion CRD +menu: + docs_{{ .version }}: + identifier: guides-druid-concepts-druidversion + name: DruidVersion + parent: guides-druid-concepts + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# DruidVersion + +## What is DruidVersion + +`DruidVersion` is a Kubernetes `Custom Resource Definitions` (CRD). It provides a declarative configuration to specify the docker images to be used for [Druid](https://druid.apache.org) database deployed with KubeDB in a Kubernetes native way. + +When you install KubeDB, a `DruidVersion` custom resource will be created automatically for every supported Druid versions. You have to specify the name of `DruidVersion` CR in `spec.version` field of [Druid](/docs/guides/druid/concepts/druid.md) crd. Then, KubeDB will use the docker images specified in the `DruidVersion` CR to create your expected database. + +Using a separate CRD for specifying respective docker images, and pod security policy names allow us to modify the images, and policies independent of KubeDB operator.This will also allow the users to use a custom image for the database. + +## DruidVersion Spec + +As with all other Kubernetes objects, a DruidVersion needs `apiVersion`, `kind`, and `metadata` fields. It also needs a `.spec` section. + +```yaml +apiVersion: catalog.kubedb.com/v1alpha1 +kind: DruidVersion +metadata: + annotations: + meta.helm.sh/release-name: kubedb + meta.helm.sh/release-namespace: kubedb + creationTimestamp: "2024-10-16T13:10:10Z" + generation: 1 + labels: + app.kubernetes.io/instance: kubedb + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/name: kubedb-catalog + app.kubernetes.io/version: v2024.9.30 + helm.sh/chart: kubedb-catalog-v2024.9.30 + name: 28.0.1 + resourceVersion: "42125" + uid: e30e23aa-febc-4029-8be7-993afaff1fc6 +spec: + db: + image: ghcr.io/appscode-images/druid:28.0.1 + initContainer: + image: ghcr.io/kubedb/druid-init:28.0.1 + securityContext: + runAsUser: 1000 + version: 28.0.1 +``` + +### metadata.name + +`metadata.name` is a required field that specifies the name of the `DruidVersion` CR. You have to specify this name in `spec.version` field of [Druid](/docs/guides/druid/concepts/druid.md) CR. + +We follow this convention for naming DruidVersion CR: + +- Name format: `{Original Druid image version}-{modification tag}` + +We use official Apache Druid release tar files to build docker images for supporting Druid versions and re-tag the image with v1, v2 etc. modification tag when there's any. An image with higher modification tag will have more features than the images with lower modification tag. Hence, it is recommended to use DruidVersion CR with the highest modification tag to enjoy the latest features. + +### spec.version + +`spec.version` is a required field that specifies the original version of Druid database that has been used to build the docker image specified in `spec.db.image` field. + +### spec.deprecated + +`spec.deprecated` is an optional field that specifies whether the docker images specified here is supported by the current KubeDB operator. + +The default value of this field is `false`. If `spec.deprecated` is set to `true`, KubeDB operator will skip processing this CRD object and will add a event to the CRD object specifying that the DB version is deprecated. + +### spec.db.image + +`spec.db.image` is a required field that specifies the docker image which will be used to create PetSet by KubeDB operator to create expected Druid database. + +### spec.initContainer.image + +`spec.initContainer.image` is a required field that specifies the image which will be used to remove `lost+found` directory and mount an `EmptyDir` data volume. + +### spec.podSecurityPolicies.databasePolicyName + +`spec.podSecurityPolicies.databasePolicyName` is a required field that specifies the name of the pod security policy required to get the database server pod(s) running. + +```bash +helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ + --namespace kubedb --create-namespace \ + --set global.featureGates.Druid=true --set global.featureGates.ZooKeeper=true \ + --set additionalPodSecurityPolicies[0]=custom-db-policy \ + --set additionalPodSecurityPolicies[1]=custom-snapshotter-policy \ + --set-file global.license=/path/to/the/license.txt \ + --wait --burst-limit=10000 --debug +``` + +## Next Steps + +- Learn about Druid CRD [here](/docs/guides/druid/concepts/druid.md). +- Deploy your first Druid database with KubeDB by following the guide [here](/docs/guides/druid/quickstart/guide/index.md). diff --git a/docs/guides/druid/configuration/_index.md b/docs/guides/druid/configuration/_index.md new file mode 100644 index 0000000000..4aaebde0ed --- /dev/null +++ b/docs/guides/druid/configuration/_index.md @@ -0,0 +1,10 @@ +--- +title: Run Druid with Custom Configuration +menu: + docs_{{ .version }}: + identifier: guides-druid-configuration + name: Custom Configuration + parent: guides-druid + weight: 40 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/configuration/config-file/images/druid-updated-ui.png b/docs/guides/druid/configuration/config-file/images/druid-updated-ui.png new file mode 100644 index 0000000000..4e53f6bb00 Binary files /dev/null and b/docs/guides/druid/configuration/config-file/images/druid-updated-ui.png differ diff --git a/docs/guides/druid/configuration/config-file/index.md b/docs/guides/druid/configuration/config-file/index.md new file mode 100644 index 0000000000..dca8653758 --- /dev/null +++ b/docs/guides/druid/configuration/config-file/index.md @@ -0,0 +1,284 @@ +--- +title: Configuring Druid Cluster +menu: + docs_{{ .version }}: + identifier: guides-druid-configuration-config-file + name: Configuration File + parent: guides-druid-configuration + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +> New to KubeDB? Please start [here](/docs/README.md). + +# Configure Druid Cluster + +In Druid cluster, there are six nodes available coordinators, overlords, brokers, routers, historicals, middleManagers. In this tutorial, we will see how to configure each node of a druid cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency with helm command. + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create namespace demo +namespace/demo created + +$ kubectl get namespace +NAME STATUS AGE +demo Active 9s +``` + +> Note: YAML files used in this tutorial are stored in [here](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/druid/configuration/yamls) in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Find Available StorageClass + +We will have to provide `StorageClass` in Druid CR specification. Check available `StorageClass` in your cluster using the following command, + +```bash +$ kubectl get storageclass +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 1h +``` + +Here, we have `standard` StorageClass in our cluster from [Local Path Provisioner](https://github.com/rancher/local-path-provisioner). + +Before deploying `Druid` cluster, we need to prepare the external dependencies. + +## Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/backup/application-level/examples/deep-storage-config.yaml +secret/deep-storage-config created +``` + +## Use Custom Configuration + +Say we want to change the default maximum number of tasks the MiddleManager can accept. Let's create the `middleManagers.properties` file with our desire configurations. + +**middleManagers.properties:** + +```properties +druid.worker.capacity=5 +``` + +and we also want to change the number of processing threads to have available for parallel processing of segments of the historicals nodes. Let's create the `historicals.properties` file with our desire configurations. + +**historicals.properties:** + +```properties +druid.processing.numThreads=3 +``` + +Let's create a k8s secret containing the above configuration where the file name will be the key and the file-content as the value: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: configsecret + namespace: demo +stringData: + middleManagers.properties: |- + druid.worker.capacity=5 + historicals.properties: |- + druid.processing.numThreads=3 +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/configuration/config-file/yamls/config-secret.yaml +secret/config-secret created +``` + +> To provide custom configuration for other nodes add values for the following `key` under `stringData`: +> - Use `common.runtime.properties` for common configurations +> - Use `coordinators.properties` for configurations of coordinators +> - Use `overlords.properties` for configurations of overlords +> - Use `brokers.properties` for configurations of brokers +> - Use `routers.properties` for configurations of routers + +Now that the config secret is created, it needs to be mentioned in the [Druid](/docs/guides/druid/concepts/druid.md) object's yaml: + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-with-config + namespace: demo +spec: + version: 28.0.1 + configSecret: + name: config-secret + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: WipeOut +``` + +Now, create the Druid object by the following command: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/configuration/config-file/yamls/druid-with-monitoring.yaml +druid.kubedb.com/druid-with-config created +``` + +Now, wait for the Druid to become ready: + +```bash +$ kubectl get dr -n demo -w +NAME TYPE VERSION STATUS AGE +druid-with-config kubedb.com/v1 3.6.1 Provisioning 5s +druid-with-config kubedb.com/v1 3.6.1 Provisioning 7s +. +. +druid-with-config kubedb.com/v1 3.6.1 Ready 2m +``` + +## Verify Configuration + +Lets exec into one of the druid middleManagers pod that we have created and check the configurations are applied or not: + +Exec into the Druid middleManagers: + +```bash +$ kubectl exec -it -n demo druid-with-config-middleManagers-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ +``` + +Now, execute the following commands to see the configurations: +```bash +bash-5.1$ cat conf/druid/cluster/data/middleManager/runtime.properties | grep druid.worker.capacity +druid.worker.capacity=5 +``` +Here, we can see that our given configuration is applied to the Druid cluster for all brokers. + +Now, lets exec into one of the druid historicals pod that we have created and check the configurations are applied or not: + +Exec into the Druid historicals: + +```bash +$ kubectl exec -it -n demo druid-with-config-historicals-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ +``` + +Now, execute the following commands to see the metadata storage directory: +```bash +bash-5.1$ cat conf/druid/cluster/data/historical/runtime.properties | grep druid.processing.numThreads +druid.processing.numThreads=3 +``` + +Here, we can see that our given configuration is applied to the historicals. + +### Verify Configuration Change from Druid UI +You can also see the configuration changes from the druid ui. For that, follow the following steps: + +First port-forward the port `8888` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-with-config-routers 8888 +Forwarding from 127.0.0.1:8888 -> 8888 +Forwarding from [::1]:8888 -> 8888 +``` + + +Now hit the `http://localhost:8888` from any browser, and you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-with-config-admin-cred -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-with-config-admin-cred -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  druid-ui +

+ + +You can see that there are 5 task slots reflecting with our provided custom configuration of `druid.worker.capacity=5`. + +## Cleanup + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +$ kubectl delete dr -n demo druid-dev + +$ kubectl delete secret -n demo configsecret-combined + +$ kubectl delete namespace demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). + diff --git a/docs/guides/druid/configuration/config-file/yamls/config-secret.yaml b/docs/guides/druid/configuration/config-file/yamls/config-secret.yaml new file mode 100644 index 0000000000..6067ee7dd2 --- /dev/null +++ b/docs/guides/druid/configuration/config-file/yamls/config-secret.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Secret +metadata: + name: new-config + namespace: demo +stringData: + middleManagers.properties: |- + druid.worker.capacity=5 + historicals.properties: |- + druid.processing.numThreads=3 diff --git a/docs/guides/druid/configuration/config-file/yamls/deep-storage-config.yaml b/docs/guides/druid/configuration/config-file/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/configuration/config-file/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/configuration/config-file/yamls/druid-with-config.yaml b/docs/guides/druid/configuration/config-file/yamls/druid-with-config.yaml new file mode 100644 index 0000000000..b2225f22b1 --- /dev/null +++ b/docs/guides/druid/configuration/config-file/yamls/druid-with-config.yaml @@ -0,0 +1,17 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-with-config + namespace: demo +spec: + version: 28.0.1 + configSecret: + name: config-secret + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: WipeOut diff --git a/docs/guides/druid/configuration/podtemplating/index.md b/docs/guides/druid/configuration/podtemplating/index.md new file mode 100644 index 0000000000..89c4ae2c68 --- /dev/null +++ b/docs/guides/druid/configuration/podtemplating/index.md @@ -0,0 +1,618 @@ +--- +title: Run Druid with Custom PodTemplate +menu: + docs_{{ .version }}: + identifier: guides-druid-configuration-podtemplating + name: Customize PodTemplate + parent: guides-druid-configuration + weight: 15 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Run Druid with Custom PodTemplate + +KubeDB supports providing custom configuration for Druid via [PodTemplate](/docs/guides/druid/concepts/druid.md#spec.topology). This tutorial will show you how to use KubeDB to run a Druid database with custom configuration using PodTemplate. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + + ```bash + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/guides/druid/configuration/podtemplating/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/druid/configuration/podtemplating/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Overview + +KubeDB allows providing a template for `leaf` and `aggregator` pod through `spec.topology.aggregator.podTemplate` and `spec.topology.leaf.podTemplate`. KubeDB operator will pass the information provided in `spec.topology.aggregator.podTemplate` and `spec.topology.leaf.podTemplate` to the `aggregator` and `leaf` PetSet created for Druid database. +KubeDB allows providing a template for all the druid pods through `spec.topology..podTemplate`. KubeDB operator will pass the information provided in `spec.topology..podTemplate` to the corresponding PetSet created for Druid database. + +KubeDB accept following fields to set in `spec.podTemplate:` + +- metadata: + - annotations (pod's annotation) + - labels (pod's labels) +- controller: + - annotations (statefulset's annotation) + - labels (statefulset's labels) +- spec: + - volumes + - initContainers + - containers + - imagePullSecrets + - nodeSelector + - affinity + - serviceAccountName + - schedulerName + - tolerations + - priorityClassName + - priority + - securityContext + - livenessProbe + - readinessProbe + - lifecycle + +Read about the fields in details in [PodTemplate concept](/docs/guides/druid/concepts/druid.md#spectopology), + + +## Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/backup/application-level/examples/deep-storage-config.yaml +secret/deep-storage-config created +``` + +## CRD Configuration + +Below is the YAML for the Druid created in this example. Here, [`spec.topology.aggregator/leaf.podTemplate.spec.args`](/docs/guides/mysql/concepts/database/index.md#specpodtemplatespecargs) provides extra arguments. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + configSecret: + name: config-secret + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + coordinators: + replicas: 1 + podTemplate: + spec: + containers: + - name: druid + resources: + limits: + memory: "2Gi" + cpu: "600m" + requests: + memory: "2Gi" + cpu: "600m" + brokers: + replicas: 1 + podTemplate: + spec: + containers: + - name: druid + resources: + limits: + memory: "2Gi" + cpu: "600m" + requests: + memory: "2Gi" + cpu: "600m" + routers: + replicas: 1 + deletionPolicy: WipeOut +``` + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/configuration/podtemplating/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait a few minutes. KubeDB operator will create necessary PVC, petset, services, secret etc. If everything goes well, we will see that `druid-cluster` is in `Ready` state. +```bash +$ kubectl get druid -n demo +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1alpha2 28.0.1 Ready 6m5s +``` + +Check that the petset's pod is running + +```bash +$ kubectl get pods -n demo -l app.kubernetes.io/instance=druid-cluster +NAME READY STATUS RESTARTS AGE +druid-cluster-brokers-0 1/1 Running 0 7m2s +druid-cluster-coordinators-0 1/1 Running 0 7m9s +druid-cluster-historicals-0 1/1 Running 0 7m7s +druid-cluster-middlemanagers-0 1/1 Running 0 7m5s +druid-cluster-routers-0 1/1 Running 0 7m +``` + +Now, we will check if the database has started with the custom configuration we have provided. + +```bash +$ kubectl get pod -n demo druid-cluster-coordinators-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "cpu": "600m", + "memory": "2Gi" + }, + "requests": { + "cpu": "600m", + "memory": "2Gi" + } +} + +$ kubectl get pod -n demo druid-cluster-brokers-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "cpu": "600m", + "memory": "2Gi" + }, + "requests": { + "cpu": "600m", + "memory": "2Gi" + } +} +``` + +Here we can see the containers of the both `coordinators` and `brokers` have the resources we have specified in the manifest. + +## Using Node Selector + +Here in this example we will use [node selector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) to schedule our druid pod to a specific node. Applying nodeSelector to the Pod involves several steps. We first need to assign a label to some node that will be later used by the `nodeSelector` . Let’s find what nodes exist in your cluster. To get the name of these nodes, you can run: + +```bash +$ kubectl get nodes --show-labels +NAME STATUS ROLES AGE VERSION LABELS +lke212553-307295-339173d10000 Ready 36m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-339173d10000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=618158120a299c6fd37f00d01d355ca18794c467,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +lke212553-307295-5541798e0000 Ready 36m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-5541798e0000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=75cfe3dbbb0380f1727efc53f5192897485e95d5,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +lke212553-307295-5b53c5520000 Ready 36m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-5b53c5520000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=792bac078d7ce0e548163b9423416d7d8c88b08f,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +``` +As you see, we have three nodes in the cluster: lke212553-307295-339173d10000, lke212553-307295-5541798e0000, and lke212553-307295-5b53c5520000. + +Next, select a node to which you want to add a label. For example, let’s say we want to add a new label with the key `disktype` and value ssd to the `lke212553-307295-5541798e0000` node, which is a node with the SSD storage. To do so, run: +```bash +$ kubectl label nodes lke212553-307295-5541798e0000 disktype=ssd +node/lke212553-307295-5541798e0000 labeled +``` +As you noticed, the command above follows the format `kubectl label nodes =` . +Finally, let’s verify that the new label was added by running: +```bash + $ kubectl get nodes --show-labels +NAME STATUS ROLES AGE VERSION LABELS +lke212553-307295-339173d10000 Ready 41m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-339173d10000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=618158120a299c6fd37f00d01d355ca18794c467,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +lke212553-307295-5541798e0000 Ready 41m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,disktype=ssd,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-5541798e0000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=75cfe3dbbb0380f1727efc53f5192897485e95d5,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +lke212553-307295-5b53c5520000 Ready 41m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-5b53c5520000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=792bac078d7ce0e548163b9423416d7d8c88b08f,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +``` +As you see, the lke212553-307295-5541798e0000 now has a new label disktype=ssd. To see all labels attached to the node, you can also run: +```bash +$ kubectl describe node "lke212553-307295-5541798e0000" +Name: lke212553-307295-5541798e0000 +Roles: +Labels: beta.kubernetes.io/arch=amd64 + beta.kubernetes.io/instance-type=g6-dedicated-4 + beta.kubernetes.io/os=linux + disktype=ssd + failure-domain.beta.kubernetes.io/region=ap-south + kubernetes.io/arch=amd64 + kubernetes.io/hostname=lke212553-307295-5541798e0000 + kubernetes.io/os=linux + lke.linode.com/pool-id=307295 + node.k8s.linode.com/host-uuid=75cfe3dbbb0380f1727efc53f5192897485e95d5 + node.kubernetes.io/instance-type=g6-dedicated-4 + topology.kubernetes.io/region=ap-south + topology.linode.com/region=ap-south +``` +Along with the `disktype=ssd` label we’ve just added, you can see other labels such as `beta.kubernetes.io/arch` or `kubernetes.io/hostname`. These are all default labels attached to Kubernetes nodes. + +Now let's create a druid with this new label as nodeSelector. Below is the yaml we are going to apply: +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-node-selector + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + coordinators: + podTemplate: + spec: + nodeSelector: + disktype: ssd + deletionPolicy: Delete +``` + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/configuration/podtemplating/yamls/druid-node-selector.yaml +druid.kubedb.com/druid-node-selector created +``` +Now, wait a few minutes. KubeDB operator will create necessary petset, services, secret etc. If everything goes well, we will see that the `druid-node-selector` instance is in `Ready` state. + +```bash +$ kubectl get druid -n demo +NAME TYPE VERSION STATUS AGE +druid-node-selector kubedb.com/v1alpha2 28.0.1 Ready 54m +``` +You can verify that by running `kubectl get pods -n demo druid-node-selector-0 -o wide` and looking at the “NODE” to which the Pod was assigned. +```bash +$ kubectl get pods -n demo druid-node-selector-cooridnators-0 -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +druid-node-selector-cooridnators-0 1/1 Running 0 3m19s 10.2.1.7 lke212553-307295-5541798e0000 +``` +We can successfully verify that our pod was scheduled to our desired node. + +## Using Taints and Tolerations + +Here in this example we will use [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to schedule our druid pod to a specific node and also prevent from scheduling to nodes. Applying taints and tolerations to the Pod involves several steps. Let’s find what nodes exist in your cluster. To get the name of these nodes, you can run: + +```bash +$ kubectl get nodes --show-labels +NAME STATUS ROLES AGE VERSION LABELS +lke212553-307295-339173d10000 Ready 36m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-339173d10000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=618158120a299c6fd37f00d01d355ca18794c467,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +lke212553-307295-5541798e0000 Ready 36m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-5541798e0000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=75cfe3dbbb0380f1727efc53f5192897485e95d5,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +lke212553-307295-5b53c5520000 Ready 36m v1.30.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=g6-dedicated-4,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south,kubernetes.io/arch=amd64,kubernetes.io/hostname=lke212553-307295-5b53c5520000,kubernetes.io/os=linux,lke.linode.com/pool-id=307295,node.k8s.linode.com/host-uuid=792bac078d7ce0e548163b9423416d7d8c88b08f,node.kubernetes.io/instance-type=g6-dedicated-4,topology.kubernetes.io/region=ap-south,topology.linode.com/region=ap-south +``` +As you see, we have three nodes in the cluster: lke212553-307295-339173d10000, lke212553-307295-5541798e0000, and lke212553-307295-5b53c5520000. + +Next, we are going to taint these nodes. +```bash +$ kubectl taint nodes lke212553-307295-339173d10000 key1=node1:NoSchedule +node/lke212553-307295-339173d10000 tainted + +$ kubectl taint nodes lke212553-307295-5541798e0000 key1=node2:NoSchedule +node/lke212553-307295-5541798e0000 tainted + +$ kubectl taint nodes lke212553-307295-5b53c5520000 key1=node3:NoSchedule +node/lke212553-307295-5b53c5520000 tainted +``` +Let's see our tainted nodes here, +```bash +$ kubectl get nodes -o json | jq -r '.items[] | select(.spec.taints != null) | .metadata.name, .spec.taints' +lke212553-307295-339173d10000 +[ + { + "effect": "NoSchedule", + "key": "key1", + "value": "node1" + } +] +lke212553-307295-5541798e0000 +[ + { + "effect": "NoSchedule", + "key": "key1", + "value": "node2" + } +] +lke212553-307295-5b53c5520000 +[ + { + "effect": "NoSchedule", + "key": "key1", + "value": "node3" + } +] +``` +We can see that our taints were successfully assigned. Now let's try to create a druid without proper tolerations. Here is the yaml of druid we are going to create. +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-without-tolerations + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/configuration/podtemplating/yamls/druid-without-tolerations.yaml +druid.kubedb.com/druid-without-tolerations created +``` +Now, wait a few minutes. KubeDB operator will create necessary petset, services, secret etc. If everything goes well, we will see that a pod with the name `druid-without-tolerations-0` has been created and running. + +Check that the petset's pod is running or not, +```bash +$ kubectl get pods -n demo -l app.kubernetes.io/instance=druid-without-tolerations +NAME READY STATUS RESTARTS AGE +druid-without-tolerations-brokers-0 0/1 Pending 0 3m35s +druid-without-tolerations-cooridnators-0 0/1 Pending 0 3m35s +druid-without-tolerations-historicals-0 0/1 Pending 0 3m35s +druid-without-tolerations-middlemanager-0 0/1 Pending 0 3m35s +druid-without-tolerations-routers-0 0/1 Pending 0 3m35s +``` +Here we can see that the pod is not running. So let's describe the pod, +```bash +$ kubectl describe pods -n demo druid-without-tolerations-coordinators-0 +Name: druid-without-tolerations-coordinators-0 +Namespace: demo +Priority: 0 +Service Account: default +Node: kind-control-plane/172.18.0.2 +Start Time: Wed, 13 Nov 2024 11:59:06 +0600 +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-without-tolerations + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com + apps.kubernetes.io/pod-index=0 + controller-revision-hash=druid-without-tolerations-coordinators-65c8c99fc7 + kubedb.com/role=coordinators + statefulset.kubernetes.io/pod-name=druid-without-tolerations-coordinators-0 +Annotations: +Status: Running +IP: 10.244.0.53 +IPs: + IP: 10.244.0.53 +Controlled By: PetSet/druid-without-tolerations-coordinators +Init Containers: + init-druid: + Container ID: containerd://62c9a2053d619dded2085e354cd2c0dfa238761033cc0483c824c1ed8ee4c002 + Image: ghcr.io/kubedb/druid-init:28.0.1@sha256:ed87835bc0f89dea923fa8e3cf1ef209e3e41cb93944a915289322035dcd8a91 + Image ID: ghcr.io/kubedb/druid-init@sha256:ed87835bc0f89dea923fa8e3cf1ef209e3e41cb93944a915289322035dcd8a91 + Port: + Host Port: + State: Terminated + Reason: Completed + Exit Code: 0 + Started: Wed, 13 Nov 2024 11:59:07 +0600 + Finished: Wed, 13 Nov 2024 11:59:07 +0600 + Ready: True + Restart Count: 0 + Limits: + memory: 512Mi + Requests: + cpu: 200m + memory: 512Mi + Environment: + DRUID_METADATA_TLS_ENABLE: false + DRUID_METADATA_STORAGE_TYPE: MySQL + Mounts: + /opt/druid/conf from main-config-volume (rw) + /opt/druid/extensions/mysql-metadata-storage from mysql-metadata-storage (rw) + /tmp/config/custom-config from custom-config (rw) + /tmp/config/operator-config from operator-config-volume (rw) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9t5kp (ro) +Containers: + druid: + Container ID: containerd://3a52f120ca09f90fcdc062c94bf404964add7a5b6ded4a372400267a9d0fd598 + Image: ghcr.io/appscode-images/druid:28.0.1@sha256:d86e424233ec5a120c1e072cf506fa169868fd9572bbb9800a85400f0c879dec + Image ID: ghcr.io/appscode-images/druid@sha256:d86e424233ec5a120c1e072cf506fa169868fd9572bbb9800a85400f0c879dec + Port: 8081/TCP + Host Port: 0/TCP + Command: + /druid.sh + coordinator + State: Running + Started: Wed, 13 Nov 2024 11:59:09 +0600 + Ready: True + Restart Count: 0 + Limits: + cpu: 600m + memory: 2Gi + Requests: + cpu: 600m + memory: 2Gi + Environment: + DRUID_ADMIN_PASSWORD: Optional: false + DRUID_METADATA_STORAGE_PASSWORD: VHJ6!hFuT8WDjcyy + DRUID_ZK_SERVICE_PASSWORD: VHJ6!hFuT8WDjcyy + Mounts: + /opt/druid/conf from main-config-volume (rw) + /opt/druid/extensions/mysql-metadata-storage from mysql-metadata-storage (rw) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9t5kp (ro) + +Conditions: + Type Status + PodReadyToStartContainers True + Initialized True + Ready True + ContainersReady True + PodScheduled True +Volumes: + data: + Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) + ClaimName: data-druid-without-tolerations-0 + ReadOnly: false + init-scripts: + Type: EmptyDir (a temporary directory that shares a pod's lifetime) + Medium: + SizeLimit: + kube-api-access-htm2z: + Type: Projected (a volume that contains injected data from multiple sources) + TokenExpirationSeconds: 3607 + ConfigMapName: kube-root-ca.crt + ConfigMapOptional: + DownwardAPI: true +QoS Class: Burstable +Node-Selectors: +Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s + node.kubernetes.io/unreachable:NoExecute op=Exists for 300s +Topology Spread Constraints: kubernetes.io/hostname:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/component=database,app.kubernetes.io/instance=druid-without-tolerations,app.kubernetes.io/managed-by=kubedb.com,app.kubernetes.io/name=druids.kubedb.com,kubedb.com/petset=standalone + topology.kubernetes.io/zone:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/component=database,app.kubernetes.io/instance=druid-without-tolerations,app.kubernetes.io/managed-by=kubedb.com,app.kubernetes.io/name=druids.kubedb.com,kubedb.com/petset=standalone +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 5m20s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {key1: node1}, 1 node(s) had untolerated taint {key1: node2}, 1 node(s) had untolerated taint {key1: node3}. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. + Warning FailedScheduling 11s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {key1: node1}, 1 node(s) had untolerated taint {key1: node2}, 1 node(s) had untolerated taint {key1: node3}. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. + Normal NotTriggerScaleUp 13s (x31 over 5m15s) cluster-autoscaler pod didn't trigger scale-up: +``` +Here we can see that the pod has no tolerations for the tainted nodes and because of that the pod is not able to scheduled. + +So, let's add proper tolerations and create another druid. Here is the yaml we are going to apply, +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + coordinators: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + brokers: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + historicals: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + middleManagers: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + deletionPolicy: Delete +``` + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/configuration/podtemplating/yamls/druid-with-tolerations.yaml +druid.kubedb.com/druid-with-tolerations created +``` +Now, wait a few minutes. KubeDB operator will create necessary petset, services, secret etc. If everything goes well, we will see that a pod with the name `druid-with-tolerations-0` has been created. + +Check that the petset's pod is running + +```bash +$ $ kubectl get pods -n demo -l app.kubernetes.io/instance=druid-cluster + +NAME READY STATUS RESTARTS AGE +druid-with-tolerations-brokers-0 1/1 Running 0 164m +druid-with-tolerations-coordinators-0 1/1 Running 0 164m +druid-with-tolerations-historicals-0 1/1 Running 0 164m +druid-with-tolerations-middlemanagers-0 1/1 Running 0 164m +druid-with-tolerations-routers-0 1/1 Running 0 164m +``` +As we see the pod is running, you can verify that by running `kubectl get pods -n demo druid-with-tolerations-0 -o wide` and looking at the “NODE” to which the Pod was assigned. +```bash +$ kubectl get pods -n demo druid-with-tolerations-coordinators-0 -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +druid-with-tolerations-coordinators-0 1/1 Running 0 3m49s 10.2.0.8 lke212553-307295-339173d10000 +``` +We can successfully verify that our pod was scheduled to the node which it has tolerations. + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete druid -n demo druid-with-tolerations + +kubectl delete ns demo +``` + +If you would like to uninstall KubeDB operator, please follow the steps [here](/docs/setup/README.md). + +## Next Steps + +- [Quickstart Druid](/docs/guides/druid/quickstart/guide/index.md) with KubeDB Operator. +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/configuration/podtemplating/yamls/deep-storage-config.yaml b/docs/guides/druid/configuration/podtemplating/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/configuration/podtemplating/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/configuration/podtemplating/yamls/druid-cluster.yaml b/docs/guides/druid/configuration/podtemplating/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..2004002096 --- /dev/null +++ b/docs/guides/druid/configuration/podtemplating/yamls/druid-cluster.yaml @@ -0,0 +1,43 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + configSecret: + name: config-secret + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + coordinators: + replicas: 1 + podTemplate: + spec: + containers: + - name: druid + resources: + limits: + memory: "2Gi" + cpu: "600m" + requests: + memory: "2Gi" + cpu: "600m" + brokers: + replicas: 1 + podTemplate: + spec: + containers: + - name: druid + resources: + limits: + memory: "2Gi" + cpu: "600m" + requests: + memory: "2Gi" + cpu: "600m" + routers: + replicas: 1 + deletionPolicy: WipeOut diff --git a/docs/guides/druid/configuration/podtemplating/yamls/druid-node-selector.yaml b/docs/guides/druid/configuration/podtemplating/yamls/druid-node-selector.yaml new file mode 100644 index 0000000000..7ad2eae717 --- /dev/null +++ b/docs/guides/druid/configuration/podtemplating/yamls/druid-node-selector.yaml @@ -0,0 +1,20 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-node-selector + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + coordinators: + podTemplate: + spec: + nodeSelector: + disktype: ssd + deletionPolicy: Delete \ No newline at end of file diff --git a/docs/guides/druid/configuration/podtemplating/yamls/druid-with-tolerations.yaml b/docs/guides/druid/configuration/podtemplating/yamls/druid-with-tolerations.yaml new file mode 100644 index 0000000000..4ef158f85a --- /dev/null +++ b/docs/guides/druid/configuration/podtemplating/yamls/druid-with-tolerations.yaml @@ -0,0 +1,58 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + coordinators: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + brokers: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + historicals: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + middleManagers: + podTemplate: + spec: + tolerations: + - key: "key1" + operator: "Equal" + value: "node1" + effect: "NoSchedule" + replicas: 1 + deletionPolicy: Delete diff --git a/docs/guides/druid/configuration/podtemplating/yamls/druid-without-tolerations.yaml b/docs/guides/druid/configuration/podtemplating/yamls/druid-without-tolerations.yaml new file mode 100644 index 0000000000..1098f3d70d --- /dev/null +++ b/docs/guides/druid/configuration/podtemplating/yamls/druid-without-tolerations.yaml @@ -0,0 +1,15 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-without-tolerations + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete diff --git a/docs/guides/druid/monitoring/_index.md b/docs/guides/druid/monitoring/_index.md new file mode 100755 index 0000000000..dd0ffbb60c --- /dev/null +++ b/docs/guides/druid/monitoring/_index.md @@ -0,0 +1,10 @@ +--- +title: Monitor Druid with Prometheus & Grafana +menu: + docs_{{ .version }}: + identifier: guides-druid-monitoring + name: Monitoring + parent: guides-druid + weight: 140 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/monitoring/images/druid-monitoring.png b/docs/guides/druid/monitoring/images/druid-monitoring.png new file mode 100644 index 0000000000..2c609a2b98 Binary files /dev/null and b/docs/guides/druid/monitoring/images/druid-monitoring.png differ diff --git a/docs/guides/druid/monitoring/images/druid-prometheus.png b/docs/guides/druid/monitoring/images/druid-prometheus.png new file mode 100644 index 0000000000..5d6d83ed1b Binary files /dev/null and b/docs/guides/druid/monitoring/images/druid-prometheus.png differ diff --git a/docs/guides/druid/monitoring/overview.md b/docs/guides/druid/monitoring/overview.md new file mode 100644 index 0000000000..752599f2e8 --- /dev/null +++ b/docs/guides/druid/monitoring/overview.md @@ -0,0 +1,139 @@ +--- +title: Druid Monitoring Overview +description: Druid Monitoring Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-monitoring-guide + name: Overview + parent: guides-druid-monitoring + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Monitoring Apache Druid with KubeDB + +KubeDB has native support for monitoring via [Prometheus](https://prometheus.io/). You can use builtin [Prometheus](https://github.com/prometheus/prometheus) scraper or [Prometheus operator](https://github.com/prometheus-operator/prometheus-operator) to monitor KubeDB managed databases. This tutorial will show you how database monitoring works with KubeDB and how to configure Database crd to enable monitoring. + +## Overview + +KubeDB uses Prometheus [exporter](https://prometheus.io/docs/instrumenting/exporters/#databases) images to export Prometheus metrics for respective databases. As KubeDB supports Druid versions in KRaft mode, and the officially recognized exporter image doesn't expose metrics for them yet - KubeDB managed Druid instances use [JMX Exporter](https://github.com/prometheus/jmx_exporter) instead. This exporter is intended to be run as a Java Agent inside Druid container, exposing a HTTP server and serving metrics of the local JVM. To Following diagram shows the logical flow of database monitoring with KubeDB. + +

+  Database Monitoring Flow +

+ +When a user creates a Druid crd with `spec.monitor` section configured, KubeDB operator provisions the respective Druid cluster while running the exporter as a Java agent inside the druid containers. It also creates a dedicated stats service with name `{database-crd-name}-stats` for monitoring. Prometheus server can scrape metrics using this stats service. + +## Configure Monitoring + +In order to enable monitoring for a database, you have to configure `spec.monitor` section. KubeDB provides following options to configure `spec.monitor` section: + +| Field | Type | Uses | +|----------------------------------------------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------| +| `spec.monitor.agent` | `Required` | Type of the monitoring agent that will be used to monitor this database. It can be `prometheus.io/builtin` or `prometheus.io/operator`. | +| `spec.monitor.prometheus.exporter.port` | `Optional` | Port number where the exporter side car will serve metrics. | +| `spec.monitor.prometheus.exporter.args` | `Optional` | Arguments to pass to the exporter sidecar. | +| `spec.monitor.prometheus.exporter.env` | `Optional` | List of environment variables to set in the exporter sidecar container. | +| `spec.monitor.prometheus.exporter.resources` | `Optional` | Resources required by exporter sidecar container. | +| `spec.monitor.prometheus.exporter.securityContext` | `Optional` | Security options the exporter should run with. | +| `spec.monitor.prometheus.serviceMonitor.labels` | `Optional` | Labels for `ServiceMonitor` crd. | +| `spec.monitor.prometheus.serviceMonitor.interval` | `Optional` | Interval at which metrics should be scraped. | + +## Sample Configuration + +A sample YAML for TLS secured Druid crd with `spec.monitor` section configured to enable monitoring with [Prometheus operator](https://github.com/prometheus-operator/prometheus-operator) is shown below. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-with-monitoring + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + monitor: + agent: prometheus.io/operator + prometheus: + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/monitoring/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +Let's deploy the above druid example by the following command: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/monitoring/yamls/druid-with-monitoring.yaml +druid.kubedb.com/druid created +``` + +Here, we have specified that we are going to monitor this server using Prometheus operator through `spec.monitor.agent: prometheus.io/operator`. KubeDB will create a `ServiceMonitor` crd in databases namespace and this `ServiceMonitor` will have `release: prometheus` label. + +## Next Steps + +- Learn how to use KubeDB to run a Apache Druid cluster [here](/docs/guides/druid/README.md). +- Deploy [dedicated topology cluster](/docs/guides/druid/clustering/overview/index.md) for Apache Druid +- Detail concepts of [DruidVersion object](/docs/guides/druid/concepts/druidversion.md). + +[//]: # (- Learn to use KubeDB managed Druid objects using [CLIs](/docs/guides/druid/cli/cli.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). \ No newline at end of file diff --git a/docs/guides/druid/monitoring/using-builtin-prometheus.md b/docs/guides/druid/monitoring/using-builtin-prometheus.md new file mode 100644 index 0000000000..69ec2aa62e --- /dev/null +++ b/docs/guides/druid/monitoring/using-builtin-prometheus.md @@ -0,0 +1,372 @@ +--- +title: Monitor Druid using Builtin Prometheus Discovery +menu: + docs_{{ .version }}: + identifier: guides-druid-monitoring-builtin-monitoring + name: Builtin Prometheus + parent: guides-druid-monitoring + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Monitoring Druid with builtin Prometheus + +This tutorial will show you how to monitor Druid cluster using builtin [Prometheus](https://github.com/prometheus/prometheus) scraper. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- If you are not familiar with how to configure Prometheus to scrape metrics from various Kubernetes resources, please read the tutorial from [here](https://github.com/appscode/third-party-tools/tree/master/monitoring/prometheus/builtin). + +- To learn how Prometheus monitoring works with KubeDB in general, please visit [here](/docs/guides/druid/monitoring/overview.md). + +- To keep Prometheus resources isolated, we are going to use a separate namespace called `monitoring` to deploy respective monitoring resources. We are going to deploy database in `demo` namespace. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Deploy Druid with Monitoring Enabled + +At first, let's deploy a Druid cluster with monitoring enabled. Below is the Druid object that we are going to create. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-with-monitoring + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + monitor: + agent: prometheus.io/builtin + prometheus: + exporter: + port: 56790 + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +Here, + +- `spec.monitor.agent: prometheus.io/builtin` specifies that we are going to monitor this server using builtin Prometheus scraper. +- `spec.monitor.prometheus.exporter.port: 56790` specifies the port where the exporter is running. + +Let's create the Druid crd we have shown above. + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/monitoring/yamls/druid-builtin-monitoring.yaml +druid.kubedb.com/druid-with-monitoring created +``` + +Now, wait for the cluster to go into `Ready` state. + +```bash +NAME TYPE VERSION STATUS AGE +druid-with-monitoring kubedb.com/v1alpha2 28.0.1 Ready 31s +``` + +KubeDB will create a separate stats service with name `{Druid crd name}-stats` for monitoring purpose. + +```bash +$ kubectl get svc -n demo --selector="app.kubernetes.io/instance=druid-with-monitoring" +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +druid-with-monitoring-brokers ClusterIP 10.96.28.252 8082/TCP 2m13s +druid-with-monitoring-coordinators ClusterIP 10.96.52.186 8081/TCP 2m13s +druid-with-monitoring-pods ClusterIP None 8081/TCP,8090/TCP,8083/TCP,8091/TCP,8082/TCP,8888/TCP 2m13s +druid-with-monitoring-routers ClusterIP 10.96.134.202 8888/TCP 2m13s +druid-with-monitoring-stats ClusterIP 10.96.222.96 56790/TCP 2m13s +``` + +Here, `druid-with-monitoring-stats` service has been created for monitoring purpose. Let's describe the service. + +```bash +$ kubectl describe svc -n druid-demo builtin-prom-stats +Name: druid-with-monitoring-stats +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-with-monitoring + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com + kubedb.com/role=stats +Annotations: monitoring.appscode.com/agent: prometheus.io/builtin + prometheus.io/path: /metrics + prometheus.io/port: 56790 + prometheus.io/scrape: true +Selector: app.kubernetes.io/instance=druid-with-monitoring,app.kubernetes.io/managed-by=kubedb.com,app.kubernetes.io/name=druids.kubedb.com +Type: ClusterIP +IP Family Policy: SingleStack +IP Families: IPv4 +IP: 10.96.222.96 +IPs: 10.96.222.96 +Port: metrics 56790/TCP +TargetPort: metrics/TCP +Endpoints: 10.244.0.31:56790,10.244.0.33:56790 +Session Affinity: None +Events: +``` + +You can see that the service contains following annotations. + +```bash +prometheus.io/path: /metrics +prometheus.io/port: 56790 +prometheus.io/scrape: true +``` + +The Prometheus server will discover the service endpoint using these specifications and will scrape metrics from the exporter. + +## Configure Prometheus Server + +Now, we have to configure a Prometheus scraping job to scrape the metrics using this service. We are going to configure scraping job similar to this [kubernetes-service-endpoints](https://github.com/appscode/third-party-tools/tree/master/monitoring/prometheus/builtin#kubernetes-service-endpoints) job that scrapes metrics from endpoints of a service. + +Let's configure a Prometheus scraping job to collect metrics from this service. + +```yaml +- job_name: 'kubedb-databases' + honor_labels: true + scheme: http + kubernetes_sd_configs: + - role: endpoints + # by default Prometheus server select all Kubernetes services as possible target. + # relabel_config is used to filter only desired endpoints + relabel_configs: + # keep only those services that has "prometheus.io/scrape","prometheus.io/path" and "prometheus.io/port" anootations + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape, __meta_kubernetes_service_annotation_prometheus_io_port] + separator: ; + regex: true;(.*) + action: keep + # currently KubeDB supported databases uses only "http" scheme to export metrics. so, drop any service that uses "https" scheme. + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] + action: drop + regex: https + # only keep the stats services created by KubeDB for monitoring purpose which has "-stats" suffix + - source_labels: [__meta_kubernetes_service_name] + separator: ; + regex: (.*-stats) + action: keep + # service created by KubeDB will have "app.kubernetes.io/name" and "app.kubernetes.io/instance" annotations. keep only those services that have these annotations. + - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name] + separator: ; + regex: (.*) + action: keep + # read the metric path from "prometheus.io/path: " annotation + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] + action: replace + target_label: __metrics_path__ + regex: (.+) + # read the port from "prometheus.io/port: " annotation and update scraping address accordingly + - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] + action: replace + target_label: __address__ + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + # add service namespace as label to the scraped metrics + - source_labels: [__meta_kubernetes_namespace] + separator: ; + regex: (.*) + target_label: namespace + replacement: $1 + action: replace + # add service name as a label to the scraped metrics + - source_labels: [__meta_kubernetes_service_name] + separator: ; + regex: (.*) + target_label: service + replacement: $1 + action: replace + # add stats service's labels to the scraped metrics + - action: labelmap + regex: __meta_kubernetes_service_label_(.+) +``` + +### Configure Existing Prometheus Server + +If you already have a Prometheus server running, you have to add above scraping job in the `ConfigMap` used to configure the Prometheus server. Then, you have to restart it for the updated configuration to take effect. + +>If you don't use a persistent volume for Prometheus storage, you will lose your previously scraped data on restart. + +### Deploy New Prometheus Server + +If you don't have any existing Prometheus server running, you have to deploy one. In this section, we are going to deploy a Prometheus server in `monitoring` namespace to collect metrics using this stats service. + +**Create ConfigMap:** + +At first, create a ConfigMap with the scraping configuration. Bellow, the YAML of ConfigMap that we are going to create in this tutorial. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config + labels: + app: prometheus-demo + namespace: monitoring +data: + prometheus.yml: |- + global: + scrape_interval: 5s + evaluation_interval: 5s + scrape_configs: + - job_name: 'kubedb-databases' + honor_labels: true + scheme: http + kubernetes_sd_configs: + - role: endpoints + # by default Prometheus server select all Kubernetes services as possible target. + # relabel_config is used to filter only desired endpoints + relabel_configs: + # keep only those services that has "prometheus.io/scrape","prometheus.io/path" and "prometheus.io/port" anootations + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape, __meta_kubernetes_service_annotation_prometheus_io_port] + separator: ; + regex: true;(.*) + action: keep + # currently KubeDB supported databases uses only "http" scheme to export metrics. so, drop any service that uses "https" scheme. + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] + action: drop + regex: https + # only keep the stats services created by KubeDB for monitoring purpose which has "-stats" suffix + - source_labels: [__meta_kubernetes_service_name] + separator: ; + regex: (.*-stats) + action: keep + # service created by KubeDB will have "app.kubernetes.io/name" and "app.kubernetes.io/instance" annotations. keep only those services that have these annotations. + - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name] + separator: ; + regex: (.*) + action: keep + # read the metric path from "prometheus.io/path: " annotation + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] + action: replace + target_label: __metrics_path__ + regex: (.+) + # read the port from "prometheus.io/port: " annotation and update scraping address accordingly + - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] + action: replace + target_label: __address__ + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + # add service namespace as label to the scraped metrics + - source_labels: [__meta_kubernetes_namespace] + separator: ; + regex: (.*) + target_label: namespace + replacement: $1 + action: replace + # add service name as a label to the scraped metrics + - source_labels: [__meta_kubernetes_service_name] + separator: ; + regex: (.*) + target_label: service + replacement: $1 + action: replace + # add stats service's labels to the scraped metrics + - action: labelmap + regex: __meta_kubernetes_service_label_(.+) +``` + +Let's create above `ConfigMap`, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/monitoring/builtin-prometheus/prom-config.yaml +configmap/prometheus-config created +``` + +**Create RBAC:** + +If you are using an RBAC enabled cluster, you have to give necessary RBAC permissions for Prometheus. Let's create necessary RBAC stuffs for Prometheus, + +```bash +$ kubectl apply -f https://github.com/appscode/third-party-tools/raw/master/monitoring/prometheus/builtin/artifacts/rbac.yaml +clusterrole.rbac.authorization.k8s.io/prometheus created +serviceaccount/prometheus created +clusterrolebinding.rbac.authorization.k8s.io/prometheus created +``` + +>YAML for the RBAC resources created above can be found [here](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/builtin/artifacts/rbac.yaml). + +**Deploy Prometheus:** + +Now, we are ready to deploy Prometheus server. We are going to use following [deployment](https://github.com/appscode/third-party-tools/blob/master/monitoring/prometheus/builtin/artifacts/deployment.yaml) to deploy Prometheus server. + +Let's deploy the Prometheus server. + +```bash +$ kubectl apply -f https://github.com/appscode/third-party-tools/raw/master/monitoring/prometheus/builtin/artifacts/deployment.yaml +deployment.apps/prometheus created +``` + +### Verify Monitoring Metrics + +Prometheus server is listening to port `9090`. We are going to use [port forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to access Prometheus dashboard. + +At first, let's check if the Prometheus pod is in `Running` state. + +```bash +$ kubectl get pod -n monitoring -l=app=prometheus +NAME READY STATUS RESTARTS AGE +prometheus-7bd56c6865-8dlpv 1/1 Running 0 28s +``` + +Now, run following command on a separate terminal to forward 9090 port of `prometheus-7bd56c6865-8dlpv` pod, + +```bash +$ kubectl port-forward -n monitoring prometheus-7bd56c6865-8dlpv 9090 +Forwarding from 127.0.0.1:9090 -> 9090 +Forwarding from [::1]:9090 -> 9090 +``` + +Now, we can access the dashboard at `localhost:9090`. Open [http://localhost:9090](http://localhost:9090) in your browser. You should see the endpoint of `druid-with-monitoring-stats` service as one of the targets. + +

+  Prometheus Target +

+ +Check the labels marked with red rectangle. These labels confirm that the metrics are coming from `Druid` cluster `druid-with-monitoring` through stats service `druid-with-monitoring-stats`. + +Now, you can view the collected metrics and create a graph from homepage of this Prometheus dashboard. You can also use this Prometheus server as data source for [Grafana](https://grafana.com/) and create beautiful dashboard with collected metrics. + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run following commands + +```bash +kubectl delete -n demo druid/druid-with-monitoring + +kubectl delete -n monitoring deployment.apps/prometheus + +kubectl delete -n monitoring clusterrole.rbac.authorization.k8s.io/prometheus +kubectl delete -n monitoring serviceaccount/prometheus +kubectl delete -n monitoring clusterrolebinding.rbac.authorization.k8s.io/prometheus + +kubectl delete ns demo +kubectl delete ns monitoring +``` + +## Next Steps + +- Learn how to configure [Druid Topology](/docs/guides/druid/clustering/overview/index.md). +- Monitor your Druid database with KubeDB using [`out-of-the-box` Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/monitoring/using-prometheus-operator.md b/docs/guides/druid/monitoring/using-prometheus-operator.md new file mode 100644 index 0000000000..abd4bb1ee6 --- /dev/null +++ b/docs/guides/druid/monitoring/using-prometheus-operator.md @@ -0,0 +1,343 @@ +--- +title: Monitor Druid using Prometheus Operator +menu: + docs_{{ .version }}: + identifier: guides-druid-monitoring-operator-monitoring + name: Prometheus Operator + parent: guides-druid-monitoring + weight: 15 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Monitoring Druid Using Prometheus operator + +[Prometheus operator](https://github.com/prometheus-operator/prometheus-operator) provides simple and Kubernetes native way to deploy and configure Prometheus server. This tutorial will show you how to use Prometheus operator to monitor Druid database deployed with KubeDB. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one locally by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- To learn how Prometheus monitoring works with KubeDB in general, please visit [here](/docs/guides/druid/monitoring/overview.md). + +- We need a [Prometheus operator](https://github.com/prometheus-operator/prometheus-operator) instance running. If you don't already have a running instance, you can deploy one using this helm chart [here](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack). + +- To keep Prometheus resources isolated, we are going to use a separate namespace called `monitoring` to deploy the prometheus operator helm chart. Alternatively, you can use `--create-namespace` flag while deploying prometheus. We are going to deploy database in `demo` namespace. + + ```bash + $ kubectl create ns monitoring + namespace/monitoring created + + $ kubectl create ns demo + namespace/demo created + ``` + + + +> Note: YAML files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Find out required labels for ServiceMonitor + +We need to know the labels used to select `ServiceMonitor` by a `Prometheus` crd. We are going to provide these labels in `spec.monitor.prometheus.serviceMonitor.labels` field of Druid crd so that KubeDB creates `ServiceMonitor` object accordingly. + +At first, let's find out the available Prometheus server in our cluster. + +```bash +$ kubectl get prometheus --all-namespaces +NAMESPACE NAME VERSION DESIRED READY RECONCILED AVAILABLE AGE +monitoring prometheus-kube-prometheus-prometheus v2.42.0 1 1 True True 2d23h +``` + +> If you don't have any Prometheus server running in your cluster, deploy one following the guide specified in **Before You Begin** section. + +Now, let's view the YAML of the available Prometheus server `prometheus` in `monitoring` namespace. + +```bash +$ kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus -o yaml +apiVersion: monitoring.coreos.com/v1 +kind: Prometheus +metadata: + annotations: + meta.helm.sh/release-name: prometheus + meta.helm.sh/release-namespace: monitoring + creationTimestamp: "2023-03-27T07:56:04Z" + generation: 1 + labels: + app: kube-prometheus-stack-prometheus + app.kubernetes.io/instance: prometheus + app.kubernetes.io/managed-by: Helm + app.kubernetes.io/part-of: kube-prometheus-stack + app.kubernetes.io/version: 45.7.1 + chart: kube-prometheus-stack-45.7.1 + heritage: Helm + release: prometheus + name: prometheus-kube-prometheus-prometheus + namespace: monitoring + resourceVersion: "638797" + uid: 0d1e7b8a-44ae-4794-ab45-95a5d7ae7f91 +spec: + alerting: + alertmanagers: + - apiVersion: v2 + name: prometheus-kube-prometheus-alertmanager + namespace: monitoring + pathPrefix: / + port: http-web + enableAdminAPI: false + evaluationInterval: 30s + externalUrl: http://prometheus-kube-prometheus-prometheus.monitoring:9090 + hostNetwork: false + image: quay.io/prometheus/prometheus:v2.42.0 + listenLocal: false + logFormat: logfmt + logLevel: info + paused: false + podMonitorNamespaceSelector: {} + podMonitorSelector: + matchLabels: + release: prometheus + portName: http-web + probeNamespaceSelector: {} + probeSelector: + matchLabels: + release: prometheus + replicas: 1 + retention: 10d + routePrefix: / + ruleNamespaceSelector: {} + ruleSelector: + matchLabels: + release: prometheus + scrapeInterval: 30s + securityContext: + fsGroup: 2000 + runAsGroup: 2000 + runAsNonRoot: true + runAsUser: 1000 + serviceAccountName: prometheus-kube-prometheus-prometheus + serviceMonitorNamespaceSelector: {} + serviceMonitorSelector: + matchLabels: + release: prometheus + shards: 1 + version: v2.42.0 + walCompression: true +status: + availableReplicas: 1 + conditions: + - lastTransitionTime: "2023-03-27T07:56:23Z" + observedGeneration: 1 + status: "True" + type: Available + - lastTransitionTime: "2023-03-30T03:39:18Z" + observedGeneration: 1 + status: "True" + type: Reconciled + paused: false + replicas: 1 + shardStatuses: + - availableReplicas: 1 + replicas: 1 + shardID: "0" + unavailableReplicas: 0 + updatedReplicas: 1 + unavailableReplicas: 0 + updatedReplicas: 1 +``` + +Notice the `spec.serviceMonitorSelector` section. Here, `release: prometheus` label is used to select `ServiceMonitor` crd. So, we are going to use this label in `spec.monitor.prometheus.serviceMonitor.labels` field of Druid crd. + +## Deploy Druid with Monitoring Enabled + +At first, let's deploy a Druid database with monitoring enabled. Below is the Druid object that we are going to create. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-with-monitoring + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + monitor: + agent: prometheus.io/operator + prometheus: + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut +``` + +Here, + +- `monitor.agent: prometheus.io/operator` indicates that we are going to monitor this server using Prometheus operator. +- `monitor.prometheus.serviceMonitor.labels` specifies that KubeDB should create `ServiceMonitor` with these labels. +- `monitor.prometheus.interval` indicates that the Prometheus server should scrape metrics from this database with 10 seconds interval. + +Let's create the druid object that we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/druid/monitoring/yamls/druid-with-monirtoring.yaml +druids.kubedb.com/druid-with-monitoring created +``` + +Now, wait for the database to go into `Running` state. + +```bash +$ kubectl get dr -n demo druid +NAME TYPE VERSION STATUS AGE +druid-with-monitoring kubedb.com/v1alpha2 3.6.1 Ready 2m24s +``` + +KubeDB will create a separate stats service with name `{Druid crd name}-stats` for monitoring purpose. + +```bash +$ kubectl get svc -n demo --selector="app.kubernetes.io/instance=druid-with-monitoring" +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +druid-with-monitoring-brokers ClusterIP 10.96.28.252 8082/TCP 2m13s +druid-with-monitoring-coordinators ClusterIP 10.96.52.186 8081/TCP 2m13s +druid-with-monitoring-pods ClusterIP None 8081/TCP,8090/TCP,8083/TCP,8091/TCP,8082/TCP,8888/TCP 2m13s +druid-with-monitoring-routers ClusterIP 10.96.134.202 8888/TCP 2m13s +druid-with-monitoring-stats ClusterIP 10.96.222.96 56790/TCP 2m13s +``` + +Here, `druid-with-monitoring-stats` service has been created for monitoring purpose. + +Let's describe this stats service. + +```bash +$ kubectl describe svc -n demo druid-with-monitoring-stats +Name: druid-with-monitoring-stats +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-with-monitoring + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com + kubedb.com/role=stats +Annotations: monitoring.appscode.com/agent: prometheus.io/operator +Selector: app.kubernetes.io/instance=druid-with-monitoring,app.kubernetes.io/managed-by=kubedb.com,app.kubernetes.io/name=druids.kubedb.com +Type: ClusterIP +IP Family Policy: SingleStack +IP Families: IPv4 +IP: 10.96.29.174 +IPs: 10.96.29.174 +Port: metrics 9104/TCP +TargetPort: metrics/TCP +Endpoints: 10.244.0.68:9104,10.244.0.71:9104,10.244.0.72:9104 + 2 more... +Session Affinity: None +Events: +``` + +Notice the `Labels` and `Port` fields. `ServiceMonitor` will use this information to target its endpoints. + +KubeDB will also create a `ServiceMonitor` crd in `demo` namespace that select the endpoints of `druid-with-monitoring-stats` service. Verify that the `ServiceMonitor` crd has been created. + +```bash +$ kubectl get servicemonitor -n demo +NAME AGE +druid-with-monitoring-stats 4m49s +``` + +Let's verify that the `ServiceMonitor` has the label that we had specified in `spec.monitor` section of Druid crd. + +```bash +$ kubectl get servicemonitor -n demo druid-with-monitoring-stats -o yaml +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + creationTimestamp: "2024-11-01T10:25:14Z" + generation: 1 + labels: + app.kubernetes.io/component: database + app.kubernetes.io/instance: druid-with-monitoring + app.kubernetes.io/managed-by: kubedb.com + app.kubernetes.io/name: druids.kubedb.com + release: prometheus + name: druid-with-monitoring-stats + namespace: demo + ownerReferences: + - apiVersion: v1 + blockOwnerDeletion: true + controller: true + kind: Service + name: druid-with-monitoring-stats + uid: b3ae48f3-476e-4cec-95f6-f8e28538b605 + resourceVersion: "597152" + uid: ff385538-eba5-48a3-91c1-1a4b15f3018a +spec: + endpoints: + - honorLabels: true + interval: 10s + path: /metrics + port: metrics + namespaceSelector: + matchNames: + - demo + selector: + matchLabels: + app.kubernetes.io/component: database + app.kubernetes.io/instance: druid-with-monitoring + app.kubernetes.io/managed-by: kubedb.com + app.kubernetes.io/name: druids.kubedb.com + kubedb.com/role: stats +``` + +Notice that the `ServiceMonitor` has label `release: prometheus` that we had specified in Druid crd. + +Also notice that the `ServiceMonitor` has selector which match the labels we have seen in the `druid-with-monitoring-stats` service. It also, target the `metrics` port that we have seen in the stats service. + +## Verify Monitoring Metrics + +At first, let's find out the respective Prometheus pod for `prometheus` Prometheus server. + +```bash +$ kubectl get pod -n monitoring -l=app.kubernetes.io/name=prometheus +NAME READY STATUS RESTARTS AGE +prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 8 (4h27m ago) 3d +``` + +Prometheus server is listening to port `9090` of `prometheus-prometheus-kube-prometheus-prometheus-0` pod. We are going to use [port forwarding](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) to access Prometheus dashboard. + +Run following command on a separate terminal to forward the port 9090 of `prometheus-kube-prometheus-prometheus` service which is pointing to the prometheus pod, + +```bash +$ kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090 +Forwarding from 127.0.0.1:9090 -> 9090 +Forwarding from [::1]:9090 -> 9090 +``` + +Now, we can access the dashboard at `localhost:9090`. Open [http://localhost:9090](http://localhost:9090) in your browser. You should see `metrics` endpoint of `druid-with-monitoring-stats` service as one of the targets. + +

+  Prometheus Target +

+ +Check the `endpoint` and `service` labels. It verifies that the target is our expected database. Now, you can view the collected metrics and create a graph from homepage of this Prometheus dashboard. You can also use this Prometheus server as data source for [Grafana](https://grafana.com/) and create a beautiful dashboard with collected metrics. + +## Cleaning up + +To clean up the Kubernetes resources created by this tutorial, run following commands + +```bash +kubectl delete -n demo dr/druid-with-monitoring +kubectl delete ns demo +``` + +## Next Steps + +- Learn how to use KubeDB to run Apache Druid cluster [here](/docs/guides/druid/README.md). +- Deploy [dedicated cluster](/docs/guides/druid/clustering/overview/index.md) for Apache Druid +[//]: # (- Deploy [combined cluster](/docs/guides/druid/clustering/combined-cluster/index.md) for Apache Druid) +- Detail concepts of [DruidVersion object](/docs/guides/druid/concepts/druidversion.md). +[//]: # (- Learn to use KubeDB managed Druid objects using [CLIs](/docs/guides/druid/cli/cli.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). \ No newline at end of file diff --git a/docs/guides/druid/monitoring/yamls/deep-storage-config.yaml b/docs/guides/druid/monitoring/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/monitoring/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/monitoring/yamls/druid-monitoring-builtin.yaml b/docs/guides/druid/monitoring/yamls/druid-monitoring-builtin.yaml new file mode 100644 index 0000000000..4962c3c536 --- /dev/null +++ b/docs/guides/druid/monitoring/yamls/druid-monitoring-builtin.yaml @@ -0,0 +1,24 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-with-monitoring + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + monitor: + agent: prometheus.io/builtin + prometheus: + exporter: + port: 56790 + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut diff --git a/docs/guides/druid/monitoring/yamls/druid-with-monitoring.yaml b/docs/guides/druid/monitoring/yamls/druid-with-monitoring.yaml new file mode 100644 index 0000000000..aa91054f8f --- /dev/null +++ b/docs/guides/druid/monitoring/yamls/druid-with-monitoring.yaml @@ -0,0 +1,23 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-with-monitoring + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + monitor: + agent: prometheus.io/operator + prometheus: + serviceMonitor: + labels: + release: prometheus + interval: 10s + deletionPolicy: WipeOut + diff --git a/docs/guides/druid/quickstart/_index.md b/docs/guides/druid/quickstart/_index.md index c99d5aad28..1be2a3045b 100644 --- a/docs/guides/druid/quickstart/_index.md +++ b/docs/guides/druid/quickstart/_index.md @@ -5,6 +5,6 @@ menu: identifier: guides-druid-quickstart name: Quickstart parent: guides-druid - weight: 15 + weight: 10 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/druid/quickstart/overview/index.md b/docs/guides/druid/quickstart/guide/index.md similarity index 97% rename from docs/guides/druid/quickstart/overview/index.md rename to docs/guides/druid/quickstart/guide/index.md index ee080604c8..4318742d55 100644 --- a/docs/guides/druid/quickstart/overview/index.md +++ b/docs/guides/druid/quickstart/guide/index.md @@ -2,8 +2,8 @@ title: Druid Quickstart menu: docs_{{ .version }}: - identifier: guides-druid-quickstart-overview - name: Overview + identifier: guides-druid-quickstart-guide + name: Druid Quickstart parent: guides-druid-quickstart weight: 10 menu_name: docs_{{ .version }} @@ -24,7 +24,7 @@ This tutorial will show you how to use KubeDB to run an [Apache Druid](https://d At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). -Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure install with helm command including the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency. +Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md) and make sure to include the flags `--set global.featureGates.Druid=true` to ensure **Druid CRD** and `--set global.featureGates.ZooKeeper=true` to ensure **ZooKeeper CRD** as Druid depends on ZooKeeper for external dependency with helm command. To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. @@ -39,7 +39,7 @@ demo Active 9s > Note: YAML files used in this tutorial are stored in [guides/druid/quickstart/overview/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/druid/quickstart/overview/yamls) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). -> We have designed this tutorial to demonstrate a production setup of KubeDB managed Apache Druid. If you just want to try out KubeDB, you can bypass some safety features following the tips [here](/docs/guides/druid/quickstart/overview/index.md#tips-for-testing). +> We have designed this tutorial to demonstrate a production setup of KubeDB managed Apache Druid. If you just want to try out KubeDB, you can bypass some safety features following the tips [here](/docs/guides/druid/quickstart/guide/index.md#tips-for-testing). ## Find Available StorageClass @@ -55,7 +55,7 @@ Here, we have `standard` StorageClass in our cluster from [Local Path Provisione ## Find Available DruidVersion -When you install the KubeDB operator, it registers a CRD named [DruidVersion](/docs/guides/druid/concepts/catalog.md). The installation process comes with a set of tested DruidVersion objects. Let's check available DruidVersions by, +When you install the KubeDB operator, it registers a CRD named [DruidVersion](/docs/guides/druid/concepts/druidversion.md). The installation process comes with a set of tested DruidVersion objects. Let's check available DruidVersions by, ```bash $ kubectl get druidversion @@ -194,7 +194,7 @@ Here, Let's create the Druid CR that is shown above: ```bash -$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/druid/quickstart/druid-quickstart.yaml +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/druid/quickstart/druid-with-monitoring.yaml druid.kubedb.com/druid-quickstart created ``` diff --git a/docs/guides/druid/reconfigure-tls/_index.md b/docs/guides/druid/reconfigure-tls/_index.md new file mode 100644 index 0000000000..f82762bb13 --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/_index.md @@ -0,0 +1,10 @@ +--- +title: Reconfigure TLS/SSL +menu: + docs_{{ .version }}: + identifier: guides-druid-reconfigure-tls + name: Reconfigure TLS/SSL + parent: guides-druid + weight: 120 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/reconfigure-tls/guide.md b/docs/guides/druid/reconfigure-tls/guide.md new file mode 100644 index 0000000000..5cd0080298 --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/guide.md @@ -0,0 +1,1539 @@ +--- +title: Reconfigure Druid TLS/SSL Encryption +menu: + docs_{{ .version }}: + identifier: guides-druid-reconfigure-tls-guide + name: Reconfigure Druid TLS/SSL Encryption + parent: guides-druid-reconfigure-tls + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Reconfigure Druid TLS/SSL (Transport Encryption) + +KubeDB supports reconfigure i.e. add, remove, update and rotation of TLS/SSL certificates for existing Druid database via a DruidOpsRequest. This tutorial will show you how to use KubeDB to reconfigure TLS/SSL encryption. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install [`cert-manger`](https://cert-manager.io/docs/installation/) v1.0.0 or later to your cluster to manage your SSL/TLS certificates. + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + + ```bash + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Add TLS to a Druid database + +Here, We are going to create a Druid without TLS and then reconfigure the database to use TLS. + +### Deploy Druid without TLS + +In this section, we are going to deploy a Druid topology cluster without TLS. In the next few sections we will reconfigure TLS using `DruidOpsRequest` CRD. Below is the YAML of the `Druid` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +Let's create the `Druid` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure-tls/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait until `druid-cluster` has status `Ready`. i.e, + +```bash +$ kubectl get dr -n demo -w +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 15s +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 37s +. +. +druid-cluster kubedb.com/v1alpha2 28.0.1 Ready 2m27s +``` + +Now, we can exec one druid broker pod and verify configuration that the TLS is disabled. + +```bash +$ kubectl exec -it -n demo druid-cluster-coordinators-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ cat conf/druid/cluster/_common/common.runtime.properties +druid.auth.authenticator.basic.authorizerName=basic +druid.auth.authenticator.basic.credentialsValidator.type=metadata +druid.auth.authenticator.basic.initialAdminPassword={"type": "environment", "variable": "DRUID_ADMIN_PASSWORD"} +druid.auth.authenticator.basic.initialInternalClientPassword=***** +druid.auth.authenticator.basic.skipOnFailure=false +druid.auth.authenticator.basic.type=basic +druid.auth.authenticatorChain=["basic"] +druid.auth.authorizer.basic.type=basic +druid.auth.authorizers=["basic"] +druid.emitter.logging.logLevel=info +druid.emitter=noop +druid.escalator.authorizerName=basic +druid.escalator.internalClientPassword=****** +druid.escalator.internalClientUsername=druid_system +druid.escalator.type=basic +druid.expressions.useStrictBooleans=true +druid.extensions.loadList=["druid-avro-extensions", "druid-kafka-indexing-service", "druid-kafka-indexing-service", "druid-datasketches", "druid-multi-stage-query", "druid-basic-security", "mysql-metadata-storage", "druid-s3-extensions"] +druid.global.http.eagerInitialization=false +druid.host=localhost +druid.indexer.logs.directory=var/druid/indexing-logs +druid.indexer.logs.type=file +druid.indexing.doubleStorage=double +druid.lookup.enableLookupSyncOnStartup=false +druid.metadata.storage.connector.connectURI=jdbc:mysql://druid-cluster-mysql-metadata.demo.svc:3306/druid +druid.metadata.storage.connector.createTables=true +druid.metadata.storage.connector.host=localhost +druid.metadata.storage.connector.password={"type": "environment", "variable": "DRUID_METADATA_STORAGE_PASSWORD"} +druid.metadata.storage.connector.port=1527 +druid.metadata.storage.connector.user=root +druid.metadata.storage.type=mysql +druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.ServiceStatusMonitor"] +druid.s3.accessKey=minio +druid.s3.enablePathStyleAccess=true +druid.s3.endpoint.signingRegion=us-east-1 +druid.s3.endpoint.url=http://myminio-hl.demo.svc.cluster.local:9000/ +druid.s3.protocol=http +druid.s3.secretKey=minio123 +druid.selectors.coordinator.serviceName=druid/coordinator +druid.selectors.indexing.serviceName=druid/overlord +druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"] +druid.sql.enable=true +druid.sql.planner.useGroupingSetForExactDistinct=true +druid.startup.logging.logProperties=true +druid.storage.baseKey=druid/segments +druid.storage.bucket=druid +druid.storage.storageDirectory=var/druid/segments +druid.storage.type=s3 +druid.zk.paths.base=/druid +druid.zk.service.host=druid-cluster-zk.demo.svc:2181 +druid.zk.service.pwd={"type": "environment", "variable": "DRUID_ZK_SERVICE_PASSWORD"} +druid.zk.service.user=super +``` + +We can verify from the above output that TLS is disabled for this cluster as there is no TLS/SSL related configs provided for it. + +#### Verify TLS/SSL is disabled using Druid UI + +First port-forward the port `8888` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-routers 8888 +Forwarding from 127.0.0.1:8888 -> 8888 +Forwarding from [::1]:8888 -> 8888 +``` + + +Now hit the `http://localhost:8888` from any browser, and you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  lifecycle +

+ +From the above screenshot, we can see that the connection is not secure now. In other words, TLS/SSL is disabled for this druid cluster. + +### Create Issuer/ ClusterIssuer + +Now, We are going to create an example `Issuer` that will be used to enable SSL/TLS in Druid. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificates using openssl. + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=ca/O=kubedb" +Generating a RSA private key +................+++++ +........................+++++ +writing new private key to './ca.key' +----- +``` + +- Now we are going to create a ca-secret using the certificate files that we have just generated. + +```bash +$ kubectl create secret tls druid-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/druid-ca created +``` + +Now, Let's create an `Issuer` using the `druid-ca` secret that we have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: druid-ca-issuer + namespace: demo +spec: + ca: + secretName: druid-ca +``` + +Let's apply the `YAML` file: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure-tls/yamls/druid-issuer.yaml +issuer.cert-manager.io/druid-ca-issuer created +``` + +### Create DruidOpsRequest + +In order to add TLS to the druid, we have to create a `DruidOpsRequest` CRO with our created issuer. Below is the YAML of the `DruidOpsRequest` CRO that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-add-tls + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + issuerRef: + name: druid-ca-issuer + kind: Issuer + apiGroup: "cert-manager.io" + certificates: + - alias: client + subject: + organizations: + - druid + organizationalUnits: + - client + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing reconfigure TLS operation on `druid-cluster` cluster. +- `spec.type` specifies that we are performing `ReconfigureTLS` on druid. +- `spec.tls.issuerRef` specifies the issuer name, kind and api group. +- `spec.tls.certificates` specifies the certificates. You can learn more about this field from [here](/docs/guides/druid/concepts/druid.md#spectls). + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure-tls/yamls/drops-add-tls.yaml +druidopsrequest.ops.kubedb.com/drops-add-tls created +``` + +#### Verify TLS Enabled Successfully + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CRO, + +```bash +$ kubectl get drops -n demo -w +NAME TYPE STATUS AGE +drops-add-tls ReconfigureTLS Progressing 39s +drops-add-tls ReconfigureTLS Progressing 44s +... +... +drops-add-tls ReconfigureTLS Successful 79s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed. + +```bash +$ kubectl describe druidopsrequest -n demo drops-add-tls +Name: drops-add-tls +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-28T09:43:13Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:timeout: + f:tls: + .: + f:certificates: + f:issuerRef: + f:type: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-28T09:43:13Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-28T09:44:32Z + Resource Version: 409889 + UID: b7f563c4-4773-49e9-aba2-17497e66f5f8 +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Timeout: 5m + Tls: + Certificates: + Alias: client + Subject: + Organizational Units: + client + Organizations: + druid + Issuer Ref: + API Group: cert-manager.io + Kind: Issuer + Name: druid-ca-issuer + Type: ReconfigureTLS +Status: + Conditions: + Last Transition Time: 2024-10-28T09:43:13Z + Message: Druid ops-request has started to reconfigure tls for druid nodes + Observed Generation: 1 + Reason: ReconfigureTLS + Status: True + Type: ReconfigureTLS + Last Transition Time: 2024-10-28T09:43:26Z + Message: Successfully synced all certificates + Observed Generation: 1 + Reason: CertificateSynced + Status: True + Type: CertificateSynced + Last Transition Time: 2024-10-28T09:43:21Z + Message: get certificate; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetCertificate + Last Transition Time: 2024-10-28T09:43:21Z + Message: check ready condition; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CheckReadyCondition + Last Transition Time: 2024-10-28T09:43:21Z + Message: issuing condition; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IssuingCondition + Last Transition Time: 2024-10-28T09:43:31Z + Message: successfully reconciled the Druid with tls configuration + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-28T09:44:32Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2024-10-28T09:43:37Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T09:43:37Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T09:43:47Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T09:43:52Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T09:43:52Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T09:43:57Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T09:44:02Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T09:44:02Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T09:44:07Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T09:44:12Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T09:44:12Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T09:44:17Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T09:44:22Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T09:44:22Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T09:44:27Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T09:44:32Z + Message: Successfully completed reconfigureTLS for druid. + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 103s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-add-tls + Normal Starting 103s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 103s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-add-tls + Warning get certificate; ConditionStatus:True 95s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 95s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 95s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Warning get certificate; ConditionStatus:True 95s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 95s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 95s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Normal CertificateSynced 95s KubeDB Ops-manager Operator Successfully synced all certificates + Warning get certificate; ConditionStatus:True 90s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 90s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 90s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Warning get certificate; ConditionStatus:True 90s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 90s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 90s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Normal CertificateSynced 90s KubeDB Ops-manager Operator Successfully synced all certificates + Normal UpdatePetSets 85s KubeDB Ops-manager Operator successfully reconciled the Druid with tls configuration + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 79s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 79s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:False; PodName:druid-cluster-historicals-0 74s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 69s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 64s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 64s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 59s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 54s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 54s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 49s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 44s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 44s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 39s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 34s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 34s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 29s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Normal RestartNodes 24s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Starting 24s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 24s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-add-tls +``` + +Now, Lets exec into a druid coordinators pod and verify the configuration that the TLS is enabled. + +```bash +$ kubectl exec -it -n demo druid-cluster-coordinators-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ cat conf/druid/cluster/_common/common.runtime.properties +druid.auth.authenticator.basic.authorizerName=basic +druid.auth.authenticator.basic.credentialsValidator.type=metadata +druid.auth.authenticator.basic.initialAdminPassword={"type": "environment", "variable": "DRUID_ADMIN_PASSWORD"} +druid.auth.authenticator.basic.initialInternalClientPassword=password2 +druid.auth.authenticator.basic.skipOnFailure=false +druid.auth.authenticator.basic.type=basic +druid.auth.authenticatorChain=["basic"] +druid.auth.authorizer.basic.type=basic +druid.auth.authorizers=["basic"] +druid.client.https.trustStorePassword={"type": "environment", "variable": "DRUID_KEY_STORE_PASSWORD"} +druid.client.https.trustStorePath=/opt/druid/ssl/truststore.jks +druid.client.https.trustStoreType=jks +druid.client.https.validateHostnames=false +druid.emitter.logging.logLevel=info +druid.emitter=noop +druid.enablePlaintextPort=false +druid.enableTlsPort=true +druid.escalator.authorizerName=basic +druid.escalator.internalClientPassword=password2 +druid.escalator.internalClientUsername=druid_system +druid.escalator.type=basic +druid.expressions.useStrictBooleans=true +druid.extensions.loadList=["druid-avro-extensions", "druid-kafka-indexing-service", "druid-kafka-indexing-service", "druid-datasketches", "druid-multi-stage-query", "druid-basic-security", "simple-client-sslcontext", "mysql-metadata-storage", "druid-s3-extensions"] +druid.global.http.eagerInitialization=false +druid.host=localhost +druid.indexer.logs.directory=var/druid/indexing-logs +druid.indexer.logs.type=file +druid.indexing.doubleStorage=double +druid.lookup.enableLookupSyncOnStartup=false +druid.metadata.storage.connector.connectURI=jdbc:mysql://druid-cluster-mysql-metadata.demo.svc:3306/druid +druid.metadata.storage.connector.createTables=true +druid.metadata.storage.connector.host=localhost +druid.metadata.storage.connector.password={"type": "environment", "variable": "DRUID_METADATA_STORAGE_PASSWORD"} +druid.metadata.storage.connector.port=1527 +druid.metadata.storage.connector.user=root +druid.metadata.storage.type=mysql +druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.ServiceStatusMonitor"] +druid.s3.accessKey=minio +druid.s3.enablePathStyleAccess=true +druid.s3.endpoint.signingRegion=us-east-1 +druid.s3.endpoint.url=http://myminio-hl.demo.svc.cluster.local:9000/ +druid.s3.protocol=http +druid.s3.secretKey=minio123 +druid.selectors.coordinator.serviceName=druid/coordinator +druid.selectors.indexing.serviceName=druid/overlord +druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"] +druid.server.https.certAlias=druid +druid.server.https.keyStorePassword={"type": "environment", "variable": "DRUID_KEY_STORE_PASSWORD"} +druid.server.https.keyStorePath=/opt/druid/ssl/keystore.jks +druid.server.https.keyStoreType=jks +druid.sql.enable=true +druid.sql.planner.useGroupingSetForExactDistinct=true +druid.startup.logging.logProperties=true +druid.storage.baseKey=druid/segments +druid.storage.bucket=druid +druid.storage.storageDirectory=var/druid/segments +druid.storage.type=s3 +druid.zk.paths.base=/druid +druid.zk.service.host=druid-cluster-zk.demo.svc:2181 +druid.zk.service.pwd={"type": "environment", "variable": "DRUID_ZK_SERVICE_PASSWORD"} +druid.zk.service.user=super + +``` + +We can see from the output above that all TLS related configs are added in the configuration file of the druid database. + +#### Verify TLS/SSL using Druid UI + +To check follow the following steps: + +Druid uses separate ports for TLS/SSL. While the plaintext port for `routers` node is `8888`. For TLS, it is `9088`. Hence, we will use that port to access the UI. + +First port-forward the port `9088` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-tls-routers 9088 +Forwarding from 127.0.0.1:9088 -> 9088 +Forwarding from [::1]:9088 -> 9088 +``` + + +Now hit the `https://localhost:9088/` from any browser. Here you may select `Advance` and then `Proceed to localhost (unsafe)` or you can add the `ca.crt` from the secret `druid-cluster-tls-client-cert` to your browser's Authorities. + +After that you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-cluster-tls-admin-cred -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-cluster-tls-admin-cred -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  lifecycle +

+ +From the above screenshot, we can see that the connection is secure. + + +## Rotate Certificate + +Now we are going to rotate the certificate of this cluster. First let's check the current expiration date of the certificate. + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-routers 9088 +Forwarding from 127.0.0.1:9088 -> 9088 +Forwarding from [::1]:9088 -> 9088 +Handling connection for 9088 +... + +$ openssl s_client -connect localhost:9088 2>/dev/null | openssl x509 -noout -enddate +notAfter=Jan 26 09:43:16 2025 GMT +``` + +So, the certificate will expire on this time `Jan 26 09:43:16 2025 GMT`. + +### Create DruidOpsRequest + +Now we are going to increase it using a DruidOpsRequest. Below is the yaml of the ops request that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-recon-tls-rotate + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + rotateCertificates: true +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing reconfigure TLS operation on `druid-cluster`. +- `spec.type` specifies that we are performing `ReconfigureTLS` on our cluster. +- `spec.tls.rotateCertificates` specifies that we want to rotate the certificate of this druid cluster. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure-tls/yamls/drops-rotate.yaml +druidopsrequest.ops.kubedb.com/drops-rotate created +``` + +#### Verify Certificate Rotated Successfully + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CRO, + +```bash +$ kubectl get druidopsrequests -n demo drops-rotate -w +NAME TYPE STATUS AGE +drops-rotate ReconfigureTLS Successful 4m4s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed. + +```bash +$ kubectl describe druidopsrequest -n demo drops-rotate +Name: drops-rotate +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-28T14:14:50Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:tls: + .: + f:rotateCertificates: + f:type: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-28T14:14:50Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-28T14:16:04Z + Resource Version: 440897 + UID: ca3532fc-6e11-4962-bddb-f9cf946d3954 +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Tls: + Rotate Certificates: true + Type: ReconfigureTLS +Status: + Conditions: + Last Transition Time: 2024-10-28T14:14:50Z + Message: Druid ops-request has started to reconfigure tls for druid nodes + Observed Generation: 1 + Reason: ReconfigureTLS + Status: True + Type: ReconfigureTLS + Last Transition Time: 2024-10-28T14:15:04Z + Message: Successfully synced all certificates + Observed Generation: 1 + Reason: CertificateSynced + Status: True + Type: CertificateSynced + Last Transition Time: 2024-10-28T14:14:58Z + Message: get certificate; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetCertificate + Last Transition Time: 2024-10-28T14:14:58Z + Message: check ready condition; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CheckReadyCondition + Last Transition Time: 2024-10-28T14:14:58Z + Message: issuing condition; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IssuingCondition + Last Transition Time: 2024-10-28T14:15:09Z + Message: successfully reconciled the Druid with tls configuration + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-28T14:16:04Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2024-10-28T14:15:14Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:15:14Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:15:19Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:15:24Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:15:24Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:15:29Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:15:34Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:15:34Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:15:39Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:15:44Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:15:44Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:15:49Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:15:54Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:15:54Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:15:59Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:16:04Z + Message: Successfully completed reconfigureTLS for druid. + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 101s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-rotate + Normal Starting 101s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 101s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-rotate + Warning get certificate; ConditionStatus:True 93s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 93s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 93s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Warning get certificate; ConditionStatus:True 93s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 93s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 93s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Normal CertificateSynced 93s KubeDB Ops-manager Operator Successfully synced all certificates + Warning get certificate; ConditionStatus:True 88s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 88s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 88s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Warning get certificate; ConditionStatus:True 88s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 88s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 88s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Normal CertificateSynced 87s KubeDB Ops-manager Operator Successfully synced all certificates + Normal UpdatePetSets 82s KubeDB Ops-manager Operator successfully reconciled the Druid with tls configuration + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 77s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 77s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 72s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 67s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 67s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 62s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 57s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 57s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 52s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 47s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 47s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 42s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 37s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 37s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 32s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Normal RestartNodes 27s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Starting 27s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 27s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-rotate +``` + +Now, let's check the expiration date of the certificate. + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-routers 9088 +Forwarding from 127.0.0.1:9088 -> 9088 +Forwarding from [::1]:9088 -> 9088 +Handling connection for 9088 +... + +$ openssl s_client -connect localhost:9088 2>/dev/null | openssl x509 -noout -enddate +notAfter=Jan 26 14:15:46 2025 GMT +``` + +As we can see from the above output, the certificate has been rotated successfully. + +## Change Issuer/ClusterIssuer + +Now, we are going to change the issuer of this database. + +- Let's create a new ca certificate and key using a different subject `CN=ca-update,O=kubedb-updated`. + +```bash +$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=ca-updated/O=kubedb-updated" +Generating a RSA private key +..............................................................+++++ +......................................................................................+++++ +writing new private key to './ca.key' +----- +``` + +- Now we are going to create a new ca-secret using the certificate files that we have just generated. + +```bash +$ kubectl create secret tls druid-new-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/druid-new-ca created +``` + +Now, Let's create a new `Issuer` using the `mongo-new-ca` secret that we have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: dr-new-issuer + namespace: demo +spec: + ca: + secretName: druid-new-ca +``` + +Let's apply the `YAML` file: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure-tls/yamls/druid-new-issuer.yaml +issuer.cert-manager.io/dr-new-issuer created +``` + +### Create DruidOpsRequest + +In order to use the new issuer to issue new certificates, we have to create a `DruidOpsRequest` CRO with the newly created issuer. Below is the YAML of the `DruidOpsRequest` CRO that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-update-issuer + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + issuerRef: + name: dr-new-issuer + kind: Issuer + apiGroup: "cert-manager.io" +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing reconfigure TLS operation on `druid-cluster` cluster. +- `spec.type` specifies that we are performing `ReconfigureTLS` on our druid. +- `spec.tls.issuerRef` specifies the issuer name, kind and api group. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure-tls/yamls/druid-update-tls-issuer.yaml +druidpsrequest.ops.kubedb.com/drops-update-issuer created +``` + +#### Verify Issuer is changed successfully + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CRO, + +```bash +$ kubectl get druidopsrequests -n demo drops-update-issuer -w +NAME TYPE STATUS AGE +drops-update-issuer ReconfigureTLS Progressing 14s +drops-update-issuer ReconfigureTLS Progressing 18s +... +... +drops-update-issuer ReconfigureTLS Successful 73s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed. + +```bash +$ kubectl describe druidopsrequest -n demo drops-update-issuer +Name: drops-update-issuer +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-28T14:24:22Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:tls: + .: + f:issuerRef: + f:type: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-28T14:24:22Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-28T14:25:35Z + Resource Version: 442332 + UID: 5089e358-2dc2-4d62-8c13-92828de7c557 +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Tls: + Issuer Ref: + API Group: cert-manager.io + Kind: Issuer + Name: dr-new-issuer + Type: ReconfigureTLS +Status: + Conditions: + Last Transition Time: 2024-10-28T14:24:22Z + Message: Druid ops-request has started to reconfigure tls for druid nodes + Observed Generation: 1 + Reason: ReconfigureTLS + Status: True + Type: ReconfigureTLS + Last Transition Time: 2024-10-28T14:24:35Z + Message: Successfully synced all certificates + Observed Generation: 1 + Reason: CertificateSynced + Status: True + Type: CertificateSynced + Last Transition Time: 2024-10-28T14:24:30Z + Message: get certificate; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetCertificate + Last Transition Time: 2024-10-28T14:24:30Z + Message: check ready condition; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CheckReadyCondition + Last Transition Time: 2024-10-28T14:24:30Z + Message: issuing condition; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IssuingCondition + Last Transition Time: 2024-10-28T14:24:40Z + Message: successfully reconciled the Druid with tls configuration + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-28T14:25:35Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2024-10-28T14:24:45Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:24:45Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:24:50Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:24:55Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:24:55Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:25:00Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:25:05Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:25:05Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:25:10Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:25:15Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:25:15Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:25:20Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:25:25Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:25:25Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:25:30Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:25:35Z + Message: Successfully completed reconfigureTLS for druid. + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 92s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-update-issuer + Normal Starting 92s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 92s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-update-issuer + Warning get certificate; ConditionStatus:True 84s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 84s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 84s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Warning get certificate; ConditionStatus:True 84s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 84s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 84s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Normal CertificateSynced 84s KubeDB Ops-manager Operator Successfully synced all certificates + Warning get certificate; ConditionStatus:True 79s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 79s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 79s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Warning get certificate; ConditionStatus:True 79s KubeDB Ops-manager Operator get certificate; ConditionStatus:True + Warning check ready condition; ConditionStatus:True 79s KubeDB Ops-manager Operator check ready condition; ConditionStatus:True + Warning issuing condition; ConditionStatus:True 79s KubeDB Ops-manager Operator issuing condition; ConditionStatus:True + Normal CertificateSynced 79s KubeDB Ops-manager Operator Successfully synced all certificates + Normal UpdatePetSets 74s KubeDB Ops-manager Operator successfully reconciled the Druid with tls configuration + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 69s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 69s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 64s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 59s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 59s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 54s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 49s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 49s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 44s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 39s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 39s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 34s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 29s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 29s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 24s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Normal RestartNodes 19s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Starting 19s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 19s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-update-issuer +``` + +Now, Lets exec into a druid node and find out the ca subject to see if it matches the one we have provided. + +```bash +$ kubectl exec -it druid-cluster-broker-0 -- bash +druid@druid-cluster-broker-0:~$ keytool -list -v -keystore /var/private/ssl/server.keystore.jks -storepass wt6f5pwxpg84 | grep 'Issuer' +Issuer: O=kubedb-updated, CN=ca-updated +Issuer: O=kubedb-updated, CN=ca-updated + +$ kubectl port-forward -n demo svc/druid-cluster-routers 9088 +Forwarding from 127.0.0.1:9088 -> 9088 +Forwarding from [::1]:9088 -> 9088 +Handling connection for 9088 +... + +$ openssl s_client -connect localhost:9088 2>/dev/null | openssl x509 -noout -issuer +issuer=CN = ca-updated, O = kubedb-updated +``` + +We can see from the above output that, the subject name matches the subject name of the new ca certificate that we have created. So, the issuer is changed successfully. + +## Remove TLS from the Database + +Now, we are going to remove TLS from this database using a DruidOpsRequest. + +### Create DruidOpsRequest + +Below is the YAML of the `DruidOpsRequest` CRO that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-remove + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + remove: true +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing reconfigure TLS operation on `druid-cluster` cluster. +- `spec.type` specifies that we are performing `ReconfigureTLS` on Druid. +- `spec.tls.remove` specifies that we want to remove tls from this cluster. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure-tls/yamls/drops-remove.yaml +druidopsrequest.ops.kubedb.com/drops-remove created +``` + +#### Verify TLS Removed Successfully + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CRO, + +```bash +$ kubectl get druidopsrequest -n demo drops-remove -w +NAME TYPE STATUS AGE +drops-remove ReconfigureTLS Progressing 25s +drops-remove ReconfigureTLS Progressing 29s +... +... +drops-remove ReconfigureTLS Successful 114s + +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed. + +```bash +$ kubectl describe druidopsrequest -n demo drops-remove +Name: drops-remove +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-28T14:31:07Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:tls: + .: + f:remove: + f:type: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-28T14:31:07Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-28T14:33:01Z + Resource Version: 443725 + UID: 27234241-c72e-471c-8dd4-16fd485956cc +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Tls: + Remove: true + Type: ReconfigureTLS +Status: + Conditions: + Last Transition Time: 2024-10-28T14:31:07Z + Message: Druid ops-request has started to reconfigure tls for druid nodes + Observed Generation: 1 + Reason: ReconfigureTLS + Status: True + Type: ReconfigureTLS + Last Transition Time: 2024-10-28T14:31:16Z + Message: successfully reconciled the Druid with tls configuration + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-28T14:33:01Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2024-10-28T14:31:21Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:31:21Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:31:26Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-historicals-0 + Last Transition Time: 2024-10-28T14:31:31Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:31:31Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:31:36Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-28T14:31:41Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:31:41Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:31:46Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-brokers-0 + Last Transition Time: 2024-10-28T14:31:51Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:31:51Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:31:56Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-routers-0 + Last Transition Time: 2024-10-28T14:32:01Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:32:01Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:32:06Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-28T14:33:01Z + Message: Successfully completed reconfigureTLS for druid. + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 2m12s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/drops-remove + Normal Starting 2m12s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 2m12s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: drops-remove + Normal UpdatePetSets 2m3s KubeDB Ops-manager Operator successfully reconciled the Druid with tls configuration + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 118s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 118s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 113s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 108s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 108s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 103s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 98s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 98s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 93s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 88s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 88s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 83s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 78s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 78s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 73s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 68s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 68s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 63s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 58s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 58s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 53s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 48s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 48s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 43s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 38s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 38s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 33s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 28s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 28s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 23s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Normal RestartNodes 18s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Starting 18s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 18s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: drops-remove +``` + +Now, Let's exec into one of the broker node and find out that TLS is disabled or not. + +```bash +$$ kubectl exec -it -n demo druid-cluster-broker-0 -- druid-configs.sh --bootstrap-server localhost:9092 --command-config /opt/druid/config/clientauth.properties --describe --entity-type brokers --all | grep 'ssl.keystore' + ssl.keystore.certificate.chain=null sensitive=true synonyms={} + ssl.keystore.key=null sensitive=true synonyms={} + ssl.keystore.location=null sensitive=false synonyms={} + ssl.keystore.password=null sensitive=true synonyms={} + ssl.keystore.type=JKS sensitive=false synonyms={DEFAULT_CONFIG:ssl.keystore.type=JKS} + ssl.keystore.certificate.chain=null sensitive=true synonyms={} + ssl.keystore.key=null sensitive=true synonyms={} + ssl.keystore.location=null sensitive=false synonyms={} + ssl.keystore.password=null sensitive=true synonyms={} +``` + +So, we can see from the above that, output that tls is disabled successfully. + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete opsrequest drops-add-tls drops-remove drops-rotate drops-update-issuer +kubectl delete druid -n demo druid-cluster +kubectl delete issuer -n demo druid-ca-issuer dr-new-issuer +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). + diff --git a/docs/guides/druid/reconfigure-tls/images/druid-ui.png b/docs/guides/druid/reconfigure-tls/images/druid-ui.png new file mode 100644 index 0000000000..f81925c59c Binary files /dev/null and b/docs/guides/druid/reconfigure-tls/images/druid-ui.png differ diff --git a/docs/guides/druid/reconfigure-tls/images/druid-with-tls.png b/docs/guides/druid/reconfigure-tls/images/druid-with-tls.png new file mode 100644 index 0000000000..9f173c38c2 Binary files /dev/null and b/docs/guides/druid/reconfigure-tls/images/druid-with-tls.png differ diff --git a/docs/guides/druid/reconfigure-tls/images/druid-without-tls.png b/docs/guides/druid/reconfigure-tls/images/druid-without-tls.png new file mode 100644 index 0000000000..07aacd32a0 Binary files /dev/null and b/docs/guides/druid/reconfigure-tls/images/druid-without-tls.png differ diff --git a/docs/guides/druid/reconfigure-tls/images/reconfigure-tls.png b/docs/guides/druid/reconfigure-tls/images/reconfigure-tls.png new file mode 100644 index 0000000000..316d1a0aa5 Binary files /dev/null and b/docs/guides/druid/reconfigure-tls/images/reconfigure-tls.png differ diff --git a/docs/guides/druid/reconfigure-tls/overview.md b/docs/guides/druid/reconfigure-tls/overview.md new file mode 100644 index 0000000000..5b55da583f --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/overview.md @@ -0,0 +1,54 @@ +--- +title: Reconfiguring TLS/SSL +menu: + docs_{{ .version }}: + identifier: guides-druid-reconfigure-tls-overview + name: Overview + parent: guides-druid-reconfigure-tls + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Reconfiguring TLS of Druid + +This guide will give an overview on how KubeDB Ops-manager operator reconfigures TLS configuration i.e. add TLS, remove TLS, update issuer/cluster issuer or Certificates and rotate the certificates of `Druid`. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + +## How Reconfiguring Druid TLS Configuration Process Works + +The following diagram shows how KubeDB Ops-manager operator reconfigures TLS of a `Druid`. Open the image in a new tab to see the enlarged version. + +
+  Reconfiguring TLS process of Druid +
Fig: Reconfiguring TLS process of Druid
+
+ +The Reconfiguring Druid TLS process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource Object (CRO). + +2. `KubeDB` Provisioner operator watches the `Druid` CRO. + +3. When the operator finds a `Druid` CR, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Then, in order to reconfigure the TLS configuration of the `Druid` database the user creates a `DruidOpsRequest` CR with desired information. + +5. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CR. + +6. When it finds a `DruidOpsRequest` CR, it pauses the `Druid` object which is referred from the `DruidOpsRequest`. So, the `KubeDB` Provisioner operator doesn't perform any operations on the `Druid` object during the reconfiguring TLS process. + +7. Then the `KubeDB` Ops-manager operator will add, remove, update or rotate TLS configuration based on the Ops Request yaml. + +8. Then the `KubeDB` Ops-manager operator will restart all the Pods of the database so that they restart with the new TLS configuration defined in the `DruidOpsRequest` CR. + +9. After the successful reconfiguring of the `Druid` TLS, the `KubeDB` Ops-manager operator resumes the `Druid` object so that the `KubeDB` Provisioner operator resumes its usual operations. + +In the next docs, we are going to show a step by step guide on reconfiguring TLS configuration of a Druid database using `DruidOpsRequest` CRD. \ No newline at end of file diff --git a/docs/guides/druid/reconfigure-tls/yamls/deep-storage-config.yaml b/docs/guides/druid/reconfigure-tls/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/reconfigure-tls/yamls/drops-add-tls.yaml b/docs/guides/druid/reconfigure-tls/yamls/drops-add-tls.yaml new file mode 100644 index 0000000000..dd3654967b --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/drops-add-tls.yaml @@ -0,0 +1,23 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-add-tls + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + issuerRef: + name: druid-ca-issuer + kind: Issuer + apiGroup: "cert-manager.io" + certificates: + - alias: client + subject: + organizations: + - druid + organizationalUnits: + - client + timeout: 5m + apply: IfReady diff --git a/docs/guides/druid/reconfigure-tls/yamls/drops-remove.yaml b/docs/guides/druid/reconfigure-tls/yamls/drops-remove.yaml new file mode 100644 index 0000000000..af42b8d00d --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/drops-remove.yaml @@ -0,0 +1,11 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-remove + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + remove: true diff --git a/docs/guides/druid/reconfigure-tls/yamls/drops-rotate.yaml b/docs/guides/druid/reconfigure-tls/yamls/drops-rotate.yaml new file mode 100644 index 0000000000..f0be918f6f --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/drops-rotate.yaml @@ -0,0 +1,11 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-recon-tls-rotate + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + rotateCertificates: true diff --git a/docs/guides/druid/reconfigure-tls/yamls/druid-ca-issuer.yaml b/docs/guides/druid/reconfigure-tls/yamls/druid-ca-issuer.yaml new file mode 100644 index 0000000000..d6298c972c --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/druid-ca-issuer.yaml @@ -0,0 +1,8 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: druid-ca-issuer + namespace: demo +spec: + ca: + secretName: druid-ca diff --git a/docs/guides/druid/reconfigure-tls/yamls/druid-cluster.yaml b/docs/guides/druid/reconfigure-tls/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..6351c2ddda --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/druid-cluster.yaml @@ -0,0 +1,16 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete + diff --git a/docs/guides/druid/reconfigure-tls/yamls/druid-new-issuer.yaml b/docs/guides/druid/reconfigure-tls/yamls/druid-new-issuer.yaml new file mode 100644 index 0000000000..ede5d5177c --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/druid-new-issuer.yaml @@ -0,0 +1,8 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: dr-new-issuer + namespace: demo +spec: + ca: + secretName: druid-new-ca \ No newline at end of file diff --git a/docs/guides/druid/reconfigure-tls/yamls/druid-update-tls-issuer.yaml b/docs/guides/druid/reconfigure-tls/yamls/druid-update-tls-issuer.yaml new file mode 100644 index 0000000000..e876f4c3b8 --- /dev/null +++ b/docs/guides/druid/reconfigure-tls/yamls/druid-update-tls-issuer.yaml @@ -0,0 +1,14 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: drops-update-issuer + namespace: demo +spec: + type: ReconfigureTLS + databaseRef: + name: druid-cluster + tls: + issuerRef: + name: dr-new-issuer + kind: Issuer + apiGroup: "cert-manager.io" diff --git a/docs/guides/druid/reconfigure/_index.md b/docs/guides/druid/reconfigure/_index.md new file mode 100644 index 0000000000..4c3cfdfe58 --- /dev/null +++ b/docs/guides/druid/reconfigure/_index.md @@ -0,0 +1,10 @@ +--- +title: Reconfigure +menu: + docs_{{ .version }}: + identifier: guides-druid-reconfigure + name: Reconfigure + parent: guides-druid + weight: 110 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/druid/reconfigure/guide.md b/docs/guides/druid/reconfigure/guide.md new file mode 100644 index 0000000000..cf85960a9c --- /dev/null +++ b/docs/guides/druid/reconfigure/guide.md @@ -0,0 +1,704 @@ +--- +title: Reconfigure Druid Topology +menu: + docs_{{ .version }}: + identifier: guides-druid-reconfigure-guide + name: Reconfigure Druid + parent: guides-druid-reconfigure + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Reconfigure Druid Cluster + +This guide will show you how to use `KubeDB` Ops-manager operator to reconfigure a Druid Topology cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. + +- Install `KubeDB` Provisioner and Ops-manager operator in your cluster following the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [Topology](/docs/guides/druid/clustering/overview/index.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Reconfigure Overview](/docs/guides/druid/reconfigure/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [/docs/guides/druid/reconfigure/yamls](/docs/guides/druid/reconfigure/yamls) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +Now, we are going to deploy a `Druid` cluster using a supported version by `KubeDB` operator. Then we are going to apply `DruidOpsRequest` to reconfigure its configuration. + +### Prepare Druid Cluster + +Now, we are going to deploy a `Druid` topology cluster with version `28.0.1`. + +#### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/restart/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +Now, lets go ahead and create a druid database. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +Let's create the `Druid` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/update-version/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +### Reconfigure using config secret + +Say we want to change the default maximum number of tasks the MiddleManager can accept. Let's create the `middleManagers.properties` file with our desire configurations. + +**middleManagers.properties:** + +```properties +druid.worker.capacity=5 +``` + +**historicals.properties:** + +```properties +druid.processing.numThreads=3 +``` + +Then, we will create a new secret with this configuration file. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: config-secret + namespace: demo +stringData: + middleManagers.properties: |- + druid.worker.capacity=5 + historicals.properties: |- + druid.processing.numThreads=3 +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/update-version/yamls/config-secret.yaml +secret/new-config created +``` + +### Check Current Configuration + +Before creating the druidOpsRequest, first +Lets exec into one of the druid middleManagers pod that we have created and check the default configuration: + +Exec into the Druid middleManagers: + +```bash +$ kubectl exec -it -n demo druid-cluster-middleManagers-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ +``` + +Now, execute the following commands to see the configurations: +```bash +bash-5.1$ cat conf/druid/cluster/data/middleManager/runtime.properties | grep druid.worker.capacity +druid.worker.capacity=2 +``` +Here, we can see that our given configuration is applied to the Druid cluster for all brokers. + +Now, lets exec into one of the druid historicals pod that we have created and check the configurations are applied or not: + +Exec into the Druid historicals: + +```bash +$ kubectl exec -it -n demo druid-cluster-historicals-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ +``` + +Now, execute the following commands to see the metadata storage directory: +```bash +bash-5.1$ cat conf/druid/cluster/data/historical/runtime.properties | grep druid.processing.numThreads +druid.processing.numThreads=2 +``` + +Here, we can see that our given configuration is applied to the historicals. + +### Check Configuration from Druid UI + +You can also see the configuration changes from the druid ui. For that, follow the following steps: + +First port-forward the port `8888` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-routers 8888 +Forwarding from 127.0.0.1:8888 -> 8888 +Forwarding from [::1]:8888 -> 8888 +``` + + +Now hit the `http://localhost:8888` from any browser, and you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-cluster-auth -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-cluster-auth -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  lifecycle +

+ +You can see that there are 2 task slots reflecting with the configuration `druid.worker.capacity=2`. + + +#### Create DruidOpsRequest + +Now, we will use this secret to replace the previous secret using a `DruidOpsRequest` CR. The `DruidOpsRequest` yaml is given below, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: reconfigure-drops + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: druid-cluster + configuration: + configSecret: + name: new-config +``` + +Here, + +- `spec.databaseRef.name` specifies that we are reconfiguring `druid-prod` database. +- `spec.type` specifies that we are performing `Reconfigure` on our database. +- `spec.configSecret.name` specifies the name of the new secret. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/reconfigure/yamls/reconfigure-druid-ops.yaml +druidopsrequest.ops.kubedb.com/reconfigure-drops created +``` + +#### Check new configuration + +If everything goes well, `KubeDB` Ops-manager operator will update the `configSecret` of `Druid` object. + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CR, + +```bash +$ kubectl get druidopsrequests -n demo +NAME TYPE STATUS AGE +reconfigure-drops Reconfigure Successful 4m55s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to reconfigure the database. + +```bash +$ kubectl describe druidopsrequest -n demo reconfigure-drops +Name: reconfigure-drops +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-08-02T05:08:37Z + Generation: 1 + Resource Version: 332491 + UID: b6e8cb1b-d29f-445e-bb01-60d29012c7eb +Spec: + Apply: IfReady + Configuration: + Config Secret: + Name: new-kf-topology-custom-config + Database Ref: + Name: druid-prod + Timeout: 5m + Type: Reconfigure +Status: + Conditions: + Last Transition Time: 2024-08-02T05:08:37Z + Message: Druid ops-request has started to reconfigure druid nodes + Observed Generation: 1 + Reason: Reconfigure + Status: True + Type: Reconfigure + Last Transition Time: 2024-08-02T05:08:45Z + Message: check reconcile; ConditionStatus:False + Observed Generation: 1 + Status: False + Type: CheckReconcile + Last Transition Time: 2024-08-02T05:09:42Z + Message: successfully reconciled the Druid with new configure + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-08-02T05:09:47Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-historicals-0 + Last Transition Time: 2024-08-02T05:09:47Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-historicals-0 + Last Transition Time: 2024-08-02T05:10:02Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-historicals-0 + Last Transition Time: 2024-08-02T05:10:07Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-historicals-1 + Last Transition Time: 2024-08-02T05:10:07Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-historicals-1 + Last Transition Time: 2024-08-02T05:10:22Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-historicals-1 + Last Transition Time: 2024-08-02T05:10:27Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-middleManagers-0 + Last Transition Time: 2024-08-02T05:10:27Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-middleManagers-0 + Last Transition Time: 2024-08-02T05:11:12Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-middleManagers-0 + Last Transition Time: 2024-08-02T05:11:17Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-middleManagers-1 + Last Transition Time: 2024-08-02T05:11:17Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-middleManagers-1 + Last Transition Time: 2024-08-02T05:11:32Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-middleManagers-1 + Last Transition Time: 2024-08-02T05:11:37Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2024-08-02T05:11:39Z + Message: Successfully completed reconfigure druid + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 3m7s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/reconfigure-drops + Normal Starting 3m7s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-prod + Normal Successful 3m7s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-prod for DruidOpsRequest: reconfigure-drops + Warning check reconcile; ConditionStatus:False 2m59s KubeDB Ops-manager Operator check reconcile; ConditionStatus:False + Normal UpdatePetSets 2m2s KubeDB Ops-manager Operator successfully reconciled the Druid with new configure + Warning get pod; ConditionStatus:True; PodName:druid-prod-historicals-0 117s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0 117s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0 112s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0 102s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-prod-historicals-1 97s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1 97s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1 92s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1 82s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1 + Warning get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 77s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 77s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0 72s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0 32s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 27s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 27s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1 22s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1 12s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Normal RestartNodes 7s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Starting 5s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-prod + Normal Successful 5s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-prod for DruidOpsRequest: reconfigure-drops +``` + +Now let's exec one of the instance and run a druid-configs.sh command to check the new configuration we have provided. + +```bash +$ kubectl exec -it -n demo druid-prod-middleManagers-0 -- druid-configs.sh --bootstrap-server localhost:9092 --command-config /opt/druid/config/clientauth.properties --describe --entity-type middleManagerss --all | grep 'log.retention.hours' + log.retention.hours=125 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.hours=125, DEFAULT_CONFIG:log.retention.hours=168} + log.retention.hours=125 sensitive=false synonyms={STATIC_BROKER_CONFIG:log.retention.hours=125, DEFAULT_CONFIG:log.retention.hours=168} +``` + +As we can see from the configuration of ready druid, the value of `log.retention.hours` has been changed from `100` to `125`. So the reconfiguration of the cluster is successful. + + +### Reconfigure using apply config + +Now we will reconfigure this cluster again to set `log.retention.hours` to `150`. This time we won't use a new secret. We will use the `applyConfig` field of the `DruidOpsRequest`. This will merge the new config in the existing secret. + +#### Create DruidOpsRequest + +Now, we will use the new configuration in the `applyConfig` field in the `DruidOpsRequest` CR. The `DruidOpsRequest` yaml is given below, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: kfops-reconfigure-apply-topology + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: druid-prod + configuration: + applyConfig: + middleManagers.properties: |- + log.retention.hours=150 + historicals.properties: |- + historicals.quorum.election.timeout.ms=4000 + historicals.quorum.fetch.timeout.ms=5000 + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are reconfiguring `druid-prod` cluster. +- `spec.type` specifies that we are performing `Reconfigure` on druid. +- `spec.configuration.applyConfig` specifies the new configuration that will be merged in the existing secret. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/druid/reconfigure/druid-reconfigure-apply-topology.yaml +druidopsrequest.ops.kubedb.com/kfops-reconfigure-apply-topology created +``` + +#### Verify new configuration + +If everything goes well, `KubeDB` Ops-manager operator will merge this new config with the existing configuration. + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CR, + +```bash +$ kubectl get druidopsrequests -n demo kfops-reconfigure-apply-topology +NAME TYPE STATUS AGE +kfops-reconfigure-apply-topology Reconfigure Successful 55s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to reconfigure the cluster. + +```bash +$ kubectl describe druidopsrequest -n demo kfops-reconfigure-apply-topology +Name: kfops-reconfigure-apply-topology +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-08-02T05:14:42Z + Generation: 1 + Resource Version: 332996 + UID: 551d2c92-9431-47a7-a699-8f8115131b49 +Spec: + Apply: IfReady + Configuration: + Apply Config: + middleManagers.properties: log.retention.hours=150 + historicals.properties: historicals.quorum.election.timeout.ms=4000 +historicals.quorum.fetch.timeout.ms=5000 + Database Ref: + Name: druid-prod + Timeout: 5m + Type: Reconfigure +Status: + Conditions: + Last Transition Time: 2024-08-02T05:14:42Z + Message: Druid ops-request has started to reconfigure druid nodes + Observed Generation: 1 + Reason: Reconfigure + Status: True + Type: Reconfigure + Last Transition Time: 2024-08-02T05:14:45Z + Message: Successfully prepared user provided custom config secret + Observed Generation: 1 + Reason: PrepareCustomConfig + Status: True + Type: PrepareCustomConfig + Last Transition Time: 2024-08-02T05:14:52Z + Message: successfully reconciled the Druid with new configure + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-08-02T05:14:57Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-historicals-0 + Last Transition Time: 2024-08-02T05:14:57Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-historicals-0 + Last Transition Time: 2024-08-02T05:15:07Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-historicals-0 + Last Transition Time: 2024-08-02T05:15:12Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-historicals-1 + Last Transition Time: 2024-08-02T05:15:12Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-historicals-1 + Last Transition Time: 2024-08-02T05:15:27Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-historicals-1 + Last Transition Time: 2024-08-02T05:15:32Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-middleManagers-0 + Last Transition Time: 2024-08-02T05:15:32Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-middleManagers-0 + Last Transition Time: 2024-08-02T05:16:07Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-middleManagers-0 + Last Transition Time: 2024-08-02T05:16:12Z + Message: get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Observed Generation: 1 + Status: True + Type: GetPod--druid-prod-middleManagers-1 + Last Transition Time: 2024-08-02T05:16:12Z + Message: evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-prod-middleManagers-1 + Last Transition Time: 2024-08-02T05:16:27Z + Message: check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-prod-middleManagers-1 + Last Transition Time: 2024-08-02T05:16:32Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2024-08-02T05:16:35Z + Message: Successfully completed reconfigure druid + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 2m6s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/kfops-reconfigure-apply-topology + Normal Starting 2m6s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-prod + Normal Successful 2m6s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-prod for DruidOpsRequest: kfops-reconfigure-apply-topology + Normal UpdatePetSets 116s KubeDB Ops-manager Operator successfully reconciled the Druid with new configure + Warning get pod; ConditionStatus:True; PodName:druid-prod-historicals-0 111s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0 111s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-historicals-0 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0 106s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0 101s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-prod-historicals-1 96s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1 96s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-historicals-1 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1 91s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-historicals-1 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1 81s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-historicals-1 + Warning get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 76s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 76s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0 71s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0 41s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 36s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Warning evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 36s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Warning check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1 31s KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-prod-middleManagers-1 + Warning check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1 21s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-prod-middleManagers-1 + Normal RestartNodes 15s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Starting 14s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-prod + Normal Successful 14s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-prod for DruidOpsRequest: kfops-reconfigure-apply-topology +``` + +Lets exec into one of the druid middleManagers pod that have updated and check the new configurations are applied or not: + +Exec into the Druid middleManagers: + +```bash +$ kubectl exec -it -n demo druid-with-config-middleManagers-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ +``` + +Now, execute the following commands to see the configurations: +```bash +bash-5.1$ cat conf/druid/cluster/data/middleManager/runtime.properties | grep druid.worker.capacity +druid.worker.capacity=5 +``` +Here, we can see that our given configuration is applied to the Druid cluster for all brokers. + +Now, lets exec into one of the druid historicals pod that have updated and check the new configurations are applied or not: + +Exec into the Druid historicals: + +```bash +$ kubectl exec -it -n demo druid-with-config-historicals-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ +``` + +Now, execute the following commands to see the metadata storage directory: +```bash +bash-5.1$ cat conf/druid/cluster/data/historical/runtime.properties | grep druid.processing.numThreads +druid.processing.numThreads=3 +``` + +Here, we can see that our given configuration is applied to the historicals. + +### Verify Configuration Change from Druid UI + +You can access the UI similarly by doing port-forward as mentioned in [Check Configuration from Druid UI](/docs/guides/druid/reconfigure/#CheckConfigurationfromDruidUI) + +You should be able to see the following changes in the UI: + +

+  lifecycle +

+ +You can see that there are 5 task slots reflecting with our provided custom configuration of `druid.worker.capacity=5`. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete kf -n demo druid-cluster +kubectl delete druidopsrequest -n demo reconfigure-drops +kubectl delete secret -n demo new-config +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/reconfigure/images/druid-ui.png b/docs/guides/druid/reconfigure/images/druid-ui.png new file mode 100644 index 0000000000..af798ee7b4 Binary files /dev/null and b/docs/guides/druid/reconfigure/images/druid-ui.png differ diff --git a/docs/guides/druid/reconfigure/images/reconfigure.svg b/docs/guides/druid/reconfigure/images/reconfigure.svg new file mode 100644 index 0000000000..84526d2735 --- /dev/null +++ b/docs/guides/druid/reconfigure/images/reconfigure.svg @@ -0,0 +1,120 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/guides/druid/reconfigure/overview.md b/docs/guides/druid/reconfigure/overview.md new file mode 100644 index 0000000000..e2ea2b268e --- /dev/null +++ b/docs/guides/druid/reconfigure/overview.md @@ -0,0 +1,54 @@ +--- +title: Reconfiguring Druid +menu: + docs_{{ .version }}: + identifier: guides-druid-reconfigure-overview + name: Overview + parent: guides-druid-reconfigure + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Reconfiguring Druid + +This guide will give an overview on how KubeDB Ops-manager operator reconfigures `Druid` components such as Combined, Broker, Controller, etc. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/kafka/concepts/kafka.md) + - [DruidOpsRequest](/docs/guides/kafka/concepts/kafkaopsrequest.md) + +## How Reconfiguring Druid Process Works + +The following diagram shows how KubeDB Ops-manager operator reconfigures `Druid` components. Open the image in a new tab to see the enlarged version. + +
+  Reconfiguring process of Druid +
Fig: Reconfiguring process of Druid
+
+ +The Reconfiguring Druid process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource (CR). + +2. `KubeDB` Provisioner operator watches the `Druid` CR. + +3. When the operator finds a `Druid` CR, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Then, in order to reconfigure the various components (ie. Coordinators, Overlords, Historicals, MiddleManagers, Brokers, Routers) of the `Druid`, the user creates a `DruidOpsRequest` CR with desired information. + +5. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CR. + +6. When it finds a `DruidOpsRequest` CR, it halts the `Druid` object which is referred from the `DruidOpsRequest`. So, the `KubeDB` Provisioner operator doesn't perform any operations on the `Druid` object during the reconfiguring process. + +7. Then the `KubeDB` Ops-manager operator will replace the existing configuration with the new configuration provided or merge the new configuration with the existing configuration according to the `MogoDBOpsRequest` CR. + +8. Then the `KubeDB` Ops-manager operator will restart the related PetSet Pods so that they restart with the new configuration defined in the `DruidOpsRequest` CR. + +9. After the successful reconfiguring of the `Druid` components, the `KubeDB` Ops-manager operator resumes the `Druid` object so that the `KubeDB` Provisioner operator resumes its usual operations. + +In the next docs, we are going to show a step-by-step guide on reconfiguring Druid components using `DruidOpsRequest` CRD. \ No newline at end of file diff --git a/docs/guides/druid/reconfigure/yamls/config-secret.yaml b/docs/guides/druid/reconfigure/yamls/config-secret.yaml new file mode 100644 index 0000000000..6067ee7dd2 --- /dev/null +++ b/docs/guides/druid/reconfigure/yamls/config-secret.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Secret +metadata: + name: new-config + namespace: demo +stringData: + middleManagers.properties: |- + druid.worker.capacity=5 + historicals.properties: |- + druid.processing.numThreads=3 diff --git a/docs/guides/druid/reconfigure/yamls/deep-storage-config.yaml b/docs/guides/druid/reconfigure/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/reconfigure/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/reconfigure/yamls/druid-cluster.yaml b/docs/guides/druid/reconfigure/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..f7a695b062 --- /dev/null +++ b/docs/guides/druid/reconfigure/yamls/druid-cluster.yaml @@ -0,0 +1,15 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: WipeOut diff --git a/docs/guides/druid/reconfigure/yamls/reconfigure-druid-ops.yaml b/docs/guides/druid/reconfigure/yamls/reconfigure-druid-ops.yaml new file mode 100644 index 0000000000..cc5f789a54 --- /dev/null +++ b/docs/guides/druid/reconfigure/yamls/reconfigure-druid-ops.yaml @@ -0,0 +1,12 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: reconfigure-drops + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: druid-cluster + configuration: + configSecret: + name: new-config \ No newline at end of file diff --git a/docs/guides/druid/restart/_index.md b/docs/guides/druid/restart/_index.md new file mode 100644 index 0000000000..7d23da5218 --- /dev/null +++ b/docs/guides/druid/restart/_index.md @@ -0,0 +1,10 @@ +--- +title: Restart Druid +menu: + docs_{{ .version }}: + identifier: guides-druid-restart + name: Restart + parent: guides-druid + weight: 130 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/druid/restart/guide.md b/docs/guides/druid/restart/guide.md new file mode 100644 index 0000000000..b172825913 --- /dev/null +++ b/docs/guides/druid/restart/guide.md @@ -0,0 +1,283 @@ +--- +title: Restart Druid +menu: + docs_{{ .version }}: + identifier: guides-druid-restart-guide + name: Restart Druid + parent: guides-druid-restart + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Restart Druid + +KubeDB supports restarting the Druid database via a DruidOpsRequest. Restarting is useful if some pods are got stuck in some phase, or they are not working correctly. This tutorial will show you how to use that. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + +```bash + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Deploy Druid + +In this section, we are going to deploy a Druid database using KubeDB. + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/restart/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +Now, lets go ahead and create a druid database. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +Let's create the `Druid` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/update-version/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +## Apply Restart opsRequest + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: restart + namespace: demo +spec: + type: Restart + databaseRef: + name: druid-cluster + timeout: 5m + apply: Always +``` + +- `spec.type` specifies the Type of the ops Request +- `spec.databaseRef` holds the name of the Druid CR. It should be available in the same namespace as the opsRequest +- The meaning of `spec.timeout` & `spec.apply` fields will be found [here](/docs/guides/druid/concepts/druidopsrequest.md#spectimeout) + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/restart/restart.yaml +druidopsrequest.ops.kubedb.com/restart created +``` + +Now the Ops-manager operator will first restart the controller pods, then broker of the referenced druid. + +```shell +$ kubectl get drops -n demo +NAME TYPE STATUS AGE +restart Restart Successful 2m11s + +$ kubectl get drops -n demo restart -oyaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + annotations: + kubectl.kubernetes.io/last-applied-configuration: | + {"apiVersion":"ops.kubedb.com/v1alpha1","kind":"DruidOpsRequest","metadata":{"annotations":{},"name":"restart","namespace":"demo"},"spec":{"apply":"Always","databaseRef":{"name":"druid-cluster"},"timeout":"5m","type":"Restart"}} + creationTimestamp: "2024-10-21T10:30:53Z" + generation: 1 + name: restart + namespace: demo + resourceVersion: "83200" + uid: 0fcbc7d4-593f-45f7-8631-7483805efe1e +spec: + apply: Always + databaseRef: + name: druid-cluster + timeout: 5m + type: Restart +status: + conditions: + - lastTransitionTime: "2024-10-21T10:30:53Z" + message: Druid ops-request has started to restart druid nodes + observedGeneration: 1 + reason: Restart + status: "True" + type: Restart + - lastTransitionTime: "2024-10-21T10:31:51Z" + message: Successfully Restarted Druid nodes + observedGeneration: 1 + reason: RestartNodes + status: "True" + type: RestartNodes + - lastTransitionTime: "2024-10-21T10:31:01Z" + message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + observedGeneration: 1 + status: "True" + type: GetPod--druid-cluster-historicals-0 + - lastTransitionTime: "2024-10-21T10:31:01Z" + message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + observedGeneration: 1 + status: "True" + type: EvictPod--druid-cluster-historicals-0 + - lastTransitionTime: "2024-10-21T10:31:06Z" + message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + observedGeneration: 1 + status: "True" + type: CheckPodRunning--druid-cluster-historicals-0 + - lastTransitionTime: "2024-10-21T10:31:11Z" + message: get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + observedGeneration: 1 + status: "True" + type: GetPod--druid-cluster-middlemanagers-0 + - lastTransitionTime: "2024-10-21T10:31:11Z" + message: evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + observedGeneration: 1 + status: "True" + type: EvictPod--druid-cluster-middlemanagers-0 + - lastTransitionTime: "2024-10-21T10:31:16Z" + message: check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + observedGeneration: 1 + status: "True" + type: CheckPodRunning--druid-cluster-middlemanagers-0 + - lastTransitionTime: "2024-10-21T10:31:21Z" + message: get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + observedGeneration: 1 + status: "True" + type: GetPod--druid-cluster-brokers-0 + - lastTransitionTime: "2024-10-21T10:31:21Z" + message: evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + observedGeneration: 1 + status: "True" + type: EvictPod--druid-cluster-brokers-0 + - lastTransitionTime: "2024-10-21T10:31:26Z" + message: check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + observedGeneration: 1 + status: "True" + type: CheckPodRunning--druid-cluster-brokers-0 + - lastTransitionTime: "2024-10-21T10:31:31Z" + message: get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + observedGeneration: 1 + status: "True" + type: GetPod--druid-cluster-routers-0 + - lastTransitionTime: "2024-10-21T10:31:31Z" + message: evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + observedGeneration: 1 + status: "True" + type: EvictPod--druid-cluster-routers-0 + - lastTransitionTime: "2024-10-21T10:31:36Z" + message: check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + observedGeneration: 1 + status: "True" + type: CheckPodRunning--druid-cluster-routers-0 + - lastTransitionTime: "2024-10-21T10:31:41Z" + message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + observedGeneration: 1 + status: "True" + type: GetPod--druid-cluster-coordinators-0 + - lastTransitionTime: "2024-10-21T10:31:41Z" + message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + observedGeneration: 1 + status: "True" + type: EvictPod--druid-cluster-coordinators-0 + - lastTransitionTime: "2024-10-21T10:31:46Z" + message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + observedGeneration: 1 + status: "True" + type: CheckPodRunning--druid-cluster-coordinators-0 + - lastTransitionTime: "2024-10-21T10:31:51Z" + message: Controller has successfully restart the Druid replicas + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete druidopsrequest -n demo restart +kubectl delete druid -n demo druid-cluster +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/restart/yamls/deep-storage-config.yaml b/docs/guides/druid/restart/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/restart/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/restart/yamls/druid-cluster.yaml b/docs/guides/druid/restart/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..6351c2ddda --- /dev/null +++ b/docs/guides/druid/restart/yamls/druid-cluster.yaml @@ -0,0 +1,16 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete + diff --git a/docs/guides/druid/restart/yamls/restart.yaml b/docs/guides/druid/restart/yamls/restart.yaml new file mode 100644 index 0000000000..7130c4c865 --- /dev/null +++ b/docs/guides/druid/restart/yamls/restart.yaml @@ -0,0 +1,11 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: restart + namespace: demo +spec: + type: Restart + databaseRef: + name: druid-cluster + timeout: 5m + apply: Always diff --git a/docs/guides/druid/scaling/_index.md b/docs/guides/druid/scaling/_index.md new file mode 100644 index 0000000000..b5da417adc --- /dev/null +++ b/docs/guides/druid/scaling/_index.md @@ -0,0 +1,10 @@ +--- +title: Scaling Druid +menu: + docs_{{ .version }}: + identifier: guides-druid-scaling + name: Scaling + parent: guides-druid + weight: 70 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/druid/scaling/horizontal-scaling/_index.md b/docs/guides/druid/scaling/horizontal-scaling/_index.md new file mode 100644 index 0000000000..73d3017f6d --- /dev/null +++ b/docs/guides/druid/scaling/horizontal-scaling/_index.md @@ -0,0 +1,10 @@ +--- +title: Horizontal Scaling +menu: + docs_{{ .version }}: + identifier: guides-druid-scaling-horizontal-scaling + name: Horizontal Scaling + parent: guides-druid-scaling + weight: 10 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/scaling/horizontal-scaling/guide.md b/docs/guides/druid/scaling/horizontal-scaling/guide.md new file mode 100644 index 0000000000..6e6648b4f8 --- /dev/null +++ b/docs/guides/druid/scaling/horizontal-scaling/guide.md @@ -0,0 +1,603 @@ +--- +title: Horizontal Scaling Druid Cluster +menu: + docs_{{ .version }}: + identifier: guides-druid-scaling-horizontal-scaling-guide + name: Druid Horizontal Scaling + parent: guides-druid-scaling-horizontal-scaling + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Horizontal Scale Druid Topology Cluster + +This guide will show you how to use `KubeDB` Ops-manager operator to scale the Druid topology cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` Provisioner and Ops-manager operator in your cluster following the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [Topology](/docs/guides/druid/clustering/overview/index.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Horizontal Scaling Overview](/docs/guides/druid/scaling/horizontal-scaling/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/examples/druid](/docs/examples/druid) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +## Apply Horizontal Scaling on Druid Cluster + +Here, we are going to deploy a `Druid` cluster using a supported version by `KubeDB` operator. Then we are going to apply horizontal scaling on it. + +### Prepare Druid Topology cluster + +Now, we are going to deploy a `Druid` topology cluster with version `28.0.1`. + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/scaling/horizontal-scaling/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +### Deploy Druid topology cluster + +In this section, we are going to deploy a Druid topology cluster. Then, in the next section we will scale the cluster using `DruidOpsRequest` CRD. Below is the YAML of the `Druid` CR that we are going to create, + + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +Let's create the `Druid` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/druid/scaling/horizontal-scaling/yamls/druid-topology.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait until `druid-cluster` has status `Ready`. i.e, + +```bash +$ kubectl get dr -n demo -w +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1aplha2 28.0.1 Provisioning 0s +druid-cluster kubedb.com/v1aplha2 28.0.1 Provisioning 24s +. +. +druid-cluster kubedb.com/v1aplha2 28.0.1 Ready 92s +``` + +Let's check the number of replicas has from druid object, number of pods the petset have, + +**Coordinators Replicas** + +```bash +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.coordinators.replicas' +1 + +$ kubectl get petset -n demo druid-cluster-coordinators -o json | jq '.spec.replicas' +1 +``` + +**Historicals Replicas** + +```bash +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.historicals.replicas' +1 + +$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.replicas' +1 +``` + +We can see from commands that the cluster has 1 replicas for both coordinators and historicals. + +### Check Replica Count from Druid UI + +You can also see the replica count of each node from the druid ui. For that, follow the following steps: + +First port-forward the port `8888` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-routers 8888 +Forwarding from 127.0.0.1:8888 -> 8888 +Forwarding from [::1]:8888 -> 8888 +``` + + +Now hit the `http://localhost:8888` from any browser, and you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  lifecycle +

+ + +Here, we can see that there is 1 replica of each node including `coordinators` and `historicals`. + +We are now ready to apply the `DruidOpsRequest` CR to scale this cluster. + +## Scale Up Replicas + +Here, we are going to scale up the replicas of the topology cluster to meet the desired number of replicas after scaling. + +### Create DruidOpsRequest + +In order to scale up the replicas of the topology cluster, we have to create a `DruidOpsRequest` CR with our desired replicas. Below is the YAML of the `DruidOpsRequest` CR that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-hscale-up + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: druid-cluster + horizontalScaling: + topology: + coordinators: 2 + historicals: 2 +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing horizontal scaling operation on `druid-cluster` cluster. +- `spec.type` specifies that we are performing `HorizontalScaling` on druid. +- `spec.horizontalScaling.topology.coordinators` specifies the desired replicas after scaling for coordinators. +- `spec.horizontalScaling.topology.historicals` specifies the desired replicas after scaling for historicals. + +> **Note:** Similarly you can scale other druid nodes horizontally by specifying the following fields: + > - For `overlords` use `spec.horizontalScaling.topology.overlords`. + > - For `brokers` use `spec.horizontalScaling.topology.brokers`. + > - For `middleManagers` use `spec.horizontalScaling.topology.middleManagers`. + > - For `routers` use `spec.horizontalScaling.topology.routers`. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-hscale-up.yaml +druidopsrequest.ops.kubedb.com/druid-hscale-up created +``` + +### Verify Topology cluster replicas scaled up successfully + +If everything goes well, `KubeDB` Ops-manager operator will update the replicas of `Druid` object and related `PetSets` and `Pods`. + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CR, + +```bash +$ watch kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +druid-hscale-up HorizontalScaling Successful 106s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to scale the cluster. + +```bash +$ kubectl describe druidopsrequests -n demo druid-hscale-up +Name: druid-hscale-up +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-21T11:32:51Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:horizontalScaling: + .: + f:topology: + .: + f:coordinators: + f:historicals: + f:type: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-21T11:32:51Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-21T11:34:02Z + Resource Version: 91877 + UID: 824356ca-eafc-4266-8af1-c372b27f6ce7 +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Horizontal Scaling: + Topology: + Coordinators: 2 + Historicals: 2 + Type: HorizontalScaling +Status: + Conditions: + Last Transition Time: 2024-10-21T11:32:51Z + Message: Druid ops-request has started to horizontally scaling the nodes + Observed Generation: 1 + Reason: HorizontalScaling + Status: True + Type: HorizontalScaling + Last Transition Time: 2024-10-21T11:33:17Z + Message: Successfully Scaled Up Broker + Observed Generation: 1 + Reason: ScaleUpCoordinators + Status: True + Type: ScaleUpCoordinators + Last Transition Time: 2024-10-21T11:33:02Z + Message: patch pet setdruid-cluster-coordinators; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: PatchPetSetdruid-cluster-coordinators + Last Transition Time: 2024-10-21T11:33:57Z + Message: node in cluster; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: NodeInCluster + Last Transition Time: 2024-10-21T11:34:02Z + Message: Successfully Scaled Up Broker + Observed Generation: 1 + Reason: ScaleUpHistoricals + Status: True + Type: ScaleUpHistoricals + Last Transition Time: 2024-10-21T11:33:22Z + Message: patch pet setdruid-cluster-historicals; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: PatchPetSetdruid-cluster-historicals + Last Transition Time: 2024-10-21T11:34:02Z + Message: Successfully completed horizontally scale druid cluster + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 95s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/druid-hscale-up + Normal Starting 95s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 95s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: druid-hscale-up + Warning patch pet setdruid-cluster-coordinators; ConditionStatus:True 84s KubeDB Ops-manager Operator patch pet setdruid-cluster-coordinators; ConditionStatus:True + Warning node in cluster; ConditionStatus:False 76s KubeDB Ops-manager Operator node in cluster; ConditionStatus:False + Warning node in cluster; ConditionStatus:True 74s KubeDB Ops-manager Operator node in cluster; ConditionStatus:True + Normal ScaleUpCoordinators 69s KubeDB Ops-manager Operator Successfully Scaled Up Broker + Warning patch pet setdruid-cluster-historicals; ConditionStatus:True 64s KubeDB Ops-manager Operator patch pet setdruid-cluster-historicals; ConditionStatus:True + Warning node in cluster; ConditionStatus:False 56s KubeDB Ops-manager Operator node in cluster; ConditionStatus:False + Warning node in cluster; ConditionStatus:True 29s KubeDB Ops-manager Operator node in cluster; ConditionStatus:True + Normal ScaleUpHistoricals 24s KubeDB Ops-manager Operator Successfully Scaled Up Broker + Normal Starting 24s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 24s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: druid-hscale-up +``` + + +Now, we are going to verify the number of replicas this cluster has from the Druid object, number of pods the petset have, + +**Coordinators Replicas** + +```bash +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.coordinators.replicas' +2 + +$ kubectl get petset -n demo druid-cluster-coordinators -o json | jq '.spec.replicas' +2 +``` + +**Historicals Replicas** + +```bash +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.historicals.replicas' +2 + +$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.replicas' +2 +``` + +Now, we are going to verify the number of replicas this cluster has from the Druid UI. + +### Verify Replica Count from Druid UI + +Verify the scaled replica count of nodes from the druid ui. To access the UI follow the steps described in the first part of this guide. [(Check Replica Count from Druid UI)](/docs/guides/druid/scaling/horizontal-scaling/#Check-Replica-Count-from-Druid-UI) + +If you follow the steps properly, you should be able to see that the replica count of both `coordinators` and `historicals` has become 2. Also as the `coordinators` is serving as the `overlords`, the count of `overlords` has also become 2. + +

+  lifecycle +

+ +## Scale Down Replicas + +Here, we are going to scale down the replicas of the druid topology cluster to meet the desired number of replicas after scaling. + +### Create DruidOpsRequest + +In order to scale down the replicas of the druid topology cluster, we have to create a `DruidOpsRequest` CR with our desired replicas. Below is the YAML of the `DruidOpsRequest` CR that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-hscale-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: druid-cluster + horizontalScaling: + topology: + coordinators: 1 + historicals: 1 +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing horizontal scaling down operation on `druid-cluster` cluster. +- `spec.type` specifies that we are performing `HorizontalScaling` on druid. +- `spec.horizontalScaling.topology.coordinators` specifies the desired replicas after scaling for the coordinators nodes. +- `spec.horizontalScaling.topology.historicals` specifies the desired replicas after scaling for the historicals nodes. + +> **Note:** Similarly you can scale other druid nodes by specifying the following fields: +> - For `overlords` use `spec.horizontalScaling.topology.overlords`. +> - For `brokers` use `spec.horizontalScaling.topology.brokers`. +> - For `middleManagers` use `spec.horizontalScaling.topology.middleManagers`. +> - For `routers` use `spec.horizontalScaling.topology.routers`. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/druid/scaling/horizontal-scaling/druid-hscale-down-topology.yaml +druidopsrequest.ops.kubedb.com/druid-hscale-down created +``` + +#### Verify Topology cluster replicas scaled down successfully + +If everything goes well, `KubeDB` Ops-manager operator will update the replicas of `Druid` object and related `PetSets` and `Pods`. + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CR, + +```bash +$ watch kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +druid-hscale-down HorizontalScaling Successful 2m32s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to scale the cluster. + +```bash +$ kubectl get druidopsrequest -n demo druid-hscale-down -oyaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + annotations: + kubectl.kubernetes.io/last-applied-configuration: | + {"apiVersion":"ops.kubedb.com/v1alpha1","kind":"DruidOpsRequest","metadata":{"annotations":{},"name":"druid-hscale-down","namespace":"demo"},"spec":{"databaseRef":{"name":"druid-cluster"},"horizontalScaling":{"topology":{"coordinators":1,"historicals":1}},"type":"HorizontalScaling"}} + creationTimestamp: "2024-10-21T12:42:09Z" + generation: 1 + name: druid-hscale-down + namespace: demo + resourceVersion: "99500" + uid: b3a81d07-be44-4adf-a8a7-36bb825f26a8 +spec: + apply: IfReady + databaseRef: + name: druid-cluster + horizontalScaling: + topology: + coordinators: 1 + historicals: 1 + type: HorizontalScaling +status: + conditions: + - lastTransitionTime: "2024-10-21T12:42:09Z" + message: Druid ops-request has started to horizontally scaling the nodes + observedGeneration: 1 + reason: HorizontalScaling + status: "True" + type: HorizontalScaling + - lastTransitionTime: "2024-10-21T12:42:33Z" + message: Successfully Scaled Down Broker + observedGeneration: 1 + reason: ScaleDownCoordinators + status: "True" + type: ScaleDownCoordinators + - lastTransitionTime: "2024-10-21T12:42:23Z" + message: reassign partitions; ConditionStatus:True + observedGeneration: 1 + status: "True" + type: ReassignPartitions + - lastTransitionTime: "2024-10-21T12:42:23Z" + message: is pet set patched; ConditionStatus:True + observedGeneration: 1 + status: "True" + type: IsPetSetPatched + - lastTransitionTime: "2024-10-21T12:42:28Z" + message: get pod; ConditionStatus:True + observedGeneration: 1 + status: "True" + type: GetPod + - lastTransitionTime: "2024-10-21T12:42:53Z" + message: Successfully Scaled Down Broker + observedGeneration: 1 + reason: ScaleDownHistoricals + status: "True" + type: ScaleDownHistoricals + - lastTransitionTime: "2024-10-21T12:42:43Z" + message: delete pvc; ConditionStatus:True + observedGeneration: 1 + status: "True" + type: DeletePvc + - lastTransitionTime: "2024-10-21T12:42:43Z" + message: get pvc; ConditionStatus:False + observedGeneration: 1 + status: "False" + type: GetPvc + - lastTransitionTime: "2024-10-21T12:42:53Z" + message: Successfully completed horizontally scale druid cluster + observedGeneration: 1 + reason: Successful + status: "True" + type: Successful + observedGeneration: 1 + phase: Successful +``` + +Now, we are going to verify the number of replicas this cluster has from the Druid object, number of pods the petset have, + +**Coordinators Replicas** + +```bash +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.coordinators.replicas' +1 + +$ kubectl get petset -n demo druid-cluster-coordinators -o json | jq '.spec.replicas' +1 +``` + +**Historicals Replicas** + +```bash +$ kubectl get druid -n demo druid-cluster -o json | jq '.spec.topology.historicals.replicas' +1 + +$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.replicas' +1 +``` + +Now, we are going to verify the number of replicas this cluster has from the Druid UI. + +### Verify Replica Count from Druid UI + +Verify the scaled replica count of nodes from the druid ui. To access the UI follow the steps described in the first part of this guide. [(Check Replica Count from Druid UI)](/docs/guides/druid/scaling/horizontal-scaling/#Check-Replica-Count-from-Druid-UI) + +If you follow the steps properly, you should be able to see that the replica count of both `coordinators` and `historicals` has become 1. Also as the `coordinators` is serving as the `overlords`, the count of `overlords` has also become 1. + +

+  lifecycle +

+ + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete dr -n demo druid-cluster +kubectl delete druidopsrequest -n demo druid-hscale-up druid-hscale-down +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/scaling/horizontal-scaling/images/dr-horizontal-scaling.png b/docs/guides/druid/scaling/horizontal-scaling/images/dr-horizontal-scaling.png new file mode 100644 index 0000000000..83615ee58f Binary files /dev/null and b/docs/guides/druid/scaling/horizontal-scaling/images/dr-horizontal-scaling.png differ diff --git a/docs/guides/druid/scaling/horizontal-scaling/images/druid-ui-scaled-up.png b/docs/guides/druid/scaling/horizontal-scaling/images/druid-ui-scaled-up.png new file mode 100644 index 0000000000..f9369cdc0d Binary files /dev/null and b/docs/guides/druid/scaling/horizontal-scaling/images/druid-ui-scaled-up.png differ diff --git a/docs/guides/druid/scaling/horizontal-scaling/images/druid-ui.png b/docs/guides/druid/scaling/horizontal-scaling/images/druid-ui.png new file mode 100644 index 0000000000..f81925c59c Binary files /dev/null and b/docs/guides/druid/scaling/horizontal-scaling/images/druid-ui.png differ diff --git a/docs/guides/druid/scaling/horizontal-scaling/overview.md b/docs/guides/druid/scaling/horizontal-scaling/overview.md new file mode 100644 index 0000000000..7158e2432a --- /dev/null +++ b/docs/guides/druid/scaling/horizontal-scaling/overview.md @@ -0,0 +1,54 @@ +--- +title: Druid Horizontal Scaling Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-scaling-horizontal-scaling-overview + name: Overview + parent: guides-druid-scaling-horizontal-scaling + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Horizontal Scaling + +This guide will give an overview on how KubeDB Ops-manager operator scales up or down `Druid` cluster replicas of various component such as Coordinators, Overlords, Historicals, MiddleManager, Brokers and Routers. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + +## How Horizontal Scaling Process Works + +The following diagram shows how KubeDB Ops-manager operator scales up or down `Druid` database components. Open the image in a new tab to see the enlarged version. + +
+ Horizontal scaling process of Druid +
Fig: Horizontal scaling process of Druid
+
+ +The Horizontal scaling process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource (CR). + +2. `KubeDB` Provisioner operator watches the `Druid` CR. + +3. When the operator finds a `Druid` CR, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Then, in order to scale the various components (i.e. Coordinators, Overlords, Historicals, MiddleManagers, Brokers, Routers) of the `Druid` cluster, the user creates a `DruidOpsRequest` CR with desired information. + +5. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CR. + +6. When it finds a `DruidOpsRequest` CR, it halts the `Druid` object which is referred from the `DruidOpsRequest`. So, the `KubeDB` Provisioner operator doesn't perform any operations on the `Druid` object during the horizontal scaling process. + +7. Then the `KubeDB` Ops-manager operator will scale the related PetSet Pods to reach the expected number of replicas defined in the `DruidOpsRequest` CR. + +8. After the successfully scaling the replicas of the related PetSet Pods, the `KubeDB` Ops-manager operator updates the number of replicas in the `Druid` object to reflect the updated state. + +9. After the successful scaling of the `Druid` replicas, the `KubeDB` Ops-manager operator resumes the `Druid` object so that the `KubeDB` Provisioner operator resumes its usual operations. + +In the next docs, we are going to show a step-by-step guide on horizontal scaling of Druid cluster using `DruidOpsRequest` CRD. \ No newline at end of file diff --git a/docs/guides/druid/scaling/horizontal-scaling/yamls/deep-storage-config.yaml b/docs/guides/druid/scaling/horizontal-scaling/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/scaling/horizontal-scaling/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-cluster.yaml b/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..6351c2ddda --- /dev/null +++ b/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-cluster.yaml @@ -0,0 +1,16 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete + diff --git a/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-hscale-down.yaml b/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-hscale-down.yaml new file mode 100644 index 0000000000..4cfa3f715d --- /dev/null +++ b/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-hscale-down.yaml @@ -0,0 +1,13 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-hscale-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: druid-cluster + horizontalScaling: + topology: + coordinators: 1 + historicals: 1 \ No newline at end of file diff --git a/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-hscale-up.yaml b/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-hscale-up.yaml new file mode 100644 index 0000000000..8063b37fb1 --- /dev/null +++ b/docs/guides/druid/scaling/horizontal-scaling/yamls/druid-hscale-up.yaml @@ -0,0 +1,13 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-hscale-up + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: druid-cluster + horizontalScaling: + topology: + coordinators: 2 + historicals: 2 \ No newline at end of file diff --git a/docs/guides/druid/scaling/vertical-scaling/_index.md b/docs/guides/druid/scaling/vertical-scaling/_index.md new file mode 100644 index 0000000000..8a9a5727c0 --- /dev/null +++ b/docs/guides/druid/scaling/vertical-scaling/_index.md @@ -0,0 +1,10 @@ +--- +title: Vertical Scaling +menu: + docs_{{ .version }}: + identifier: guides-druid-scaling-vertical-scaling + name: Vertical Scaling + parent: guides-druid-scaling + weight: 20 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/scaling/vertical-scaling/guide.md b/docs/guides/druid/scaling/vertical-scaling/guide.md new file mode 100644 index 0000000000..cedf35b076 --- /dev/null +++ b/docs/guides/druid/scaling/vertical-scaling/guide.md @@ -0,0 +1,454 @@ +--- +title: Vertical Scaling Druid Cluster +menu: + docs_{{ .version }}: + identifier: guides-druid-scaling-vertical-scaling-guide + name: Druid Vertical Scaling + parent: guides-druid-scaling-vertical-scaling + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Vertical Scale Druid Topology Cluster + +This guide will show you how to use `KubeDB` Ops-manager operator to update the resources of a Druid topology cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` Provisioner and Ops-manager operator in your cluster following the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [Topology](/docs/guides/druid/clustering/overview/index.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Vertical Scaling Overview](/docs/guides/druid/scaling/vertical-scaling/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/examples/druid](/docs/examples/druid) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +## Apply Vertical Scaling on Topology Cluster + +Here, we are going to deploy a `Druid` topology cluster using a supported version by `KubeDB` operator. Then we are going to apply vertical scaling on it. + +### Prepare Druid Topology Cluster + +Now, we are going to deploy a `Druid` topology cluster database with version `28.0.1`. + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/scaling/vertical-scaling/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +### Deploy Druid Cluster + +In this section, we are going to deploy a Druid topology cluster. Then, in the next section we will update the resources of the database using `DruidOpsRequest` CRD. Below is the YAML of the `Druid` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +Let's create the `Druid` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/druid/scaling/vertical-scaling/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait until `druid-cluster` has status `Ready`. i.e, + +```bash +$ kubectl get dr -n demo -w +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1aplha2 28.0.1 Provisioning 0s +druid-cluster kubedb.com/v1aplha2 28.0.1 Provisioning 24s +. +. +druid-cluster kubedb.com/v1aplha2 28.0.1 Ready 92s +``` + +Let's check the Pod containers resources for both `coordinators` and `historicals` of the Druid topology cluster. Run the following command to get the resources of the `coordinators` and `historicals` containers of the Druid topology cluster + +```bash +$ kubectl get pod -n demo druid-cluster-coordinators-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} +``` + +```bash +$ kubectl get pod -n demo druid-cluster-historicals-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} +``` +This is the default resources of the Druid topology cluster set by the `KubeDB` operator. + +We are now ready to apply the `DruidOpsRequest` CR to update the resources of this database. + +### Vertical Scaling + +Here, we are going to update the resources of the topology cluster to meet the desired resources after scaling. + +#### Create DruidOpsRequest + +In order to update the resources of the database, we have to create a `DruidOpsRequest` CR with our desired resources. Below is the YAML of the `DruidOpsRequest` CR that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-vscale + namespace: demo +spec: + type: VerticalScaling + databaseRef: + name: druid-cluster + verticalScaling: + coordinators: + resources: + requests: + memory: "1.2Gi" + cpu: "0.6" + limits: + memory: "1.2Gi" + cpu: "0.6" + historicals: + resources: + requests: + memory: "1.1Gi" + cpu: "0.6" + limits: + memory: "1.1Gi" + cpu: "0.6" + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing vertical scaling operation on `druid-cluster` cluster. +- `spec.type` specifies that we are performing `VerticalScaling` on druid. +- `spec.VerticalScaling.coordinators` specifies the desired resources of `coordinators` node after scaling. +- `spec.VerticalScaling.historicals` specifies the desired resources of `historicals` node after scaling. + +> **Note:** Similarly you can scale other druid nodes vertically by specifying the following fields: +> - For `overlords` use `spec.verticalScaling.overlords`. +> - For `brokers` use `spec.verticalScaling.brokers`. +> - For `middleManagers` use `spec.verticalScaling.middleManagers`. +> - For `routers` use `spec.verticalScaling.routers`. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/scaling/vertical-scaling/yamls/druid-vscale.yaml +druidopsrequest.ops.kubedb.com/druid-vscale created +``` + +#### Verify Druid cluster resources have been updated successfully + +If everything goes well, `KubeDB` Ops-manager operator will update the resources of `Druid` object and related `PetSets` and `Pods`. + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CR, + +```bash +$ kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +druid-vscale VerticalScaling Successful 3m56s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to scale the cluster. + +```bash +$ kubectl describe druidopsrequest -n demo druid-vscale +Name: druid-vscale +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-21T12:53:55Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:timeout: + f:type: + f:verticalScaling: + .: + f:coordinators: + .: + f:resources: + .: + f:limits: + .: + f:cpu: + f:memory: + f:requests: + .: + f:cpu: + f:memory: + f:historicals: + .: + f:resources: + .: + f:limits: + .: + f:cpu: + f:memory: + f:requests: + .: + f:cpu: + f:memory: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-21T12:53:55Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-21T12:54:23Z + Resource Version: 102002 + UID: fe8bb22f-02e8-4a10-9a78-fc211371d581 +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Timeout: 5m + Type: VerticalScaling + Vertical Scaling: + Coordinators: + Resources: + Limits: + Cpu: 0.6 + Memory: 1.2Gi + Requests: + Cpu: 0.6 + Memory: 1.2Gi + Historicals: + Resources: + Limits: + Cpu: 0.6 + Memory: 1.1Gi + Requests: + Cpu: 0.6 + Memory: 1.1Gi +Status: + Conditions: + Last Transition Time: 2024-10-21T12:53:55Z + Message: Druid ops-request has started to vertically scale the Druid nodes + Observed Generation: 1 + Reason: VerticalScaling + Status: True + Type: VerticalScaling + Last Transition Time: 2024-10-21T12:53:58Z + Message: Successfully updated PetSets Resources + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-21T12:54:23Z + Message: Successfully Restarted Pods With Resources + Observed Generation: 1 + Reason: RestartPods + Status: True + Type: RestartPods + Last Transition Time: 2024-10-21T12:54:03Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-21T12:54:03Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-21T12:54:08Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-21T12:54:13Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-21T12:54:13Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-21T12:54:18Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-historicals-0 + Last Transition Time: 2024-10-21T12:54:23Z + Message: Successfully completed the vertical scaling for RabbitMQ + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 67s KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/druid-vscale + Normal Starting 67s KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 67s KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: druid-vscale + Normal UpdatePetSets 64s KubeDB Ops-manager Operator Successfully updated PetSets Resources + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 59s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 59s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 54s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 49s KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 49s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 44s KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Normal RestartPods 39s KubeDB Ops-manager Operator Successfully Restarted Pods With Resources + Normal Starting 39s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 39s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: druid-vscale +``` +Now, we are going to verify from one of the Pod yaml whether the resources of the topology cluster has updated to meet up the desired state, Let's check, + +```bash +$ kubectl get pod -n demo druid-cluster-coordinators-0 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "cpu": "600m", + "memory": "1288490188800m" + }, + "requests": { + "cpu": "600m", + "memory": "1288490188800m" + } +} +$ kubectl get pod -n demo druid-cluster-historicals-1 -o json | jq '.spec.containers[].resources' +{ + "limits": { + "cpu": "600m", + "memory": "1181116006400m" + }, + "requests": { + "cpu": "600m", + "memory": "1181116006400m" + } +} +``` + +The above output verifies that we have successfully scaled up the resources of the Druid topology cluster. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete dr -n demo druid-cluster +kubectl delete druidopsrequest -n demo druid-vscale +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/scaling/vertical-scaling/images/dr-vertical-scaling.png b/docs/guides/druid/scaling/vertical-scaling/images/dr-vertical-scaling.png new file mode 100644 index 0000000000..552bb0fb30 Binary files /dev/null and b/docs/guides/druid/scaling/vertical-scaling/images/dr-vertical-scaling.png differ diff --git a/docs/guides/druid/scaling/vertical-scaling/overview.md b/docs/guides/druid/scaling/vertical-scaling/overview.md new file mode 100644 index 0000000000..2ddd690601 --- /dev/null +++ b/docs/guides/druid/scaling/vertical-scaling/overview.md @@ -0,0 +1,54 @@ +--- +title: Druid Vertical Scaling Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-scaling-vertical-scaling-overview + name: Overview + parent: guides-druid-scaling-vertical-scaling + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Vertical Scaling + +This guide will give an overview on how KubeDB Ops-manager operator updates the resources(for example CPU and Memory etc.) of the `Druid`. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/kafka/concepts/kafka.md) + - [DruidOpsRequest](/docs/guides/kafka/concepts/kafkaopsrequest.md) + +## How Vertical Scaling Process Works + +The following diagram shows how KubeDB Ops-manager operator updates the resources of the `Druid`. Open the image in a new tab to see the enlarged version. + +
+  Vertical scaling process of Druid +
Fig: Vertical scaling process of Druid
+
+ +The vertical scaling process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource (CR). + +2. `KubeDB` Provisioner operator watches the `Druid` CR. + +3. When the operator finds a `Druid` CR, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Then, in order to update the resources(for example `CPU`, `Memory` etc.) of the `Druid` cluster, the user creates a `DruidOpsRequest` CR with desired information. + +5. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CR. + +6. When it finds a `DruidOpsRequest` CR, it halts the `Druid` object which is referred from the `DruidOpsRequest`. So, the `KubeDB` Provisioner operator doesn't perform any operations on the `Druid` object during the vertical scaling process. + +7. Then the `KubeDB` Ops-manager operator will update resources of the PetSet Pods to reach desired state. + +8. After the successful update of the resources of the PetSet's replica, the `KubeDB` Ops-manager operator updates the `Druid` object to reflect the updated state. + +9. After the successful update of the `Druid` resources, the `KubeDB` Ops-manager operator resumes the `Druid` object so that the `KubeDB` Provisioner operator resumes its usual operations. + +In the next docs, we are going to show a step by step guide on updating resources of Druid database using `DruidOpsRequest` CRD. \ No newline at end of file diff --git a/docs/guides/druid/scaling/vertical-scaling/yamls/deep-storage-config.yaml b/docs/guides/druid/scaling/vertical-scaling/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/scaling/vertical-scaling/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/scaling/vertical-scaling/yamls/druid-cluster.yaml b/docs/guides/druid/scaling/vertical-scaling/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..7a89d0dc91 --- /dev/null +++ b/docs/guides/druid/scaling/vertical-scaling/yamls/druid-cluster.yaml @@ -0,0 +1,15 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete diff --git a/docs/guides/druid/scaling/vertical-scaling/yamls/druid-vscale.yaml b/docs/guides/druid/scaling/vertical-scaling/yamls/druid-vscale.yaml new file mode 100644 index 0000000000..38cf25d3ca --- /dev/null +++ b/docs/guides/druid/scaling/vertical-scaling/yamls/druid-vscale.yaml @@ -0,0 +1,28 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-vscale + namespace: demo +spec: + type: VerticalScaling + databaseRef: + name: druid-cluster + verticalScaling: + coordinators: + resources: + requests: + memory: "1.2Gi" + cpu: "0.6" + limits: + memory: "1.2Gi" + cpu: "0.6" + historicals: + resources: + requests: + memory: "1.1Gi" + cpu: "0.6" + limits: + memory: "1.1Gi" + cpu: "0.6" + timeout: 5m + apply: IfReady diff --git a/docs/guides/druid/tls/_index.md b/docs/guides/druid/tls/_index.md new file mode 100755 index 0000000000..2bf445ceea --- /dev/null +++ b/docs/guides/druid/tls/_index.md @@ -0,0 +1,10 @@ +--- +title: Run Druid with TLS +menu: + docs_{{ .version }}: + identifier: guides-druid-tls + name: TLS/SSL Encryption + parent: guides-druid + weight: 90 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/druid/tls/guide.md b/docs/guides/druid/tls/guide.md new file mode 100644 index 0000000000..ead99ecbd5 --- /dev/null +++ b/docs/guides/druid/tls/guide.md @@ -0,0 +1,307 @@ +--- +title: Druid Combined TLS/SSL Encryption +menu: + docs_{{ .version }}: + identifier: guides-druid-tls-guide + name: Druid TLS/SSL + parent: guides-druid-tls + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Run Druid with TLS/SSL (Transport Encryption) + +KubeDB supports providing TLS/SSL encryption for Druid. This tutorial will show you how to use KubeDB to run a Druid cluster with TLS/SSL encryption. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install [`cert-manger`](https://cert-manager.io/docs/installation/) v1.0.0 or later to your cluster to manage your SSL/TLS certificates. + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + + ```bash + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Overview + +KubeDB uses following crd fields to enable SSL/TLS encryption in Druid. + +- `spec:` + - `enableSSL` + - `tls:` + - `issuerRef` + - `certificate` + +Read about the fields in details in [druid concept](/docs/guides/druid/concepts/druid.md), + +`tls` is applicable for all types of Druid (i.e., `combined` and `topology`). + +Users must specify the `tls.issuerRef` field. KubeDB uses the `issuer` or `clusterIssuer` referenced in the `tls.issuerRef` field, and the certificate specs provided in `tls.certificate` to generate certificate secrets. These certificate secrets are then used to generate required certificates including `ca.crt`, `tls.crt`, `tls.key`, `keystore.jks` and `truststore.jks`. + +## Create Issuer/ ClusterIssuer + +We are going to create an example `Issuer` that will be used throughout the duration of this tutorial to enable SSL/TLS in Druid. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating you ca certificates using openssl. + +```bash +openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=druid/O=kubedb" +``` + +- Now create a ca-secret using the certificate files you have just generated. + +```bash +kubectl create secret tls druid-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +``` + +Now, create an `Issuer` using the `ca-secret` you have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: druid-ca-issuer + namespace: demo +spec: + ca: + secretName: druid-ca +``` + +Apply the `YAML` file: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/tls/yamls/druid-ca-issuer.yaml +issuer.cert-manager.io/druid-ca-issuer created +``` + +## TLS/SSL encryption in Druid Cluster + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/tls/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +Now, lets go ahead and create a druid database. + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster-tls + namespace: demo +spec: + version: 28.0.1 + enableSSL: true + tls: + issuerRef: + apiGroup: "cert-manager.io" + kind: Issuer + name: druid-ca-issuer + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +### Deploy Druid Topology Cluster with TLS/SSL + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/tls/yamls/druid-cluster-tls.yaml +druid.kubedb.com/druid-cluster-tls created +``` + +Now, wait until `druid-cluster-tls created` has status `Ready`. i.e, + +```bash +$ kubectl get druid -n demo -w + +Every 2.0s: kubectl get druid -n demo aadee: Fri Sep 6 12:34:51 2024 +NAME TYPE VERSION STATUS AGE +druid-cluster-tls kubedb.com/v1alpha2 28.0.1 Ready 20s +druid-cluster-tls kubedb.com/v1alpha2 28.0.1 Provisioning 1m +... +... +druid-cluster-tls kubedb.com/v1alpha2 28.0.1 Ready 38m +``` + +### Verify TLS/SSL in Druid Cluster + +```bash +$ kubectl describe secret druid-cluster-tls-client-cert -n demo +Name: druid-cluster-tls-client-cert +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=druid-cluster-tls + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=druids.kubedb.com + controller.cert-manager.io/fao=true +Annotations: cert-manager.io/alt-names: + *.druid-cluster-tls-pods.demo.svc.cluster.local,druid-cluster-tls-brokers-0.druid-cluster-tls-pods.demo.svc.cluster.local:8282,druid-clust... + cert-manager.io/certificate-name: druid-cluster-tls-client-cert + cert-manager.io/common-name: druid-cluster-tls-pods.demo.svc + cert-manager.io/ip-sans: 127.0.0.1 + cert-manager.io/issuer-group: cert-manager.io + cert-manager.io/issuer-kind: Issuer + cert-manager.io/issuer-name: druid-ca-issuer + cert-manager.io/uri-sans: + +Type: kubernetes.io/tls + +Data +==== +ca.crt: 1147 bytes +keystore.jks: 3720 bytes +tls-combined.pem: 3835 bytes +tls.crt: 2126 bytes +tls.key: 1708 bytes +truststore.jks: 865 bytes +``` + +Now, Lets exec into a druid coordinators pod and verify the configuration that the TLS is enabled. + +```bash +$ kubectl exec -it -n demo druid-cluster-tls-coordinators-0 -- bash +Defaulted container "druid" out of: druid, init-druid (init) +bash-5.1$ cat conf/druid/cluster/_common/common.runtime.properties +druid.client.https.trustStorePassword={"type": "environment", "variable": "DRUID_KEY_STORE_PASSWORD"} +druid.client.https.trustStorePath=/opt/druid/ssl/truststore.jks +druid.client.https.trustStoreType=jks +druid.emitter=noop +druid.enablePlaintextPort=false +druid.enableTlsPort=true +druid.metadata.mysql.ssl.clientCertificateKeyStorePassword=password +druid.metadata.mysql.ssl.clientCertificateKeyStoreType=JKS +druid.metadata.mysql.ssl.clientCertificateKeyStoreUrl=/opt/druid/ssl/metadata/keystore.jks +druid.metadata.mysql.ssl.useSSL=true +druid.server.https.certAlias=druid +druid.server.https.keyStorePassword={"type": "environment", "variable": "DRUID_KEY_STORE_PASSWORD"} +druid.server.https.keyStorePath=/opt/druid/ssl/keystore.jks +druid.server.https.keyStoreType=jks +``` + +We can see from the above output that, all the TLS related configuration is added. Here the `MySQL` and `ZooKeeper` deployed with Druid is also TLS secure and their connection configs are added as well. + +#### Verify TLS/SSL using Druid UI + +To check follow the following steps: + +Druid uses separate ports for TLS/SSL. While the plaintext port for `routers` node is `8888`. For TLS, it is `9088`. Hence, we will use that port to access the UI. + +First port-forward the port `9088` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-tls-routers 9088 +Forwarding from 127.0.0.1:9088 -> 9088 +Forwarding from [::1]:9088 -> 9088 +``` + + +Now hit the `https://localhost:9088/` from any browser. Here you may select `Advance` and then `Proceed to localhost (unsafe)` or you can add the `ca.crt` from the secret `druid-cluster-tls-client-cert` to your browser's Authorities. + +After that you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-cluster-tls-admin-cred -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-cluster-tls-admin-cred -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  lifecycle +

+ +From the above output, we can see that the connection is secure. + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete druid -n demo druid-cluster-tls +kubectl delete issuer -n demo druid-ca-issuer +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Monitor your Druid cluster with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). +- Monitor your Druid cluster with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md). + +[//]: # (- Use [kubedb cli](/docs/guides/druid/cli/cli.md) to manage databases like kubectl for Kubernetes.) +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/tls/images/druid-ui.png b/docs/guides/druid/tls/images/druid-ui.png new file mode 100644 index 0000000000..9f173c38c2 Binary files /dev/null and b/docs/guides/druid/tls/images/druid-ui.png differ diff --git a/docs/guides/druid/tls/images/tls.png b/docs/guides/druid/tls/images/tls.png new file mode 100644 index 0000000000..7f21589742 Binary files /dev/null and b/docs/guides/druid/tls/images/tls.png differ diff --git a/docs/guides/druid/tls/overview.md b/docs/guides/druid/tls/overview.md new file mode 100644 index 0000000000..2022e51955 --- /dev/null +++ b/docs/guides/druid/tls/overview.md @@ -0,0 +1,70 @@ +--- +title: Druid TLS/SSL Encryption Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-tls-overview + name: Overview + parent: guides-druid-tls + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid TLS/SSL Encryption + +**Prerequisite :** To configure TLS/SSL in `Druid`, `KubeDB` uses `cert-manager` to issue certificates. So first you have to make sure that the cluster has `cert-manager` installed. To install `cert-manager` in your cluster following steps [here](https://cert-manager.io/docs/installation/kubernetes/). + +To issue a certificate, the following crd of `cert-manager` is used: + +- `Issuer/ClusterIssuer`: Issuers, and ClusterIssuers represent certificate authorities (CAs) that are able to generate signed certificates by honoring certificate signing requests. All cert-manager certificates require a referenced issuer that is in a ready condition to attempt to honor the request. You can learn more details [here](https://cert-manager.io/docs/concepts/issuer/). + +- `Certificate`: `cert-manager` has the concept of Certificates that define a desired x509 certificate which will be renewed and kept up to date. You can learn more details [here](https://cert-manager.io/docs/concepts/certificate/). + +**Druid CRD Specification :** + +KubeDB uses following crd fields to enable SSL/TLS encryption in `Druid`. + +- `spec:` + - `enableSSL` + - `tls:` + - `issuerRef` + - `certificates` + +Read about the fields in details from [druid concept](/docs/guides/druid/concepts/druid.md), + +When, `enableSSL` is set to `true`, the users must specify the `tls.issuerRef` field. `KubeDB` uses the `issuer` or `clusterIssuer` referenced in the `tls.issuerRef` field, and the certificate specs provided in `tls.certificate` to generate certificate secrets using `Issuer/ClusterIssuers` specification. These certificates secrets including `ca.crt`, `tls.crt` and `tls.key` etc. are used to configure `druid` server and clients. + +## How TLS/SSL configures in Druid + +The following figure shows how `KubeDB` enterprise used to configure TLS/SSL in Druid. Open the image in a new tab to see the enlarged version. + +
+Deploy Druid with TLS/SSL +
Fig: Deploy Druid with TLS/SSL
+
+ +Deploying Druid with TLS/SSL configuration process consists of the following steps: + +1. At first, a user creates a `Issuer/ClusterIssuer` cr. + +2. Then the user creates a `Druid` CR which refers to the `Issuer/ClusterIssuer` CR that the user created in the previous step. + +3. `KubeDB` Provisioner operator watches for the `Druid` cr. + +4. When it finds one, it creates `Secret`, `Service`, etc. for the `Druid` cluster. + +5. `KubeDB` Ops-manager operator watches for `Druid`(5c), `Issuer/ClusterIssuer`(5b), `Secret` and `Service`(5a). + +6. When it finds all the resources(`Druid`, `Issuer/ClusterIssuer`, `Secret`, `Service`), it creates `Certificates` by using `tls.issuerRef` and `tls.certificates` field specification from `Druid` cr. + +7. `cert-manager` watches for certificates. + +8. When it finds one, it creates certificate secrets `tls-secrets`(server, client, exporter secrets etc.) that holds the actual certificate signed by the CA. + +9. `KubeDB` Provisioner operator watches for the Certificate secrets `tls-secrets`. + +10. When it finds all the tls-secret, it creates the related `PetSets` so that Druid database can be configured with TLS/SSL. + +In the next doc, we are going to show a step-by-step guide on how to configure a `Druid` cluster with TLS/SSL. \ No newline at end of file diff --git a/docs/guides/druid/tls/yamls/deep-storage-config.yaml b/docs/guides/druid/tls/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/tls/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/tls/yamls/druid-ca-issuer.yaml b/docs/guides/druid/tls/yamls/druid-ca-issuer.yaml new file mode 100644 index 0000000000..d6298c972c --- /dev/null +++ b/docs/guides/druid/tls/yamls/druid-ca-issuer.yaml @@ -0,0 +1,8 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: druid-ca-issuer + namespace: demo +spec: + ca: + secretName: druid-ca diff --git a/docs/guides/druid/tls/yamls/druid-cluster-tls.yaml b/docs/guides/druid/tls/yamls/druid-cluster-tls.yaml new file mode 100644 index 0000000000..902b5b36d4 --- /dev/null +++ b/docs/guides/druid/tls/yamls/druid-cluster-tls.yaml @@ -0,0 +1,21 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster-tls + namespace: demo +spec: + version: 28.0.1 + enableSSL: true + tls: + issuerRef: + apiGroup: "cert-manager.io" + kind: Issuer + name: druid-ca-issuer + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete diff --git a/docs/guides/druid/update-version/_index.md b/docs/guides/druid/update-version/_index.md new file mode 100644 index 0000000000..26c6ab4da1 --- /dev/null +++ b/docs/guides/druid/update-version/_index.md @@ -0,0 +1,10 @@ +--- +title: Update Version +menu: + docs_{{ .version }}: + identifier: guides-druid-update-version + name: Update Version + parent: guides-druid + weight: 60 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/druid/update-version/guide.md b/docs/guides/druid/update-version/guide.md new file mode 100644 index 0000000000..f5c31ca64d --- /dev/null +++ b/docs/guides/druid/update-version/guide.md @@ -0,0 +1,448 @@ +--- +title: Update Version of Druid +menu: + docs_{{ .version }}: + identifier: guides-druid-update-version-guide + name: Update Druid Version + parent: guides-druid-update-version + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- +> New to KubeDB? Please start [here](/docs/README.md). + +# Update version of Druid + +This guide will show you how to use `KubeDB` Ops-manager operator to update the version of `Druid` Combined or Topology. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install `KubeDB` Provisioner and Ops-manager operator in your cluster following the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Updating Overview](/docs/guides/druid/update-version/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/examples/druid](/docs/examples/druid) directory of [kubedb/docs](https://github.com/kube/docs) repository. + +## Prepare Druid + +Now, we are going to deploy a `Druid` cluster with version `28.0.1`. + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash + +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/update-version/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +### Deploy Druid + +In this section, we are going to deploy a Druid topology cluster. Then, in the next section we will update the version using `DruidOpsRequest` CRD. Below is the YAML of the `Druid` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-quickstart + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete +``` + +Let's create the `Druid` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/update-version/yamls/druid-cluster.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait until `druid-cluster` created has status `Ready`. i.e, + +```bash +$ kubectl get dr -n demo -w +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1aplha2 28.0.1 Provisioning 0s +druid-cluster kubedb.com/v1aplha2 28.0.1 Provisioning 55s +. +. +druid-cluster kubedb.com/v1aplha2 28.0.1 Ready 119s +``` + +We are now ready to apply the `DruidOpsRequest` CR to update. + +#### Check Druid Version from UI: + +You can also see the version of druid cluster from the druid ui. For that, follow the following steps: + +First, port-forward the port `8888` to local machine: + +```bash +$ kubectl port-forward -n demo svc/druid-cluster-routers 8888 +Forwarding from 127.0.0.1:8888 -> 8888 +Forwarding from [::1]:8888 -> 8888 +``` + +Now hit the `http://localhost:8888` from any browser, and you will be prompted to provide the credential of the druid database. By following the steps discussed below, you can get the credential generated by the KubeDB operator for your Druid database. + +**Connection information:** + +- Username: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.username}' | base64 -d + admin + ``` + +- Password: + + ```bash + $ kubectl get secret -n demo druid-cluster-admin-cred -o jsonpath='{.data.password}' | base64 -d + LzJtVRX5E8MorFaf + ``` + +After providing the credentials correctly, you should be able to access the web console like shown below. + +

+  lifecycle +

+ + +Here, we can see that the version of the druid cluster is `28.0.1`. + +### Update Druid Version + +Here, we are going to update `Druid` from `28.0.1` to `30.0.0`. + +#### Create DruidOpsRequest: + +In order to update the version, we have to create a `DruidOpsRequest` CR with your desired version that is supported by `KubeDB`. Below is the YAML of the `DruidOpsRequest` CR that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-update-version + namespace: demo +spec: + type: UpdateVersion + databaseRef: + name: druid-cluster + updateVersion: + targetVersion: 30.0.0 + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing operation on `druid-cluster` Druid. +- `spec.type` specifies that we are going to perform `UpdateVersion` on our database. +- `spec.updateVersion.targetVersion` specifies the expected version of the database `30.0.0`. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/update-version/yamls/druid-hscale-up.yaml +druidopsrequest.ops.kubedb.com/druid-update-version created +``` + +#### Verify Druid version updated successfully + +If everything goes well, `KubeDB` Ops-manager operator will update the image of `Druid` object and related `PetSets` and `Pods`. + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CR, + +```bash +$ watch kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +druid-update-version UpdateVersion Successful 2m6s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to update the database version. + +```bash +$ kubectl describe druidopsrequest -n demo druid-update-version +Name: druid-update-version +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-21T13:04:51Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:timeout: + f:type: + f:updateVersion: + .: + f:targetVersion: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-21T13:04:51Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-21T13:08:46Z + Resource Version: 103855 + UID: 5d470e24-37fd-4e16-b7a3-33040dcefe3d +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Timeout: 5m + Type: UpdateVersion + Update Version: + Target Version: 30.0.0 +Status: + Conditions: + Last Transition Time: 2024-10-21T13:04:51Z + Message: Druid ops-request has started to update version + Observed Generation: 1 + Reason: UpdateVersion + Status: True + Type: UpdateVersion + Last Transition Time: 2024-10-21T13:04:56Z + Message: successfully reconciled the Druid with updated version + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-21T13:08:46Z + Message: Successfully Restarted Druid nodes + Observed Generation: 1 + Reason: RestartPods + Status: True + Type: RestartPods + Last Transition Time: 2024-10-21T13:05:01Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-21T13:05:01Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-historicals-0 + Last Transition Time: 2024-10-21T13:08:01Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-historicals-0 + Last Transition Time: 2024-10-21T13:08:06Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-21T13:08:06Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-21T13:08:11Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-middlemanagers-0 + Last Transition Time: 2024-10-21T13:08:16Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-21T13:08:16Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-brokers-0 + Last Transition Time: 2024-10-21T13:08:21Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-brokers-0 + Last Transition Time: 2024-10-21T13:08:26Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-21T13:08:26Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-routers-0 + Last Transition Time: 2024-10-21T13:08:31Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-routers-0 + Last Transition Time: 2024-10-21T13:08:36Z + Message: get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: GetPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-21T13:08:36Z + Message: evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: EvictPod--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-21T13:08:41Z + Message: check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Observed Generation: 1 + Status: True + Type: CheckPodRunning--druid-cluster-coordinators-0 + Last Transition Time: 2024-10-21T13:08:46Z + Message: Successfully completed update druid version + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 21m KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/druid-update-version + Normal UpdatePetSets 21m KubeDB Ops-manager Operator successfully reconciled the Druid with updated version + Warning get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 21m KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 21m KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:False; PodName:druid-cluster-historicals-0 21m KubeDB Ops-manager Operator check pod running; ConditionStatus:False; PodName:druid-cluster-historicals-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 18m KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-historicals-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 18m KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 18m KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 18m KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-middlemanagers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 17m KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 17m KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 17m KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-brokers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 17m KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 17m KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 17m KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-routers-0 + Warning get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 17m KubeDB Ops-manager Operator get pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 17m KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Warning check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 17m KubeDB Ops-manager Operator check pod running; ConditionStatus:True; PodName:druid-cluster-coordinators-0 + Normal RestartPods 17m KubeDB Ops-manager Operator Successfully Restarted Druid nodes + Normal Starting 17m KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 17m KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: druid-update-version +``` + +Now, we are going to verify whether the `Druid` and the related `PetSets` and their `Pods` have the new version image. Let's check, + +```bash +$ kubectl get dr -n demo druid-cluster -o=jsonpath='{.spec.version}{"\n"}' +30.0.0 + +$ kubectl get petset -n demo druid-cluster-brokers -o=jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/druid:30.0.0@sha256:4cd60a1dc6a124e27e91ec52ca39e2b9ca6809df915ae2dd712a2dd7462626d7 + +$ kubectl get pods -n demo druid-cluster-brokers-0 -o=jsonpath='{.spec.containers[0].image}{"\n"}' +ghcr.io/appscode-images/druid:30.0.0 +``` + +You can see from above, our `Druid` has been updated with the new version. So, the updateVersion process is successfully completed. + +#### Verify updated Druid Version from UI: + +You can also see the version of druid cluster from the druid ui by following the steps described previously in this tutorial. [Check Druid Version from UI](/docs/guides/druid/update-version/guide.md/#Check-Druid-Version-from-UI) + +If you follow the steps properly, you should be able to see that the version is upgraded to `30.0.0` from the druid console as shown below. + +

+  lifecycle +

+ +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete druidopsrequest -n demo druid-update-version +kubectl delete dr -n demo druid-cluster +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). + +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/update-version/images/dr-update-version.png b/docs/guides/druid/update-version/images/dr-update-version.png new file mode 100644 index 0000000000..b61db35bc0 Binary files /dev/null and b/docs/guides/druid/update-version/images/dr-update-version.png differ diff --git a/docs/guides/druid/update-version/images/druid-ui-28.png b/docs/guides/druid/update-version/images/druid-ui-28.png new file mode 100644 index 0000000000..d5c74a4ad2 Binary files /dev/null and b/docs/guides/druid/update-version/images/druid-ui-28.png differ diff --git a/docs/guides/druid/update-version/images/druid-ui-30.png b/docs/guides/druid/update-version/images/druid-ui-30.png new file mode 100644 index 0000000000..f1da2d8ef1 Binary files /dev/null and b/docs/guides/druid/update-version/images/druid-ui-30.png differ diff --git a/docs/guides/druid/update-version/overview.md b/docs/guides/druid/update-version/overview.md new file mode 100644 index 0000000000..b1c2f21ef5 --- /dev/null +++ b/docs/guides/druid/update-version/overview.md @@ -0,0 +1,53 @@ +--- +title: Update Version Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-update-version-overview + name: Overview + parent: guides-druid-update-version + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Update Version Overview + +This guide will give you an overview on how KubeDB Ops-manager operator update the version of `Druid`. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + +## How update version Process Works + +The following diagram shows how KubeDB Ops-manager operator used to update the version of `Druid`. Open the image in a new tab to see the enlarged version. + +
+  updating Process of Druid +
Fig: updating Process of Druid
+
+ +The updating process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource (CR). + +2. `KubeDB` Provisioner operator watches the `Druid` CR. + +3. When the operator finds a `Druid` CR, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Then, in order to update the version of the `Druid` database the user creates a `DruidOpsRequest` CR with the desired version. + +5. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CR. + +6. When it finds a `DruidOpsRequest` CR, it halts the `Druid` object which is referred from the `DruidOpsRequest`. So, the `KubeDB` Provisioner operator doesn't perform any operations on the `Druid` object during the updating process. + +7. By looking at the target version from `DruidOpsRequest` CR, `KubeDB` Ops-manager operator updates the images of all the `PetSets`. + +8. After successfully updating the `PetSets` and their `Pods` images, the `KubeDB` Ops-manager operator updates the image of the `Druid` object to reflect the updated state of the database. + +9. After successfully updating of `Druid` object, the `KubeDB` Ops-manager operator resumes the `Druid` object so that the `KubeDB` Provisioner operator can resume its usual operations. + +In the next doc, we are going to show a step by step guide on updating of a Druid database using updateVersion operation. \ No newline at end of file diff --git a/docs/guides/druid/update-version/yamls/deep-storage-config.yaml b/docs/guides/druid/update-version/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/update-version/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/update-version/yamls/druid-cluster.yaml b/docs/guides/druid/update-version/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..7a89d0dc91 --- /dev/null +++ b/docs/guides/druid/update-version/yamls/druid-cluster.yaml @@ -0,0 +1,15 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + routers: + replicas: 1 + deletionPolicy: Delete diff --git a/docs/guides/druid/update-version/yamls/update-version-ops.yaml b/docs/guides/druid/update-version/yamls/update-version-ops.yaml new file mode 100644 index 0000000000..a6aaa91063 --- /dev/null +++ b/docs/guides/druid/update-version/yamls/update-version-ops.yaml @@ -0,0 +1,13 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: druid-update-version + namespace: demo +spec: + type: UpdateVersion + databaseRef: + name: druid-cluster + updateVersion: + targetVersion: 30.0.0 + timeout: 5m + apply: IfReady \ No newline at end of file diff --git a/docs/guides/druid/volume-expansion/_index.md b/docs/guides/druid/volume-expansion/_index.md new file mode 100644 index 0000000000..50632cd875 --- /dev/null +++ b/docs/guides/druid/volume-expansion/_index.md @@ -0,0 +1,10 @@ +--- +title: Volume Expansion +menu: + docs_{{ .version }}: + identifier: guides-druid-volume-expansion + name: Volume Expansion + parent: guides-druid + weight: 80 +menu_name: docs_{{ .version }} +--- \ No newline at end of file diff --git a/docs/guides/druid/volume-expansion/guide.md b/docs/guides/druid/volume-expansion/guide.md new file mode 100644 index 0000000000..d9a110aa18 --- /dev/null +++ b/docs/guides/druid/volume-expansion/guide.md @@ -0,0 +1,498 @@ +--- +title: Druid Topology Volume Expansion +menu: + docs_{{ .version }}: + identifier: guides-druid-volume-expansion-guide + name: Druid Volume Expansion + parent: guides-druid-volume-expansion + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Topology Volume Expansion + +This guide will show you how to use `KubeDB` Ops-manager operator to expand the volume of a Druid Topology Cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. + +- You must have a `StorageClass` that supports volume expansion. + +- Install `KubeDB` Provisioner and Ops-manager operator in your cluster following the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [Topology](/docs/guides/druid/clustering/overview/index.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + - [Volume Expansion Overview](/docs/guides/druid/volume-expansion/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> Note: The yaml files used in this tutorial are stored in [docs/examples/druid](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/druid) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Expand Volume of Topology Druid Cluster + +Here, we are going to deploy a `Druid` topology using a supported version by `KubeDB` operator. Then we are going to apply `DruidOpsRequest` to expand its volume. + +### Prepare Druid Topology Cluster + +At first verify that your cluster has a storage class, that supports volume expansion. Let's check, + +```bash +$ kubectl get storageclass +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 28h +longhorn (default) driver.longhorn.io Delete Immediate true 27h +longhorn-static driver.longhorn.io Delete Immediate true 27h +``` + +We can see from the output the `longhorn` storage class has `ALLOWVOLUMEEXPANSION` field as true. So, this storage class supports volume expansion. We can use it. + +### Create External Dependency (Deep Storage) + +Before proceeding further, we need to prepare deep storage, which is one of the external dependency of Druid and used for storing the segments. It is a storage mechanism that Apache Druid does not provide. **Amazon S3**, **Google Cloud Storage**, or **Azure Blob Storage**, **S3-compatible storage** (like **Minio**), or **HDFS** are generally convenient options for deep storage. + +In this tutorial, we will run a `minio-server` as deep storage in our local `kind` cluster using `minio-operator` and create a bucket named `druid` in it, which the deployed druid database will use. + +```bash +$ helm repo add minio https://operator.min.io/ +$ helm repo update minio +$ helm upgrade --install --namespace "minio-operator" --create-namespace "minio-operator" minio/operator --set operator.replicaCount=1 + +$ helm upgrade --install --namespace "demo" --create-namespace druid-minio minio/tenant \ +--set tenant.pools[0].servers=1 \ +--set tenant.pools[0].volumesPerServer=1 \ +--set tenant.pools[0].size=1Gi \ +--set tenant.certificate.requestAutoCert=false \ +--set tenant.buckets[0].name="druid" \ +--set tenant.pools[0].name="default" + +``` + +Now we need to create a `Secret` named `deep-storage-config`. It contains the necessary connection information using which the druid database will connect to the deep storage. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" +``` + +Let’s create the `deep-storage-config` Secret shown above: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/volume-expansion/yamls/deep-storage-config.yaml +secret/deep-storage-config created +``` + +Now, we are going to deploy a `Druid` combined cluster with version `28.0.1`. + +### Deploy Druid + +In this section, we are going to deploy a Druid topology cluster for historicals and middleManagers with 1GB volume. Then, in the next section we will expand its volume to 2GB using `DruidOpsRequest` CRD. Below is the YAML of the `Druid` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + historicals: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + middleManagers: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + routers: + replicas: 1 + deletionPolicy: Delete +``` + +Let's create the `Druid` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/volume-expansion/yamls/druid-topology.yaml +druid.kubedb.com/druid-cluster created +``` + +Now, wait until `druid-cluster` has status `Ready`. i.e, + +```bash +$ kubectl get dr -n demo -w +NAME TYPE VERSION STATUS AGE +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 0s +druid-cluster kubedb.com/v1alpha2 28.0.1 Provisioning 9s +. +. +druid-cluster kubedb.com/v1alpha2 28.0.1 Ready 3m26s +``` + +Let's check volume size from petset, and from the persistent volume, + +```bash +$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"1Gi" + +$ kubectl get petset -n demo druid-cluster-middleManagers -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"1Gi" + +$ kubectl get pv -n demo +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE +pvc-0bf49077-1c7a-4943-bb17-1dffd1626dcd 1Gi RWO Delete Bound demo/druid-cluster-segment-cache-druid-cluster-historicals-0 longhorn 10m +pvc-59ed4914-53b3-4f18-a6aa-7699c2b738e2 1Gi RWO Delete Bound demo/druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 longhorn 10m +``` + +You can see the petsets have 1GB storage, and the capacity of all the persistent volumes are also 1GB. + +We are now ready to apply the `DruidOpsRequest` CR to expand the volume of this database. + +### Volume Expansion + +Here, we are going to expand the volume of the druid topology cluster. + +#### Create DruidOpsRequest + +In order to expand the volume of the database, we have to create a `DruidOpsRequest` CR with our desired volume size. Below is the YAML of the `DruidOpsRequest` CR that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: dr-volume-exp + namespace: demo +spec: + type: VolumeExpansion + databaseRef: + name: druid-cluster + volumeExpansion: + historicals: 2Gi + middleManagers: 2Gi + mode: Offline +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing volume expansion operation on `druid-cluster`. +- `spec.type` specifies that we are performing `VolumeExpansion` on our database. +- `spec.volumeExpansion.historicals` specifies the desired volume size for historicals node. +- `spec.volumeExpansion.middleManagers` specifies the desired volume size for middleManagers node. +- `spec.volumeExpansion.mode` specifies the desired volume expansion mode(`Online` or `Offline`). + +During `Online` VolumeExpansion KubeDB expands volume without pausing database object, it directly updates the underlying PVC. And for `Offline` volume expansion, the database is paused. The Pods are deleted and PVC is updated. Then the database Pods are recreated with updated PVC. + +> If you want to expand the volume of only one node, you can specify the desired volume size for that node only. + +Let's create the `DruidOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/druid/volume-expansion/yamls/druid-volume-expansion-topology.yaml +druidopsrequest.ops.kubedb.com/dr-volume-exp created +``` + +#### Verify Druid Topology volume expanded successfully + +If everything goes well, `KubeDB` Ops-manager operator will update the volume size of `Druid` object and related `PetSets` and `Persistent Volumes`. + +Let's wait for `DruidOpsRequest` to be `Successful`. Run the following command to watch `DruidOpsRequest` CR, + +```bash +$ kubectl get druidopsrequest -n demo +NAME TYPE STATUS AGE +dr-volume-exp VolumeExpansion Successful 3m1s +``` + +We can see from the above output that the `DruidOpsRequest` has succeeded. If we describe the `DruidOpsRequest` we will get an overview of the steps that were followed to expand the volume of druid. + +```bash +$ kubectl describe druidopsrequest -n demo dr-volume-exp +Name: dr-volume-exp +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: DruidOpsRequest +Metadata: + Creation Timestamp: 2024-10-25T09:22:02Z + Generation: 1 + Managed Fields: + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:metadata: + f:annotations: + .: + f:kubectl.kubernetes.io/last-applied-configuration: + f:spec: + .: + f:apply: + f:databaseRef: + f:type: + f:volumeExpansion: + .: + f:historicals: + f:middleManagers: + f:mode: + Manager: kubectl-client-side-apply + Operation: Update + Time: 2024-10-25T09:22:02Z + API Version: ops.kubedb.com/v1alpha1 + Fields Type: FieldsV1 + fieldsV1: + f:status: + .: + f:conditions: + f:observedGeneration: + f:phase: + Manager: kubedb-ops-manager + Operation: Update + Subresource: status + Time: 2024-10-25T09:24:35Z + Resource Version: 221378 + UID: 2407cfa7-8d3b-463e-abf7-1910249009bd +Spec: + Apply: IfReady + Database Ref: + Name: druid-cluster + Type: VolumeExpansion + Volume Expansion: + Historicals: 2Gi + Middle Managers: 2Gi + Mode: Offline +Status: + Conditions: + Last Transition Time: 2024-10-25T09:22:02Z + Message: Druid ops-request has started to expand volume of druid nodes. + Observed Generation: 1 + Reason: VolumeExpansion + Status: True + Type: VolumeExpansion + Last Transition Time: 2024-10-25T09:22:10Z + Message: get pet set; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPetSet + Last Transition Time: 2024-10-25T09:22:10Z + Message: is pet set deleted; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsPetSetDeleted + Last Transition Time: 2024-10-25T09:22:30Z + Message: successfully deleted the petSets with orphan propagation policy + Observed Generation: 1 + Reason: OrphanPetSetPods + Status: True + Type: OrphanPetSetPods + Last Transition Time: 2024-10-25T09:22:35Z + Message: get pod; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPod + Last Transition Time: 2024-10-25T09:22:35Z + Message: is ops req patched; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsOpsReqPatched + Last Transition Time: 2024-10-25T09:22:35Z + Message: create pod; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CreatePod + Last Transition Time: 2024-10-25T09:22:40Z + Message: get pvc; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: GetPvc + Last Transition Time: 2024-10-25T09:22:40Z + Message: is pvc patched; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: IsPvcPatched + Last Transition Time: 2024-10-25T09:23:50Z + Message: compare storage; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CompareStorage + Last Transition Time: 2024-10-25T09:23:00Z + Message: create; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: Create + Last Transition Time: 2024-10-25T09:23:08Z + Message: is druid running; ConditionStatus:False + Observed Generation: 1 + Status: False + Type: IsDruidRunning + Last Transition Time: 2024-10-25T09:23:20Z + Message: successfully updated middleManagers node PVC sizes + Observed Generation: 1 + Reason: UpdateMiddleManagersNodePVCs + Status: True + Type: UpdateMiddleManagersNodePVCs + Last Transition Time: 2024-10-25T09:24:15Z + Message: successfully updated historicals node PVC sizes + Observed Generation: 1 + Reason: UpdateHistoricalsNodePVCs + Status: True + Type: UpdateHistoricalsNodePVCs + Last Transition Time: 2024-10-25T09:24:30Z + Message: successfully reconciled the Druid resources + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2024-10-25T09:24:35Z + Message: PetSet is recreated + Observed Generation: 1 + Reason: ReadyPetSets + Status: True + Type: ReadyPetSets + Last Transition Time: 2024-10-25T09:24:35Z + Message: Successfully completed volumeExpansion for Druid + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Starting 10m KubeDB Ops-manager Operator Start processing for DruidOpsRequest: demo/dr-volume-exp + Normal Starting 10m KubeDB Ops-manager Operator Pausing Druid databse: demo/druid-cluster + Normal Successful 10m KubeDB Ops-manager Operator Successfully paused Druid database: demo/druid-cluster for DruidOpsRequest: dr-volume-exp + Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Warning is pet set deleted; ConditionStatus:True 10m KubeDB Ops-manager Operator is pet set deleted; ConditionStatus:True + Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Warning is pet set deleted; ConditionStatus:True 10m KubeDB Ops-manager Operator is pet set deleted; ConditionStatus:True + Warning get pet set; ConditionStatus:True 10m KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Normal OrphanPetSetPods 9m59s KubeDB Ops-manager Operator successfully deleted the petSets with orphan propagation policy + Warning get pod; ConditionStatus:True 9m54s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 9m54s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning create pod; ConditionStatus:True 9m54s KubeDB Ops-manager Operator create pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 9m49s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 9m49s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning is pvc patched; ConditionStatus:True 9m49s KubeDB Ops-manager Operator is pvc patched; ConditionStatus:True + Warning compare storage; ConditionStatus:False 9m49s KubeDB Ops-manager Operator compare storage; ConditionStatus:False + Warning get pod; ConditionStatus:True 9m44s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 9m44s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 9m39s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 9m39s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 9m34s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 9m34s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 9m29s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 9m29s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning compare storage; ConditionStatus:True 9m29s KubeDB Ops-manager Operator compare storage; ConditionStatus:True + Warning create; ConditionStatus:True 9m29s KubeDB Ops-manager Operator create; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 9m29s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning get pod; ConditionStatus:True 9m24s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is druid running; ConditionStatus:False 9m21s KubeDB Ops-manager Operator is druid running; ConditionStatus:False + Warning get pod; ConditionStatus:True 9m19s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 9m14s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Normal UpdateMiddleManagersNodePVCs 9m9s KubeDB Ops-manager Operator successfully updated middleManagers node PVC sizes + Warning get pod; ConditionStatus:True 9m4s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 9m4s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning create pod; ConditionStatus:True 9m4s KubeDB Ops-manager Operator create pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m59s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 8m59s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning is pvc patched; ConditionStatus:True 8m59s KubeDB Ops-manager Operator is pvc patched; ConditionStatus:True + Warning compare storage; ConditionStatus:False 8m59s KubeDB Ops-manager Operator compare storage; ConditionStatus:False + Warning get pod; ConditionStatus:True 8m54s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 8m54s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m49s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 8m49s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m44s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 8m44s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m39s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pvc; ConditionStatus:True 8m39s KubeDB Ops-manager Operator get pvc; ConditionStatus:True + Warning compare storage; ConditionStatus:True 8m39s KubeDB Ops-manager Operator compare storage; ConditionStatus:True + Warning create; ConditionStatus:True 8m39s KubeDB Ops-manager Operator create; ConditionStatus:True + Warning is ops req patched; ConditionStatus:True 8m39s KubeDB Ops-manager Operator is ops req patched; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m34s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning is druid running; ConditionStatus:False 8m31s KubeDB Ops-manager Operator is druid running; ConditionStatus:False + Warning get pod; ConditionStatus:True 8m29s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m24s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Warning get pod; ConditionStatus:True 8m19s KubeDB Ops-manager Operator get pod; ConditionStatus:True + Normal UpdateHistoricalsNodePVCs 8m14s KubeDB Ops-manager Operator successfully updated historicals node PVC sizes + Normal UpdatePetSets 7m59s KubeDB Ops-manager Operator successfully reconciled the Druid resources + Warning get pet set; ConditionStatus:True 7m54s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Warning get pet set; ConditionStatus:True 7m54s KubeDB Ops-manager Operator get pet set; ConditionStatus:True + Normal ReadyPetSets 7m54s KubeDB Ops-manager Operator PetSet is recreated + Normal Starting 7m54s KubeDB Ops-manager Operator Resuming Druid database: demo/druid-cluster + Normal Successful 7m54s KubeDB Ops-manager Operator Successfully resumed Druid database: demo/druid-cluster for DruidOpsRequest: dr-volume-exp +``` + +Now, we are going to verify from the `Petset`, and the `Persistent Volumes` whether the volume of the database has expanded to meet the desired state, Let's check, + +```bash +$ kubectl get petset -n demo druid-cluster-historicals -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"3Gi" + +$ kubectl get petset -n demo druid-cluster-middleManagers -o json | jq '.spec.volumeClaimTemplates[].spec.resources.requests.storage' +"2Gi" + +$ kubectl get pv -n demo +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE +pvc-0bf49077-1c7a-4943-bb17-1dffd1626dcd 2Gi RWO Delete Bound demo/druid-cluster-segment-cache-druid-cluster-historicals-0 longhorn 23m +pvc-59ed4914-53b3-4f18-a6aa-7699c2b738e2 2Gi RWO Delete Bound demo/druid-cluster-base-task-dir-druid-cluster-middlemanagers-0 longhorn 23m +``` + +The above output verifies that we have successfully expanded the volume of the Druid. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete druidopsrequest -n demo dr-volume-exp +kubectl delete dr -n demo druid-cluster +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Druid object](/docs/guides/druid/concepts/druid.md). +- Different Druid topology clustering modes [here](/docs/guides/druid/clustering/_index.md). +- Monitor your Druid database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/druid/monitoring/using-prometheus-operator.md). +- +[//]: # (- Monitor your Druid database with KubeDB using [out-of-the-box builtin-Prometheus](/docs/guides/druid/monitoring/using-builtin-prometheus.md).) +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/druid/volume-expansion/images/druid-volume-expansion.png b/docs/guides/druid/volume-expansion/images/druid-volume-expansion.png new file mode 100644 index 0000000000..9e2b77d6cd Binary files /dev/null and b/docs/guides/druid/volume-expansion/images/druid-volume-expansion.png differ diff --git a/docs/guides/druid/volume-expansion/overview.md b/docs/guides/druid/volume-expansion/overview.md new file mode 100644 index 0000000000..a9290612ff --- /dev/null +++ b/docs/guides/druid/volume-expansion/overview.md @@ -0,0 +1,56 @@ +--- +title: Druid Volume Expansion Overview +menu: + docs_{{ .version }}: + identifier: guides-druid-volume-expansion-overview + name: Overview + parent: guides-druid-volume-expansion + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Druid Volume Expansion + +This guide will give an overview on how KubeDB Ops-manager operator expand the volume of various component of `Druid` like:. (Combined and Topology). + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Druid](/docs/guides/druid/concepts/druid.md) + - [DruidOpsRequest](/docs/guides/druid/concepts/druidopsrequest.md) + +## How Volume Expansion Process Works + +The following diagram shows how KubeDB Ops-manager operator expand the volumes of `Druid` database components. Open the image in a new tab to see the enlarged version. + +
+  Volume Expansion process of Druid +
Fig: Volume Expansion process of Druid
+
+ +The Volume Expansion process consists of the following steps: + +1. At first, a user creates a `Druid` Custom Resource (CR). + +2. `KubeDB` Provisioner operator watches the `Druid` CR. + +3. When the operator finds a `Druid` CR, it creates required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Each PetSet creates a Persistent Volume according to the Volume Claim Template provided in the petset configuration. This Persistent Volume will be expanded by the `KubeDB` Ops-manager operator. + +5. Then, in order to expand the volume of the druid data components (ie. Historicals, MiddleManagers) of the `Druid`, the user creates a `DruidOpsRequest` CR with desired information. + +6. `KubeDB` Ops-manager operator watches the `DruidOpsRequest` CR. + +7. When it finds a `DruidOpsRequest` CR, it halts the `Druid` object which is referred from the `DruidOpsRequest`. So, the `KubeDB` Provisioner operator doesn't perform any operations on the `Druid` object during the volume expansion process. + +8. Then the `KubeDB` Ops-manager operator will expand the persistent volume to reach the expected size defined in the `DruidOpsRequest` CR. + +9. After the successful Volume Expansion of the related PetSet Pods, the `KubeDB` Ops-manager operator updates the new volume size in the `Druid` object to reflect the updated state. + +10. After the successful Volume Expansion of the `Druid` components, the `KubeDB` Ops-manager operator resumes the `Druid` object so that the `KubeDB` Provisioner operator resumes its usual operations. + +In the next docs, we are going to show a step-by-step guide on Volume Expansion of various Druid database components using `DruidOpsRequest` CRD. diff --git a/docs/guides/druid/volume-expansion/yamls/deep-storage-config.yaml b/docs/guides/druid/volume-expansion/yamls/deep-storage-config.yaml new file mode 100644 index 0000000000..3612595828 --- /dev/null +++ b/docs/guides/druid/volume-expansion/yamls/deep-storage-config.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: deep-storage-config + namespace: demo +stringData: + druid.storage.type: "s3" + druid.storage.bucket: "druid" + druid.storage.baseKey: "druid/segments" + druid.s3.accessKey: "minio" + druid.s3.secretKey: "minio123" + druid.s3.protocol: "http" + druid.s3.enablePathStyleAccess: "true" + druid.s3.endpoint.signingRegion: "us-east-1" + druid.s3.endpoint.url: "http://myminio-hl.demo.svc.cluster.local:9000/" + diff --git a/docs/guides/druid/volume-expansion/yamls/druid-cluster.yaml b/docs/guides/druid/volume-expansion/yamls/druid-cluster.yaml new file mode 100644 index 0000000000..cb8e321237 --- /dev/null +++ b/docs/guides/druid/volume-expansion/yamls/druid-cluster.yaml @@ -0,0 +1,34 @@ +apiVersion: kubedb.com/v1alpha2 +kind: Druid +metadata: + name: druid-cluster + namespace: demo +spec: + version: 28.0.1 + deepStorage: + type: s3 + configSecret: + name: deep-storage-config + topology: + historicals: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + middleManagers: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageType: Durable + routers: + replicas: 1 + deletionPolicy: Delete + diff --git a/docs/guides/druid/volume-expansion/yamls/volume-expansion-ops.yaml b/docs/guides/druid/volume-expansion/yamls/volume-expansion-ops.yaml new file mode 100644 index 0000000000..b5ad80546b --- /dev/null +++ b/docs/guides/druid/volume-expansion/yamls/volume-expansion-ops.yaml @@ -0,0 +1,13 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: DruidOpsRequest +metadata: + name: dr-volume-exp + namespace: demo +spec: + type: VolumeExpansion + databaseRef: + name: druid-cluster + volumeExpansion: + historicals: 2Gi + middleManagers: 2Gi + mode: Offline \ No newline at end of file