Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Commit

Permalink
feature/en289 Add a one-volume option for persistence using subPath (#…
Browse files Browse the repository at this point in the history 
…301)

Signed-off-by: Michael Dent <[email protected]>
Co-authored-by: Michael Dent <[email protected]>
  • Loading branch information
@emrge-michaeld
emrge-michaeld and Michael Dent authored Jun 15, 2023
1 parent 944db6d commit f296a8e
Show file tree
Hide file tree
Showing 3 changed files with 101 additions and 25 deletions.
9 changes: 6 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ The following items can be set via `--set` flag during installation or configure

#### Configure how to persist data

- **Disable**: The data does not survive the termination of a pod.
- **Persistent Volume Claim(default)**: A default `StorageClass` is needed in the Kubernetes cluster to dynamically provision the volumes. Specify another StorageClass in the `storageClass` or set `existingClaim` if you have already existing persistent volumes to use.
- **Disable(default)**: The data does not survive the termination of a pod.
- **Persistent Volume Claim**: Enable persistence so that data survives termination of the pod. There is the choice of using one large persistent volume (using subPath) or seven separate persistent volumes for config, data, logs, repos, etc.
A default `StorageClass` is needed in the Kubernetes cluster to dynamically provision the volumes. Specify another StorageClass in the `persistence.storageClass` setting.

#### Configure authentication

Expand Down Expand Up @@ -184,14 +185,16 @@ The following table lists the configurable parameters of the nifi chart and the
| `persistence.enabled` | Use persistent volume to store data | `false` |
| `persistence.storageClass` | Storage class name of PVCs (use the default type if unset) | `nil` |
| `persistence.accessMode` | ReadWriteOnce or ReadOnly | `[ReadWriteOnce]` |
| `persistence.subPath.enabled` | Use only one persistent volume with subPath instead of seven separate persistent volumes | `false` |
| `persistence.subPath.name` | Name of the one persistent volume claim when using subPath | `data` |
| `persistence.subPath.size` | Size of the one persistent volume claim when using subPath | `36Gi` |
| `persistence.configStorage.size` | Size of persistent volume claim | `100Mi` |
| `persistence.authconfStorage.size` | Size of persistent volume claim | `100Mi` |
| `persistence.dataStorage.size` | Size of persistent volume claim | `1Gi` |
| `persistence.flowfileRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.contentRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.provenanceRepoStorage.size` | Size of persistent volume claim | `10Gi` |
| `persistence.logStorage.size` | Size of persistent volume claim | `5Gi` |
| `persistence.existingClaim` | Use an existing PVC to persist data | `nil` |
| **jvmMemory** |
| `jvmMemory` | bootstrap jvm size | `2g` |
| **SideCar** |
Expand Down
106 changes: 85 additions & 21 deletions templates/statefulset.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -453,20 +453,55 @@ spec:
tcpSocket:
port: {{ .Values.properties.httpsPort }}
volumeMounts:
- name: "logs"
mountPath: /opt/nifi/nifi-current/logs
- name: "data"
mountPath: /opt/nifi/data
- name: "auth-conf"
mountPath: /opt/nifi/nifi-current/auth-conf/
- name: "config-data"
mountPath: /opt/nifi/nifi-current/config-data
- name: "flowfile-repository"
mountPath: /opt/nifi/flowfile_repository
- name: "content-repository"
mountPath: /opt/nifi/content_repository
- name: "provenance-repository"
mountPath: /opt/nifi/provenance_repository
- mountPath: /opt/nifi/nifi-current/logs
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
- mountPath: /opt/nifi/data
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: data
{{- else }}
name: "data"
{{- end }}
- mountPath: /opt/nifi/nifi-current/auth-conf/
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: auth-conf
{{- else }}
name: "auth-conf"
{{- end }}
- mountPath: /opt/nifi/nifi-current/config-data
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: config-data
{{- else }}
name: "config-data"
{{- end }}
- mountPath: /opt/nifi/flowfile_repository
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: flowfile-repository
{{- else }}
name: "flowfile-repository"
{{- end }}
- mountPath: /opt/nifi/content_repository
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: content-repository
{{- else }}
name: "content-repository"
{{- end }}
- mountPath: /opt/nifi/provenance_repository
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: provenance-repository
{{- else }}
name: "provenance-repository"
{{- end }}
- name: "bootstrap-conf"
mountPath: /opt/nifi/nifi-current/conf/bootstrap.conf
subPath: "bootstrap.conf"
Expand Down Expand Up @@ -541,8 +576,13 @@ spec:
resources:
{{ toYaml .Values.logresources | indent 10 }}
volumeMounts:
- name: logs
mountPath: /var/log
- mountPath: /var/log
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
- name: bootstrap-log
imagePullPolicy: {{ .Values.sidecar.imagePullPolicy | default "Always" | quote }}
image: "{{ .Values.sidecar.image }}:{{ .Values.sidecar.tag }}"
Expand All @@ -553,8 +593,13 @@ spec:
resources:
{{ toYaml .Values.logresources | indent 10 }}
volumeMounts:
- name: logs
mountPath: /var/log
- mountPath: /var/log
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
- name: user-log
imagePullPolicy: {{ .Values.sidecar.imagePullPolicy | default "Always" | quote }}
image: "{{ .Values.sidecar.image }}:{{ .Values.sidecar.tag }}"
Expand All @@ -565,8 +610,13 @@ spec:
resources:
{{ toYaml .Values.logresources | indent 10 }}
volumeMounts:
- name: logs
mountPath: /var/log
- mountPath: /var/log
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
name: {{ .Values.persistence.subPath.name }}
subPath: logs
{{- else }}
name: "logs"
{{- end }}
{{- if .Values.certManager.enabled }}
- name: cert-manager
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
Expand Down Expand Up @@ -786,7 +836,21 @@ spec:
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | indent 6 }}
{{- end }}
{{- if .Values.persistence.enabled }}
{{- if and .Values.persistence.enabled .Values.persistence.subPath.enabled }}
volumeClaimTemplates:
- metadata:
name: {{ .Values.persistence.subPath.name }}
spec:
accessModes:
{{- range .Values.persistence.accessModes }}
- {{ . | quote }}
{{- end }}
storageClassName: {{ .Values.persistence.storageClass | quote }}
resources:
requests:
storage: {{ .Values.persistence.subPath.size }}
{{- end }}
{{- if and .Values.persistence.enabled (not .Values.persistence.subPath.enabled) }}
volumeClaimTemplates:
- metadata:
name: logs
Expand Down
11 changes: 10 additions & 1 deletion values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,16 @@ persistence:
# The default storage class is used if this variable is not set.

accessModes: [ReadWriteOnce]
## Storage Capacities for persistent volumes

## Use subPath and have 1 persistent volume instead of 7 volumes - use when your k8s nodes have limited volume slots, to limit waste of space,
## or your available volume sizes are quite large
# The one disk will have a directory folder for each volumeMount, but this is hidden. Run 'mount' to view each mount.
subPath:
enabled: false
name: data
size: 30Gi

## Storage Capacities for persistent volumes (these are ignored if using one volume with subPath)
configStorage:
size: 100Mi
authconfStorage:
Expand Down

0 comments on commit f296a8e

Please sign in to comment.