Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add datalake network to clickhouse #339

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions analytics-datastore-clickhouse/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ services:
public:
reverse-proxy:
default:
datalake:


volumes:
Expand Down
12 changes: 6 additions & 6 deletions datalake/docker-compose.cluster.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ services:
minio-02:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" --address ":9090" http://minio-0{1...4}/data{1...2}'
drizzentic marked this conversation as resolved.
Show resolved Hide resolved
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
test: [ "CMD", "curl", "-f", "http://localhost:9090/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
Expand All @@ -32,12 +32,12 @@ services:
minio-03:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" --address ":9090" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
test: [ "CMD", "curl", "-f", "http://localhost:9090/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
Expand All @@ -54,12 +54,12 @@ services:
minio-04:
image: ${MINIO_IMAGE}
entrypoint: sh
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" http://minio-0{1...4}/data{1...2}'
command: -c 'mkdir -p /data1/loki /data2/loki && minio server --console-address ":9001" --address ":9090" http://minio-0{1...4}/data{1...2}'
environment:
MINIO_ROOT_USER: ${MO_SECURITY_ADMIN_USER}
MINIO_ROOT_PASSWORD: ${MO_SECURITY_ADMIN_PASSWORD}
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
test: [ "CMD", "curl", "-f", "http://localhost:9090/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
Expand Down
2 changes: 1 addition & 1 deletion monitoring/loki/loki-config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ storage_config:
resync_interval: 30s
shared_store: s3
aws:
s3: http://${MO_SECURITY_ADMIN_USER}:${MO_SECURITY_ADMIN_PASSWORD}@minio-01.:9000/loki
s3: http://${MO_SECURITY_ADMIN_USER}:${MO_SECURITY_ADMIN_PASSWORD}@minio-01.:9090/loki
s3forcepathstyle: true
drizzentic marked this conversation as resolved.
Show resolved Hide resolved

compactor:
Expand Down
2 changes: 1 addition & 1 deletion openfn/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ services:
- WORKER_SECRET=${OPENFN_WORKER_SECRET}
- KAFKA_TRIGGERS_ENABLED=${OPENFN_KAFKA_TRIGGERS_ENABLED}
healthcheck:
test: '${ DOCKER_WEB_HEALTHCHECK_TEST:-curl localhost:4000/health_check}'
test: '${DOCKER_WEB_HEALTHCHECK_TEST:-curl localhost:4000/health_check}'
interval: '10s'
timeout: '3s'
start_period: '5s'
Expand Down
10 changes: 5 additions & 5 deletions test/cucumber/features/cluster-mode/clickhouse.cluster.feature
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,16 @@ Feature: Analytics Datastore Clickhouse?
When I launch the platform with params
Then The service "analytics-datastore-clickhouse-01" should be started with 1 replica
And The service "analytics-datastore-clickhouse-01" should be connected to the networks
| clickhouse_public | clickhouse_default |
| clickhouse_public | clickhouse_default | minio_public |
And The service "analytics-datastore-clickhouse-02" should be started with 1 replica
And The service "analytics-datastore-clickhouse-02" should be connected to the networks
| clickhouse_public | clickhouse_default |
| clickhouse_public | clickhouse_default | minio_public |
And The service "analytics-datastore-clickhouse-03" should be started with 1 replica
And The service "analytics-datastore-clickhouse-03" should be connected to the networks
| clickhouse_public | clickhouse_default |
| clickhouse_public | clickhouse_default | minio_public |
And The service "analytics-datastore-clickhouse-04" should be started with 1 replica
And The service "analytics-datastore-clickhouse-04" should be connected to the networks
| clickhouse_public | clickhouse_default |
| clickhouse_public | clickhouse_default | minio_public |
And The service "clickhouse-config-importer" should be removed
And There should be 4 services

Expand All @@ -30,4 +30,4 @@ Feature: Analytics Datastore Clickhouse?
And There should be 0 volume
And There should be 0 config
And There should not be network
| clickhouse_public |
| clickhouse_public | minio_public |
9 changes: 0 additions & 9 deletions test/cucumber/features/cluster-mode/monitoring.feature
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,7 @@ Feature: Monitoring package?
And The service "node-exporter" should be started with 3 replica
And The service "loki" should be started with 1 replica
And The service "promtail" should be started with 3 replica
And The service "minio-01" should be started with 1 replica
And The service "minio-01" should be connected to the networks
| reverse-proxy_public | monitoring_default |
And The service "minio-02" should be started with 1 replica
And The service "minio-03" should be started with 1 replica
And The service "minio-04" should be started with 1 replica
And The service "prometheus_backup" should be started with 1 replica
And The service "prometheus_backup" should be connected to the networks
| prometheus_public | monitoring_default |
Expand All @@ -33,10 +28,6 @@ Feature: Monitoring package?
And The service "node-exporter" should be removed
And The service "loki" should be removed
And The service "promtail" should be removed
And The service "minio-01" should be removed
And The service "minio-02" should be removed
And The service "minio-03" should be removed
And The service "minio-04" should be removed
And The service "prometheus_backup" should be removed
And There should be 0 service
And There should be 0 volume
Expand Down
4 changes: 2 additions & 2 deletions test/cucumber/features/single-mode/clickhouse.feature
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Feature: Analytics Datastore Clickhouse?
When I launch the platform with params
Then The service "analytics-datastore-clickhouse" should be started with 1 replica
And The service "analytics-datastore-clickhouse" should be connected to the networks
| clickhouse_public | clickhouse_default |
| clickhouse_public | clickhouse_default | minio_public |
And The service "clickhouse-config-importer" should be removed
And There should be 1 service
And There should be 1 volume
Expand All @@ -19,4 +19,4 @@ Feature: Analytics Datastore Clickhouse?
And There should be 0 volume
And There should be 0 config
And There should not be network
| clickhouse_public |
| clickhouse_public | minio_public |
6 changes: 1 addition & 5 deletions test/cucumber/features/single-mode/monitoring.feature
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,11 @@ Feature: Monitoring package?
And The service "cadvisor" should have healthy containers
And The service "loki" should be started with 1 replica
And The service "promtail" should be started with 1 replica
And The service "minio-01" should be started with 1 replica
And The service "grafana" should be connected to the networks
| reverse-proxy_public | keycloak_public | monitoring_default |
And The service "prometheus" should be connected to the networks
| prometheus_public | monitoring_default |
And The service "minio-01" should be connected to the networks
| reverse-proxy_public | monitoring_default |
And There should be 6 volumes
And There should be 3 volumes

Scenario: Destroy Monitoring package
Given I use parameters "package destroy -n=monitoring --dev --env-file=.env.local"
Expand All @@ -29,7 +26,6 @@ Feature: Monitoring package?
And The service "node-exporter" should be removed
And The service "loki" should be removed
And The service "promtail" should be removed
And The service "minio-01" should be removed
And There should be 0 service
And There should be 0 volume
And There should be 0 config
Expand Down
3 changes: 1 addition & 2 deletions test/cucumber/features/single-mode/recipe.feature
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,9 @@ Feature: CDR-DW recipe?
And The service "cadvisor" should have healthy containers
And The service "loki" should be started with 1 replica
And The service "promtail" should be started with 1 replica
And The service "minio-01" should be started with 1 replica
And The service "dashboard-visualiser-superset" should be started with 1 replica
And The service "analytics-datastore-clickhouse" should be started with 1 replica
And The service "openfn" should be started with 1 replica
And The service "minio-01" should be started with 1 replica

Scenario: Send Fhir bundle and store the clinical data in the Fhir datastore, the patient info in the CR
Given I have configured the cdr
Expand Down
Loading