Moving Dalgo airbyte to kubernetes

[Ishankoradia]

Dec 5, 2024

Experiment

Problem/Challenge
Problem at hand was to figure out why airbyte is not saving sync (job) logs to s3. The syncs run successful but the logs come up empty.

What we tried
I made a minor change in values.yaml and wanted to see if that helped. I ran the command helm upgrade --install airbyte airbyte/airbyte -n airbyte --values values0.62.0.yaml --version 0.62.0 --description "" and we ran into a series of issues

• It complained about the role and rolebinding Error: UPGRADE FAILED: Unable to continue with update: Role "airbyte-admin-role" in namespace "airbyte" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "airbyte"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "airbyte". So we manually deleted the role and rolebinding resource. Also we set create service account to false

serviceAccount:
  # -- Specifies whether a ServiceAccount should be created
  create: false
  # -- Annotations for service account. Evaluated as a template. Only used if `create` is `true`.
  annotations: {}
  # -- Name of the service account to use. If not set and create is true, a name is generated using the fullname template.
  name: *service-account-name

• After that we ran into, this issue that complained about env keys being invalid. UPGRADE FAILED: failed to create patch: The order in patch list: . Looked like helm was not able to patch the changes in values.yaml we did for some reason.

We decided to uninstall and install airbyte again

After doing this we ran into some other issue which is an open issue in airbyte . java.lang.IllegalStateException: baseUrl is invalid.

We tried a couple of things for this.

• Update the airbyte helm repo by running helm repo update since people mentioned issues with chart version. This would fetch the latest chart versions of different airbyte services. After doing this we saw some weird behaviour.

  • An old connection (lets say A) stopped sycning. It didn't even reach the worker.
  • We created a new connection B, which synced successfully.
  • Then we tried syncing A again and it worked now.
    This didn't resolve

• The airbyte issue attached above does say that version 0.62.0 has this issue and upgrading might work. For some people downgrading to 0.60.1 worked.

• Tried creating a fresh bucket and using that in airbyte-config-secrets.yaml but nada.

[Ishankoradia]

Dec 6, 2024

What we tried

• Airbyte logs are still not syncing to the s3 bucket. We tried giving the Node group role S3FullAccess. But that didn't work too.

Debugging

• airbyte-server pod logs show this trail.

io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 18 and attempt no: 0
	at io.airbyte.commons.server.handlers.AttemptHandler.getAttemptCombinedStats(AttemptHandler.java:231) ~[io.airbyte-airbyte-commons-server-0.62.0.jar:?]
	at io.airbyte.server.apis.AttemptApiController.lambda$getAttemptCombinedStats$2(AttemptApiController.java:69) ~[io.airbyte-airbyte-server-0.62.0.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:28) ~[io.airbyte-airbyte-server-0.62.0.jar:?]
	at io.airbyte.server.apis.AttemptApiController.getAttemptCombinedStats(AttemptApiController.java:69) ~[io.airbyte-airbyte-server-0.62.0.jar:?]
	at io.airbyte.server.apis.$AttemptApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.62.0.jar:?]
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461) ~[micronaut-inject-4.4.10.jar:4.4.10]
	at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4232) ~[micronaut-inject-4.4.10.jar:4.4.10]
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:271) ~[micronaut-router-4.4.10.jar:4.4.10]
	at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:488) ~[micronaut-http-server-4.4.10.jar:4.4.10]
	at io.micronaut.http.server.RouteExecutor.lambda$callRoute$6(RouteExecutor.java:465) ~[micronaut-http-server-4.4.10.jar:4.4.10]
	at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87) ~[micronaut-core-4.4.10.jar:4.4.10]
	at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211) ~[micronaut-core-4.4.10.jar:4.4.10]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-12-05 11:13:15 ERROR i.a.c.s.e.IdNotFoundExceptionHandler(handle):31 - Not found exception

• airbyte-workload-launcher pods has two errors.

2024-12-05 11:12:25 ERROR i.a.a.c.c.ClientSupportFactory(generateDefaultRetryPolicy$lambda$7):147 - Retry attempts exceeded.
java.net.UnknownHostException: airbyte-workload-api-server-svc: Name or service not known

2024-12-05 11:12:25 ERROR i.a.a.c.c.ClientSupportFactory(generateDefaultRetryPolicy$lambda$3):123 - Log message invocation failed: java.lang.NullPointerException: Cannot invoke "okhttp3.Response.request()" because the return value of "dev.failsafe.event.ExecutionCompletedEvent.getResult()" is null

[Ishankoradia]

Dec 9, 2024

Problem/Challenge
Problem at hand was to figure out why airbyte is not saving sync (job) logs to s3. The syncs run successful but the logs come up empty.

What we tried

• Another change we made was under the global section where storage type was changed from s3 to S3.

• Debugging through kubectl and looking at various pods. We first saw this issue airbyte-workload-api-server-svc: Name or service not known. We tried to describe the workfload-launcher pod via kubectl describe pod airbyte-launcher-<id> -n airbyte and found that somewhere at the top , image version was of the latest airbyte version 1.2.0

Image: airbyte/workload-launcher:1.2.0

• We realized that the helm chart versions was not according to the airbyte version we were trying to install. And the culprint was this command helm install airbyte airbyte/airbyte --namespace airbyte --values values0.62.0.yaml. It didn't specify the helm chart version and so by default it pulled the latesst ones. Then running this command fixed the chart versions
  helm install airbyte airbyte/airbyte --namespace airbyte --values values0.62.0.yaml --version 0.62.0. We were under the assumption that version: 0.62.0 inside the values.yaml would be enough but clearly not. Helm requires that argument.

Unexpected things

• While fixing the --version for helm. We were trying to do a clean install and by mistake we had this in values.yaml version: 0.63.9 and so i upgraded out database also. And then we tried to rollback to version: 0.62.0, it clearly did not work. So we manually went and deleted all the tables public schemas from airbyte, temporal and temporal_visibility. We also had to run GRANT ALL privileges ON SCHEMA public to airbyte_user. For some reason, temporal still had some psql. data access exceptions, so dropped both the temporal dbs and let airbyte create it, which seem to work.

• We also had to increase our max connections limit on RDS to 120 before we could start airbyte with fresh db.

Coming back to the s3 logs problem

• After fixing the chart versions with a fresh database, we ran into the problem we have been seeing again and again. java.lang.IllegalStateException: baseUrl is invalid.. Airbyte's git issue. This error is seen in the airbyte-worker pod.

• We ran into the above error while creating a connection which failed with Schema discorvery failed in the UI.

• Many people in the above git issue have mentioned that 0.62.X has this issue and for some downgrading worked while for some upgrading worked.

• Our best bet now is to upgrade airbyte to 0.64.7. The last version before 1.0.0

Upgrading airbyte to 0.64.7

• Go to airbyte-platform repo and change the branch to 0.64.7.

• The template for values.yaml is at airbyte-platform/charts/airbyte/values.yaml. Download this and fill out the configuration values.

• Dropped the current dbs airbyte, temporal and temporal_visibility

• Ran the command helm install airbyte airbyte/airbyte --namespace airbyte --values values0.64.7.yaml --version 0.64.7 but got an error. INSTALLATION FAILED: chart "airbyte" matching 0.64.7 not found in airbyte index. (try 'helm repo update'): no chart version found for airbyte-0.64.7

• After search through helm chart repo for airbyte version vs chart versions helm search repo airbyte --versions. I found out the corresponding chart version and then ran helm install airbyte airbyte/airbyte --namespace airbyte --values values0.64.7.yaml --version 0.654.0. Got this error - Error: INSTALLATION FAILED: YAML parse error on airbyte/charts/server/templates/deployment.yaml: error converting YAML to JSON: yaml: line 214: could not find expected ':'
  This a known issue in airbyte , here.

Updated the storage secret from storageSecretName to secretName

• Ran into permission denied for schema public. Tried giving permission via GRANT USAGE ON SCHEMA public TO airbyte_user; but did not work. Then tried GRANT ALL PRIVILEGES on SCHEMA public to airbyte_user; and it worked. Airbyte is up.

• Airbyte came up but 0.62.0 came up. Bleh !!!

• Removed this from values.yaml version: 0.64.7 and did a fresh install. Looks like if you specify the chart version correctly, it picks up the right airbyte version. So basically --version is the chart version

• Airbyte 0.64.7 is up.

• Testing 0.64.7

• Create a postgres destination successfully.

• Create a gsheet source successuflly.

• Create a connection successfullly. No schema discover issues.

• The sync logs came up nicely.

• Sync ran successfully. Took 3mins to finish the sync which is reasonable.

• I also see the logs of the above test sync in our s3 bucket !!

[Ishankoradia]

Dec 10, 2024

Problem/Challenge
Although airbyte version 0.64.7 works nicely for us. The docker compose setup doesn't support this version and we can do a migration from docker setup to Kubernetes. Airbyte version 0.63.13 is the last version docker compose supports (slack bot mentioned 1.0 but clealy it was wrong)

So our plan/experiment is to run kubernetes airbyte on 0.63.13. Then upgrade docker compose setup on dalgo staging to 0.63.13 (its currently on 0.58.0). Then run an experiment where we switch docker compose setup with the kubernetes. If this is successful, it means we would upgrade production (currently on 0.58.0) to 0.63.13 and then do the switch over to Kuberentes.

We could do the above experiment on 0.58.0 also instead of 0.63.13. But we have faced some issues with 0.58.0 and are hoping we dont see them in 0.63.13

Switchin dalgo staging airbyte to kubernetes

• Setup kubernetes airbyte to run on airbyte version 0.63.13. Do a fresh setup, drop the dbs and follow setups in the previous command for upgrade.

• 0.63.13 has multiple chart versions. This is the latesst one 0.398.0. Used this command to install helm install airbyte airbyte/airbyte --namespace airbyte --values values0.63.13.yaml --version 0.398.0

• Airbyte came up without any errors.

• Values.yaml for 0.63.13 vs 0.64.7 has some diffs. Will check in the yaml file.

• Testing 0.63.13

• Create a postgres destination successfully.

• Create a gsheet source successuflly.

• Create a connection successfullly. No schema discover issues.

• The sync logs came up nicely.

• Sync ran successfully. Took 3mins to finish the sync which is reasonable.

• Upgraded the old staging to 0.63.13 using run-ab-platform.sh . Works nicely.

• Run this command helm list -n airbyte to figure out which is the current chart and/or airbyte version deployed in the namespace.

• Setup google secrets manager for eks staging 2. Since the old staging is running on gsm. This didn't work for airbyte version 0.63.13 with chart version 0.398.0. The workload-launcher pod runs into The secret not found in the store !. Going to try other chart versions of airbyte 0.63.13

• None of the chart versions of 0.63.13 including 0.398.0, 0.394.0, 0.390.0 & 0.383.1 work with google secret manager.

• Switch old staging (run-ab-platform.sh) to staging 2 (eks). Copy airbyte, temporal and temporal_visibility dbs to the new rds and try to bring up the eks airbyte [In Progress]

[fatchat]

Step 1: Run Airbyte 0.63.13 on staging

• upgrade staging-1 to airbyte 0.63.13 using run-ab-platform
• upgrade staging-2 to airbyte 0.63.13 using helm
• copy the airbyte and dalgo databases from staging-1 to the staging-2 RDS
• connect all dalgo components on staging-2 to these copied databases and run

at this point dalgo on staging-2 "looks just like" dalgo on staging-1 except that its airbyte is running on EKS

• test staging-1 on 0.63.13 for a week
• test staging-2 on 0.63.13 for a week

at this point we have tested airbyte 0.63.13 for one week, both using docker-compose as well as kubernetes. the databases would have diverged; we don't care

• (cleanup) take down staging-1, EC2 and RDS

Step 2: Run Airbyte 0.63.13 on production

• upgrade production-1 to airbyte 0.63.13 using run-ab-platform
• run production-2 on EKS using airbyte 0.63.13. no dalgo components, only airbyte

at this point we have two airbyte configuration databses with the same table schemas, ready to copy from the old one to the new one

Step 3: Upgrade staging on Kubernetes

• upgrade staging-2 airbyte from 0.63.13 to 0.64.7 using helm

Step 4: Switch Dalgo production to Kubernetes

• copy the airbyte and dalgo databases from production-1 to the production-2 RDS
• connect all dalgo components on production-2 to these copied databases and run

at this point dalgo production-2 "looks just like" dalgo production-1
note that production-1 MUST BE SHUT DOWN

at this point dalgo production is running with airbyte 0.63.13 on EKS

Step 5: Upgrade production on Kubernetes

• upgrade production-2 airbyte from 0.63.13 to 0.64.7 using helm

[fatchat]

Suggested schedule:

• Step 1 Dec 13
• Step 2 Dec 20
• Step 3 Dec 20
• Step 4 Jan 3 (???)
• Step 5 Jan 10

[fatchat]

0.63.13 doesn't connect to GSM, we saw issues with concurrent syncs on 0.64.7
going straight to 1.1 now
new plan will appear below

[Ishankoradia]

Dec 16/17, 2024

Building on Dec 10 2024's work

Switchin dalgo staging airbyte to kubernetes

• Our first approach was to figure out a version less than 0.63.20 (last ver where docker compose works) for which we can have a stable eks airbyte running.

• Tried many version of airbyte from 0.62.x to 0.63.x but either the gsm didn't work or the sync had a weird issue of 0 records syncing in a successful sync.

• Looks like our best bet would be to just go to 1.x. So now i am trying to start the airbyte 1.1.0 with the backup of staging airbyte which was on version 0.58.0

  • Created a new db airbyte in the RDS.
  • Restored the staging 0.58.0 backup to this. Made sure the airbyte_user role has all privelges to this restored db.
  • Fetched the values.yaml template from https://artifacthub.io/packages/helm/airbyte/airbyte/1.1.1 .
  • Updated it with the configuration we have been using for s3 / gsm /external db etc.
  • Ran helm install airbyte airbyte/airbyte --namespace airbyte --values values1.1.0.yaml
  • Airbyte came up successfully. It looks like it also upgraded the migrations but portfowarding the webapp gave this error External Postgres Database Setup Issues airbytehq/airbyte#49304
  • It also created temporal and temporal_visibility dbs by itself.
  • Since i didn't specify the version it tried to installed airbyte 1.3.1 - always default to the latest
  • Trying to run helm install airbyte airbyte/airbyte --namespace airbyte --values values1.1.0.yaml --version 1.1.1
  • Okay that worked. I can see all the configuration and workspaces we had from before on staging.
  • Created a new source, a new destination and ran a sync. I am able to see logs also.

[Ishankoradia]

Dec 18, 2024 - Present

Testing Staging 2 Running on airbyte eks 1.1.0 (migrated from 0.58.0)

• Ran a couple of old gsheet connection with biguqery and postgres destination. They synced perfectly fine in comparable amount of time as before
• Able to create a new source (gsheet)
• Able to create a new destination (postgres)
• Gsm works because all sources/destination from 0.58.0 were using the same gsm. And i checked the backend for a new destination and confirmed that the secret fields are being stored in the gsm.
• Logs are visible for new syncs but the history is lost since they were being pushed to docker volume in 0.58.0 and in 1.1.0 they go in s3
• Test kobotoolbox custom connector
  - Looks like the connector version was outdated on staging. Image tag was 0.2.0 but our docker hub has the latest one at 0.2.2. So creating the source failed.
  - I tried upgrading the connector from settings. Changed the docker tag from 0.2.0 to 0.2.2 but it didn't work. Error in server pod Failed to fetch registry entry for tech4dev/source-kobotoolbox:0.2.2
  - EKS nodes are running arm64 machines while the images we currenty have on our docker hub are of amd64 architecture. This could be the reason for fetch failing.
  - Next step is to build the arm64 image for each connector and push it to our docker hub.
  - One of our machine has the arm architecture. I followed these steps to build the docker image there
  - Cloned our airbyte repo.
  - Switched to the right branch for kobotoolbox connector.
  - Run docker buildx build -t tech4dev/source-kobotoolbox:0.2.3-aarch --platform linux/aarch64 . on the arm machine. This worked I am able to create the connector and sync the data.

[Ishankoradia]

• Test commcare connector
  • Some of airbyte-ci works for commcare connector for building images. I switched to the tech4dev/commcare branch.
  • Ran airbyte-ci connectors --name=source-commcare build --architecture=linux/arm64 --tag=0.2.0-aarch from the root directory of our airbyte repo
  • Ran docker tag airbyte/source-commcare:0.2.0-aarch tech4dev/source-commcare:0.2.0-aarch to change the repository to ours so we can push it
  • Ran docker push tech4dev/source-commcare:0.2.0-aarch
  • airbyte-ci is able to build arm64 images on an amd64 machine. I dont understand yet, how but its amazing.
  • Went to the airbyte workspace and updated tag from 0.1.0 to 0.2.0-aarch. This was successful
  • Create a source and a connection
  • Synced successfully. Checked the records in the warehouse.

[fatchat]

Step 1: Run Airbyte 1.1.0 on staging

• run staging-2 on airbyte 1.1.0 using helm and with the 0.58.0 config database
• copy the airbyte and dalgo databases from staging-1 to the staging-2 RDS
• connect all dalgo components on staging-2 to these copied databases and run

at this point dalgo on staging-2 "looks just like" dalgo on staging-1 except that its airbyte is running on EKS

• test staging-2 on 1.1.0 for a week

• test all custom connectors on staging-2

• connect our GitHub CI/CD to staging-2

• (cleanup) take down staging-1, EC2 and RDS

Goal: Dec 31 2024

Step 2: Run Airbyte 1.1.0 on production

• copy the airbyte and dalgo databases from production-1 to the production-2 RDS
• run production-2 on EKS using airbyte 1.1.0 with the copied database from 0.58.0
• run all dalgo components on production-2 with these copied databases

Goal: Jan 11 2024

[fatchat]

"test all custom connectors" is a bit of an unknown, if they don't work we will need to figure out why and fix