Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Figure out abctl #110

Closed
fatchat opened this issue Nov 29, 2024 · 13 comments
Closed

Figure out abctl #110

fatchat opened this issue Nov 29, 2024 · 13 comments
Assignees
@fatchat

Comments

@fatchat
Copy link

fatchat commented Nov 29, 2024

Topic

No response

Relevant information

No response
@fatchat
Copy link
Author

fatchat commented Nov 29, 2024 •
edited by github-actions bot
Loading

  1. First, stop your current Airbyte instance that's running with run-ab-platform.

  2. Install abctl by following the instructions in the OSS Quickstart guide.

  3. If you want to migrate your existing data from the Docker Compose setup to abctl, you can use the --migrate flag when installing with abctl. Run the following command:

abctl local install --migrate

This will migrate your existing data to the new abctl setup Migrating from Docker Compose.

Upgrading with abctl

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat fatchat added this to Dalgo Nov 29, 2024
@fatchat fatchat moved this to In Progress in Dalgo Nov 29, 2024
@fatchat fatchat self-assigned this Nov 29, 2024
@fatchat
Copy link
Author

fatchat commented Nov 29, 2024 •
edited by github-actions bot
Loading 

ubuntu@ip-172-31-19-164:~$ abctl local install --migrate

  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /home/ubuntu/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 26.1.3
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created
 SUCCESS  Port 8000 appears to be available

Creating cluster airbyte-abctl

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Nov 29, 2024 •
edited by github-actions bot
Loading 

SUCCESS  Cluster 'airbyte-abctl' created
  INFO    Pulling image airbyte/bootloader:1.2.0
  INFO    Pulling image airbyte/connector-builder-server:1.2.0
  INFO    Pulling image airbyte/connector-sidecar:1.2.0
  INFO    Pulling image airbyte/container-orchestrator:1.2.0
  INFO    Pulling image airbyte/cron:1.2.0
  INFO    Pulling image airbyte/db:1.2.0
  INFO    Pulling image airbyte/mc:latest
  INFO    Pulling image airbyte/server:1.2.0
  INFO    Pulling image airbyte/webapp:1.2.0
  INFO    Pulling image airbyte/worker:1.2.0
  INFO    Pulling image airbyte/workload-api-server:1.2.0
  INFO    Pulling image airbyte/workload-init-container:1.2.0
  INFO    Pulling image airbyte/workload-launcher:1.2.0
  INFO    Pulling image bitnami/kubectl:1.28.9
  INFO    Pulling image busybox:1.35
  INFO    Pulling image busybox:latest
  INFO    Pulling image curlimages/curl:8.1.1
  INFO    Pulling image minio/minio:RELEASE.2023-11-20T22-40-07Z
  INFO    Pulling image temporalio/auto-setup:1.23.0

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Nov 29, 2024 •
edited by github-actions bot
Loading 

$ find .airbyte/
.airbyte/
.airbyte/abctl
.airbyte/abctl/.helmrepo
.airbyte/abctl/abctl.kubeconfig
.airbyte/abctl/data
.airbyte/abctl/.helmcache
.airbyte/abctl/.helmcache/airbyte-index.yaml
.airbyte/abctl/.helmcache/airbyte-1.2.0.tgz
.airbyte/abctl/.helmcache/airbyte-charts.txt
.airbyte/analytics.yml

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Nov 29, 2024 •
edited by github-actions bot
Loading 

  INFO    Namespace 'airbyte-abctl' created
  INFO    Persistent volume 'airbyte-minio-pv' created
  INFO    Persistent volume 'airbyte-volume-db' created
  ERROR   Failed to migrate data from previous Airbyte installation
  ERROR   Unable to install Airbyte locally
  ERROR   unable to migrate data from previous airbyte installation: unable to update postgres user: container '61f24da9f5fe5083a32995ea33ba2658891184efc56a80223d0137aff4fd29a5' exec exited with non-zero exit code: 2

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Nov 29, 2024 •
edited by github-actions bot
Loading

When using abctl to migrate an existing Airbyte installation with a remote database, you need to provide the database connection information. The --migrate flag alone isn't sufficient in your case. Instead, you should use a values.yaml file to customize your installation and specify the external database details.

Here's how you can proceed:

  1. Create a values.yaml file with your database connection information. You can use this template as a starting point:
global:
  database:
    type: external
    host: "your-database-host"
    port: 5432
    database: "your-database-name"
    user: "your-database-user"
    secretName: "airbyte-config-secrets"
    passwordSecretKey: "database-password"
  1. Create a secrets.yaml file to store your database password securely:
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  database-password: "your-database-password"
  1. Now, you can install Airbyte with abctl using these configuration files:

abctl local install --values ./values.yaml --secret ./secrets.yaml --migrate

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Nov 29, 2024 •
edited by github-actions bot
Loading

kubectl --kubeconfig=/home/ubuntu/.airbyte/abctl/abctl.kubeconfig [COMMAND]

also running abctl local install using sudo

still seeing errors though
Error: couldn't find key airbyte062 in Secret airbyte-abctl/airbyte-config-secrets

the secrets available are

NAME                                  TYPE                 DATA   AGE
airbyte-abctl-airbyte-secrets         Opaque               5      3m30s
airbyte-config-secrets                Opaque               1      11m
sh.helm.release.v1.airbyte-abctl.v1   helm.sh/release.v1   1      11m
sh.helm.release.v1.airbyte-abctl.v2   helm.sh/release.v1   1      3m30s

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Dec 1, 2024 •
edited by github-actions bot
Loading

Oh we're not supposed to use --migrate with an external database

Running abctl local install --values ./values.yaml --secret ./secrets.yaml

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Dec 1, 2024 •
edited by github-actions bot
Loading

that failed. the secrets.yaml was incorrect, containing the key stringData.database-password instead of stringData.airbyte062 (where airbyte062 is the value for the key global.database.passwordSecretKey in values.yaml)

retrying

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Dec 2, 2024 •
edited by github-actions bot
Loading

still failed with

Encountered an issue deploying Airbyte:
            Pod: airbyte-db-0.180d1b38827ccfdc
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(56723dff-2fb2-4cd4-90db-29a0189efb37)
            Count: 12
            Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied
          chown: /var/lib/postgresql/data/pgdata: No such file or directory
          mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Dec 2, 2024 •
edited by github-actions bot
Loading

attempting with local db

abctl local install

this succeeded with

  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /home/ubuntu/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 26.1.3
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created
 SUCCESS  Port 8000 appears to be available
 SUCCESS  Cluster 'airbyte-abctl' created
  INFO    Pulling image airbyte/bootloader:1.2.0
  INFO    Pulling image airbyte/connector-builder-server:1.2.0
  INFO    Pulling image airbyte/connector-sidecar:1.2.0
  INFO    Pulling image airbyte/container-orchestrator:1.2.0
  INFO    Pulling image airbyte/cron:1.2.0
  INFO    Pulling image airbyte/db:1.2.0
  INFO    Pulling image airbyte/mc:latest
  INFO    Pulling image airbyte/server:1.2.0
  INFO    Pulling image airbyte/webapp:1.2.0
  INFO    Pulling image airbyte/worker:1.2.0
  INFO    Pulling image airbyte/workload-api-server:1.2.0
  INFO    Pulling image airbyte/workload-init-container:1.2.0
  INFO    Pulling image airbyte/workload-launcher:1.2.0
  INFO    Pulling image bitnami/kubectl:1.28.9
  INFO    Pulling image busybox:1.35
  INFO    Pulling image busybox:latest
  INFO    Pulling image curlimages/curl:8.1.1
  INFO    Pulling image minio/minio:RELEASE.2023-11-20T22-40-07Z
  INFO    Pulling image temporalio/auto-setup:1.23.0
  INFO    Namespace 'airbyte-abctl' created
  INFO    Persistent volume 'airbyte-minio-pv' created
  INFO    Persistent volume 'airbyte-volume-db' created
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' created
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created
  INFO    Starting Helm Chart installation of 'airbyte/airbyte' (version: 1.2.0)
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-7d4d454fd-b98jv.180d3a69aea886e1
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.14:8001/api/v1/health": dial tcp 10.244.0.14:8001: connect: connection refused
            Count: 6
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-7d4d454fd-b98jv.180d3a69aea886e1
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.14:8001/api/v1/health": dial tcp 10.244.0.14:8001: connect: connection refused
            Count: 7
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-7d4d454fd-b98jv.180d3a69aea886e1
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.14:8001/api/v1/health": dial tcp 10.244.0.14:8001: connect: connection refused
            Count: 8
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-abctl-server-7d4d454fd-b98jv.180d3a69aea886e1
            Reason: Unhealthy
            Message: Readiness probe failed: Get "http://10.244.0.14:8001/api/v1/health": dial tcp 10.244.0.14:8001: connect: connection refused
            Count: 9
 SUCCESS  Installed Helm Chart airbyte/airbyte:
            Name: airbyte-abctl
            Namespace: airbyte-abctl
            Version: 1.2.0
            AppVersion: 1.2.0
            Release: 1
  INFO    Starting Helm Chart installation of 'nginx/ingress-nginx' (version: 4.11.3)
 SUCCESS  Installed Helm Chart nginx/ingress-nginx:
            Name: ingress-nginx
            Namespace: ingress-nginx
            Version: 4.12.0-beta.0
            AppVersion: 1.12.0-beta.0
            Release: 1
  INFO    No existing Ingress found, creating one
 SUCCESS  Ingress created
 WARNING  Failed to launch web-browser.
          Please launch your web-browser to access http://localhost:8000
 SUCCESS  Airbyte installation complete.
            A password may be required to login. The password can by found by running
            the command abctl local credentials

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Dec 2, 2024 •
edited by github-actions bot
Loading

kubectl --kubeconfig /home/ubuntu/.airbyte/abctl/abctl.kubeconfig get pods --namespace airbyte-abctl

NAME                                                      READY   STATUS      RESTARTS       AGE
airbyte-abctl-airbyte-bootloader                          0/1     Completed   0              4m42s
airbyte-abctl-connector-builder-server-67d5c749dd-6dtr8   1/1     Running     0              4m7s
airbyte-abctl-cron-6d7498fb94-zjvln                       1/1     Running     0              4m7s
airbyte-abctl-pod-sweeper-544d944547-vkjvt                1/1     Running     0              4m7s
airbyte-abctl-server-7d4d454fd-b98jv                      1/1     Running     1 (3m6s ago)   4m7s
airbyte-abctl-temporal-6c49f7bd99-krbwd                   1/1     Running     0              4m7s
airbyte-abctl-webapp-7fcfb9659b-z4zn7                     1/1     Running     0              4m7s
airbyte-abctl-worker-864f9c48fc-d4gk6                     1/1     Running     1 (3m5s ago)   4m7s
airbyte-abctl-workload-api-server-8d9f7c98b-6q5t9         1/1     Running     0              4m7s
airbyte-abctl-workload-launcher-698bff4d45-klmqh          1/1     Running     1 (3m5s ago)   4m7s
airbyte-db-0                                              1/1     Running     0              4m43s
airbyte-minio-0                                           1/1     Running     0              4m43s

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat
Copy link
Author

fatchat commented Dec 10, 2024 •
edited by github-actions bot
Loading

abctl is only needed for running Airbyte on our dev laptops. When we do this we won't connect to a remote database

So all we need is e.g.

abctl local install --chart-version 0.398.0

with the right helm chart version. Versions can be found by looking through the output of

helm search repo airbyte --versions

, there will be multiple helm chart versions for each version of airbyte

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>

@fatchat fatchat closed this as completed Dec 10, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Dalgo Dec 10, 2024
@fatchat fatchat moved this from Done to R&D in Dalgo Dec 10, 2024
@fatchat fatchat changed the title Upgrade Airbyte to 0.62.2 Figure out abctl Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fatchat fatchat

Labels
None yet
Projects
Dalgo
Status: R&D
Milestone
No milestone
Development

No branches or pull requests
1 participant
@fatchat