Skip to content

Latest commit

 

History

History
106 lines (67 loc) · 4.83 KB

getting-started-tutorial.md

File metadata and controls

106 lines (67 loc) · 4.83 KB
Raw
description
Start syncing data in minutes with Airbyte

Getting Started

Let's see how you can spin up a local instance of Airbyte and syncing data from one Postgres database to another.

Here's a 6-minute video showing you how you can do it.

{% embed url="https://www.youtube.com/watch?v=Rcpt5SVsMpk" caption="" %}

First of all, make sure you have Docker and Docker Compose installed. Then run the following commands:

git clone https://github.com/airbytehq/airbyte.git
cd airbyte
docker-compose up

Once you see an Airbyte banner, the UI is ready to go at http://localhost:8000/.

1. Set up your preferences

You should see an onboarding page. Enter your email if you want updates about Airbyte and continue.

2. Set up your first connection

Now you will see a wizard that allows you choose the data you want to send through Airbyte.

As of our alpha launch, we have one database source (Postgres) and two API sources (an exchange rate API and the Stripe API). We're currently building an integration framework that makes it easy to create sources and destinations, so you should expect many more soon. Please reach out to us if you need a specific integration or would like to help build one.

For now, we will start out with a Postgres source and destination.

In the case you don't have a readily available database to sync

Before we configure anything in the UI, we need databases and data. Run the following commands in a new terminal window to start backgrounded source and destination databases:

docker run --rm --name airbyte-source -e POSTGRES_PASSWORD=password -p 2000:5432 -d postgres
docker run --rm --name airbyte-destination -e POSTGRES_PASSWORD=password -p 3000:5432 -d postgres

Add a table with a few rows to the source database:

docker exec -it airbyte-source psql -U postgres -c "CREATE TABLE users(id SERIAL PRIMARY KEY, col1 VARCHAR(200));"
docker exec -it airbyte-source psql -U postgres -c "INSERT INTO public.users(col1) VALUES('record1');"
docker exec -it airbyte-source psql -U postgres -c "INSERT INTO public.users(col1) VALUES('record2');"
docker exec -it airbyte-source psql -U postgres -c "INSERT INTO public.users(col1) VALUES('record3');"

Return to the UI and configure a source Postgres database. Use the name airbyte-source for the name and Postgresas the type. Fill in the configuration fields as follows:

Host: localhost
Port: 2000
User: postgres
Password: password
DB Name: postgres

Click on Set Up Source and the wizard should move on to allow you to configure a destination. We currently support BigQuery, Postgres, and file output for debugging, but Redshift, Snowflake, and other destinations are coming soon. For now, configure the destination Postgres database:

Host: localhost
Port: 3000
User: postgres
Password: password
DB Name: postgres

After adding the destination, you can choose what tables and columns you want to sync.

For this demo, we recommend leaving the defaults and selecting "Every 5 Minutes" as the frequency. Click Set Up Connection to finish setting up the sync.

3. Check the logs of your first sync

You should now see a list of sources with the source you just added. Click on it to find more information about your connection. This is the page where you can update any settings about this source and how it syncs. There should be a Completed job under the history section. If you click on that run, it will show logs from that run.

One of biggest problems we've seen in tools like Fivetran is the lack of visibility when debugging. In Airbyte, allowing full log access and the ability to debug and fix integration problems is one of our highest priorities. We'll be working hard to make these logs accessible and understandable.

4. Check if the syncing actually worked

Now let's verify that this worked. Let's output the contents of the destination db:

docker exec airbyte-destination psql -U postgres -c "SELECT * FROM public.users;"

You should see the rows from the source database inside the destination database!

And there you have it. You've taken data from one database and replicated it to another. All of the actual configuration for this replication only took place in the UI.

That's it! This is just the beginning of Airbyte. If you have any questions at all, please reach out to us on Slack. We’re still in alpha, so if you see any rough edges or want to request an integration you need, please create an issue on our Github or leave a thumbs up on an existing issue.

Thank you and we hope you enjoy using Airbyte.