Skip to content

Latest commit

 

History

History
64 lines (44 loc) · 1.61 KB

full-refresh-append.md

File metadata and controls

64 lines (44 loc) · 1.61 KB

Full Refresh - Append

Overview

The Full Refresh modes are the simplest methods that Airbyte uses to sync data, as they always retrieve all available data requested from the source, regardless of whether it has been synced before. This contrasts with Incremental sync, which does not sync data that has already been synced before.

In the Append variant, new syncs will take all data from the sync and append it to the destination table. Therefore, if syncing similar information multiple times, every sync will create duplicates of already existing data.

Example Behavior

On the nth sync of a full refresh connection:

Add new data to the same table. Do not touch existing data.

data in the destination before the nth sync:

Languages
Python
Java

new data:

Languages
Python
Java
Ruby

data in the destination after the nth sync:

Languages
Python
Java
Python
Java
Ruby

This could be useful when we are interested to know about deletion of data in the source. This is possible if we also consider the date, or the batch id from which the data was written to the destination:

new data at the n+1th sync:

Languages
Python
Ruby

data in the destination after the n+1th sync:

Languages batch id
Python 1
Java 1
Python 2
Java 2
Ruby 2
Python 3
Ruby 3

In the future

We will consider making a better detection of deletions in the source, especially with Incremental, and Change Data Capture based sync modes for example.