The Full Refresh modes are the simplest methods that Airbyte uses to sync data, as they always retrieve all available information requested from the source, regardless of whether it has been synced before. This contrasts with Incremental sync, which does not sync data that has already been synced before.
In the Overwrite variant, new syncs will destroy all data in the existing destination table and then pull the new data in. Therefore, data that has been removed from the source after an old sync will be deleted in the destination table.
On the nth sync of a full refresh connection:
data in the destination before the sync:
| Languages |
|---|
| Python |
| Java |
new data:
| Languages |
|---|
| Python |
| Java |
| Ruby |
data in the destination after the sync:
| Languages |
|---|
| Python |
| Java |
| Ruby |
Note: This is how Singer target-bigquery does it.
We will consider making other flavors of full refresh configurable as first-class citizens in Airbyte. e.g. On new data, copy old data to a new table with a timestamp, and then replace the original table with the new data. As always, we will focus on adding these options in such a way that the behavior of each connector is both well documented and predictable.