Skip to content

Commit

Permalink
Added alation conn arg docs (open-metadata#15495)
Browse files Browse the repository at this point in the history
  • Loading branch information
OnkarVO7 authored Mar 8, 2024
1 parent 226db58 commit 1f8e232
Showing 1 changed file with 20 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,20 @@ The access token created using the steps mentioned [here](https://developer.alat

{% /codeInfo %}

{% codeInfo srNumber=21 %}

**connectionArguments**: These are additional parameters for Alation. If not specified the ingestion will use the predefined pagination logic.
The following arguments are intended to be used in conjunction and are specifically for Alation DataSource APIs:
- skip: This parameter determines the count of records to bypass at the start of the dataset. When set to 0, as in this case, it means that no records will be bypassed. If set to 10, it will bypass the first 10 records.

- limit: This argument specifies the maximum number of records to return. Here, it's set to 10, meaning only the first 10 records will be returned.

To perform incremental ingestion, these arguments should be used together. For instance, if there are a total of 30 datasources in Alation, the ingestion can be configured to execute three times, with each execution ingesting 10 datasources.
- 1st execution: {"skip": 0, "limit": 10}
- 2nd execution: {"skip": 10, "limit": 10}
- 3rd execution: {"skip": 20, "limit": 10}
{% /codeInfo %}

#### Sink Configuration

{% codeInfo srNumber=20 %}
Expand Down Expand Up @@ -176,6 +190,12 @@ source:
```yaml {% srNumber=18 %}
alationTagClassificationName: alationTags
```
```yaml {% srNumber=21 %}
connectionArguments: {
"skip": 0,
"limit": 10,
}
```
```yaml {% srNumber=19 %}
sourceConfig:
config:
Expand Down

0 comments on commit 1f8e232

Please sign in to comment.