Skip to content

Commit

Permalink
v0.2.1
Browse files Browse the repository at this point in the history 
Add 1Password API
  • Loading branch information
@8naama
8naama authored Aug 18, 2024
2 parents 2e5eb3e + c529979 commit 673e3f2
Show file tree
Hide file tree
Showing 11 changed files with 307 additions and 30 deletions.
50 changes: 38 additions & 12 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,19 @@ Configure your API inputs under `apis`. For every API, mention the input type un
For structuring custom API calls use type `general` API with the parameters below.

## Configuration Options
| Parameter Name | Description | Required/Optional | Default |
|--------------------|-----------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in next requests based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| Parameter Name | Description | Required/Optional | Default |
|--------------------|---------------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| next_body | If needed to update the body in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |

## Pagination Configuration Options
If needed, you can configure pagination.
Expand Down Expand Up @@ -207,6 +208,27 @@ By default `cloudflare` API type:
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| pagination_off | True if builtin pagination should be off, False otherwise | Optional | `False` |

</details>
<details>
<summary>
<span><a href="./src/apis/onepassword/README.md">1Password</a></span>
</summary>

By default `1password` API type has built in pagination settings and sets the `response_data_path` to `items` field.

## Configuration Options
| Parameter Name | Description | Required/Optional | Default |
|--------------------------|--------------------------------------------------------------------------------------------------------------|-------------------|-------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| onepassword_bearer_token | The 1Password Bearer token | Required | - |
| url | The request URL | Required | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | - |
| days_back_fetch | The amount of days to fetch back in the first request. Applies a filter on 1password `start_time` parameter. | Optional | - |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| onepassword_limit | 1Password limit for number of events to return in a single request (allowed range: 100 to 1000) | Optional | 100 |
| pagination_off | True if builtin pagination should be off, False otherwise | Optional | `False` |

</details>


Expand Down Expand Up @@ -262,6 +284,10 @@ docker stop -t 30 logzio-api-fetcher
```

## Changelog:
- **0.2.1**:
- Add 1Password Support
- Add `next_body` support to allow more customization in general settings
- Support integers and boolean as values in pagination 'equals' stop condition
- **0.2.0**:
- **Breaking changes!!**
- Deprecate configuration fields:
Expand Down
24 changes: 21 additions & 3 deletions src/apis/general/Api.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import logging
from pydantic import BaseModel, Field
import requests
from typing import Union
from typing import Union, Optional

from src.utils.processing_functions import extract_vars, substitute_vars
from src.apis.general.PaginationSettings import PaginationSettings, PaginationType
Expand All @@ -30,31 +30,37 @@ class ApiFetcher(BaseModel):
:param method: Optional, the method to use for the request (default: GET)
:param pagination_settings: Optional, PaginationSettings object that defines how to perform pagination
:param next_url: Optional, If needed update a param in the url according to the response as we go
:param next_body: Optional, If needed update a param in the body according to the response as we go
:param response_data_path: Optional, The path to find the data within the response.
:param additional_fields: Optional, 'key: value' pairs that should be added to the API logs.
:param scrape_interval_minutes: the interval between scraping jobs.
:param url_vars: Not passed to the class, array of params that is generated based on next_url.
:param body_vars: Not passed to the class, array of params that is generated based on next_body.
"""
name: str = Field(default="")
url: str
headers: dict = Field(default={})
body: Union[str, dict, list] = Field(default=None)
method: ReqMethod = Field(default=ReqMethod.GET, frozen=True)
pagination_settings: PaginationSettings = Field(default=None, frozen=True, alias="pagination")
pagination_settings: Optional[PaginationSettings] = Field(default=None, frozen=True, alias="pagination")
next_url: str = Field(default=None)
next_body: Union[str, dict, list] = Field(default=None)
response_data_path: str = Field(default=None, frozen=True)
additional_fields: dict = Field(default={})
scrape_interval_minutes: int = Field(default=1, alias="scrape_interval", ge=1)
url_vars: list = Field(default=[], init=False, init_var=True)
body_vars: list = Field(default=[], init=False, init_var=True)

def __init__(self, **data):
"""
Makes sure to format the body and generate the url_vars based on next_url.
Makes sure to format the body and generate the url_vars based on next_url and body_vars based on next_body.
:param data: the fields for creation of the class.
"""
super().__init__(**data)
self.body = self._format_body(self.body)
self.next_body = self._format_body(self.next_body)
self.url_vars = extract_vars(self.next_url)
self.body_vars = extract_vars(self.next_body)
if not self.name:
self.name = self.url
if not self.additional_fields.get("type"):
Expand Down Expand Up @@ -208,6 +214,14 @@ def update_next_url(self, new_next_url):
self.next_url = new_next_url
self.url_vars = extract_vars(self.next_url)

def update_next_body(self, new_next_body):
"""
Supports updating the next request body format to make sure the 'self.body_vars' is updated accordingly.
:param new_next_body: new format for the next body. (if in future some customized APIs will need it supported)
"""
self.next_body = new_next_body
self.body_vars = extract_vars(self.next_body)

def send_request(self):
"""
Manages the request:
Expand Down Expand Up @@ -236,4 +250,8 @@ def send_request(self):
# Update the url if needed
if self.next_url:
self.url = substitute_vars(self.next_url, self.url_vars, r)

# Update the body if needed
if self.next_body:
self.body = substitute_vars(self.next_body, self.body_vars, r)
return responses
25 changes: 13 additions & 12 deletions src/apis/general/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,19 @@ For structuring custom API calls use type `general` API with the parameters belo
- [Example](#example)

## Configuration
| Parameter Name | Description | Required/Optional | Default |
|--------------------|-----------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in next requests based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| Parameter Name | Description | Required/Optional | Default |
|--------------------|---------------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| url | The request URL | Required | - |
| headers | The request Headers | Optional | `{}` |
| body | The request body | Optional | - |
| method | The request method (`GET` or `POST`) | Optional | `GET` |
| pagination | Pagination settings if needed (see [options below](#pagination-configuration-options)) | Optional | - |
| next_url | If needed to update the URL in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| next_body | If needed to update the body in the next request based on the last response. Supports using variables ([see below](#using-variables)) | Optional | - |
| response_data_path | The path to the data inside the response | Optional | response root |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | Add `type` as `api-fetcher` |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |

## Pagination Configuration Options
If needed, you can configure pagination.
Expand Down
5 changes: 3 additions & 2 deletions src/apis/general/StopPaginationSettings.py
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from enum import Enum
import logging
from pydantic import BaseModel, Field, model_validator
from typing import Union

logger = logging.getLogger(__name__)

Expand All @@ -23,7 +24,7 @@ class StopPaginationSettings(BaseModel):
"""
field: str
condition: StopCondition
value: str = Field(default=None, frozen=True)
value: Union[str, int, bool] = Field(default=None, frozen=True)

@model_validator(mode='after')
def _check_conditional_fields(self):
Expand All @@ -32,7 +33,7 @@ def _check_conditional_fields(self):
if we got condition as 'contains' or 'equals' >> that we also got value
:return: self
"""
if self.condition in (StopCondition.EQUALS, StopCondition.CONTAINS) and not self.value:
if self.condition in (StopCondition.EQUALS, StopCondition.CONTAINS) and self.value is None:
raise ValueError(f"Used stop condition {self.condition} but missing required 'value' field.")
return self

Expand Down
Loading

0 comments on commit 673e3f2

Please sign in to comment.