Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some fixes in API fetcher documentation #622

Merged
merged 2 commits into from
Aug 20, 2024
Merged

Some fixes in API fetcher documentation #622

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b2e1e5d
add some fixes
8naama Aug 19, 2024
3a607ce
add cloudflare changes that weren't applied
8naama Aug 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 94 additions & 2 deletions docs/shipping/Network/cloudflare.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@ drop_filter: []

The Cloudflare web application firewall (WAF) protects your internet property against malicious attacks that aim to exploit vulnerabilities such as SQL injection attacks, cross-site scripting, and cross-site forgery requests.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

<Tabs>
<TabItem value="use-s3" label="Send logs using S3" default>


For an overview of Cloudflare logs, and the related S3 and Logpush configuration procedures, click [here](https://developers.cloudflare.com/logs/).


Expand All @@ -35,8 +42,6 @@ Before you begin, ensure that you have:
+ [Enabled the Cloudflare Logppush service](https://developers.cloudflare.com/logs/get-started/logpush-dashboard) for the assets you want to monitor in Cloudflare, via **Analytics > Logs > Connect a service**.




##### Configure Logpush to send logs to the S3 bucket

To configure Logpush to stream logs of Cloudflare's datasets to your cloud service in batches, follow the [Cloudflare procedure](https://developers.cloudflare.com/logs/get-started/enable-destinations/aws-s3/) to enable the Logpush service to access Amazon S3. <!-- deprecated link (https://developers.cloudflare.com/logs/logpush/aws-s3 -->
Expand All @@ -48,6 +53,93 @@ For an overview of the Logpush service, [click here](https://developers.cloudfla
Use [our procedure](https://docs.logz.io/docs/shipping/aws/aws-s3-bucket/#configure-logzio-to-fetch-logs-from-an-s3-bucket) to configure Logz.io to fetch logs from your S3 bucket.


</TabItem>
<TabItem value="use-cf-api" label="Send logs using Cloudflare API" default>

You can send available logs from the Cloudflare API with Logzio API fetcher.

## Pull Docker Image
Download the logzio-api-fetcher image:

```shell
docker pull logzio/logzio-api-fetcher
```

## Configuration
Create a local config file `config.yaml`.

```yaml
apis:
- name: cloudflare example
type: cloudflare
cloudflare_account_id: <<CLOUDFLARE_ACCOUNT_ID>>
cloudflare_bearer_token: <<CLOUDFLARE_BEARER_TOKEN>>
url: https://api.cloudflare.com/client/v4/accounts/{account_id}/alerting/v3/history
next_url: https://api.cloudflare.com/client/v4/accounts/{account_id}/alerting/v3/history?since={res.result.[0].sent}
days_back_fetch: 7
additional_fields:
type: cloudflare

logzio:
url: https://<<LISTENER-HOST>>:8071
token: <<LOGZIO_SHIPPING_TOKEN>>
```

:::note
You can customize the endpoints to collect data from by adding extra API configurations under `apis`.
:::

### Cloudflare configuration options
| Parameter Name | Description | Required/Optional | Default |
|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|-------------------|-------------------|
| name | Name of the API (custom name) | Optional | the defined `url` |
| cloudflare_account_id | The CloudFlare Account ID | Required | - |
| cloudflare_bearer_token | The Cloudflare Bearer token | Required | - |
| url | The request URL | Required | - |
| next_url | If needed to update the URL in next requests based on the last response. Supports using variables. | Optional | - |
| additional_fields | Additional custom fields to add to the logs before sending to logzio | Optional | - |
| days_back_fetch | The amount of days to fetch back in the first request. Applies a filter on `since` parameter. | Optional | - |
| scrape_interval | Time interval to wait between runs (unit: `minutes`) | Optional | 1 (minute) |
| pagination_off | True if builtin pagination should be off, False otherwise | Optional | `False` |

### Logzio output configuration options
| Parameter Name | Description | Required/Optional | Default |
|----------------|-----------------------------|-------------------|---------------------------------|
| url | The logzio Listener address (You can find the relevant `<<LISTENER-HOST>>` [here](https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping?product=logs).) | Optional | `https://listener.logz.io:8071` |
| token | The logzio shipping token | Required | - |


## Run The Docker Container
In the path where you saved your `config.yaml`, run:
```shell
docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher
```

:::note
To run in Debug mode add `--level` flag to the command:
```shell
docker run --name logzio-api-fetcher \
-v "$(pwd)":/app/src/shared \
logzio/logzio-api-fetcher \
--level DEBUG
```
Available Options: `INFO`, `WARN`, `ERROR`, `DEBUG`
:::

### Stopping the container
When you want to stop the container, to make sure it will finish the iteration on time, please give it a grace period of 30 seconds when you run the docker stop command:

```shell
docker stop -t 30 logzio-api-fetcher
```


</TabItem>
</Tabs>


##### Check Logz.io for your logs

Give your Cloudflare data some time to get from your system to ours, and then open [Open Search Dashboards](https://app.logz.io/#/dashboard/osd).
Expand Down
10 changes: 7 additions & 3 deletions docs/shipping/Security/one-password.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ metrics_alerts: []
drop_filter: []
---

You can ship logs available from the 1Password API with Logzio-api-fetcher.
You can send available logs from the 1Password API with Logzio API fetcher.

## Pull Docker Image
Download the logzio-api-fetcher image:
Expand All @@ -28,7 +28,7 @@ Create a local config file `config.yaml`.

```yaml
apis:
- name: 1Password test
- name: 1Password Audit Events
type: 1password
onepassword_bearer_token: <<1PASSWORD_BEARER_TOKEN>>
url: https://events.1password.com/api/v1/auditevents
Expand All @@ -40,9 +40,13 @@ apis:

logzio:
url: https://<<LISTENER-HOST>>:8071
token: <<SHIPPING_TOKEN>>
token: <<LOG-SHIPPING-TOKEN>>
```

:::note
You can customize the endpoints to collect data from by adding extra API configurations under `apis`.
:::

### 1Password configuration options
| Parameter Name | Description | Required/Optional | Default |
|--------------------------|-------------------------------------------------------------------------------------------------|-------------------|-------------------|
Expand Down
6 changes: 4 additions & 2 deletions docs/user-guide/log-management/api-fetcher.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ image: https://dytvr9ot2sszz.cloudfront.net/logz-docs/social-assets/docs-social.
keywords: [logz.io, opensearch dashboards, log analysis, observability]
---

The Logz.io API Fetcher supports both auth and OAuth APIs and includes specific implementations for Azure Graph, Cisco Secure X, and Office365 Message Trace reports.
The Logz.io API Fetcher supports both auth and OAuth APIs and includes specific implementations for Azure Graph, Office365 Message Trace reports, Cloudflare and 1Password.

This guide outlines the steps for configuring the Logz.io API Fetcher to fetch and send data to Logz.io. Our aim is to develop the API Fetcher as a generic tool capable of fetching data from any API endpoint. However, this presents significant challenges. If you encounter difficulties configuring the API Fetcher with a particular API endpoint, please reach out to our support team for assistance.
This guide outlines the steps for configuring the Logz.io API Fetcher to fetch and send data to Logz.io. Our aim is to develop the API Fetcher as a generic tool capable of fetching data from any API endpoint. However, this presents significant challenges. If you encounter difficulties configuring the API Fetcher with a particular API endpoint, please reach out to our support team for assistance.

Below is a sample configuration template, as found in our documentation and on GitHub:

Expand Down Expand Up @@ -88,6 +88,7 @@ apis:
```

## Configuration
Create a local config file `config.yaml`.

### Add Your Logz.io Listener and Token

Expand All @@ -100,6 +101,7 @@ logzio:
```

{@include: ../../_include/log-shipping/log-shipping-token.md}

{@include: ../../_include/log-shipping/listener-var.html}

### Configure your APIs
Expand Down