Skip to content

Commit

Permalink
Add Makefile, some simple readme info. Remove python2 mentioning
Browse files Browse the repository at this point in the history
There are no changes which mean python2 is actually unsupportd, but we don't want to guarantee this going forwards so remove it from the README
  • Loading branch information
Limess committed Dec 16, 2020
1 parent a60b437 commit 78795c0
Show file tree
Hide file tree
Showing 6 changed files with 651 additions and 181 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
targets
8 changes: 8 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
.PHONY: format
format:
poetry run black .

.PHONY: dev-start
dev-start:
@mkdir -p ./targets
poetry run python discoverecs.py --directory $$PWD/targets
106 changes: 69 additions & 37 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,62 +1,92 @@
# prometheus-ecs-sd

ECS Service Discovery for Prometheus

## Info

This tool provides Prometheus service discovery for Docker containers running on AWS ECS. You can easily instrument your app using a Prometheus
client and enable discovery adding an ENV variable at the Service Task Definition. Your container will then be added
to the list of Prometheus targets to be scraped. Requires python2 or python3 and boto3. Works with Prometheus 2.x. It supports bridge, host
and awsvpc (EC2 and Fargate) network modes.
to the list of Prometheus targets to be scraped.

Bridge, host and awsvpc (EC2 and Fargate) network modes are supported.

Requires

- `python3`
- The `boto3` library
- Prometheus `2.x`.

## Developing

Local development requires the [poetry](https://python-poetry.org/) tool, check the documentation for installation instructions.

To start the service run

```shell
AWS_PROFILE=<your_aws_profile> make dev-start
```

any AWS configuration supported by boto3 is supported, e.g. individual access/secret keys.

To format code run

```shell
make format
```

## Setup

`discoverecs.py` should run alongside the Prometheus server. It generates targets using JSON file service discovery. It can
be started by running:

```
```shell
python discoverecs.py --directory /opt/prometheus-ecs
```

Where `/opt/prometheus-ecs` is defined in your Prometheus config as a `file_sd_config` job:
note that the directory must already exist.

The output directory is then `/opt/prometheus-ecs` defined in your Prometheus config as a `file_sd_config` job:

```yaml
- job_name: 'ecs-1m'
scrape_interval: 1m
file_sd_configs:
- files:
- /opt/prometheus-ecs/1m-tasks.json
- files:
- /opt/prometheus-ecs/1m-tasks.json
relabel_configs:
- source_labels: [metrics_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [metrics_path]
action: replace
target_label: __metrics_path__
regex: (.+)
```
You can also specify a discovery interval with `--interval` (in seconds). Default is 60s. We also provide caching to minimize hitting query
rate limits with the AWS ECS API. `discoverecs.py` runs in a loop until interrupted and will output target information to stdout.

To make your application discoverable by Prometheus, you need to set the following environment variable in your task definition:

```
{"name": "PROMETHEUS", "value": "true"}
```json
{ "name": "PROMETHEUS", "value": "true" }
```

Metric path and scrape interval is supported via `PROMETHEUS_ENDPOINT`:

```
```text
"interval:/metric_path,..."
```

Examples:

```
```text
"5m:/mymetrics,30s:/mymetrics2"
"/mymetrics"
"30s:/mymetrics1,/mymetrics2"
```

Under ECS task definition (`task.json`):

```
{"name": "PROMETHEUS_ENDPOINT", "value": "5m:/mymetrics,30s:/mymetrics2"}
```json
{ "name": "PROMETHEUS_ENDPOINT", "value": "5m:/mymetrics,30s:/mymetrics2" }
```

Available scrape intervals: `15s`, `30s`, `1m`, `5m`.
Expand All @@ -66,50 +96,51 @@ The default metric path is `/metrics`. The default scrape interval is `1m`.
The following Prometheus configuration should be used to support all available intervals:

```yaml
- job_name: 'ecs-15s'
scrape_interval: 15s
file_sd_configs:
- job_name: 'ecs-15s'
scrape_interval: 15s
file_sd_configs:
- files:
- /opt/prometheus-ecs/15s-tasks.json
relabel_configs:
- /opt/prometheus-ecs/15s-tasks.json
relabel_configs:
- source_labels: [metrics_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- job_name: 'ecs-30s'
scrape_interval: 30s
file_sd_configs:
- job_name: 'ecs-30s'
scrape_interval: 30s
file_sd_configs:
- files:
- /opt/prometheus-ecs/30s-tasks.json
relabel_configs:
- /opt/prometheus-ecs/30s-tasks.json
relabel_configs:
- source_labels: [metrics_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- job_name: 'ecs-1m'
scrape_interval: 1m
file_sd_configs:
- job_name: 'ecs-1m'
scrape_interval: 1m
file_sd_configs:
- files:
- /opt/prometheus-ecs/1m-tasks.json
relabel_configs:
- /opt/prometheus-ecs/1m-tasks.json
relabel_configs:
- source_labels: [metrics_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- job_name: 'ecs-5m'
scrape_interval: 5m
file_sd_configs:
- job_name: 'ecs-5m'
scrape_interval: 5m
file_sd_configs:
- files:
- /opt/prometheus-ecs/5m-tasks.json
relabel_configs:
- /opt/prometheus-ecs/5m-tasks.json
relabel_configs:
- source_labels: [metrics_path]
action: replace
target_label: __metrics_path__
regex: (.+)
```

## EC2 IAM Policy

The following IAM Policy should be added when running `discoverecs.py` in EC2:
Expand All @@ -135,6 +166,7 @@ resource "aws_iam_role_policy_attachment" "prometheus-server-role-ec2-read-only"
```

## Special cases

For skipping labels, set `PROMETHEUS_NOLABELS` to `true`.
This is useful when you use "blackbox" exporters or Pushgateway in a task
and metrics are exposed at a service level. This way, no EC2/ECS labels
Expand Down
Loading

0 comments on commit 78795c0

Please sign in to comment.