AWS Security Group exporter for Prometheus

A dockerized^[1] Prometheus exporter that compares desired/wanted IPv4/IPv6 CIDR against currently applied inbound CIDR rules by protocol and port number in your AWS security group(s) per region.

^[1]: If you want to use this exporter without Docker jump here: Usage without Docker

Motivation

Some IP addresses ranges such as Cloudfront edge nodes or SaaS hosts might change frequently and you possibly want to ensure that those are always in sync with what you have currently defined in your security group. This exporter does exactly this and can easily be hooked up with Alertmanager to trigger alerts in case you get out of sync.

How does it work

Desired/Wanted IP address CIDR

You have to provide a command, which is parsable by bash's eval function and evalutes newline-separated to your desired/wanted IP address CIDR. As a few examples:

# Note that for single IP addresses, AWS requires '/32' to be appended
eval "dig +short nat.travisci.net | xargs -n1 -I% echo \"%/32\""
eval "printf \"10.13.23.23/32\n192.168.0.0/24\n\""

Applied security Group CIDR

You have to provide the following in order to fetch your currently applied sg rules:

Security group name (The Name tag)^[1]
AWS region where the security group resides
Security group rule protocol (e.g.: tcp, udp, icmp, ...)
Security group rule from port (e.g.: 80, 443, ...)

^[1]: The * wildcard is supported for the name, but you have to ensure to match exactly one security group

Output

The exporter will then output Prometheus readable information as such:

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="sg-name",region="us-east-1",proto="tcp",from_port="80",ip="v4",cidr="10.4.1.1/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="sg-name",region="us-east-1",proto="tcp",from_port="80",ip="v4",cidr="10.4.1.5/32",sg_id="sg-xxxxx",errno="0",error=""} 0

A value of 1 means the desired/wanted IP CIDR is applied to the security group
A value of 0 means the desired/wanted IP CIDR is not applied to the security group

See Metrics for an indepth description.

Requirements

You will need AWS access key and secret with the following permission:

ec2:DescribeSecurityGroups

Docker settings

Tagging

Ensure to use Docker image tags (which are the same as git tags from this repository) to prevent any backwards incompatible changes. The latest tag should only be used for testing purposes.

Additionally do not blindly update Docker image tags before having tested it. Security group rule checks are an important matter and you want to ensure your alerting is reliable.

Environment variables

You can specify up to 4 security group checks: SG1_*, SG2_*, SG3_* and SG4_*.

Variable	Description
`AWS_ACCESS_KEY_ID`	The AWS access key (required to connect to AWS to check the sg rules)
`AWS_SECRET_ACCESS_KEY`	The AWS secret key (required to connect to AWS to check the sg rules)
`AWS_SESSION_TOKEN`	(Optional) The AWS session token

`UPDATE_TIME`	Time interval in sec for how often to update metrics (default: `60`)

`SG1_NAME`	Name of the security group on AWS
`SG1_REGION`	Region the security group resides in
`SG1_PROTO`	Security group rule protocol: `tcp`, `udp`, `icmp` or a protocol number
`SG1_FROM_PORT`	Security group rule from port
`SG1_IP4_CMD`	The command that evaluates to newline-separated IPv4 IP address CIDR ^[1]
`SG1_IP6_CMD`	The command that evaluates to newline-separated IPv6 IP address CDIR ^[1]

`SG2_NAME`	Name of the security group on AWS
`SG2_REGION`	Region the security group resides in
`SG2_PROTO`	Security group rule protocol: `tcp`, `udp`, `icmp` or a protocol number
`SG2_FROM_PORT`	Security group rule from port
`SG2_IP4_CMD`	The command that evaluates to newline-separated IPv4 IP address CIDR ^[1]
`SG2_IP6_CMD`	The command that evaluates to newline-separated IPv6 IP address CDIR ^[1]

`SG3_NAME`	Name of the security group on AWS
`SG3_REGION`	Region the security group resides in
`SG3_PROTO`	Security group rule protocol: `tcp`, `udp`, `icmp` or a protocol number
`SG3_FROM_PORT`	Security group rule from port
`SG3_IP4_CMD`	The command that evaluates to newline-separated IPv4 IP address CIDR ^[1]
`SG3_IP6_CMD`	The command that evaluates to newline-separated IPv6 IP address CDIR ^[1]

`SG4_NAME`	Name of the security group on AWS
`SG4_REGION`	Region the security group resides in
`SG4_PROTO`	Security group rule protocol: `tcp`, `udp`, `icmp` or a protocol number
`SG4_FROM_PORT`	Security group rule from port
`SG4_IP4_CMD`	The command that evaluates to newline-separated IPv4 IP address CIDR ^[1]
`SG4_IP6_CMD`	The command that evaluates to newline-separated IPv6 IP address CDIR ^[1]

^[1]: SG*_IP4_CMD and SG*_IP6_CMD are mutually exclusive. Also note that evaluated IP address CIDR are only checked against security group rules that match the protocol (SG*_PROTO) and also match the from port (SG*_FROM_PORT).

Mount points

None - it's fully stateless

Exposed ports

External	Internal	Description
Up to you	`8080`	Where the `aws-ec2-sg-exporter` provides metrics via HTTP

Metrics

This exporter outputs metrics in the following format:

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="sg-name",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="10.4.1.1/32",sg_id="sg-xxxxx",errno="0",error=""} 1

The following table describes each of the key/value paris:

Key	Value
`name`	The security group name as specified by `SG*_NAME`
`region`	The security group region as specified by `SG*_REGION`
`proto`	The security group rule protocol as specified by `SG*_PROTO`
`from_port`	The security group rule from port as specified by `SG*_FROM_PORT`
`ip`	IP version of desired/wanted CIDR to be available in your security group by `proto` and `from_port`
`cidr`	The desired/wanted IP to be available in your security group by `proto` and `from_port`
`sg_id`	The security group ID found by `name` and `region`. If this is empty then either zero or more multiple security groups were found.
`errno`	0: One security group was found (OK) 1: No security group was found (ERR) 2: Multiple security groups were found (ERR)
`error`	The corresponding error message for `errno`

A value of 1 means the desired/wanted IP CIDR is applied to the security group
A value of 0 means the desired/wanted IP CIDR is not applied to the security group

Examples

Scenario 1 - Travis

Check if your security group named my-sg (in us-east-1) allows all inbound IPv4 addresses from Travis-CI via tcp on https.

Desired/wanted IP CIDR

Ensure you have a working command which can be interpretated by eval and that outputs CIDR (with /[0-9]+ appended) of your desired ranges:

$ eval "dig +short nat.travisci.net | xargs -n1 -I% echo \"%/32\""

35.184.226.236/32
35.188.1.99/32
35.188.73.34/32
35.192.85.2/32
35.192.136.167/32
...

Run `aws-ec2-sg-exporter`

docker run -it --rm \
	-p 9000:8080 \
	\
	-e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
	-e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
	\
	-e SG1_NAME="my-sg" \
	-e SG1_REGION="us-east-1" \
	-e SG1_PROTO="tcp" \
	-e SG1_FROM_PORT="443" \
	-e SG1_IP4_CMD="dig +short nat.travisci.net | xargs -n1 -I% echo \"%/32\"" \
	cytopia/aws-ec2-sg-exporter

Check output

Check the output via curl:

$ curl localhost:9000`

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.184.226.236/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.188.1.99/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.188.73.34/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.192.85.2/32",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="35.192.136.167/32",sg_id="sg-xxxxx",errno="0",error=""} 0
...

As you can see, the last line returns a 0, which means this IP CIDR is missing in your security group.

Scenario 2 - Cloudfront

Check if your security group named my-sg4 (in us-east-1) allows all inbound IPv4 addresses from Cloudfront edge-nodes via tcp on https.
Check if your security group named my-sg6 (in us-east-1) allows all inbound IPv6 addresses from Cloudfront edge-nodes via tcp on https.

Desired/wanted IP CIDR

Ensure you have a working command which can be interpretated by eval and that outputs CIDR (with /[0-9]+ appended) of your desired ranges:

$ eval "curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json \
	| jq -r '.prefixes \
		| sort_by(.ip_prefix)[] \
		| select( .service | contains(\"CLOUDFRONT\")) \
		| select ( .region | test(\"^(GLOBAL|us-|eu-)\")) \
		| .ip_prefix'"

13.224.0.0/14
13.249.0.0/16
13.32.0.0/15
13.35.0.0/16
13.52.204.0/23
...

$ eval "curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json \
	| jq -r '.ipv6_prefixes \
		| sort_by(.ipv6_prefixes)[] \
		| select( .service | contains(\"CLOUDFRONT\")) \
		| select ( .region | test(\"^(GLOBAL|us-|eu-)\")) \
		| .ipv6_prefix'"

2600:9000:eee::/48
2600:9000:4000::/36
2600:9000:3000::/36
2600:9000:f000::/36
2600:9000:fff::/48
...

Run `aws-ec2-sg-exporter`

docker run -it --rm \
	-p 9000:8080 \
	\
	-e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
	-e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
	\
	-e SG1_NAME="my-sg4" \
	-e SG1_REGION="us-east-1" \
	-e SG1_PROTO="tcp" \
	-e SG1_FROM_PORT="443" \
	-e SG1_IP4_CMD="curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes | sort_by(.ip_prefix)[] | select( .service | contains(\"CLOUDFRONT\")) | select ( .region | test(\"^(GLOBAL|us-|eu-)\")) | .ip_prefix'" \
	\
	-e SG2_NAME="my-sg6" \
	-e SG2_REGION="us-east-1" \
	-e SG2_PROTO="tcp" \
	-e SG2_FROM_PORT="443" \
	-e SG2_IP6_CMD="curl -sS https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.ipv6_prefixes | sort_by(.ipv6_prefixes)[] | select( .service | contains(\"CLOUDFRONT\")) | select ( .region | test(\"^(GLOBAL|us-|eu-)\")) | .ipv6_prefix'" \
	cytopia/aws-ec2-sg-exporter

Check output

Check the output via curl:

$ curl localhost:9000`

# HELP aws_ec2_sg_compare Determines If CIDR is applied to security group.
# TYPE aws_ec2_sg_compare counter
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.224.0.0/14",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.249.0.0/16",sg_id="sg-xxxxx",errno="0",error=""} 0
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.32.0.0/15",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.35.0.0/16",sg_id="sg-xxxxx",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg4",region="us-east-1",proto="tcp",from_port="443",ip="v4",cidr="13.52.204.0/23",sg_id="sg-xxxxx",errno="0",error=""} 1
...
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:eee::/48",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:4000::/36",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:3000::/36",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:f000::/36",sg_id="sg-yyyyy",errno="0",error=""} 1
aws_ec2_sg_compare{name="my-sg6",region="us-east-1",proto="tcp",from_port="443",ip="v6",cidr="2600:9000:fff::/48",sg_id="sg-yyyyy",errno="0",error=""} 0
...

As you can see, the second line ipv4 address returns a 0 and the last ipv6 address returns a 0, which means these IP CIDR are missing in your security groups.

Grafana setup

Graphs

Align the Min time interval with what you have set UPDATE_TIME to.
Add you metrics by the name of your specified security group name
Set the legend to {{ cidr }} to have only the CIDR displayed

Once this is done, your graph will look similar to this one:

Single Stat

Align the Min time interval with what you have set UPDATE_TIME to.
Add you metrics by the name of your specified security group name
sum() gives your the total sum of values (0 and 1) and count() will give you the total number of available IP addresses

Once this is done, your single stat will look similar to this one:

Usage without Docker

Docker is not necessarily required and you can simply use the exporter bash script: aws-ec2-sg-exporter.

By doing so, you need to ensure you have all requirements installed on your system (aws and jq binary as well as bash itself).

Additionally you will have to make sure the script's stdout will somehow be served by a webserver. The recommended method is to have some mechanism which writes the script's output atomically to a static html file and a web server will simply serve it.

aws-ec2-sg-exporter will read all configuration from the shell's environment, so in order to use this script you need to export all values to your env. See Environment variables for possible values.

Error handling

The exporter writes all errors to stderr regardless of using Docker or the standalone aws-ec2-sg-exporter script.

Expected errors

An error occurred (RequestExpired) when calling the DescribeSecurityGroups operation: Request has expired.

In case you are using IAM roles, your session has simply been expired and needs to be renewed. It is recommended to user IAM users instead without session.

[ERR] 2019-08-18 10:55:11 (aws-ec2-sg-exporter): No sg found by name: sg-name22 in region: us-east-1

A security group could not be found by name and region. The exporter will continue to run and output Prometheus metrics, but will mark all desired/wanted IP CIDR as not found in your security group.

[ERR] 2019-08-18 10:56:17 (aws-ec2-sg-exporter): Multiple sg found by name: sg-name-* in region: us-east-1: sg-xxxxx,sg-yyyyy

Multiple security groups have been found by the specified name and region. The exporter will continue to run and output Prometheus metrics, but will mark all desired/wanted IP CIDR as not found in your security group.

Unexpected errors

write error: Broken pipe

This is a very rare condition and will most likely be caused by using broken shell pipes (|) in your commands specified via SG*_IP4_CMD or SG*_IP6_CMD.

In case you are using something like this:

curl http://some-page.tld | grep -E '^[.0-9]+/[0-9]+$';

Consider to add a buffer in between:

curl http://some-page.tld | dd obs=1M 2>/dev/null | grep -E '^[.0-9]+/[0-9]+$';

See here: https://superuser.com/a/642932/705357

License

MIT License

Name		Name	Last commit message	Last commit date
Latest commit History 26 Commits
data		data
doc		doc
.travis.yml		.travis.yml
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

AWS Security Group exporter for Prometheus

Motivation

How does it work

Desired/Wanted IP address CIDR

Applied security Group CIDR

Output

Requirements

Docker settings

Tagging

Environment variables

Mount points

Exposed ports

Metrics

Examples

Scenario 1 - Travis

Desired/wanted IP CIDR

Run `aws-ec2-sg-exporter`

Check output

Scenario 2 - Cloudfront

Desired/wanted IP CIDR

Run `aws-ec2-sg-exporter`

Check output

Grafana setup

Graphs

Single Stat

Usage without Docker

Error handling

Expected errors

Unexpected errors

License

About

Releases 1

Packages

Languages

License

cytopia/aws-ec2-sg-exporter

Folders and files

Latest commit

History

Repository files navigation

AWS Security Group exporter for Prometheus

Motivation

How does it work

Desired/Wanted IP address CIDR

Applied security Group CIDR

Output

Requirements

Docker settings

Tagging

Environment variables

Mount points

Exposed ports

Metrics

Examples

Scenario 1 - Travis

Desired/wanted IP CIDR

Run aws-ec2-sg-exporter

Check output

Scenario 2 - Cloudfront

Desired/wanted IP CIDR

Run aws-ec2-sg-exporter

Check output

Grafana setup

Graphs

Single Stat

Usage without Docker

Error handling

Expected errors

Unexpected errors

License

About

Topics

Resources

License

Stars

Watchers

Forks

Releases 1

Packages 0

Languages

Run `aws-ec2-sg-exporter`

Run `aws-ec2-sg-exporter`

Packages