Skip to content

Commit

Permalink
Add some links, correcto mundo the decryption piece
Browse files Browse the repository at this point in the history
  • Loading branch information
@KevinHock
KevinHock committed Nov 27, 2023
1 parent b8f5dce commit 581d7e4
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 13 deletions.
12 changes: 7 additions & 5 deletions ...23-11-26-preventing-accidental-internet-exposure-of-aws-resources-part-1-vpc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ The reason for this is as follows. When you make an interface VPC endpoint with

However, suppose you are willing to do a lot of heavy lifting that is orthogonal to AWS primitives.

In that case, you _can_ use these with [a proxy](https://eng.lyft.com/internet-egress-filtering-of-services-at-lyft-72e99e29a4d9) to accomplish centralized egress. This works because the destination IP of outbound traffic won't be the Internet, but a private IP, due to deploying, e.g., iptables to re-route Internet-destined traffic on every host.
In that case, you _can_ use these with [an outbound proxy](https://eng.lyft.com/internet-egress-filtering-of-services-at-lyft-72e99e29a4d9) to accomplish centralized egress. This works because the destination IP of outbound traffic won't be the Internet, but a private IP, due to deploying, e.g., iptables to re-route Internet-destined traffic on every host.

Some reasons you may not want to do this are:
- Significant effort
Expand All @@ -116,7 +116,7 @@ With that said, AWS does not have a primitive to perform Egress filtering,[^99]

[^985]: A peering connection cannot be selected as a [traffic mirror source or target](https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-targets.html), but a network interface can. However, only an ENI belonging to an EC2 instance can be a mirror source, not an ENI belonging to an Interface endpoint. The documentation doesn't mention this anywhere I could find.

[^99]: It has [AWS Network Firewall](https://aws.amazon.com/network-firewall/faqs/), which can be fooled via SNI spoofing. So it is, at best, a stepping stone to keep an inventory of your Egress traffic if you can’t get a proxy up and running short-term and are not using TLS 1.3 with encrypted client hello (ECH) or encrypted SNI (ESNI). I cringe at how [the FAQ](https://aws.amazon.com/network-firewall/faqs/) says these are not supported rather than a bypass of the product. Sadly, this euphemism [isn't unique to AWS](https://i.imgur.com/dPyFaNK.png).
[^99]: It has [AWS Network Firewall](https://aws.amazon.com/network-firewall/faqs/), which is [just managed Suricata](https://docs.suricata.io/en/latest/rules/tls-keywords.html) and can be fooled via SNI spoofing. So it is, at best, a stepping stone to keep an inventory of your Egress traffic if you can’t get a proxy or real firewall running short-term and [are not using TLS 1.3 with encrypted client hello (ECH) or encrypted SNI (ESNI)](https://docs.aws.amazon.com/network-firewall/latest/developerguide/tls-inspection-considerations.html).

### Option 3: Centralized Egress via Gateway Load Balancer (GWLB) with Firewall

Expand All @@ -131,8 +131,6 @@ Gateway Load Balancer endpoint ENIs _have this check disabled_ to support their
![alt text](https://i.imgur.com/tIQaTa0.png)
(No NAT Gateway is necessary here, as the firewall is running in a public subnet and performing NAT. [AWS](https://aws.amazon.com/blogs/networking-and-content-delivery/best-practices-for-deploying-gateway-load-balancer/) and [others](https://networkgeekstuff.com/networking/basic-load-balancer-scenarios-explained/) call this two-arm mode.)

Unfortunately, unless they do decryption, firewalls can't filter on URL paths -- for example, you can't block all of github.com except for github.com/mycompany. The better vendors don’t seem to offer decryption; you’re probably better off using a proxy (i.e., the previous option) if you want to filter on URL paths.

The firewall must support Geneve encapsulation, be invulnerable to SNI spoofing, fast, reliable, not [susceptible to IP address mismatches](https://chasersystems.com/discriminat/faq/#are-the-out-of-band-dns-lookups-susceptible-to-ip-address-mismatches), and preferably perform NAT to eliminate the need for NAT Gateways, so building an open-source alternative is not easy.

Regarding specific vendors, [DiscrimiNAT](https://github.com/ChaserSystems/terraform-aws-discriminat-gwlb#deployment-examples) seems much easier to configure compared to e.g. Palo Alto Firewall,[^99351] as all you do is [add FQDNs to security group descriptions](https://chasersystems.com/docs/discriminat/aws/quick-start/#viii-configuring-a-whitelist). However, DiscrimiNAT would need to add subaccount support for the diagram above to function to read the security groups in the 'spoke' account.
Expand Down Expand Up @@ -223,11 +221,15 @@ AWS Billing Cost | <span style="color:red">High</span> | Low
Complexity* | Medium | <span style="color:red">High</span> | Medium | Low | Medium
Scalability* | High | High | High | Low | Medium
Flexibility* | High | High | High | Medium | <span style="color:red">Lowest</span>
Filtering Granularity | None | FQDN or URL Path | FQDN | None | None
Filtering Granularity | None | FQDN (or URL Path\*\*) | FQDN (or URL Path\*\*) | None | None
Will Prevent Org Migration | False | False | False | <span style="color:red">True</span> | False

\* = YMMV

\*\* = URL path is only available to filter on if MITM is performed.[^91512]

[^91512]: See the ["Man-in-the-Middle" section](https://eng.lyft.com/internet-egress-filtering-of-services-at-lyft-72e99e29a4d9) of Lyft's post, for some thoughts around this.

## FAQ

### Can you walk through the cost details around Option 1?
Expand Down
19 changes: 11 additions & 8 deletions _site/2023/11/26/preventing-accidental-internet-exposure-of-aws-resources-part-1-vpc.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ <h3 id="option-2-centralized-egress-via-privatelink-or-vpc-peering-with-proxy">O

<p>However, suppose you are willing to do a lot of heavy lifting that is orthogonal to AWS primitives.</p>

<p>In that case, you <em>can</em> use these with <a href="https://eng.lyft.com/internet-egress-filtering-of-services-at-lyft-72e99e29a4d9">a proxy</a> to accomplish centralized egress. This works because the destination IP of outbound traffic won’t be the Internet, but a private IP, due to deploying, e.g., iptables to re-route Internet-destined traffic on every host.</p>
<p>In that case, you <em>can</em> use these with <a href="https://eng.lyft.com/internet-egress-filtering-of-services-at-lyft-72e99e29a4d9">an outbound proxy</a> to accomplish centralized egress. This works because the destination IP of outbound traffic won’t be the Internet, but a private IP, due to deploying, e.g., iptables to re-route Internet-destined traffic on every host.</p>

<p>Some reasons you may not want to do this are:</p>
<ul>
Expand All @@ -253,8 +253,6 @@ <h3 id="option-3-centralized-egress-via-gateway-load-balancer-gwlb-with-firewall
<p><img src="https://i.imgur.com/tIQaTa0.png" alt="alt text" />
(No NAT Gateway is necessary here, as the firewall is running in a public subnet and performing NAT. <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/best-practices-for-deploying-gateway-load-balancer/">AWS</a> and <a href="https://networkgeekstuff.com/networking/basic-load-balancer-scenarios-explained/">others</a> call this two-arm mode.)</p>

<p>Unfortunately, unless they do decryption, firewalls can’t filter on URL paths – for example, you can’t block all of github.com except for github.com/mycompany. The better vendors don’t seem to offer decryption; you’re probably better off using a proxy (i.e., the previous option) if you want to filter on URL paths.</p>

<p>The firewall must support Geneve encapsulation, be invulnerable to SNI spoofing, fast, reliable, not <a href="https://chasersystems.com/discriminat/faq/#are-the-out-of-band-dns-lookups-susceptible-to-ip-address-mismatches">susceptible to IP address mismatches</a>, and preferably perform NAT to eliminate the need for NAT Gateways, so building an open-source alternative is not easy.</p>

<p>Regarding specific vendors, <a href="https://github.com/ChaserSystems/terraform-aws-discriminat-gwlb#deployment-examples">DiscrimiNAT</a> seems much easier to configure compared to e.g. Palo Alto Firewall,<sup id="fnref:99351" role="doc-noteref"><a href="#fn:99351" class="footnote" rel="footnote">7</a></sup> as all you do is <a href="https://chasersystems.com/docs/discriminat/aws/quick-start/#viii-configuring-a-whitelist">add FQDNs to security group descriptions</a>. However, DiscrimiNAT would need to add subaccount support for the diagram above to function to read the security groups in the ‘spoke’ account.</p>
Expand Down Expand Up @@ -382,8 +380,8 @@ <h3 id="tradeoffs">Tradeoffs</h3>
<tr>
<td>Filtering Granularity</td>
<td>None</td>
<td>FQDN or URL Path</td>
<td>FQDN</td>
<td>FQDN (or URL Path**)</td>
<td>FQDN (or URL Path**)</td>
<td>None</td>
<td>None</td>
</tr>
Expand All @@ -400,6 +398,8 @@ <h3 id="tradeoffs">Tradeoffs</h3>

<p>* = YMMV</p>

<p>** = URL path is only available to filter on if MITM is performed.<sup id="fnref:91512" role="doc-noteref"><a href="#fn:91512" class="footnote" rel="footnote">12</a></sup></p>

<h2 id="faq">FAQ</h2>

<h3 id="can-you-walk-through-the-cost-details-around-option-1">Can you walk through the cost details around Option 1?</h3>
Expand Down Expand Up @@ -563,13 +563,13 @@ <h3 id="what-happens-if-an-ec2-instance-in-a-private-subnet-gets-a-public-ip">Wh

<p>As for why it cannot respond to traffic, that is more interesting!</p>

<p>For a private subnet, the route table – which is only consulted for outgoing traffic – will have a path to a NAT Gateway, not the IGW. So response packets will reach the NAT Gateway, <a href="https://www.youtube.com/watch?app=desktop&amp;v=UP7wDBjZ37o&amp;t=35m20s">which does connection/flow tracking</a>,<sup id="fnref:91426" role="doc-noteref"><a href="#fn:91426" class="footnote" rel="footnote">12</a></sup> and get dropped because there is no existing connection.<sup id="fnref:9133" role="doc-noteref"><a href="#fn:9133" class="footnote" rel="footnote">13</a></sup></p>
<p>For a private subnet, the route table – which is only consulted for outgoing traffic – will have a path to a NAT Gateway, not the IGW. So response packets will reach the NAT Gateway, <a href="https://www.youtube.com/watch?app=desktop&amp;v=UP7wDBjZ37o&amp;t=35m20s">which does connection/flow tracking</a>,<sup id="fnref:91426" role="doc-noteref"><a href="#fn:91426" class="footnote" rel="footnote">13</a></sup> and get dropped because there is no existing connection.<sup id="fnref:9133" role="doc-noteref"><a href="#fn:9133" class="footnote" rel="footnote">14</a></sup></p>

<p>If the EC2 has UDP ports open, an attacker can receive responses, and you have a security problem. (A NACL will not help, as an Ingress deny rule blocking the Internet from hitting the EC2 will also block responses from the Internet to Egress Traffic.)</p>

<h2 id="conclusion">Conclusion</h2>

<p>Let me know how it goes limiting your Internet-exposed attack surface in an easy to understand, secure-by-default way.<sup id="fnref:92442" role="doc-noteref"><a href="#fn:92442" class="footnote" rel="footnote">14</a></sup></p>
<p>Let me know how it goes limiting your Internet-exposed attack surface in an easy to understand, secure-by-default way.<sup id="fnref:92442" role="doc-noteref"><a href="#fn:92442" class="footnote" rel="footnote">15</a></sup></p>

<p>You might still get breached, but hopefully in a more interesting way.</p>

Expand All @@ -591,7 +591,7 @@ <h2 id="footnotes">Footnotes</h2>
<p>A peering connection cannot be selected as a <a href="https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-targets.html">traffic mirror source or target</a>, but a network interface can. However, only an ENI belonging to an EC2 instance can be a mirror source, not an ENI belonging to an Interface endpoint. The documentation doesn’t mention this anywhere I could find. <a href="#fnref:985" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
</li>
<li id="fn:99" role="doc-endnote">
<p>It has <a href="https://aws.amazon.com/network-firewall/faqs/">AWS Network Firewall</a>, which can be fooled via SNI spoofing. So it is, at best, a stepping stone to keep an inventory of your Egress traffic if you can’t get a proxy up and running short-term and are not using TLS 1.3 with encrypted client hello (ECH) or encrypted SNI (ESNI). I cringe at how <a href="https://aws.amazon.com/network-firewall/faqs/">the FAQ</a> says these are not supported rather than a bypass of the product. Sadly, this euphemism <a href="https://i.imgur.com/dPyFaNK.png">isn’t unique to AWS</a>. <a href="#fnref:99" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
<p>It has <a href="https://aws.amazon.com/network-firewall/faqs/">AWS Network Firewall</a>, which is <a href="https://docs.suricata.io/en/latest/rules/tls-keywords.html">just managed Suricata</a> and can be fooled via SNI spoofing. So it is, at best, a stepping stone to keep an inventory of your Egress traffic if you can’t get a proxy or real firewall running short-term and <a href="https://docs.aws.amazon.com/network-firewall/latest/developerguide/tls-inspection-considerations.html">are not using TLS 1.3 with encrypted client hello (ECH) or encrypted SNI (ESNI)</a>. <a href="#fnref:99" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
</li>
<li id="fn:99328" role="doc-endnote">
<p>The usual suspects <a href="https://awsteele.com/blog/2022/01/20/aws-gwlb-deep-packet-manipulation.html">Aidan Steele</a>, <a href="https://web.archive.org/web/20220129101637/https://www.sentiatechblog.com/geneveproxy-an-aws-gateway-load-balancer-reference-application">Luc van Donkersgoed</a>, and <a href="https://www.lastweekinaws.com/blog/what-i-dont-get-about-the-aws-gateway-load-balancer/">Corey Quinn</a> have written about GWLB. <a href="#fnref:99328" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
Expand All @@ -611,6 +611,9 @@ <h2 id="footnotes">Footnotes</h2>
<li id="fn:996221" role="doc-endnote">
<p>See Stage 1 of Scott’s <a href="https://summitroute.com/downloads/aws_security_maturity_roadmap-Summit_Route.pdf">AWS Security Maturity Roadmap</a>, for example. <a href="#fnref:996221" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
</li>
<li id="fn:91512" role="doc-endnote">
<p>See the <a href="https://eng.lyft.com/internet-egress-filtering-of-services-at-lyft-72e99e29a4d9">“Man-in-the-Middle” section</a> of Lyft’s post, for some thoughts around this. <a href="#fnref:91512" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
</li>
<li id="fn:91426" role="doc-endnote">
<p>According to that re:Invent session from <a href="https://twitter.com/colmmacc?lang=en">Colm MacCárthaigh</a>, and me testing <a href="https://nmap.org/book/scan-methods-ack-scan.html#:~:text=ACK%20scan%20is%20enabled%20by,both%20return%20a%20RST%20packet.">ACK scanning</a> does not work through a NAT Gateway. <a href="#fnref:91426" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
</li>
Expand Down

0 comments on commit 581d7e4

Please sign in to comment.