Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible interop problems with other DoH servers. #59

Open
complex2d opened this issue Dec 24, 2020 · 9 comments
Open

Possible interop problems with other DoH servers. #59

complex2d opened this issue Dec 24, 2020 · 9 comments
Labels
bug Something isn't working

Comments

@complex2d
Copy link

I've made a couple of attempts to get Flamethrower working against DoH servers in a test harness environment running Ubuntu 20.04.1 LTS, and so far I'm not having any luck.

My first attempt was to deploy Apache 2.4.41 as the HTTP/2 service handler/muxer and DoH Server desribed here: https://github.com/m13253/dns-over-https

The only major differences in my deplpyment are the DoH server and Apache are not on the same box (mostly for tcpdump and troubleshooting reasons), and the fact I'm not using OSCP stapling or a real CA signed cert, which as far as I know should be perfectly fine for an isolated test harness.

My Apache 2.4.41 config is very simple, and just terminates the H2 connection and proxies the query to the DoH server:

<IfModule mod_ssl.c>
        SSLProtocol TLSv1.2
        SSLHonorCipherOrder On
        SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+3DES:!aNULL:!MD5:!DSS:!eNULL:!EXP:!LOW:!MD5
        # SSLUseStapling on
        # SSLStaplingCache shmcb:/var/lib/apache2/stapling_cache(512000)

<VirtualHost _default_:443>
        ServerName kurochan
        Protocols h2 http/1.
        ProxyPass /dns-query http://10.10.20.20:8053/dns-query
        ProxyPassReverse /dns-query http://10.10.20.20:8053/dns-query

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on

        #   A self-signed (snakeoil) certificate can be created by installing
        #   the ssl-cert package. See
        #   /usr/share/doc/apache2.2-common/README.Debian.gz for more info.
        #   If both key and certificate are stored in the same file, only the
        #   SSLCertificateFile directive is needed.
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

        ErrorLog /var/log/apache2/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel debug

        CustomLog /var/log/apache2/ssl_access.log combined

</VirtualHost>
</IfModule>

When I use a DoH test client such as the doh-client that comes with facebookexperimental's doh-proxy (https://github.com/facebookexperimental/doh-proxy), it works just fine, as shown here:

root@planck2:~# doh-client-fb --qname www.doh1.com --qtype A --domain 10.10.20.21 --remote-address 10.10.20.21 --insecure
2020-12-23 11:35:53,462:    DEBUG: Opening connection to 10.10.20.21
2020-12-23 11:35:53,472:    DEBUG: Query parameters: {'dns': 'AAABAAABAAAAAAAAA3d3dwRkb2gxA2NvbQAAAQAB'}
2020-12-23 11:35:53,473:    DEBUG: Stream ID: 1 / Total streams: 0
2020-12-23 11:35:53,478:    DEBUG: Response headers: [(':status', '200'), ('date', 'Wed, 23 Dec 2020 19:35:53 GMT'), ('server', 'DNS-over-HTTPS/2.2.5 (+https://github.com/m13253/dns-over-https)'), ('access-control-allow-headers', 'Content-Type'), ('access-control-allow-methods', 'GET, HEAD, OPTIONS, POST'), ('access-control-allow-origin', '*'), ('access-control-max-age', '3600'), ('cache-control', 'private, max-age=500'), ('content-type', 'application/dns-message'), ('expires', 'Wed, 23 Dec 2020 19:44:13 GMT'), ('last-modified', 'Wed, 23 Dec 2020 19:35:53 GMT'), ('vary', 'Accept'), ('x-powered-by', 'DNS-over-HTTPS/2.2.5 (+https://github.com/m13253/dns-over-https)'), ('content-length', '234')]
id 0
opcode QUERY
rcode NOERROR
flags QR AA RD RA
edns 0
payload 4096
option ECS 10.10.20.0/24 scope/0
;QUESTION
www.doh1.com. IN A
;ANSWER
www.doh1.com. 1800 IN A 10.10.10.12
www.doh1.com. 1800 IN A 10.10.10.10
www.doh1.com. 1800 IN A 10.10.10.13
www.doh1.com. 1800 IN A 10.10.10.14
www.doh1.com. 1800 IN A 10.10.10.11
;AUTHORITY
doh1.com. 1800 IN NS ns.
;ADDITIONAL
ns. 500 IN A 1.1.1.1
2020-12-23 11:35:53,485:    DEBUG: Response trailers: {}

Apache ssl_access.log:

10.10.20.20 - - [23/Dec/2020:11:35:53 -0800] "GET /dns-query?dns=AAABAAABAAAAAAAAA3d3dwRkb2gxA2NvbQAAAQAB HTTP/2.0" 200 587 "-" "-"

However when I try to run Flamethrower with extremely light serial queries pointed at the Apache server, Apache rejects the HTTP/2. It appears it does not like something about the HEADERS frame, but I'm not entirely sure what:

root@planck2:~# docker run ns1labs/flame -P doh https://10.10.20.21/dns-query -r www.doh1.com -q 1 -c 1 -d 100
binding to 0.0.0.0
flaming target(s) [10.10.20.21] on port 443 with 1 concurrent generators, each sending 1 queries every 100ms on protocol doh
query generator [static] contains 1 record(s)
0.995918s: send: 1, avg send: 1, recv: 0, avg recv: 0, min/avg/max resp: 0/-nan/0ms, in flight: 1, timeouts: 0
^C1.18096s: send: 0, avg send: 1, recv: 0, avg recv: 0, min/avg/max resp: 0/-nan/0ms, in flight: 1, timeouts: 0
stopping, waiting up to 3s for in flight to finish...

------
run id      : 7ffd9bda0320
run start   : 2020-12-23T19:50:19Z
runtime     : 4.18091 s
total sent  : 1
total rcvd  : 0
min resp    : 0 ms
avg resp    : -nan ms
max resp    : 0 ms
avg r qps   : 0
avg s qps   : 1
avg pkt     : 55 bytes
tcp conn.   : 1
timeouts    : 1 (100%)
bad recv    : 0
net errors  : 0

Apache error log:

[Wed Dec 23 11:50:19.539183 2020] [ssl:info] [pid 13986:tid 140707071219456] [client 10.10.20.20:37529] AH01964: Connection to child 13 established (server kurochan:443)
[Wed Dec 23 11:50:19.539398 2020] [ssl:debug] [pid 13986:tid 140707071219456] ssl_engine_kernel.c(2372): [client 10.10.20.20:37529] AH02645: Server name not provided via TLS extension (using default/first virtual host)
[Wed Dec 23 11:50:19.539464 2020] [ssl:debug] [pid 13986:tid 140707071219456] ssl_engine_kernel.c(2372): [client 10.10.20.20:37529] AH02645: Server name not provided via TLS extension (using default/first virtual host)
[Wed Dec 23 11:50:19.539591 2020] [core:debug] [pid 13986:tid 140707071219456] protocol.c(2257): [client 10.10.20.20:37529] AH03155: select protocol from h2,http/1., choices=h2 for server kurochan
[Wed Dec 23 11:50:19.539709 2020] [core:debug] [pid 13986:tid 140707071219456] protocol.c(2302): [client 10.10.20.20:37529] AH03156: select protocol, proposals=h2 preferences=h2,http/1. configured=h2,http/1.
[Wed Dec 23 11:50:19.539743 2020] [core:debug] [pid 13986:tid 140707071219456] protocol.c(2320): [client 10.10.20.20:37529] AH03157: selected protocol=h2
[Wed Dec 23 11:50:19.585575 2020] [ssl:debug] [pid 13986:tid 140707071219456] ssl_engine_kernel.c(2233): [client 10.10.20.20:37529] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Wed Dec 23 11:50:19.585748 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(958): [client 10.10.20.20:37529] AH03200: h2_session(13,INIT,0): created, max_streams=100, stream_mem=32768, workers_limit=6, workers_max=37, push_diary(type=1,N=256)
[Wed Dec 23 11:50:19.585811 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1040): [client 10.10.20.20:37529] AH03201: h2_session(13,INIT,0): start, INITIAL_WINDOW_SIZE=65535, MAX_CONCURRENT_STREAMS=100
[Wed Dec 23 11:50:19.585846 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(2134): [client 10.10.20.20:37529] AH03079: h2_session(13,INIT,0): started on kurochan:443
[Wed Dec 23 11:50:19.585869 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,BUSY,0): transit [INIT] -- init --> [BUSY]
[Wed Dec 23 11:50:19.585913 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,0): sent FRAME[SETTINGS[length=6, stream=0]], frames=0/1 (r/s)
[Wed Dec 23 11:50:19.585966 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,0): sent FRAME[WINDOW_UPDATE[stream=0, incr=2147418112]], frames=0/2 (r/s)
[Wed Dec 23 11:50:19.586062 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,IDLE,0): transit [BUSY] -- no io (keepalive) --> [IDLE]
[Wed Dec 23 11:50:19.629428 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(341): [client 10.10.20.20:37529] AH03066: h2_session(13,IDLE,0): recv FRAME[SETTINGS[length=6, stream=0]], frames=0/2 (r/s)
[Wed Dec 23 11:50:19.629545 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(341): [client 10.10.20.20:37529] AH03066: h2_session(13,IDLE,0): recv FRAME[SETTINGS[ack=1, stream=0]], frames=1/2 (r/s)
[Wed Dec 23 11:50:19.629616 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_stream.c(543): [client 10.10.20.20:37529] AH03082: h2_stream(13-1,IDLE): created
[Wed Dec 23 11:50:19.629650 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,BUSY,1): transit [IDLE] -- stream change --> [BUSY]
[Wed Dec 23 11:50:19.629697 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(221): [client 10.10.20.20:37529] AH03063: h2_session(13,BUSY,1): recv invalid FRAME[HEADERS[length=71, hend=1, stream=1, eos=1]], frames=2/2 (r/s)
[Wed Dec 23 11:50:19.629738 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,1): sent FRAME[SETTINGS[ack=1, stream=0]], frames=2/3 (r/s)
[Wed Dec 23 11:50:19.629762 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,BUSY,1): sent FRAME[RST_STREAM[length=4, flags=0, stream=1]], frames=2/4 (r/s)
[Wed Dec 23 11:50:19.629798 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(267): [client 10.10.20.20:37529] AH03065: h2_stream(13-1,CLOSED): closing with err=1 protocol error
[Wed Dec 23 11:50:19.629895 2020] [http2:debug] [pid 13986:tid 140707071219456] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,IDLE,0): transit [BUSY] -- no io (keepalive) --> [IDLE]
[Wed Dec 23 11:50:22.548659 2020] [ssl:info] [pid 13986:tid 140707062826752] (70014)End of file found: [client 10.10.20.20:37529] AH01991: SSL input filter read failed.
[Wed Dec 23 11:50:22.548889 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(2192): (70014)End of file found: [client 10.10.20.20:37529] AH03403: h2_session(13,IDLE,0): no data, error
[Wed Dec 23 11:50:22.548927 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(1788): [client 10.10.20.20:37529] AH03401: h2_session(13,IDLE,0): conn error -> shutdown
[Wed Dec 23 11:50:22.548966 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(593): [client 10.10.20.20:37529] AH03068: h2_session(13,IDLE,0): sent FRAME[GOAWAY[error=0, reason='timeout', last_stream=1]], frames=2/5 (r/s)
[Wed Dec 23 11:50:22.549066 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(753): [client 10.10.20.20:37529] AH03069: h2_session(13,IDLE,0): sent GOAWAY, err=0, msg=timeout
[Wed Dec 23 11:50:22.549109 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,DONE,0): transit [IDLE] -- local goaway --> [DONE]
[Wed Dec 23 11:50:22.549166 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_conn.c(213): (70014)End of file found: [client 10.10.20.20:37529] AH03045: h2_session(13,DONE,0): process, closing conn
[Wed Dec 23 11:50:22.549205 2020] [http2:debug] [pid 13986:tid 140707062826752] h2_session.c(1698): [client 10.10.20.20:37529] AH03078: h2_session(13,CLEANUP,0): transit [DONE] -- pre_close --> [CLEANUP]
[Wed Dec 23 11:50:22.549305 2020] [ssl:debug] [pid 13986:tid 140707062826752] ssl_engine_io.c(1106): [client 10.10.20.20:37529] AH02001: Connection closed to child 14 with standard shutdown (server kurochan:443)

Apache really doesn't like something about the HTTP/2, claiming it received an invalid frame.

My final attempt was to give up on Apache and just spin up the facebookexperimental doh-proxy, but pointing Flamethrower to that also fails.

Launching doh-proxy:

NOTE: doh-proxy was slightly modified to print out the headers so I could see what doh-proxy was getting from the Flamethrower client.

root@planck2:~# doh-proxy --certfile /etc/ssl/certs/ssl-cert-snakeoil.pem --keyfile /etc/ssl/private/ssl-cert-snakeoil.key --listen-address 10.10.20.20  --upstream-resolver localhost
2020-12-23 13:45:03,403:     INFO: Serving on <Server sockets=[<socket.socket fd=8, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.10.20.20', 443)>]>

Running Flamethrower:

root@planck2:~# docker run ns1labs/flame -P doh https://10.10.20.20/dns-query -r www.doh1.com -q 1 -c 1 -d 100
binding to 0.0.0.0
flaming target(s) [10.10.20.20] on port 443 with 1 concurrent generators, each sending 1 queries every 100ms on protocol doh
query generator [static] contains 1 record(s)
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data
malformed data

Output from doh-proxy (printing the headers list):

[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
[(':method', 'GET'), (':scheme', ''), (':authority', ''), (':path', '?dns=M5sBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]

As you can see above, Flamethrower is claiming it's getting back malformed data which could be true, but it also looks like the headers being sent by Flamethrower might be an issue. The path header is supposed to contain the complete URI, including the /dns-query part, but it's missing for some reason. Scheme and authority should probably also be set. Any ideas why this might be the case?

Am I doing something wrong here in how I'm executing Flamethrower from Docker? Any help/guidance would be super cool.

Thanks in advance, and happy holidays! :)

@weyrick
Copy link
Contributor

weyrick commented Dec 29, 2020

@complex2d Thanks for the detailed report! It may take a few days due to the holidays but we will investigate this. cc @jwijenbergh for visibility as well.

@jwijenbergh
Copy link
Contributor

jwijenbergh commented Dec 29, 2020

@weyrick Thanks for the visibility.
@complex2d
Thanks for the very detailed report! I am trying to reproduce this with:

doh-proxy --upstream-resolver 8.8.8.8 --certfile cert.pem --keyfile key.pem --listen-address 0.0.0.0 (modified to use port 1337)

where cert.pem and key.pem are self signed using openssl.

flame -q 1 -c 1 -n 1 -p 1337 -r google.com -P doh -M GET https://0.0.0.0/dns-query

This ends up with a correct query and response.

And also the headers that are being send with flamethrower seem to be correct

PATH: /dns-query?dns=rigBAAABAAAAAAABBmdvb2dsZQNjb20AAAEAAQAAKQTQAAAAAAAA
SCHEME: https
AUTHORITY: 0.0.0.0

I had a feeling that the url parsing library that we use might be faulty when an ip is supplied, this does not seem to be the case.

@complex2d
can you add the following prints to httpssession.ccp?

diff --git a/flame/httpssession.cpp b/flame/httpssession.cpp
index 1a9b98a..3e4f06d 100644
--- a/flame/httpssession.cpp
+++ b/flame/httpssession.cpp
@@ -54,6 +54,9 @@ std::unique_ptr<http2_stream_data> HTTPSSession::create_http2_stream_data(std::u
         path.append("?dns=");
         path.append(data.get(), len);
     }
+    std::cout << "PATH: " << path << std::endl;
+    std::cout << "SCHEME: " << scheme << std::endl;
+    std::cout << "AUTHORITY: " << authority << std::endl;
     std::string streamData(data.get(), len);
     auto root = std::make_unique<http2_stream_data>(scheme, authority, path, stream_id, streamData);
     return root;

This is to verify if the headers are actually sent correctly (which as you stated they probably aren't). Hopefully I will get some more helpful information to help debug this.

@complex2d
Copy link
Author

I'm on holiday right now, but I will do this as soon as I possibly can and get back to you. If it's at all helpful, the version I was using was the one pulled from the public docker image, rather than one I built myself. I will try building from source with the additional prints above asap.

@complex2d
Copy link
Author

I built from source with the logging requested, and it works fine now... I feared that might be the case:

To Apache:

root@planck2:/var/tmp/flamethrower-master/build# ./flame -P doh https://10.10.20.21/dns-query -r www.doh1.com -q 1 -c 1 -d 100
binding traffic generators to 0.0.0.0
flaming target(s) [10.10.20.21] on port 443 with 1 concurrent generators, each sending 1 queries every 100ms on protocol doh
query generator [static] contains 1 record(s)
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
0.968474s: send: 9, avg send: 9, recv: 9, avg recv: 9, min/avg/max resp: 2.228/3.97549/13.7565ms, in flight: 1, timeouts: 0
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
PATH: /dns-query?dns=6-ABAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.21
^C1.19164s: send: 2, avg send: 5, recv: 2, avg recv: 5, min/avg/max resp: 2.592/2.60703/2.62206ms, in flight: 1, timeouts: 0

------
run id      : 7ffcb8694280
run start   : 2021-01-06T00:12:42Z
runtime     : 1.1945 s
total sent  : 11
total rcvd  : 11
min resp    : 2.228 ms
avg resp    : 3.29126 ms
max resp    : 13.7565 ms
avg r qps   : 5
avg s qps   : 5
avg pkt     : 55 bytes
tcp conn.   : 11
timeouts    : 0 (0%)
bad recv    : 0
net errors  : 0
responses   :
  NOERROR: 11

(Apache log omitted because it's fine)

To doh-proxy:

root@planck2:/var/tmp/flamethrower-master/build# ./flame -P doh https://10.10.20.20/dns-query -r www.doh1.com -q 1 -c 1 -d 100
binding traffic generators to 0.0.0.0
flaming target(s) [10.10.20.20] on port 443 with 1 concurrent generators, each sending 1 queries every 100ms on protocol doh
query generator [static] contains 1 record(s)
PATH: /dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.20
PATH: /dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.20
PATH: /dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.20
PATH: /dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.20
PATH: /dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA
SCHEME: https
AUTHORITY: 10.10.20.20
^C0.588232s: send: 5, avg send: 5, recv: 5, avg recv: 5, min/avg/max resp: 6.18294/6.80179/8.78524ms, in flight: 1, timeouts: 0

------
run id      : 7fff23f09bf0
run start   : 2021-01-06T00:13:43Z
runtime     : 0.591063 s
total sent  : 5
total rcvd  : 5
min resp    : 6.18294 ms
avg resp    : 6.80179 ms
max resp    : 8.78524 ms
avg r qps   : 5
avg s qps   : 5
avg pkt     : 55 bytes
tcp conn.   : 6
timeouts    : 0 (0%)
bad recv    : 0
net errors  : 0
responses   :
  NOERROR: 5

output from doh-proxy:

root@planck2:/etc/apache2/sites-enabled# doh-proxy --certfile /etc/ssl/certs/ssl-cert-snakeoil.pem --keyfile /etc/ssl/private/ssl-cert-snakeoil.key --listen-address 10.10.20.20  --upstream-resolver localhost
2021-01-05 16:12:25,730:     INFO: Serving on <Server sockets=[<socket.socket fd=6, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.10.20.20', 443)>]>

[(':method', 'GET'), (':scheme', 'https'), (':authority', '10.10.20.20'), (':path', '/dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
2021-01-05 16:13:43,097:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 RD
2021-01-05 16:13:43,099:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 36972 RD
2021-01-05 16:13:43,101:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 36972 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 2ms
2021-01-05 16:13:43,102:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 4ms
[(':method', 'GET'), (':scheme', 'https'), (':authority', '10.10.20.20'), (':path', '/dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
2021-01-05 16:13:43,211:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 RD
2021-01-05 16:13:43,212:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 63679 RD
2021-01-05 16:13:43,214:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 63679 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 1ms
2021-01-05 16:13:43,214:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 3ms
[(':method', 'GET'), (':scheme', 'https'), (':authority', '10.10.20.20'), (':path', '/dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
2021-01-05 16:13:43,328:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 RD
2021-01-05 16:13:43,329:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 8862 RD
2021-01-05 16:13:43,331:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 8862 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 1ms
2021-01-05 16:13:43,332:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 3ms
[(':method', 'GET'), (':scheme', 'https'), (':authority', '10.10.20.20'), (':path', '/dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
2021-01-05 16:13:43,445:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 RD
2021-01-05 16:13:43,447:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 41425 RD
2021-01-05 16:13:43,448:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 41425 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 1ms
2021-01-05 16:13:43,449:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 3ms
[(':method', 'GET'), (':scheme', 'https'), (':authority', '10.10.20.20'), (':path', '/dns-query?dns=n7MBAAABAAAAAAABA3d3dwRkb2gxA2NvbQAAAQABAAApBNAAAAAAAAA'), ('accept', 'application/dns-message')]
2021-01-05 16:13:43,563:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 RD
2021-01-05 16:13:43,564:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 64792 RD
2021-01-05 16:13:43,566:     INFO: [DNS] 10.10.20.20 www.doh1.com. A IN 64792 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 1ms
2021-01-05 16:13:43,566:     INFO: [HTTPS] 10.10.20.20 www.doh1.com. A IN 40883 QR/AA/RD/RA 5/0/0 0/0/4096 NOERROR 3ms

Given that, I suspect the issue might be with the public docker image. I can revert right back to the strange non-working behavior if I return to using the docker image rather than the version I just built from source. I'm fairly confident my docker environment is fine, because I can run ns1labs/flame without issue, and it works great for standard DNS, just not DoH.
For now I am unblocked though, and can continue using my locally built copy. If you want me to test anything else, let me know and I'll try to make some time.

@weyrick
Copy link
Contributor

weyrick commented Jan 10, 2021

Interesting, I will investigate the public docker image to see if I can find a problem with the build. Thanks again for the report.

@weyrick
Copy link
Contributor

weyrick commented Feb 8, 2021

@complex2d can you let me know what versions of libraries you built with locally, especially nghttp2?

@complex2d
Copy link
Author

@complex2d can you let me know what versions of libraries you built with locally, especially nghttp2?

root@planck2:/etc/apache2/sites-enabled# dpkg -l | grep nghttp; dpkg -l | grep cmake; dpkg -l | grep libuv; dpkg -l | grep libldns; dpkg -l | grep gnutls; dpkg -l | grep g++;
ii libnghttp2-14:amd64 1.40.0-1build1 amd64 library implementing HTTP/2 protocol (shared library)
ii libnghttp2-dev 1.40.0-1build1 amd64 library implementing HTTP/2 protocol (development files)
ii nghttp2 1.40.0-1build1 all server, proxy and client implementing HTTP/2
ii nghttp2-client 1.40.0-1build1 amd64 client implementing HTTP/2 protocol
ii nghttp2-proxy 1.40.0-1build1 amd64 reverse proxy implementing HTTP/2 protocol
ii nghttp2-server 1.40.0-1build1 amd64 server implementing HTTP/2 protocol
ii cmake 3.16.3-1ubuntu1 amd64 cross-platform, open-source make system
ii cmake-data 3.16.3-1ubuntu1 all CMake data files (modules, templates and documentation)
ii libuv1:amd64 1.34.2-1ubuntu1.1 amd64 asynchronous event notification library - runtime library
ii libuv1-dev:amd64 1.34.2-1ubuntu1.1 amd64 asynchronous event notification library - development files
ii libldns-dev:amd64 1.7.0-4.1ubuntu1 amd64 ldns library for DNS programming
ii libldns2:amd64 1.7.0-4.1ubuntu1 amd64 ldns library for DNS programming
ii gnutls-bin 3.6.13-2ubuntu1.3 amd64 GNU TLS library - commandline utilities
ii libcurl3-gnutls:amd64 7.68.0-1ubuntu2.4 amd64 easy-to-use client-side URL transfer library (GnuTLS flavour)
ii libgnutls-dane0:amd64 3.6.13-2ubuntu1.3 amd64 GNU TLS library - DANE security support
ii libgnutls-openssl27:amd64 3.6.13-2ubuntu1.3 amd64 GNU TLS library - OpenSSL wrapper
ii libgnutls28-dev:amd64 3.6.13-2ubuntu1.3 amd64 GNU TLS library - development files
ii libgnutls30:amd64 3.6.13-2ubuntu1.3 amd64 GNU TLS library - main runtime library
ii libgnutlsxx28:amd64 3.6.13-2ubuntu1.3 amd64 GNU TLS library - C++ runtime library
ii g++ 4:9.3.0-1ubuntu2 amd64 GNU C++ compiler
ii g++-7 7.5.0-6ubuntu2 amd64 GNU C++ compiler
ii g++-9 9.3.0-17ubuntu1~20.04 amd64 GNU C++ compiler

If you see anything missing there, let me know. As a side note, I've been meaning to spend more time with this and look into some other issues, but I've been absolutely slammed with work lately. I may be a bit slow to respond.

@weyrick weyrick added the bug Something isn't working label Jun 3, 2021
@arbv
Copy link
Contributor

arbv commented Jun 9, 2021

There is a possibility that this MR has solved at least some of the interoperability problems:

#67

@complex2d
Copy link
Author

A few years late, but wanted to circle back around and mention that I've largely left the industry. I doubt I'll be able to help out with any testing from here on out. If you believe this is resolved, please feel free to close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants