docker + dat deployment

[joehand]

Current status on docker + dat networking troubleshooting. This is a general issue to keep visibility and update progress, not related directly to cli.

We're still having trouble accessing exposing ports directly.e.g.:

docker run -d --name some-image -p 3282 -p 3282/udp

We had an issue with using IPTables before (#503), which docker uses. So it may be related to that, not Docker directly. This is what some of our IPTables rules (generated via docker) look like:

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  tcp  --  172.19.0.7           172.19.0.7           tcp dpt:3282
MASQUERADE  udp  --  172.19.0.7           172.19.0.7           udp dpt:3282

Chain DOCKER (2 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             anywhere             tcp dpt:32769 to:172.19.0.7:3282
DNAT       udp  --  anywhere             anywhere             udp dpt:32768 to:172.19.0.7:3282

See also:

• Ports to expose when running dat doctor in docker? #841 (may not be docker related)
• exposing docker ports
• docker networking

[joehand]

Seems like it is related to a bug in discovery-swarm/channel (mafintosh/discovery-swarm#32) where the internal port doesn't match the external port and the wrong one gets reported.

[joehand]

Switching Docker networking to "host" mode works. This will use the OS ports instead of Docker's networking interface, getting around the external/internal port bug.

[mitar]

I think you are running it wrong. If you just do -p 3282 then the host port will be randomly chosen. Which means that port dat inside will think it has will be different on the outside.

I am running hypercored inside Docker using -p 3282:3282/tcp -p 3282:3282/udp and it works perfectly.

[SvenDowideit]

interestingly, I've been having issues with dat sharing that are related to this:

docker run --rm -it -p 3282:3282/udp -p 3282:3282/tcp -v share_share:/data -v share_secretkeys:/root/.dat svendowideit/dat share /data

pretty much doesn't work. I'm going to compare to running hypercored in a bit - cos this is ew!

whereas using --net host does...

[SvenDowideit]

I'm kinda wondering if there's not enough error reporting going on - I was able to run 2 dat shares on one computer - how are they both listening on the same ports?

[mitar]

@SvenDowideit see #945 about port question.

[SvenDowideit]

@mitar merci! - that would pretty much make issues when using it in docker impossible to debug

[mitar]

Also see #947.

[nettiopsu]

I have experienced the same thing while trying to dockerize dat in my local machine. I tried exposing port 3282 and with --net host as well, but local host machine does not connect to dat in docker. I did not manage to connect to it from dat cli neither from Beaker browser. However, non dockerized dat works fine in local machine and you can access it directly e.g. from Beaker.

You can connect also from any remote machine directly to local dat docker (!). It is enough just to expose 3282, no need in --net host

Agree here with @SvenDowideit that it is almost impossible to understand where dat fails without proper debugging messages. I was able to guess that only local machine is not working by trying and guessing

[pfrazee]

Did you try running dat with the DEBUG env var set to *? That’ll give a lot of logging

…

On Jul 24, 2018, at 3:53 AM, nettiopsu ***@***.***> wrote: I have experienced the same thing while trying to dockerize dat. I tried exposing ports and with --net host as well, but it seems that host machine does not see it for some reason. dat doctor runs well with first tests without any errors, second test starts to use random ports and host machine fails to find it Agree here with @SvenDowideit that it is almost impossible to understand where dat fails without proper debugging messages — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[rjcorwin]

I'm facing the same issue, made worse by the fact I can't use --net host because my app relies on linked containers for the database. Watch out for the dreaded undefined behavior ;).

docker: Error response from daemon: conflicting options: host type networking can't be used with links. This would result in undefined behavior.

[max-mapper]

I looked into this a while ago and found something about the docker NAT implementation having a bug with UDP hole punching. Can't find the github thread now but thought I'd comment in case that ends up being relevant.

[rjcorwin]

Thanks @maxogden I saw your status from two years ago to remind us two years in the future :-) https://twitter.com/maxogden/status/913874394421719040

This issue that @joehand references at the top may be the issue you are thinking of #841

For the time being I'm mounting a volume and running dat-party (https://github.com/rjsteinert/dat-party) on the host machine so that any folder the container places in that volume will be shared as a Dat Archive from the host machine without having to punch through Docker NAT. It's a funny workaround.

[nettiopsu]

@pfrazee yes I tried, but it lacks still some data for understanding, sometimes it does not show even URL which dat is sharing

By the way, @rjsteinert reminded me about other Docker problem with dat share if you run it in MacOS. Docker has problems with sending notifications about file change (inotify) in this system. So if you run dat share for shared volume in Docker machine, then if you make changes in this volume from host machine, then dat will not get this changes neither share them in network. Though dat share works if changes were made from the same Docker machine, not from host. Taking this into account, it maybe wise for dat cli not to rely always on inotify, but also run sometimes in low priority scan of all files in folder for changes (?) at least for Docker

[mitar]

I didn't have problems running dat inside docker with regular Docker NAT, the problem was just that dat is picking random ports. If you could configure which port dat really uses, things work great.

[thadguidry]

@mitar but it does ? It picks first available, if the default port is already in use: https://github.com/datproject/dat/blob/master/bin/cli.js#L39

[mitar]

Yes, and so it is tricky to know that default port is not being used. So it is tricky to debug. I would prefer a mode which would be "use given port or die".

i have had issues in Docker only because my dat daemon was listening on ports I have not had mapped. And it was hard to debug.

[joehand]

Yes, and so it is tricky to know that default port is not being used. So it is tricky to debug. I would prefer a mode which would be "use given port or die".

@mitar, the latest release of the dat cli fixed this =). Thanks for the suggestions on that.

[mitar]

Then it should work. (If nothing else changed. I have not used dat for few months now.)

I tried exposing port 3282

@nettiopsu, did you expose both UDP and TCP ports? Use -p 3282:3282/tcp -p 3282:3282/udp.

[nettiopsu]

@mitar just tried, it did not help, same situation - can not connect from host machine, but can connect from a remote one

I have tried also this new port parameter to use some another port, but then even remote machine stopped working

[mitar]

Oh, yes, from host it does not work. This is a general issue with NAT. I missed that in your original comment.

[drelephant]

Would you mind explaining more about what the general issue with NAT is?

[mitar]

I mean, it depends how your Docker is configured, but if whole Docker network is behind one IP (NAT), you might not be able to access a particular IP of the container from outside (host). You can map port to that one IP and then access the port on it.