Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lancache Epic and RIOT (probably some others too, steam is fine) downloading keep doing start and stop, start and pause etc. #192

Open
coddec opened this issue Jun 9, 2024 · 19 comments

Comments

@coddec
Copy link

coddec commented Jun 9, 2024

Describe the issue you are having

Case 1
Without lancache, Epic downloads GTA fine with full WAN speed

Case 2
With lancache, Epic prefill (https://github.com/tpill90/steam-lancache-prefill) for GTA will download at full WAN speed no stop or pause.

Case 3
With lancache, Epic and RIOT games downloading via Epic client or RIOT client will download at full speed at start then drop to 0, stay at 0 for sometime then back to full speed, then repeat.
While this is happening lancache will have error log entries like shown below
(I can see that the container is not downloading too, certainly not that container is downloading the client is waiting case. In other words not this https://lancache.net/docs/common-issues/#slow-download-speeds)
The Epic client also show error message "We're sorry. It looks like we're having trouble connecting. We'll continue to retry" Click on it lead to https://www.epicgames.com/help/en-US/c-Category_EpicGamesStore/c-EpicGamesStore_ErrorCodes/is-0005-unable-to-connect-a000084850 stating "IS-0005: Unable to connect, This error is caused when your computer is not connected to the internet."
image

My internet is certainly working and there's no fluctuation, otherwise Epic prefill will experience the same issue.
Client computer to LAN and internet connection is also working fine.
Content are cached though. Enough disk space, enough CPU and RAM etc. Client PC is fine too.
12MB/s should not upset NIC since tested on 1g NIC and 10g sfp+ dac.
This issue makes the download taking way longer than it should be.

Case 4
Steam download works fine. Content are cached.

error.log
2024/06/09 12:09:40 [error] 1842#1842: *481 upstream prematurely closed connection while reading upstream, client: 192.168.0.11, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk HTTP/1.1", upstream: "http://127.0.0.1:3128/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk", host: "cloudflare.epicgamescdn.com"
2024/06/09 12:09:44 [error] 1834#1834: *1128 upstream prematurely closed connection while reading upstream, client: 192.168.0.11, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk HTTP/1.1", upstream: "http://127.0.0.1:3128/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk", host: "cloudflare.epicgamescdn.com"
upstream-error.log
2024/06/09 12:09:37 [error] 1869#1869: *829 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/84/D93D5307BC2E3B0A_E1FA523C40934DA164061687A71F786F.chunk HTTP/1.0", upstream: "http://104.18.50.190:80/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/84/D93D5307BC2E3B0A_E1FA523C40934DA164061687A71F786F.chunk", host: "cloudflare.epicgamescdn.com"
2024/06/09 12:09:40 [error] 1899#1899: *889 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk HTTP/1.0", upstream: "http://104.18.50.190:80/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/34/4FC5209EAD7B5D6D_F9414B27445C8D727F27F8810CE708A1.chunk", host: "cloudflare.epicgamescdn.com"
2024/06/09 12:09:44 [error] 1831#1831: *1241 upstream timed out (110: Connection timed out) while reading upstream, client: 127.0.0.1, server: , request: "GET /Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk HTTP/1.0", upstream: "http://104.18.50.184:80/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/88/5FB167B913870FEC_8C85A34245324631437C68AD8C23C984.chunk", host: "cloudflare.epicgamescdn.com"
stream-error.log
2024/06/09 12:49:00 [error] 1831#1831: *11952 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 192.168.0.11, server: 0.0.0.0:443, upstream: "52.169.159.67:443", bytes from/to client:1076/3101, bytes from/to upstream:3101/1296
2024/06/09 12:49:03 [error] 1831#1831: *11951 recv() failed (104: Connection reset by peer) while proxying and reading from upstream, client: 192.168.0.11, server: 0.0.0.0:443, upstream: "13.74.139.29:443", bytes from/to client:1077/3101, bytes from/to upstream:3101/1297
C:\Users\[username]\AppData\Local\EpicGamesLauncher\Saved\Logs\EpicGamesLauncher.log
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/21/6319E1D547314493_723E74A54B7801F60D864397C8A55F75.chunk
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: request failed, libcurl error: 0 (No error)
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x261a5f33c50 [can pipeline])
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 1 (Re-using existing connection! (#639) with host cloudflare.epicgamescdn.com)
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#639))
[2024.06.10-04.24.51:453][353]LogHttp: Warning: 00000261AFC8E930: libcurl info message cache 3 (Closing connection 639)
[2024.06.10-04.24.51:488][354]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/21/6319E1D547314493_723E74A54B7801F60D864397C8A55F75.chunk
[2024.06.10-04.27.35:943][418]LogEOSMessageService: Error: Unable to find port.
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/27/F9BB3A429600D379_D84F1160496C7259D4D650BC32B1B74F.chunk
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: request failed, libcurl error: 0 (No error)
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x261a5f33c50 [can pipeline])
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 1 (Re-using existing connection! (#644) with host cloudflare.epicgamescdn.com)
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#644))
[2024.06.10-04.28.13:571][847]LogHttp: Warning: 00000261AFC8F6E0: libcurl info message cache 3 (Closing connection 644)
[2024.06.10-04.28.13:637][849]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/27/F9BB3A429600D379_D84F1160496C7259D4D650BC32B1B74F.chunk
C:\Users\[username]\AppData\Local\EpicGamesLauncher\Saved\Logs\EpicGamesLauncher.log
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/36/E07192FAF958A2D6_F6023FE84A6C3AAB76E8818FA9596FD3.chunk
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: request failed, libcurl error: 0 (No error)
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x239b9683e50 [can pipeline])
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 1 (Re-using existing connection! (#37) with host cloudflare.epicgamescdn.com)
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#37))
[2024.06.10-06.35.55:019][657]LogHttp: Warning: 00000239BBC54980: libcurl info message cache 3 (Closing connection 37)
[2024.06.10-06.35.55:052][659]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/36/E07192FAF958A2D6_F6023FE84A6C3AAB76E8818FA9596FD3.chunk
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: HTTP request timed out after 30.00 seconds URL=http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/58/F6295F141ABBF1C9_CC10965F49FE379F76A96094FF9791B3.chunk
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: request failed, libcurl error: 0 (No error)
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 0 (Found bundle for host cloudflare.epicgamescdn.com: 0x239b9683e50 [can pipeline])
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 1 (Re-using existing connection! (#23) with host cloudflare.epicgamescdn.com)
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 2 (Connected to cloudflare.epicgamescdn.com (192.168.0.11) port 80 (#23))
[2024.06.10-06.35.59:256][912]LogHttp: Warning: 00000239BBC5ADE0: libcurl info message cache 3 (Closing connection 23)
[2024.06.10-06.35.59:273][913]LogCloudChunkSource: FAILED: http://cloudflare.epicgamescdn.com/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/58/F6295F141ABBF1C9_CC10965F49FE379F76A96094FF9791B3.chunk
[2024.06.10-06.35.59:273][913]LogPortal: Error: FAlertReporter: Reporting Alert code: IS-0005
[2024.06.10-06.35.59:273][913]LogAlertListener: No user messaging in AlertMessageRepository for: IS-0005
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: HTTP request timed out after 30.00 seconds URL=http://epicgames-download1.akamaized.net/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/08/96490E5B268F9C52_2C9FCF2C4948373FD3010E8E1B9AF54A.chunk
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: request failed, libcurl error: 0 (No error)
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 0 (  Trying 192.168.0.11...)
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 1 (TCP_NODELAY set)
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 2 (Connected to epicgames-download1.akamaized.net (192.168.0.11) port 80 (#39))
[2024.06.10-06.36.06:302][286]LogHttp: Warning: 00000239BBC5BB90: libcurl info message cache 3 (Closing connection 39)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: HTTP request timed out after 30.00 seconds URL=http://epicgames-download1.akamaized.net/Builds/Org/o-dhz7kpvrvqngzdpnx5vn3jrljj53ca/68d2cc08f9a94b8fb51af4f5cfa6d41b/default/ChunksV4/04/2E4D5B054046D34A_B4913E714BD8A998C6763C8BB0E8F513.chunk
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: request failed, libcurl error: 0 (No error)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 0 (Found bundle for host epicgames-download1.akamaized.net: 0x239b9680ed0 [serially])
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 1 (Hostname epicgames-download1.akamaized.net was found in DNS cache)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 2 (  Trying 192.168.0.11...)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 3 (TCP_NODELAY set)
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 4 (Connected to epicgames-download1.akamaized.net (192.168.0.11) port 80 (#41))
[2024.06.10-06.36.06:967][306]LogHttp: Warning: 00000239BBC5C4B0: libcurl info message cache 5 (Closing connection 41)

Tried to clear all cache
Tried to change slice size between 1M - 8M
Tried to disable slice, by setting to 0
Add multiple IP to lancache host tested, no help
No client RAM, CPU, Disk issue
No lancache host RAM, CPU, Disk issue
...

How are you running the container(s)

Case 1
Container in OMV VM, OMV VM in PVE
monolithic container using host network

Case 2
Container in LXC container in PVE

Case 3
Container in Ubuntu on another laptop bare metal
monolithic container using docker0 network

Same results as described.

DNS Configuration

Unbound on pfsense as main DNS with custom DNS settings like below covering all uklancache defined domains
local-zone: "cloudflare.epicgamescdn.com" redirect
local-data: "cloudflare.epicgamescdn.com 3600 IN A 192.168.0.2"

Output of container(s)

Refer to error log entries in the beginning

Any help appreciated.
TIA

@coddec coddec changed the title Lancache Epic and RIOT downloading keep doing start and stop, start and pause etc. Lancache Epic and RIOT (probably some others too, steam is fine) downloading keep doing start and stop, start and pause etc. Jun 9, 2024
@coddec
Copy link
Author

coddec commented Jun 10, 2024

For record.
Disabling nginx buffer makes the 0KB time form 10 seconds to none, however nothing is cached.

/etc/nginx/sites-enabled/20_upstream.conf

 Upstream server to proxy and handle inconsistent 302 redirects
# All cache traffic is passed through this proxy to allow rewriting of redirects without caching

# This is particularly important for sony / ps5 as upstreams redirect between them which confuses slice map on caching

server {

  # Internal bind on 3128, this should not be externally mapped
  listen localhost:3128 reuseport;
proxy_request_buffering off;
proxy_buffering off;

  # No access_log tracking as all requests to this instance are already logged through monolithic

  # access_log /data/logs/upstream-access.log cachelog;
  error_log /data/logs/upstream-error.log;

  include /etc/nginx/sites-available/upstream.conf.d/*.conf;
}
/etc/nginx/sites-enabled/10_generic.conf

# Primary monolithic cache engine

server {
  listen 80 reuseport;
proxy_request_buffering off;
proxy_buffering off;

  access_log /data/logs/access.log cachelog;
  error_log /data/logs/error.log;

  include /etc/nginx/sites-available/cache.conf.d/*.conf;
}

@Redrrx
Copy link

Redrrx commented Jun 11, 2024

I second this issue, same situation right now.

@coddec
Copy link
Author

coddec commented Jun 12, 2024

Tried every fix I can think of. None worked.
Squid forward proxy can easily cache epic downloads, though default config will have cache miss problem if the CDN changes without storid program.
Squid can't cache Steam by default, as steam uses https by default.
Can't point lancache.steamcontent.com to squid as it is in forward proxy mode.
Hope lancache can get it fixed, so that we can still use it for game caching.
Thanks,

@AlexanderSalge
Copy link

also having this issue

@Redrrx
Copy link

Redrrx commented Jun 21, 2024

also having this issue

Tried every fix I can think of. None worked. Squid forward proxy can easily cache epic downloads, though default config will have cache miss problem if the CDN changes without storid program. Squid can't cache Steam by default, as steam uses https by default. Can't point lancache.steamcontent.com to squid as it is in forward proxy mode. Hope lancache can get it fixed, so that we can still use it for game caching. Thanks,

Disabled nginx slicing on my end i had little improvements but its still not reliable as it used to be unfortunately.

@AlexanderSalge
Copy link

AlexanderSalge commented Jun 25, 2024

Do they have any version tags to use other than latest?

Edit1:
Switching to a "latest" release from couple of weeks months ago doesn't fix it either.
Since lancache doesn't do version tags I used the only other sha256 I could find in a hurry from the truecharts release, where they switched to upstream images instead of their own: truecharts/charts@e6a70ee

here the tag i used:
lancachenet/monolithic:latest@sha256:b72d6b909b9e3fb7b521e90aab97479f7977bf6bee97e89a095e1afdbd6d3b85

Edit2:
same with 7077061f078ebf2208a0906153cf13f9a979f47cab19619b28c47b35b3d8eaf5
and with tpill90's version with updated base images. He's got a PR open for those changes (#191)

@VibroAxe
Copy link
Member

VibroAxe commented Jul 8, 2024

Where are you all located who are having this issue. We've been running the latest version of lancache all weekend at stratlan (~150 clients) and haven't seen this issue once. I'm wondering if there's a regional issue on one of epic's upstream.

The errors you are seeing implies that the upstream server is disconnecting from lancache before it's finished downloading and the epic client is doing a 10s walkback before trying again.

@AlexanderSalge
Copy link

Frankfurt, Germany.

@coddec
Copy link
Author

coddec commented Jul 8, 2024

Australia

@VibroAxe
Copy link
Member

VibroAxe commented Jul 8, 2024

I'm wondering if this is CDN's (badly) handling capacity and attempting to throttle lancaches. What event size are you guys running as well

Note for future me> Investigate proxy_next_upstream_tries
Start the rabbit hole here> https://serverfault.com/questions/1151654/nginx-retry-proxy-pass-for-a-period-of-time-on-connection-refused-from-upstream

@coddec
Copy link
Author

coddec commented Jul 8, 2024

Only one client in my LAN.

@AlexanderSalge
Copy link

AlexanderSalge commented Jul 8, 2024

Only two clients on LAN for now, too. Planning a party in two weeks

I was also having this issue with Steam downloads, but I've since forced rerun the prefill script and it filled/redownloaded around 10% of the library (not entire games, only partial) which i found interesting. Since then I haven't encountered any issues with Steam. Haven't had the time to test other publishers. But will try and confirm the continuation of this issue in the comming days.

@Risusama
Copy link

Risusama commented Jul 9, 2024

I've noticed this on Riot launcher for almost a year now. I'm located in Finland.
Didn't report it before since I thought it was my own lancache instance being bad, but apparently not. If I switch my clients DNS to point to 1.1.1.1 or 8.8.8.8, downloads from Riot are back to normal, expected speeds.

@raishack
Copy link

raishack commented Aug 5, 2024

Same problem here with RIOT client since last two months

@starcraft66
Copy link

starcraft66 commented Aug 7, 2024

Have been having the same issue with Riot games for at least 2 years. Atrocious download speed, 0kbps most of the time. I had to stop using lan cache altogether. I never saw an issue about it open so I figured something was wrong on my end. (I'm in Montreal, Canada if that makes a difference)

@raishack
Copy link

raishack commented Aug 7, 2024

Can i stop caching riot games?

@raishack
Copy link

raishack commented Aug 7, 2024

I recommend that you do this
https://lancache.net/docs/containers/dns/
imagen_2024-08-08_004435187

@tpill90
Copy link
Contributor

tpill90 commented Aug 14, 2024

Linking this issue which has some additional discussion : uklans/cache-domains#209

@drobskind
Copy link

I experienced the same issue last weekend "We're sorry. It looks like we're having trouble connecting. We'll continue to retry" on my little LAN Party with 4 Clients.
It was located near Berlin / Germany.
Steam was working fine.
Lancache/Lancache DNS running in Docker on Lenovo Workstation with Xeon and 128GB Ram running Linux Debian.
I have no access to the machine at the moment. so cannot provide any additional details or logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants