Repo mirrors are painfully slow #8

toreanderson · 2020-05-15T10:57:13Z

The SourceForge mirrors seem to be extremely slow or even totally unreachable. This makes UnitedRPMs slow down essentially all dnf operations.

It's been like this for a very long time. Years. I've held back on submitting an issue since it is not really an UnitedRPMs bug per se, but it makes having the UnitedRPMs repo enabled by default very painful. Perhaps it would be possible to find another mirror system.

For the record, I have a fast Internet connection (500Mbps FTTH), and all other DNF repos (both the upstream Fedora ones as well as multiple other third-party ones). It is only the UnitedRPMs repos that are slow.

A few examples from today:

curl simply getting a connect timeout

[:~] 35 $ curl --verbose --trace-time https://vorboss.dl.sourceforge.net/project/unitedrpms/31/x86_64/repodata/repomd.xml.asc
12:43:55.869747 *   Trying 5.10.152.194:443...
12:43:55.870048 * TCP_NODELAY set
12:46:07.959269 * connect to 5.10.152.194 port 443 failed: Connection timed out
12:46:07.959462 * Failed to connect to vorboss.dl.sourceforge.net port 443: Connection timed out
12:46:07.959635 * Closing connection 0
curl: (7) Failed to connect to vorboss.dl.sourceforge.net port 443: Connection timed out

(I got this URL from a failing dnf command, for what it is worth.)

same command as above retried, but this time it eventually connected, but hit some other timeout

[:~] $ curl --verbose --trace-time https://vorboss.dl.sourceforge.net/project/unitedrpms/31/x86_64/repodata/repomd.xml.asc
12:40:32.915722 *   Trying 5.10.152.194:443...
12:40:32.916334 * TCP_NODELAY set
12:41:04.900047 * Connected to vorboss.dl.sourceforge.net (5.10.152.194) port 443 (#0)
12:41:04.904025 * ALPN, offering h2
12:41:04.904140 * ALPN, offering http/1.1
12:41:04.931467 * successfully set certificate verify locations:
12:41:04.931653 *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
12:41:04.932427 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
12:42:18.626711 * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to vorboss.dl.sourceforge.net:443 
12:42:18.626927 * Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to vorboss.dl.sourceforge.net:443

dnf giving up on downloads because of transfer speed dropping below 1Kb/s

[:~] $ sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=unitedrpms update
[...]
Downloading Packages:
[MIRROR] chromium-freeworld-libs-81.0.4044.138-604.1.x86_64.rpm: Curl error (28): Timeout was reached for https://netcologne.dl.sourceforge.net/project/unitedrpms/31/x86_64/chromium-freeworld-libs-81.0.4044.138-604.1.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
[MIRROR] chromium-pepper-flash-32.0.0.371-2.fc31.x86_64.rpm: Curl error (28): Timeout was reached for https://netcologne.dl.sourceforge.net/project/unitedrpms/31/x86_64/chromium-pepper-flash-32.0.0.371-2.fc31.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
(1/4): chromium-freeworld-81.0.4044.138-604.1.x86_64.rpm      609 kB/s | 121 MB     03:23    
(2/4): chromium-freeworld-libs-81.0.4044.138-604.1.x86_64.rpm  39 kB/s | 8.4 MB     03:43    
(3/4): chromium-pepper-flash-32.0.0.371-2.fc31.x86_64.rpm      35 kB/s | 8.0 MB     03:52    
(4/4): unitedrpms-31-16.fc31.noarch.rpm                       1.6 kB/s |  11 kB     00:06    
----------------------------------------------------------------------------------------------
Total                                                         587 kB/s | 137 MB     03:59     
[...]

dnf update with only unitedrpms repo enabled: `timer: sack setup: 17664 ms`

[:~] $ time sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=unitedrpms update
timer: config: 3 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
Command: dnf -d 10 --refresh --disablerepo=* --enablerepo=unitedrpms update 
Installroot: /
Releasever: 31
cachedir: /var/cache/dnf
Base command: update
Extra commands: ['-d', '10', '--refresh', '--disablerepo=*', '--enablerepo=unitedrpms', 'update']
User-Agent: constructed: 'libdnf (Fedora 31; generic; Linux.x86_64)'
unitedrpms 31 - x86_64                                         51  B/s | 870  B     00:17    
reviving: 'unitedrpms' can be revived - repomd matches.
unitedrpms: using metadata from fr. 15. mai 2020 kl. 11.57 +0200.
timer: sack setup: 17664 ms
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
timer: depsolve: 39 ms
Dependencies resolved.
Nothing to do.
Complete!
Cleaning up.

real    0m18,107s
user    0m1,039s
sys     0m0,199s

compare the above with only the (much larger!) fedora repo enabled: `timer: sack setup: 1900 ms`

[:~] $ time sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=fedora update
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
Command: dnf -d 10 --refresh --disablerepo=* --enablerepo=fedora update 
Installroot: /
Releasever: 31
cachedir: /var/cache/dnf
Base command: update
Extra commands: ['-d', '10', '--refresh', '--disablerepo=*', '--enablerepo=fedora', 'update']
User-Agent: constructed: 'libdnf (Fedora 31; generic; Linux.x86_64)'
Fedora 31 - x86_64                                             46 kB/s |  25 kB     00:00    
reviving: 'fedora' can be revived - metalink checksums match.
fedora: using metadata from to. 24. okt. 2019 kl. 00.52 +0200.
timer: sack setup: 1900 ms
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
timer: depsolve: 523 ms
Dependencies resolved.
Nothing to do.
Complete!
Cleaning up.

real    0m3,119s
user    0m2,277s
sys     0m0,246s

The text was updated successfully, but these errors were encountered:

sergiomb2 · 2020-05-15T16:06:42Z

yeah I also noticed this issue , why other mirror is not the main mirror ?
Thank you

kuboosoft · 2020-05-15T17:43:28Z

The mirror is already reported. I detected various mirrors down recently (I reported it; and some hours they solved); I am not sure the cause; maybe overload in this pandemic in some mirrors.

kuboosoft · 2020-05-15T17:53:04Z

@sergiomb2 Issues in mirrorlist + unnecessary space with debug files... Our second mirror reduced quota... (Ho yes, we have actually big size) 😺

toreanderson · 2020-05-15T19:01:56Z

The UnitedRPM mirrors have been very slow since way before the pandemic struck, for what it is worth. Even with the vorboss one disabled, the UnitedRPMs repo is still is way slower than the much larger fedora repo. To me it seems like the SourceForge mirror network is just very slow overall.

unitedrpms repo enabled only

[:~] $ time sudo dnf -v --refresh --disablerepo='*' --enablerepo=unitedrpms update
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
cachedir: /var/cache/dnf
User-Agent: constructed: 'libdnf (Fedora 32; generic; Linux.x86_64)'
unitedrpms 32 - x86_64                                                                                                                                                                                         104  B/s | 870  B     00:08    
reviving: 'unitedrpms' can be revived - repomd matches.
unitedrpms: using metadata from fr. 15. mai 2020 kl. 10.11 +0200.
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
Dependencies resolved.
Nothing to do.
Complete!

real    0m9,108s
user    0m0,790s
sys     0m0,134s

fedora repo enabled only

[:~] $ time sudo dnf -d 10 --refresh --disablerepo='*' --enablerepo=fedora update
timer: config: 2 ms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
DNF version: 4.2.21
Command: dnf -d 10 --refresh --disablerepo=* --enablerepo=fedora update 
Installroot: /
Releasever: 32
cachedir: /var/cache/dnf
Base command: update
Extra commands: ['-d', '10', '--refresh', '--disablerepo=*', '--enablerepo=fedora', 'update']
User-Agent: constructed: 'libdnf (Fedora 32; generic; Linux.x86_64)'
countme: no event for fedora: window already counted
Fedora 32 - x86_64                                                                                                                                                                                              20 kB/s |  24 kB     00:01    
reviving: 'fedora' can be revived - metalink checksums match.
fedora: using metadata from to. 23. april 2020 kl. 00.22 +0200.
timer: sack setup: 2265 ms
Completion plugin: Generating completion cache...
--> Starting dependency resolution
--> Finished dependency resolution
timer: depsolve: 509 ms
Dependencies resolved.
Nothing to do.
Complete!
Cleaning up.

real    0m3,460s
user    0m2,010s
sys     0m0,245s

sergiomb2 · 2020-05-15T21:18:15Z

(Ho yes, we have actually big size)

What we are talking about ? 100 Gigas ? , so just mirror non-debug rpms on https://osdn.net/projects/unitedrpms/ ... I guess , It is possible , is it possible .

Anyway mirrorlist issues was fixed
Anyway 2, metalink with mirror manager are now in use in Fedora and RPMFusion which is the preferred choice, because works well

Thanks

kuboosoft · 2020-05-15T23:30:51Z

@sergiomb2 aprox... we don't need debug files (checking the traffic of each debug). Next step, debug of the debug?... 😄 Yes is possible avoid debug (some hours needs).

About metalink; in Fedora and other third-party; is irrelevant; the people choice the useful for him/her... and UnitedRPMs has great and big software.

@toreanderson If anybody want speed mirrors, collaborate with it is a great idea (We need mirrors)... the mirrors in SF uses automatic geographic election; and here we only can report it.
What do you expect from a self-sustaining and collaborative infrastructure? 😄

sergiomb2 · 2020-05-16T03:58:50Z

@sergiomb2 aprox... we don't need debug files (checking the traffic of each debug). Next step, debug of the debug?... smile Yes is possible avoid debug (some hours needs).

Debug files are separated because we save a lot of bandwidth , disk and even performance etc and for common use is rare that someone install it and we can leave without it. if we don't want debug app crashes .

we just need it to debug a crash / segfault , to find the symbols and lines of code that make the crash of the computer .

I sometime install it , usually on Beta version under Virtualbox, but after debug it , I remove it again to save disk space and bandwith , yeah I bearably update debug packages after test it I remove it , it always gigas and gigas of downloads .

sergiomb2 · 2020-05-16T04:31:21Z

https://github.com/UnitedRPMs/unitedrpms/blob/master/unitedrpms.repo#L4 where we can use

baseurl=https://osdn.net/projects/unitedrpms/storage/$releasever/$basearch/

hum and why osdn.net is not in first on https://raw.githubusercontent.com/UnitedRPMs/unitedrpms/master/mirrorlist_x86_64.txt ? I use mirrorlist in my dnf configuration btw

and we may remove debug files from osdn.net if we remove it also from here https://github.com/UnitedRPMs/unitedrpms/blob/master/mirrorlist_debug_30.txt . and maybe do the same to the sources rpms ?

What do you think ?

toreanderson · 2020-05-16T07:38:38Z

What do you expect from a self-sustaining and collaborative infrastructure? smile

To be clear, I do not have any demands or hard expectations per se. I am fully aware that I get what I pay for when using UnitedRPMs. So the reason for submitting this issue is just to make you aware of it, in case you weren't already. If the conclusion is that «yeah, SourceForge is slow, but it is nevertheless the best free alternative, so we'll keep using it», then that is fair enough as far as I am concerned.

sergiomb2 · 2020-05-16T22:38:51Z

@toreanderson thank you for these words

kuboosoft · 2020-05-17T04:05:41Z

@toreanderson I am testing some changes; we need secure the changes. Thanks.

@sergiomb2 I need secure the limits and if you orbserve, osdn is incomplete (repodata) and others. But is possible enable again the mirrorlist; I remember is programmed a sync in repositories, cross fingers with some lines changed in the sync task.

kuboosoft · 2020-05-18T19:58:20Z

@toreanderson Done. Can you test now the speed?

toreanderson · 2020-05-19T07:34:18Z

It seems much faster now! The sudo dnf -v --refresh --disablerepo='*' --enablerepo=unitedrpms update commands seems to complete in around 4 seconds now, which is a major speedup compared to how it used to be.

However, it is still (slightly) slower than the much larger fedora repo. While watching the below screen recording (play locally with asciinema play https://asciinema.org/a/irlpk0mlsiA2kvulym5vY81M4), you can see how the progress bar starts out stalled with --- B/s transfer speed). Indeed, it seems like the vast majority of the dnf execution time is spent in that state - once it starts moving, it finishes almost instantaneously.

By comparison, when updating the fedora repo, things start moving right away.

Unfortunately I have not found out how to make dnf spit out sufficient amounts of debugging info to figure out what exactly it is waiting for when it is stalled. Any tips here would be appreciated.

kuboosoft · 2020-05-19T20:02:51Z

@toreanderson Hi, thanks. Is only conjectures about the delay; the repository/repodata is signed (for security). Read about it and about how to works repo_gpgcheck here:

https://blog.centos.org/2018/07/improving-centos-package-delivery-security-with-signed-repository-metadata/

https://linux.die.net/man/5/yum.conf#repo_gpgcheck

"mirrorlist" + "repo_gpgcheck" caused in 2017-2018 our repository down... Reason was urgent necesary go back to "baseurl"; and now I checking the issue was solved and works again. Hopefully not happen other regression.

kuboosoft added the enhancement label May 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repo mirrors are painfully slow #8

Repo mirrors are painfully slow #8

toreanderson commented May 15, 2020

sergiomb2 commented May 15, 2020 •

edited

Loading

kuboosoft commented May 15, 2020

kuboosoft commented May 15, 2020

toreanderson commented May 15, 2020

sergiomb2 commented May 15, 2020 •

edited

Loading

kuboosoft commented May 15, 2020

sergiomb2 commented May 16, 2020 •

edited

Loading

sergiomb2 commented May 16, 2020

toreanderson commented May 16, 2020

sergiomb2 commented May 16, 2020

kuboosoft commented May 17, 2020

kuboosoft commented May 18, 2020

toreanderson commented May 19, 2020 •

edited

Loading

kuboosoft commented May 19, 2020 •

edited

Loading

Repo mirrors are painfully slow #8

Repo mirrors are painfully slow #8

Comments

toreanderson commented May 15, 2020

curl simply getting a connect timeout

same command as above retried, but this time it eventually connected, but hit some other timeout

dnf giving up on downloads because of transfer speed dropping below 1Kb/s

dnf update with only unitedrpms repo enabled: timer: sack setup: 17664 ms

compare the above with only the (much larger!) fedora repo enabled: timer: sack setup: 1900 ms

sergiomb2 commented May 15, 2020 • edited Loading

kuboosoft commented May 15, 2020

kuboosoft commented May 15, 2020

toreanderson commented May 15, 2020

unitedrpms repo enabled only

fedora repo enabled only

sergiomb2 commented May 15, 2020 • edited Loading

kuboosoft commented May 15, 2020

sergiomb2 commented May 16, 2020 • edited Loading

sergiomb2 commented May 16, 2020

toreanderson commented May 16, 2020

sergiomb2 commented May 16, 2020

kuboosoft commented May 17, 2020

kuboosoft commented May 18, 2020

toreanderson commented May 19, 2020 • edited Loading

kuboosoft commented May 19, 2020 • edited Loading

dnf update with only unitedrpms repo enabled: `timer: sack setup: 17664 ms`

compare the above with only the (much larger!) fedora repo enabled: `timer: sack setup: 1900 ms`

sergiomb2 commented May 15, 2020 •

edited

Loading

sergiomb2 commented May 15, 2020 •

edited

Loading

sergiomb2 commented May 16, 2020 •

edited

Loading

toreanderson commented May 19, 2020 •

edited

Loading

kuboosoft commented May 19, 2020 •

edited

Loading