CI: On `i686-linux-gnu`, the `Downloads` test set is failing frequently (but not always) with `nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100` #46159

DilumAluthge · 2022-07-25T01:14:37Z

The error seems to be: nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100

Relevant log snippet:

Error in testset Downloads:
Error During Test at /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/runtests.jl:355
  Got exception outside of a @test
  TaskFailedException
  Stacktrace:
    [1] macro expansion
      @ ./experimental.jl:102 [inlined]
    [2] macro expansion
      @ ./timing.jl:394 [inlined]
    [3] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/runtests.jl:366 [inlined]
    [4] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Test/src/Test.jl:1388 [inlined]
    [5] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/runtests.jl:356 [inlined]
    [6] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Test/src/Test.jl:1388 [inlined]
    [7] top-level scope
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/runtests.jl:4
    [8] include
      @ ./Base.jl:424 [inlined]
    [9] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/test/testdefs.jl:24 [inlined]
   [10] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Test/src/Test.jl:1388 [inlined]
   [11] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/test/testdefs.jl:23 [inlined]
   [12] macro expansion
      @ ./timing.jl:475 [inlined]
   [13] runtests(name::String, path::String, isolate::Bool; seed::UInt128)
      @ Main /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/test/testdefs.jl:21
   [14] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, UInt128, Tuple{Symbol}, NamedTuple{(:seed,), Tuple{UInt128}}})
      @ Base ./essentials.jl:810
   [15] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285
   [16] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
   [17] macro expansion
      @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:285 [inlined]
   [18] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:501
      nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100
      Stacktrace:
        [1] (::Downloads.var"#9#18"{IOBuffer, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, Bool, String, Bool, Bool})(easy::Downloads.Curl.Easy)
          @ Downloads /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/src/Downloads.jl:388
        [2] with_handle(f::Downloads.var"#9#18"{IOBuffer, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, Bool, String, Bool, Bool}, handle::Downloads.Curl.Easy)
          @ Downloads.Curl /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/src/Curl/Curl.jl:88
        [3] #8
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/src/Downloads.jl:329 [inlined]
        [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, Bool, String, Bool, Bool}, arg::IOBuffer)
          @ ArgTools /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/ArgTools/src/ArgTools.jl:134
        [5] #7
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/src/Downloads.jl:328 [inlined]
        [6] arg_read
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/ArgTools/src/ArgTools.jl:76 [inlined]
        [7] request(url::String; input::Nothing, output::IOBuffer, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Nothing, verbose::Bool, debug::Nothing, throw::Bool, downloader::Downloads.Downloader)
          @ Downloads /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/src/Downloads.jl:327
        [8] #3
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/src/Downloads.jl:233 [inlined]
        [9] arg_write(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, Downloads.Downloader, String}, arg::IOBuffer)
          @ ArgTools /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/ArgTools/src/ArgTools.jl:134
       [10] #download#2
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/src/Downloads.jl:232 [inlined]
       [11] (::Main.Test42Main_Downloads.var"#2#3"{Base.Pairs{Symbol, Downloads.Downloader, Tuple{Symbol}, NamedTuple{(:downloader,), Tuple{Downloads.Downloader}}}, String})(output::IOBuffer)
          @ Main.Test42Main_Downloads /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/setup.jl:13
       [12] sprint
          @ ./strings/io.jl:107 [inlined]
       [13] #download_body#1
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/setup.jl:12 [inlined]
       [14] #download_json#4
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/setup.jl:18 [inlined]
       [15] macro expansion
          @ /cache/build/default-amdci5-0/julialang/julia-master/julia-9bdaabdc82/share/julia/stdlib/v1.9/Downloads/test/runtests.jl:368 [inlined]
       [16] (::Main.Test42Main_Downloads.var"#24#50"{Downloads.Downloader, String, Int32})()
          @ Main.Test42Main_Downloads ./task.jl:501

Example log: https://buildkite.com/julialang/julia-master/builds/14198#01822720-ae76-4ff4-8a35-cec678dfa6a7

The text was updated successfully, but these errors were encountered:

vtjnash · 2022-07-25T01:23:26Z

JuliaCI/julia-buildkite#185 may fix this, so would be good to check

giordano · 2022-07-25T19:59:20Z

This Win64 job failed with

      nested task error: RequestError: Failed to connect to httpbingo.julialang.org port 443 after 21058 ms: Timed out while requesting https://httpbingo.julialang.org/delay/2?id=40

which looks similar enough?

DilumAluthge · 2022-07-27T05:19:47Z

Sadly, I am still seeing the original failure (nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=98) on i686-linux-gnu even after I merged JuliaCI/julia-buildkite#185.

Example log: https://buildkite.com/julialang/julia-master/builds/14331#01823cdc-7767-4c4b-bcc8-976db68868de

In the above log, it says that JULIA_NUM_THREADS was set to 1, and that the value of nthreads() was 1, so it does look like threads were disabled in that job.

DilumAluthge · 2022-07-27T07:57:40Z

This Win64 job failed with

      nested task error: RequestError: Failed to connect to httpbingo.julialang.org port 443 after 21058 ms: Timed out while requesting https://httpbingo.julialang.org/delay/2?id=40

which looks similar enough?

Hmmm. I don't think that's the same failure.

DilumAluthge · 2022-07-27T08:04:56Z

I did see the exact same failure as the original post (nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=1) just now on Buildbot on i686-w64-mingw32.

Log: https://build.julialang.org/#/builders/42/builds/7253

DilumAluthge · 2022-07-27T08:11:44Z

I should point out that even though this failure didn't go away once I merged JuliaCI/julia-buildkite#185, it certainly did become less frequent.

DilumAluthge · 2022-08-05T05:12:24Z

I continue to see this on i686-linux-gnu.

@vtjnash @StefanKarpinski Could this be a libuv issue? We are using libuv's getaddrinfo, right?

vtjnash · 2022-09-28T16:01:53Z

Looking closer, the test fails at 20-30 seconds, but the expected maximum time is 180 seconds: JuliaLang/Downloads.jl#204

Thus, we are simply running too many downloads concurrently, and mbedtls is too slow to keep up.

StefanKarpinski · 2022-09-28T16:52:04Z

Interestingly, same error as I saw here: JuliaLang/Downloads.jl#196 (comment). Seems like there's some race condition in libcurl that we're hitting in some cases. In the link, putting a try/catch in a cfunction someone causes it to be triggered more easily.

vtjnash · 2022-09-28T17:31:10Z

Ah, sorry, I was responding to #46159 (comment), which is a different issue (that happens to be in roughly the same place, for related but not identical reasons)

StefanKarpinski · 2022-10-01T21:39:17Z

This semaphore may help: JuliaLang/Downloads.jl#205.

vtjnash · 2022-10-03T00:35:24Z

You can test this on your own machine (both issues here should turn out to have the same fix) by changing the 100 concurrent attempts to 1500 (or even higher) in this test

DilumAluthge added the ci Continuous integration label Jul 25, 2022

DilumAluthge mentioned this issue Jul 25, 2022

Revert broadening of arithmetic Any methods (revert #44564 and #45320) #45489

Merged

DilumAluthge mentioned this issue Aug 14, 2022

Split the i686-linux-gnu tests up into two groups: "all tests except Downloads and Pkg" and "only the Downloads tests" JuliaCI/julia-buildkite#192

Merged

vtjnash mentioned this issue Oct 12, 2022

Too short timeout in "concurrent requests" stress test JuliaLang/Downloads.jl#204

Open

vtjnash closed this as completed Oct 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: On `i686-linux-gnu`, the `Downloads` test set is failing frequently (but not always) with `nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100` #46159

CI: On `i686-linux-gnu`, the `Downloads` test set is failing frequently (but not always) with `nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100` #46159

DilumAluthge commented Jul 25, 2022

vtjnash commented Jul 25, 2022

giordano commented Jul 25, 2022

DilumAluthge commented Jul 27, 2022 •

edited

Loading

DilumAluthge commented Jul 27, 2022 •

edited

Loading

DilumAluthge commented Jul 27, 2022 •

edited

Loading

DilumAluthge commented Jul 27, 2022

DilumAluthge commented Aug 5, 2022 •

edited

Loading

vtjnash commented Sep 28, 2022 •

edited

Loading

StefanKarpinski commented Sep 28, 2022

vtjnash commented Sep 28, 2022 •

edited

Loading

StefanKarpinski commented Oct 1, 2022

vtjnash commented Oct 3, 2022

CI: On i686-linux-gnu, the Downloads test set is failing frequently (but not always) with nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100 #46159

CI: On i686-linux-gnu, the Downloads test set is failing frequently (but not always) with nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100 #46159

Comments

DilumAluthge commented Jul 25, 2022

vtjnash commented Jul 25, 2022

giordano commented Jul 25, 2022

DilumAluthge commented Jul 27, 2022 • edited Loading

DilumAluthge commented Jul 27, 2022 • edited Loading

DilumAluthge commented Jul 27, 2022 • edited Loading

DilumAluthge commented Jul 27, 2022

DilumAluthge commented Aug 5, 2022 • edited Loading

vtjnash commented Sep 28, 2022 • edited Loading

StefanKarpinski commented Sep 28, 2022

vtjnash commented Sep 28, 2022 • edited Loading

StefanKarpinski commented Oct 1, 2022

vtjnash commented Oct 3, 2022

CI: On `i686-linux-gnu`, the `Downloads` test set is failing frequently (but not always) with `nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100` #46159

CI: On `i686-linux-gnu`, the `Downloads` test set is failing frequently (but not always) with `nested task error: RequestError: getaddrinfo() thread failed to start while requesting https://httpbingo.julialang.org/delay/2?id=100` #46159

DilumAluthge commented Jul 27, 2022 •

edited

Loading

DilumAluthge commented Jul 27, 2022 •

edited

Loading

DilumAluthge commented Jul 27, 2022 •

edited

Loading

DilumAluthge commented Aug 5, 2022 •

edited

Loading

vtjnash commented Sep 28, 2022 •

edited

Loading

vtjnash commented Sep 28, 2022 •

edited

Loading