Software Heritage fallback fixes #6036

rjbou · 2024-06-19T17:47:18Z

Fix archive cooking request for wget, the wrong option was given
Fix SWH liveness check, no POST request. It is a hidden bug, it was failing with no error code for curl and as wget had the wrong POST request option it wasn't showing.
Update SWH API request: vault/directory is deprecated in favor to vault/flat
Some doc
fixes opam does not wait for the Software Heritage fallback archives to be "cooked" #5721

dra27

A couple of questions, and I wonder if we may as well strictly follow the protocol for vault/flat, given that the code makes it relatively easy for us to do so.

src/repository/opamDownload.ml

dra27

LGTM - I'm puzzled by the slightly arcane way swh seems to specify vault/flat, but that doesn't need to affect the code here!

rjbou · 2024-11-15T17:00:43Z

Updated!
Some outputs:

wget & not cooked

> ./opam install  -vvv --debug -y swh-wget-not-cooked
[...]

[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-wget-not-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-wget-not-cooked.1: http]
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1102792-e86235/fkdjhfskjfhsjkdhfake.o.r.g.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://fkdjhfskjfhsjkdhfake.o.r.g/"
- --2024-11-15 16:08:51--  https://fkdjhfskjfhsjkdhfake.o.r.g/
- Resolving fkdjhfskjfhsjkdhfake.o.r.g (fkdjhfskjfhsjkdhfake.o.r.g)... failed: Name or service not known.
- wget: unable to resolve host address ‘fkdjhfskjfhsjkdhfake.o.r.g’
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/ping/"
- --2024-11-15 16:08:51--  https://archive.softwareheritage.org/api/1/ping/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:08:51 GMT
-   Content-Type: application/json
-   Content-Length: 6
-   Vary: Accept, origin, Cookie
-   Allow: HEAD, GET, OPTIONS, OPTIONS
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 113
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 47910447
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 6 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100% 8.32M=0s
- 
- 2024-11-15 16:08:51 (8.32 MB/s) - written to stdout [6/6]
- 
- "pong"
00:14.316  CURL(SWHID)            API is working
Source https://fkdjhfskjfhsjkdhfake.o.r.g/ is not available. Do you want to try to retrieve it from Software Heritage cache
(https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:14.317  CURL(SWHID)            SWH fallback for swh-wget-not-cooked.1 with swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331
Processing  1/3:
+ /usr/bin/wget "--content-on-error" "-t" "3" "-O" "/opamroot/log/swh-out-1102792-c8bbc0" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:08:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 404 Not Found
-   Date: Fri, 15 Nov 2024 15:08:51 GMT
-   Content-Type: application/json
-   Content-Length: 121
-   Expires: Fri, 15 Nov 2024 15:08:51 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 59
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 50336527
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Connection: keep-alive
- Saving to: ‘/opamroot/log/swh-out-1102792-c8bbc0’
- 
-      0K                                                       100%  224M=0s
- 
- 2024-11-15 16:08:51 ERROR 404: Not Found.
- 
00:14.528  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is uncooked, request cooking
+ /usr/bin/wget "--method" "POST" "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:08:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:08:52 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:08:52 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 119
-   X-RateLimit-Reset: 1731686931
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 53773176
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  938M=0s
- 
- 2024-11-15 16:08:52 (938 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:14.867  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:02--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:02 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:02 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 58
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 45750401
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  288M=0s
- 
- 2024-11-15 16:09:02 (288 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:25.063  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:12--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:12 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:12 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: GET, POST, OPTIONS
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 57
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 53283606
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  300M=0s
- 
- 2024-11-15 16:09:12 (300 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:35.263  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:22--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:22 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:22 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: GET, POST, OPTIONS
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 56
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 55281750
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  245M=0s
- 
- 2024-11-15 16:09:22 (245 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:45.481  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:32--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:32 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:32 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, POST, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 55
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 52529512
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  272M=0s
- 
- 2024-11-15 16:09:33 (272 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:55.864  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:43--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:43 GMT
-   Content-Type: application/json
-   Content-Length: 241
-   Expires: Fri, 15 Nov 2024 15:09:43 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: GET, OPTIONS, POST
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 54
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 39989269
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 241 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100% 1.18G=0s
- 
- 2024-11-15 16:09:43 (1.18 GB/s) - written to stdout [241/241]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"done","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
01:06.211  CURL(SWHID)            Downloading swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 for swh-wget-not-cooked.1
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/"
- --2024-11-15 16:09:43--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 302 Found
-   Date: Fri, 15 Nov 2024 15:09:43 GMT
-   Content-Type: text/html; charset=utf-8
-   Content-Length: 0
-   Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/f2391ef00cb8cab822ba66e4f9bdb0f782b84bfd?se=2024-11-16T15%3A09%3A43Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz%22&sig=3N2KiPjDIRFVfpGo6PLwr8syJab5Ye7Q8jKJW66G0oM%3D
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, GET, OPTIONS, HEAD
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 112
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 52431141
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Connection: keep-alive
- Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/f2391ef00cb8cab822ba66e4f9bdb0f782b84bfd?se=2024-11-16T15%3A09%3A43Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz%22&sig=3N2KiPjDIRFVfpGo6PLwr8syJab5Ye7Q8jKJW66G0oM%3D [following]
- --2024-11-15 16:09:43--  https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/f2391ef00cb8cab822ba66e4f9bdb0f782b84bfd?se=2024-11-16T15%3A09%3A43Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz%22&sig=3N2KiPjDIRFVfpGo6PLwr8syJab5Ye7Q8jKJW66G0oM%3D
- Resolving swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)... 20.209.11.33
- Connecting to swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)|20.209.11.33|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Content-Length: 15559
-   Content-Type: application/octet-stream
-   Content-MD5: 7HYlM4kpCKEi2SPr0xMKyA==
-   Last-Modified: Fri, 15 Nov 2024 15:09:38 GMT
-   Accept-Ranges: bytes
-   ETag: "0x8DD058785ED331E"
-   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
-   x-ms-request-id: d6c83cff-a01e-0017-3b70-3757dc000000
-   x-ms-version: 2024-11-04
-   x-ms-creation-time: Fri, 15 Nov 2024 15:09:38 GMT
-   x-ms-lease-status: unlocked
-   x-ms-lease-state: available
-   x-ms-blob-type: BlockBlob
-   Content-Disposition: attachment; filename="swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz"
-   x-ms-server-encrypted: true
-   Access-Control-Allow-Origin: *
-   Date: Fri, 15 Nov 2024 15:09:43 GMT
- Length: 15559 (15K) [application/octet-stream]
- Saving to: ‘/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331.part’
- 
-      0K .......... .....                                      100% 1.31G=0s
- 
- 2024-11-15 16:09:44 (1.31 GB/s) - ‘/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331.part’ saved [15559/15559]
- 
+ /usr/bin/tar "xfz" "/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331" "-C" "/tmp/opam-1102792-b35983"
⬇ retrieved swh-wget-not-cooked.1  (SWH fallback)
[...]

cURL & not cooked

> ./opam install  -vvv --debug -y swh-curl-not-cooked
[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-curl-not-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-curl-not-cooked.1: http]
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-360556-43cf30/i-am-not-available-fake.o.r.g.part" "--" "https://i-am-not-available-fake.o.r.g/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 2 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 1 retries left.
- 000
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/ping/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100     6  100     6    0     0     49      0 --:--:-- --:--:-- --:--:--    50
- "pong"200
00:17.873  CURL(SWHID)            API is working
Source https://i-am-not-available-fake.o.r.g/ is not available. Do you want to try to retrieve it from
Software Heritage cache (https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:17.873  CURL(SWHID)            SWH fallback for swh-curl-not-cooked.1 with swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b
Processing  1/3:
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/opamroot/log/swh-out-360556-b9353e" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
- 404
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   121  100   121    0     0    741      0 --:--:-- --:--:-- --:--:--   746
00:18.065  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is uncooked, request cooking
+ /usr/bin/curl "--request" "POST" "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   240  100   240    0     0    863      0 --:--:-- --:--:-- --:--:--   866
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":null,"id":416941408,"status":"new","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:18.371  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is cooking...
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   255  100   255    0     0   1735      0 --:--:-- --:--:-- --:--:--  1734
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":"Processing...","id":416941408,"status":"pending","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:28.550  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is cooking...
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   255  100   255    0     0   1850      0 --:--:-- --:--:-- --:--:--  1861
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":"Processing...","id":416941408,"status":"pending","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:38.718  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is cooking...
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   241  100   241    0     0   1726      0 --:--:-- --:--:-- --:--:--  1733
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":null,"id":416941408,"status":"done","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:48.886  CURL(SWHID)            Downloading swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b for swh-curl-not-cooked.1
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-360556-ffbc05/7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b.part" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/"
- 200
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 12495  100 12495    0     0  29676      0 --:--:-- --:--:-- --:--:-- 29676
+ /usr/bin/tar "xfz" "/tmp/opam-360556-ffbc05/7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b" "-C" "/tmp/opam-360556-0a2c9e"
⬇ retrieved swh-curl-not-cooked.1  (SWH fallback)
[...]

wget & cooked

> ./opam install  -vvv --debug -y swh-wget-cooked
[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-wget-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-wget-cooked.1: http]
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1100962-6df63b/fkdjhfskjfhsjkdhfake.o.r.g.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://fkdjhfskjfhsjkdhfake.o.r.g/"
- --2024-11-15 16:05:51--  https://fkdjhfskjfhsjkdhfake.o.r.g/
- Resolving fkdjhfskjfhsjkdhfake.o.r.g (fkdjhfskjfhsjkdhfake.o.r.g)... failed: Name or service not known.
- wget: unable to resolve host address ‘fkdjhfskjfhsjkdhfake.o.r.g’
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/ping/"
- --2024-11-15 16:05:51--  https://archive.softwareheritage.org/api/1/ping/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:05:51 GMT
-   Content-Type: application/json
-   Content-Length: 6
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, GET, OPTIONS, HEAD
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 115
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 54232414
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 6 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100% 22.9M=0s
- 
- 2024-11-15 16:05:51 (22.9 MB/s) - written to stdout [6/6]
- 
- "pong"
00:13.858  CURL(SWHID)            API is working
Source https://fkdjhfskjfhsjkdhfake.o.r.g/ is not available. Do you want to try to retrieve it from
Software Heritage cache (https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:13.858  CURL(SWHID)            SWH fallback for swh-wget-cooked.1 with swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4
Processing  1/3:
+ /usr/bin/wget "--content-on-error" "-t" "3" "-O" "/opamroot/log/swh-out-1100962-7f8485" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/"
- --2024-11-15 16:05:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:05:51 GMT
-   Content-Type: application/json
-   Content-Length: 241
-   Expires: Fri, 15 Nov 2024 15:05:51 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 59
-   X-RateLimit-Reset: 1731683211
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 47910295
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 241 [application/json]
- Saving to: ‘/opamroot/log/swh-out-1100962-7f8485’
- 
-      0K                                                       100%  143M=0s
- 
- 2024-11-15 16:05:51 (143 MB/s) - ‘/opamroot/log/swh-out-1100962-7f8485’ saved [241/241]
- 
00:14.101  CURL(SWHID)            swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 is cooked, requesting url
00:14.101  CURL(SWHID)            Downloading swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 for swh-wget-cooked.1
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/raw/"
- --2024-11-15 16:05:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/raw/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 302 Found
-   Date: Fri, 15 Nov 2024 15:05:52 GMT
-   Content-Type: text/html; charset=utf-8
-   Content-Length: 0
-   Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/94f715535502200f7a4ede59a85d22dda9fc3510?se=2024-11-16T15%3A05%3A52Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz%22&sig=9DEL7O0yEMhxz3Wgvo3QgJ6180mkDIjuV4rKoBCTRTc%3D
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, OPTIONS, GET, HEAD
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 114
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 52136120
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Connection: keep-alive
- Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/94f715535502200f7a4ede59a85d22dda9fc3510?se=2024-11-16T15%3A05%3A52Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz%22&sig=9DEL7O0yEMhxz3Wgvo3QgJ6180mkDIjuV4rKoBCTRTc%3D [following]
- --2024-11-15 16:05:52--  https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/94f715535502200f7a4ede59a85d22dda9fc3510?se=2024-11-16T15%3A05%3A52Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz%22&sig=9DEL7O0yEMhxz3Wgvo3QgJ6180mkDIjuV4rKoBCTRTc%3D
- Resolving swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)... 20.209.11.33
- Connecting to swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)|20.209.11.33|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Content-Length: 4213
-   Content-Type: application/octet-stream
-   Content-MD5: +jpEuytdGqLxCrKdtXLUMg==
-   Last-Modified: Wed, 19 Jun 2024 21:00:34 GMT
-   Accept-Ranges: bytes
-   ETag: "0x8DC90A2DD052E92"
-   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
-   x-ms-request-id: 041ed48b-601e-0027-456f-37e913000000
-   x-ms-version: 2024-11-04
-   x-ms-creation-time: Wed, 19 Jun 2024 21:00:34 GMT
-   x-ms-lease-status: unlocked
-   x-ms-lease-state: available
-   x-ms-blob-type: BlockBlob
-   Content-Disposition: attachment; filename="swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz"
-   x-ms-server-encrypted: true
-   Access-Control-Allow-Origin: *
-   Date: Fri, 15 Nov 2024 15:05:52 GMT
- Length: 4213 (4.1K) [application/octet-stream]
- Saving to: ‘/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4.part’
- 
-      0K ....                                                  100%  268M=0s
- 
- 2024-11-15 16:05:52 (268 MB/s) - ‘/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4.part’ saved [4213/4213]
- 
+ /usr/bin/tar "xfz" "/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4" "-C" "/tmp/opam-1100962-c8b6c4"
⬇ retrieved swh-wget-cooked.1  (SWH fallback)
[...]

cURL & cooked

> ./opam install  -vvv --debug -y swh-curl-cooked
[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-curl-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-curl-cooked.1: http]
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-1090321-caeea4/i-am-not-available-fake.o.r.g.part" "--" "https://i-am-not-available-fake.o.r.g/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 2 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 1 retries left.
- 000
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/ping/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100     6  100     6    0     0     10      0 --:--:-- --:--:-- --:--:--    10
- "pong"200
00:21.231  CURL(SWHID)            API is working
Source https://i-am-not-available-fake.o.r.g/ is not available. Do you want to try to retrieve it from
Software Heritage cache (https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:21.231  CURL(SWHID)            SWH fallback for swh-curl-cooked.1 with swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4
Processing  1/3:
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/opamroot/log/swh-out-1090321-b8392a" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/"
- 200
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   241  100   241    0     0   1173      0 --:--:-- --:--:-- --:--:--  1169
00:21.955  CURL(SWHID)            swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 is cooked, requesting url
00:21.956  CURL(SWHID)            Downloading swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 for swh-curl-cooked.1
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-1090321-fb6640/6b700f4b287aee509adbc723d0303098188604f4.part" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/raw/"
- 200
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  4213  100  4213    0     0   7233      0 --:--:-- --:--:-- --:--:--  7233
+ /usr/bin/tar "xfz" "/tmp/opam-1090321-fb6640/6b700f4b287aee509adbc723d0303098188604f4" "-C" "/tmp/opam-1090321-5ea6ea"
⬇ retrieved swh-curl-cooked.1  (SWH fallback)
[...]

Some logging enhancement were done after the output: counting of tries, and change swhid is cooked to swhid is cooked or cooking as it may be currently cooking.

kit-ty-kate

is the download: parse http error code for wget too and permit to retrieve it. commit absolutely necessary for this fix or can it be moved to a separate PR and be discussed separately? If this is possible i would rather review this change separately (with repository: add re as a dependency of opam-repository as both are related) so that this fix gets merged quicker (e.g. i personally don't think it's a good idea to parse --server-response)

kit-ty-kate · 2024-11-16T00:28:58Z

src/repository/opamDownload.ml

@@ -261,7 +261,7 @@ let get_output ~post ?(args=[]) url =
  let cmd_args =
    if post then
      match cmd_args with
-      | ("curl" as cmd)::args -> Some (cmd :: ["-X"; "POST"] @ args)
+      | ("curl" as cmd)::args -> Some (cmd :: ["--request"; "POST"] @ args)


what is the reasoning behind this change?

It's a personal view: i think it's better to use long form of options, it's easier to understand.

src/repository/opamDownload.ml

kit-ty-kate

apart from the minor suggestion, lgtm. I'll review it more properly tomorrow but it looks very good. I especially like the change to a match in tool_return, it looks better i think.

src/repository/opamDownload.ml

kit-ty-kate

full review is still positive, i have a bit more suggestions though

master_changes.md

src/repository/opamDownload.ml

…grained curl handling

…emporary files: 'with_tmp_file' and 'with_tmp_file_job'

* Update to new API * More fine-grained workflow: check is archive is already cooked: if not request cooking, otherwise retrieve url of the archive * Use only post request when needed

src/core/opamFilename.ml

rjbou · 2024-11-28T14:50:36Z

src/repository/opamDownload.ml

+    | Some out -> out
+    | None ->
+      (* Shouldn't happen, we already checked that a post tool is used *)
+      (* XXX change to an assert false ? *)


I don't think this display is needed, as in this module the main entry archive_fallback checks that.

in that case i agree this could be replaced by assert false

src/repository/opamDownload.ml

…arser http error is NaN

.gitignore

Co-authored-by: R. Boujbel <[email protected]>

Co-authored-by: Kate <[email protected]>

tests/reftests/swhid.unix.test

rjbou · 2024-11-28T18:59:47Z

master_changes.md

@@ -161,6 +165,9 @@ users)
  * `OpamArg.InvalidCLI`: export exception [#6150 @rjbou]

 ## opam-repository
+  * `OpamDownload.get_output`: fix `wget` option for `POST` requests [#6036 @rjbou]
+  * `OpamDownload.get_output`: use long form for `curl` `POST` request option [#6036 @rjbou]
+  * `OpamDownload.download`: more fine grained HTTP request error code detection for curl [#6036 @rjbou]


Suggested change

* `OpamDownload.download`: more fine grained HTTP request error code detection for curl [#6036 @rjbou]

* `OpamDownload.download`: more fine grained HTTP request error code detection for curl [#6036 @rjbou]

* `OpamDownload`: remove internal `check_post_tool` and `post_tools` as it was only used to check tools for Software availaibility fallback. With wget support removal, it is no more needed. [#6036]

those functions were not exported so i don't think this should be part of the changelog

rjbou added this to the 2.2.0~rc1 milestone Jun 19, 2024

rjbou requested a review from kit-ty-kate June 19, 2024 17:47

rjbou linked an issue Jun 19, 2024 that may be closed by this pull request

opam does not wait for the Software Heritage fallback archives to be "cooked" #5721

Open

rjbou added the KIND: BUG label Jun 19, 2024

dra27 reviewed Jun 19, 2024

View reviewed changes

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

rjbou force-pushed the swhid-test branch from 1442734 to 1e2f479 Compare June 19, 2024 21:12

dra27 approved these changes Jun 20, 2024

View reviewed changes

rjbou added the PR: NEEDS UPDATE label Jun 20, 2024

rjbou removed this from the 2.2.0~rc1 milestone Jun 21, 2024

rjbou added this to the 2.3 milestone Jun 27, 2024

kit-ty-kate approved these changes Jul 9, 2024

View reviewed changes

rjbou marked this pull request as draft July 10, 2024 14:15

kit-ty-kate modified the milestones: 2.3.0~alpha1, 2.4.0~alpha1 Sep 18, 2024

rjbou force-pushed the swhid-test branch from 1e2f479 to 40fe666 Compare November 15, 2024 16:58

rjbou added PR: WAITING FOR REVIEW and removed PR: NEEDS UPDATE labels Nov 15, 2024

rjbou force-pushed the swhid-test branch from 40fe666 to 34a5ec0 Compare November 15, 2024 17:06

rjbou requested review from dra27 and kit-ty-kate November 15, 2024 17:07

rjbou marked this pull request as ready for review November 15, 2024 17:07

kit-ty-kate force-pushed the swhid-test branch from 34a5ec0 to 9c7635b Compare November 16, 2024 00:27

kit-ty-kate reviewed Nov 16, 2024

View reviewed changes

rjbou added PR: NEEDS UPDATE and removed PR: WAITING FOR REVIEW labels Nov 18, 2024

kit-ty-kate reviewed Nov 27, 2024

View reviewed changes

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

kit-ty-kate reviewed Nov 27, 2024

View reviewed changes

rjbou added 16 commits November 28, 2024 15:36

download: fix POST option for wget

c2657c9

download: use long format option for curl POST request

5eb4e78

download: remove wget support for SWH fallback to focus on more fine-…

fb1cc77

…grained curl handling

download: fix SWH liveness check

888381f

download: fix SWH cooking request and status retrieval API request

02b3022

download: add internal function to retrieve http code for curl.

abcae23

core: factorise temporary file and directory names creation

94985ec

core: add 'OpamSystem' and 'OpamFilename' function to use and clean t…

971b9e1

…emporary files: 'with_tmp_file' and 'with_tmp_file_job'

download: rework SWH retrieval fallback

d7f5f1e

* Update to new API * More fine-grained workflow: check is archive is already cooked: if not request cooking, otherwise retrieve url of the archive * Use only post request when needed

!fixup rework SWH: use with_tmpfile function

2e59790

!fixup rework SWH : remove wget

9a1ed1a

!fixup rework SWH use string function instead of list parsing (optim)

d1e4b08

swhid: add some debug logging

cd06588

gitignore: add release binary output directory

cf801e4

!fixup rework SWH remove open opam process

1195490

!fixup rework SWH: remove wget in message

433c92b

rjbou force-pushed the swhid-test branch from 349dba9 to 433c92b Compare November 28, 2024 14:48

rjbou commented Nov 28, 2024

View reviewed changes

src/core/opamFilename.ml Show resolved Hide resolved

rjbou commented Nov 28, 2024

View reviewed changes

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

!fixup retrieve http code for curl : don't output code 999 when the p…

26bdbea

…arser http error is NaN

kit-ty-kate reviewed Nov 28, 2024

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

kit-ty-kate and others added 3 commits November 28, 2024 17:20

fixup! rm question

04c68b0

Co-authored-by: R. Boujbel <[email protected]>

!fixup remove wget

5f6b75e

!fixup gitignore

8eb3332

Co-authored-by: Kate <[email protected]>

rjbou commented Nov 28, 2024

View reviewed changes

tests/reftests/swhid.unix.test Outdated Show resolved Hide resolved

!fixup remove wget

a61576f

rjbou commented Nov 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Software Heritage fallback fixes #6036

Software Heritage fallback fixes #6036

rjbou commented Jun 19, 2024 •

edited

Loading

dra27 left a comment

dra27 left a comment

rjbou commented Nov 15, 2024

kit-ty-kate left a comment

kit-ty-kate Nov 16, 2024

rjbou Nov 18, 2024

kit-ty-kate left a comment

kit-ty-kate left a comment

rjbou Nov 28, 2024

kit-ty-kate Nov 28, 2024

rjbou Nov 28, 2024

kit-ty-kate Nov 28, 2024

	* `OpamDownload.download`: more fine grained HTTP request error code detection for curl [#6036 @rjbou]
	* `OpamDownload.download`: more fine grained HTTP request error code detection for curl [#6036 @rjbou]
	* `OpamDownload`: remove internal `check_post_tool` and `post_tools` as it was only used to check tools for Software availaibility fallback. With wget support removal, it is no more needed. [#6036]

Software Heritage fallback fixes #6036

Are you sure you want to change the base?

Software Heritage fallback fixes #6036

Conversation

rjbou commented Jun 19, 2024 • edited Loading

dra27 left a comment

Choose a reason for hiding this comment

dra27 left a comment

Choose a reason for hiding this comment

rjbou commented Nov 15, 2024

kit-ty-kate left a comment

Choose a reason for hiding this comment

kit-ty-kate Nov 16, 2024

Choose a reason for hiding this comment

rjbou Nov 18, 2024

Choose a reason for hiding this comment

kit-ty-kate left a comment

Choose a reason for hiding this comment

kit-ty-kate left a comment

Choose a reason for hiding this comment

rjbou Nov 28, 2024

Choose a reason for hiding this comment

kit-ty-kate Nov 28, 2024

Choose a reason for hiding this comment

rjbou Nov 28, 2024

Choose a reason for hiding this comment

kit-ty-kate Nov 28, 2024

Choose a reason for hiding this comment

rjbou commented Jun 19, 2024 •

edited

Loading