From 8e8bdfee310d14129646af146444e8da421652b0 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Tue, 9 Apr 2024 13:35:46 -0700 Subject: [PATCH] Add a forward about the index and caching behavior --- rubygems-org-compact-index-api.md | 47 ++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/rubygems-org-compact-index-api.md b/rubygems-org-compact-index-api.md index bba9964..f43419f 100644 --- a/rubygems-org-compact-index-api.md +++ b/rubygems-org-compact-index-api.md @@ -10,6 +10,47 @@ next: /rubygems-org-rate-limits Compact Index API ----------------- +The Compact Index API is considered a stable public API. + +The primary index file is the `versions` file, which provides the name, versions and the MD5 checksum of the per-gem `info` files. + +If you need to collect all info about all gems, please use the [public data dumps](https://rubygems.org/pages/data) of the rubygems.org database. + +### Fetching and Caching + +Example Response (with some headers ignored): + + $ curl -I https://rubygems.org/info/bundler + + HTTP/2 200 + last-modified: Fri, 22 Mar 2024 13:09:59 GMT + etag: "40148273a7c7cd16b49d00a9935fd445" + cache-control: max-age=60, public + content-type: text/plain; charset=utf-8 + repr-digest: sha-256="pf7Ts7NTRac//JxO9ke3IYbcWx8kcVn9N0mbm5EJpP0=" + accept-ranges: bytes + +All compact index endpoints support ETags with the `If-None-Match` header. + + If-None-Match: "40148273a7c7cd16b49d00a9935fd445" + +The compact index is designed to be fetched using the HTTP `Range` header. +When a previously fetched copy is present, a ranged request to take advantage of the appended line pattern. + + Range: bytes=#{range_start}- + +Responses from the `/versions` and `/info/[GEM]` endpoints will include the `Repr-Digest` header. +The digests will have at least the SHA256 checksum of the entire file whether the response includes the full file or a partial response. +*Please note that a `Digest` header is present, but it is deprecated and may be removed without notice.* + +When the Range header is satisfied, append the contents at exactly the starting byte, then computer the SHA256 checksum of the resulting file. +If the result matches the `Repr-Digest` header's SHA256 checksum, the file is considered complete and up-to-date. + +Each line in the `/versions` file includes the latest MD5 calculation of the matching `/info` file for that gem at the time the line was written. +The latest MD5, closest to the end of the file, will match the MD5 of the up-to-date `/info` file. + +Exact implementation details are available in the [Bundler CompactIndexClient::Updater](https://github.com/rubygems/rubygems/blob/master/bundler/lib/bundler/compact_index_client/updater.rb). + ### GET - `/versions` Returns a custom text based format of all versions of all gems. @@ -81,7 +122,11 @@ Returns a custom text based format for a single gem name with a line for each ve #### `info` File Format -The format of the `info` file uses one line per version. It is recalculated when a version is yanked., with additional lines appended to the end that may include new or yanked versions of a gem already present earlier in the file. +The format of the `info` file uses one line per version. +When a new version is added, additional lines are appended to the end of the file. + +When a version is yanked, the `info` file is recalculated. +The file will never list a version as yanked. It will only be excluded from the file. The file starts with three minus characters on a new line.