Add a forward about the index and caching behavior

rubygems-org-compact-index-api.md

@@ -10,6 +10,47 @@ next: /rubygems-org-rate-limits
 Compact Index API
 -----------------
 
+The Compact Index API is considered a stable public API.
+
+The primary index file is the `versions` file, which provides the name, versions and the MD5 checksum of the per-gem `info` files.
+
+If you need to collect all info about all gems, please use the [public data dumps](https://rubygems.org/pages/data) of the rubygems.org database.
+
+### Fetching and Caching
+
+Example Response (with some headers ignored):
+
+    $ curl -I https://rubygems.org/info/bundler
+
+    HTTP/2 200
+    last-modified: Fri, 22 Mar 2024 13:09:59 GMT
+    etag: "40148273a7c7cd16b49d00a9935fd445"
+    cache-control: max-age=60, public
+    content-type: text/plain; charset=utf-8
+    repr-digest: sha-256="pf7Ts7NTRac//JxO9ke3IYbcWx8kcVn9N0mbm5EJpP0="
+    accept-ranges: bytes
+
+All compact index endpoints support ETags with the `If-None-Match` header.
+
+    If-None-Match: "40148273a7c7cd16b49d00a9935fd445"
+
+The compact index is designed to be fetched using the HTTP `Range` header.
+When a previously fetched copy is present, a ranged request to take advantage of the appended line pattern.
+
+    Range: bytes=#{range_start}-
+
+Responses from the `/versions` and `/info/[GEM]` endpoints will include the `Repr-Digest` header.
+The digests will have at least the SHA256 checksum of the entire file whether the response includes the full file or a partial response.
+*Please note that a `Digest` header is present, but it is deprecated and may be removed without notice.*
+
+When the Range header is satisfied, append the contents at exactly the starting byte, then computer the SHA256 checksum of the resulting file.
+If the result matches the `Repr-Digest` header's SHA256 checksum, the file is considered complete and up-to-date.
+
+Each line in the `/versions` file includes the latest MD5 calculation of the matching `/info` file for that gem at the time the line was written.
+The latest MD5, closest to the end of the file, will match the MD5 of the up-to-date `/info` file.
+
+Exact implementation details are available in the [Bundler CompactIndexClient::Updater](https://github.com/rubygems/rubygems/blob/master/bundler/lib/bundler/compact_index_client/updater.rb).
+
 ### GET - `/versions`
 
 Returns a custom text based format of all versions of all gems.
@@ -81,7 +122,11 @@ Returns a custom text based format for a single gem name with a line for each ve
 
 #### `info` File Format
 
-The format of the `info` file uses one line per version. It is recalculated when a version is yanked., with additional lines appended to the end that may include new or yanked versions of a gem already present earlier in the file.
+The format of the `info` file uses one line per version.
+When a new version is added, additional lines are appended to the end of the file.
+
+When a version is yanked, the `info` file is recalculated.
+The file will never list a version as yanked. It will only be excluded from the file.
 
 The file starts with three minus characters on a new line.