Issue with Cloning Google Cloud Ops Agent Repository via apt-mirror

[godunko-v]

Describe the bug
I am trying to set up a mirror of the Google Cloud Ops Agent repository (deb https://packages.cloud.google.com/apt google-cloud-ops-agent-noble-all main) for future installation of the Google Cloud Ops Agent on my instances. However, I encountered an issue when using apt-mirror.

Steps to reproduce the behavior:

1. Configure apt-mirror to clone the repository:
   deb https://packages.cloud.google.com/apt google-cloud-ops-agent-noble-all main
2. Run apt-mirror to download the repository.
3. Encounter the error message:

apt-mirror: can't open index packages.cloud.google.com/apt//dists/google-cloud-ops-agent-noble-all/main/binary-amd64/Packages in process_index at /usr/bin/apt-mirror line 891, chunk 48

Observations:
I believe the issue is related to the Packages file type. When I run curl -I https://packages.cloud.google.com/apt/dists/google-cloud-ops-agent-noble-all/main/binary-amd64/Packages, the response returns a Content-Type: text/html instead of the expected application/octet-stream or other appropriate binary content type.

Expected behavior

• The Packages file should be served with the correct Content-Type (e.g., application/octet-stream for binary files).
• apt-mirror should be able to download and process the Packages file without errors.
• The repository should mirror successfully, allowing me to use the mirrored repository for installation.

Environment (please complete the following information):

• VM distro / OS: ubuntu 24.04 LTS

Additional context

• The URL used for the repository is correct and accessible, but the Packages file is not being served in the expected format for mirror synchronization.

I would appreciate it if you could investigate this issue and help resolve it.

Thank you!

[jefferbrecht]

Hi @godunko-v, thanks for reporting this! We've opened an internal bug to get this resolved and I'll update you once a fix is rolled out. Apologies for the inconvenience.

[annakuo]

Hi @godunko-v , the fix has been rolled out. Can you confirm if the issue has been resolved?

[godunko-v]

Hello, @annakuo

Thank you for your prompt response and for updating the file type for the Packages file to application/octet-stream. However, it appears that the issue is still not resolved.

I tested with the following apt-mirror configuration:

## Ubuntu security sources
deb http://security.ubuntu.com/ubuntu noble-security main
deb http://security.ubuntu.com/ubuntu noble-security universe
deb http://security.ubuntu.com/ubuntu noble-security restricted
deb http://security.ubuntu.com/ubuntu noble-security multiverse

## Google Ops Agent
deb https://packages.cloud.google.com/apt google-cloud-ops-agent-noble-all main

## Wazuh sources
deb https://packages.wazuh.com/4.x/apt/ stable main

## PHP ppa
deb https://ppa.launchpadcontent.net/ondrej/php/ubuntu/ noble main
clean https://ppa.launchpadcontent.net/ondrej/php/ubuntu/

Everything is working fine except for google-cloud-ops-agent-noble-all. When running apt-mirror, I still encounter the following error:

Processing indexes: [PPPPPPPPPPPPPPPPapt-mirror: can't open index packages.cloud.google.com/apt//dists/google-cloud-ops-agent-noble-all/main/binary-amd64/Packages in process_index at /usr/bin/apt-mirror line 891.
PPPPPPP]

While the Packages file type is now corrected, it seems this was not the main issue. apt-mirror is still unable to download the packages from the repository.

Could you please investigate further? Let me know if additional details are required.

Thank you!

[jefferbrecht]

Hey @godunko-v, thanks for the troubleshooting & sorry this is still happening! We dug a bit deeper and think we've found the issue. TL;DR apt-mirror does not support repos that only serve an uncompressed Packages index (we found an upstream issue here for it).

I can't give a timeline for when we might be able to start serving compressed indices, which would resolve this at the source assuming this does not get fixed upstream, but I can offer a few workarounds in the meantime:

• A commenter in the linked issue mentions that apt-mirror2 can handle an uncompressed Packages index
• You could patch your apt-mirror locally so that it supports uncompressed indices:

  cat << 'EOF' > ./apt-mirror.patch
  --- a
  +++ b
  @@ -445,6 +445,7 @@
                   add_url_to_download( $url . $_ . "/Contents-" . $arch . ".xz" );
               }
               add_url_to_download( $url . $_ . "/binary-" . $arch . "/Release" );
  +            add_url_to_download( $url . $_ . "/binary-" . $arch . "/Packages" );
               add_url_to_download( $url . $_ . "/binary-" . $arch . "/Packages.gz" );
               add_url_to_download( $url . $_ . "/binary-" . $arch . "/Packages.bz2" );
               add_url_to_download( $url . $_ . "/binary-" . $arch . "/Packages.xz" );
  EOF
  sudo patch /usr/bin/apt-mirror -o /usr/local/bin/apt-mirror-patched -i ./apt-mirror.patch
  sudo chmod 0755 /usr/local/bin/apt-mirror-patched
  
  # Now run apt-mirror-patched

[godunko-v]

Hello,

Thank you for digging deeper into this issue and for providing a detailed explanation and workarounds! I tried the patch for apt-mirror, and it is indeed working.

I can now successfully mirror the google-cloud-ops-agent-noble-all repository. I really appreciate your support and the quick resolution.

Thanks again for the help!