Client Authentication Fail (HTTP 401) during registration to RMT-Server

[rafaprata]

I am encountering the following scenario:

• RMT-Server v2.14 (I also tested with versions 2.15 and 2.17) using helm-chart
• Kubernetes v1.24.10
• Clients running SLES 15 SP5 and 15 SP3

The server has already mirrored the necessary repositories for both clients, and when registering the clients on the RMT-Server, I observe two different outcomes:

1. SLES 15 SP3 - This is the easier scenario, as running the client setup works correctly. The repositories point to the RMT server, and I can perform the necessary upgrades on the client.
2. SLES 15 SP5 - This scenario is peculiar. Given the same context, the only significant difference is the client's operating system. It returns an authentication error when running the command /usr/bin/zypper --non-interactive refs SUSE_Linux_Enterprise_Server_x86_64, with the RMT server returning a 401 error, causing the registration to fail.

I have reviewed the logs and everything possible to understand this issue, even contacting SUSE support for assistance. However, I have not been able to resolve it. I noticed that during the SUSEConnect --write-config --url <RMT-SERVER-URL> command, it modifies repositories, services, and connections to the RMT-Server. At some point in this process, it executes a refresh services command (zypper refs), passing the services individually. This command makes an HTTP request to the RMT server. In both scenarios, the first request returns a 401 error, but in scenario 1, the second request succeeds (200), while in scenario 2, it returns 401.

I do not understand why the password generated when the system in scenario 2 was registered on the RMT server does not work for this connection. I observe that the command correctly registers the information in the /etc/zypp/credentials.d directory, and this information can be verified both in this directory and in the RMT server database.

Thank you.
Rafael Souza

[digitaltom]

RMT in its default config does not authorize access to repositories, so it's unexpected that you run into a 401 error. Could you provide the console output from zypper? Maybe it fails on a repository that is not hosted on RMT?

[rafaprata]

For sure @digitaltom.

Here is the error I'm facing:

Registering system to registration proxy https://<rmt-server-url>/

Announcing system to https://<rmt-server-url>/ ...

Activating SLES 15.5 x86_64 ...
-> Adding service to system ...
command '/usr/bin/zypper --non-interactive refs SUSE_Linux_Enterprise_Server_x86_64' failed
Error: zypper returned 1 with 'Login failed. (http://<rmt-server-url>/services/2465/repo/repoindex.xml?credentials=SUSE_Linux_Enterprise_Server_x86_64): The requested URL returned error: 401
Unexpected exception.
Unknown error reading from 'http://<rmt-server-url>/services/2465?credentials=SUSE_Linux_Enterprise_Server_x86_64'
History:
 - ABORT request: Aborting requested by user
 - Login failed. (http://<rmt-server-url>/services/2465/repo/repoindex.xml?credentials=SUSE_Linux_Enterprise_Server_x86_64): The requested URL returned error: 401

Please file a bug report about this.
See http://en.opensuse.org/Zypper/Troubleshooting for instructions.' (exit status 1)

And here is the repos mirrored inside rmt-server:

+------+--------------------------------------------+---------+--------+---------+-------------------------+
| ID   | Product                                    | Version | Arch   | Mirror? | Last mirrored           |
+------+--------------------------------------------+---------+--------+---------+-------------------------+
| 2474 | Basesystem Module                          | 15 SP5  | x86_64 | Mirror  | 2024-07-23 16:55:26 UTC |
|      | sle-module-basesystem/15.5/x86_64          |         |        |         |                         |
| 2504 | Public Cloud Module                        | 15 SP5  | x86_64 | Mirror  | 2024-07-23 17:00:42 UTC |
|      | sle-module-public-cloud/15.5/x86_64        |         |        |         |                         |
| 2536 | Python 3 Module                            | 15 SP5  | x86_64 | Mirror  | 2024-07-23 17:01:10 UTC |
|      | sle-module-python3/15.5/x86_64             |         |        |         |                         |
| 2482 | Server Applications Module                 | 15 SP5  | x86_64 | Mirror  | 2024-07-23 16:56:58 UTC |
|      | sle-module-server-applications/15.5/x86_64 |         |        |         |                         |
| 2465 | SUSE Linux Enterprise Server               | 15 SP5  | x86_64 | Mirror  | 2024-07-23 16:42:19 UTC |
|      | SLES/15.5/x86_64                           |         |        |         |                         |
+------+--------------------------------------------+---------+--------+---------+-------------------------+
Only enabled products are shown by default. Use the '--all' option to see all products.

[digitaltom]

Could you compare the credentials on the client system (/etc/zypp/credentials.d/SCCcredentials) with the RMT database:

cd /usr/share/rmt; RAILS_ENV=production bin/rails console
System.find_by(login: '<login from file>')

Also, I saw a similar symptom when there is a proxy used that filters basic auth headers, could that be the case?

[rafaprata]

Yeah for sure!

Here is the comparison, it looks the same:

# Client SCCcredentials file
ops-master01:/etc/zypp/credentials.d # cat SCCcredentials
username=SCC_261865521d4242e5bf188590f0529391
password=207d424fe12f4e0f
system_token=bb1ba621-b94b-4e55-b1b2-c61dc7bdaaca

# RMT-Server verification:
rmt-server-app-6f458777b5-426hq:/usr/share/rmt # RAILS_ENV=production bin/rails console
irb(main):001:0> System.find_by(login: 'SCC_261865521d4242e5bf188590f0529391')
D, [2024-07-25T09:34:14.353724 #221] DEBUG -- :   System Load (1.1ms)  SELECT `systems`.* FROM `systems` WHERE `systems`.`login` = 'SCC_261865521d4242e5bf188590f0529391' LIMIT 1
=> #<System id: 1, login: "SCC_261865521d4242e5bf188590f0529391", password: [FILTERED], hostname: "ops-master01", registered_at: "2024-07-24 10:20:34.000000000 +0000", last_seen_at: "2024-07-25 08:53:09.000000000 +0000", created_at: "2024-07-24 10:20:34.000000000 +0000", updated_at: "2024-07-25 08:53:09.000000000 +0000", scc_registered_at: nil, scc_system_id: nil, proxy_byos: false, system_token: "bb1ba621-b94b-4e55-b1b2-c61dc7bdaaca", system_information: "{\"hostname\":\"ops-master01\",\"cpus\":4,\"sockets\":4,\"h...", instance_data: nil>
irb(main):002:0> 

About the basic auth headers, I don't know about it. Because the scenario that the error happens is a scenario that a client VM try to connect to RMT server that is deployed on a Kubernetes cluster. This client is one control-plane node of this kubernetes cluster. Basically the client VM was sending a request to it self basically.
Another point is that, when I'm trying to register to RMT other VM in the same network, with the same configuration [except 2 things: 1) The OS (SLES 15 SP3) and 2) This new VM can reach to the Kubernetes cluster but is not part of it ] works! I didn't configured anything else just running the same command on the both VMs.

To deploy on kubernetes I'm using the helm chart recommended on the RMT documentation. If were there any configuration about the headers that I can add, if you help me, I will agree to try on my side.