Using 'az ssh config' generated key and cert results in 'error: AuthorizedKeysCommand /usr/sbin/aad_certhandler'

[lmpeiris]

Describe the bug

I'm using 'az ssh config' command to generate ssh config file with private key and cert.

This method works for some servers, but not for others. All are azure provided ubuntu 22.04.3 LTS, including client.

'az ssh vm' command works without issues, hence this is not related to #4026 although looks similar. I'm trying to use ssh -F / scp -F command to use the ssh-config to log in.

Following workarounds already applied:

in client side ssh_config:
PubkeyAcceptedKeyTypes [email protected]

in server side (only applied in server side: with or without this it doesn't work):
HostKeyAlgorithms +ssh-rsa
PubkeyAcceptedKeyTypes +ssh-rsa

Below are ssh client and server logs (redacted) for both scenarios.

Issue server

client side:

debug1: Offering public key: /home/user/scp_ssh_keys/az_ssh_config/AD-TEST-AD-TEST/id_rsa.pub-aadcert.pub RSA-CERT SHA256:tS/9gTUU+OcD4a6irJfRG1bapWI1w/U69DGcvYNPIR4 explicit
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: /home/user/scp_ssh_keys/az_ssh_config/AD-TEST-AD-TEST/id_rsa RSA SHA256:tS/9gTUU+OcD4a6irJfRG1bapWI1w/U69DGcvYNPIR4 explicit
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]@172.27.3.17: Permission denied (publickey).
lost connection

server side:

Oct 25 09:43:50 AD-TEST aad_certhandler[30421]: Version: 1.0.023850001; user: [email protected]
Oct 25 09:43:50 AD-TEST aad_certhandler[30421]: The public key is of type ssh-rsa, not a certificate.
Oct 25 09:43:50 AD-TEST sshd[30419]: error: AuthorizedKeysCommand /usr/sbin/aad_certhandler [email protected] AAAAxxxxxxxxxxxxxxxxxxxxxxxxx

Working server

client side:

debug1: Offering public key: /home/user/scp_ssh_keys/az_ssh_config/AD-TEST-AD-TEST3/id_rsa.pub-aadcert.pub RSA-CERT SHA256:0jnu1u7X66BPOJIH+jr2Leli36yaG7CrdTjGyqT7THw explicit
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /home/user/scp_ssh_keys/az_ssh_config/AD-TEST-AD-TEST3/id_rsa.pub-aadcert.pub RSA-CERT SHA256:0jnu1u7X66BPOJIH+jr2Leli36yaG7CrdTjGyqT7THw explicit
debug2: sign_and_send_pubkey: using private key "/home/user/scp_ssh_keys/az_ssh_config/AD-TEST-AD-TEST3/id_rsa" for certificate
Authenticated to 172.27.3.18 ([172.27.3.18]:22) using "publickey".

server side:

Oct 25 09:41:49 AD-TEST3 aad_certhandler[1743973]: Version: 1.0.023850001; user: [email protected]
Oct 25 09:41:49 AD-TEST3 aad_certhandler[1743973]: This is an Azure machine
Oct 25 09:41:49 AD-TEST3 sshd[1743970]: Certificate extension "[email protected]" is not supported
Oct 25 09:41:49 AD-TEST3 sshd[1743970]: Certificate extension "[email protected]" is not supported
Oct 25 09:41:49 AD-TEST3 sshd[1743970]: Certificate extension "[email protected]" is not supported
Oct 25 09:41:49 AD-TEST3 sshd[1743970]: pam_aad(sshd:account): AadAuthorize, Version: 1.0.023850001; CorrelationId: c25e6714-c34a-4ca6-b4c9-42621285bee9
Oct 25 09:41:49 AD-TEST3 sshd[1743970]: pam_aad(sshd:account): This is an Azure machine
Oct 25 09:41:51 AD-TEST3 sshd[1743970]: pam_aad(sshd:account): Login granted for [email protected] as an admin.
Oct 25 09:41:51 AD-TEST3 sshd[1743970]: Accepted publickey for [email protected] from 172.27.0.149 port 41746 ssh2: RSA-CERT SHA256:0jnu1u7X66BPOJIH+jr2Leli36yaG7CrdTjGyqT7THw ID 4025d326-304e-4fc9-acb3-3194b2495023@50d9a2a9-a260-4175-8c40-a96f35dce44c (serial 0) CA RSA SHA256:n0oGIqYl9RlHxCr2bNmfFBLrKEMO/1yeZ8AkuW4YQqo
Oct 25 09:41:51 AD-TEST3 systemd[1743980]: pam_unix(systemd-user:session): session opened for user [email protected] by (uid=0)
Oct 25 09:41:51 AD-TEST3 sshd[1743970]: pam_unix(sshd:session): session opened for user [email protected] by (uid=0)
Oct 25 09:41:51 AD-TEST3 sshd[1743988]: Received disconnect from 172.27.0.149 port 41746:11: disconnected by user
Oct 25 09:41:51 AD-TEST3 sshd[1743988]: Disconnected from user [email protected] 172.27.0.149 port 41746
Oct 25 09:41:51 AD-TEST3 sshd[1743970]: pam_unix(sshd:session): session closed for user [email protected]

Related command

az ssh config --file "$ssh_config_file" --subscription $subscription -g $resource_group -n "$vm_name" --prefer-private-ip

Errors

client side:
debug1: send_pubkey_test: no mutual signature algorithm

server side:

Oct 25 09:43:50 AD-TEST aad_certhandler[30421]: The public key is of type ssh-rsa, not a certificate.
Oct 25 09:43:50 AD-TEST sshd[30419]: error: AuthorizedKeysCommand /usr/sbin/aad_certhandler [email protected] AAAAxxxxxxxxxxxxxxxxxxxxxxxxx

Issue script & Debug output

can be provided if the issue seems reasonable to investigate

Expected behavior

ssh / scp via ssh key / certificate should be successful in to vms with AAD extension enabled

Environment Summary

azure-cli 2.53.0 *

core 2.53.0 *
telemetry 1.1.0

Extensions:
ssh 2.0.2

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

Python location '/opt/az/bin/python3'
Extensions directory '/home/malshan/.azure/cliextensions'

Python (Linux) 3.10.10 (main, Sep 20 2023, 06:07:38) [GCC 11.4.0]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

servers and client: ubuntu LTS 22.04.3

[yonzhan]

Thank you for opening this issue, we will look into it.

[lmpeiris]

i would be glad if i can get an update on this issue.

[GBBx]

I'm facing the same issue. I see there have been new aadsshlogin releases recently but has this problem been solved?