Certificate issues when connecting session in Android app

[SilverMira]

logcat looks something like this when trying to Session::connect

logcat

11-19 16:46:46.873 14890 14927 V rust_lib_frb_base::ap..: Connecting to librespot
11-19 16:46:46.873 14890 14927 D librespot_core::apres..: new ApResolver
11-19 16:46:46.873 14890 14927 D librespot_core::http_..: Requesting https://apresolve.spotify.com/?type=accesspoint&type=dealer&type=spclient
11-19 16:46:46.874 14890 14927 W librespot_core::apres..: Failed to resolve all access points, using fallbacks
11-19 16:46:46.874 14890 14927 W librespot_core::apres..: Resolve access points error: Requested entity was not found { no native root CA certificates found (errors: []) }
11-19 16:46:46.874 14890 14927 I librespot_core::session: Connecting to AP "ap.spotify.com:443"
11-19 16:46:47.499 14890 14930 D librespot_core::conne..: Authenticating with AP using AUTHENTICATION_SPOTIFY_TOKEN
11-19 16:46:47.779 14890 14930 W librespot_core::session: Instructed to try another access point...
11-19 16:46:47.780 14890 14930 D librespot_core::http_..: Requesting https://apresolve.spotify.com/?type=accesspoint&type=dealer&type=spclient
11-19 16:46:47.780 14890 14930 W librespot_core::apres..: Failed to resolve all access points, using fallbacks
11-19 16:46:47.780 14890 14930 W librespot_core::apres..: Resolve access points error: Requested entity was not found { no native root CA certificates found (errors: []) }
11-19 16:46:47.780 14890 14930 I librespot_core::session: Connecting to AP "ap.spotify.com:443"
11-19 16:46:48.057 14890 14914 D EGL_emulation: app_time_stats: avg=19.43ms min=1.18ms max=500.33ms count=38
11-19 16:46:48.406 14890 14930 D librespot_core::conne..: Authenticating with AP using AUTHENTICATION_SPOTIFY_TOKEN
11-19 16:46:48.674 14890 14930 W librespot_core::session: Instructed to try another access point...
11-19 16:46:48.674 14890 14930 D librespot_core::http_..: Requesting https://apresolve.spotify.com/?type=accesspoint&type=dealer&type=spclient
11-19 16:46:48.674 14890 14930 W librespot_core::apres..: Failed to resolve all access points, using fallbacks
11-19 16:46:48.674 14890 14930 W librespot_core::apres..: Resolve access points error: Requested entity was not found { no native root CA certificates found (errors: []) }
11-19 16:46:48.674 14890 14930 I librespot_core::session: Connecting to AP "ap.spotify.com:443"
11-19 16:46:49.059 14890 14914 D EGL_emulation: app_time_stats: avg=500.72ms min=500.23ms max=501.21ms count=2
11-19 16:46:49.214 14890 14930 D librespot_core::conne..: Authenticating with AP using AUTHENTICATION_SPOTIFY_TOKEN
11-19 16:46:49.479 14890 14930 W librespot_core::session: Instructed to try another access point...
11-19 16:46:49.479 14890 14930 D librespot_core::http_..: Requesting https://apresolve.spotify.com/?type=accesspoint&type=dealer&type=spclient
11-19 16:46:49.480 14890 14930 W librespot_core::apres..: Failed to resolve all access points, using fallbacks
11-19 16:46:49.480 14890 14930 W librespot_core::apres..: Resolve access points error: Requested entity was not found { no native root CA certificates found (errors: []) }
11-19 16:46:49.480 14890 14930 I librespot_core::session: Connecting to AP "ap.spotify.com:443"
11-19 16:46:49.914 14890 14930 D librespot_core::conne..: Authenticating with AP using AUTHENTICATION_SPOTIFY_TOKEN
11-19 16:46:50.011 14890 14930 W librespot_core::session: Instructed to try another access point...
11-19 16:46:50.011 14890 14930 D librespot_core::http_..: Requesting https://apresolve.spotify.com/?type=accesspoint&type=dealer&type=spclient
11-19 16:46:50.011 14890 14930 W librespot_core::apres..: Failed to resolve all access points, using fallbacks
11-19 16:46:50.011 14890 14930 W librespot_core::apres..: Resolve access points error: Requested entity was not found { no native root CA certificates found (errors: []) }
11-19 16:46:50.011 14890 14930 I librespot_core::session: Connecting to AP "ap.spotify.com:443"
11-19 16:46:50.555 14890 14930 D librespot_core::conne..: Authenticating with AP using AUTHENTICATION_SPOTIFY_TOKEN
11-19 16:46:50.557 14890 14914 D EGL_emulation: app_time_stats: avg=499.41ms min=498.30ms max=499.97ms count=3
11-19 16:46:50.823 14890 14930 W librespot_core::session: Instructed to try another access point...
11-19 16:46:50.823 14890 14930 D librespot_core::http_..: Requesting https://apresolve.spotify.com/?type=accesspoint&type=dealer&type=spclient
11-19 16:46:50.823 14890 14930 W librespot_core::apres..: Failed to resolve all access points, using fallbacks
11-19 16:46:50.823 14890 14930 W librespot_core::apres..: Resolve access points error: Requested entity was not found { no native root CA certificates found (errors: []) }
11-19 16:46:50.824 14890 14930 I librespot_core::session: Connecting to AP "ap.spotify.com:443"
11-19 16:46:51.416 14890 14930 D librespot_core::conne..: Authenticating with AP using AUTHENTICATION_SPOTIFY_TOKEN
11-19 16:46:51.558 14890 14914 D EGL_emulation: app_time_stats: avg=500.38ms min=499.46ms max=501.31ms count=2
11-19 16:46:51.670 14890 14930 E librespot_core::session: Tried too many access points
11-19 16:46:51.670 14890 14930 E rust_lib_frb_base::ap..: Error connecting to librespot: Error { kind: PermissionDenied, error: LoginFailed(TryAnotherAP) }
11-19 16:46:51.671 14890 14930 D librespot_core::session: drop Session
11-19 16:46:51.671 14890 14930 D librespot_core::apres..: drop ApResolver

I'm suspecting we are hitting this issue, where rustls-native-certs doesn't actually support Android/IOS and instead recommends using rustls-platform-verifier instead.

Relevant rustls-native-certs usage here

Originally posted by @SilverMira in #1277 (comment)

A repro is available here. The app is tested to have working playback on Windows.
I couldn't verify the app to be working on Linux since WSL is giving Timed out errors (unsure whether it's just my WSL acting up). Turns out it's my WSL acting up with DNS queries failing, verified the same application works on linux too with playback.

Relevant logic to librespot here, which is about the same as examples/play.rs

[SilverMira]

I tested to compile against webpki-roots feature for all rustls dependency like such. That has made the certificate errors disappear, will putting a new feature flag to control using either webpki-roots or native-certs be a solution here?

Still, even after the certificate errors went away, I still couldn't get Session to connect, all 6 APs seems to just return TryAnotherAP error every time I tried on Android, tested the webpki-roots branch on other platform and Sessions are connecting as expected, so I'm a little lost why this is the case, even verified that the Login packet sent to AP is identical (except for the few unique ID fields) but still no luck.

[kingosticks]

I've not checked but isn't part of the login packet platform dependent? Lots of Spotify now works very differently on Android so it wouldn't be surprising that the same thing doesn't work there (anymore).

You might be better off pretending to be x86 everywhere. I'm not convinced there's any benefit in us reporting the platform honestly, just more complexity.

[SilverMira]

I've not checked but isn't part of the login packet platform dependent?

Yeah, I'm aware there are device dependent fields sent.

Just for the heck of it, I spoofed the OS field to return Windows as well as CPU family to return X86_64 here, but that didn't work either, all 6 APs still errored with TryAnotherAP

Logging the spoofed login packet before it was sent yields something like this on Android.

ClientResponseEncrypted { login_credentials: MessageField(Some(LoginCredentials { username: None, typ: Some(AUTHENTICATION_SPOTIFY_TOKEN), auth_data: Some([token stuff]), special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } })), account_creation: None, fingerprint_response: MessageField(None), peer_ticket: MessageField(None), system_info: MessageField(Some(SystemInfo { cpu_family: Some(CPU_X86_64), cpu_subtype: None, cpu_ext: None, brand: None, brand_flags: None, os: Some(OS_WINDOWS), os_version: None, os_ext: None, system_information_string: Some("librespot-6c2bf8c-FywWxfn6"), device_id: Some("d8c05d3c-a4fc-4be3-818d-e336f8cd064c"), special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } })), platform_model: None, version_string: Some("librespot 0.6.0-dev"), appkey: MessageField(None), client_info: MessageField(None), special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }

For comparison, running the same application on my host Windows machine results in basically same login packet below, excluding the unique IDs, even used the same access token. But somehow it does connect on Windows 😅

ClientResponseEncrypted { login_credentials: MessageField(Some(LoginCredentials { username: None, typ: Some(AUTHENTICATION_SPOTIFY_TOKEN), auth_data: Some([token stuff]), special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } })), account_creation: None, fingerprint_response: MessageField(None), peer_ticket: MessageField(None), system_info: MessageField(Some(SystemInfo { cpu_family: Some(CPU_X86_64), cpu_subtype: None, cpu_ext: None, brand: None, brand_flags: None, os: Some(OS_WINDOWS), os_version: None, os_ext: None, system_information_string: Some("librespot-6c2bf8c-sG3VBtfO"), device_id: Some("eb421dc8-c417-4b55-aac2-fd97fc7234fb"), special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } })), platform_model: None, version_string: Some("librespot 0.6.0-dev"), appkey: MessageField(None), client_info: MessageField(None), special_fields: SpecialFields { unknown_fields: UnknownFields { fields: None }, cached_size: CachedSize { size: 0 } } }

On another note, the "arm64" comparison for cpu_family is actually supposed to be "aarch64" according to std::env::consts::ARCH docs.

[SilverMira]

I have just noticed, even with certificate errors from using native-certs, the error is also TryAnotherAP. Does any sort of handshaking failure just makes the AP telling librespot to try another AP?

Update: Spoofing all instance of unique values deriving from "android" made playback work. Now to bisect what being sent is causing spotify to reject the client...

[SilverMira]

From fiddling around, I found that if the client_hello packet's platform is Platform::PLATFORM_ANDROID_ARM, all APs will return TryAnotherAP, no matter if the access token is obtained via keymaster's client id, or Android specific client id.

Spoofing just client_hello to send Platform::PLATFORM_LINUX_X86_64 without any other modifications made AP authenticate fails instead with PremiumAccountRequired (I definitely have premium).

I'm just trying various things out currently and hoping it will stick.
If anyone have more insight into how Spotify do their authentication, it will be super helpful!

[kingosticks]

There will be a load of places to change it and stay consistent, including agent type in http requests etc. Their server can only know what you send it, pretending to be non-android should be possible. I've no experience running librespot on android.

[SilverMira]

There will be a load of places to change it and stay consistent, including agent type in http requests etc. Their server can only know what you send it, pretending to be non-android should be possible. I've no experience running librespot on android.

Do you think librespot should spoof Android devices as Linux desktop? It definitely does work but I'm not sure whether the maintainers will agree to spoofing the device details like such. I can create a PR if this is the approach that's to be taken.

[kingosticks]

It seems a reasonable workaround if we can't find anyone to fix it properly.

But if you want to try and fix it properly there might be info in the mega thread at #1308 or in the go/java ports of librespot or at https://github.com/3052/platform/blob/v1.4.9/spotify/spotify.go

How did you obtain your access token?.

[SilverMira]

there might be info in the mega thread at #1308

Alright, will take a look through the ports / thread.

How did you obtain your access token?.

I'm getting them via rspotify, since librespot's oauth flow doesn't currently have a way for app integration (since it either listens on localhost, or ask for input from stdin), and won't work at all when using Android's client ID to perform oauth, where the redirect URI must be https://auth-callback.spotify.com/r/android/music/login rather than localhost.

I generate the PKCE auth_url with the appropriate redirect_url like such using rspotify (scopes "streaming" only), then with a managed webview from flutter, I'm able to extract the code whenever the webview navigates to the redirect URL, and pass the code back to rspotify to complete the oauth flow.

Using the access token from rspotify, I'm able to stream playback as per normal on at least Windows/Linux (keymaster client ID), but no luck on Android (android client ID)

[kingosticks]

We should fix the oauth limitations. Thanks for highlighting.

[SilverMira]

I have created 2 PRs for making things work on Android again.

Both PRs will need to get merged before librespot can stream on Android again.