API ergonomics

[bugadani]

Hi!

Let me try to describe how I misuse jsonwebtokens. Unfortunately I can't share the source code directly, but I think I can describe it. Now, this is rather low priority, but I hope some of these points may give you some idea how other people (mis)use jsonwebtokens.

My use case

I'm perhaps misusing JWT, but I store a different secret for each of my tokens. This means I need to look the secret up with every token validation.
I also store a token type (in fact, my token is an enum with different fields for each variant). This means I need to build different verifier objects depending on the data itself.

I solve this currently by slicing the input token up with split_token(), I decode (with decode_json_token_slice and decoding with serde_json) the header to get the signature algo, decode the claims to get the token's unique id and look up the secret. I could probably put the id into the header, but that's a different question.

After this, I look up the token secret and I verify the whole token with a Verifier instance.

My problems with what I need to do:

• This (deconstructing the token by hand) is a lot of code, but at least it works.
• I need to deserialize the token (well, parts of the token) twice.
• Even though the algorithm is encoded in my token, I need to mine it out. To allow users to change the signature without
  invalidating issued tokens, jsonwebtokens could/should probably detect the algo automatically, maybe even by default.

Alternatives

I have some alternative usage patterns in mind, that may be more elegant in my case:

1: Let me look up the secret in the verifier. It's still not ideal for me (I actually need to build different verifiers based on some token property), but it probably is for others:

Verifier::create()
    .auto_algo_from_header() // ideally, this could be an implicit default
    .using_secret(|unverified_claims| async { me.lookup.secret(...).await }) // or simply a string, or whatever
    .build()
    .verify()

2: Let me decode an unverified token into whatever my type is, and let the Verifier verify that. Just as I write this, I realized this is pretty hard to do without deserializing the token twice.

2, again: let me decode an unverified token into an intermediate type I can work with, then let the Verifier verify that and decode into my concrete type.

[rib]

Cool, thanks for writing up your use case, I'll try and quote and reply to some bits of it...

My use case

I'm perhaps misusing JWT, but I store a different secret for each of my tokens. This means I need to look the secret up with every token validation.

okey, yep, that's not too unusual I think - in particular it's quite common for the "kid" in the header to be used as an id for looking up public keys within a Json Web Key Set, which is something I do within https://github.com/rib/jsonwebtokens-cognito

I also store a token type (in fact, my token is an enum with different fields for each variant). This means I need to build different verifier objects depending on the data itself.

So I suppose I wonder if this is done with an entirely non-standard payload for the token, or maybe you put your custom type fields inside your "claims" and still also keep some of the other standard fields like "exp", "aud", "sub"?

I solve this currently by slicing the input token up with split_token(), I decode (with decode_json_token_slice and decoding with serde_json) the header to get the signature algo, decode the claims to get the token's unique id and look up the secret. I could probably put the id into the header, but that's a different question.

After this, I look up the token secret and I verify the whole token with a Verifier instance.

My problems with what I need to do:

• This (deconstructing the token by hand) is a lot of code, but at least it works.

For destructuring it's hopefully possible to do something like this:

use serde_json::value::Value;
let TokenSlices {message, signature, header, claims } = raw::split_token(token)?;
let header: Value = raw::decode_json_token_slice(header)?;
let claims: Value = raw::decode_json_token_slice(claims)?;

I suppose since this is a more manual break down and verify route I'm not quite sure it can be reduced much more but depending on what you do with the claims/header the type might be inferred. Were you thinking it'd be nice to somehow reduce this even further?

• I need to deserialize the token (well, parts of the token) twice.

Right, this should hopefully be possible to avoid - funnily enough one of the reasons I ended up making jsonwebtokens in the first place was because I was a bit annoyed with having to parse the tokens multiple times for my use case with other libs :)

Since you're needing to manually split your token and decode the different parts then you can use the raw:: apis for checking the signature and verifying claims to avoid redundant parsing hopefully.

A completely manual split; decode of parts; signature verification and claims verification could look something like:

let alg = Algorithm::new_hmac(AlgorithmID::HS256, "secret")?;
let verifier = Verifier::create()
    // snip
    .build()?;

let TokenSlices {message, signature, header, claims } = raw::split_token(token)?;
let header = raw::decode_json_token_slice(header)?;
raw::verify_signature_only(&header, message, signature, &alg)?;
let claims = raw::decode_json_token_slice(claims)?;
verifier.verify_claims_only(&claims, time_now)?;

With the enum structure and needing to check the enum type for completing verification, I'm not currently sure whether you'd be wanting to handle verifying those as a separate step after verifying standard claims, or maybe your enum type even affects how you want to verify standard claims too?

I could certainly imagine wanting some final custom steps with something like

<snip>
verifier.verify_claims_only(&claims, time_now)?;
let custom: MyClaims = serde_json::from_value(&claims);
let ok = match custom.my_type {
   ... extra verification depending on enum type
}

I can see how the Verifiers customization api is not going to be really well suited to handling the dependency on the enum type here. Currently The most flexible escape hatch is the claim_callback api that lets you specify a closure that can be used to verify a claim, but it's only really geared for checking one claim in isolation.

If your enum type should affect verifying standard claims then I guess, hopefully its not too many secret+enum-type combinations and I suppose you define a seperate Verifier for each combo. In that case your (secret,enum-type) could act like a key for finding the right Verifier you need and those are the only two bits of information your read before verifying claims (besides checking the signature).

• Even though the algorithm is encoded in my token, I need to mine it out. To allow users to change the signature without
  invalidating issued tokens, jsonwebtokens could/should probably detect the algo automatically, maybe even by default.

This is a tricky detail perhaps. It was a conscious decision to be very explicit about what algorithm is used considering that there have been a number of security exploits documented as a result of automatic algorithm inference. There are two notable traps that come to mind for libraries that might try and infer/detect the algorithm:

1. In situations where JWT authentication is normally based on public (e.g. RSA) keys where it's not uncommon for a "kid" to be used to pick a key then an attacker might try sending a malformed token that would pick a symmetric algorithm like "alg": "HS256" and they would use the publicly available RSA key as a symmetric key on the off chance that the verification might blindly trust the "alg" choice and still honor the "kid" choice too for the public key (thus treating the publicly available key as the symmetric secret).

2. JWT includes a standard "none" algorithm, and trusting the "alg" state in a token may leave verification vulnerable to getting a malformed token that simply removes the signature and specifies "alg": "none" which could potentially short circuit any verification.

These were two notable dangers I found documented in the wild and so thought it would be worth designing this so you have to specify your algorithm (conceptually nothing about the JWT should be considered trusted until the signature is verified so it seems good to minimize what influence it may have over how you verify a token)

In your situation you may even want to double/tripple check just in case you might be ending up in a risky space by using the embedded "alg" as a selector. An alternative to consider could be that you would associate the algorithm with the various keys/secrets you have, and once you get the key ID from the token then you look up your key but only trust your own key information for determining the algorithm instead of trusting the token.

Hopefully with this background this design choice makes sense, but I also might not have fully followed the nuance of your use case to know if there could still be a case for auto selection in some case (I'm not exactly sure what it means to let users change the signature without invalidating a token). I'm aware that some libraries such as jsonwebtokens-node allow auto selection, but in the past they were also vulnerable to the above issues - and arguably jsonwebtokens-node is still treading a very dangerous/fine line to being vulnerable to "alg": "none" abuse still.

Alternatives

I have some alternative usage patterns in mind, that may be more elegant in my case:

1: Let me look up the secret in the verifier. It's still not ideal for me (I actually need to build different verifiers based on some token property), but it probably is for others:

Verifier::create()
    .auto_algo_from_header() // ideally, this could be an implicit default
    .using_secret(|unverified_claims| async { me.lookup.secret(...).await }) // or simply a string, or whatever
    .build()
    .verify()

If possible, it sounds like it could be good to use the standard "kid" value in a JWT header for identifying your secrets. Moving this into the header could also help minimize what needs to be decoded / read before verifying your signature and claims.

Based on the thoughts, above I'd be quite worried about the combination of auto_algo and looking up the secret from unverified claims. In general with jsonwebtokens it does try to provide access to the lower-level raw:: details since I don't really want to be opinionated enough to get in the way of whatever different use cases people would have but also want to try and strike a balance that is safe by default and errs towards making it difficult to shoot yourself in the foot hopefully.

2: Let me decode an unverified token into whatever my type is, and let the Verifier verify that. Just as I write this, I realized this is pretty hard to do without deserializing the token twice.

Yeah, this relates to one of the reasons I essentially ended up diverging from jsonwebtoken (without an 's') it's easy to paint into a corner where you're forced to double parse. My general design principle had been to provide a high level api for common cases which can handle all the destructuring and verification itself and finally also deserialize to any custom struct you have if wanted, but for more manual stuff, then it works with Value as a lower common denominator and then it's up to you if you want to temporarily deserialize to your own struct or maybe just at the very end deserialize to your own struct, but this way it's at least in your control whether you do some extra deserializing.

So to clarify; you can decode an unverified token into whatever type you like (the decode apis are generic and will rely on type inference to determine what to deserialize into) - but if you end up needing the raw apis for more manual use cases then they basically require you to start out by destructuring / deserializing into Value, but you could always choose to additionally deserialize into a custom type for some extra verification steps and wouldn't really be penalized from the pov that we needed the Value at the very least anyway as a lowest common denominator that jsonwebtokens works with internally.

2, again: let me decode an unverified token into an intermediate type I can work with, then let the Verifier verify that and decode into my concrete type.

I think this should be possible with the current api, e.g. I think you should be able to do something like:

let TokenSlices {message, signature, header, claims } = raw::split_token(token)?;
struct MyCustomClaims {
    foo: u32
}
let claims = raw::decode_json_token_slice(claims)?;

or rather I suppose you mean you just want your custom/concrete type at the end, and in that case the 'intermediate' type would be a Value which can be passed to verifier.verify_claims_only(&claims, time_now)?; and although it's not provided directly by jsonwebtokens itself at the very end you could perhaps use serde_json::from_value() to then deserialize into your concrete type when finished?

Thanks again for summarising your use case. I can see how the conditional verification based on your token type adds some extra complexity where it's maybe not obvious if it's best to handle that using the Verifier api or maybe handle as a step afterwards - I guess the main decider might be whether the type affects verification of standard claims.

I have a feeling that with something like this I'd maybe try and use the standard"kid" header value if possible for looking up secrets and could imagine combining that with the enum type to create key for looking up your (secret+pre-built Verifier) and in that way avoid needing to query any algorithm choice from an unverified token if possible (guessing each secret presumably implies a specific algorithm). It might not really be an issue for you but one of the ideas with the original design was to allow the creation of Verifiers to be decoupled from runtime verification so that keys/secrets only need to be decoded once (mainly in cases dealing with RSA or EC keys in PEM format to avoid re-parsing them repeatedly -which might not matter for your case)

Hope a few of my rambling thoughts/pointers make sense here. :) (let me know if not) I think the main thing it's got me pondering about is whether there's a practical way to help with conditional claim verification - like with your enum type. Off the top of my head I think it could easily get out of hand for the Verifier api to really get overly flexible/complex but maybe there's another kind of simple escape hatch that could be added to help with that kind of thing.

[bugadani]

Thanks for the detailed response, I'll need some time to properly go through it and I'll get back to you with my thoughts once that's done.

I think I'm currently doing most of what you wrote, and I also think you found me a security issue - thanks for that, I'll need to allowlist the algos I accept. Which would probably make the automatic algo detection work, but also raises the question if that would make sense in any form.

[bugadani]

While reading your response I realized that I might not have been clear. You are completely right in what you say, and the availability of the low level API is why I went with jsonwebtokens. I think I was just so focused on getting my stuff done and I was trying to put a square peg through a round hole by trying to use the high level functionality, where in fact I could have rolled my own based on the raw::.

Oh, after reading what you wrote, the design makes actually a lot of sense.

So, my takeaways:

• I should have focused more on the raw API. I was already using it sort of the way you outlined, but not exactly.
• I can build my own verification method to suit my needs (duh, obvious enough for me to completely ignore).
• I can also see a hammer and treat everything as a nail.

Thanks for your time and insight and let's see if I can up the efficiency of my code :)

Late edit: I was actually able to refactor my code to a reasonably clean version, just by using the Value objects handed to me. Right now I feel a bit sad that I'm only using the Verifier to check the few standard fields I have in my tokens 😅 But thanks again for your time and the suggestions :)

[rib]

Glad my thoughts were of some help! Yeah, from your use case it sounded like you were off the beaten path enough to warrant using the ::raw api. It's a bit of a shame to not be able to handle your custom claims but it seems that the real challenge there is the state interdependencies (with the enum type) and I imagine that generalizing Verifier to be able to declare that interdependency could easily turn out more complex than handling those custom/application-specific claims directly.

Thanks again for writing up your use case. I'm thinking atm, when I get a chance, I may spin a 1.0 release from master, after a version bump for simple_asn1. (I wanted to initially hold out to look at your use case in case it could have made sense to tweak the api for something - but it sounds like you're all set for now 🤞 )