Removal of Token Persistence

[chamilaadhi]

Problem

Current API-M deployments utilize JWT access tokens since the required number of queries for the validation is much lesser than the opaque tokens. But due to the following use cases, a reference called the JTI is persisted at the token table for each generated access token.

• To revocate an access token before the actual expiry time.
• To use refresh token in refresh-grant flow

In large-scale deployments of WSO2 API Manager, this can cause a bottleneck in the persistence layer and result in reduced TPS for token generation. This can negatively impact the overall performance of the deployment. A solution is needed to improve TPS for token generation in these high-load scenarios.

Solution

The proposed solution aims to improve token management by removing the persistence of tokens in the database.

[chamilaadhi]

Initial POC was done by Dushani and it is documented in https://docs.google.com/document/d/1Owp_Ams5Y_xLX0xFk2t50rFCeNxL0T6CxIs4ITHTMc4/edit#

Following areas need to be handled

• Access Token Generation
• Refresh Token Generation
• Refresh Token Grant Type related changes
• Handling Revocation Token Endpoint
• Token Validation
• Revocations due to internal events

[chamilaadhi]

Following are the PRs for Identitiy repo containing newly introduced interfaces

Access Token Generation wso2-extensions/identity-inbound-auth-oauth#2041
Refresh Token Generation wso2-extensions/identity-inbound-auth-oauth#2042
Refresh Token Grant Type related changes wso2-extensions/identity-inbound-auth-oauth#2043

[chamilaadhi]

Interface and code refactor for revoke flow wso2-extensions/identity-inbound-auth-oauth#2044

[chamilaadhi]

Add new component for keymanager extension to add new code repo wso2-extensions/apim-km-wso2is#96

[chamilaadhi]

New DB is introduced to store persistence related data

  CREATE TABLE IF NOT EXISTS AM_INVALID_TOKENS (
     UUID VARCHAR(255) NOT NULL,
     SIGNATURE VARCHAR(2048) NOT NULL,
     CONSUMER_KEY VARCHAR(255) NOT NULL,
     TOKEN_TYPE VARCHAR(15) NOT NULL,
     TIME_CREATED TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
     EXPIRY_TIMESTAMP BIGINT NOT NULL,
     PRIMARY KEY (UUID)
  ) ENGINE=InnoDB;

Need to add the following datasource in deployment.toml to point to this

  [[datasource]]
  id = "WSO2_PERSISTENCE_DB"
  url = "jdbc:mysql://localhost:3306/invalid_token_db?useSSL=false"
  username = "root"
  password = "Admin123"
  driver = "com.mysql.jdbc.Driver"
  validationQuery = "SELECT 1"
  pool_options.validationInterval = "30000"
  pool_options.defaultAutoCommit = "true"
  pool_options.maxActive = "50"
  pool_options.maxWait = "60000"
  pool_options.testOnBorrow = "true"

Add db util to access new db wso2-extensions/apim-km-wso2is#97

[chamilaadhi]

identity-inbound-auth-oauth repo prs for master branch with new interfaces
wso2-extensions/identity-inbound-auth-oauth#2054
wso2-extensions/identity-inbound-auth-oauth#2055
wso2-extensions/identity-inbound-auth-oauth#2056

[chamilaadhi]

initial implementation for apim keymanager connectors wso2-extensions/apim-km-wso2is#100

[chamilaadhi]

Add mechanism to handle existing opaque refresh tokens during refresh token grant flow. New refresh token will be generated as JWT token. PR related wso2-extensions/apim-km-wso2is#101

[chamilaadhi]

In APIM, portals work with opaque tokens. This needs to be changed to JWT tokens. Currently, JWT tokens are not supported when invoking product APIs. JWT based implementation is introduced to Choreo but is not working with APIM. Currently working on getting this to work with APIM

[chamilaadhi]

Set the product apis to work with JWT tokens wso2/carbon-apimgt#11976

With the above change, we need to increase the maxHttpHeaderSize in the catalina-server.xml. We need to increase this because the refresh token is sent as a JWT and without this, following error is thrown when login in to the publisher

    [2023-04-20 13:11:50,731] ERROR - Http11Processor Error processing request
    org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.
        at org.apache.coyote.http11.Http11OutputBuffer.checkLengthBeforeWrite(Http11OutputBuffer.java:473) ~[tomcat_9.0.70.wso2v1.jar:?]
        at org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:426) ~[tomcat_9.0.70.wso2v1.jar:?]

Increase the maxHttpHeaderSize using following

    [transport.https.properties]
    maxHttpHeaderSize = "12288"

[chamilaadhi]

Add cleanup mechanism to remove expired tokens wso2-extensions/apim-km-wso2is@386ee33

Created a doc with instruction on setting up https://docs.google.com/document/d/16uISJGhspJfQ31iYaAflwdoNNSRbJABbEdsBpt7EaII/edit

[dushaniw]

2023.11.06
Fixed the PR review comments in wso2-extensions/identity-inbound-auth-oauth#2091

[dushaniw]

Following are the remaining tasks of the feature.

• Fix PR comments
• Create design outline doc
• Handle token introspection
• Handle userinfo endpoint
• Handle indirect revocations in refresh grant and token revocation
• Refactor connector code
• Analyze impact of oauth2 tokens | Competitor analysis
• Handle oauth tokens parallely with a config
• Refactor gateway validation code
• Move connector code to IS connector
• Smoke test
• Performance tests

[dushaniw]

Task List Updated as of 2023.11.04

✅ Check whether app updates, delete are revoking tokens in default IS case and non persistence case
🕐 Test scenarios in the Testing CheckList

• Add java doc comments
• Refactor lakith’s APIM code
• Connect with APIM code
• Test IS-KM scenario + backend JWT
• Refactor APIM code changes
• Define unit test cases
• Define integration test cases
• Prepare doc for IS discussion
• Prepare doc for code review
• Write unit test cases
• Write integration test cases
• Prepare and test db scripts for all databases

To be discussed

• Handle tenant delete events for token revocation

Testing Checklist

Tested scenarios
✅ Internal user event for revocation + introspection
✅ Internal consumer app event for revocation + introspection
✅ Access token generation - password + client cred grants
✅ Refresh token generation - password grant
✅ Refresh grant handler basic scenarios
✅ Consented claim in refresh and access token generation + userinfo endpoint
✅ Refresh grant handler - error path for access token
✅ Refresh token (valid, invalid) + introspection endpoint
✅ Refresh grant handler - revoked refresh token
✅ Authorization code grant handler basic scenarios
✅ Token revocation direct access token + introspection + userinfo endpoint
✅ Token revocation direct refresh token + introspection + userinfo endpoint
✅ Internal userevent for revocation + client cred grant token + introspection endpoint
✅ Internal consumer app event for revocation + client cred grant token + introspection endpoint

TODO

• Refresh grant handler - renew existing token = false
• Refresh grant handler - opaque refresh token
• Authorization code grant consented claim + userinfo endpoint
• Authorization code grant cache behavior
• Authorization code grant with id token
• Implicit grant basic scenarios with consented claim + userinfo endpoint
• Implicit grant basic scenarios with id token
• Token introspection for opaque refresh tokens
• Token revocation for opaque refresh tokens
• Federated user + auth code grant + introspection + userinfo
• JWT Bearer grant
• SAML Bearer grant

[dushaniw]

The pending and in-progress items of the task are as follows for the current scope :

• Add recently suggested changes to IS
  • introduce new claim to keep token ID in JWT claims
  • retrieve user from userID from authz user session table. carbon-identity-framework#5241, Handle federated users for token non persistence token refresh wso2-extensions/identity-inbound-auth-oauth#2284
• Run integration tests in IS
• Run performance tests used for IS release for token performance
  • https://github.com/wso2/performance-is/tree/master/benchmarks

Once kernel and identity versions are bumped in latest carbon-apimgt following needs to be done :

• Merge carbon-apimgt related PR from Lakith
• Test API-M portals with token persistence removal
• Fix API-M portal client side to maintain the indirect revoked token event map

[dushaniw]

Backporting feature for IS 6.1 related identity branches.

• Initial extension points for revocation, refresh grant - Porting Add extensions for token persistence removal feature wso2-extensions/identity-inbound-auth-oauth#2091
• Extension points for token introspection and user info - Porting Add extension support for token validation during token introspection and user info endpoints wso2-extensions/identity-inbound-auth-oauth#2176
• Add config and new claims needed for JWTS - Porting Add proposed changes for token persistence removal wso2-extensions/identity-inbound-auth-oauth#2208,
• Template token persistence config. carbon-identity-framework#5073
• Handle consented claim during non persistence - Porting Handle consented flows for no persistence scenarios wso2-extensions/identity-inbound-auth-oauth#2241
• Introduce new claims for JWT for token persistence removal wso2-extensions/identity-inbound-auth-oauth#2253
• Refactor jwt utils wso2-extensions/identity-inbound-auth-oauth#2254
• Config for auth code persistence - Porting Add config for auth code persistence carbon-identity-framework#5162
• Properties to identify migrated tokens from content - Porting Add properties to identify migrated tokens | Add config to deprecate revoked token headers in revocation response. wso2-extensions/identity-inbound-auth-oauth#2267
• Config to remove revoked token headers from response - Porting add config for enabling revoked token headers in revoke endpoint response carbon-identity-framework#5206
• add method to session store to retrieve user by id - Porting retrieve user from userID from authz user session table. carbon-identity-framework#5241
• Handle user attribute cache for token refresh in federated non provisioned scenario - Porting Handle user attribute cache for token refresh wso2-extensions/identity-inbound-auth-oauth#2317
• Fix identity versions in Token persistence Removal wso2-extensions/apim-km-wso2is#112
• Add abstractions for user event changes for token revocations - Porting Add abstractions to indirect revocations due to internal user events wso2-extensions/identity-inbound-auth-oauth#2179, Add method call for newly added methods wso2-extensions/identity-data-publisher-oauth#101
• Add abstractions for app event changes for token revocations - Porting https://github.com/wso2-extensions/identity-inbound-auth-oauth/pull/2089/files

[dushaniw]

The PRs for the feature has been merged. The bugs found while further testing needs to be fixed. There tracked in TokenPersistenceRemoval bugs for token persistence removal

Will close the issue once the doc PR is merged

[tharikaGitHub]

Closing as the task is completed.