Enable to use userId as OAuth2 sub

.env.dev

@@ -91,7 +91,6 @@ OAUTH2_AUTH_URL=http://localhost:8083/o/authorize
 OAUTH2_TOKEN_URL=http://wagtail:8000/o/token/
 OAUTH2_CALLBACK_URL=http://localhost:3000/client/authn/oauth2-provider/callback
 OAUTH2_INTROSPECTION_URL=http://wagtail:8000/o/introspect/
-OAUTH2_INTROSPECTION_SUB_KEY=username
 OAUTH2_USER_PROFILE_URL=http://wagtail:8000/profile
 
 

.remarkrc

@@ -9,7 +9,6 @@
       "skipLocalhost": true,
       "skipUrlPatterns": [
         "https://security.stackexchange.com",
-        "https://help.liferay.com",
         "https://github.com/LayerManager/layman/issues/",
         "http://www.plantuml.com/plantuml/proxy"
       ]

CHANGELOG.md

@@ -9,6 +9,14 @@
   ```
 - Stop using environment variable `LAYMAN_GS_ROLE_SERVICE`, it has no effect to Layman anymore. Layman now uses [role service](doc/security.md#role-service) identified by new environment variable [LAYMAN_ROLE_SERVICE_URI](doc/env-settings.md#LAYMAN_ROLE_SERVICE_URI). The service is called `layman_role_service` on GeoServer.
 - Set new environment variable [LAYMAN_ROLE_SERVICE_URI](doc/env-settings.md#LAYMAN_ROLE_SERVICE_URI)
+- Stop using environment variable [`OAUTH2_INTROSPECTION_SUB_KEY`](https://github.com/LayerManager/layman/blob/v1.22.3/doc/env-settings.md#oauth2_introspection_sub_key), it has no effect to Layman anymore. 
+- If you are using Wagtail as OAuth2 provider
+  - After running `make upgrade-demo` or `make-upgrade-demo-full`, run also script `v1_23_change_oauth2_sub_username_to_user_id.py`:
+    ```bash
+    docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps -u root -e LAYMAN_WAGTAIL_DB_URI=<URI_of_Wagtail_db> layman bash -c "cd src && python3 -B v1_23_change_oauth2_sub_username_to_user_id.py"
+    ```
+      - `URI_of_Wagtail_db` is PostgreSQL connection URI to Wagtail database, e.g. `postgresql://user:[email protected]:5432/wagtail_db_name`
+      - The script changes OAuth2 "sub" values in Layman prime DB schema from Wagtail usernames to Wagtail user IDs. See [940](https://github.com/LayerManager/layman/issues/940).
 ### Migrations and checks
 #### Schema migrations
 - [#165](https://github.com/LayerManager/layman/issues/165) Add column `role_name` to table `rights` in prime DB schema. Add constraint that exactly one of columns `role_name` and `id_user` is not null.
@@ -38,6 +46,8 @@
   - GET Workspace [Layers](doc/rest.md#get-workspace-layers)/[Maps](doc/rest.md#get-workspace-maps)
   - GET [Layers](doc/rest.md#get-layers)/[Maps](doc/rest.md#get-maps)/[Publications](doc/rest.md#get-publications)
 - [#165](https://github.com/LayerManager/layman/issues/165) Name of [users](doc/models.md#username) and [public workspaces](doc/models.md#public-workspace) are from now on restricted to a maximum length of 59 characters.
+- [940](https://github.com/LayerManager/layman/issues/940) Use `userId` as OAuth2 "sub" instead of `username`. This is suitable for Wagtail.
+- [940](https://github.com/LayerManager/layman/issues/940) Stop supporting Liferay as OAuth2 provider.
 - [941](https://github.com/LayerManager/layman/issues/941) Wagtail database is now persistent when restarting Layman or Wagtail.
 - All changes from [v1.22.1](#v1221), [v1.22.2](#v1222) and [v1.22.3](#v1223).
 - [#960](https://github.com/LayerManager/layman/issues/960) Handle WMS requests with HTTP error more efficiently in timgen.
@@ -187,7 +197,7 @@
     OAUTH2_INTROSPECTION_SUB_KEY=username
     OAUTH2_USER_PROFILE_URL=http://wagtail:8000/profile
     ```
-  - unset environment variable `OAUTH2_SCOPE` (previously `OAUTH2_LIFERAY_SCOPE`)
+  - unset environment variable `OAUTH2_SCOPE` (previously [`OAUTH2_LIFERAY_SCOPE`](https://github.com/LayerManager/layman/blob/v1.20.1/doc/env-settings.md#oauth2_liferay_scope))
   - after [usual dev upgrade commands](README.md#upgrade) run also
     ```
     make wagtail-build
@@ -368,8 +378,8 @@ make client-build
 - [#541](https://github.com/LayerManager/layman/issues/541) Layer name and map name can start with numbers.
 - Maximum length of layer and map name is 210 characters.
 - [#606](https://github.com/LayerManager/layman/issues/606) Fix filtering and ordering publications by bounding box in case of publication with whole world bounding box in database.
-- New environment variable [OAUTH2_LIFERAY_SCOPE](doc/env-settings.md#oauth2_scope). Introduced in v1.16.2.
-- New environment variable [OAUTH2_LIFERAY_INTROSPECTION_SUB_KEY](doc/env-settings.md#oauth2_introspection_sub_key). Introduced in v1.16.1.
+- New environment variable [OAUTH2_LIFERAY_SCOPE](https://github.com/LayerManager/layman/blob/v1.20.1/doc/env-settings.md#oauth2_liferay_scope). Introduced in v1.16.2.
+- New environment variable [OAUTH2_LIFERAY_INTROSPECTION_SUB_KEY](https://github.com/LayerManager/layman/blob/v1.22.3/doc/env-settings.md#oauth2_introspection_sub_key). Introduced in v1.16.1.
 - [#599](https://github.com/LayerManager/layman/issues/599) Layman supports uploading data files with upper or mixed case extensions. Introduced in v1.16.1.
 - [#541](https://github.com/LayerManager/layman/issues/541) Vector layers are stored in DB table with name in form `layer_<UUID>`, e.g. `layer_96b918c6_d88c_42d8_b999_f3992b826958`, previously the name of the table was the same as name of the layer.
 
@@ -386,7 +396,7 @@ make client-build
   ```
   LAYMAN_CLIENT_VERSION=v1.11.0
   ```
-- If you are using Liferay as OAuth2 provider, set new environment variable [OAUTH2_LIFERAY_SCOPE](doc/env-settings.md#oauth2_scope):
+- If you are using Liferay as OAuth2 provider, set new environment variable [OAUTH2_LIFERAY_SCOPE](https://github.com/LayerManager/layman/blob/v1.20.1/doc/env-settings.md#oauth2_liferay_scope):
   ```
   OAUTH2_LIFERAY_SCOPE=liferay-json-web-services.everything.read.userprofile
   ```
@@ -396,14 +406,14 @@ make client-build
   make client-build
   ```
 ### Changes
-- New environment variable [OAUTH2_LIFERAY_SCOPE](doc/env-settings.md#oauth2_scope).
+- New environment variable [OAUTH2_LIFERAY_SCOPE](https://github.com/LayerManager/layman/blob/v1.20.1/doc/env-settings.md#oauth2_liferay_scope).
 
 ## v1.16.1
  2022-02-25
 ### Changes
 - Fix infinity loop when generating map thumbnail. One of consequences was that such infinity loops consumed all celery workers and it was not possible to complete POST/PATCH map or layer.
 - Fix empty map thumbnail. In some cases, map thumbnail was generated as if anonymous user asks for the map. Now the thumbnail is generated as if user with writing rights asks for the map.
-- New environment variable [OAUTH2_LIFERAY_INTROSPECTION_SUB_KEY](doc/env-settings.md#oauth2_introspection_sub_key).
+- New environment variable [OAUTH2_LIFERAY_INTROSPECTION_SUB_KEY](https://github.com/LayerManager/layman/blob/v1.22.3/doc/env-settings.md#oauth2_introspection_sub_key).
 - [#599](https://github.com/LayerManager/layman/issues/599) Layman supports uploading data files with upper or mixed case extensions.
 
 ## v1.16.0

Makefile

@@ -271,6 +271,9 @@ postgresql-psql:
 postgresql-psql-test:
 	docker compose -f docker-compose.deps.yml run -e PGPASSWORD=docker --entrypoint "psql -U docker -p 5432 -h postgresql layman_test" --rm postgresql
 
+postgresql-bash-exec:
+	docker compose -f docker-compose.deps.yml exec -e PGPASSWORD=docker postgresql "bash"
+
 redis-cli-db:
 	docker compose -f docker-compose.deps.yml exec redis redis-cli -h redis -p 6379 -n 0
 

doc/env-settings.md

@@ -61,7 +61,7 @@ List of dotted paths to Python modules to be used for authentication. Paths are
 Secret name of HTTP header used for authentication internally (e.g. when generating private map thumbnail). Only combination of lowercase characters and numbers must be used for the value. At demo configuration, the HTTP header is automatically removed by Nginx on every request to Layman REST API or to GeoServer WMS/WFS/OWS.
 
 ### OAUTH2_CLIENT_ID
-Client ID of Layman's Test Client registered at OAuth2 provider (e.g. Wagtail or Liferay).
+Client ID of Layman's Test Client registered at OAuth2 provider (e.g. Wagtail).
 
 ### OAUTH2_CLIENT<n>_ID
 Client ID of another Layman's client registered at OAuth2 provider. The **n** must be integer starting from `1`. In case of more clients other than LTC, list of **n**s must be uninterrupted series of integers.
@@ -84,15 +84,9 @@ URL of LTC OAuth2 callback endpoint to be called after successful OAuth2 authori
 ### OAUTH2_INTROSPECTION_URL
 URL of OAuth2 Introspection endpoint.
 
-### OAUTH2_INTROSPECTION_SUB_KEY
-Name of the key in OAuth2 introspection response whose value is OAuth2 subject (also known as "sub"). Value `username` is suitable for Wagtail. If not set or set to empty string, `sub` is used, that is suitable option for Liferay.
-
 ### OAUTH2_USER_PROFILE_URL
 URL of User Profile endpoint used to obtain user's ID, name, email, etc.
 
-### OAUTH2_SCOPE
-Comma-separated list of requested OAuth2 scopes. Do not set this variable at all (not even to empty string) if you don't want to request scope; this is suitable option for Wagtail. Value `liferay-json-web-services.everything.read.userprofile` is suitable for Liferay.
-
 ### GRANT_CREATE_PUBLIC_WORKSPACE
 List of [users](models.md#user) and [roles](models.md#role) giving them permission to create new [public workspace](models.md#public-workspace). It must be subset of (or equal to) GRANT_PUBLISH_IN_PUBLIC_WORKSPACE.
 

doc/oauth2/index.md

@@ -32,9 +32,8 @@ From [RFC6749](https://tools.ietf.org/html/rfc6749#section-1.1):
 Layman acts as *resource server*. On every request to REST API, Layman accepts OAuth2 [access token](https://tools.ietf.org/html/rfc6749#section-1.4) from a *client* and validates access token against *authorization server* to authenticate *resource owner* (i.e. end-user). The access token is validated token against *authorization server* by OAuth2 mechanism called [Token Introspection](https://oauth.net/2/token-introspection/) (RFC 7662). Furthermore, Layman is responsible for fetching user-related metadata from *authorization server* using provider-specific endpoint.
 
 ### Authorization Server
-There are currently two supported *authorization servers* (OAuth2 providers):
+There is currently one supported *authorization server* (OAuth2 provider):
 - [Django OAuth2 Toolkit](https://django-oauth-toolkit.readthedocs.io/en/latest/) as plugin of [Wagtail CRX](https://docs.coderedcorp.com/wagtail-crx/), this is preferred option
-- [Liferay Portal](https://help.liferay.com/hc/en-us/articles/360018176491-OAuth-2-0)
 
 Supporting [other OAuth2 providers](https://en.wikipedia.org/wiki/List_of_OAuth_providers) (e.g. Google or Facebook) should be quite straightforward in the future.
 
@@ -130,26 +129,3 @@ Check following environment variables of LTC:
 - OAUTH2_TOKEN_URL: URL of [Token Endpoint](https://tools.ietf.org/html/rfc6749#section-3.2). In case of Django OAuth Toolkit (Wagtail), it's something like `<http or https>://<wagtail domain and port>/o/token`
 - OAUTH2_CALLBACK_URL: URL of [Redirection Endpoint](https://tools.ietf.org/html/rfc6749#section-3.1.2), the value is `<http or https>://<LTC domain, port, and path prefix>/auth/oauth2-provider/callback`.
 - OAUTH2_USER_PROFILE_URL: URL of Layman's [GET Current User](../rest.md#get-current-user)
-
-### Liferay Settings
-Every *client* must be registered in Liferay as *application*, as described in [Liferay documentation](https://help.liferay.com/hc/en-us/articles/360018176491-OAuth-2-0#creating-an-application). For LTC, fill in following settings:
-- **Website URL** should point to application's home page, e.g. `http://localhost:3000/`.
-- **Callback URIs** must contain URL of OAuth2 [Redirection Endpoint](https://tools.ietf.org/html/rfc6749#section-3.1.2). In case of LTC, the value is the same as LTC setting OAUTH2_CALLBACK_URL.
-- **Client Profile**: Web Application
-- **Allowed Authorization Types**:
-    - Authorization Code
-    - Refresh Token
-- **Supported Features**:
-    - Token Introspection
-
-Furthermore, check "read your personal user data" (liferay-json-web-services.everything.read.userprofile) in **Scopes** tab. This scope will enable `/api/jsonws/user/get-current-user` endpoint to provide user-related metadata to Layman.
-
-By default, only Liferay users with Administrator role have enough privileges to use OAuth2 authorization. To enable other roles to use OAuth2 (e.g. User role), you need to
-- add **View** permission for **Authorize Screen URL** to desired roles
-  - **Authorize Screen URL** can be found in *Configuration > System Settings > OAuth 2 > Authorize Screen*
-  - to open permissions of default Authorize Screen URL `/?p_p_id=com_liferay_oauth2_provider_web_internal_portlet_OAuth2AuthorizePortlet&p_p_state=maximized`, visit [this URL](http://localhost:8080/?p_p_id=com_liferay_portlet_configuration_web_portlet_PortletConfigurationPortlet&p_p_state=pop_up&_com_liferay_portlet_configuration_web_portlet_PortletConfigurationPortlet_mvcPath=%2Fedit_permissions.jsp&_com_liferay_portlet_configuration_web_portlet_PortletConfigurationPortlet_portletConfiguration=true&_com_liferay_portlet_configuration_web_portlet_PortletConfigurationPortlet_portletResource=com_liferay_oauth2_provider_web_internal_portlet_OAuth2AuthorizePortlet&_com_liferay_portlet_configuration_web_portlet_PortletConfigurationPortlet_resourcePrimKey=com_liferay_oauth2_provider_web_internal_portlet_OAuth2AuthorizePortlet)
-    - see [Workaround #1](https://issues.liferay.com/browse/OAUTH2-202) for details
-- add **View** and **Create token** permissions for each registered OAuth2 application to desired roles
-  - to open permissions, visit *Configuration > OAuth 2 Administration*, click on three dots for desired application and select *Permissions*
-
-After registration, add **Client ID** and **Client Secret** pair to Layman's setting OAUTH2_CLIENTS.

src/layman/authn/oauth2/__init__.py

@@ -72,6 +72,15 @@ def authenticate():
                 # current_app.logger.info(f"r_json={r_json}")
                 if r_json['active'] is True and r_json.get('token_type', 'Bearer') == 'Bearer':
                     valid_resp = r_json
+                    if settings.OAUTH2_INTROSPECTION_USE_SUB_KEY_FROM_USER_PROFILE:
+                        assert USER_PROFILE_URL is not None, f"USER_PROFILE_URL is None"
+                        response = requests.get(USER_PROFILE_URL, headers={
+                            'Authorization': f'Bearer {access_token}',
+                        }, timeout=settings.DEFAULT_CONNECTION_TIMEOUT)
+                        response.raise_for_status()
+                        user_profile_json = response.json()
+                        # current_app.logger.info(f"user_profile_json={user_profile_json}")
+                        valid_resp[INTROSPECTION_SUB_KEY] = user_profile_json[INTROSPECTION_SUB_KEY]
                     break
             except ValueError:
                 continue

src/layman/authn/oauth2_test.py

@@ -150,7 +150,7 @@ def test_unexisting_introspection_url(client, headers):
         f'{TOKEN_HEADER}': 'Bearer abc',
     }
 ])
-@pytest.mark.usefixtures('app_context', 'inactive_token_introspection_url', 'ensure_layman')
+@pytest.mark.usefixtures('app_context', 'inactive_token_introspection_url', 'user_profile_url')
 def test_token_inactive(client, headers):
     username = 'testuser1'
     url = url_for('rest_workspace_layers.get', workspace=username)
@@ -166,7 +166,7 @@ def test_token_inactive(client, headers):
         f'{TOKEN_HEADER}': 'Bearer abc',
     }
 ])
-@pytest.mark.usefixtures('app_context', 'active_token_introspection_url', 'ensure_layman')
+@pytest.mark.usefixtures('app_context', 'active_token_introspection_url', 'user_profile_url')
 def test_token_active(client, headers):
     username = 'testuser1'
     url = url_for('rest_workspace_layers.get', workspace=username)
@@ -176,7 +176,7 @@ def test_token_active(client, headers):
     assert resp_json['code'] == 40
 
 
-@pytest.mark.usefixtures('app_context', 'active_token_introspection_url', 'user_profile_url', 'ensure_layman')
+@pytest.mark.usefixtures('app_context', 'active_token_introspection_url', 'user_profile_url')
 def test_authn_get_current_user_without_username(client):
     rest_path = url_for('rest_current_user.get')
     response = client.get(rest_path, headers={

src/layman_settings.py

@@ -214,8 +214,9 @@ class EnumWfsWmsStatus(Enum):
     if len(u) > 0
 ]
 OAUTH2_INTROSPECTION_URL = os.getenv('OAUTH2_INTROSPECTION_URL', None)
-OAUTH2_INTROSPECTION_SUB_KEY = os.getenv('OAUTH2_INTROSPECTION_SUB_KEY') or 'sub'
+OAUTH2_INTROSPECTION_SUB_KEY = 'userId'
 OAUTH2_USER_PROFILE_URL = os.getenv('OAUTH2_USER_PROFILE_URL', None)
+OAUTH2_INTROSPECTION_USE_SUB_KEY_FROM_USER_PROFILE = True
 OAUTH2_CLIENTS = [
     d for d in read_clients_dict_from_env()
     if len(d['id']) > 0

src/v1_23_change_oauth2_sub_username_to_user_id.py

@@ -0,0 +1,54 @@
+import os
+import layman_settings as settings
+from db import util as db_util
+
+
+def main():
+    assert settings.OAUTH2_INTROSPECTION_SUB_KEY == 'userId', f"OAUTH2_INTROSPECTION_SUB_KEY is expected to be `userId`"
+    assert settings.OAUTH2_INTROSPECTION_USE_SUB_KEY_FROM_USER_PROFILE is True, f"OAUTH2_INTROSPECTION_USE_SUB_KEY_FROM_USER_PROFILE is expected to be `true`"
+
+    wagtail_db_uri = os.getenv('LAYMAN_WAGTAIL_DB_URI', None)
+    assert wagtail_db_uri is not None, f"LAYMAN_WAGTAIL_DB_URI must be set"
+
+    wagtail_user_rows = db_util.run_query(f"select id, username from auth_user;", uri_str=wagtail_db_uri)
+    assert len(set(r[0] for r in wagtail_user_rows)) == len(wagtail_user_rows), f"Wagtail userIds are expected to be unique"
+    assert len(set(r[1] for r in wagtail_user_rows)) == len(wagtail_user_rows), f"Wagtail usernames are expected to be unique"
+    wagtail_username_to_id = {
+        username: f"{user_id}"
+        for user_id, username in wagtail_user_rows
+    }
+
+    layman_user_rows = db_util.run_query(f"""
+select u.id, u.issuer_id, u.sub, w.name as username
+from {settings.LAYMAN_PRIME_SCHEMA}.users u
+  inner join {settings.LAYMAN_PRIME_SCHEMA}.workspaces w on u.id_workspace = w.id
+    """, uri_str=settings.PG_URI_STR)
+
+    print(f"Found {len(wagtail_user_rows)} Wagtail users:")
+    for user_id, username in wagtail_user_rows:
+        print(f"  {username}, id={user_id}")
+
+    print(f"Found {len(layman_user_rows)} Layman users with username registered.")
+    print(f'Processing Layman users ...')
+
+    changed_subs = 0
+    for layman_user_id, issuer_id, sub, username in layman_user_rows:
+        print(f"  {username} (id={layman_user_id}, sub={sub}, issuer_id={issuer_id})")
+        if issuer_id != 'layman.authn.oauth2':
+            print(f"    WARNING: User has unexpected issuer_id, skipping.")
+            continue
+        new_sub = wagtail_username_to_id.get(sub)
+        if new_sub is None:
+            print(f"    WARNING: Sub of the user was not found among Wagtail usernames, skipping.")
+            continue
+        print(f'    Changing sub from `{sub}` to `{new_sub}`')
+        db_util.run_statement(
+            f"UPDATE {settings.LAYMAN_PRIME_SCHEMA}.users set sub = %s where id = %s",
+            data=(new_sub, layman_user_id), uri_str=settings.PG_URI_STR)
+        changed_subs += 1
+        print(f'      Changed!')
+    print(f'Processing finished, changed {changed_subs} OAuth2 subs of {len(layman_user_rows)} Layman users.')
+
+
+if __name__ == '__main__':
+    main()