[ENH] - Design import/export connector for Keycloak API #2519
Assignees
Labels
area:backup-and-restore 💾
All items related to backup and restore
area:developer-experience 👩🏻💻
area: integration/keycloak
type: enhancement 💅🏼
New feature or request
Milestone
Comments
Proposed Design:Nebari uses Keycloak to manage users and groups. Keycloak also manages clients for services like Jupyterhub, argo-server-sso, conda_store, etc. Nebari creates These clients automatically, so we don't have to expose them. The Nebari API will consolidate responses from the internal Keycloak API and return a consolidated JSON response.
With this context, to accurately re-create the Nebari Keycloak, we need the following endpoints: Endpoints for backup
|
|
That's a nice description of all the concepts thanks, @pt247, Have you started working on this yet? I am considering scheduling a meeting for this as well |
viniciusdc
added
type: enhancement 💅🏼
|
Thanks @viniciusdc
No, I have not started. Meeting only foucsing on Keycloak backup and restore would help. |
pt247
changed the title
|
interesting link: |
|
@tylergraff I assume you have some experience in backup and restore. I was wondering how you migrate credentials? |
|
@kalpanachinnappan We are starting work on backup and restore. Keycloak is the first component we are considering. We would like your input on this as well. |
|
As this solution may rely on the Keycloak API, @viniciusdc suggested we might want to compare the current Keycloak API with the latest version of Keycloak. |
|
yes, mainly due to more flexibility of the exporting/importing features that the newer Keycloak versions allow (it might help us handle the credentials issue) |
|
Just as a refresher here, to recap what has been discussed so far, and use this as a foundation for the overall architecture and development strategies for other services later on: Goals of the Backup and Restore
ComponentsThe major endpoints have been outlined above in previous comments, but the general overview of the data we will be looking for so far will be:
This will be made customizable so that this list can be extended, in a modular way. Overall structuregraph TD;
subgraph Keycloak
EM([Export Mechanism])
IM([Import Mechanism])
end
BnR[Backup & Restore]
B([Backup])
R([Restore])
BnR --> B
BnR --> R
B --> EM
R --> IM
For the implementation details:
Now some details that might be interesting to have in the perspective of the BARe service:
|
Action items
|
Quick updates:
Action items from this:
|
import json
from functools import cached_property
import psycopg
from psycopg.rows import dict_row
class KeycloakPasswordFetcher:
@cached_property
def pg_connection(self):
"""Establish a direct connection to the Keycloak PostgreSQL database."""
return psycopg.connect(
f"dbname={KEYCLOAK_DATABASE_NAME} "
f"user={KEYCLOAK_DATABASE_USER} "
f"password={KEYCLOAK_DATABASE_PASSWORD} "
f"host={KEYCLOAK_DATABASE_HOST} "
f"port={KEYCLOAK_DATABASE_PORT}",
row_factory=dict_row,
)
def get_keycloak_password(self, user_id):
"""Retrieve secret data from the Keycloak PostgreSQL database (not accessible via REST API)."""
query = """
SELECT secret_data, credential_data, created_date
FROM credential
WHERE user_id=%(user_id)s
ORDER BY created_date DESC;
"""
with self.pg_connection.cursor() as cur:
cur.execute(query, {"user_id": user_id})
row = cur.fetchone()
if row:
secret_data = json.loads(row["secret_data"])
credential_data = json.loads(row["credential_data"])
return {
"algorithm": credential_data["algorithm"],
"iterations": credential_data["hashIterations"],
"salt": secret_data["salt"],
"hash": secret_data["value"],
}
raise ValueError(f"No credential found for user_id {user_id}")
fetcher = KeycloakPasswordFetcher()
user_id = "<UUID>"
password = "{algorithm}${iterations}${salt}${hash}".format(
**fetcher.get_keycloak_password(user_id=user_id),
) |
6 tasks
viniciusdc
changed the title
|
Based on what have been discussed above this is the general layout of how the keycloak import/export connectors will look like: import os
import json
import requests
import requests_cache
from typing import List, Dict, Any
from pydantic import BaseModel, Field, Optional
class RoleSchema(BaseModel):
id: Optional[str] = None
name: str
description: Optional[str] = None
composite: bool = False
clientRole: bool = Field(False, alias="clientRole")
containerId: Optional[str] = Field(None, alias="containerId")
class Config:
populate_by_name = True
class KeycloakSkeleton(BaseModel):
roles: List[RoleSchema] = Field(default_factory=list, depends_on=["clients"])
class KeycloakAuth(BaseModel):
url: str = Field(os.getenv("KEYCLOAK_URL"), alias="auth_url")
realm: str = Field(os.getenv("KEYCLOAK_REALM"), alias="realm")
client_id: str = Field(os.getenv("KEYCLOAK_CLIENT_ID"), alias="client_id")
client_secret: str = Field(os.getenv("KEYCLOAK_CLIENT_SECRET"), alias="client_secret")
class Config:
populate_name = True
class KeycloakAPIClient:
def __init__(self, auth: Dict[str, str]):
self.auth = self._validate_auth(auth)
self.token = None
def _validate_auth(self, auth) -> None:
required_keys = ["url", "realm", "client_id", "client_secret"]
missing_keys = [key for key in required_keys if key not in auth]
if missing_keys:
raise ValueError(f"Missing required authentication parameters: {missing_keys}")
return auth
def _authenticate(self) -> None:
if self.token and self._is_token_valid():
return
response = requests.post(
url=f"{self.auth['url']}/realms/{self.auth['realm']}/protocol/openid-connect/token",
data={
"client_id": self.auth["client_id"],
"client_secret": self.auth["client_secret"],
"grant_type": "client_credentials",
},
headers={"Content-Type": "application/x-www-form-urlencoded"},
)
response.raise_for_status()
self.token = response.json()["access_token"]
def _is_token_valid(self) -> bool:
introspection_response = requests.post(
url=f"{self.auth['url']}/realms/{self.auth['realm']}/protocol/openid-connect/token/introspect",
data={
"client_id": self.auth["client_id"],
"client_secret": self.auth["client_secret"],
"token": self.token,
},
headers={"Content-Type": "application/x-www-form-urlencoded"},
)
introspection_response.raise_for_status()
return introspection_response.json().get("active", False)
def get(self, endpoint: str) -> List[Dict[str, Any]]:
self._authenticate()
response = requests.get(
url=f"{self.auth['url']}{endpoint}",
headers={"Authorization": f"Bearer {self.token}"},
)
response.raise_for_status()
return response.json()
def post(self, endpoint: str, json: Dict[str, Any]) -> None:
self._authenticate()
response = requests.post(
url=f"{self.auth['url']}{endpoint}",
json=json,
headers={"Authorization": f"Bearer {self.token}"},
)
response.raise_for_status()
class KeycloakExport:
def __init__(self, api_client: KeycloakAPIClient, state: KeycloakSkeleton):
self.api_client = api_client
self.state = state
def _export_roles(self) -> List:
print("Exporting role data from Keycloak...")
data = self.api_client.get(f"/auth/admin/realms/{self.state.realm}/roles")
roles = [RoleSchema(**item).dict() for item in data]
self._save_to_file(roles)
return roles
def _save_to_file(self, roles: List[Dict[str, Any]]) -> None:
os.makedirs('keycloak', exist_ok=True)
with open('keycloak/roles.json', 'w') as file:
json.dump(roles, file, indent=4)
print("Roles have been saved to 'keycloak/roles.json'.")
class KeycloakImport:
def __init__(self, api_client: KeycloakAPIClient, state: KeycloakSkeleton):
self.api_client = api_client
self.state = state
def _import_roles(self) -> None:
print("Importing role data into Keycloak...")
roles = self._load_from_file()
roles_schema = [RoleSchema(**item) for item in roles]
for role in roles_schema:
self.api_client.post(
f"/auth/admin/realms/{self.state.realm}/roles", json=role.dict()
)
print("Roles have been imported successfully.")
def _load_from_file(self) -> List[Dict[str, Any]]:
with open('keycloak/roles.json', 'r') as file:
roles = json.load(file)
return roles
class Keycloak:
"""
Main service class for interacting with Keycloak's API, exposing data export and import.
"""
state = KeycloakSkeleton()
def __init__(self, auth: Dict[str, str] = {}):
self.api_client = KeycloakAPIClient(KeycloakAuth(**auth).dict())
self.keycloak_export = KeycloakExport(self.api_client, self.state)
self.keycloak_import = KeycloakImport(self.api_client, self.state)
# Enable caching for the requests library
requests_cache.install_cache("keycloak_cache", expire_after=300)
# Dynamically expose methods from KeycloakExport and KeycloakImport
self._expose_methods(self.keycloak_export, "export")
self._expose_methods(self.keycloak_import, "import")
def _expose_methods(self, obj, prefix):
for method_name in dir(obj):
if callable(getattr(obj, method_name)) and not method_name.startswith("__"):
setattr(self, f"{prefix}_{method_name}", getattr(obj, method_name))
# Example usage
if __name__ == "__main__":
auth_params = {
"url": "http://localhost:8080",
"realm": "myrealm",
"client_id": "myclient",
"client_secret": "mysecret",
}
keycloak = Keycloak(auth=auth_params)
# Export roles
exported_roles = keycloak.export_roles()
print("Exported Roles:", exported_roles)
# Import roles
keycloak.import_roles()
print("Roles Imported Successfully")This is a miniature version of the general structure so that it can be executed and tested; the complete code has all the required resources (Users, Roles, ...) and its reconciliation method to handle duplicated entries and the inner dependencies between the users/groups data. The main structure for each serializable service follows the same standard as below. In this example, we outlined the backup path, though restoration will rely on the same objects: More information about the general responsibilities of the components outlined in the diagram above can be found in the #2650 For future ref. While we talked about each route and their endpoint at the beginning of this issue, a quick summary to outline what the "Export Mechanism" and "Import Mechanism" actually do is required: The "Export Mechanism" ("Import Mechanism") is a gateway for the targeted components' endpoints available in keycloaks that require exporting (importing). Meanwhile, storage is managed through a higher process of the backup-restore application. Keycloak export sequenceClick to expandsequenceDiagram
participant KeycloakServer
participant ExportProcess
participant ExportData
par Export Roles
ExportProcess->>KeycloakServer: Get Roles (ROLE)
KeycloakServer-->>ExportProcess: Roles Data
ExportProcess->>ExportData: Store Roles Data
and Export Groups
ExportProcess->>KeycloakServer: Get Groups (GROUP)
KeycloakServer-->>ExportProcess: Groups Data
ExportProcess->>ExportData: Store Groups Data
and Export Providers
ExportProcess->>KeycloakServer: Get IDPS (PROVIDERS)
KeycloakServer-->>ExportProcess: Providers Data
ExportProcess->>ExportData: Store Providers Data
and Export Users
ExportProcess->>KeycloakServer: Get Users (USER)
KeycloakServer-->>ExportProcess: Users Data
ExportProcess->>ExportData: Store Users Data
and Export Clients
ExportProcess->>KeycloakServer: Get Clients (CLIENT)
KeycloakServer-->>ExportProcess: Clients Data
ExportProcess->>ExportData: Store Clients Data
end
Keycloak import sequenceClick to expandsequenceDiagram
participant StoredData
participant ImportProcess
participant KeycloakServer
StoredData->>ImportProcess: Start Import Process
ImportProcess->>KeycloakServer: Create/Update Groups (GROUP)
KeycloakServer-->>ImportProcess: Response
ImportProcess->>KeycloakServer: Create/Update Users (USER)
KeycloakServer-->>ImportProcess: Response
ImportProcess->>KeycloakServer: Assign Users to Groups (USER_GROUP)
KeycloakServer-->>ImportProcess: Response
ImportProcess->>KeycloakServer: Create/Update Clients (CLIENT)
KeycloakServer-->>ImportProcess: Response
ImportProcess->>KeycloakServer: Assign Roles to Clients (CLIENT_ROLE)
KeycloakServer-->>ImportProcess: Response
ImportProcess->>KeycloakServer: Get Roles for Clients
KeycloakServer-->>ImportProcess: Response
ImportProcess->>KeycloakServer: Create Missing Roles (ROLE)
KeycloakServer-->>ImportProcess: Response
ImportProcess->>KeycloakServer: Configure Identity Providers (IDENTITY_PROVIDER)
KeycloakServer-->>ImportProcess: Response
|
viniciusdc
removed
the
status: in progress 🏗
|
This was moved to the implementation phase and now works as expected. #2657 will handle the required modifications to make it available on nebari |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature description
Backup and Restore RFD
Value and/or benefit
Nebari Admin can access Keycloak API and query user data.
We need to create a backup controller to expose the existing Keycloak API to authenticated users.
This can be a REST API using FastAPI.
/api/v1/keycloak: Expose existing Keycload Admin REST API - docsSo for example:
GET
/api/v1/keycloak/admin/realms/nebari/usersshould get us a list of all the non-admin users.Anything else?
Related META issue- #2518
The text was updated successfully, but these errors were encountered: