diff --git a/README.md b/README.md index 5cd26bb..2316953 100644 --- a/README.md +++ b/README.md @@ -8,239 +8,21 @@ The extension includes features such as chart creation, chart editing, chart emb With ckanext-charts, users can easily generate interactive and visually appealing charts to enhance data analysis and presentation in CKAN. -This extension offers two plugins: `charts_view` and `charts_builder_view`. The first plugin is a standard chart builder designed for system administrators or anyone with permission to edit or create a resource view. Creating a preview using this plugin generates a static chart with saved settings. This chart retains the settings and will always be rendered in the same way. +## Quick start -![User chart builder](doc/chart-builder.gif) - -The charts_builder_view plugin does not have any special settings during the creation or editing stages. Instead, it renders a chart builder JS widget on a resource page, allowing any user to interact with the resource data and create a chart that meets their needs. In the future, we plan to implement a feature that will allow users to save their charts. - -![User chart builder](doc/chart-user-builder.gif) - -The user chart builder could be restricted to only authenticated users. See the config settings section for more information. In this case, user will be forced to log in to build a chart. After the user logs in, user will be redirected to the chart builder page back. - -![Restrict user chart builder for anon](doc/anon-restrict.png) - -## Requirements - -Requires Redis 7+ - -Compatibility with core CKAN versions: - -| CKAN version | Compatible? | -| --------------- | ------------- | -| 2.9 and earlier | no | -| 2.10+ | yes | - - -## Installation - -- Install it with `PyPi` with `pip install ckanext-charts` +- Install it with `PyPi` with `pip install ckanext-charts[pyarrow]` - Add `charts_view` to the list of plugins in your CKAN config (`ckan.plugins = charts_view charts_builder_view`) -## Config settings - -List of config options: - - # Caching strategy for chart data (required, default: redis). - # Available options: redis, file_orc, file_csv - ckanext.charts.cache_strategy = file_csv - - # Time to live for the Redis cache in seconds. Set 0 to disable cache (default: 3600) - ckanext.charts.redis_cache_ttl = 7200 - - # Time to live for the File cache in seconds. Set 0 to disable cache. - ckanext.charts.file_cache_ttl = 0 - - # Enable cache for the charts (default: true) - ckanext.charts.enable_cache = true - - # Include HTMX assets. Could be disabled if another plugins are including it (default: true) - ckanext.charts.include_htmx_asset - - # Reinit CKAN JS modules after HTMX swap (default: false) - ckanext.charts.reinit_ckan_js_modules - - # Allow anonymous users to build charts (default: false) - ckanext.charts.allow_anon_building_charts - - -## Admin config page - -The extension provides an admin configuration page where you can set all the listed configuration options. The admin page available only -using an extension [`ckanext-admin-panel`](https://github.com/DataShades/ckanext-admin-panel). The admin panel is a separate extension that provides an alternative admin interface for CKAN. It allows you to manage CKAN settings and other extensions settings through the web interface. - -![Admin config page](doc/admin-page.jpg) - -This page could be accessed by the following URL: `/admin-panel/charts/config`, but it will be registered only if the `ckanext-admin-panel` is installed and enabled. - -## Cache - -The extension implement a cache strategy to store the data fetched from the different sources. There are three cache strategies available: `redis`, `file_orc` and `file_csv`. The file cache works by storing the data in an `orc` or `csv` file in the filesystem. The redis cache stores the data in a Redis database. The cache strategy can be changed at the CKAN configuration level through the admin interface or in a configuration file. - -The cache TTL can be set in the CKAN configuration file. The default value is 3600 seconds (1 hour). The cache TTL can be set to 0 to disable the cache. - -The `redis` and `file-type` cache has separate TTL settings. The `redis` cache TTL can be set with the `ckanext.charts.redis_cache_ttl` configuration option. The `file` cache TTL can be set with the `ckanext.charts.file_cache_ttl` configuration option. - -You need to have a Redis server running to use the `redis` cache strategy. - -The `file-type` cache strategy stores the data in a file in the filesystem. The file cache is stored in the `ckanext-charts` directory in the CKAN storage path. The file cache is stored in an `orc` or `csv` file format. - -Cache could be disabled by setting `ckanext.charts.enable_cache` to `false`. In this case the data will be fetched from the source every time the chart is rendered. It could be useful for debugging purposes. But using it in production is not recommended, as it could lead to performance issues. - -### File ORC cache strategy - -This strategy requires `pyarrow` python library to be installed. - -## Implementing new fetchers - -Fetchers are responsible for fetching data from different sources (DataStore, URL, file system, hardcoded data). -To register new fetchers, you need to create a new class that inherits from `DataFetcherStrategy` and implements the `fetch_data` and `make_cache_key` methods. -The `fetch_data` method should return a `pandas` `DataFrame` object with the data that should be displayed in the chart. -The `make_cache_key` method should return a unique string that will be used as a key to store the data in the cache. - -## Implementing new chart engines support - -Implementing support for a new chart engine includes multiple steps and changes in Python, HTML, and JavaScript. Starting from the Python code: - -1. Create a new builder class at `ckanext.charts.chart_builder` that inherits from `BaseChartBuilder` and implements the `get_supported_forms` method. This method must return a list of classes that represent supported chart types forms. - -2. Each form type builder must be connected with a respective chart type builder. - -3. The chart type builder must implement a `to_json` method that will return a dumped JSON data, which will be passed to a JS script. - -4. The form type builder must implement a `get_form_fields` method that will return a list of all form fields that will be rendered for the user, allowing them to provide all the necessary information for a chart. - -5. Register your chart engine by adding the builder class to `get_chart_engines` in `ckanext.charts.chart_builder.__init__.py`. - -A full example of an implementation of `bar` chart for `obvervable plot` library. - -```py -from __future__ import annotations - -import json -from typing import Any - -import ckanext.charts.exception as exception -from ckanext.charts.chart_builders.base import BaseChartBuilder, BaseChartForm - - -class ObservableBuilder(BaseChartBuilder): - @classmethod - def get_supported_forms(cls) -> list[type[Any]]: - return [ObservableBarForm] - - -class ObservableBarBuilder(ObservableBuilder): - def to_json(self) -> str: - return json.dumps( - { - "type": "bar", - "data": self.df.to_dict(orient="records"), - "settings": self.settings, - } - ) - - -class ObservableBarForm(BaseChartForm): - name = "Bar" - builder = ObservableBarBuilder - - def fill_field(self, choices: list[dict[str, str]]) -> dict[str, str]: - field = self.color_field(choices) - field.update({"field_name": "fill", "label": "Fill"}) - - return field - - def get_form_fields(self): - columns = [{"value": col, "label": col} for col in self.df.columns] - chart_types = [ - {"value": form.name, "label": form.name} - for form in self.builder.get_supported_forms() - ] - - return [ - self.title_field(), - self.description_field(), - self.engine_field(), - self.type_field(chart_types), - self.x_axis_field(columns), - self.y_axis_field(columns), - self.fill_field(columns), - self.opacity_field(), - self.limit_field(), - ] -``` - -Another step is to register JS/CSS vendor libraries of the chart you want to use. Refer to [CKAN documentation](https://docs.ckan.org/en/latest/theming/webassets.html) to read about adding CSS and JavaScript files using Webassets. - -You also will need a CKAN JS module, that will be responsible for rendering the Chart. This module must be registered inside a `webassets.yml` as well. -```js - ckan.module("charts-render-observable", function ($, _) { - "use strict"; - - return { - options: { - config: null - }, - - initialize: function () { - $.proxyAll(this, /_/); - - if (!this.options.config) { - console.error("No configuration provided"); - return; - } - - var plot; - - switch (this.options.config.type) { - case "bar": - plot = Plot.barY(this.options.config.data, this.options.config.settings).plot(); - break; - default: - return; - } - - this.el[0].replaceChildren(plot); - } - }; - }); -``` - -And an HTML file, that will provide a proper container and include your JS module with `data-module`. - -```html - {% asset "charts/observable" %} - - {% if chart %} -
- {% else %} -- {{ _("Cannot build chart with current settings") }} -
- {% endif %} -``` - -Note, that we should add `{% asset "charts/observable" %}` not only here, but in `charts_form.html` too. - -The reason for having a separate `HTML` file and `JS` module is that different libraries may require different types of container elements (such as div, canvas, etc.) to initialize or may need additional boilerplate code to build a chart. There's no easy way to abstract this, so you have to implement these things yourself. ## Developer installation -To install ckanext-charts for development, activate your CKAN virtualenv and +To install `ckanext-charts` for development, activate your CKAN virtualenv and do: git clone https://github.com/DataShades/ckanext-charts.git cd ckanext-charts - python setup.py develop - pip install -r dev-requirements.txt - -## Troubleshooting - -**ImportError: lxml.html.clean module is now a separate project lxml_html_clean** - -Install `lxml[html_clean]` or `lxml_html_clean` directly using pip. + pip install -e '.[dev]' ## Tests diff --git a/ckanext/charts/cache.py b/ckanext/charts/cache.py index a369117..a423da7 100644 --- a/ckanext/charts/cache.py +++ b/ckanext/charts/cache.py @@ -21,29 +21,58 @@ class CacheStrategy(ABC): - """Cache strategy interface""" + """Cache strategy interface. + + Defines the abstracts methods for cache strategies. + """ @abstractmethod def get_data(self, key: str) -> pd.DataFrame | None: + """Return data from cache if exists. + + Args: + key: The cache key to retrieve the data. + + Returns: + The data if exists, otherwise None. + """ pass @abstractmethod def set_data(self, key: str, data: pd.DataFrame): + """Store data to cache. + + Args: + key: The cache key to store the data. + data: The data to be stored. + """ pass @abstractmethod def invalidate(self, key: str) -> None: + """Invalidate cache by key. + + Args: + key: The cache key to invalidate. + """ pass class RedisCache(CacheStrategy): - """Cache data to Redis""" + """Cache data to Redis as a CSV string""" def __init__(self): self.client = connect_to_redis() def get_data(self, key: str) -> pd.DataFrame | None: - """Return data from cache if exists""" + """Return data from cache if exists. + + Args: + key: The cache key to retrieve the data. + + Returns: + The data if exists, otherwise None. + """ raw_data = self.client.get(key) if not raw_data: @@ -52,7 +81,15 @@ def get_data(self, key: str) -> pd.DataFrame | None: return pd.read_csv(BytesIO(raw_data)) # type: ignore def set_data(self, key: str, data: pd.DataFrame): - """Serialize data and save to redis""" + """Serialize data and save to Redis. + + Args: + key: The cache key to store the data. + data: The data to be stored. + + Raises: + Exception: If failed to save data to Redis. + """ cache_ttl = config.get_redis_cache_ttl() try: @@ -64,11 +101,19 @@ def set_data(self, key: str, data: pd.DataFrame): log.exception("Failed to save data to Redis") def invalidate(self, key: str): + """Remove data from cache. + + Args: + key: The cache key to invalidate. + """ self.client.delete(key) class FileCache(CacheStrategy): - """Cache data as file""" + """Cache data as file. + + We store the cached files in a separate folder in the CKAN storage. + """ FILE_FORMAT = "" @@ -76,7 +121,14 @@ def __init__(self): self.directory = get_file_cache_path() def get_data(self, key: str) -> pd.DataFrame | None: - """Return data from cache if exists""" + """Return data from cache if exists. + + Args: + key: The cache key to retrieve the data. + + Returns: + The data if exists, otherwise None. + """ file_path = self.make_file_path_from_key(key) @@ -91,40 +143,87 @@ def get_data(self, key: str) -> pd.DataFrame | None: @abstractmethod def read_data(self, file: IO) -> pd.DataFrame | None: + """Read cached data from a file object. + + Args: + file: The file object to read the data. + + Returns: + The data if exists, otherwise None. + """ pass def set_data(self, key: str, data: pd.DataFrame) -> None: - """Save data to cache. The data will be stored as an ORC file.""" + """Store data to cache. + + Args: + key: The cache key to store the data. + data: The data to be stored. + """ file_path = self.make_file_path_from_key(key) self.write_data(file_path, data) @abstractmethod def write_data(self, file_path: str, data: pd.DataFrame) -> None: + """Defines how to write data to a file. + + Args: + file_path: The path to the file. + data: The data to be stored. + """ pass def invalidate(self, key: str) -> None: - """Remove data from cache""" + """Remove data from cache. + + Args: + key: The cache key to invalidate. + """ file_path = self.make_file_path_from_key(key) if os.path.exists(file_path): os.remove(file_path) def make_file_path_from_key(self, key: str) -> str: + """Generate file path based on the key + + Args: + key: The cache key to generate the file path. + + Returns: + The file path. + """ return os.path.join( self.directory, f"{self.generate_unique_consistent_filename(key)}.{self.FILE_FORMAT}", ) def generate_unique_consistent_filename(self, key: str) -> str: - """Generate unique and consistent filename based on the key""" + """Generate unique and consistent filename based on the key. + + Args: + key: The cache key to generate the filename. + + Returns: + The filename. + """ hash_object = hashlib.sha256() hash_object.update(key.encode("utf-8")) return hash_object.hexdigest() @staticmethod def is_file_cache_expired(file_path: str) -> bool: - """Check if file cache is expired. If TTL is 0 then cache never expires.""" + """Check if file cache is expired. + + If TTL is 0 then cache never expires. + + Args: + file_path: The path to the file. + + Returns: + True if file cache is expired, otherwise False. + """ file_ttl = config.get_file_cache_ttl() if not file_ttl: @@ -139,11 +238,25 @@ class FileCacheORC(FileCache): FILE_FORMAT = "orc" def read_data(self, file: IO) -> pd.DataFrame | None: + """Read cached data from an ORC file. + + Args: + file: The file object to read the data. + + Returns: + The data if exists, otherwise None. + """ from pyarrow import orc return orc.ORCFile(file).read().to_pandas() def write_data(self, file_path: str, data: pd.DataFrame) -> None: + """Write data to an ORC file. + + Args: + file_path: The path to the file. + data: The data to be stored. + """ for col in data.select_dtypes(include=["object"]).columns: data[col] = data[col].astype(str) @@ -156,9 +269,23 @@ class FileCacheCSV(FileCache): FILE_FORMAT = "csv" def read_data(self, file: IO) -> pd.DataFrame | None: + """Read cached data from a CSV file. + + Args: + file: The file object to read the data. + + Returns: + The data if exists, otherwise None. + """ return pd.read_csv(file) def write_data(self, file_path: str, data: pd.DataFrame) -> None: + """Write data to a CSV file. + + Args: + file_path: The path to the file. + data: The data to be stored. + """ data.to_csv(file_path, index=False) diff --git a/ckanext/charts/chart_builders/chartjs.py b/ckanext/charts/chart_builders/chartjs.py index 37eb8a5..a853e62 100644 --- a/ckanext/charts/chart_builders/chartjs.py +++ b/ckanext/charts/chart_builders/chartjs.py @@ -398,8 +398,7 @@ def _calculate_bubble_radius(self, data_series: pd.Series, size_max: int) -> int data_series_size = np.nan_to_num(data_series[size_column], nan=0) bubble_radius = (data_series_size / size_max) * 30 - if bubble_radius < self.min_bubble_radius: - bubble_radius = self.min_bubble_radius + bubble_radius = max(bubble_radius, self.min_bubble_radius) return self.convert_to_native_types(bubble_radius) diff --git a/ckanext/charts/fetchers.py b/ckanext/charts/fetchers.py index 8ea70d5..155c14e 100644 --- a/ckanext/charts/fetchers.py +++ b/ckanext/charts/fetchers.py @@ -25,21 +25,44 @@ def __init__(self, cache_strategy: str | None = None) -> None: @abstractmethod def fetch_data(self) -> pd.DataFrame: + """This method should implement the data fetch logic. + + All the necessary information should be provided in the constructor. + + Returns: + pd.DataFrame: The fetched data + """ pass @abstractmethod def make_cache_key(self) -> str: + """This method should generate a cache key for the fetched data. + + Every data fetcher should implement this method to support caching. + + Returns: + str: The cache key + """ pass def invalidate_cache(self): + """Invalidate the cache for the data fetcher.""" self.cache.invalidate(self.make_cache_key()) def get_cached_data(self) -> pd.DataFrame | None: + """Fetch data from the cache. + + Returns: + pd.DataFrame | None: The cached data or None if not found + """ return self.cache.get_data(self.make_cache_key()) class DatastoreDataFetcher(DataFetcherStrategy): - """Fetch data from the DataStore""" + """Fetch dataset resource data from the DataStore. + + This fetcher is used to fetch data from the DataStore using the resource ID. + """ def __init__( self, @@ -47,14 +70,22 @@ def __init__( limit: int = 2000000, cache_strategy: str | None = None, ): + """Initialize the DatastoreDataFetcher. + + Args: + resource_id (str): The ID of the resource to fetch data for. + limit (int, optional): The maximum number of rows to fetch. + cache_strategy (str, optional): The cache strategy to use. If not provided, + the configured cache strategy will be used. + """ + super().__init__(cache_strategy=cache_strategy) self.resource_id = resource_id self.limit = limit def fetch_data(self) -> pd.DataFrame: - """We are working with resources, that are stored with DataStore in - a separate table. + """Fetch data from the DataStore. Returns: pd.DataFrame: Data from the DataStore @@ -97,10 +128,27 @@ def fetch_data(self) -> pd.DataFrame: return df def make_cache_key(self) -> str: + """Generate a cache key for the DataStore data fetcher. + + Uses the resource ID as the part of a cache key. + + Returns: + str: The cache key + """ return f"ckanext-charts:datastore:{self.resource_id}" class URLDataFetcher(DataFetcherStrategy): + """Fetch data from a URL. + + This fetcher is used to fetch data from a URL. + + Supported formats: + - `CSV` + - `XLSX` + - `XLS` + - `XML` + """ SUPPORTED_FORMATS = ["csv", "xlsx", "xls", "xml"] def __init__( @@ -110,6 +158,15 @@ def __init__( timeout: int = 0, cache_strategy: str | None = None, ): + """Initialize the URLDataFetcher. + + Args: + url (str): The URL to fetch data from. + file_format (str, optional): The format of the file. + timeout (int, optional): The timeout for the request in seconds. + cache_strategy (str, optional): The cache strategy to use. If not provided, + the configured cache strategy will be used. + """ super().__init__(cache_strategy=cache_strategy) self.url = url @@ -117,6 +174,11 @@ def __init__( self.timeout = timeout def fetch_data(self) -> pd.DataFrame: + """Fetch data from the URL. + + Returns: + pd.DataFrame: Data fetched from the URL + """ if config.is_cache_enabled(): cached_df = self.get_cached_data() @@ -148,10 +210,24 @@ def fetch_data(self) -> pd.DataFrame: return df def make_cache_key(self) -> str: + """Generate a cache key for the URL data fetcher. + + Uses the URL as the part of a cache key. + + Returns: + str: The cache key + """ return f"ckanext-charts:url:{self.url}" def make_request(self) -> bytes: - """Make a request to the URL and return the response text""" + """Make a request to the URL and return the response content. + + Returns: + bytes: The response content + + Raises: + DataFetchError: If an error occurs during the request + """ try: response = requests.get(self.url) response.raise_for_status() @@ -174,6 +250,16 @@ def make_request(self) -> bytes: class FileSystemDataFetcher(DataFetcherStrategy): + """Fetch data from the file system. + + This fetcher is used to fetch data from a file on the file system. + + Supported formats: + - `CSV` + - `XLSX` + - `XLS` + - `XML` + """ SUPPORTED_FORMATS = ["csv", "xlsx", "xls", "xml"] def __init__( @@ -182,13 +268,25 @@ def __init__( file_format: str = "csv", cache_strategy: str | None = None, ): + """Initialize the FileSystemDataFetcher. + + Args: + file_path (str): The path to the file. + file_format (str, optional): The format of the file. + cache_strategy (str, optional): The cache strategy to use. If not provided, + the configured cache strategy will be used. + """ super().__init__(cache_strategy=cache_strategy) self.file_path = file_path self.file_format = file_format def fetch_data(self) -> pd.DataFrame: - """Fetch data from the file system""" + """Fetch data from the file system. + + Returns: + pd.DataFrame: Data fetched from the file system + """ if config.is_cache_enabled(): cached_df = self.get_cached_data() @@ -225,15 +323,36 @@ def fetch_data(self) -> pd.DataFrame: return df def make_cache_key(self) -> str: + """Generate a cache key for the FileSystem data fetcher. + + Uses the file path as the part of a cache key. + + Returns: + str: The cache key + """ return f"ckanext-charts:url:{self.file_path}" class HardcodedDataFetcher(DataFetcherStrategy): + """Fetch hardcoded data. + + This fetcher is used to make a dataframe from hardcoded data, so you can + build a chart from it. + """ def __init__(self, data: dict[str, list[Any]]): + """Initialize the HardcodedDataFetcher. + + Args: + data (dict[str, list[Any]]): The hardcoded data. + """ self.data = data def fetch_data(self) -> pd.DataFrame: - """Transform arbitrary data into a dataframe""" + """Transform arbitrary data into a dataframe. + + Returns: + pd.DataFrame: The hardcoded data as a dataframe + """ try: df = pd.DataFrame(self.data) except ValueError as e: diff --git a/ckanext/charts/helpers.py b/ckanext/charts/helpers.py index 5dc9d78..9a9f3c6 100644 --- a/ckanext/charts/helpers.py +++ b/ckanext/charts/helpers.py @@ -10,33 +10,60 @@ from ckanext.charts.fetchers import DatastoreDataFetcher -def get_redis_cache_size(): - """Get the size of the Redis cache in a human-readable format.""" +def get_redis_cache_size() -> str: + """Get the size of the Redis cache in a human-readable format. + + Returns: + str: Human-readable Redis cache size + """ return utils.printable_file_size(count_redis_cache_size()) -def get_file_cache_size(): - """Get the size of the file cache in a human-readable format.""" +def get_file_cache_size() -> str: + """Get the size of the file cache in a human-readable format. + + Returns: + str: Human-readable file cache size + """ return utils.printable_file_size(count_file_cache_size()) -def get_available_chart_engines_options(): - """Get the available chart engines.""" +def get_available_chart_engines_options() -> list[dict[str, str]]: + """Get the available chart engines. + + Returns: + List of chart engines options + """ return [{"value": engine, "text": engine} for engine in get_chart_engines()] def charts_include_htmx_asset() -> bool: - """Include HTMX asset if enabled.""" + """Checks if the HTMX asset should be included. + + Returns: + bool: True if the HTMX asset should be included, False otherwise. + """ return config.include_htmx_asset() def charts_reinit_ckan_js_modules() -> bool: - """Reinitialize CKAN JS modules.""" + """Checks if CKAN JS modules should be reinitialized. + + Returns: + bool: True if CKAN JS modules should be reinitialized, False otherwise. + """ return config.reinit_ckan_js_modules() def charts_get_resource_columns(resource_id: str) -> str: - """Get the columns of the given resource.""" + """Get the columns of the given resource. + + Args: + resource_id: Resource ID + + Returns: + str: JSON string of columns options + """ fetcher = DatastoreDataFetcher(resource_id) return json.dumps( @@ -45,10 +72,18 @@ def charts_get_resource_columns(resource_id: str) -> str: def charts_user_is_authenticated() -> bool: - """Check if the user is authenticated.""" + """Check if the user is authenticated. + + Returns: + bool: True if the user is authenticated, False otherwise. + """ return tk.current_user.is_authenticated def charts_allow_anon_building_charts() -> bool: - """Allow anonymous users to build charts.""" + """Check if anonymous users are allowed to build charts. + + Returns: + bool: True if anonymous users are allowed to build charts, False otherwise. + """ return config.allow_anon_building_charts() diff --git a/ckanext/charts/i18n/.gitignore b/ckanext/charts/i18n/.gitignore deleted file mode 100644 index e69de29..0000000 diff --git a/ckanext/charts/logic/schema.py b/ckanext/charts/logic/schema.py index a54de9d..e54ac35 100644 --- a/ckanext/charts/logic/schema.py +++ b/ckanext/charts/logic/schema.py @@ -8,5 +8,5 @@ @validator_args -def settings_schema(validate_chart_extras) -> Schema: - return {"__extras": [validate_chart_extras]} +def settings_schema(charts_validate_extras) -> Schema: + return {"__extras": [charts_validate_extras]} diff --git a/ckanext/charts/logic/validators.py b/ckanext/charts/logic/validators.py index 8b10827..464c8db 100644 --- a/ckanext/charts/logic/validators.py +++ b/ckanext/charts/logic/validators.py @@ -10,6 +10,14 @@ def float_validator(value: Any) -> float: + """A validator for decimal numbers. + + Args: + value (Any): The value to validate + + Returns: + float: The value as a float + """ try: return float(value) except ValueError: @@ -17,7 +25,17 @@ def float_validator(value: Any) -> float: def charts_if_empty_same_as(other_key: str) -> Callable[..., Any]: - """A custom version of if_empty_same_as validator for charts""" + """A custom version of if_empty_same_as validator for charts. + + This validator is used to set the value of a field to the value of another + field if it is empty or missing. + + Args: + other_key (str): The key of the field to copy the value from + + Returns: + Callable[..., Any]: The validator function + """ def callable(key, data, errors, context): value = data.get(key) @@ -31,6 +49,17 @@ def callable(key, data, errors, context): def charts_strategy_support(strategy: str) -> str: + """Check if the cache strategy is supported. + + Args: + strategy (str): The cache strategy + + Returns: + str: The cache strategy if it is supported + + Raises: + tk.Invalid: If the cache strategy is not supported + """ if strategy not in const.SUPPORTED_CACHE_STRATEGIES: raise tk.Invalid(tk._("Invalid cache strategy")) @@ -48,13 +77,20 @@ def charts_strategy_support(strategy: str) -> str: return strategy -def validate_chart_extras( +def charts_validate_extras( key: types.FlattenKey, data: types.FlattenDataDict, errors: types.FlattenErrorDict, context: types.Context, ): - """Use a custom validation schema for specific chart types.""" + """Validate charts settings according to the chart type and engine schema. + + Args: + key (types.FlattenKey): The key of the field + data (types.FlattenDataDict): The data to validate + errors (types.FlattenErrorDict): The errors dict + context (types.Context): The context + """ settings = _extract_setting(data) if "engine" not in settings or "type" not in settings: @@ -93,15 +129,29 @@ def _extract_setting(data: types.FlattenDataDict) -> dict[str, Any]: def charts_to_list_if_string(value: Any) -> Any: - """Convert a string to a list""" + """Convert a string to a list. + + Args: + value (Any): The value to convert + + Returns: + list[Any]: The value in a list + """ if isinstance(value, str): return [value] return value -def charts_list_to_csv(data: list[str] | str): - """Convert a list of strings to a CSV string""" +def charts_list_to_csv(data: list[str] | str) -> str: + """Convert a list of strings to a CSV string. + + Args: + data (list[str] | str): The data to convert + + Returns: + str: The comma separated string + """ if not isinstance(data, list): return data @@ -109,6 +159,14 @@ def charts_list_to_csv(data: list[str] | str): def charts_list_length_validator(max_length: int) -> Callable[..., Any]: + """A validator to check the length of a list. + + Args: + max_length (int): The maximum length of the list + + Returns: + Callable[..., Any]: The validator function + """ def callable( key: types.FlattenKey, data: types.FlattenDataDict, diff --git a/ckanext/charts/plugin.py b/ckanext/charts/plugin.py index 40e01c9..b287842 100644 --- a/ckanext/charts/plugin.py +++ b/ckanext/charts/plugin.py @@ -71,7 +71,7 @@ def info(self) -> dict[str, Any]: } def can_view(self, data_dict: dict[str, Any]) -> bool: - return utils.can_view_be_viewed(data_dict) + return utils.can_view(data_dict) def setup_template_variables( self, @@ -224,7 +224,7 @@ def info(self) -> dict[str, Any]: } def can_view(self, data_dict: dict[str, Any]) -> bool: - return utils.can_view_be_viewed(data_dict) + return utils.can_view(data_dict) def setup_template_variables( self, diff --git a/ckanext/charts/tests/conftest.py b/ckanext/charts/tests/conftest.py index 7de52ed..a98df36 100644 --- a/ckanext/charts/tests/conftest.py +++ b/ckanext/charts/tests/conftest.py @@ -4,12 +4,12 @@ from ckanext.charts.cache import drop_file_cache -@pytest.fixture() +@pytest.fixture def clean_file_cache(): drop_file_cache() -@pytest.fixture() +@pytest.fixture def data_frame(): return pd.DataFrame( { diff --git a/ckanext/charts/utils.py b/ckanext/charts/utils.py index 4dd64e3..9085910 100644 --- a/ckanext/charts/utils.py +++ b/ckanext/charts/utils.py @@ -13,14 +13,35 @@ def get_column_options(resource_id: str) -> list[dict[str, str]]: - """Get column options for the given resource""" + """Get column options for the given resource. + + Args: + resource_id: Resource ID + + Returns: + List of column options + """ df = DatastoreDataFetcher(resource_id).fetch_data() return [{"text": col, "value": col} for col in df.columns] def printable_file_size(size_bytes: int) -> str: - """Convert file size in bytes to human-readable format""" + """Convert file size in bytes to human-readable format. + + Args: + size_bytes: File size in bytes + + Returns: + str: Human-readable file size + + Examples: + >>> printable_file_size(123456789) + '117.7 MB' + + >>> printable_file_size(7777) + '7.6 KB' + """ if size_bytes == 0: return "0 bytes" @@ -33,6 +54,7 @@ def printable_file_size(size_bytes: int) -> str: def get_chart_form_builder(engine: str, chart_type: str): + """Get form builder for the given engine and chart type.""" builders = get_chart_engines() if engine not in builders: @@ -41,13 +63,31 @@ def get_chart_form_builder(engine: str, chart_type: str): return builders[engine].get_form_for_type(chart_type) -def build_chart_for_data(settings: dict[str, Any], data: pd.DataFrame): - """Build chart for the given dataframe""" +def build_chart_for_data(settings: dict[str, Any], data: pd.DataFrame) -> str | None: + """Build chart for the given dataframe and settings. + + Args: + settings: Chart settings + data: Dataframe with data + + Returns: + Chart config as JSON string + """ return _build_chart(settings, data) -def build_chart_for_resource(settings: dict[str, Any], resource_id: str): - """Build chart for the given resource ID""" +def build_chart_for_resource(settings: dict[str, Any], resource_id: str) -> str | None: + """Build chart for the given resource ID. + + Uses a DatastoreDataFetcher to fetch data from the resource. + + Args: + settings: Chart settings + resource_id: Resource ID + + Returns: + str | None: Chart config as JSON string or None if the chart can't be built + """ settings.pop("__extras", None) try: @@ -77,7 +117,17 @@ def _build_chart(settings: dict[str, Any], dataframe: pd.DataFrame) -> str | Non return chart_config -def can_view_be_viewed(data_dict: dict[str, Any]) -> bool: +def can_view(data_dict: dict[str, Any]) -> bool: + """Check if the resource can be viewed as a chart. + + For now, we work only with resources stored with the DataStore. + + Args: + data_dict: Resource data dictionary + + Returns: + bool: True if the resource can be viewed as a chart, False otherwise + """ if data_dict["resource"].get("datastore_active"): return True diff --git a/docs/caching/base.md b/docs/caching/base.md new file mode 100644 index 0000000..2545cc9 --- /dev/null +++ b/docs/caching/base.md @@ -0,0 +1,54 @@ +# Base Cache Strategy + +To implement a custom cache strategy, you need to create a new class that extends the `CacheStrategy` class and implement the abstract methods. + +See a naive implementation of a memcached cache strategy below: +```python +from io import BytesIO + +from pymemcache.client import base +import pandas as pd + +import ckanext.charts.config as config +from ckanext.charts.cache import CacheStrategy + +class MemcachedCache(CacheStrategy): + """Cache data to Memcached""" + + def __init__(self): + self.client = base.Client(('localhost', 11211)) + + def get_data(self, key: str) -> pd.DataFrame | None: + """Return data from cache if exists""" + try: + raw_data = self.client.get(key) + + if not raw_data: + return None + + return pd.read_csv(BytesIO(raw_data)) + except Exception: + log.exception(f"Failed to get data for key: {key}") + return None + + def set_data(self, key: str, data: pd.DataFrame): + """Serialize data and save to Memcached""" + cache_ttl = config.get_memcached_cache_ttl() + + try: + serialized_data = data.to_csv(index=False).encode('utf-8') + self.client.set(key, serialized_data, expire=cache_ttl) + except Exception: + log.exception(f"Failed to save data to Memcached for key: {key}") + + def invalidate(self, key: str): + """Invalidate cache by key""" + try: + self.client.delete(key) + except Exception: + log.exception(f"Failed to invalidate cache for key: {key}") +``` + +::: charts.cache.CacheStrategy + options: + show_source: false diff --git a/docs/caching/file.md b/docs/caching/file.md new file mode 100644 index 0000000..8282623 --- /dev/null +++ b/docs/caching/file.md @@ -0,0 +1,16 @@ +# File Cache + +::: charts.cache.FileCache + options: + show_source: false + show_root_heading: true + +::: charts.cache.FileCacheORC + options: + show_source: false + show_root_heading: true + +::: charts.cache.FileCacheCSV + options: + show_source: false + show_root_heading: true diff --git a/docs/caching/index.md b/docs/caching/index.md new file mode 100644 index 0000000..6e14e88 --- /dev/null +++ b/docs/caching/index.md @@ -0,0 +1,41 @@ +# Caching + +The extension implement a cache strategy to store the data fetched from the different sources. + +There are three cache strategies available: + +1. `redis` +2. `file_orc` +3. `file_csv`. + +## File cache + +The file cache works by storing the data in an `orc` or `csv` file in the filesystem. The redis cache stores the data in a Redis database. The cache strategy can be changed at the CKAN configuration level through the admin interface or in a configuration file. + +The `file-type` cache strategy stores the data in a file in the filesystem. The file cache is stored in the `ckanext-charts` directory in the CKAN storage path. The file cache is stored in an `orc` or `csv` file format. + +???+ Warning + Using `file_orc` cache strategy requires the `pyarrow` python library to be installed. + +## Redis cache + +The `redis` cache strategy stores the data in a Redis database. + +Each redis key has a `ckanext-charts:*` prefix and store the data as a CSV string. + +???+ Note + You need to have a Redis server running to use the `redis` cache strategy. + +## Cache TTL + +The cache TTL can be set in the CKAN configuration file. The default value is 3600 seconds (1 hour). The cache TTL can be set to `0` to disable the cache. + +The `redis` and `file-type` cache has separate TTL settings: + +* The `redis` cache TTL can be set with the `ckanext.charts.redis_cache_ttl` configuration option. +* The `file` cache TTL can be set with the `ckanext.charts.file_cache_ttl` configuration option. + +## Disable cache + +Cache could be disabled by setting `ckanext.charts.enable_cache` to `false`. In this case the data will be fetched from the source every time the chart is rendered. It could be useful for debugging purposes. But using it in production is not recommended, as it could lead to performance issues. + diff --git a/docs/caching/redis.md b/docs/caching/redis.md new file mode 100644 index 0000000..91294f2 --- /dev/null +++ b/docs/caching/redis.md @@ -0,0 +1,5 @@ +# Redis Cache + +::: charts.cache.RedisCache + options: + show_source: false diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..9a8f2d1 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,103 @@ +Configuration options + +There is a number of configuration options available for the extension. You can set them in the CKAN configuration file or using the admin configuration page. + +## Configuration options + +### Cache strategy +**`ckanext.charts.cache_strategy`** [__optional__] + +Cache strategy for chart data. + +**Options**: `redis`, `file_orc`, `file_csv` + +**Type**: `str` + +**Default**: `redis` + +----- + +### Redis cache TTL + +**`ckanext.charts.redis_cache_ttl`** [__optional__] + +Time to live for the Redis cache in seconds. Set to `0` to disable cache. + +**Type**: `int` + +**Default**: `3600` + +----- + +### File cache TTL + +**`ckanext.charts.file_cache_ttl`** [__optional__] + +Time to live for the File cache in seconds. Set to `0` to disable cache. + +**Type**: `int` + +**Default**: `3600` + +----- + +### Enable cache + +**`ckanext.charts.enable_cache`** [__optional__] + +Enable cache for the charts. + +???+ Warning + If the cache is disabled, the chart data will be fetched from the data source every time the chart is rendered. + It could lead to performance issues if the data source is slow or the data is too large. + +**Type**: `bool` + +**Default**: `true` + +----- + +### Include HTMX + +**`ckanext.charts.include_htmx_asset`** [__optional__] + +Include HTMX assets. Could be disabled if another plugins are including it. + +**Type**: `bool` + +**Default**: `true` + +----- + +### Reinit CKAN JS + +**`ckanext.charts.reinit_ckan_js_modules`** [__optional__] + +Reinit CKAN JS modules after HTMX swap. + +**Type**: `bool` + +**Default**: `false` + +----- + +### Allow anonymous users to build charts + +**`ckanext.charts.allow_anon_building_charts`** [__optional__] + +By default, only authenticated users can build charts. If you want to allow anonymous users to build charts, set this option to `true`. + +**Type**: `bool` + +**Default**: `false` + +----- + +## Admin config page + +The extension provides an admin configuration page where you can set all the listed configuration options. The admin page available only +using an extension [`ckanext-admin-panel`](https://github.com/DataShades/ckanext-admin-panel). The admin panel is a separate extension that provides an alternative admin interface for CKAN. It allows you to manage CKAN settings and other extensions settings through the web interface. + +![Admin config page](./image/admin-page.jpg) + +This page could be accessed by the following URL: `/admin-panel/charts/config`, but it will be registered only if the `ckanext-admin-panel` is installed and enabled. diff --git a/docs/engines/custom.md b/docs/engines/custom.md new file mode 100644 index 0000000..f54a9f5 --- /dev/null +++ b/docs/engines/custom.md @@ -0,0 +1,134 @@ +# Custom chart engine + +## Implementing new chart engines support + +Implementing support for a new chart engine includes multiple steps and changes in Python, HTML, and JavaScript. Starting from the Python code: + +1. Create a new builder class at `ckanext.charts.chart_builder` that inherits from `BaseChartBuilder` and implements the `get_supported_forms` method. This method must return a list of classes that represent supported chart types forms. + +2. Each form type builder must be connected with a respective chart type builder. + +3. The chart type builder must implement a `to_json` method that will return a dumped JSON data, which will be passed to a JS script. + +4. The form type builder must implement a `get_form_fields` method that will return a list of all form fields that will be rendered for the user, allowing them to provide all the necessary information for a chart. + +5. Register your chart engine by adding the builder class to `get_chart_engines` in `ckanext.charts.chart_builder.__init__.py`. + +A full example of an implementation of `bar` chart for `obvervable plot` library. + +```py +from __future__ import annotations + +import json +from typing import Any + +import ckanext.charts.exception as exception +from ckanext.charts.chart_builders.base import BaseChartBuilder, BaseChartForm + + +class ObservableBuilder(BaseChartBuilder): + @classmethod + def get_supported_forms(cls) -> list[type[Any]]: + return [ObservableBarForm] + + +class ObservableBarBuilder(ObservableBuilder): + def to_json(self) -> str: + return json.dumps( + { + "type": "bar", + "data": self.df.to_dict(orient="records"), + "settings": self.settings, + } + ) + + +class ObservableBarForm(BaseChartForm): + name = "Bar" + builder = ObservableBarBuilder + + def fill_field(self, choices: list[dict[str, str]]) -> dict[str, str]: + field = self.color_field(choices) + field.update({"field_name": "fill", "label": "Fill"}) + + return field + + def get_form_fields(self): + columns = [{"value": col, "label": col} for col in self.df.columns] + chart_types = [ + {"value": form.name, "label": form.name} + for form in self.builder.get_supported_forms() + ] + + return [ + self.title_field(), + self.description_field(), + self.engine_field(), + self.type_field(chart_types), + self.x_axis_field(columns), + self.y_axis_field(columns), + self.fill_field(columns), + self.opacity_field(), + self.limit_field(), + ] +``` + +## Vendor and custom JS + +Another step is to register JS/CSS vendor libraries of the chart you want to use. Refer to [CKAN documentation](https://docs.ckan.org/en/latest/theming/webassets.html) to read about adding CSS and JavaScript files using Webassets. + +You also will need a CKAN JS module, that will be responsible for rendering the Chart. This module will work with the vendor library and will be responsible for rendering the chart in the container. + +This module must be registered inside a `webassets.yml` as well. +```js + ckan.module("charts-render-observable", function ($, _) { + "use strict"; + + return { + options: { + config: null + }, + + initialize: function () { + $.proxyAll(this, /_/); + + if (!this.options.config) { + console.error("No configuration provided"); + return; + } + + var plot; + + switch (this.options.config.type) { + case "bar": + plot = Plot.barY(this.options.config.data, this.options.config.settings).plot(); + break; + default: + return; + } + + this.el[0].replaceChildren(plot); + } + }; + }); +``` + +## HTML container + +And an HTML file, that will provide a proper container and include your JS module with `data-module`. + +```html + {% asset "charts/observable" %} + + {% if chart %} + + {% else %} ++ {{ _("Cannot build chart with current settings") }} +
+ {% endif %} +``` + +Note, that we should add `{% asset "charts/observable" %}` not only here, but in `charts_form.html` too. + +The reason for having a separate `HTML` file and `JS` module is that different libraries may require different types of container elements (such as div, canvas, etc.) to initialize or may need additional boilerplate code to build a chart. There's no easy way to abstract this, so you have to implement these things yourself. diff --git a/docs/engines/index.md b/docs/engines/index.md new file mode 100644 index 0000000..ebb0cad --- /dev/null +++ b/docs/engines/index.md @@ -0,0 +1,8 @@ +The `ckanext-charts` implements supports different chart engines, such as `plotly`, `observable` and `chartjs`. + +Each engine consists of two classes - chart builder and form builder. The chart builder is responsible for generating a JSON-string representation of the chart data, that will be passed to a respective JS module, that will render a chart based on the data. + +The form builder is responsible for generating a form that allows users to configure the chart from the UI. + +Read more about how to create a custom chart engine in the [documentation](./custom.md). + diff --git a/docs/fetchers/base.md b/docs/fetchers/base.md new file mode 100644 index 0000000..41d2006 --- /dev/null +++ b/docs/fetchers/base.md @@ -0,0 +1,7 @@ +# Base Fetcher + +All fetchers must inherit from the `DataFetcherStrategy` class. This class provides the basic structure for fetchers and defines the methods that must be implemented by all fetchers. + +::: charts.fetchers.DataFetcherStrategy + options: + show_bases: false diff --git a/docs/fetchers/datastore.md b/docs/fetchers/datastore.md new file mode 100644 index 0000000..7ba5e26 --- /dev/null +++ b/docs/fetchers/datastore.md @@ -0,0 +1,7 @@ +# Datastore Fetcher + +::: charts.fetchers.DatastoreDataFetcher + options: + show_source: false + show_bases: false + show_inheritance_diagram: true diff --git a/docs/fetchers/file.md b/docs/fetchers/file.md new file mode 100644 index 0000000..534e656 --- /dev/null +++ b/docs/fetchers/file.md @@ -0,0 +1,5 @@ +# File System Data Fetcher + +::: charts.fetchers.FileSystemDataFetcher + options: + show_source: false diff --git a/docs/fetchers/hardcode.md b/docs/fetchers/hardcode.md new file mode 100644 index 0000000..311c155 --- /dev/null +++ b/docs/fetchers/hardcode.md @@ -0,0 +1,5 @@ +# Hardcoded Data Fetcher + +::: charts.fetchers.HardcodedDataFetcher + options: + show_source: false diff --git a/docs/fetchers/index.md b/docs/fetchers/index.md new file mode 100644 index 0000000..4677a31 --- /dev/null +++ b/docs/fetchers/index.md @@ -0,0 +1,17 @@ +# Fetchers + +Fetchers are responsible for fetching data from different sources (DataStore, URL, file system, hardcoded data). + +For the current implementation, we're working with resources that are uploaded to the DataStore, so the fetcher will be responsible for fetching the data from the DataStore. + +But it might come in handy to have fetchers for other sources, like URL, file system, etc. + +## Implementing new fetchers + +To register new fetchers, you need to create a new class that inherits from `DataFetcherStrategy` and implements the `fetch_data` and `make_cache_key` methods. + +The `fetch_data` method should return a `pandas` `DataFrame` object with the data that should be displayed in the chart. + +The `make_cache_key` method should return a unique string that will be used as a key to store the data in the cache. + +See the [base class](./base.md) `DataFetcherStrategy` for more information. diff --git a/docs/fetchers/url.md b/docs/fetchers/url.md new file mode 100644 index 0000000..7d615fd --- /dev/null +++ b/docs/fetchers/url.md @@ -0,0 +1,5 @@ +# URL Data Fetcher + +::: charts.fetchers.URLDataFetcher + options: + show_source: false diff --git a/docs/helpers.md b/docs/helpers.md new file mode 100644 index 0000000..5187367 --- /dev/null +++ b/docs/helpers.md @@ -0,0 +1,5 @@ +# Helper Functions + +::: charts.helpers + options: + show_source: true diff --git a/doc/admin-page.jpg b/docs/image/admin-page.jpg similarity index 100% rename from doc/admin-page.jpg rename to docs/image/admin-page.jpg diff --git a/doc/anon-restrict.png b/docs/image/anon-restrict.png similarity index 100% rename from doc/anon-restrict.png rename to docs/image/anon-restrict.png diff --git a/doc/chart-builder.gif b/docs/image/chart-builder.gif similarity index 100% rename from doc/chart-builder.gif rename to docs/image/chart-builder.gif diff --git a/doc/chart-user-builder.gif b/docs/image/chart-user-builder.gif similarity index 100% rename from doc/chart-user-builder.gif rename to docs/image/chart-user-builder.gif diff --git a/docs/image/favicon.ico b/docs/image/favicon.ico new file mode 100644 index 0000000..0d9295c Binary files /dev/null and b/docs/image/favicon.ico differ diff --git a/docs/image/logo.png b/docs/image/logo.png new file mode 100644 index 0000000..3f631bf Binary files /dev/null and b/docs/image/logo.png differ diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..612c7a5 --- /dev/null +++ b/docs/index.md @@ -0,0 +1 @@ +--8<-- "README.md" diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 0000000..cb4ef01 --- /dev/null +++ b/docs/install.md @@ -0,0 +1,40 @@ +# Installation + +## Requirements + +Requires **Redis 7+** + +Compatibility with core CKAN versions: + +| CKAN version | Compatible? | +| --------------- | ------------- | +| 2.9 and earlier | no | +| 2.10+ | yes | + +## Installation + +1. Install the extension from `PyPI`: + ```sh + pip install ckanext-charts + ``` + + If you want to use `ORC` file cache, you have to install the extension with the `pyarrow` extra: + ```sh + pip install ckanext-charts[pyarrow] + ``` + +2. Enable the main plugin and harvesters you want to use in your CKAN configuration file (e.g. `ckan.ini` or `production.ini`): + + ```ini + ckan.plugins = ... charts_view charts_builder_view ... + ``` + +## Dependencies + +The extension requires the following CKAN extensions to be installed and enabled: + +1. [ckanext-scheming](https://github.com/ckan/ckanext-scheming): +We're using the scheming extension to create custom forms for the chart builders. + +2. [ckanext-admin-panel](https://github.com/DataShades/ckanext-admin-panel) (__optional__): +If you want to use the admin configuration page, you need to install and enable this extension. The admin panel is a separate extension that provides an alternative admin interface for CKAN. It allows you to manage CKAN settings and other extensions settings through the web interface and significantly extends the default CKAN admin interface. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..bba6068 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,3 @@ +**ImportError: lxml.html.clean module is now a separate project lxml_html_clean** + +Install `lxml[html_clean]` or `lxml_html_clean` directly using pip. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..543d606 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,15 @@ +## Basic usage + +This extension offers two plugins: `charts_view` and `charts_builder_view`. The first plugin is a standard chart builder designed for system administrators or anyone with permission to edit or create a resource view. Creating a preview using this plugin generates a static chart with saved settings. This chart retains the settings and will always be rendered in the same way. + +![User chart builder](./image/chart-builder.gif) + +The `charts_builder_view` plugin does not have any special settings during the creation or editing stages. Instead, it renders a chart builder JS widget on a resource page, allowing any user to interact with the resource data and create a chart that meets their needs. In the future, we plan to implement a feature that will allow users to save their charts. + +![User chart builder](./image/chart-user-builder.gif) + +## Restrict user chart builder for anonymous users + +The user chart builder could be restricted to only authenticated users. See the config settings section for more information. In this case, user will be forced to log in to build a chart. After the user logs in, user will be redirected to the chart builder page back. + +![Restrict user chart builder for anon](./image/anon-restrict.png) diff --git a/docs/utils.md b/docs/utils.md new file mode 100644 index 0000000..8423085 --- /dev/null +++ b/docs/utils.md @@ -0,0 +1,5 @@ +# Utility Functions + +::: charts.utils + options: + show_source: true diff --git a/docs/validators.md b/docs/validators.md new file mode 100644 index 0000000..593dc09 --- /dev/null +++ b/docs/validators.md @@ -0,0 +1,5 @@ +# Validators + +::: charts.logic.validators + options: + show_source: true diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..2e14764 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,103 @@ +site_name: ckanext-charts +site_url: https://.github.io/ckanext-charts/ + +repo_url: https://github.com/DataShades/ckanext-charts + +markdown_extensions: + - pymdownx.snippets + - admonition + - pymdownx.details + - pymdownx.emoji: + emoji_index: !!python/name:material.extensions.emoji.twemoji + emoji_generator: !!python/name:material.extensions.emoji.to_svg + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format + - pymdownx.tabbed: + alternate_style: true + + +plugins: + - mkdocstrings: + handlers: + python: + paths: ["ckanext"] + options: + show_root_full_path: false + show_root_toc_entry: false + show_symbol_type_heading: true + show_source: false + docstring_section_style: spacy + - search: + separator: '[\s\-\.\_]+' + +watch: + - README.md + - ckanext/charts/ + +theme: + name: material + logo: image/logo.png + favicon: image/favicon.ico + palette: + - media: "(prefers-color-scheme)" + toggle: + icon: material/brightness-auto + name: Switch to light mode + + # Palette toggle for light mode + - media: "(prefers-color-scheme: light)" + scheme: default + primary: blue + toggle: + icon: material/brightness-7 + name: Switch to dark mode + + # Palette toggle for dark mode + - media: "(prefers-color-scheme: dark)" + scheme: slate + toggle: + icon: material/brightness-4 + name: Switch to system preference + + features: + - navigation.instant + - navigation.instant.prefetch + - navigation.instant.progress + - navigation.footer + - navigation.indexes + - navigation.top + - content.code.copy + - content.code.select + - content.code.annotate + +nav: + - index.md + - install.md + - usage.md + - configuration.md + + - Engines: + - engines/index.md + - engines/custom.md + + - Caching: + - caching/index.md + - caching/base.md + - caching/redis.md + - caching/file.md + + - Fetchers: + - fetchers/index.md + - fetchers/base.md + - fetchers/datastore.md + - fetchers/url.md + - fetchers/file.md + - fetchers/hardcode.md + + - validators.md + - utils.md + - helpers.md + - troubleshooting.md diff --git a/site/404.html b/site/404.html new file mode 100644 index 0000000..20a9c4b --- /dev/null +++ b/site/404.html @@ -0,0 +1,869 @@ + + + + + + + + + + + + + + + + + + +