diff --git a/README.md b/README.md
index 6255883..22f4ed9 100644
--- a/README.md
+++ b/README.md
@@ -33,6 +33,21 @@ To run the tests, do:
pytest --ckan-ini=test.ini
+## Building the documentation
+
+We are using `mkdocs` to build the documentation. To build and deploy the documentation, do:
+
+ CHARTS_FIELDS=1 mkdocs build && mkdocs gh-deploy
+
+We need to set the `CHARTS_FIELDS` for our custom handler, that autogenerate documentation
+for chart types fields.
+
+If you're working on the documentation, you can run the following command to start a live-reloading server without
+gathering the chart types fields data. It will speed up the process significantly, as we won't need to wait for the
+CKAN initialization:
+
+ mkdocs serve -a 127.0.0.1:8001
+
## License
[AGPL](https://www.gnu.org/licenses/agpl-3.0.en.html)
diff --git a/docs/install.md b/docs/install.md
index cb4ef01..694fd88 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -23,7 +23,7 @@ Compatibility with core CKAN versions:
pip install ckanext-charts[pyarrow]
```
-2. Enable the main plugin and harvesters you want to use in your CKAN configuration file (e.g. `ckan.ini` or `production.ini`):
+2. Enable the view and builder plugins in your CKAN configuration file (e.g. `ckan.ini` or `production.ini`):
```ini
ckan.plugins = ... charts_view charts_builder_view ...
diff --git a/mkdocstrings_handlers/ChartFieldsHandler/handler.py b/mkdocstrings_handlers/ChartFieldsHandler/handler.py
index ceb97ea..37e207e 100644
--- a/mkdocstrings_handlers/ChartFieldsHandler/handler.py
+++ b/mkdocstrings_handlers/ChartFieldsHandler/handler.py
@@ -1,7 +1,6 @@
from __future__ import annotations
import os
-import importlib
from typing import Mapping, MutableMapping, Any
from unittest.mock import patch, MagicMock
@@ -19,7 +18,7 @@
class ChartFieldsHandler(BaseHandler):
- """Custom handler for documenting the chart engine fields according to the
+ """Custom handler for documenting different chart types fields according to the
form fields schema."""
def collect(
self, identifier: str, config: MutableMapping[str, Any]
diff --git a/site/404.html b/site/404.html
index 7a1d7bb..c127875 100644
--- a/site/404.html
+++ b/site/404.html
@@ -12,7 +12,7 @@
-
+
diff --git a/site/assets/_mkdocstrings.css b/site/assets/_mkdocstrings.css
index 8b83f70..e03e464 100644
--- a/site/assets/_mkdocstrings.css
+++ b/site/assets/_mkdocstrings.css
@@ -26,20 +26,33 @@
float: right;
}
+/* Parameter headings must be inline, not blocks. */
+.doc-heading-parameter {
+ display: inline;
+}
+
+/* Prefer space on the right, not the left of parameter permalinks. */
+.doc-heading-parameter .headerlink {
+ margin-left: 0 !important;
+ margin-right: 0.2rem;
+}
+
/* Backward-compatibility: docstring section titles in bold. */
.doc-section-title {
font-weight: bold;
}
/* Symbols in Navigation and ToC. */
-:root,
+:root, :host,
[data-md-color-scheme="default"] {
+ --doc-symbol-parameter-fg-color: #df50af;
--doc-symbol-attribute-fg-color: #953800;
--doc-symbol-function-fg-color: #8250df;
--doc-symbol-method-fg-color: #8250df;
--doc-symbol-class-fg-color: #0550ae;
--doc-symbol-module-fg-color: #5cad0f;
+ --doc-symbol-parameter-bg-color: #df50af1a;
--doc-symbol-attribute-bg-color: #9538001a;
--doc-symbol-function-bg-color: #8250df1a;
--doc-symbol-method-bg-color: #8250df1a;
@@ -48,12 +61,14 @@
}
[data-md-color-scheme="slate"] {
+ --doc-symbol-parameter-fg-color: #ffa8cc;
--doc-symbol-attribute-fg-color: #ffa657;
--doc-symbol-function-fg-color: #d2a8ff;
--doc-symbol-method-fg-color: #d2a8ff;
--doc-symbol-class-fg-color: #79c0ff;
--doc-symbol-module-fg-color: #baff79;
+ --doc-symbol-parameter-bg-color: #ffa8cc1a;
--doc-symbol-attribute-bg-color: #ffa6571a;
--doc-symbol-function-bg-color: #d2a8ff1a;
--doc-symbol-method-bg-color: #d2a8ff1a;
@@ -68,6 +83,15 @@ code.doc-symbol {
font-weight: bold;
}
+code.doc-symbol-parameter {
+ color: var(--doc-symbol-parameter-fg-color);
+ background-color: var(--doc-symbol-parameter-bg-color);
+}
+
+code.doc-symbol-parameter::after {
+ content: "param";
+}
+
code.doc-symbol-attribute {
color: var(--doc-symbol-attribute-fg-color);
background-color: var(--doc-symbol-attribute-bg-color);
diff --git a/site/caching/base/index.html b/site/caching/base/index.html
index c8ed579..6c5571e 100644
--- a/site/caching/base/index.html
+++ b/site/caching/base/index.html
@@ -18,7 +18,7 @@
-
+
@@ -1071,6 +1071,11 @@
We need to set the CHARTS_FIELDS for our custom handler, that autogenerate documentation
+for chart types fields.
+
If you're working on the documentation, you can run the following command to start a live-reloading server without
+gathering the chart types fields data. It will speed up the process significantly, as we won't need to wait for the
+CKAN initialization:
This extension, ckanext-charts, provides additional functionality for working with charts in CKAN. It allows users to create, manage, and visualize charts based on data stored in CKAN datasets.
The extension includes features such as chart creation, chart editing, chart embedding, and chart sharing. It also supports various chart types, including bar charts, line charts, pie charts, and more.
With ckanext-charts, users can easily generate interactive and visually appealing charts to enhance data analysis and presentation in CKAN.
There is a number of configuration options available for the extension. You can set them in the CKAN configuration file or using the admin configuration page.
If the cache is disabled, the chart data will be fetched from the data source every time the chart is rendered. It could lead to performance issues if the data source is slow or the data is too large.
The extension provides an admin configuration page where you can set all the listed configuration options. The admin page available only using an extension ckanext-admin-panel. The admin panel is a separate extension that provides an alternative admin interface for CKAN. It allows you to manage CKAN settings and other extensions settings through the web interface.
This page could be accessed by the following URL: /admin-panel/charts/config, but it will be registered only if the ckanext-admin-panel is installed and enabled.
Check if anonymous users are allowed to build charts.
RETURNS DESCRIPTION bool
True if anonymous users are allowed to build charts, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_allow_anon_building_charts() -> bool:\n \"\"\"Check if anonymous users are allowed to build charts.\n\n Returns:\n bool: True if anonymous users are allowed to build charts, False otherwise.\n \"\"\"\n return config.allow_anon_building_charts()\n
def charts_get_resource_columns(resource_id: str) -> str:\n \"\"\"Get the columns of the given resource.\n\n Args:\n resource_id: Resource ID\n\n Returns:\n str: JSON string of columns options\n \"\"\"\n fetcher = DatastoreDataFetcher(resource_id)\n\n return json.dumps(\n [{\"id\": col, \"title\": col} for col in fetcher.fetch_data().columns],\n )\n
True if the HTMX asset should be included, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_include_htmx_asset() -> bool:\n \"\"\"Checks if the HTMX asset should be included.\n\n Returns:\n bool: True if the HTMX asset should be included, False otherwise.\n \"\"\"\n return config.include_htmx_asset()\n
Checks if CKAN JS modules should be reinitialized.
RETURNS DESCRIPTION bool
True if CKAN JS modules should be reinitialized, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_reinit_ckan_js_modules() -> bool:\n \"\"\"Checks if CKAN JS modules should be reinitialized.\n\n Returns:\n bool: True if CKAN JS modules should be reinitialized, False otherwise.\n \"\"\"\n return config.reinit_ckan_js_modules()\n
True if the user is authenticated, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_user_is_authenticated() -> bool:\n \"\"\"Check if the user is authenticated.\n\n Returns:\n bool: True if the user is authenticated, False otherwise.\n \"\"\"\n return tk.current_user.is_authenticated\n
def get_available_chart_engines_options() -> list[dict[str, str]]:\n \"\"\"Get the available chart engines.\n\n Returns:\n List of chart engines options\n \"\"\"\n return [{\"value\": engine, \"text\": engine} for engine in get_chart_engines()]\n
Get the size of the file cache in a human-readable format.
RETURNS DESCRIPTION str
Human-readable file cache size
TYPE: str
Source code in ckanext/charts/helpers.py
def get_file_cache_size() -> str:\n \"\"\"Get the size of the file cache in a human-readable format.\n\n Returns:\n str: Human-readable file cache size\n \"\"\"\n return utils.printable_file_size(count_file_cache_size())\n
Get the size of the Redis cache in a human-readable format.
RETURNS DESCRIPTION str
Human-readable Redis cache size
TYPE: str
Source code in ckanext/charts/helpers.py
def get_redis_cache_size() -> str:\n \"\"\"Get the size of the Redis cache in a human-readable format.\n\n Returns:\n str: Human-readable Redis cache size\n \"\"\"\n return utils.printable_file_size(count_redis_cache_size())\n
The extension requires the following CKAN extensions to be installed and enabled:
ckanext-scheming: We're using the scheming extension to create custom forms for the chart builders.
ckanext-admin-panel (optional): If you want to use the admin configuration page, you need to install and enable this extension. The admin panel is a separate extension that provides an alternative admin interface for CKAN. It allows you to manage CKAN settings and other extensions settings through the web interface and significantly extends the default CKAN admin interface.
This extension offers two plugins: charts_view and charts_builder_view. The first plugin is a standard chart builder designed for system administrators or anyone with permission to edit or create a resource view. Creating a preview using this plugin generates a static chart with saved settings. This chart retains the settings and will always be rendered in the same way.
The charts_builder_view plugin does not have any special settings during the creation or editing stages. Instead, it renders a chart builder JS widget on a resource page, allowing any user to interact with the resource data and create a chart that meets their needs. In the future, we plan to implement a feature that will allow users to save their charts.
"},{"location":"usage/#restrict-user-chart-builder-for-anonymous-users","title":"Restrict user chart builder for anonymous users","text":"
The user chart builder could be restricted to only authenticated users. See the config settings section for more information. In this case, user will be forced to log in to build a chart. After the user logs in, user will be redirected to the chart builder page back.
Uses a DatastoreDataFetcher to fetch data from the resource.
PARAMETER DESCRIPTION settings
Chart settings
TYPE: dict[str, Any]
resource_id
Resource ID
TYPE: str
RETURNS DESCRIPTION str | None
str | None: Chart config as JSON string or None if the chart can't be built
Source code in ckanext/charts/utils.py
def build_chart_for_resource(settings: dict[str, Any], resource_id: str) -> str | None:\n \"\"\"Build chart for the given resource ID.\n\n Uses a DatastoreDataFetcher to fetch data from the resource.\n\n Args:\n settings: Chart settings\n resource_id: Resource ID\n\n Returns:\n str | None: Chart config as JSON string or None if the chart can't be built\n \"\"\"\n settings.pop(\"__extras\", None)\n\n try:\n df = DatastoreDataFetcher(resource_id).fetch_data()\n except tk.ValidationError:\n return None\n\n return _build_chart(settings, df)\n
For now, we work only with resources stored with the DataStore.
PARAMETER DESCRIPTION data_dict
Resource data dictionary
TYPE: dict[str, Any]
RETURNS DESCRIPTION bool
True if the resource can be viewed as a chart, False otherwise
TYPE: bool
Source code in ckanext/charts/utils.py
def can_view(data_dict: dict[str, Any]) -> bool:\n \"\"\"Check if the resource can be viewed as a chart.\n\n For now, we work only with resources stored with the DataStore.\n\n Args:\n data_dict: Resource data dictionary\n\n Returns:\n bool: True if the resource can be viewed as a chart, False otherwise\n \"\"\"\n if data_dict[\"resource\"].get(\"datastore_active\"):\n return True\n\n # TODO: Add support for XML, XLS, XLSX, and other formats tabular data?\n # if data_dict[\"resource\"][\"format\"].lower() == \"xml\":\n # return True\n\n return False\n
Get form builder for the given engine and chart type.
Source code in ckanext/charts/utils.py
def get_chart_form_builder(engine: str, chart_type: str):\n \"\"\"Get form builder for the given engine and chart type.\"\"\"\n builders = get_chart_engines()\n\n if engine not in builders:\n raise NotImplementedError(f\"Engine {engine} is not supported\")\n\n return builders[engine].get_form_for_type(chart_type)\n
def get_column_options(resource_id: str) -> list[dict[str, str]]:\n \"\"\"Get column options for the given resource.\n\n Args:\n resource_id: Resource ID\n\n Returns:\n List of column options\n \"\"\"\n df = DatastoreDataFetcher(resource_id).fetch_data()\n\n return [{\"text\": col, \"value\": col} for col in df.columns]\n
A custom version of if_empty_same_as validator for charts.
This validator is used to set the value of a field to the value of another field if it is empty or missing.
PARAMETER DESCRIPTION other_key
The key of the field to copy the value from
TYPE: str
RETURNS DESCRIPTION Callable[..., Any]
Callable[..., Any]: The validator function
Source code in ckanext/charts/logic/validators.py
def charts_if_empty_same_as(other_key: str) -> Callable[..., Any]:\n \"\"\"A custom version of if_empty_same_as validator for charts.\n\n This validator is used to set the value of a field to the value of another\n field if it is empty or missing.\n\n Args:\n other_key (str): The key of the field to copy the value from\n\n Returns:\n Callable[..., Any]: The validator function\n \"\"\"\n\n def callable(key, data, errors, context):\n value = data.get(key)\n if not value or value is tk.missing:\n try:\n data[key] = data[key[:-1] + (other_key,)]\n except KeyError:\n data[key] = data.get((\"__extras\",), {}).get(other_key, \"\")\n\n return callable\n
def charts_list_length_validator(max_length: int) -> Callable[..., Any]:\n \"\"\"A validator to check the length of a list.\n\n Args:\n max_length (int): The maximum length of the list\n\n Returns:\n Callable[..., Any]: The validator function\n \"\"\"\n def callable(\n key: types.FlattenKey,\n data: types.FlattenDataDict,\n errors: types.FlattenErrorDict,\n context: types.Context,\n ):\n if len(data[key]) > max_length:\n raise tk.Invalid(tk._(\"Length must be less than {0}\").format(max_length))\n\n return callable\n
def charts_list_to_csv(data: list[str] | str) -> str:\n \"\"\"Convert a list of strings to a CSV string.\n\n Args:\n data (list[str] | str): The data to convert\n\n Returns:\n str: The comma separated string\n \"\"\"\n if not isinstance(data, list):\n return data\n\n return \", \".join(data)\n
def charts_strategy_support(strategy: str) -> str:\n \"\"\"Check if the cache strategy is supported.\n\n Args:\n strategy (str): The cache strategy\n\n Returns:\n str: The cache strategy if it is supported\n\n Raises:\n tk.Invalid: If the cache strategy is not supported\n \"\"\"\n if strategy not in const.SUPPORTED_CACHE_STRATEGIES:\n raise tk.Invalid(tk._(\"Invalid cache strategy\"))\n\n if strategy == const.CACHE_FILE_ORC:\n try:\n from pyarrow import orc as _ # noqa\n except ImportError:\n raise tk.Invalid(\n tk._(\"Can't use File Orc cache strategy. PyArrow is not installed\"),\n ) from None\n\n if not strategy:\n return const.DEFAULT_CACHE_STRATEGY\n\n return strategy\n
def charts_to_list_if_string(value: Any) -> Any:\n \"\"\"Convert a string to a list.\n\n Args:\n value (Any): The value to convert\n\n Returns:\n list[Any]: The value in a list\n \"\"\"\n if isinstance(value, str):\n return [value]\n\n return value\n
Validate charts settings according to the chart type and engine schema.
PARAMETER DESCRIPTION key
The key of the field
TYPE: FlattenKey
data
The data to validate
TYPE: FlattenDataDict
errors
The errors dict
TYPE: FlattenErrorDict
context
The context
TYPE: Context
Source code in ckanext/charts/logic/validators.py
def charts_validate_extras(\n key: types.FlattenKey,\n data: types.FlattenDataDict,\n errors: types.FlattenErrorDict,\n context: types.Context,\n):\n \"\"\"Validate charts settings according to the chart type and engine schema.\n\n Args:\n key (types.FlattenKey): The key of the field\n data (types.FlattenDataDict): The data to validate\n errors (types.FlattenErrorDict): The errors dict\n context (types.Context): The context\n \"\"\"\n settings = _extract_setting(data)\n\n if \"engine\" not in settings or \"type\" not in settings:\n builder = DEFAULT_CHART_FORM\n else:\n builder = utils.get_chart_form_builder(settings[\"engine\"], settings[\"type\"])\n\n settings, err = tk.navl_validate(\n settings,\n builder(settings[\"resource_id\"]).get_validation_schema(\n context.get(\"_for_show\", False),\n ),\n {},\n )\n\n # TODO: do we have a better way to handle this? Seems like a hack\n for k, v in settings.items():\n data[(k,)] = v\n\n for k, v in settings.pop(\"__extras\", {}).items():\n data[(k,)] = v\n\n for k, v in err.items():\n errors[(k,)] = v\n
def float_validator(value: Any) -> float:\n \"\"\"A validator for decimal numbers.\n\n Args:\n value (Any): The value to validate\n\n Returns:\n float: The value as a float\n \"\"\"\n try:\n return float(value)\n except ValueError:\n raise tk.Invalid(tk._(\"Must be a decimal number\")) from None\n
The file cache works by storing the data in an orc or csv file in the filesystem. The redis cache stores the data in a Redis database. The cache strategy can be changed at the CKAN configuration level through the admin interface or in a configuration file.
The file-type cache strategy stores the data in a file in the filesystem. The file cache is stored in the ckanext-charts directory in the CKAN storage path. The file cache is stored in an orc or csv file format.
Warning
Using file_orc cache strategy requires the pyarrow python library to be installed.
The cache TTL can be set in the CKAN configuration file. The default value is 3600 seconds (1 hour). The cache TTL can be set to 0 to disable the cache.
The redis and file-type cache has separate TTL settings:
The redis cache TTL can be set with the ckanext.charts.redis_cache_ttl configuration option.
The file cache TTL can be set with the ckanext.charts.file_cache_ttl configuration option.
Cache could be disabled by setting ckanext.charts.enable_cache to false. In this case the data will be fetched from the source every time the chart is rendered. It could be useful for debugging purposes. But using it in production is not recommended, as it could lead to performance issues.
To implement a custom cache strategy, you need to create a new class that extends the CacheStrategy class and implement the abstract methods.
See a naive implementation of a memcached cache strategy below:
from io import BytesIO\n\nfrom pymemcache.client import base\nimport pandas as pd\n\nimport ckanext.charts.config as config\nfrom ckanext.charts.cache import CacheStrategy\n\nclass MemcachedCache(CacheStrategy):\n \"\"\"Cache data to Memcached\"\"\"\n\n def __init__(self):\n self.client = base.Client(('localhost', 11211))\n\n def get_data(self, key: str) -> pd.DataFrame | None:\n \"\"\"Return data from cache if exists\"\"\"\n try:\n raw_data = self.client.get(key)\n\n if not raw_data:\n return None\n\n return pd.read_csv(BytesIO(raw_data))\n except Exception:\n log.exception(f\"Failed to get data for key: {key}\")\n return None\n\n def set_data(self, key: str, data: pd.DataFrame):\n \"\"\"Serialize data and save to Memcached\"\"\"\n cache_ttl = config.get_memcached_cache_ttl()\n\n try:\n serialized_data = data.to_csv(index=False).encode('utf-8')\n self.client.set(key, serialized_data, expire=cache_ttl)\n except Exception:\n log.exception(f\"Failed to save data to Memcached for key: {key}\")\n\n def invalidate(self, key: str):\n \"\"\"Invalidate cache by key\"\"\"\n try:\n self.client.delete(key)\n except Exception:\n log.exception(f\"Failed to invalidate cache for key: {key}\")\n
Bases: ABC
Cache strategy interface.
Defines the abstracts methods for cache strategies.
The ckanext-charts implements supports different chart engines, such as plotly, observable and chartjs.
Each engine consists of two classes - chart builder and form builder. The chart builder is responsible for generating a JSON-string representation of the chart data, that will be passed to a respective JS module, that will render a chart based on the data.
The form builder is responsible for generating a form that allows users to configure the chart from the UI.
Also, you can implement support of a chart library of your choice by creating a custom chart engine. Read more about it here.
The bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars can be plotted vertically or horizontally. For a horizontal bar chart, use the Horizontal bar chart chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select one or more columns for the Y-axis
Type: List[str]
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/chartjs/#horizontal-bar-chart","title":"Horizontal bar chart","text":"
The horizontal bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars are plotted horizontally.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The line chart is a chart that displays information as a series of data points called 'markers' connected by straight line segments. It is useful for showing trends over time.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The pie chart is a circular statistical graphic that is divided into slices to illustrate numerical proportions. The arc length of each slice is proportional to the quantity it represents.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The scatter plot is a chart that uses Cartesian coordinates to display values for two variables for a set of data. The data points are represented as individual dots.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The radar chart is a chart that displays multivariate data in the form of a two-dimensional chart of three or more quantitative variables represented on axes starting from the same point. The data points are connected by a line to form a polygon.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
Names (names) [Required]
Type: str
Values (values) [Required]
Select 3 or more different categorical variables (dimensions)
Type: List[str]
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/custom/","title":"Custom chart engine","text":""},{"location":"engines/custom/#implementing-new-chart-engines-support","title":"Implementing new chart engines support","text":"
Implementing support for a new chart engine includes multiple steps and changes in Python, HTML, and JavaScript. Starting from the Python code:
Create a new builder class at ckanext.charts.chart_builder that inherits from BaseChartBuilder and implements the get_supported_forms method. This method must return a list of classes that represent supported chart types forms.
Each form type builder must be connected with a respective chart type builder.
The chart type builder must implement a to_json method that will return a dumped JSON data, which will be passed to a JS script.
The form type builder must implement a get_form_fields method that will return a list of all form fields that will be rendered for the user, allowing them to provide all the necessary information for a chart.
Register your chart engine by adding the builder class to get_chart_engines in ckanext.charts.chart_builder.__init__.py.
A full example of an implementation of bar chart for obvervable plot library.
from __future__ import annotations\n\nimport json\nfrom typing import Any\n\nimport ckanext.charts.exception as exception\nfrom ckanext.charts.chart_builders.base import BaseChartBuilder, BaseChartForm\n\n\nclass ObservableBuilder(BaseChartBuilder):\n @classmethod\n def get_supported_forms(cls) -> list[type[Any]]:\n return [ObservableBarForm]\n\n\nclass ObservableBarBuilder(ObservableBuilder):\n def to_json(self) -> str:\n return json.dumps(\n {\n \"type\": \"bar\",\n \"data\": self.df.to_dict(orient=\"records\"),\n \"settings\": self.settings,\n }\n )\n\n\nclass ObservableBarForm(BaseChartForm):\n name = \"Bar\"\n builder = ObservableBarBuilder\n\n def fill_field(self, choices: list[dict[str, str]]) -> dict[str, str]:\n field = self.color_field(choices)\n field.update({\"field_name\": \"fill\", \"label\": \"Fill\"})\n\n return field\n\n def get_form_fields(self):\n columns = [{\"value\": col, \"label\": col} for col in self.df.columns]\n chart_types = [\n {\"value\": form.name, \"label\": form.name}\n for form in self.builder.get_supported_forms()\n ]\n\n return [\n self.title_field(),\n self.description_field(),\n self.engine_field(),\n self.type_field(chart_types),\n self.x_axis_field(columns),\n self.y_axis_field(columns),\n self.fill_field(columns),\n self.opacity_field(),\n self.limit_field(),\n ]\n
"},{"location":"engines/custom/#vendor-and-custom-js","title":"Vendor and custom JS","text":"
Another step is to register JS/CSS vendor libraries of the chart you want to use. Refer to CKAN documentation to read about adding CSS and JavaScript files using Webassets.
You also will need a CKAN JS module, that will be responsible for rendering the Chart. This module will work with the vendor library and will be responsible for rendering the chart in the container.
This module must be registered inside a webassets.yml as well.
ckan.module(\"charts-render-observable\", function ($, _) {\n \"use strict\";\n\n return {\n options: {\n config: null\n },\n\n initialize: function () {\n $.proxyAll(this, /_/);\n\n if (!this.options.config) {\n console.error(\"No configuration provided\");\n return;\n }\n\n var plot;\n\n switch (this.options.config.type) {\n case \"bar\":\n plot = Plot.barY(this.options.config.data, this.options.config.settings).plot();\n break;\n default:\n return;\n }\n\n this.el[0].replaceChildren(plot);\n }\n };\n });\n
Note, that we should add {% asset \"charts/observable\" %} not only here, but in charts_form.html too.
The reason for having a separate HTML file and JS module is that different libraries may require different types of container elements (such as div, canvas, etc.) to initialize or may need additional boilerplate code to build a chart. There's no easy way to abstract this, so you have to implement these things yourself.
The bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars can be plotted vertically or horizontally. For a horizontal bar chart, use the Horizontal bar chart chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Fill (fill)
Select a column for the color
Type: str
Opacity (opacity)
Opacity level of the chart
Type: float
Default: 1
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/observable/#horizontal-bar-chart","title":"Horizontal bar chart","text":"
The horizontal bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars are plotted horizontally.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The line chart is a chart that displays information as a series of data points called 'markers' connected by straight line segments. It is useful for showing trends over time.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The pie chart is a circular statistical graphic that is divided into slices to illustrate numerical proportions. The arc length of each slice is proportional to the quantity it represents.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The scatter plot is a chart that uses Cartesian coordinates to display values for two variables for a set of data. The data points are represented as individual dots.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The auto chart is a chart that automatically selects the best chart type based on the data. It is useful for quickly visualizing data without having to manually select a chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars can be plotted vertically or horizontally. For a horizontal bar chart, use the Horizontal bar chart chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
Type: bool
Color (color)
Select a column for the color
Type: str
Animation Frame (animation_frame)
Select a column for the animation frame
Type: str
Opacity (opacity)
Opacity level of the chart
Type: float
Default: 1
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/plotly/#horizontal-bar-chart","title":"Horizontal bar chart","text":"
The horizontal bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars are plotted horizontally.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
The line chart is a chart that displays information as a series of data points called 'markers' connected by straight line segments. It is useful for showing trends over time.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
The pie chart is a circular statistical graphic that is divided into slices to illustrate numerical proportions. The arc length of each slice is proportional to the quantity it represents.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
The scatter plot is a chart that uses Cartesian coordinates to display values for two variables for a set of data. The data points are represented as individual dots.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
Fetchers are responsible for fetching data from different sources (DataStore, URL, file system, hardcoded data).
For the current implementation, we're working with resources that are uploaded to the DataStore, so the fetcher will be responsible for fetching the data from the DataStore.
But it might come in handy to have fetchers for other sources, like URL, file system, etc.
"},{"location":"fetchers/#implementing-new-fetchers","title":"Implementing new fetchers","text":"
To register new fetchers, you need to create a new class that inherits from DataFetcherStrategy and implements the fetch_data and make_cache_key methods.
The fetch_data method should return a pandasDataFrame object with the data that should be displayed in the chart.
The make_cache_key method should return a unique string that will be used as a key to store the data in the cache.
See the base class DataFetcherStrategy for more information.
All fetchers must inherit from the DataFetcherStrategy class. This class provides the basic structure for fetchers and defines the methods that must be implemented by all fetchers.
Make a request to the URL and return the response content.
RETURNS DESCRIPTION bytes
The response content
TYPE: bytes
RAISES DESCRIPTION DataFetchError
If an error occurs during the request
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-\\.\\_]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#ckanext-charts","title":"ckanext-charts","text":"
This extension, ckanext-charts, provides additional functionality for working with charts in CKAN. It allows users to create, manage, and visualize charts based on data stored in CKAN datasets.
The extension includes features such as chart creation, chart editing, chart embedding, and chart sharing. It also supports various chart types, including bar charts, line charts, pie charts, and more.
With ckanext-charts, users can easily generate interactive and visually appealing charts to enhance data analysis and presentation in CKAN.
We need to set the CHARTS_FIELDS for our custom handler, that autogenerate documentation for chart types fields.
If you're working on the documentation, you can run the following command to start a live-reloading server without gathering the chart types fields data. It will speed up the process significantly, as we won't need to wait for the CKAN initialization:
There is a number of configuration options available for the extension. You can set them in the CKAN configuration file or using the admin configuration page.
If the cache is disabled, the chart data will be fetched from the data source every time the chart is rendered. It could lead to performance issues if the data source is slow or the data is too large.
The extension provides an admin configuration page where you can set all the listed configuration options. The admin page available only using an extension ckanext-admin-panel. The admin panel is a separate extension that provides an alternative admin interface for CKAN. It allows you to manage CKAN settings and other extensions settings through the web interface.
This page could be accessed by the following URL: /admin-panel/charts/config, but it will be registered only if the ckanext-admin-panel is installed and enabled.
Check if anonymous users are allowed to build charts.
RETURNS DESCRIPTION bool
True if anonymous users are allowed to build charts, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_allow_anon_building_charts() -> bool:\n \"\"\"Check if anonymous users are allowed to build charts.\n\n Returns:\n bool: True if anonymous users are allowed to build charts, False otherwise.\n \"\"\"\n return config.allow_anon_building_charts()\n
def charts_get_resource_columns(resource_id: str) -> str:\n \"\"\"Get the columns of the given resource.\n\n Args:\n resource_id: Resource ID\n\n Returns:\n str: JSON string of columns options\n \"\"\"\n fetcher = DatastoreDataFetcher(resource_id)\n\n return json.dumps(\n [{\"id\": col, \"title\": col} for col in fetcher.fetch_data().columns],\n )\n
True if the HTMX asset should be included, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_include_htmx_asset() -> bool:\n \"\"\"Checks if the HTMX asset should be included.\n\n Returns:\n bool: True if the HTMX asset should be included, False otherwise.\n \"\"\"\n return config.include_htmx_asset()\n
Checks if CKAN JS modules should be reinitialized.
RETURNS DESCRIPTION bool
True if CKAN JS modules should be reinitialized, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_reinit_ckan_js_modules() -> bool:\n \"\"\"Checks if CKAN JS modules should be reinitialized.\n\n Returns:\n bool: True if CKAN JS modules should be reinitialized, False otherwise.\n \"\"\"\n return config.reinit_ckan_js_modules()\n
True if the user is authenticated, False otherwise.
TYPE: bool
Source code in ckanext/charts/helpers.py
def charts_user_is_authenticated() -> bool:\n \"\"\"Check if the user is authenticated.\n\n Returns:\n bool: True if the user is authenticated, False otherwise.\n \"\"\"\n return tk.current_user.is_authenticated\n
def get_available_chart_engines_options() -> list[dict[str, str]]:\n \"\"\"Get the available chart engines.\n\n Returns:\n List of chart engines options\n \"\"\"\n return [{\"value\": engine, \"text\": engine} for engine in get_chart_engines()]\n
Get the size of the file cache in a human-readable format.
RETURNS DESCRIPTION str
Human-readable file cache size
TYPE: str
Source code in ckanext/charts/helpers.py
def get_file_cache_size() -> str:\n \"\"\"Get the size of the file cache in a human-readable format.\n\n Returns:\n str: Human-readable file cache size\n \"\"\"\n return utils.printable_file_size(count_file_cache_size())\n
Get the size of the Redis cache in a human-readable format.
RETURNS DESCRIPTION str
Human-readable Redis cache size
TYPE: str
Source code in ckanext/charts/helpers.py
def get_redis_cache_size() -> str:\n \"\"\"Get the size of the Redis cache in a human-readable format.\n\n Returns:\n str: Human-readable Redis cache size\n \"\"\"\n return utils.printable_file_size(count_redis_cache_size())\n
The extension requires the following CKAN extensions to be installed and enabled:
ckanext-scheming: We're using the scheming extension to create custom forms for the chart builders.
ckanext-admin-panel (optional): If you want to use the admin configuration page, you need to install and enable this extension. The admin panel is a separate extension that provides an alternative admin interface for CKAN. It allows you to manage CKAN settings and other extensions settings through the web interface and significantly extends the default CKAN admin interface.
This extension offers two plugins: charts_view and charts_builder_view. The first plugin is a standard chart builder designed for system administrators or anyone with permission to edit or create a resource view. Creating a preview using this plugin generates a static chart with saved settings. This chart retains the settings and will always be rendered in the same way.
The charts_builder_view plugin does not have any special settings during the creation or editing stages. Instead, it renders a chart builder JS widget on a resource page, allowing any user to interact with the resource data and create a chart that meets their needs. In the future, we plan to implement a feature that will allow users to save their charts.
"},{"location":"usage/#restrict-user-chart-builder-for-anonymous-users","title":"Restrict user chart builder for anonymous users","text":"
The user chart builder could be restricted to only authenticated users. See the config settings section for more information. In this case, user will be forced to log in to build a chart. After the user logs in, user will be redirected to the chart builder page back.
Uses a DatastoreDataFetcher to fetch data from the resource.
PARAMETER DESCRIPTION settings
Chart settings
TYPE: dict[str, Any]
resource_id
Resource ID
TYPE: str
RETURNS DESCRIPTION str | None
str | None: Chart config as JSON string or None if the chart can't be built
Source code in ckanext/charts/utils.py
def build_chart_for_resource(settings: dict[str, Any], resource_id: str) -> str | None:\n \"\"\"Build chart for the given resource ID.\n\n Uses a DatastoreDataFetcher to fetch data from the resource.\n\n Args:\n settings: Chart settings\n resource_id: Resource ID\n\n Returns:\n str | None: Chart config as JSON string or None if the chart can't be built\n \"\"\"\n settings.pop(\"__extras\", None)\n\n try:\n df = DatastoreDataFetcher(resource_id).fetch_data()\n except tk.ValidationError:\n return None\n\n return _build_chart(settings, df)\n
For now, we work only with resources stored with the DataStore.
PARAMETER DESCRIPTION data_dict
Resource data dictionary
TYPE: dict[str, Any]
RETURNS DESCRIPTION bool
True if the resource can be viewed as a chart, False otherwise
TYPE: bool
Source code in ckanext/charts/utils.py
def can_view(data_dict: dict[str, Any]) -> bool:\n \"\"\"Check if the resource can be viewed as a chart.\n\n For now, we work only with resources stored with the DataStore.\n\n Args:\n data_dict: Resource data dictionary\n\n Returns:\n bool: True if the resource can be viewed as a chart, False otherwise\n \"\"\"\n if data_dict[\"resource\"].get(\"datastore_active\"):\n return True\n\n # TODO: Add support for XML, XLS, XLSX, and other formats tabular data?\n # if data_dict[\"resource\"][\"format\"].lower() == \"xml\":\n # return True\n\n return False\n
Get form builder for the given engine and chart type.
Source code in ckanext/charts/utils.py
def get_chart_form_builder(engine: str, chart_type: str):\n \"\"\"Get form builder for the given engine and chart type.\"\"\"\n builders = get_chart_engines()\n\n if engine not in builders:\n raise NotImplementedError(f\"Engine {engine} is not supported\")\n\n return builders[engine].get_form_for_type(chart_type)\n
def get_column_options(resource_id: str) -> list[dict[str, str]]:\n \"\"\"Get column options for the given resource.\n\n Args:\n resource_id: Resource ID\n\n Returns:\n List of column options\n \"\"\"\n df = DatastoreDataFetcher(resource_id).fetch_data()\n\n return [{\"text\": col, \"value\": col} for col in df.columns]\n
A custom version of if_empty_same_as validator for charts.
This validator is used to set the value of a field to the value of another field if it is empty or missing.
PARAMETER DESCRIPTION other_key
The key of the field to copy the value from
TYPE: str
RETURNS DESCRIPTION Callable[..., Any]
Callable[..., Any]: The validator function
Source code in ckanext/charts/logic/validators.py
def charts_if_empty_same_as(other_key: str) -> Callable[..., Any]:\n \"\"\"A custom version of if_empty_same_as validator for charts.\n\n This validator is used to set the value of a field to the value of another\n field if it is empty or missing.\n\n Args:\n other_key (str): The key of the field to copy the value from\n\n Returns:\n Callable[..., Any]: The validator function\n \"\"\"\n\n def callable(key, data, errors, context):\n value = data.get(key)\n if not value or value is tk.missing:\n try:\n data[key] = data[key[:-1] + (other_key,)]\n except KeyError:\n data[key] = data.get((\"__extras\",), {}).get(other_key, \"\")\n\n return callable\n
def charts_list_length_validator(max_length: int) -> Callable[..., Any]:\n \"\"\"A validator to check the length of a list.\n\n Args:\n max_length (int): The maximum length of the list\n\n Returns:\n Callable[..., Any]: The validator function\n \"\"\"\n def callable(\n key: types.FlattenKey,\n data: types.FlattenDataDict,\n errors: types.FlattenErrorDict,\n context: types.Context,\n ):\n if len(data[key]) > max_length:\n raise tk.Invalid(tk._(\"Length must be less than {0}\").format(max_length))\n\n return callable\n
def charts_list_to_csv(data: list[str] | str) -> str:\n \"\"\"Convert a list of strings to a CSV string.\n\n Args:\n data (list[str] | str): The data to convert\n\n Returns:\n str: The comma separated string\n \"\"\"\n if not isinstance(data, list):\n return data\n\n return \", \".join(data)\n
def charts_strategy_support(strategy: str) -> str:\n \"\"\"Check if the cache strategy is supported.\n\n Args:\n strategy (str): The cache strategy\n\n Returns:\n str: The cache strategy if it is supported\n\n Raises:\n tk.Invalid: If the cache strategy is not supported\n \"\"\"\n if strategy not in const.SUPPORTED_CACHE_STRATEGIES:\n raise tk.Invalid(tk._(\"Invalid cache strategy\"))\n\n if strategy == const.CACHE_FILE_ORC:\n try:\n from pyarrow import orc as _ # noqa\n except ImportError:\n raise tk.Invalid(\n tk._(\"Can't use File Orc cache strategy. PyArrow is not installed\"),\n ) from None\n\n if not strategy:\n return const.DEFAULT_CACHE_STRATEGY\n\n return strategy\n
def charts_to_list_if_string(value: Any) -> Any:\n \"\"\"Convert a string to a list.\n\n Args:\n value (Any): The value to convert\n\n Returns:\n list[Any]: The value in a list\n \"\"\"\n if isinstance(value, str):\n return [value]\n\n return value\n
Validate charts settings according to the chart type and engine schema.
PARAMETER DESCRIPTION key
The key of the field
TYPE: FlattenKey
data
The data to validate
TYPE: FlattenDataDict
errors
The errors dict
TYPE: FlattenErrorDict
context
The context
TYPE: Context
Source code in ckanext/charts/logic/validators.py
def charts_validate_extras(\n key: types.FlattenKey,\n data: types.FlattenDataDict,\n errors: types.FlattenErrorDict,\n context: types.Context,\n):\n \"\"\"Validate charts settings according to the chart type and engine schema.\n\n Args:\n key (types.FlattenKey): The key of the field\n data (types.FlattenDataDict): The data to validate\n errors (types.FlattenErrorDict): The errors dict\n context (types.Context): The context\n \"\"\"\n settings = _extract_setting(data)\n\n if \"engine\" not in settings or \"type\" not in settings:\n builder = DEFAULT_CHART_FORM\n else:\n builder = utils.get_chart_form_builder(settings[\"engine\"], settings[\"type\"])\n\n settings, err = tk.navl_validate(\n settings,\n builder(settings[\"resource_id\"]).get_validation_schema(\n context.get(\"_for_show\", False),\n ),\n {},\n )\n\n # TODO: do we have a better way to handle this? Seems like a hack\n for k, v in settings.items():\n data[(k,)] = v\n\n for k, v in settings.pop(\"__extras\", {}).items():\n data[(k,)] = v\n\n for k, v in err.items():\n errors[(k,)] = v\n
def float_validator(value: Any) -> float:\n \"\"\"A validator for decimal numbers.\n\n Args:\n value (Any): The value to validate\n\n Returns:\n float: The value as a float\n \"\"\"\n try:\n return float(value)\n except ValueError:\n raise tk.Invalid(tk._(\"Must be a decimal number\")) from None\n
The file cache works by storing the data in an orc or csv file in the filesystem. The redis cache stores the data in a Redis database. The cache strategy can be changed at the CKAN configuration level through the admin interface or in a configuration file.
The file-type cache strategy stores the data in a file in the filesystem. The file cache is stored in the ckanext-charts directory in the CKAN storage path. The file cache is stored in an orc or csv file format.
Warning
Using file_orc cache strategy requires the pyarrow python library to be installed.
The cache TTL can be set in the CKAN configuration file. The default value is 3600 seconds (1 hour). The cache TTL can be set to 0 to disable the cache.
The redis and file-type cache has separate TTL settings:
The redis cache TTL can be set with the ckanext.charts.redis_cache_ttl configuration option.
The file cache TTL can be set with the ckanext.charts.file_cache_ttl configuration option.
Cache could be disabled by setting ckanext.charts.enable_cache to false. In this case the data will be fetched from the source every time the chart is rendered. It could be useful for debugging purposes. But using it in production is not recommended, as it could lead to performance issues.
To implement a custom cache strategy, you need to create a new class that extends the CacheStrategy class and implement the abstract methods.
See a naive implementation of a memcached cache strategy below:
from io import BytesIO\n\nfrom pymemcache.client import base\nimport pandas as pd\n\nimport ckanext.charts.config as config\nfrom ckanext.charts.cache import CacheStrategy\n\nclass MemcachedCache(CacheStrategy):\n \"\"\"Cache data to Memcached\"\"\"\n\n def __init__(self):\n self.client = base.Client(('localhost', 11211))\n\n def get_data(self, key: str) -> pd.DataFrame | None:\n \"\"\"Return data from cache if exists\"\"\"\n try:\n raw_data = self.client.get(key)\n\n if not raw_data:\n return None\n\n return pd.read_csv(BytesIO(raw_data))\n except Exception:\n log.exception(f\"Failed to get data for key: {key}\")\n return None\n\n def set_data(self, key: str, data: pd.DataFrame):\n \"\"\"Serialize data and save to Memcached\"\"\"\n cache_ttl = config.get_memcached_cache_ttl()\n\n try:\n serialized_data = data.to_csv(index=False).encode('utf-8')\n self.client.set(key, serialized_data, expire=cache_ttl)\n except Exception:\n log.exception(f\"Failed to save data to Memcached for key: {key}\")\n\n def invalidate(self, key: str):\n \"\"\"Invalidate cache by key\"\"\"\n try:\n self.client.delete(key)\n except Exception:\n log.exception(f\"Failed to invalidate cache for key: {key}\")\n
Bases: ABC
Cache strategy interface.
Defines the abstracts methods for cache strategies.
The ckanext-charts implements supports different chart engines, such as plotly, observable and chartjs.
Each engine consists of two classes - chart builder and form builder. The chart builder is responsible for generating a JSON-string representation of the chart data, that will be passed to a respective JS module, that will render a chart based on the data.
The form builder is responsible for generating a form that allows users to configure the chart from the UI.
Also, you can implement support of a chart library of your choice by creating a custom chart engine. Read more about it here.
The bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars can be plotted vertically or horizontally. For a horizontal bar chart, use the Horizontal bar chart chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select one or more columns for the Y-axis
Type: List[str]
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/chartjs/#horizontal-bar-chart","title":"Horizontal bar chart","text":"
The horizontal bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars are plotted horizontally.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The line chart is a chart that displays information as a series of data points called 'markers' connected by straight line segments. It is useful for showing trends over time.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The pie chart is a circular statistical graphic that is divided into slices to illustrate numerical proportions. The arc length of each slice is proportional to the quantity it represents.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The scatter plot is a chart that uses Cartesian coordinates to display values for two variables for a set of data. The data points are represented as individual dots.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
The radar chart is a chart that displays multivariate data in the form of a two-dimensional chart of three or more quantitative variables represented on axes starting from the same point. The data points are connected by a line to form a polygon.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Doughnut, Scatter, Bubble, Radar
Names (names) [Required]
Type: str
Values (values) [Required]
Select 3 or more different categorical variables (dimensions)
Type: List[str]
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/custom/","title":"Custom chart engine","text":""},{"location":"engines/custom/#implementing-new-chart-engines-support","title":"Implementing new chart engines support","text":"
Implementing support for a new chart engine includes multiple steps and changes in Python, HTML, and JavaScript. Starting from the Python code:
Create a new builder class at ckanext.charts.chart_builder that inherits from BaseChartBuilder and implements the get_supported_forms method. This method must return a list of classes that represent supported chart types forms.
Each form type builder must be connected with a respective chart type builder.
The chart type builder must implement a to_json method that will return a dumped JSON data, which will be passed to a JS script.
The form type builder must implement a get_form_fields method that will return a list of all form fields that will be rendered for the user, allowing them to provide all the necessary information for a chart.
Register your chart engine by adding the builder class to get_chart_engines in ckanext.charts.chart_builder.__init__.py.
A full example of an implementation of bar chart for obvervable plot library.
from __future__ import annotations\n\nimport json\nfrom typing import Any\n\nimport ckanext.charts.exception as exception\nfrom ckanext.charts.chart_builders.base import BaseChartBuilder, BaseChartForm\n\n\nclass ObservableBuilder(BaseChartBuilder):\n @classmethod\n def get_supported_forms(cls) -> list[type[Any]]:\n return [ObservableBarForm]\n\n\nclass ObservableBarBuilder(ObservableBuilder):\n def to_json(self) -> str:\n return json.dumps(\n {\n \"type\": \"bar\",\n \"data\": self.df.to_dict(orient=\"records\"),\n \"settings\": self.settings,\n }\n )\n\n\nclass ObservableBarForm(BaseChartForm):\n name = \"Bar\"\n builder = ObservableBarBuilder\n\n def fill_field(self, choices: list[dict[str, str]]) -> dict[str, str]:\n field = self.color_field(choices)\n field.update({\"field_name\": \"fill\", \"label\": \"Fill\"})\n\n return field\n\n def get_form_fields(self):\n columns = [{\"value\": col, \"label\": col} for col in self.df.columns]\n chart_types = [\n {\"value\": form.name, \"label\": form.name}\n for form in self.builder.get_supported_forms()\n ]\n\n return [\n self.title_field(),\n self.description_field(),\n self.engine_field(),\n self.type_field(chart_types),\n self.x_axis_field(columns),\n self.y_axis_field(columns),\n self.fill_field(columns),\n self.opacity_field(),\n self.limit_field(),\n ]\n
"},{"location":"engines/custom/#vendor-and-custom-js","title":"Vendor and custom JS","text":"
Another step is to register JS/CSS vendor libraries of the chart you want to use. Refer to CKAN documentation to read about adding CSS and JavaScript files using Webassets.
You also will need a CKAN JS module, that will be responsible for rendering the Chart. This module will work with the vendor library and will be responsible for rendering the chart in the container.
This module must be registered inside a webassets.yml as well.
ckan.module(\"charts-render-observable\", function ($, _) {\n \"use strict\";\n\n return {\n options: {\n config: null\n },\n\n initialize: function () {\n $.proxyAll(this, /_/);\n\n if (!this.options.config) {\n console.error(\"No configuration provided\");\n return;\n }\n\n var plot;\n\n switch (this.options.config.type) {\n case \"bar\":\n plot = Plot.barY(this.options.config.data, this.options.config.settings).plot();\n break;\n default:\n return;\n }\n\n this.el[0].replaceChildren(plot);\n }\n };\n });\n
Note, that we should add {% asset \"charts/observable\" %} not only here, but in charts_form.html too.
The reason for having a separate HTML file and JS module is that different libraries may require different types of container elements (such as div, canvas, etc.) to initialize or may need additional boilerplate code to build a chart. There's no easy way to abstract this, so you have to implement these things yourself.
The bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars can be plotted vertically or horizontally. For a horizontal bar chart, use the Horizontal bar chart chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Fill (fill)
Select a column for the color
Type: str
Opacity (opacity)
Opacity level of the chart
Type: float
Default: 1
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/observable/#horizontal-bar-chart","title":"Horizontal bar chart","text":"
The horizontal bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars are plotted horizontally.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The line chart is a chart that displays information as a series of data points called 'markers' connected by straight line segments. It is useful for showing trends over time.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The pie chart is a circular statistical graphic that is divided into slices to illustrate numerical proportions. The arc length of each slice is proportional to the quantity it represents.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The scatter plot is a chart that uses Cartesian coordinates to display values for two variables for a set of data. The data points are represented as individual dots.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The auto chart is a chart that automatically selects the best chart type based on the data. It is useful for quickly visualizing data without having to manually select a chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Line, Pie, Scatter, Auto
The bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars can be plotted vertically or horizontally. For a horizontal bar chart, use the Horizontal bar chart chart type.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
Type: bool
Color (color)
Select a column for the color
Type: str
Animation Frame (animation_frame)
Select a column for the animation frame
Type: str
Opacity (opacity)
Opacity level of the chart
Type: float
Default: 1
Limit (limit)
Limit the number of rows to show in the chart
Type: int
Default: 100
"},{"location":"engines/plotly/#horizontal-bar-chart","title":"Horizontal bar chart","text":"
The horizontal bar chart is a chart with rectangular bars with lengths proportional to the values that they represent. The bars are plotted horizontally.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
The line chart is a chart that displays information as a series of data points called 'markers' connected by straight line segments. It is useful for showing trends over time.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
The pie chart is a circular statistical graphic that is divided into slices to illustrate numerical proportions. The arc length of each slice is proportional to the quantity it represents.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
The scatter plot is a chart that uses Cartesian coordinates to display values for two variables for a set of data. The data points are represented as individual dots.
Title (title)
Title of the chart view
Type: str
Description (description)
Description of the chart view
Type: str
Engine (engine) [Required]
Select the chart engine to use
Type: str
Default: plotly
Choices: plotly, observable, chartjs
Type (type) [Required]
Select the type of the chart, e.g. Line, Bar, Scatter, etc.
Type: str
Default: Line
Choices: Bar, Horizontal Bar, Pie, Line, Scatter
X Axis (x) [Required]
Select a column for the X-axes
Type: str
Y Axis (y) [Required]
Select a column for the Y-axis
Type: str
Log-scale X-axis (log_x)
Use log scale for the X-axis
Type: bool
Log-scale Y-axis (log_y)
Use log scale for the Y-axis
Type: bool
Sort X-axis (sort_x)
Sort the X-axis values
Type: bool
Sort Y-axis (sort_y)
Sort the Y-axis values
Type: bool
Skip N/A and NULL values (skip_null_values)
Entries of the data with missing values will not be graphed or will be skipped
Fetchers are responsible for fetching data from different sources (DataStore, URL, file system, hardcoded data).
For the current implementation, we're working with resources that are uploaded to the DataStore, so the fetcher will be responsible for fetching the data from the DataStore.
But it might come in handy to have fetchers for other sources, like URL, file system, etc.
"},{"location":"fetchers/#implementing-new-fetchers","title":"Implementing new fetchers","text":"
To register new fetchers, you need to create a new class that inherits from DataFetcherStrategy and implements the fetch_data and make_cache_key methods.
The fetch_data method should return a pandasDataFrame object with the data that should be displayed in the chart.
The make_cache_key method should return a unique string that will be used as a key to store the data in the cache.
See the base class DataFetcherStrategy for more information.
All fetchers must inherit from the DataFetcherStrategy class. This class provides the basic structure for fetchers and defines the methods that must be implemented by all fetchers.