From 9b3fc8e34ba8ca15e04b31a75c3a30a8cc42dde9 Mon Sep 17 00:00:00 2001 From: pseusys Date: Fri, 13 Oct 2023 12:06:16 +0200 Subject: [PATCH 01/18] user functions guide --- docs/source/user_guides/user_functions.rst | 48 ++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 docs/source/user_guides/user_functions.rst diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst new file mode 100644 index 000000000..a9cb24c36 --- /dev/null +++ b/docs/source/user_guides/user_functions.rst @@ -0,0 +1,48 @@ +User functions guide +-------------------- + +Overview +~~~~~~~~ + +Dialog flow franework allows user to define custom functions for implementaing custom behavior +in several aspects. +This tutorial summarizes the custom functions use cases, specifies their arguments and return +types, warns about several common exception handling. + +``Actor`` handlers +~~~~~~~~~~~~~~~~~~ + +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ constructor accepts ``handlers`` +parameter, that is either ``None`` or dictionary attributing lists of functions to different +`ActorStage <../api/dff.script.core.types#ActorStage>`_ values. + +These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`_ +lifecycle. +Each of these functions have to have the following signature: + +.. code-block:: python + + def handler(ctx: Context, pipeline: Pipeline) -> Any: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value can be anything (it is not used). + +Pre-transition processors +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each script `Node <../api/dff.script.core.script#Node>`_ has a property called ``pre_transitions_processing``. +That is a dictionary, associating functions to their names (that can be any hashable object). + +These functions are run before transition from the previous node to the current node. +Each of these functions have to have the following signature: + +.. code-block:: python + + def handler(ctx: Context, pipeline: Pipeline) -> Context: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value is the modified Context value. From 281497d5498b5cb1d8b79583967f7a3b31fbbc87 Mon Sep 17 00:00:00 2001 From: pseusys Date: Tue, 17 Oct 2023 05:19:54 +0200 Subject: [PATCH 02/18] actor user defined functions done --- dff/pipeline/pipeline/actor.py | 2 +- docs/source/user_guides.rst | 12 +- docs/source/user_guides/user_functions.rst | 122 ++++++++++++++++++++- 3 files changed, 131 insertions(+), 5 deletions(-) diff --git a/dff/pipeline/pipeline/actor.py b/dff/pipeline/pipeline/actor.py index 1bd5878b0..850621cbf 100644 --- a/dff/pipeline/pipeline/actor.py +++ b/dff/pipeline/pipeline/actor.py @@ -392,7 +392,7 @@ def validate_script(self, pipeline: Pipeline, verbose: bool = True): def default_condition_handler( condition: Callable, ctx: Context, pipeline: Pipeline, *args, **kwargs -) -> Callable[[Context, Pipeline, Any, Any], bool]: +) -> bool: """ The simplest and quickest condition handler for trivial condition handling returns the callable condition: diff --git a/docs/source/user_guides.rst b/docs/source/user_guides.rst index 8724a1489..0db776df3 100644 --- a/docs/source/user_guides.rst +++ b/docs/source/user_guides.rst @@ -17,9 +17,17 @@ for exploring the telemetry data collected from your conversational services. We show how to plug in the telemetry collection and configure the pre-built Superset dashboard shipped with DFF. +:doc:`User defined functions guide <./user_guides/user_functions>` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``user defined functions guide`` tutorial describes useage of all user-defined functions +that can be used for framework extension. +For each of these functions use cases are defined, execution orser is specified +and also signature and return value types are highlighted. + .. toctree:: :hidden: + :glob: - user_guides/basic_conceptions - user_guides/superset_guide + user_guides/* diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index a9cb24c36..4ce38a957 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -18,7 +18,7 @@ parameter, that is either ``None`` or dictionary attributing lists of functions These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`_ lifecycle. -Each of these functions have to have the following signature: +Each of these functions has the following signature: .. code-block:: python @@ -36,7 +36,7 @@ Each script `Node <../api/dff.script.core.script#Node>`_ has a property called ` That is a dictionary, associating functions to their names (that can be any hashable object). These functions are run before transition from the previous node to the current node. -Each of these functions have to have the following signature: +Each of these functions has the following signature: .. code-block:: python @@ -46,3 +46,121 @@ Each of these functions have to have the following signature: where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value is the modified Context value. + +Pre-response processors +~~~~~~~~~~~~~~~~~~~~~~~ + +Each script `Node <../api/dff.script.core.script#Node>`_ has a property called ``pre_response_processing``. +That is a dictionary, associating functions to their names (that can be any hashable object). + +These functions are run before acquiring the response, after the current node is processed. +Each of these functions has the following signature: + +.. code-block:: python + + def handler(ctx: Context, pipeline: Pipeline) -> Context: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value is the modified Context value. + +Script conditions and condition handlers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each transition between different nodes in `Script <../api/dff.script.core.script#Script>`_ +has a condition function. +They are provided in script dictionary (TODO: parser?) in form of Python functions. + +These functions are executed on pipeline startup if ``validation_stage`` parameter of +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and before transition +from the corresponding node in script in runtime. +Each of these functions has the following signature: + +.. code-block:: python + + def condition(ctx: Context, pipeline: Pipeline) -> bool: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value is ``True`` if transition should be made and ``False`` otherwise. + +There is a set of `standard condition functions <../api/dff.script.conditions.std_conditions>`_ defined. + +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ constructor also accepts +condition handler - that is a special function that executes conditions. + +This function is invoked every time condition should be checked, it launches and checks condition. +This function has the following signature: + +.. code-block:: python + + def condition_handler(condition: Callable[[Context, Pipeline], bool], ctx: Context, pipeline: Pipeline) -> bool: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value is ``True`` if transition should be made and ``False`` otherwise. + +The simplest `default condition handler <../api/dff.pipeline.pipeline.actor#default_condition_handler>`_ +just invokes the condition function and returns the result. + +Labels +~~~~~~ + +Some of the transitions between nodes in `Script <../api/dff.script.core.script#Script>`_ +do not have "absolute" node targets specified. +For instance, that might be useful in case it is required to stay in the same node or transition +to the previous node. +For such cases special function node labels can be used. + +These functions are executed on pipeline startup if ``validation_stage`` parameter of +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and before transition +from the corresponding node in script in runtime. +Each of these functions has the following signature: + +.. code-block:: python + + def label(ctx: Context, pipeline: Pipeline) -> Tuple[str, str, float]: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value is an instance of `NodeLabel3Type <../api/dff.script.core.types#NodeLabel3Type>`, +that is a tuple of target flow name (``str``), node name (``str``) and priority (``float``). + +There is a set of `standard label functions <../api/dff.script.conditions.std_labels>`_ defined. + +Responses +~~~~~~~~~ + +For some of the nodes in `Script <../api/dff.script.core.script#Script>`_ returning constant values +might be not enough. +For these cases each return value can be represented as a Python function. + +These functions are executed on pipeline startup if ``validation_stage`` parameter of +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and in the end +of any node processing in runtime. +Each of these functions has the following signature: + +.. code-block:: python + + def response(ctx: Context, pipeline: Pipeline) -> Message: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value is an instance of `Message <../api/dff.script.core.message#Message>`. + +Extra handlers +~~~~~~~~~~~~~~ + +Service handlers +~~~~~~~~~~~~~~~~ + +Service conditions +~~~~~~~~~~~~~~~~~~ + +Statistics extractors +~~~~~~~~~~~~~~~~~~~~~ From 37232aaa3c8accbb32fa3930da034c0eb4e1e427 Mon Sep 17 00:00:00 2001 From: pseusys Date: Wed, 18 Oct 2023 10:14:41 +0200 Subject: [PATCH 03/18] pipeline functions done --- docs/source/user_guides/user_functions.rst | 71 ++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 4ce38a957..b07a8d066 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -156,11 +156,82 @@ and the return value is an instance of `Message <../api/dff.script.core.message# Extra handlers ~~~~~~~~~~~~~~ +For some (or all) services in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ special +extra handler functions can be added. +These functions can handle statistics collection, input data transformation +or other pipeline functionality extension. + +These functions can be either added to `pipeline dict <../api/dff.pipeline.types#PipelineBuilder>`_ +or added to all services at once with `add_global_handler <../api/dff.pipeline.pipeline.pipeline#add_global_handler>`_ +function. +The handlers can be executed before or after pipeline services. +Each of them has one of the following signatures: + +.. code-block:: python + + async def handler(ctx: Context) -> Any: + ... + + async def handler(ctx: Context, pipeline: Pipeline) -> Any: + ... + + async def handler(ctx: Context, pipeline: Pipeline, runtime_info: Dict) -> Any: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_, +where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_ +and the return value can be anything (it is not used). + Service handlers ~~~~~~~~~~~~~~~~ +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_) +should be represented as functions. +These functions can be run sequentially or combined into several asynchronous groups. +The handlers can, for instance, process data, make web requests, read and write files, etc. + +The services are executed on every `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ run, +they can happen before or after `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_ execution. +Each of them has one of the following signatures: + +.. code-block:: python + + async def handler(ctx: Context) -> Any: + ... + + async def handler(ctx: Context, pipeline: Pipeline) -> Any: + ... + + async def handler(ctx: Context, pipeline: Pipeline, runtime_info: Dict) -> Any: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_, +where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_ +and the return value can be anything (it is not used). + Service conditions ~~~~~~~~~~~~~~~~~~ +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_) +can be executed conditionally. +For that some special conditions should be used (that are in a way similar to `Script conditions and condition handlers`_). +However, there is no such thing as ``condition handler`` function in pipeline. + +These conditions are only run before services they are related to, that can be any services **except for Actor**. +Each of these functions has the following signature: + +.. code-block:: python + + def condition(ctx: Context, pipeline: Pipeline) -> bool: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and the return value is ``True`` if the service should be run and ``False`` otherwise. + +There is a set of `standard condition functions <../api/dff.pipeline.conditions>`_ defined. + Statistics extractors ~~~~~~~~~~~~~~~~~~~~~ From 4bef9ae258300c8e94010789ec3ba1d7c912df44 Mon Sep 17 00:00:00 2001 From: pseusys Date: Wed, 18 Oct 2023 10:30:23 +0200 Subject: [PATCH 04/18] extractors added --- docs/source/user_guides/user_functions.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index b07a8d066..6c8245ddc 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -235,3 +235,21 @@ There is a set of `standard condition functions <../api/dff.pipeline.conditions> Statistics extractors ~~~~~~~~~~~~~~~~~~~~~ + +`OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`_ has some wrapper functions, +added to it on ``instrument`` call. +These functions can extract and process telemetry statistics. + +The extractors are run upon ``__call__`` of the instrumentor. +They have the following signature: + +.. code-block:: python + + def extractor(ctx: Context, _: ???, runtime_info: Dict) -> None: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +and ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_. + +There is a set of `standard statistics extractors <../api/dff.stats.default_extractors>`_ defined. From 91f300412c6bbeeb477220c317f995f39adcaabb Mon Sep 17 00:00:00 2001 From: pseusys Date: Fri, 20 Oct 2023 11:22:54 +0200 Subject: [PATCH 05/18] exception handling descriptions added --- docs/source/user_guides/user_functions.rst | 175 +++++++++++++++++++-- 1 file changed, 164 insertions(+), 11 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 6c8245ddc..0ffab7164 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -12,13 +12,18 @@ types, warns about several common exception handling. ``Actor`` handlers ~~~~~~~~~~~~~~~~~~ +Description +=========== + `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ constructor accepts ``handlers`` parameter, that is either ``None`` or dictionary attributing lists of functions to different `ActorStage <../api/dff.script.core.types#ActorStage>`_ values. These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`_ lifecycle. -Each of these functions has the following signature: + +Signature +========= .. code-block:: python @@ -29,14 +34,27 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value can be anything (it is not used). +Exceptions +========== + +If an exception occurs during this function execution, it will be handeled on pipeline level, +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +will be set to ``FAILED``. +These exceptions **are not raised** during script validation. + Pre-transition processors ~~~~~~~~~~~~~~~~~~~~~~~~~ +Description +=========== + Each script `Node <../api/dff.script.core.script#Node>`_ has a property called ``pre_transitions_processing``. That is a dictionary, associating functions to their names (that can be any hashable object). These functions are run before transition from the previous node to the current node. -Each of these functions has the following signature: + +Signature +========= .. code-block:: python @@ -47,14 +65,26 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value is the modified Context value. +Exceptions +========== + +If an exception occurs during this function execution, it will be handeled internally, +only an exception message will be printed to ``stdout``. +These exceptions **are not raised** during script validation. + Pre-response processors ~~~~~~~~~~~~~~~~~~~~~~~ +Description +=========== + Each script `Node <../api/dff.script.core.script#Node>`_ has a property called ``pre_response_processing``. That is a dictionary, associating functions to their names (that can be any hashable object). These functions are run before acquiring the response, after the current node is processed. -Each of these functions has the following signature: + +Signature +========= .. code-block:: python @@ -65,9 +95,19 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value is the modified Context value. +Exceptions +========== + +If an exception occurs during this function execution, it will be handeled internally, +only an exception message will be printed to ``stdout``. +These exceptions **are not raised** during script validation. + Script conditions and condition handlers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Description of condition +======================== + Each transition between different nodes in `Script <../api/dff.script.core.script#Script>`_ has a condition function. They are provided in script dictionary (TODO: parser?) in form of Python functions. @@ -75,7 +115,9 @@ They are provided in script dictionary (TODO: parser?) in form of Python functio These functions are executed on pipeline startup if ``validation_stage`` parameter of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and before transition from the corresponding node in script in runtime. -Each of these functions has the following signature: + +Signature of condition +====================== .. code-block:: python @@ -86,13 +128,29 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value is ``True`` if transition should be made and ``False`` otherwise. +Standard conditions +=================== + There is a set of `standard condition functions <../api/dff.script.conditions.std_conditions>`_ defined. +Exceptions in conditions +======================== + +If an exception occurs during this function execution, it will be reported during script validation stage +(if any) and also will be handeled on pipeline level, +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +will be set to ``FAILED``. + +Description of condition handler +================================ + `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ constructor also accepts condition handler - that is a special function that executes conditions. This function is invoked every time condition should be checked, it launches and checks condition. -This function has the following signature: + +Signature of condition handler +============================== .. code-block:: python @@ -103,12 +161,26 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value is ``True`` if transition should be made and ``False`` otherwise. +Standard condition handler +========================== + The simplest `default condition handler <../api/dff.pipeline.pipeline.actor#default_condition_handler>`_ just invokes the condition function and returns the result. +Exceptions in condrition handler +================================ + +If an exception occurs during this function execution, it will be reported during script validation stage +(if any), otherwise it will be handeled on pipeline level, +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +will be set to ``FAILED``. + Labels ~~~~~~ +Description +=========== + Some of the transitions between nodes in `Script <../api/dff.script.core.script#Script>`_ do not have "absolute" node targets specified. For instance, that might be useful in case it is required to stay in the same node or transition @@ -118,7 +190,9 @@ For such cases special function node labels can be used. These functions are executed on pipeline startup if ``validation_stage`` parameter of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and before transition from the corresponding node in script in runtime. -Each of these functions has the following signature: + +Signature +========= .. code-block:: python @@ -130,11 +204,23 @@ where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pip and the return value is an instance of `NodeLabel3Type <../api/dff.script.core.types#NodeLabel3Type>`, that is a tuple of target flow name (``str``), node name (``str``) and priority (``float``). +Standard +======== + There is a set of `standard label functions <../api/dff.script.conditions.std_labels>`_ defined. +Exceptions +========== + +If an exception occurs during this function execution, it will be reported during script validation stage +(if any), otherwise it will be handeled internally, only an exception message will be printed to ``stdout``. + Responses ~~~~~~~~~ +Description +=========== + For some of the nodes in `Script <../api/dff.script.core.script#Script>`_ returning constant values might be not enough. For these cases each return value can be represented as a Python function. @@ -142,7 +228,9 @@ For these cases each return value can be represented as a Python function. These functions are executed on pipeline startup if ``validation_stage`` parameter of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and in the end of any node processing in runtime. -Each of these functions has the following signature: + +Signature +========= .. code-block:: python @@ -153,9 +241,20 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value is an instance of `Message <../api/dff.script.core.message#Message>`. +Exceptions +========== + +If an exception occurs during this function execution, it will be reported during script validation stage +(if any), otherwise it will be handeled on pipeline level, +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +will be set to ``FAILED``. + Extra handlers ~~~~~~~~~~~~~~ +Description +=========== + For some (or all) services in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ special extra handler functions can be added. These functions can handle statistics collection, input data transformation @@ -165,7 +264,9 @@ These functions can be either added to `pipeline dict <../api/dff.pipeline.types or added to all services at once with `add_global_handler <../api/dff.pipeline.pipeline.pipeline#add_global_handler>`_ function. The handlers can be executed before or after pipeline services. -Each of them has one of the following signatures: + +Signatures +========== .. code-block:: python @@ -183,9 +284,20 @@ where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pip where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_ and the return value can be anything (it is not used). +Exceptions +========== + +If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted +and an exception message will be printed to ``stdout``. +If any other exception occures during this function execution, it **will not** be handeled on pipeline level, +it will either be reported in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`_ or interrupt pipeline execution. + Service handlers ~~~~~~~~~~~~~~~~ +Description +=========== + `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_) should be represented as functions. These functions can be run sequentially or combined into several asynchronous groups. @@ -193,7 +305,9 @@ The handlers can, for instance, process data, make web requests, read and write The services are executed on every `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ run, they can happen before or after `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_ execution. -Each of them has one of the following signatures: + +Signatures +========== .. code-block:: python @@ -211,16 +325,30 @@ where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pip where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_ and the return value can be anything (it is not used). +Exceptions +========== + +If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted +in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`_ and an exception message will be printed to ``stdout``. +If any other exception occures during this function execution, it will be handeled on pipeline level, +exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +will be set to ``FAILED``. + Service conditions ~~~~~~~~~~~~~~~~~~ +Description +=========== + `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_) can be executed conditionally. For that some special conditions should be used (that are in a way similar to `Script conditions and condition handlers`_). However, there is no such thing as ``condition handler`` function in pipeline. These conditions are only run before services they are related to, that can be any services **except for Actor**. -Each of these functions has the following signature: + +Signature +========= .. code-block:: python @@ -231,17 +359,32 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and the return value is ``True`` if the service should be run and ``False`` otherwise. +Standard +======== + There is a set of `standard condition functions <../api/dff.pipeline.conditions>`_ defined. +Exceptions +========== + +If any other exception occures during this function execution, it will be handeled on pipeline level, +exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +will be set to ``FAILED``. + Statistics extractors ~~~~~~~~~~~~~~~~~~~~~ +Description +=========== + `OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`_ has some wrapper functions, added to it on ``instrument`` call. These functions can extract and process telemetry statistics. The extractors are run upon ``__call__`` of the instrumentor. -They have the following signature: + +Signature +========= .. code-block:: python @@ -252,4 +395,14 @@ where ``ctx`` is the current instance of `Context <../api/dff.script.core.contex where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ and ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_. +Standard +======== + There is a set of `standard statistics extractors <../api/dff.stats.default_extractors>`_ defined. + +Exceptions +========== + +If an exception occures during this function execution, it is not hangled and will be thrown +during `OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`_ ``__call__`` +function execution. From 8af5e5022eae994b96c591618287d89b8e33f620 Mon Sep 17 00:00:00 2001 From: pseusys Date: Fri, 20 Oct 2023 12:00:52 +0200 Subject: [PATCH 06/18] some of RST errors fixed --- dff/pipeline/pipeline/actor.py | 4 +--- docs/source/user_guides/user_functions.rst | 6 +++--- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/dff/pipeline/pipeline/actor.py b/dff/pipeline/pipeline/actor.py index 850621cbf..56c902672 100644 --- a/dff/pipeline/pipeline/actor.py +++ b/dff/pipeline/pipeline/actor.py @@ -390,9 +390,7 @@ def validate_script(self, pipeline: Pipeline, verbose: bool = True): return error_msgs -def default_condition_handler( - condition: Callable, ctx: Context, pipeline: Pipeline, *args, **kwargs -) -> bool: +def default_condition_handler(condition: Callable, ctx: Context, pipeline: Pipeline, *args, **kwargs) -> bool: """ The simplest and quickest condition handler for trivial condition handling returns the callable condition: diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 0ffab7164..8e2694733 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -131,7 +131,7 @@ and the return value is ``True`` if transition should be made and ``False`` othe Standard conditions =================== -There is a set of `standard condition functions <../api/dff.script.conditions.std_conditions>`_ defined. +There is a set of `standard script condition functions <../api/dff.script.conditions.std_conditions>`_ defined. Exceptions in conditions ======================== @@ -362,7 +362,7 @@ and the return value is ``True`` if the service should be run and ``False`` othe Standard ======== -There is a set of `standard condition functions <../api/dff.pipeline.conditions>`_ defined. +There is a set of `standard service condition functions <../api/dff.pipeline.conditions>`_ defined. Exceptions ========== @@ -388,7 +388,7 @@ Signature .. code-block:: python - def extractor(ctx: Context, _: ???, runtime_info: Dict) -> None: + def extractor(ctx: Context, _: TODO, runtime_info: Dict) -> None: ... where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, From b74223e9acefa13a9219f428fd7b1625a7fa330a Mon Sep 17 00:00:00 2001 From: pseusys Date: Fri, 20 Oct 2023 12:35:45 +0200 Subject: [PATCH 07/18] links made anonymous --- docs/source/user_guides/user_functions.rst | 120 ++++++++++----------- 1 file changed, 60 insertions(+), 60 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 8e2694733..969d81246 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -15,11 +15,11 @@ types, warns about several common exception handling. Description =========== -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ constructor accepts ``handlers`` +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor accepts ``handlers`` parameter, that is either ``None`` or dictionary attributing lists of functions to different -`ActorStage <../api/dff.script.core.types#ActorStage>`_ values. +`ActorStage <../api/dff.script.core.types#ActorStage>`__ values. -These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`_ +These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`__ lifecycle. Signature @@ -30,15 +30,15 @@ Signature def handler(ctx: Context, pipeline: Pipeline) -> Any: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value can be anything (it is not used). Exceptions ========== If an exception occurs during this function execution, it will be handeled on pipeline level, -exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. These exceptions **are not raised** during script validation. @@ -48,7 +48,7 @@ Pre-transition processors Description =========== -Each script `Node <../api/dff.script.core.script#Node>`_ has a property called ``pre_transitions_processing``. +Each script `Node <../api/dff.script.core.script#Node>`__ has a property called ``pre_transitions_processing``. That is a dictionary, associating functions to their names (that can be any hashable object). These functions are run before transition from the previous node to the current node. @@ -61,8 +61,8 @@ Signature def handler(ctx: Context, pipeline: Pipeline) -> Context: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value is the modified Context value. Exceptions @@ -78,7 +78,7 @@ Pre-response processors Description =========== -Each script `Node <../api/dff.script.core.script#Node>`_ has a property called ``pre_response_processing``. +Each script `Node <../api/dff.script.core.script#Node>`__ has a property called ``pre_response_processing``. That is a dictionary, associating functions to their names (that can be any hashable object). These functions are run before acquiring the response, after the current node is processed. @@ -91,8 +91,8 @@ Signature def handler(ctx: Context, pipeline: Pipeline) -> Context: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value is the modified Context value. Exceptions @@ -108,12 +108,12 @@ Script conditions and condition handlers Description of condition ======================== -Each transition between different nodes in `Script <../api/dff.script.core.script#Script>`_ +Each transition between different nodes in `Script <../api/dff.script.core.script#Script>`__ has a condition function. They are provided in script dictionary (TODO: parser?) in form of Python functions. These functions are executed on pipeline startup if ``validation_stage`` parameter of -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and before transition +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ is ``True`` and before transition from the corresponding node in script in runtime. Signature of condition @@ -124,27 +124,27 @@ Signature of condition def condition(ctx: Context, pipeline: Pipeline) -> bool: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value is ``True`` if transition should be made and ``False`` otherwise. Standard conditions =================== -There is a set of `standard script condition functions <../api/dff.script.conditions.std_conditions>`_ defined. +There is a set of `standard script condition functions <../api/dff.script.conditions.std_conditions>`__ defined. Exceptions in conditions ======================== If an exception occurs during this function execution, it will be reported during script validation stage (if any) and also will be handeled on pipeline level, -exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. Description of condition handler ================================ -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ constructor also accepts +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor also accepts condition handler - that is a special function that executes conditions. This function is invoked every time condition should be checked, it launches and checks condition. @@ -157,14 +157,14 @@ Signature of condition handler def condition_handler(condition: Callable[[Context, Pipeline], bool], ctx: Context, pipeline: Pipeline) -> bool: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value is ``True`` if transition should be made and ``False`` otherwise. Standard condition handler ========================== -The simplest `default condition handler <../api/dff.pipeline.pipeline.actor#default_condition_handler>`_ +The simplest `default condition handler <../api/dff.pipeline.pipeline.actor#default_condition_handler>`__ just invokes the condition function and returns the result. Exceptions in condrition handler @@ -172,7 +172,7 @@ Exceptions in condrition handler If an exception occurs during this function execution, it will be reported during script validation stage (if any), otherwise it will be handeled on pipeline level, -exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. Labels @@ -181,14 +181,14 @@ Labels Description =========== -Some of the transitions between nodes in `Script <../api/dff.script.core.script#Script>`_ +Some of the transitions between nodes in `Script <../api/dff.script.core.script#Script>`__ do not have "absolute" node targets specified. For instance, that might be useful in case it is required to stay in the same node or transition to the previous node. For such cases special function node labels can be used. These functions are executed on pipeline startup if ``validation_stage`` parameter of -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and before transition +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ is ``True`` and before transition from the corresponding node in script in runtime. Signature @@ -199,15 +199,15 @@ Signature def label(ctx: Context, pipeline: Pipeline) -> Tuple[str, str, float]: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value is an instance of `NodeLabel3Type <../api/dff.script.core.types#NodeLabel3Type>`, that is a tuple of target flow name (``str``), node name (``str``) and priority (``float``). Standard ======== -There is a set of `standard label functions <../api/dff.script.conditions.std_labels>`_ defined. +There is a set of `standard label functions <../api/dff.script.conditions.std_labels>`__ defined. Exceptions ========== @@ -221,12 +221,12 @@ Responses Description =========== -For some of the nodes in `Script <../api/dff.script.core.script#Script>`_ returning constant values +For some of the nodes in `Script <../api/dff.script.core.script#Script>`__ returning constant values might be not enough. For these cases each return value can be represented as a Python function. These functions are executed on pipeline startup if ``validation_stage`` parameter of -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ is ``True`` and in the end +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ is ``True`` and in the end of any node processing in runtime. Signature @@ -237,8 +237,8 @@ Signature def response(ctx: Context, pipeline: Pipeline) -> Message: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value is an instance of `Message <../api/dff.script.core.message#Message>`. Exceptions @@ -246,7 +246,7 @@ Exceptions If an exception occurs during this function execution, it will be reported during script validation stage (if any), otherwise it will be handeled on pipeline level, -exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. Extra handlers @@ -255,13 +255,13 @@ Extra handlers Description =========== -For some (or all) services in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ special +For some (or all) services in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ special extra handler functions can be added. These functions can handle statistics collection, input data transformation or other pipeline functionality extension. -These functions can be either added to `pipeline dict <../api/dff.pipeline.types#PipelineBuilder>`_ -or added to all services at once with `add_global_handler <../api/dff.pipeline.pipeline.pipeline#add_global_handler>`_ +These functions can be either added to `pipeline dict <../api/dff.pipeline.types#PipelineBuilder>`__ +or added to all services at once with `add_global_handler <../api/dff.pipeline.pipeline.pipeline#add_global_handler>`__ function. The handlers can be executed before or after pipeline services. @@ -279,9 +279,9 @@ Signatures async def handler(ctx: Context, pipeline: Pipeline, runtime_info: Dict) -> Any: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_, -where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__, +where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`__ and the return value can be anything (it is not used). Exceptions @@ -290,7 +290,7 @@ Exceptions If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted and an exception message will be printed to ``stdout``. If any other exception occures during this function execution, it **will not** be handeled on pipeline level, -it will either be reported in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`_ or interrupt pipeline execution. +it will either be reported in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`__ or interrupt pipeline execution. Service handlers ~~~~~~~~~~~~~~~~ @@ -298,13 +298,13 @@ Service handlers Description =========== -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_) +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`__) should be represented as functions. These functions can be run sequentially or combined into several asynchronous groups. The handlers can, for instance, process data, make web requests, read and write files, etc. -The services are executed on every `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ run, -they can happen before or after `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_ execution. +The services are executed on every `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ run, +they can happen before or after `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`__ execution. Signatures ========== @@ -320,18 +320,18 @@ Signatures async def handler(ctx: Context, pipeline: Pipeline, runtime_info: Dict) -> Any: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_, -where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__, +where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`__ and the return value can be anything (it is not used). Exceptions ========== If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted -in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`_ and an exception message will be printed to ``stdout``. +in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`__ and an exception message will be printed to ``stdout``. If any other exception occures during this function execution, it will be handeled on pipeline level, -exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. Service conditions @@ -340,9 +340,9 @@ Service conditions Description =========== -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`_) +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`__) can be executed conditionally. -For that some special conditions should be used (that are in a way similar to `Script conditions and condition handlers`_). +For that some special conditions should be used (that are in a way similar to `Script conditions and condition handlers`__). However, there is no such thing as ``condition handler`` function in pipeline. These conditions are only run before services they are related to, that can be any services **except for Actor**. @@ -355,20 +355,20 @@ Signature def condition(ctx: Context, pipeline: Pipeline) -> bool: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value is ``True`` if the service should be run and ``False`` otherwise. Standard ======== -There is a set of `standard service condition functions <../api/dff.pipeline.conditions>`_ defined. +There is a set of `standard service condition functions <../api/dff.pipeline.conditions>`__ defined. Exceptions ========== If any other exception occures during this function execution, it will be handeled on pipeline level, -exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`_ +exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. Statistics extractors @@ -377,7 +377,7 @@ Statistics extractors Description =========== -`OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`_ has some wrapper functions, +`OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`__ has some wrapper functions, added to it on ``instrument`` call. These functions can extract and process telemetry statistics. @@ -391,18 +391,18 @@ Signature def extractor(ctx: Context, _: TODO, runtime_info: Dict) -> None: ... -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`_, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`_ -and ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`_. +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ +and ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`__. Standard ======== -There is a set of `standard statistics extractors <../api/dff.stats.default_extractors>`_ defined. +There is a set of `standard statistics extractors <../api/dff.stats.default_extractors>`__ defined. Exceptions ========== If an exception occures during this function execution, it is not hangled and will be thrown -during `OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`_ ``__call__`` +during `OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`__ ``__call__`` function execution. From 23f3f5bd7e04af7e8f2d80d45342857180ff0ae4 Mon Sep 17 00:00:00 2001 From: pseusys Date: Fri, 20 Oct 2023 12:56:36 +0200 Subject: [PATCH 08/18] title underscore length fixed --- docs/source/user_guides.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guides.rst b/docs/source/user_guides.rst index 2d7f5071a..8e80e68e7 100644 --- a/docs/source/user_guides.rst +++ b/docs/source/user_guides.rst @@ -24,7 +24,7 @@ We show how to plug in the telemetry collection and configure the pre-built Superset dashboard shipped with DFF. :doc:`User defined functions guide <./user_guides/user_functions>` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``user defined functions guide`` tutorial describes useage of all user-defined functions that can be used for framework extension. From d041903e539147947f5038843818bd61f1951fbd Mon Sep 17 00:00:00 2001 From: pseusys Date: Fri, 20 Oct 2023 13:34:20 +0200 Subject: [PATCH 09/18] link fix attempt --- docs/source/user_guides/user_functions.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 969d81246..8de9af4c2 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -201,7 +201,7 @@ Signature where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ -and the return value is an instance of `NodeLabel3Type <../api/dff.script.core.types#NodeLabel3Type>`, +and the return value is an instance of `NodeLabel3Type <../api/dff.script.core.types#NodeLabel3Type>`__, that is a tuple of target flow name (``str``), node name (``str``) and priority (``float``). Standard @@ -239,7 +239,7 @@ Signature where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ -and the return value is an instance of `Message <../api/dff.script.core.message#Message>`. +and the return value is an instance of `Message <../api/dff.script.core.message#Message>`__. Exceptions ========== @@ -342,7 +342,7 @@ Description `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ services (other than `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`__) can be executed conditionally. -For that some special conditions should be used (that are in a way similar to `Script conditions and condition handlers`__). +For that some special conditions should be used (that are in a way similar to `Script conditions and condition handlers`_). However, there is no such thing as ``condition handler`` function in pipeline. These conditions are only run before services they are related to, that can be any services **except for Actor**. From a089fc9ee10b7b9636956fe23033fa808fcadb62 Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Fri, 20 Oct 2023 14:41:39 +0300 Subject: [PATCH 10/18] remove todo --- docs/source/user_guides/user_functions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 8de9af4c2..5bec88f9c 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -388,7 +388,7 @@ Signature .. code-block:: python - def extractor(ctx: Context, _: TODO, runtime_info: Dict) -> None: + def extractor(ctx: Context, pipeline: Pipeline, runtime_info: Dict) -> None: ... where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, From 0f9b5ea2462c55bea34047ef0f016be1ab82f861 Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Fri, 20 Oct 2023 14:41:56 +0300 Subject: [PATCH 11/18] fix typos --- docs/source/user_guides/user_functions.rst | 26 +++++++++++----------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 5bec88f9c..e88bc6a86 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -4,7 +4,7 @@ User functions guide Overview ~~~~~~~~ -Dialog flow franework allows user to define custom functions for implementaing custom behavior +Dialog flow framework allows user to define custom functions for implementing custom behavior in several aspects. This tutorial summarizes the custom functions use cases, specifies their arguments and return types, warns about several common exception handling. @@ -37,7 +37,7 @@ and the return value can be anything (it is not used). Exceptions ========== -If an exception occurs during this function execution, it will be handeled on pipeline level, +If an exception occurs during this function execution, it will be handled on pipeline level, exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. These exceptions **are not raised** during script validation. @@ -68,7 +68,7 @@ and the return value is the modified Context value. Exceptions ========== -If an exception occurs during this function execution, it will be handeled internally, +If an exception occurs during this function execution, it will be handled internally, only an exception message will be printed to ``stdout``. These exceptions **are not raised** during script validation. @@ -98,7 +98,7 @@ and the return value is the modified Context value. Exceptions ========== -If an exception occurs during this function execution, it will be handeled internally, +If an exception occurs during this function execution, it will be handled internally, only an exception message will be printed to ``stdout``. These exceptions **are not raised** during script validation. @@ -137,7 +137,7 @@ Exceptions in conditions ======================== If an exception occurs during this function execution, it will be reported during script validation stage -(if any) and also will be handeled on pipeline level, +(if any) and also will be handled on pipeline level, exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. @@ -167,11 +167,11 @@ Standard condition handler The simplest `default condition handler <../api/dff.pipeline.pipeline.actor#default_condition_handler>`__ just invokes the condition function and returns the result. -Exceptions in condrition handler +Exceptions in condition handler ================================ If an exception occurs during this function execution, it will be reported during script validation stage -(if any), otherwise it will be handeled on pipeline level, +(if any), otherwise it will be handled on pipeline level, exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. @@ -213,7 +213,7 @@ Exceptions ========== If an exception occurs during this function execution, it will be reported during script validation stage -(if any), otherwise it will be handeled internally, only an exception message will be printed to ``stdout``. +(if any), otherwise it will be handled internally, only an exception message will be printed to ``stdout``. Responses ~~~~~~~~~ @@ -245,7 +245,7 @@ Exceptions ========== If an exception occurs during this function execution, it will be reported during script validation stage -(if any), otherwise it will be handeled on pipeline level, +(if any), otherwise it will be handled on pipeline level, exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. @@ -289,7 +289,7 @@ Exceptions If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted and an exception message will be printed to ``stdout``. -If any other exception occures during this function execution, it **will not** be handeled on pipeline level, +If any other exception occurs during this function execution, it **will not** be handled on pipeline level, it will either be reported in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`__ or interrupt pipeline execution. Service handlers @@ -330,7 +330,7 @@ Exceptions If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`__ and an exception message will be printed to ``stdout``. -If any other exception occures during this function execution, it will be handeled on pipeline level, +If any other exception occurs during this function execution, it will be handled on pipeline level, exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. @@ -367,7 +367,7 @@ There is a set of `standard service condition functions <../api/dff.pipeline.con Exceptions ========== -If any other exception occures during this function execution, it will be handeled on pipeline level, +If any other exception occurs during this function execution, it will be handled on pipeline level, exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. @@ -403,6 +403,6 @@ There is a set of `standard statistics extractors <../api/dff.stats.default_extr Exceptions ========== -If an exception occures during this function execution, it is not hangled and will be thrown +If an exception occurs during this function execution, it is not handled and will be thrown during `OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`__ ``__call__`` function execution. From e8e887c7b03ffa87ba2b5abf92bf6639e313ba76 Mon Sep 17 00:00:00 2001 From: pseusys Date: Sun, 22 Oct 2023 17:11:26 +0200 Subject: [PATCH 12/18] merge + reorder --- docs/source/user_guides/user_functions.rst | 156 +++++++++------------ 1 file changed, 64 insertions(+), 92 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index e88bc6a86..f1c0c1c83 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -9,79 +9,18 @@ in several aspects. This tutorial summarizes the custom functions use cases, specifies their arguments and return types, warns about several common exception handling. -``Actor`` handlers -~~~~~~~~~~~~~~~~~~ - -Description -=========== - -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor accepts ``handlers`` -parameter, that is either ``None`` or dictionary attributing lists of functions to different -`ActorStage <../api/dff.script.core.types#ActorStage>`__ values. - -These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`__ -lifecycle. - -Signature -========= - -.. code-block:: python - - def handler(ctx: Context, pipeline: Pipeline) -> Any: - ... - -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ -and the return value can be anything (it is not used). - -Exceptions -========== - -If an exception occurs during this function execution, it will be handled on pipeline level, -exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ -will be set to ``FAILED``. -These exceptions **are not raised** during script validation. - -Pre-transition processors -~~~~~~~~~~~~~~~~~~~~~~~~~ +Pre-transition and pre-response processors +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Description =========== -Each script `Node <../api/dff.script.core.script#Node>`__ has a property called ``pre_transitions_processing``. +Each script `Node <../api/dff.script.core.script#Node>`__ has a properties called +``pre_transitions_processing`` and ``pre_response_processing``. That is a dictionary, associating functions to their names (that can be any hashable object). -These functions are run before transition from the previous node to the current node. - -Signature -========= - -.. code-block:: python - - def handler(ctx: Context, pipeline: Pipeline) -> Context: - ... - -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ -and the return value is the modified Context value. - -Exceptions -========== - -If an exception occurs during this function execution, it will be handled internally, -only an exception message will be printed to ``stdout``. -These exceptions **are not raised** during script validation. - -Pre-response processors -~~~~~~~~~~~~~~~~~~~~~~~ - -Description -=========== - -Each script `Node <../api/dff.script.core.script#Node>`__ has a property called ``pre_response_processing``. -That is a dictionary, associating functions to their names (that can be any hashable object). - -These functions are run before acquiring the response, after the current node is processed. +Pre-transition proccessors are run before transition from the previous node to the current node. +Pre-response proccessors are run before acquiring the response, after the current node is processed. Signature ========= @@ -249,48 +188,38 @@ If an exception occurs during this function execution, it will be reported durin exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. -Extra handlers -~~~~~~~~~~~~~~ +``Actor`` handlers +~~~~~~~~~~~~~~~~~~ Description =========== -For some (or all) services in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ special -extra handler functions can be added. -These functions can handle statistics collection, input data transformation -or other pipeline functionality extension. +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor accepts ``handlers`` +parameter, that is either ``None`` or dictionary attributing lists of functions to different +`ActorStage <../api/dff.script.core.types#ActorStage>`__ values. -These functions can be either added to `pipeline dict <../api/dff.pipeline.types#PipelineBuilder>`__ -or added to all services at once with `add_global_handler <../api/dff.pipeline.pipeline.pipeline#add_global_handler>`__ -function. -The handlers can be executed before or after pipeline services. +These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`__ +lifecycle. -Signatures -========== +Signature +========= .. code-block:: python - async def handler(ctx: Context) -> Any: - ... - - async def handler(ctx: Context, pipeline: Pipeline) -> Any: - ... - - async def handler(ctx: Context, pipeline: Pipeline, runtime_info: Dict) -> Any: + def handler(ctx: Context, pipeline: Pipeline) -> Any: ... where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__, -where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`__ +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ and the return value can be anything (it is not used). Exceptions ========== -If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted -and an exception message will be printed to ``stdout``. -If any other exception occurs during this function execution, it **will not** be handled on pipeline level, -it will either be reported in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`__ or interrupt pipeline execution. +If an exception occurs during this function execution, it will be handled on pipeline level, +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ +will be set to ``FAILED``. +These exceptions **are not raised** during script validation. Service handlers ~~~~~~~~~~~~~~~~ @@ -371,6 +300,49 @@ If any other exception occurs during this function execution, it will be handled exception message will be printed to ``stdout`` and the service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. +Extra handlers +~~~~~~~~~~~~~~ + +Description +=========== + +For some (or all) services in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ special +extra handler functions can be added. +These functions can handle statistics collection, input data transformation +or other pipeline functionality extension. + +These functions can be either added to `pipeline dict <../api/dff.pipeline.types#PipelineBuilder>`__ +or added to all services at once with `add_global_handler <../api/dff.pipeline.pipeline.pipeline#add_global_handler>`__ +function. +The handlers can be executed before or after pipeline services. + +Signatures +========== + +.. code-block:: python + + async def handler(ctx: Context) -> Any: + ... + + async def handler(ctx: Context, pipeline: Pipeline) -> Any: + ... + + async def handler(ctx: Context, pipeline: Pipeline, runtime_info: Dict) -> Any: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__, +where ``runtime_info`` is a `runtime info dictionary <../api/dff.pipeline.types#ExtraHandlerRuntimeInfo>`__ +and the return value can be anything (it is not used). + +Exceptions +========== + +If this function exceeds timeout (that implies that ``TimeoutError`` is thrown), it will be interrupted +and an exception message will be printed to ``stdout``. +If any other exception occurs during this function execution, it **will not** be handled on pipeline level, +it will either be reported in parent `ServiceGroup <../api/dff.pipeline.service.group#ServiceGroup>`__ or interrupt pipeline execution. + Statistics extractors ~~~~~~~~~~~~~~~~~~~~~ From 0269d98badd83b9f41318576d39cf0ce4dcd8b6c Mon Sep 17 00:00:00 2001 From: pseusys Date: Mon, 23 Oct 2023 09:26:34 +0200 Subject: [PATCH 13/18] First use-case section added --- docs/source/user_guides/user_functions.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index f1c0c1c83..b754cf3eb 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -22,6 +22,9 @@ That is a dictionary, associating functions to their names (that can be any hash Pre-transition proccessors are run before transition from the previous node to the current node. Pre-response proccessors are run before acquiring the response, after the current node is processed. +Use-cases +========= + Signature ========= From bf44d72cad7ca54dca8087d50e7166cf524a292b Mon Sep 17 00:00:00 2001 From: pseusys Date: Tue, 24 Oct 2023 16:30:37 +0200 Subject: [PATCH 14/18] tutorial links added --- docs/source/user_guides/user_functions.rst | 78 ++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index b754cf3eb..8bba0338e 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -25,6 +25,13 @@ Pre-response proccessors are run before acquiring the response, after the curren Use-cases ========= +TODO + +See tutorials: + +#. `About pre-transition processors <../tutorials/tutorials.script.core.9_pre_transitions_processing>`__ +#. `About pre-response processors <../tutorials/tutorials.script.core.7_pre_response_processing>`__ + Signature ========= @@ -58,6 +65,13 @@ These functions are executed on pipeline startup if ``validation_stage`` paramet `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ is ``True`` and before transition from the corresponding node in script in runtime. +Use-cases +========= + +TODO + +See `tutorial about conditions <../tutorials/tutorials.script.core.2_conditions>`__. + Signature of condition ====================== @@ -91,6 +105,11 @@ condition handler - that is a special function that executes conditions. This function is invoked every time condition should be checked, it launches and checks condition. +Use-cases +========= + +TODO + Signature of condition handler ============================== @@ -133,6 +152,13 @@ These functions are executed on pipeline startup if ``validation_stage`` paramet `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ is ``True`` and before transition from the corresponding node in script in runtime. +Use-cases +========= + +TODO + +See `tutorial about transition functions <../tutorials/tutorials.script.core.4_transitions>`__. + Signature ========= @@ -171,6 +197,13 @@ These functions are executed on pipeline startup if ``validation_stage`` paramet `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ is ``True`` and in the end of any node processing in runtime. +Use-cases +========= + +TODO + +See `tutorial about response functions <../tutorials/tutorials.script.core.3_responses>`__. + Signature ========= @@ -204,6 +237,11 @@ parameter, that is either ``None`` or dictionary attributing lists of functions These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`__ lifecycle. +Use-cases +========= + +TODO + Signature ========= @@ -238,6 +276,18 @@ The handlers can, for instance, process data, make web requests, read and write The services are executed on every `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ run, they can happen before or after `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`__ execution. +Use-cases +========= + +TODO + +See tutorials: + +#. `About synchronous services (basic) <../tutorials/tutorials.pipeline.3_pipeline_dict_with_services_basic>`__ +#. `About synchronous services (advanced) <../tutorials/tutorials.pipeline.3_pipeline_dict_with_services_full>`__ +#. `About asynchronous services (basic) <../tutorials/tutorials.pipeline.5_asynchronous_groups_and_services_basic>`__ +#. `About asynchronous services (advanced) <../tutorials/tutorials.pipeline.5_asynchronous_groups_and_services_full>`__ + Signatures ========== @@ -279,6 +329,16 @@ However, there is no such thing as ``condition handler`` function in pipeline. These conditions are only run before services they are related to, that can be any services **except for Actor**. +Use-cases +========= + +TODO + +See tutorials: + +#. `About service groups and conditions (basic) <../tutorials/tutorials.pipeline.4_groups_and_conditions_basic>`__ +#. `About service groups and conditions (advanced) <../tutorials/tutorials.pipeline.4_groups_and_conditions_full>`__ + Signature ========= @@ -319,6 +379,17 @@ or added to all services at once with `add_global_handler <../api/dff.pipeline.p function. The handlers can be executed before or after pipeline services. +Use-cases +========= + +TODO + +See tutorials: + +#. `About extra handlers (basic) <../tutorials/tutorials.pipeline.7_extra_handlers_basic>`__ +#. `About extra handlers (advanced) <../tutorials/tutorials.pipeline.7_extra_handlers_full>`__ +#. `About extra handlers and extensions <../tutorials/tutorials.pipeline.8_extra_handlers_and_extensions>`__ + Signatures ========== @@ -358,6 +429,13 @@ These functions can extract and process telemetry statistics. The extractors are run upon ``__call__`` of the instrumentor. +Use-cases +========= + +TODO + +See `tutorial about extractor functions <../tutorials/tutorials.stats.1_extractor_functions>`__. + Signature ========= From 0fea18c9b0e49b8a47cf435aaa45ddad2f60731b Mon Sep 17 00:00:00 2001 From: pseusys Date: Tue, 24 Oct 2023 16:36:27 +0200 Subject: [PATCH 15/18] comment fixes accepted --- docs/source/user_guides/user_functions.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 8bba0338e..d949ef689 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -7,7 +7,7 @@ Overview Dialog flow framework allows user to define custom functions for implementing custom behavior in several aspects. This tutorial summarizes the custom functions use cases, specifies their arguments and return -types, warns about several common exception handling. +types, warns about several common exception handling cases. Pre-transition and pre-response processors ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -189,9 +189,9 @@ Responses Description =========== -For some of the nodes in `Script <../api/dff.script.core.script#Script>`__ returning constant values +For some of the nodes in `Script <../api/dff.script.core.script#Script>`__ yielding constant response values might be not enough. -For these cases each return value can be represented as a Python function. +For these cases each response can be represented as a Python function. These functions are executed on pipeline startup if ``validation_stage`` parameter of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ is ``True`` and in the end @@ -274,7 +274,7 @@ These functions can be run sequentially or combined into several asynchronous gr The handlers can, for instance, process data, make web requests, read and write files, etc. The services are executed on every `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ run, -they can happen before or after `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`__ execution. +they can run before or after `Actor <../api/dff.pipeline.pipeline.pipeline#ACTOR>`__ execution. Use-cases ========= @@ -369,7 +369,7 @@ Extra handlers Description =========== -For some (or all) services in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ special +For any service in a `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ special extra handler functions can be added. These functions can handle statistics collection, input data transformation or other pipeline functionality extension. @@ -423,11 +423,12 @@ Statistics extractors Description =========== -`OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`__ has some wrapper functions, -added to it on ``instrument`` call. -These functions can extract and process telemetry statistics. +Statistics module introduces the concept of extractor functions. +These are equivalent to pipeline's regular extra handlers (described above) except that +they have to be decorated by Opentelemetry's `OtelInstrumentor <../api/dff.stats.instrumentor#OtelInstrumentor>`_ +class that proxies the output of these handlers and redirects it to Opentelemetry services. -The extractors are run upon ``__call__`` of the instrumentor. +See the `extractor function tutorial <../tutorials/tutorials.stats.1_extractor_functions.py>`__ for reference. Use-cases ========= From 1d321821e9be102d9da4a8c8881d2ccc44790de0 Mon Sep 17 00:00:00 2001 From: pseusys Date: Wed, 8 Nov 2023 20:52:26 +0100 Subject: [PATCH 16/18] user guide sections added --- docs/source/user_guides/user_functions.rst | 82 ++++++++++++---------- 1 file changed, 44 insertions(+), 38 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index d949ef689..66dd13cc2 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -2,13 +2,54 @@ User functions guide -------------------- Overview -~~~~~~~~ +++++++++ Dialog flow framework allows user to define custom functions for implementing custom behavior in several aspects. This tutorial summarizes the custom functions use cases, specifies their arguments and return types, warns about several common exception handling cases. +``Actor`` handlers +++++++++++++++++++ + +Description +=========== + +`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor accepts ``handlers`` +parameter, that is either ``None`` or dictionary attributing lists of functions to different +`ActorStage <../api/dff.script.core.types#ActorStage>`__ values. + +These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`__ +lifecycle. + +Use-cases +========= + +TODO + +Signature +========= + +.. code-block:: python + + def handler(ctx: Context, pipeline: Pipeline) -> Any: + ... + +where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, +where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ +and the return value can be anything (it is not used). + +Exceptions +========== + +If an exception occurs during this function execution, it will be handled on pipeline level, +exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ +will be set to ``FAILED``. +These exceptions **are not raised** during script validation. + +Script functions +++++++++++++++++ + Pre-transition and pre-response processors ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -224,43 +265,8 @@ If an exception occurs during this function execution, it will be reported durin exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ will be set to ``FAILED``. -``Actor`` handlers -~~~~~~~~~~~~~~~~~~ - -Description -=========== - -`Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor accepts ``handlers`` -parameter, that is either ``None`` or dictionary attributing lists of functions to different -`ActorStage <../api/dff.script.core.types#ActorStage>`__ values. - -These functions are run at specific point in `Actor <../api/dff.pipeline.pipeline.actor#Actor>`__ -lifecycle. - -Use-cases -========= - -TODO - -Signature -========= - -.. code-block:: python - - def handler(ctx: Context, pipeline: Pipeline) -> Any: - ... - -where ``ctx`` is the current instance of `Context <../api/dff.script.core.context#Context>`__, -where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ -and the return value can be anything (it is not used). - -Exceptions -========== - -If an exception occurs during this function execution, it will be handled on pipeline level, -exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ -will be set to ``FAILED``. -These exceptions **are not raised** during script validation. +Service functions ++++++++++++++++++ Service handlers ~~~~~~~~~~~~~~~~ From c4e59bb8fcc9cc7852dbe272497c1eeec3d56fd3 Mon Sep 17 00:00:00 2001 From: pseusys Date: Fri, 1 Dec 2023 15:28:12 +0100 Subject: [PATCH 17/18] heading levels changed --- docs/source/user_guides/user_functions.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index 66dd13cc2..b02e484e7 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -13,7 +13,7 @@ types, warns about several common exception handling cases. ++++++++++++++++++ Description -=========== +########### `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor accepts ``handlers`` parameter, that is either ``None`` or dictionary attributing lists of functions to different @@ -23,12 +23,12 @@ These functions are run at specific point in `Actor <../api/dff.pipeline.pipelin lifecycle. Use-cases -========= +######### TODO Signature -========= +######### .. code-block:: python @@ -40,7 +40,7 @@ where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pip and the return value can be anything (it is not used). Exceptions -========== +########## If an exception occurs during this function execution, it will be handled on pipeline level, exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__ From aba92e8e7a284280d526f0a6da0fd376e687612f Mon Sep 17 00:00:00 2001 From: pseusys Date: Tue, 5 Dec 2023 01:25:05 +0100 Subject: [PATCH 18/18] headers are fixed (I hope) --- docs/source/user_guides/user_functions.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guides/user_functions.rst b/docs/source/user_guides/user_functions.rst index b02e484e7..b7e680322 100644 --- a/docs/source/user_guides/user_functions.rst +++ b/docs/source/user_guides/user_functions.rst @@ -13,7 +13,7 @@ types, warns about several common exception handling cases. ++++++++++++++++++ Description -########### +~~~~~~~~~~~ `Pipeline <../api/dff.pipeline.pipeline.pipeline#Pipeline>`__ constructor accepts ``handlers`` parameter, that is either ``None`` or dictionary attributing lists of functions to different @@ -23,12 +23,12 @@ These functions are run at specific point in `Actor <../api/dff.pipeline.pipelin lifecycle. Use-cases -######### +~~~~~~~~~ TODO Signature -######### +~~~~~~~~~ .. code-block:: python @@ -40,7 +40,7 @@ where ``pipeline`` is the current instance of `Pipeline <../api/dff.pipeline.pip and the return value can be anything (it is not used). Exceptions -########## +~~~~~~~~~~ If an exception occurs during this function execution, it will be handled on pipeline level, exception message will be printed to ``stdout`` and the actor service `state <../api/dff.pipeline.types#ComponentExecutionState>`__