em-odp v2.9.0

Event Machine (EM) on ODP v2.9.0

- Support for EM API v2.9 (em-odp/include), see API changes in
  em-odp/include/event_machine/README_API.
  * Minor API change: documentation clean up and a define value increase.

- EM config file options - config/em-odp.conf:
 - Config file version number bumped up to "0.0.14", see new options below.

- EM pool statistics update
  - EM Pool statistics to rely on ODP pool statistics.
    EM does not have to maintain pool statistics since ODP provides the
    same (or extended) information. The EM API em_pool_info() remains the same,
    it now uses ODP pool statistics under the hood instead.

- Dispatch, init: enforce ctrl queue polling on first dispatch.
  - The first call to em_dispatch() on each core will trigger a poll of the
    internal unscheduled ctrl queues.
  - See "Changed EM internal communication" in the release notes for v2.8.0
    below for more details on EM's unscheduled ctrl queues.

- EM Command Line Interface (EM CLI) update
  - Further EM CLI commands added, use the command "help" to see all available
    commands:
	(<em-odp.conf>: cli.enable = true)
        $> telnet localhost 55555
        EM-ODP> help
        Commands available:
          help                 Show available commands
          quit                 Disconnect
          logout               Disconnect
          exit                 Exit from current mode
          history              Show a list of previously run commands
          enable               Turn on privileged commands
          ...
          <odp-cmds>
          ...
          em_agrp_print        [a|i <ag id>|n <ag name>|h]
          em_eo_print          [a|i <eo id>|n <eo name>|h]
          em_egrp_print
          em_info_print        [a|p|c|h]
          em_pool_print        [a|i <pool id>|n <pool name>|h]
          em_queue_print       [a|c|h]
          em_qgrp_print        [a|i <qgrp id>|n <qgrp name>|h]
          em_core_print
  - See "EM Command Line Interface (EM CLI)" in the release notes for v2.8.0
    below for more details on usage.

- Rate limit EM's input-poll and output-drain callback functions.
  - Config file options to control how often the user provided callback
    functions are called (see config/em-odp.conf for full documentation):
    - Core local interval for calling input poll and output drain functions
      (in dispatch rounds):
      poll_drain_interval = 1
    - Core local interval for calling input poll and output drain functions
      (in nanoseconds):
      poll_drain_interval_ns = 1000000L
    - Default values currently selected so that functionality is unchanged
      compared to earlier releases.

- EM timer updated to use new ODP APIs
  - EM timer updated to internally use odp_timer_start() (instead of the now
    deprecated odp_timer_set_abs())

-  Test files and scripts moved into the em-odp repo (robot-tests/, scripts/)

.github/workflows/ci-pipeline.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout EM-ODP
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           # Full git history is needed to get a proper list of changed files
           # within `super-linter`
@@ -35,7 +35,7 @@ jobs:
   Build-Doxygen:
     runs-on: ubuntu-20.04
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Install dependencies
       run: |
         sudo apt-get update
@@ -61,7 +61,7 @@ jobs:
         sudo apt-get install libcli-dev
         sudo apt-get install telnet
     - name: Checkout EM-ODP
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
     - name: Build EM-ODP
       run: ./scripts/build.sh
     - name: Run Robot Tests
@@ -73,7 +73,7 @@ jobs:
   Cross-Compile-for-arm64:
     runs-on: ubuntu-20.04
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Cross-compile EM-ODP for arm64
       run: >
           sudo docker run

.github/workflows/gh-pages.yml

@@ -14,7 +14,7 @@ jobs:
   Documentation:
     runs-on: ubuntu-20.04
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: Install dependencies
       run: |
         sudo apt-get update

CHANGE_NOTES

@@ -52,6 +52,73 @@ Examples:
 - See em-odp/README for usage and compilation instructions.
 - See em-odp/include/event_machine/README_API for API changes
 
+--------------------------------------------------------------------------------
+Event Machine (EM) on ODP v2.9.0
+--------------------------------------------------------------------------------
+- Support for EM API v2.9 (em-odp/include), see API changes in
+  em-odp/include/event_machine/README_API.
+  * Minor API change: documentation clean up and a define value increase.
+
+- EM config file options - config/em-odp.conf:
+ - Config file version number bumped up to "0.0.14", see new options below.
+
+- EM pool statistics update
+  - EM Pool statistics to rely on ODP pool statistics.
+    EM does not have to maintain pool statistics since ODP provides the
+    same (or extended) information. The EM API em_pool_info() remains the same,
+    it now uses ODP pool statistics under the hood instead.
+
+- Dispatch, init: enforce ctrl queue polling on first dispatch.
+  - The first call to em_dispatch() on each core will trigger a poll of the
+    internal unscheduled ctrl queues.
+  - See "Changed EM internal communication" in the release notes for v2.8.0
+    below for more details on EM's unscheduled ctrl queues.
+
+- EM Command Line Interface (EM CLI) update
+  - Further EM CLI commands added, use the command "help" to see all available
+    commands:
+	(<em-odp.conf>: cli.enable = true)
+        $> telnet localhost 55555
+        EM-ODP> help
+        Commands available:
+          help                 Show available commands
+          quit                 Disconnect
+          logout               Disconnect
+          exit                 Exit from current mode
+          history              Show a list of previously run commands
+          enable               Turn on privileged commands
+          ...
+          <odp-cmds>
+          ...
+          em_agrp_print        [a|i <ag id>|n <ag name>|h]
+          em_eo_print          [a|i <eo id>|n <eo name>|h]
+          em_egrp_print
+          em_info_print        [a|p|c|h]
+          em_pool_print        [a|i <pool id>|n <pool name>|h]
+          em_queue_print       [a|c|h]
+          em_qgrp_print        [a|i <qgrp id>|n <qgrp name>|h]
+          em_core_print
+  - See "EM Command Line Interface (EM CLI)" in the release notes for v2.8.0
+    below for more details on usage.
+
+- Rate limit EM's input-poll and output-drain callback functions.
+  - Config file options to control how often the user provided callback
+    functions are called (see config/em-odp.conf for full documentation):
+    - Core local interval for calling input poll and output drain functions
+      (in dispatch rounds):
+      poll_drain_interval = 1
+    - Core local interval for calling input poll and output drain functions
+      (in nanoseconds):
+      poll_drain_interval_ns = 1000000L
+    - Default values currently selected so that functionality is unchanged
+      compared to earlier releases.
+
+- EM timer updated to use new ODP APIs
+  - EM timer updated to internally use odp_timer_start() (instead of the now
+    deprecated odp_timer_set_abs())
+
+-  Test files and scripts moved into the em-odp repo (robot-tests/, scripts/)
+
 --------------------------------------------------------------------------------
 Event Machine (EM) on ODP v2.8.0
 --------------------------------------------------------------------------------

README

@@ -101,15 +101,17 @@ with the same -march/-mcpu options for best results:
 ===============================================================================
 Debug compilation
 ===============================================================================
-For debugging activities it may be useful to compile separate libs/executables
-of ODP and EM-ODP. The configuration script has to be run with different
-options, other commands as above.
+For debugging activities it may be useful to compile separate debug libs and
+executables of ODP and EM-ODP. The configuration script has to be run with
+different options, other commands as above.
 
 ODP debug configuration:
-> ../configure --prefix=<odp debug install path> CFLAGS='-O0 -g3' ...
+> ../configure --prefix=<odp debug install path> \
+  --enable-debug --enable-helper-debug CFLAGS='-O0 -g3' ...
 EM-ODP debug configuration:
 > ../configure --prefix=<em-odp debug install path> \
-  --with-odp-path=<odp install path> CFLAGS='-O0 -g3' --enable-check-level=3...
+  --with-odp-path=<odp install path> \
+  --enable-check-level=3 --enable-esv --enable-debug-print CFLAGS='-O0 -g3' ...
 
 ===============================================================================
 Packet-I/O (ODP-linux-generic + socket, dpdk or netmap based io)

config/em-odp.conf

@@ -16,7 +16,7 @@
 
 # Mandatory fields
 em_implementation = "em-odp"
-config_file_version = "0.0.13"
+config_file_version = "0.0.14"
 
 # Pool options
 pool: {
@@ -285,4 +285,31 @@ dispatch: {
 	# while this option serves to give more control over the polling rate in
 	# nanoseconds.
 	poll_ctrl_interval_ns = 1000000L # poll max every 1ms
+
+	# Core local interval for calling input poll and output drain functions
+	# (in dispatch rounds)
+	#
+	# Rate limit EM poll and drain functions:
+	# Call the poll and/or drain functions for events every
+	# N:th dispatch round on a core (where N is 'poll_drain_interval' here).
+	#
+	# 1) If 'poll_drain_interval = 1': polling and draining is done every
+	#    dispatch round.
+	# 2) If 'poll_drain_interval = N (>1)': Every N:th dispatch round check
+	#    whether 'poll_drain_interval_ns' nanoseconds has passed since the
+	#    previous poll/drain, and if it has passed then call the poll/drain
+	#    functions.
+	poll_drain_interval = 1
+
+	# Core local interval for calling input poll and output drain functions
+	# (in nanoseconds)
+	#
+	# Works together with 'poll_drain_interval' to limit polling:
+	# When 'poll_drain_interval' is larger than 1, use this option to limit
+	# the poll/drain rate more exactly in nanoseconds per core.
+	# The previous option 'poll_drain_interval' is intended to limit the need
+	# to check absolute time(in ns) and thus maybe save some on performance,
+	# while this option serves to give more control over the poll/drain rate
+	# in nanoseconds.
+	poll_drain_interval_ns = 1000000L
 }

configure.ac

@@ -3,7 +3,7 @@ AC_PREREQ([2.69])
 # Version
 ############################
 m4_define([em_version_api_major], [2])
-m4_define([em_version_api_minor], [8])
+m4_define([em_version_api_minor], [9])
 m4_define([em_version_implementation], [0])
 m4_define([em_version_fix], [0])
 
@@ -536,7 +536,7 @@ AC_MSG_RESULT([
 	$PACKAGE $VERSION
 	=================
 	EM API version:		${EM_VERSION_API}
-	EM ODP version:		${EM_VERSION}
+	EM (em-odp) version:	${EM_VERSION}
 	Library version:	${EM_VERSION_LIBSO}
 
 	host:			${host}

include/event_machine/README_API

@@ -5,6 +5,14 @@ EM API Release Notes
 - See em-odp/README for usage and compilation instructions.
 - See em-odp/CHANGE_NOTES for changed and added features.
 
+-------------------------------------------------------------------------------
+API 2.9 (EM_VERSION_API_MAJOR=2, EM_VERSION_API_MINOR=9)
+-------------------------------------------------------------------------------
+Backwards compatible with EM 2.8 API
+
+1. API documentation clean-up.
+2. The define EM_MAX_QUEUES increased from 960 to 1024.
+
 -------------------------------------------------------------------------------
 API 2.8 (EM_VERSION_API_MAJOR=2, EM_VERSION_API_MINOR=8)
 -------------------------------------------------------------------------------

include/event_machine/api/.gitignore

@@ -0,0 +1 @@
+event_machine_version.h

include/event_machine/api/event_machine_eo.h

@@ -176,7 +176,6 @@ typedef void (*em_receive_func_t)(void *eo_ctx,
  * @param eo_ctx  EO context data given to em_eo_create_multircv(),
  *                EM does not dereference.
  * @param events  Event handles: events[num]
- * @param types   Event types: types[num] - types[i] is related to event[i]
  * @param num     Number of events received
  *                (0 to 'max_events' of em_eo_create_multircv())
  * @param queue   Queue from which the event was dequeued
@@ -543,9 +542,6 @@ em_eo_add_queue(em_eo_t eo, em_queue_t queue,
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param eo            EO handle
@@ -598,9 +594,6 @@ em_eo_remove_queue(em_eo_t eo, em_queue_t queue,
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param eo            EO handle
@@ -651,9 +644,6 @@ em_eo_remove_queue_all(em_eo_t eo, int delete_queues,
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param eo              EO handle
@@ -764,9 +754,6 @@ em_eo_start(em_eo_t eo, em_status_t *result, const em_eo_conf_t *conf,
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param      eo      EO handle
@@ -828,9 +815,6 @@ em_eo_stop(em_eo_t eo, int num_notif, const em_notif_t notif_tbl[]);
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param eo            EO handle

include/event_machine/api/event_machine_error.h

@@ -38,14 +38,14 @@
  * @defgroup em_error Error management
  *  Error/Exception management
  *
- *  OpenEM provides two basic mechanisms to manage operational exceptions
+ *  EM provides two basic mechanisms to manage operational exceptions
  *
  *  -# Return value
  *    - Most API calls return a status which can be checked by the application
  *
  *  -# Error handler
  *    - An optional error handler function can be registered. The given
- *      function is called by the EM implementation as an error occurs. This
+ *      function is called by the EM implementation when an error occurs. This
  *      makes it possible to centralize the error management or even modify
  *      some API behavior as an error handler can modify the original API call
  *      return value as well. For instance, a global error handler could modify
@@ -71,8 +71,7 @@ extern "C" {
  * The user can register EO specific and/or EM global error handlers. When an
  * error occurs, EM calls the EO specific error handler, if registered. If
  * there's no EO specific handler registered or the error occurs outside of an
- * EO context, EM calls the global error handler. If no error handlers are
- * found, EM just returns the error code depending on the API function.
+ * EO context, EM calls the global error handler.
  *
  * The error handler is called with the original error code (that's about to be
  * returned) from the API call or em_error(). The error scope identifies the

include/event_machine/api/event_machine_event_group.h

@@ -163,7 +163,7 @@ em_status_t em_event_group_delete(em_event_group_t event_group);
  * This function sets the event count and notification parameters for the event
  * group. After it returns, events sent or assigned to the event group are
  * counted against the current count value. Notification events are sent when
- * all (counted) events have been processed (count is decrement at EO receive
+ * all (counted) events have been processed (count is decremented at EO receive
  * return or by calling em_event_group_processing_end()). A new apply call is
  * needed to re-use the event group for another cycle (with a new count and
  * notifications).
@@ -328,7 +328,7 @@ void em_event_group_processing_end(void);
  * Assign will fail if there already is an active current event group, i.e.
  * only one event group can be active at a time (per core).
  *
- * This needs to use used with care, i.e. match the amount of events applied
+ * This needs to be used with care, i.e. match the amount of events applied
  * and assigned.
  *
  * @param event_group   An applied event group to assign to

include/event_machine/api/event_machine_queue_group.h

@@ -117,9 +117,6 @@ em_queue_group_create(const char *name, const em_core_mask_t *mask,
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param name       Queue group name (optional, NULL ok)
@@ -166,9 +163,6 @@ em_queue_group_delete(em_queue_group_t queue_group,
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param queue_group  Queue group to delete
@@ -229,9 +223,6 @@ em_queue_group_modify(em_queue_group_t queue_group,
  * While one core is calling a sync-API function, the others must be running the
  * EM dispatch loop to be able to receive and handle the sync-API request events
  * sent internally.
- * Only one sync-API of any kind, i.e. of APIs named "em_..._sync()", can be
- * called at a time: EM will report and return an error from the API to avoid
- * deadlock if multiple cores simultaneously try to call synchronous APIs.
  * Use the sync-APIs mainly to simplify application start-up or teardown.
  *
  * @param queue_group  Queue group to modify

include/event_machine/platform/event_machine_config.h

@@ -122,7 +122,7 @@ extern "C" {
  * @def EM_MAX_QUEUES
  * Maximum total number of queues
  */
-#define EM_MAX_QUEUES  960  /* Should be <= odp-max-queues */
+#define EM_MAX_QUEUES  1024  /* Should be <= odp-max-queues */
 
 /**
  * @def EM_QUEUE_NAME_LEN