Copyright (c) 2013-2023, Nokia Solutions and Networks
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:

  * Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.
  * Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.
  * Neither the name of the copyright holder nor the names of its
    contributors may be used to endorse or promote products derived
    from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

===============================================================================
Event Machine on Open Data Plane - em-odp
===============================================================================
em-odp is an implementation of the Event Machine (EM) framework on top of
Open Data Plane (ODP). Many ODP implementations exist, both generic and HW or
SoC optimized, and all should be suitable for em-odp to utilize and run upon.
For development, e.g. odp for linux-generic could be most suitable, while a SoC
optimized version would provide the best performance on a specific target.
em-odp provides the EM API for the applications while ODP acts as a portability
layer as well as providing APIs and services outside the scope of EM, e.g.
odp pkt-io or crypto APIs.

===============================================================================
Compilation
===============================================================================
First clone the ODP code (here using the ODP linux-generic version):
> git clone https://github.com/OpenDataPlane/odp.git
Go to the odp folder, create a build folder (optional) and configure,
compile and install odp (optionally with a user specified installation folder):
> ./bootstrap
> mkdir build && cd build
> ../configure --prefix=<odp install path>
If testing for performance, also give the --disable-abi-compat option to enable
more inlining of code. Try also --disable-shared if a static lib is OK for you.
(> ../configure --prefix=<odp install path> --disable-abi-compat \
      --disable-shared ...)
Enabling Link Time Optimization (LTO) for EM, ODP and the application might
give further performance benefits (--enable-lto).
See further configure options: ../configure --help
> make && make install

Clone EM-ODP code:
> git clone <em-odp repo>
Here <em-odp repo> can be e.g. https://github.com/openeventmachine/em-odp.git.
Go to the em-odp folder, optionally create a build folder, then configure,
compile and install em-odp. Separate build folders are useful if you want to run
em-odp with different odp installations or configurations.
> ./bootstrap
> mkdir build && cd build
> ../configure --prefix=<em-odp install path> \
    --with-odp-path=<odp install path> [--with-odp-lib=libodp-linux] \
    [--enable-esv --enable-check-level=3] ...
Use the options --enable-check-level=3 --enable-esv to help catch bugs
during development.
Try --disable-shared also for EM if a static lib is OK for you.
Enabling Link Time Optimization for EM, ODP and the application might give
further performance benefits (--enable-lto).
See further configure options: ../configure --help
> make && make install

Run some example application to test the functionality (path from build/), e.g.
> ./programs/example/fractal/fractal -c 0xe -t
Optionally change ODP and/or EM run-time options by specifying your own
runtime config files:
> ODP_CONFIG_FILE=my-odp.conf EM_CONFIG_FILE=my-em.conf \
  ./programs/example/hello/hello -c 0xe -t

Stop by pressing Ctrl-C.

===============================================================================
ODP configuration changes for EM (em-odp)
===============================================================================
The ODP configuration file lets the user tweak ODP settings.
For odp-linux, some settings are better modified when using EM (em-odp), read
about config file usage here: <odp>/config/README

Either change the default values in the config files and recompile or
override the defaults by providing your own config files at startup:
> ODP_CONFIG_FILE=my-odp.conf EM_CONFIG_FILE=my-em.conf \
  ./programs/example/hello/hello -c 0xe -t

odp-linux default configuration file: <odp>/config/odp-linux-generic.conf
Usually EM work better with the following config changes:

- Timer: Use inline timer implementation with EM.
    timer.inline = 0 -> 1

- Scheduler:
  - Priority level spread, the optimal value is typically the number of threads
    using the scheduler:
      sched_basic.prio_spread = 4 -> 'number of EM cores used'

  - Disable ODP automatically updated schedule groups, EM does not need them.
    (These options are actually DEPRECATED so prefer using the ODP API function
     odp_schedule_config() instead)
      sched_basic.group_enable: {all     = 1 -> 0
                                 worker  = 1 -> 0
                                 control = 1 -> 0}

  - Ordered queue reorder stash size. ODP might drop events if the
    stash/queue becomes full, causing EM ESV failure since event(s) dropped
    outside of EM can't be tracked, thus disable this.
      sched_basic.order_stash_size = 512 -> 0

- Stash: Disable strict stash size.
  Rely on stash 'put' return value to determine operation status.
     stash.strict_size = 1 -> 0

===============================================================================
CPU Architecture specific options
===============================================================================
Arch specific compilation options should be passed to EM and ODP via configure.
Example for a made up ARMv8 SoC called "soc-x":
odp-soc-x/build> ../configure … --with-platform=soc-x [default=linux-generic] …
   em-odp/build> ../configure … CFLAGS=”-O3 -march=armv8.2-a+… -mcpu=soc-x” \
                   --with-odp-path=<odp-soc-x install path> \
		   --with-odp-lib=libodp-soc-x …
The whole chain of used SW libs/dependencies should preferably be compiled
with the same -march/-mcpu options for best results:
  other-libs/build> ../configure … CFLAGS=”-O3 -march=armv8.2-a+… -mcpu=soc-x”

===============================================================================
Debug compilation
===============================================================================
For debugging activities it may be useful to compile separate debug libs and
executables of ODP and EM-ODP. The configuration script has to be run with
different options, other commands as above.

ODP debug configuration:
> ../configure --prefix=<odp debug install path> \
  --enable-debug --enable-helper-debug CFLAGS='-O0 -g3' ...
EM-ODP debug configuration:
> ../configure --prefix=<em-odp debug install path> \
  --with-odp-path=<odp install path> \
  --enable-check-level=3 --enable-esv --enable-debug-print CFLAGS='-O0 -g3' ...

===============================================================================
Packet-I/O (ODP-linux-generic + socket or dpdk based io)
===============================================================================
The default ODP for linux-generic has pkt-io support based on linux networking
and sockets that is quite slow but works nicely for development purposes.
The linux-generic version of ODP also supports dpdk based packet io. Use dpdk
based packet io on x86 targets for better performance.
Alternatively, try XDP sockets (AF_XDP) that has been added to odp-linux.
See odp-linux documentation for setup.

dpdk-io:
DPDK can be used to accelerate packet io on x86 targets with the linux-generic
version of ODP.
This is different from the separate odp-dpdk implementation described below,
but requires the same DPDK installation.
DPDK can be installed via e.g. apt-get or yum or compiled from source code from
https://www.dpdk.org/.
See the supported dpdk versions from the odp documentation.

DPDK compilation from source code (example for dpdk v19.11):
> cd dpdk
> make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
... edit x86_64-native-linuxapp-gcc/.config to your liking ...
> make install T=x86_64-native-linuxapp-gcc DESTDIR=<dpdk install path> \
  EXTRA_CFLAGS="-fPIC" (fPIC optional)
See ODP and DPDK instructions for more detailed info - e.g. uio vs vfio usage.
Note that 'DESTDIR' is required when compiling to generate a similar
installation as installing via apt-get would generate, use e.g.
DESTDIR=./install
After DPDK installation, configure and compile odp linux-generic:
> cd odp/build/ #assume odp/bootstrap already run
> ../configure --prefix=<odp install path> --with-dpdk-path=<dpdk install path>
> make && make install
Recompile em-odp and make sure the --with-odp-path points to the odp with
dpdk-pktio installed above, then start an example packet io application.
Note that DPDK interfaces are accessed using only indexes and not "ethX" names.
> cd em-odp/build
> sudo ./programs/packet_io/loopback -c 0xe -t -i 0,1

===============================================================================
Using odp-dpdk (note: different from odp linux-generic + dpdk pkt-io)
===============================================================================
There is also an ODP version that is optimized to be run on top of Intel DPDK
- this version is called odp-dpdk.
Note that also odp for linux-generic can use dpdk for packet-io, but the
optimized odp-dpdk utilizes dpdk internally for other functionality also.

DPDK instructions can be found from http://dpdk.org/doc/guides/

Get the odp-dpdk code:
> git clone https://github.com/OpenDataPlane/odp-dpdk.git
The odp-dpdk readme file (<odp-dpdk>/platform/linux-dpdk/README) describes how
to configure and compile dpdk and odp-dpdk on top of it.
ODP compilations as above but different installation directories i.e.
> ../configure --prefix=<odp-dpdk install path> ...

EM-ODP compilation with odp-dpdk:
> mkdir build--odp-dpdk && cd build--odp-dpdk
> ../configure --with-odp-path=<odp-dpdk install path> \
>              --with-odp-lib=libodp-dpdk --prefix=...