All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
empty
- CMakeLists.txt: use C++17 standard and use FindHDF5 module for linking with HDF5 libraries.
- Update the NOTICE file and prepare for the first public.
- sender: add support to select any event range within a list of files.
- GenericClient, GenericServer: enable TCP keepalive and set the related socket options, which default to 60s idle time.
- add nginx proxy for dashboard.
- Image feature extraction algorithms including global mean/RMS calculation and peak finding (based on MSSE method) has been implemented by CUDA in ingester.
- Online data filtering logic and the related dynamic configuration parameters in ingester have been applied and added.
- Change deployment cpu affinity considering the hardware NUMA configuration, i.e. the position of IB network card.
- Build and deloy all components utilizing the GitLab CI and Registry instead of Docker Hub.
- Added support to set the port number of UDP/TCP client, then the bandwidth of two 10Gb data links under bond mode 4 with option xmit_hash_policy=layer3+4 can be fully utilized.
- dispatcher: added configurations dgram_queue_size and other_cpu_list.
- dispatcher: added support to set cpu affinity for all other worker threads except the UDP packet receiving thread.
- Use 4 combiners for deployment instread of 2.
- Use taskset to set cpu affinity for the deployment of combiner, ingester and monitor, in order to minimize the effect of the high load of these processes on the UDP packet receiving thread and thus reduce the packet loss rate.
- Use ByteBuffer for dynamic network data caching instead of std::vector<char>. ByteBuffer is implemented by malloc() which only allocate memory but not initialize the data in it like the STL default allocator, therefore it has better performance especially when the buffer size is large.
- Added two libraries utilities and algorithms in diffraflow/shared folder, and library common was also moved into this folder.
- Added class ImageDataField to represent ImageDataType::Field, and it can be included in CUDA code.
- Added file nvidia-mps.yaml for deploying CUDA MPS on k8s and its function was tested.
- ingester: Cache memory for the data processing pipeline was changed from dynamic to static using ring buffer which was implemented by class IngImgFtrBuffer.
- ingester: Copying image data in/out GPU and doing calibraion on GPU were added and tested on a k8s cluster.
- CMake minimum required version was changed to 3.18 which has the native support for CUDAToolkit.
- Removed class ImageWithFeature.
- CMake scripts are optimized and use ninja build by default.
- Version information can be generated by cmake and git during the build process instead of hard-coding in a header file.
- Add CUDA support for the build system of both cmake and docker.
- Add a project testing/cudatest to test the CUDA capability of docker and k8s, and it works.
- The build system was switched from Gradle to CMake.
- README.md files were revised.
- testing/console_test (a Java project) was removed.
- sender: can repeatedly send unlimited number of images using limited dataset.
- ingester: the number of images writting into each data file can be randomized, in order to avoid many file open/close operations that can occur at very similar time.
- ingester: add support of non-blocking cache for image data saving. When the speed of data saving is lower than that of processing, the cache queue will be full and some images may be lost.
- combiner: add support of non-blocking cache for image data processing. When the speed of data processing is lower than that of time alignment, the cache queue will be full and some images may be lost.
- Add deployment files for local Docker Registry and Docker Registry UI.
- Add script bunch_id_2_event_num.py.
- Use local Docker Registry instead of Docker Hub for the deployment of all core components.
- Make it default to deploy two combiners on each GPU node.
- aggregator: add configuration for subscription name and add support for consuming compacted topics from Pulsar.
- Add deployment files for dashboard which is maintained in another project.
- Add header 'Access-Control-Allow-Origin: *' for all http responses.
- controller: add code to handle HTTP OPTIONS requests from web browser by fetch API.
- The code of aggregator is simplified: use a common consumer class for all components, while different components use different instances.
- deploy trigger, aggregator, controller and dashboard on different nodes instead of the same one.
- monitor: use fixed energy range for image data conversion for visualization instead of dynamic one.
- sender and dispatcher: buffer size of UDP socket can be configured by configuration file.
- monitor: fix a energy range mistake in image data conversion function.
- combiner: fix a deadlock problem related to clear worker thread.
- BlockingQueue: fix the forgetting of thread notifying in offer(), get(), try_offer() and try_get() methods.
From this version, all core components including dispatcher, combiner, ingester, monitor, controller and aggregator, and the necessary testing components including sender and trigger, are functional and tested on an 8-node kubernetes cluster.
- draw_current_event.py: fetch current image data and analysis results from monitor or controller and draw the image.
- aggregator is implemented and tested, and the deploying YAML files are added.
- scripts/deploy/{deploy_services.sh, undeploy_services.sh}: scripts to quickly deploy and undeploy services including ZooKeeper, BookKeeper and Pulsar Broker.
- scripts/config_net.sh: configure network parameters for better UDP data transporting.
- GenericDgramReceiver: the size of receiving buffer is changed from 64 MiB to 32 MiB.
- controller: added support for metrics reporting via REST and message queue.
- Enabled object deletion protection via mutex for the server manager of all components.
- ImageDataSmall: fixed an index related bug for private method copy_from_().
- trigger and sender were implemented. The format of raw data sent from sender to dispatcher is defined by docs/rawdata-format.txt.
- Added support to transfer data from sender to dispatcher by UDP and it was tested using a 8-node kubernetes cluster.
- GenericConnection, GenericClient: added support to receive data into dynamically allocated vector, which is helpful for reducing data copying.
- ImageFrameRaw, ImageDataRaw: added these two classes to support doing event alignment before decoding.
- ImageDataSmall, ImageVisObject: these two classes were added to convert and send reduced image data to web client for visualization.
- Added support to transfer data from combiner to ingester using Unix Domain Socket.
- Data calibration in ingester was implemented and tested.
- Changed the data structure of ImageFrame and ImageData according to the format defined in docs/rawdata-format.txt.
- The way of data transferring from combiner to ingester was changed from a request-response mode to a long-pulling like mode, in order to increase data throughput.
- Raw data decoding was changed to be done after event alighment and moved from combiner to ingester, and the MessagePack serialization between combiner and ingester was cancelled, instead combiner directly send the raw data of image frames to ingester.
- Instead of sending the full image data with float data type, monitor converts and sends the image data by using only one byte to represent each pixel to reduce data size and the calculation overhead on client side.
- Data compression for data transferring from dispatcher to combiner was removed, as data compression will use a lot of CPU time and actually reduce the data transfering speed.
- Added file .clang-format and applied clang-format for all C++ code.
- Added the YAML deployment files for ingester, monitor and controller, and together with dispatcher and combiner, the deployments of all these 5 components on demo machine are tested.
- Added some programs to convert the EuXFEL data published on CXIDB for the purpose of using it as the test data for this project. These programs include scripts/{calib_gen.py, alignment_index_proc.py, event_select_proc.py, event_view_proc.py, raw_data_gen.py} and testing/generator. And the raw data format is defined by file docs/rawdata-format.txt.
- Added some kubernetes job deployment files for doing the data generating and converting works on demo machine.
- Dockerfile: switched to use Pulsar v2.5.2 instead of v2.5.0.
- combiner now will force doing alignment if the module data in cache stays for a relatively long time. This is useful to clear the cache after a run finished but there is still some partial data remained in cache.
- Finished the code of monitor to fetch image data from ingester by an HTTP client, as well as the online data analysis template code, the HTTP server to provide image data and analysis results on request.
- Finished the load balancer in controller for multiple monitors running on different nodes, and the support for ZooKeeper operations via RESTful API, including HTTP verbs GET, POST, PUT, PATCH and DELETE.
- Added the code to collect metrics for combiner, ingester and monitor.
- Finished the template code to do data processing pipeline in ingester, and the specific data processing algorithms in each step of the pipeline need to be implemented later.
- Added classes to read/write image data from/into HDF5 files, in which the SWMR mode is supported. Those classes are used to sink data into storage in ingester.
- ingester starts an HTTP server when it starts to provide the latest image data on the request from monitor.
- DynamicConfiguration: update, delete and fetch operations of config_map on ZooKeeper now support version check.
- CmbImgCache: the code to do time alignment is added, but only tested with single-detector data.
- The data exchanging logic between combiner and ingester is added and tested, and it is ready to write data processing pipeline in ingester.
- DynamicConfiguration: use JSON instead of MessagePack to serialize configuration data stored in ZooKeeper.
- GenericServer: use async and future to start and stop server, this make it convenient to run multiple servers in one executable.
- BlockingQueue: data consuming functions return false only when the internal queue is empty, even if the queue is stopped.
- Add code to report runtime metrics. Two methods are applied. One is using Pulsar C++ client to periodically send runtime metrics to a Pulsar message queue cluster. The other one is using C++ REST SDK to start a http server, which can reply the latest runtime metrics on the GET request from client. The data of runtime metrics is aggregated and serialized by JSON.
- Data transerring code using socket is optimized. The related classes include GenericClient, GenericConnection and DspSender.
- Use MSGPACK_DEFINE_MAP to do serialization using MessagePack, which will serialize C++ object into JSON object and keep the variable name in the serialized data.
- Add LZ4 and ZSTD compression support for the data transferring from dispatcher to combiner.
- Add deploy folder, which contains the Kuberenetes deployment YAML files for all components and services. The deployment of dispatcher and combiner on a local vm-built Kubernetes cluster was tested, and also for that of ZooKeeper, BookKeeper and Pulsar.
- Install Apache Pulsar C++ client in Dockerfile, as Pulsar is planned to be used for system runtime monitoring.
- Add class DynamicConfiguration which can dynamically update configurations at runtime. This class works based on ZooKeeper.
- Add controller which is responsible for updating the configurations of ingester and monitor in ZooKeeper at runtime, and fetching online analysis results from monitor in a round-robin way on the request from front-end client.
- use MessagePack library instead self-writting code to serialize ImageFrame and ImageData.
- log library is switched from boost.log to log4cxx.
- use a specific OptionsManager to parse command line parameters
- dispatcher: combiner addresses file is now set by a command line option instead of inside configuration file.
- The Dockerfile is optimized, and the CI workflow is setup.
- GenericServer now can listen on a specific host.
- Add Dockerfile and start to use Docker for automated building.
- The code (using Snappy) to compress the data transferred from dispatcher to combiner is added.
- Data compression between dispatcher and combiner can be switched on or off by dispatcher's configuration file using compress_flag.
- Serialization: use try-catch clause to check exceptions instead of using lots of if-else.
- 'dispatcher' is rewritten by C++, and the Java version is removed.
- Use an unified data exchange protocol between different components, and the protocol is defined by docs/protocol.txt.
- The data transferred from 'dispatcher' to 'combiner' can be compressed, but now the compression code is not yet added.
- Add this changelog file.