[RELEASE] Morpheus v24.10.00 #1965
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Forward-merge triggered by push to branch-24.06 that creates a PR to keep branch-24.10 up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See [forward-merger docs](https://docs.rapids.ai/maintainers/forward-merger/) for more info.
* Update the `CMakeLists.txt` file for the two C++ developer guide examples * After updating `dependencies.yaml` run `rapids-dependency-file-generator` to update the Conda environment yamls * Update manifest.yaml * Update version string embedded in `*.pyi` files (unrelated change) Closes #1779 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1801
* Add a unique column to output of the `log_parsing` pipeline allowing for the output to be easily validated with the `scripts/compare_data_files.py` script. * Remove unused variables from `examples/log_parsing/postprocessing.py` Closes #1789 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1795
…th the `--file_source` flag (#1800) * Replace "./morpheus/data/*" as a data source for `vdb_upload` which is not a valid data source. * Add new serialized dataframes to `examples/data/vdb_upload` * `doca_guides.jsonlines`: Serialized Dataframe from data in `examples/doca/vdb_realtime/sender/dataset` * `nvidia_blogs.jsonlines`: Serialized Dataframe from two Nvidia blog posts: - https://blogs.nvidia.com/blog/mlperf-training-benchmarks/ - https://blogs.nvidia.com/blog/ai-security-steps/ Closes #1790 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1800
* Invoke `morpheus_utils_initialize_install_prefix` function after calling `project` Closes #1776 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1815
* Copy built documentation into the `docs` dir rather than the documentation source. * Add a new docker stage `build_docs` Closes #1788 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1806
* Avoids warning when executing `docker compose` commands Closes #1771 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1808
Currently CLI args always take precedence over Yaml config values, however since most of the CLI args have a default value in practice the Yaml config values are always ignored. * Differentiate the explicit CLI args the user specified on the command line from the CLI args which include default values the user didn't specify. * Move default values out of the code blocks into global dicts * Fix bug type-o causing Yaml schema definitions to be ignored. * Resulting code is 100 lines shorter Precedence order: 1. Explicit CLI args 2. Yaml config (if there is one) 3. Default CLI args Closes #1752 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1813
… builds (#1814) * Replace symlinks to markdown files under `examples/` with stub files using `include` directives. * Replace existing `https` to relative urls * Add stub file for the devcontainer instructions Closes #1783 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1814
* Create a models Docker image located at `models/docker/Dockerfile` along with a helper script `models/docker/build_container.sh` * Remove models from release container * Adjust examples documentation to use new image * Move the ransomeware models from `examples/ransomware_detection/models` into `models/ransomware-models` Closes #552 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - https://github.com/raykallen - Michael Demoret (https://github.com/mdemoret-nv) URL: #1804
…1809) * Fixes two related bugs which occur when running the `log_parsing` example with unexpected data. * Fix error in `LogParsingPostProcessingStage::__get_label_dicts` where there is a possibility that the `new_label` and `new_confidence` variables are undefined. * Fix error where the `tokenize_text_series` method can return a sequence_ids list with duplicate ids when a text series is passed in and one of the values has a length longer than `seq_len` and `truncation=False`, this prevents a down-stream error in the inference stage ``` RuntimeError: Inconsistent ID column. Last element in 'seq_ids' tensor, [1], must not extend beyond last message, [0] ``` Not sure if this is the best place to catch this error. Closes #1765 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1809
Closes #1818 Removed dependency on NVTabular ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1825
…1807) * Rather than copy directly from the host's repo, perform a `git clone` ensuring that any local build artifacts or uncommitted changes don't find their way into the release image. Closes #1769 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1807
* Fix bug in `DocaSourceStage` ring buffer initialization (#1820) * Incoming packet data is buffered into a `mrc::BufferedChannel`, limiting the amount of times the convert stage needs to acquire the GIL. * Launch the `_packet_gather_payload_kernel` kernel with a 2D grid, treating the byte-offset as the second axis. * No longer convert 32bit int IP data into strings in the gather kernel, instead buffer 32bit int data, and let cuDF convert the data all in one big pass prior to constructing the cuDF DataFrame. * No longer outputs fixed data sizes. * Perform the sizes to offsets calculation with MatX rather than using a deprecated cuDF method. * Add `PacketDataBuffer` struct to wrap the three pieces of data needed to be buffered: header, payload and payload sizes. * Split `doca_stages.hpp` into `doca_source_stage.hpp` and `doca_convert_stage.hpp` * Rename `doca_convert.cpp`->`doca_convert_stage.cpp` and `doca_source.cpp`->`doca_source_stage.cpp` * Add command line flags to `examples/doca/run_udp_convert.py` and `examples/doca/vdb_realtime/sender/send.py` * Move constants in `morpheus/_lib/doca/include/morpheus/doca/common.hpp` into the `morpheus::doca` namespace (was in global). * Fix CI checks for DOCA based code * Remove unused code Closes #1820 This PR punts on fixing #1827 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1731
* Add [vale](https://github.com/errata-ai/vale) as a documentation linter to CI * Add custom vocabulary (`ci/vale/styles/config/vocabularies/morpheus/accept.txt`). * Add sphinx's linkcheck to the documentation builds * Fix spelling & grammar errors found in existing documentation Closes #545 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - https://github.com/hsin-c URL: #1821
* This was needed to prevent documentation builds from loading merlin, which would require a GPU. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1830
… raise an exception rather than segfaulting (#1829) * MRC PR nv-morpheus/MRC#402 fixed this issue, this PR adds some unittests to ensure this. * Update `Pipeline.build_and_start` to re-raise any exceptions raised during build, this prevents `Pipeline.join` from being called which avoids a failed assert in `join` since the expected future was not created. Closes #953 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1829
* Improved type hints for both the input parameters and the return types Closes #1812 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1831
This is the first PR to update the source layout. It moves the morpheus source code to a new sub-tree that will allow breaking out morpheus functional blocks from the morpheus-base - ``` . |___ CMakeLists.txt |___ python | |___ CMakeLists.txt | |___ morpheus | | |___ CMakeLists.txt | | |___ setup.py | | |___ setup.cfg | | |___ MANIFEST.in | | |___ morpheus | | | |___ __init__.py | | | |___ _lib | | | |___ etc. | |___ morpheus_N ``` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - David Gardner (https://github.com/dagardner-nv) - Christopher Harris (https://github.com/cwharris) URL: #1836
* Prevents isort from moving import os `_utils` to the third-party section ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #1842
* Fixes a bug when C++ mode is enabled, the file would be written twice. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1840
* Ensure that Python sources which run indefinitely actually shutdown when requested by the pipline (ex: when the user hits Ctrl+C). * The `KafkaSourceStage` already handled this properly, move the `stop` method from this class into the `SingleOutputSource`. * Add `request_stop` and `is_stop_requested` methods to `SingleOutputSource`. * Update all existing source stages which run indefinitely. * Add new `should_stop_fn` constructor argument to `DirectoryWatcher`, sources which use the watcher (`AutoencoderSourceStage` & `AppShieldSourceStage`) pass in their `is_stop_requested` method allowing the watcher to know when to shut down. * Move the RSS source generator method from the RSS source module into the `RSSController`, and update the `RSSSourceStage` to use the `RSSController` directly rather than the RSS module. This avoids the problem where modules require all config values to be JSON serializable, preventing a callback function from being passed to a module. Closes #1837 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #1839
Closes ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #1843
* Fixes isort vscode integration * Fixes an issue where `tests` was listed in `src_paths` but this was expected to be relative to `python/morpheus` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #1854
Update `isort` settings file path in `fix_all.sh` to `${PROJ_TOML}` so that this script finds the config path correctly.
This is a follow-up PR of #1854.
## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.
Authors:
- Yuchen Zhang (https://github.com/yczhang-nv)
Approvers:
- David Gardner (https://github.com/dagardner-nv)
- Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)
URL: #1855
Fixes Kafka download script to use an available Kafka version. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #1856
…celled" (#1860) CIs are failing because of #1736 not playing well with rapidsai/shared-workflows#237 As a stopgap I am reverting the changes in #1736 Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #1860
This PR breaks out the LLM source code into -
1. A separate subdirectory - python/morpheus_llm
2. Compiled as a separate shared lib -
libmorpheus_llm.so
|_ llm/__init__.cpython-310-x86_64-linux-gnu.so
3. And setup as a separate python package
a. morpheus_llm - all llm modules are now in this namespace
b. with the following distribution -
./build/python/morpheus_llm/dist/morpheus_llm-24.10.0a0+46.g125a6be9.dirty-py3-none-any.whl
To build and install all components in a dev env you can use these steps (steps also updated in contributing.md) -
1. ./scripts/compile_all.sh
2. pip install -e python/morpheus
3. pip install -e python/morpheus_llm
Items of note -
To ensure right order of build-n-install morpheus_llm has dependencies on the following targets -
1. cudf_helpers
2. morpheus
3. all "py_morpheus" targets
Ref: python/morpheus_llm/morpheus_llm/_lib/cmake/libmorpheus_llm.cmake
Pending (major) items -
1. Need to breakout llm related build/conda environment requirements from the base morpheus requirements
2. Need to breakout llm python tests (cpp tests have been moved as a part of this commit).
Authors:
- Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)
Approvers:
- David Gardner (https://github.com/dagardner-nv)
URL: #1853
Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: #1998
- Dependencies for DFP container were recently moved from `examples/digital_fingerprinting/production/conda_env.yaml` to `conda/environments/examples_cuda-125_arch-x86_64.yaml` but `python-logging-loki` was not included. - Add `python-logging-loki` to `example-dfp-prod` in `dependencies.yaml`. - Remove `conda_env.yaml` since it's no longer being used anywhere. Closes #1917 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #1977
* Fixes an issue where an `int` is being returned in the DFP rolling window stage, likely caused by the recent update to the version of pandas being used. * Update file paths to reflect the move of the `dfp_duo_pipeline.py`, `dfp_azure_pipeline.py` & `dfp_integrated_training_batch_pipeline.py` scripts * Document installing the missing dependencies needed for the DFP Grafana example Closes #1994 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #1995
) Closes #1934 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #2004
This is to account for the changes done during DFP refactoring Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #2003
* The deserialize stage no longer actually deserializes, but instead chunks incoming `MessageMeta` objects and converts to `ControlMessage` Closes #2007 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: #2009
Closes #1999 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #2008
…2006) Closes #2002 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Christopher Harris (https://github.com/cwharris) URL: #2006
…#2016) Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: #2016
Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: #2017
…e correct rows for each batch (#2015) * `DeserializeStage` now performs a copy of the rows needed for each batch, ensuring each output `ControlMessage` now has a unique `MessageMeta` and underlying `DataFrame` * Works-around issue where calling `SlicedMessageMeta::get_mutable_info().checkout_obj()` returns the entire `DataFrame` * This unfortunately means that `DeserializeStage` will require the GIL. * Remove unused `make_output_message` method * Avoid some redundant code in `SlicedMessageMeta` (unrelated but I was in this part of the code). Closes #2002 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) URL: #2015
Earlier the morpheus-pipeline stage was using the morpheus-runtime container as the base image. This change switches the base container to nvidia/cuda and installs the morpheus-dfp library. There is a minor docs fixup in conda_packages.md that I pull in based on the review comments. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #1971
* Updates the version of the Triton Inference server container being used by Morpheus from `23.06` to `24.09`. * Includes a re-trained `ransomw-model-short-rf` (PR #2023), the original model was not compatible with Triton 24.09 * Update the output dimensions of the `log-parsing-onnx` model per #1727 Closes #2000 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) - Tad ZeMicheal (https://github.com/tzemicheal) Approvers: - https://github.com/raykallen - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: #2001
Requires changes from MRC PR: nv-morpheus/MRC#502 Closes #1519 Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #1963
This bug is caused by the transition from `MultiMessage` to `ControlMessage`. `inference_stage.py::InferenceStage::_build_single` calls `_convert_one_response` in a loop for a batch, and the argument is passing is the same for the whole batch, but inside `_convert_one_response` we're grabbing the tensors and assigning starting at zero, so the tensors are overwriting each other and causes the issue. Added `batch_offset` variable to keep track where the next incoming tensor should start writing to the output message. Closes #2019 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Christopher Harris (https://github.com/cwharris) URL: #2024
* Use the index of the incoming DF as the source of the `doc` field not the first column of the sequence_ids * Fix typeo in docstring for `TableInfo::num_indices` (unrelated) * Remove declaration for unimplemtned/unused function in `python/morpheus/morpheus/_lib/include/morpheus/stages/deserialize.hpp` (unrelated) Closes #2029 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: #2031
- Fixes benchmark tests that were broken from latest code updates - Add `pynvml` dependency back to conda dev environment. It was removed when `nvtabular` was removed. - Remove broken E2E autoencoder test since Starter DFP is being removed. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #1958
Closes #2030 This is needed to run root_cause_analysis in the release container. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Christopher Harris (https://github.com/cwharris) URL: #2032
See #2026 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #2027
At some point these got out of sync where the CLI was using add-scores, but the Python API was using the filter stage, and some of the docs were using add-scores as well ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #2037
Closes ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - https://github.com/raykallen URL: #2036
…rors (#2039) Closes #2038 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #2039
mdemoret-nv
approved these changes
Nov 1, 2024
AnuradhaKaruppiah
approved these changes
Nov 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-24.10andv24.10releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-24.10until release (merging of this PR).All other development PRs should be retargeted towards the next release branch:
branch-25.02.What is the purpose of this PR?
branch-24.10intomainfor the release