Skip to content

Commit

Permalink
Populate all the LFS data needed for running examples within the rele…
Browse files Browse the repository at this point in the history
…ase container (nv-morpheus#1976)

Move the data files needed for log-parsing from models to examples/data to avoid copying models to the release container

Authors:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: nv-morpheus#1976
  • Loading branch information
AnuradhaKaruppiah authored Oct 25, 2024
1 parent cf1ec22 commit be7e90f
Show file tree
Hide file tree
Showing 5 changed files with 11 additions and 6 deletions.
3 changes: 1 addition & 2 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ RUN --mount=type=bind,source=.,target=/opt/host_repo \
git clone file:///opt/host_repo/${MORPHEUS_ROOT_HOST} /tmp/morpheus_repo &&\
cd /tmp/morpheus_repo &&\
git lfs install &&\
./scripts/fetch_data.py fetch datasets examples
./scripts/fetch_data.py fetch examples

# ============ Stage: runtime ============
# Setup container for runtime environment
Expand All @@ -360,7 +360,6 @@ COPY --from=git_clone "/tmp/morpheus_repo/conda/environments/*.yaml" "./conda/en
COPY --from=git_clone "/tmp/morpheus_repo/docker" "./docker"
COPY --from=build_docs "/workspace/${MORPHEUS_ROOT_HOST}/build/docs/html" "./docs"
COPY --from=git_clone "/tmp/morpheus_repo/examples" "./examples"
COPY --from=git_clone "/tmp/morpheus_repo/models/datasets" "./models/datasets"
COPY --from=git_clone "/tmp/morpheus_repo/scripts" "./scripts"
COPY --from=git_clone "/tmp/morpheus_repo/*.md" "./"
COPY --from=git_clone "/tmp/morpheus_repo/LICENSE" "./"
Expand Down
3 changes: 3 additions & 0 deletions examples/data/log-parsing-config-20220418.json
Git LFS file not shown
3 changes: 3 additions & 0 deletions examples/data/log-parsing-validation-data-input.csv
Git LFS file not shown
6 changes: 3 additions & 3 deletions examples/log_parsing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,11 @@ Run the following from the root of the Morpheus repo to start the log parsing pi

```bash
python examples/log_parsing/run.py \
--input_file=./models/datasets/validation-data/log-parsing-validation-data-input.csv \
--input_file=./examples/data/log-parsing-validation-data-input.csv \
--model_vocab_hash_file=data/bert-base-cased-hash.txt \
--model_vocab_file=./models/training-tuning-scripts/sid-models/resources/bert-base-cased-vocab.txt \
--model_vocab_file=data/bert-base-cased-vocab.txt \
--model_name log-parsing-onnx \
--model_config_file=./models/log-parsing-models/log-parsing-config-20220418.json
--model_config_file=./examples/data/log-parsing-config-20220418.json
```

Use `--help` to display information about the command line options:
Expand Down
2 changes: 1 addition & 1 deletion examples/log_parsing/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
help="Model vocab hash file to use for pre-processing.")
@click.option('--model_vocab_file',
required=True,
type=click.Path(exists=True, dir_okay=False),
type=MorpheusRelativePath(exists=True, dir_okay=False),
help="Model vocab file to use for post-processing.")
@click.option("--model_seq_length",
default=256,
Expand Down

0 comments on commit be7e90f

Please sign in to comment.