Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve logging for MLflow metrics #42

Merged
merged 4 commits into from
Feb 15, 2024

Conversation

anujsinha3
Copy link
Collaborator

@anujsinha3 anujsinha3 commented Feb 14, 2024

Change Description

#41 - Improving the MLflow to handle (prevent overriding) metrics logging when running eval() or eval_prompts() with multiple prompts.

Example Run: https://ml.azure.com/experiments/id/e1882f37-87e8-4ff9-930a-7aa050693072/runs/gray_coat_sy48gpw0jc?wsid=/subscriptions/e4f4cccd-ac89-41b8-aae6-896e5a8471e4/resourcegroups/AutoRADoc/providers/Microsoft.MachineLearningServices/workspaces/AutoRAML&tid=f6b6dd5b-f02f-441a-99a0-162ac5060bd2#overview

  • My PR includes a link to the issue that I am addressing

Solution Description

Code Quality

  • I have read the Contribution Guide
  • My code follows the code style of this project
  • My code builds (or compiles) cleanly without any errors or warnings
  • My code contains relevant comments and necessary documentation

Project-Specific Pull Request Checklists

Bug Fix Checklist

  • My fix includes a new test that breaks as a result of the bug (if possible)
  • My change includes a breaking change
    • My change includes backwards compatibility and deprecation warnings (if possible)

New Feature Checklist

  • I have added or updated the docstrings associated with my feature using the NumPy docstring format
  • I have updated the tutorial to highlight my new feature (if appropriate)
  • I have added unit/End-to-End (E2E) test cases to cover my new feature
  • My change includes a breaking change
    • My change includes backwards compatibility and deprecation warnings (if possible)

Documentation Change Checklist

Build/CI Change Checklist

  • If required or optional dependencies have changed (including version numbers), I have updated the README to reflect this
  • If this is a new CI setup, I have added the associated badge to the README

Other Change Checklist

  • Any new or updated docstrings use the NumPy docstring format.
  • I have updated the tutorial to highlight my new feature (if appropriate)
  • I have added unit/End-to-End (E2E) test cases to cover any changes
  • My change includes a breaking change
    • My change includes backwards compatibility and deprecation warnings (if possible)

@anujsinha3 anujsinha3 marked this pull request as ready for review February 14, 2024 23:03
@anujsinha3 anujsinha3 requested a review from carlosgjs February 14, 2024 23:03
@codecov-commenter
Copy link

codecov-commenter commented Feb 14, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (aac9880) 98.65% compared to head (8db2bc0) 98.63%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #42      +/-   ##
==========================================
- Coverage   98.65%   98.63%   -0.03%     
==========================================
  Files           8        8              
  Lines         298      292       -6     
==========================================
- Hits          294      288       -6     
  Misses          4        4              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@anujsinha3 anujsinha3 enabled auto-merge (squash) February 14, 2024 23:06
Copy link
Collaborator

@carlosgjs carlosgjs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See inline comment. Can you also add a link to an AzureML run in the PR description? Thx!

src/autora/doc/pipelines/main.py Outdated Show resolved Hide resolved
@anujsinha3
Copy link
Collaborator Author

Also updated the code to use mlflow.logMetrics() - allows for multiple metrics logging via dictionary instead of log.metric() - which logs only one key-value pair at a time.

@anujsinha3 anujsinha3 requested a review from carlosgjs February 15, 2024 02:26
@anujsinha3 anujsinha3 merged commit daf1b1c into main Feb 15, 2024
9 checks passed
@anujsinha3 anujsinha3 deleted the perf-improve-ML-flow-logging branch February 15, 2024 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants