Executable Tutorial Submission - samkh, miladsf

[M1l0d]

Assignment Proposal

Title

Monitoring ML Model Predictions with Prometheus

Names and KTH ID

• Sam Khosravi ([email protected])
• Milad Farahani ([email protected])

Deadline

• Task 3

Category

• Executable tutorial

Description

In this tutorial we will set up a python script that logs a ML models predictions.
This is integrated with Prometheus which will mointor them.
The goal is to learn how to integrate Prometheus with a Python application, which is then will be able to monitor the model to understand if it for example will work well when put into production.

Relevance

Monitoring ML model prediction with Prometheus is highly relevant to DevOps because it introduces automation and observability to the machine learning lifecycle.
In DevOps, ensuring that systems are continously monitored for performance and reliability is crucial and this extends to machine learning models in production.
By integrating Prometheus to track metrics like prediction accuracy or latency, teams can proactively identify issues to reduce downtime and improve model reliability.

KillerCoda: https://killercoda.com/m1l0d/scenario/extut

Github: https://github.com/M1l0d/killercoda

[dd2482-bot]

Readme is not correctly formatted
Need exactly: ['Assignment Proposal', 'Title', 'Names and KTH ID', 'Deadline', 'Category', 'Description']

Got: ['Assignment Submission', 'Title', 'Names and KTH ID', 'Deadline', 'Category', 'Description', 'Links']

[dd2482-bot]

Readme is not correctly formatted
Need exactly: ['Assignment Proposal', 'Title', 'Names and KTH ID', 'Deadline', 'Category', 'Description']

Got: ['Assignment Proposal', 'Title', 'Names and KTH ID', 'Deadline', 'Category', 'Description', 'Links']

[bjornthiberg]

Feedback for “Monitoring ML Model Predictions with Prometheus”

I/We certify that generative AI, incl. ChatGPT, has not been used to write this feedback. Using generative AI without permission is considered academic misconduct.

Feedback given by:

• Björn Thiberg ([email protected])
• Omid Fattahi Mehr ([email protected])

High-level strengths

The tutorial is well-paced, has an appropriate amount of humor and is relevant to DevOps. Specifically, we appreciate the code the user writes procedurally grows in chunks, rather than just being provided all at once. It also immediately gets into the technical nit-and-grit of writing code and monitoring something in Prometheus. Perhaps most importantly, the tutorial is easy to follow and there is never any confusion as to what the next step is.

High-level weaknesses

There tutorial could benefit from some additional visualization, as it is occasionally difficult to know what is happening behind the scenes for someone with a limited understanding of the used tools. Perhaps the Prometheus dashboard could be viewed (KillerCoda allows for opening localhost webpages) or a screenshot of Prometheus could be included. Also, for someone that has never worked with an ML workflow, is the lack of a clear problem statement. It would be great and essential to explain why we need Prometheus in the first place. For example, what the specific use cases are and when this tool would be really useful.

Specific feedback, per section

Intro:

Good humor, well written. 3 suggestions for improvement:

• State clearly what Prometheus is, for those that do not know (just one sentence).
• State the “intended learning outcomes” more clearly in the intro (part of the mandatory requirements), perhaps as bullet points.
• Give a little bit more information about the Wine dataset, what the features are and what the targets are, so we know what the model is actually learning.

Step 1

Clear and specific instructions. Especially good how the “ubuntu $” line is included, to show how the user should know when the pip command is completed.

It was also a good decision to have the explanations for what has been installed after the installation instructions, so that the user can read it while waiting.

A potential improvement: Add {{copy}} to the pip3 install… command to allow the user to copy it by clicking (or make it inline instead of a code block). This is the same for several steps in the tutorial. KillerCoda has code examples for scenarios here:

• https://github.com/killercoda/scenario-examples (specifially, the code-actions and code-snippets directories/tutorials).

Step 2

You might want to consider making these into one command, and add {{copy}} to enable multi-line copy. Or split into two commands so that they can be copied individually:

mkdir ~/ML_prometheus
cd ~/ML_prometheus

Step 3

Nothing to add here!

Step 4

We appreciate the fact that the code to be copied is provided in several steps, even though it’s to be copied at the same time. This makes it much more readable.

Code comments are also good. For someone who would have never worked with these tools, it is quite intuitive, and the code is well-structured.

Step 5

You could consider splitting the second paragraph into bullet points, or use some other method to make the explanation of what we are going to do more clear.

Perhaps you could also include a short explanation as to why we are doing this (something like “we are going to create a counter, so that we have a metric we can monitor in Prometheus).

Step 6

Very nice with the “#new” blocks provided like that. Makes it obvious to the user what is going on. Also, the fact that each new line of code has been commented makes it easy for the user to follow what happens.

Step 7

Perhaps you could add some more context and an explanation as to why we would want to add more monitors here:

“Based on the amount of monitor needed, you can add additional run_model() lines”

Also, you can access killer coda-hosted sites in the Traffic/Ports page in the top right hamburger menu. So we can see the Prometheus GUI. But perhaps it was intended to not use this to keep the tutorial stream-lined.

Finish

In the “What we can see from the output is” part, one could include specific lines from the printed output, and write below to explain it step-by-step. It is difficult as a user to look back and forth between output and the explanations.

It would have been great to put more emphasis on this part as it pertains a lot more to the Prometheus tool, which is the main focus of the tutorial. In the end it felt a little bit short, but what has been presented feels easy to understand without getting additional documentation and is relevant.