Week 4 presentation proposal

[leegrash]

Assignment Proposal

Title

A/B testing of LLMs to increase user experience on a platform using Optimizely

Names and KTH ID

• Gustaf Svensson ([email protected])
• Carl Lavö ([email protected])

Deadline

• Week 4

Category

• Presentation

Description

We want to show the advantages of refining the user experience of LLMs on a platform by A/B testing using the tool Optimizely. The presentation will show the strengths of using A/B testing to analyze usage of the platform and show metrics that can be gathered through Optimizely to tune the LLM specifically to the platforms needs.

Relevance

We think that understanding comparative testing methods is important to being able to actually use ML tools in practice. According to the MLOps wikipedia page, very few users of MLOps-practices actually found any real world benefits and being able to reliably test is key in determining whether or not to use it.

[sofiabobadilla]

Hi @leegrash
The idea of the proposal has potential, yet I would like you to improve both the description and the proposal focusing on how this will be connected to MLOps.

As it is now, the description makes me think you are working on the wrong week (week 2), while it should be week 4.

FYI: We have seen A/B testing today on week 2 so consider this when designing the presentation.

[leegrash]

Hi!

I understand. The way we see it, A/B testing could either be used to test different iterations LLMS on a platform with users. This is one way we could focus our presentation. Another way is to focus on the systematical testing advantages of LLMS with incremental changes to variables. What way would you suggest us to focus in order for us not to overlap with other presentations or topic weeks?

[sofiabobadilla]

A/B testing for LLMs by itself is not very original.
But if you talk about A/B testing for LLMs using X tool that can be more interesting.

For example this talk from week 2: Property-based testing in Python using Hypothesis

Also, in 2020 someone presented a similar topic (link) . Maybe you can get more inspiration for the presentation from it :)

[leegrash]

I understand. I updated the topic proposal to be more specific regarding tools and purpose. Is this okay?

[sofiabobadilla]

The proposal is clear now.
I'll merge.
@leegrash, FYI: I'll send you a calendar invite for your case
Good luck!