Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smarter action shuffler #39

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pbruski
Copy link

@pbruski pbruski commented Jan 28, 2020

No description provided.

@pbruski pbruski force-pushed the smarter-action-shuffler branch 2 times, most recently from 1a0566f to 0b4e0f6 Compare January 28, 2020 10:44
@@ -43,6 +43,9 @@ dependencies {
implementation("org.apache.commons:commons-math3:3.6.1")
implementation("com.atlassian.performance.tools:concurrency:[1.0.0,2.0.0)")
implementation("com.atlassian.performance:selenium-js:[1.0.0,2.0.0)")

runtimeOnly("org.jetbrains.kotlin:kotlin-reflect:${kotlinVersion}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uh oh, reflection = danger zone. We're bypassing a lot of compilation-time guarantees.

}

companion object {
fun createRandomisedScenario(seededRandom: SeededRandom, actionProportions: Map<Action, Int>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the problem we're trying to solve here?

Do skipped actions malform the ActionMetric.label proportions too much? Our initial workaround was to run the test for a longer time so they get averaged out.
If that is not effective enough, then we can avoid action skips by using a self-loading Memory, e.g. pass this to an action during Memory.recall:

fun recall(): String {
    if (issueKeys.isEmpty()) {
        SearchJqlAction(jira, throwawayActionMeter, IssueJqlMemory(), this).run()
    }
}

This way you can avoid skips with local logic rather than a global order-controlling scheme.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I want to preserve the requested proportions. During our runs, ~1500 view issue calls are skipped. This looks like a minimally invasive way of achieving this, without altering the way scenarios are created.
The problem with hiding the query inside a different action is that you lose control over proportions - if there are permission issues, the JQL action can be executed as often as view issue, which we don't want. If we stick to sorting, everything is transparent and there are no surprises hiding in the implicit initialisation.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same problem affects ViewBoard, btw.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try a different question: is there any benefit to keeping a fully-randomised shuffle in the scenario?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔
I'll come back to this next week

@pbruski pbruski force-pushed the smarter-action-shuffler branch 2 times, most recently from 4d10d91 to d9a2e73 Compare February 15, 2020 21:24
@pbruski pbruski requested a review from dagguh February 15, 2020 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants