Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send deactivate flush if need_flush is set #1573

Merged
merged 3 commits into from
Nov 27, 2024
Merged

Conversation

mkeeter
Copy link
Contributor

@mkeeter mkeeter commented Nov 26, 2024

Otherwise, deactivate can fail in a case where there are no active jobs but need_flush is set: it will deactivate right away (because there are no active jobs), then the automatic flush will be sent to a Downstairs in an invalid state (Deactivated or New).

This PR will hopefully fix #1571

@mkeeter mkeeter requested review from jmpesp and leftwo November 26, 2024 18:44
Copy link
Contributor

@leftwo leftwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming this is what james has been testing in CA, yeah.


// Send exactly IO_CACHED_MAX_JOBS to force a Barrier to be sent. The
// barrier empties out the active list, so deactivation may proceed.
for _ in 0..10000 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should this instead be IO_CACHED_MAX_JOBS + N

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IO_CACHED_MAX_JOBS isn't public in the crucible crate, because it's (mostly) an implementation detail. I'm ambivalent about making it public versus hard-coding it here, let me know if you have strong opinions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...and it must be an exact multiple of IO_CACHED_MAX_JOBS, because otherwise there would be leftover jobs in can_deactivate_immediately and deactivation wouldn't skip the final flush.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it must be exactly that value, I'd say make it public or have a way to return it (less excited about that option)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alternatively, mimic what omicron does and use

#[cfg(any(test, feature = "testing"))]

to guard if it's pub or not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, done in 7dc0be1

There's not a particularly clean way to enable / disable the pub visibility qualifier, so I added a dedicated module that re-exports the constant.

Copy link
Contributor

@jmpesp jmpesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm running this right now, but I think the test is definitive. 🚢

@leftwo
Copy link
Contributor

leftwo commented Nov 27, 2024

I've also been running this on the dublin racklett setup. I (accidentally) verified I could reproduce the problem, which is good. I'm now at 17 instances created and no issues.

@leftwo
Copy link
Contributor

leftwo commented Nov 27, 2024

Overnight testing of 40 more instance starts, still good.

@leftwo leftwo merged commit 2cfc7e0 into main Nov 27, 2024
16 checks passed
@leftwo leftwo deleted the mkeeter/deactivate-need-flush branch November 27, 2024 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Propolis panic from crucible: enqueue should not be called from state New
3 participants