Implement record based Crucible reference counting #6805

jmpesp · 2024-10-09T00:29:36Z

Crucible volumes are created by layering read-write regions over a hierarchy of read-only resources. Originally only a region snapshot could be used as a read-only resource for a volume. With the introduction of read-only regions (created during the region snapshot replacement process) this is no longer true!

Read-only resources can be used by many volumes, and because of this they need to have a reference count so they can be deleted when they're not referenced anymore. The region_snapshot table uses a volume_references column, which counts how many uses there are. The region table does not have this column, and more over a simple integer works for reference counting but does not tell you what volume that use is from. This can be determined (see omdb's validate volume references command) but it's information that is tossed out, as Nexus knows what volumes use what resources! Instead, record what read-only resources a volume uses in a new table.

As part of the schema change to add the new volume_resource_usage table, a migration is included that will create the appropriate records for all region snapshots.

In testing, a few bugs were found: the worst being that read-only regions did not have their read_only column set to true. This would be a problem if read-only regions are created, but they're currently only created during region snapshot replacement. To detect if any of these regions were created, find all regions that were allocated for a snapshot volume:

SELECT id FROM region
WHERE volume_id IN (SELECT volume_id FROM snapshot);

A similar bug was found in the simulated Crucible agent.

This commit also reverts #6728, enabling region snapshot replacement again - it was disabled due to a lack of read-only region reference counting, so it can be enabled once again.

Crucible volumes are created by layering read-write regions over a hierarchy of read-only resources. Originally only a region snapshot could be used as a read-only resource for a volume. With the introduction of read-only regions (created during the region snapshot replacement process) this is no longer true! Read-only resources can be used by many volumes, and because of this they need to have a reference count so they can be deleted when they're not referenced anymore. The region_snapshot table uses a `volume_references` column, which counts how many uses there are. The region table does not have this column, and more over a simple integer works for reference counting but does not tell you _what_ volume that use is from. This can be determined (see omdb's validate volume references command) but it's information that is tossed out, as Nexus knows what volumes use what resources! Instead, record what read-only resources a volume uses in a new table. As part of the schema change to add the new `volume_resource_usage` table, a migration is included that will create the appropriate records for all region snapshots. In testing, a few bugs were found: the worst being that read-only regions did not have their read_only column set to true. This would be a problem if read-only regions are created, but they're currently only created during region snapshot replacement. To detect if any of these regions were created, find all regions that were allocated for a snapshot volume: SELECT id FROM region WHERE volume_id IN (SELECT volume_id FROM snapshot); A similar bug was found in the simulated Crucible agent. This commit also reverts oxidecomputer#6728, enabling region snapshot replacement again - it was disabled due to a lack of read-only region reference counting, so it can be enabled once again.

leftwo

I may have some more questions after I finish going through nexus/tests/integration_tests/volume_management.rs but I thought I should give you what I have so far.

leftwo · 2024-10-09T23:00:59Z

nexus/db-model/src/volume_resource_usage.rs

+/// this column, and more over a simple integer works for reference counting but
+/// does not tell you _what_ volume that use is from. This can be determined
+/// (see omdb's validate volume references command) but it's information that is
+/// tossed out, as Nexus knows what volumes use what resources! Instead, record


The wording of the comment here, I'm having trouble tracking what the "Instead" is referencing.

We are recording what read-only resources a volume uses instead of doing what?

leftwo · 2024-10-09T23:15:34Z

nexus/db-model/src/volume_resource_usage.rs

+ VolumeResourceUsage::ReadOnlyRegion {
+ region_id: record
+ .region_id
+ .expect("valid read-only region usage record"),


For this and the .expects below, this message is printed when we panic, right? If so, should it be saying that we did not find a valid region usage record?

Conservatively, this should maybe be a TryFrom implementation. As it exists today, someone could modify a column in the database and cause Nexus to panic with these .expects

leftwo · 2024-10-09T23:43:54Z

nexus/db-queries/src/db/datastore/volume.rs

+ conn: &async_bb8_diesel::Connection<DbConnection>,
+ err: OptionalError<VolumeCreationError>,
+ volume: Volume,
+ crucible_targets: CrucibleTargets,


What are the contents of CrucibleTargets? I know it's a String, but what is it really?
Ah, okay, it looks like it's just a Vec of the String of the targets for only the read only parents. It's not the best name for that structure, but I see it existed before this PR so I can't complain too much about it here.

leftwo · 2024-10-10T17:47:09Z

nexus/db-queries/src/db/datastore/volume.rs

+ }));
+ }
+
+ // If the resource was hard-deleted, or in the off chance that the


Is this choice expected often? Do we want to log a message if we don't expect to see it, but do?

leftwo · 2024-10-10T17:49:15Z

nexus/db-queries/src/db/datastore/volume.rs

+ .optional()
+ }
+
+ async fn read_only_target_to_volume_resource_usage(


This is a 1-1 situation, where there should only ever be one VolumeResourceUsage for a given read only target?

leftwo · 2024-10-10T21:23:08Z

nexus/db-queries/src/db/datastore/volume.rs

+ for read_only_target in crucible_targets.read_only_targets {
+ let sub_err = OptionalError::new();
+
+ let maybe_usage = Self::read_only_target_to_volume_resource_usage(


Nit: This line and a few others here seem to have escaped cargo fmt as they are super long.

leftwo · 2024-10-10T21:29:55Z

nexus/db-queries/src/db/datastore/volume.rs

@@ -2409,24 +2956,188 @@ impl DataStore {
 read_only_parent: None,
 };

- let volume_data = serde_json::to_string(&vcr)
+ let volume_data = serde_json::to_string(&vcr)


Man, there is a lot going on inside this transaction :)

leftwo · 2024-10-10T22:29:37Z

nexus/db-queries/src/db/datastore/volume.rs

+ )
+ })
+ })?
+ // XXX be smart enough to .filter the above query


make this a TODO, you can get smart enough!

leftwo · 2024-10-10T22:35:51Z

nexus/db-queries/src/db/datastore/volume.rs

+ while let Some(vcr_part) = parts.pop_front() {
+ match vcr_part {
+ VolumeConstructionRequest::Volume { sub_volumes, .. } => {
+ for sub_volume in sub_volumes {


Nice, this should work for multiple sub-volumes.

leftwo · 2024-10-10T22:54:53Z

nexus/src/app/sagas/disk_create.rs

+ .transaction_retry_wrapper("no_volume_resource_usage_records_exist")
+ .transaction(&conn, |conn| async move {
+ conn.batch_execute_async(
+ nexus_test_utils::db::ALLOW_FULL_TABLE_SCAN_SQL,


Don't let dap see this.

smklein · 2024-10-14T16:34:47Z

schema/crdb/dbinit.sql

+ 'region_snapshot'
+);
+
+CREATE TABLE IF NOT EXISTS omicron.public.volume_resource_usage (


I know it exists elsewhere in this PR, but I think this table would benefit from some text explaining what it is.

smklein · 2024-10-14T16:44:58Z

nexus/db-model/src/volume_resource_usage.rs

+ pub region_id: Option<Uuid>,
+
+ pub region_snapshot_dataset_id: Option<Uuid>,
+ pub region_snapshot_region_id: Option<Uuid>,
+ pub region_snapshot_snapshot_id: Option<Uuid>,


I had no idea reading the DB schema that these were two groups of columns, and "either one or the other is non-null".

If that's the intent -- and it seems to be, based on VolumeResourceUsage as an enum -- maybe we could add a CHECK on the table validating this?

Something like:

CONSTRAINT exactly_one_usage_source CHECK ( ( (usage_type = 'readonlyregion') AND (region_id IS NOT NULL) AND (region_snapshot_dataset_id IS NULL AND region_snaphshot_region_id IS NULL AND region_snapshot_snapshot_id IS NULL) ) OR ( (usage_type = 'regionsnapshot') AND (region_id NOT NULL) AND (region_snapshot_dataset_id IS NOT NULL AND region_snaphshot_region_id IS NOT NULL AND region_snapshot_snapshot_id IS NOT NULL) ) )

smklein · 2024-10-14T16:53:45Z

nexus/db-model/src/volume_resource_usage.rs

+ VolumeResourceUsage::ReadOnlyRegion {
+ region_id: record
+ .region_id
+ .expect("valid read-only region usage record"),


Conservatively, this should maybe be a TryFrom implementation. As it exists today, someone could modify a column in the database and cause Nexus to panic with these .expects

smklein · 2024-10-14T16:54:41Z

nexus/db-queries/src/db/datastore/region.rs

@@ -264,7 +264,7 @@ impl DataStore {
 block_size,
 blocks_per_extent,
 extent_count,
- read_only: false,
+ read_only: maybe_snapshot_id.is_some(),


Was this a bug before this PR?

smklein · 2024-10-14T16:55:29Z

nexus/db-queries/src/db/datastore/volume.rs

- enum VolumeCreationError {
- #[error("Error from Volume creation: {0}")]
- Public(Error),
+ let maybe_volume: Option<Volume> = dsl::volume


Volume has a time_deleted column -- do we not care about that here?

smklein · 2024-10-14T17:55:52Z

nexus/db-queries/src/db/datastore/volume.rs

+ // This function may be called with a replacement volume
+ // that is completely blank, to be filled in later by this
+ // function. `volume_create` will have been called but will
+ // not have added any volume resource usage records, because
+ // it was blank!


I'm confused about this comment, in this location - namely, I'm not sure who is calling volume_create in this situation where we're saying "volume_create will have been called". Is this something we're doing within the body of this function, and I'm not seeing it? Or is this something someone else could concurrently be doing?

smklein · 2024-10-14T17:56:30Z

nexus/db-queries/src/db/datastore/volume.rs

+ // not have added any volume resource usage records, because
+ // it was blank!
+ //
+ // The indention leaving this transaction is that the


Suggested change

// The indention leaving this transaction is that the

// The intention leaving this transaction is that the

smklein · 2024-10-14T17:58:05Z

nexus/db-queries/src/db/datastore/volume.rs

+ // We don't support a pure Region VCR at the volume
+ // level in the database, so this choice should
+ // never be encountered.
+ panic!("Region not supported as a top level volume");


This seems like a footgun to me, to have a pub fn with an undocumented panic?

smklein · 2024-10-14T18:02:51Z

schema/crdb/crucible-ref-count-records/up08.sql

+ OVERLAY(
+ OVERLAY(
+ MD5(volume.id::TEXT || dataset_id::TEXT || region_id::TEXT || snapshot_id::TEXT || snapshot_addr || volume_references::TEXT)
+ PLACING '4' from 13


What is this doing? Why '4'? Why from 13?

smklein · 2024-10-14T18:03:07Z

schema/crdb/crucible-ref-count-records/up08.sql

+ MD5(volume.id::TEXT || dataset_id::TEXT || region_id::TEXT || snapshot_id::TEXT || snapshot_addr || volume_references::TEXT)
+ PLACING '4' from 13
+ )
+ PLACING TO_HEX(volume_references) from 17


Why from 17?

jmpesp requested review from smklein and leftwo October 9, 2024 00:29

jmpesp added 2 commits October 9, 2024 00:44

Merge branch 'main' into read_only_regions_need_ref_counting_too

66c6797

full table scan required during clean slate check

410c79d

jmpesp mentioned this pull request Oct 9, 2024

Pantry crashed when processing a region with mismatched information #6353

Open

leftwo reviewed Oct 11, 2024

View reviewed changes

smklein reviewed Oct 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement record based Crucible reference counting #6805

Implement record based Crucible reference counting #6805

jmpesp commented Oct 9, 2024

leftwo left a comment •

edited

Loading

leftwo Oct 9, 2024

leftwo Oct 9, 2024

smklein Oct 14, 2024

leftwo Oct 9, 2024

leftwo Oct 10, 2024

leftwo Oct 10, 2024

leftwo Oct 10, 2024

leftwo Oct 10, 2024

leftwo Oct 10, 2024

leftwo Oct 10, 2024

leftwo Oct 10, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

smklein Oct 14, 2024

	// The indention leaving this transaction is that the
	// The intention leaving this transaction is that the

Implement record based Crucible reference counting #6805

Are you sure you want to change the base?

Implement record based Crucible reference counting #6805

Conversation

jmpesp commented Oct 9, 2024

leftwo left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leftwo left a comment •

edited

Loading