Add omdb command to forcefully quiesce db_metadata_nexus records

[smklein]

Builds on #9034

Adds a command to handle permanent Nexus failure during handoff.

Fixes #9008

[davepacheco]

It would be nice to do a confirmation and maybe an attempt to check whether Nexus is up, similar to what we do for dangerous saga operations:

omicron/dev-tools/omdb/src/bin/omdb/db/saga.rs

Lines 173 to 226 in 3d2f118

	// Add a confirmation prompt reminding the caller of the risks of this
	// injection
	let text = r#"
	WARNING: Injecting an error into a saga will (hopefully) cause it to be
	unwound, but if the actions into which errors are injected have taken effect,
	those effects will not be undone. This can result in corruption of control
	plane state, even if the Nexus assigned to this saga is not currently running.
	You should only do this if:
	- you've stopped Nexus and then verified that the currently-running nodes
	either have no side effects, have not made any changes to the system, or
	you've already undone them by hand
	- this is a development system whose state can be wiped
	"#;
	if should_print_color {
	println!("{}", text.red().bold());
	} else {
	println!("{text}");
	}
	// Before doing anything: find the current SEC for the saga, and ping it to
	// ensure that the Nexus is down.
	if !args.bypass_sec_check {
	let saga: Saga = {
	use nexus_db_schema::schema::saga::dsl;
	dsl::saga
	.filter(dsl::id.eq(args.saga_id))
	.first_async(&*conn)
	.await?
	};
	let status = get_saga_sec_status(omdb, opctx, &saga).await;
	status.display_message(should_print_color);
	status.into_result()?;
	} else {
	let text = "Skipping check of whether the Nexus assigned to this saga \
	is running. If this Nexus is running, the control plane state managed \
	by this saga may become corrupted!";
	if should_print_color {
	println!("{}", text.red().bold());
	} else {
	println!("{text}");
	}
	}
	// Before making any changes, ask for confirmation
	let mut prompt = ConfirmationPrompt::new();
	prompt.read_and_validate("y/N", "y")?;
	drop(prompt);

There are enough warning signs here, and I don't think this is likely a tool that people will stumble on accidentally, that I wouldn't call this a blocker.

[smklein]

It would be nice to do a confirmation and maybe an attempt to check whether Nexus is up, similar to what we do for dangerous saga operations:

omicron/dev-tools/omdb/src/bin/omdb/db/saga.rs

Lines 173 to 226 in 3d2f118

	// Add a confirmation prompt reminding the caller of the risks of this
	// injection
	let text = r#"
	WARNING: Injecting an error into a saga will (hopefully) cause it to be
	unwound, but if the actions into which errors are injected have taken effect,
	those effects will not be undone. This can result in corruption of control
	plane state, even if the Nexus assigned to this saga is not currently running.
	You should only do this if:
	- you've stopped Nexus and then verified that the currently-running nodes
	either have no side effects, have not made any changes to the system, or
	you've already undone them by hand
	- this is a development system whose state can be wiped
	"#;
	if should_print_color {
	println!("{}", text.red().bold());
	} else {
	println!("{text}");
	}
	// Before doing anything: find the current SEC for the saga, and ping it to
	// ensure that the Nexus is down.
	if !args.bypass_sec_check {
	let saga: Saga = {
	use nexus_db_schema::schema::saga::dsl;
	dsl::saga
	.filter(dsl::id.eq(args.saga_id))
	.first_async(&*conn)
	.await?
	};
	let status = get_saga_sec_status(omdb, opctx, &saga).await;
	status.display_message(should_print_color);
	status.into_result()?;
	} else {
	let text = "Skipping check of whether the Nexus assigned to this saga \
	is running. If this Nexus is running, the control plane state managed \
	by this saga may become corrupted!";
	if should_print_color {
	println!("{}", text.red().bold());
	} else {
	println!("{text}");
	}
	}
	// Before making any changes, ask for confirmation
	let mut prompt = ConfirmationPrompt::new();
	prompt.read_and_validate("y/N", "y")?;
	drop(prompt);

There are enough warning signs here, and I don't think this is likely a tool that people will stumble on accidentally, that I wouldn't call this a blocker.

I went ahead and incorporated a confirmation prompt, but this breaks automated tests, so I also added an optional skip_confirmation argument.