ref(config): Introduce multi-topic config, deprecate old fields

[untitaker]

ref STREAM-1042

We want to eventually support consuming multiple topics from a single taskbroker replica; https://app.notion.com/p/sentry/Multi-topic-management-for-taskbroker-3728b10e4b5d80adba06e127c5fac612 -- this is mainly for raw mode, so that we can consume multiple raw topics.

This is the first part that migrates the config to the new format. It is fully backwards compatible and old fields will print deprecation warnings. Multi-topic support is not actually implemented and defining more than one topic will result in a validation error.

Fields are migrated in Config::validate(), and it's renamed to normalize_and_validate. consumable_topic() has been introduced to make migration of the application less miserable, but when the actual support for multi-topic is implemented, that method has to go away.

Config::default() no longer produces a valid config. Instead, one has to run Config::default() (what figment uses), then always call normalize_and_validate afterwards to get a config that the rest of the app code can actually deal with.

What has been migrated is:

• retry topic config
• DLQ settings (kafka_deadletter_cluster is now just... a cluster in kafka_clusters)
• main consumer topic

What has not been migrated:

• kafka_consume_retry_topic has been removed entirely because the migration is too complicated. It is not currently used and I think until we have proper multi-topic support, we can work without it.
• Kafka session timeout and commit interval remain global options since there is currently no need to configure them per-topic (or... per-cluster?)
• the global settings auto offset reset, session timeout, and commit interval can now also be configured at a per-topic level, however the old fields are not deprecated. they serve as defaults instead.

Other changes:

While grinding through bugbot comments, I had to add a ton of extra validation just to deal with the fact that we're sharing a producer across many topics. The old config was more restrictive in what kind of values it allowed to set (for example, retry topic didn't have a dedicated retry cluster setting), and the new format structurally allows for too many possibilities that are not actually handled in code.

I think we should follow up and actually create separate producers for DLQ vs retry topic vs demoted namespaces, then this unnecessary restriction also goes away. It's also a footgun anyway since the k8s deployment templates don't validate against this either.

[linear-code]

STREAM-1042

STREAM-1085

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit fb61fe7. Configure here.}

[evanh]

Why not have normalize_and_validate be called in the default() function directly? Why make it separate?

[untitaker]

Why not have normalize_and_validate be called in the default() function directly? Why make it separate?

because the flow is like this:

1. default is created
2. figment is overwriting based on envvars
3. we validate (this kind of happens implicitly when we use the config, we have various assertions around the codebase)

that's how it works in main. with this PR this doesn't really change:

1. default
2. figment
3. normalize and validate

if we normalize before figment, or validate before it, we'll just not validate or normalize any user input.

same in tests: we get a default config, overwrite whatever is necessary, then validate/normalize.