ref(taskbroker): Migrate to new kafka_clusters/kafka_topics config format#4363
Merged
untitaker merged 4 commits intoJun 3, 2026
Conversation
…rmat Move the taskbroker service (and the external-kafka patch variant) off the deprecated kafka_cluster/kafka_deadletter_cluster fields onto the new kafka_clusters/kafka_topics format. Single consumed topic (taskworker) plus produce-only deadletter; retries fall back to the consumed topic. The external-kafka patch now layers per-cluster auth onto the shared default cluster. ref STREAM-1095 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move topic declarations into a mounted ./taskbroker/config.yml (like symbolicator) instead of TASKBROKER_KAFKA_TOPICS__* env vars, so topic names can keep their hyphen (taskworker-dlq) — env-var config keys can't contain hyphens. The cluster address/auth stay as env vars (no hyphens needed, and the external-kafka patch overrides them). Regenerated the external-kafka patch accordingly; verified it applies with patch -p0. ref STREAM-1095 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The regenerated external-kafka patch dropped the tab+timestamp from its --- / +++ header lines, which made GNU patch (Linux) resolve the target to the wrong filename. Restore the original header format so 'patch -p0' targets docker-compose.yml. Also trim the config.yml comment to a link to the docs PR. ref STREAM-1095 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lf-hosted-to-new-config-format
evanh
approved these changes
Jun 3, 2026
untitaker
deleted the
markusunterwaditzer/stream-1095-migrate-self-hosted-to-new-config-format
branch
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See getsentry/taskbroker#667 for details.
The short version of this is, we are making some changes to kafka configuration that affect every user. The change is already out in the latest version of self-hosted and users should already get deprecation warnings as per getsentry/taskbroker#663
The config change is desired so we can configure multiple kafka topics and clusters for a single taskbroker container. All of this can be configured in envvars, too, but at this point it'll be more readable to move to yaml (which taskbroker supported all along)
ALso,
taskworker-dlqcan't be defined as a topic via envvars (hyphens aren't permitted there), which is the main reason this moves the topic config into a mounted yaml file.