Skip to content

chore: upgrade Flink to 2.2.0 and CP to 8.2.0 in Docker deployment #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Dave Troiano (davetroiano) wants to merge 1 commit into
base: master
Choose a base branch
from
Open

chore: upgrade Flink to 2.2.0 and CP to 8.2.0 in Docker deployment #136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 40 additions & 36 deletions docker/docker-compose-flinksql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,61 +1,50 @@
services:
broker:
image: confluentinc/cp-kafka:7.4.1
image: confluentinc/cp-kafka:8.2.0
hostname: broker
container_name: broker
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_NODE_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:9092'
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_NODE_ID: 1
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
KAFKA_LISTENERS: PLAINTEXT://broker:9092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:29092
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LOG_DIRS: /tmp/kraft-combined-logs
CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk
KAFKA_JMX_PORT: 9101
KAFKA_JMX_HOSTNAME: 'localhost'
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
KAFKA_LISTENERS: 'PLAINTEXT://broker:9092,CONTROLLER://broker:29093'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
# Replace CLUSTER_ID with a unique base64 UUID using "bin/kafka-storage.sh random-uuid"
# See https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#kafka-storage-sh
CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'

schema-registry:
image: confluentinc/cp-schema-registry:7.3.0
image: confluentinc/cp-schema-registry:8.2.0
hostname: schema-registry
container_name: schema-registry
depends_on:
- broker
ports:
- 8081:8081
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: broker:9092
flink-sql-client:
image: cnfldemos/flink-sql-client-kafka:1.19.1-scala_2.12-java17
hostname: flink-sql-client
container_name: flink-sql-client
depends_on:
- flink-jobmanager
environment:
FLINK_JOBMANAGER_HOST: flink-jobmanager
volumes:
- ./settings/:/settings
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: 'broker:9092'
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

flink-jobmanager:
image: cnfldemos/flink-kafka:1.19.1-scala_2.12-java17
image: cnfldemos/flink-kafka:2.2.0-scala_2.12-java17
hostname: flink-jobmanager
container_name: flink-jobmanager
ports:
- 9081:9081
- 8081:8081
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: flink-jobmanager
rest.bind-port: 9081

flink-taskmanager:
image: cnfldemos/flink-kafka:1.19.1-scala_2.12-java17
image: cnfldemos/flink-kafka:2.2.0-scala_2.12-java17
hostname: flink-taskmanager
container_name: flink-taskmanager
depends_on:
Expand All @@ -67,3 +56,18 @@ services:
FLINK_PROPERTIES=
jobmanager.rpc.address: flink-jobmanager
taskmanager.numberOfTaskSlots: 10

flink-sql-client:
image: cnfldemos/flink-sql-client-kafka:2.2.0-scala_2.12-java17
hostname: flink-sql-client
container_name: flink-sql-client
depends_on:
- flink-jobmanager
- flink-taskmanager
- broker
- schema-registry
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: flink-jobmanager
taskmanager.numberOfTaskSlots: 10
Comment on lines +70 to +73
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FLINK_PROPERTIES here duplicates settings that appear to be baked into the image via /opt/flink/conf/config.yaml (e.g., jobmanager.rpc.address and taskmanager.numberOfTaskSlots). Keeping the same config in both places increases the chance they drift; consider relying on a single source of truth (either the config file or container-specific FLINK_PROPERTIES) or adding a comment explaining why overrides are needed.

Copilot uses AI. Check for mistakes.
11 changes: 5 additions & 6 deletions docker/flink/Dockerfile.flink-kafka
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
FROM flink:1.19.1-scala_2.12-java11
FROM flink:2.2.0-scala_2.12-java17

LABEL io.confluent.docker=true

# Download connector libraries
RUN wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/3.2.0-1.19/flink-sql-connector-kafka-3.2.0-1.19.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/1.19.1/flink-json-1.19.1.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/1.19.1/flink-sql-avro-confluent-registry-1.19.1.jar;
RUN wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/4.0.1-2.0/flink-sql-connector-kafka-4.0.1-2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar;
Comment on lines +6 to +7
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These images download connector JARs from Maven Central during the Docker build without any integrity verification (checksum/signature). This is a supply-chain risk and can also lead to non-reproducible builds if artifacts are ever replaced. Consider pinning and verifying checksums (or vendoring the artifacts / using a build that resolves deps via Gradle/Maven and copies them into the image).

Suggested change
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar;
wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/4.0.1-2.0/flink-sql-connector-kafka-4.0.1-2.0.jar.sha512; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar.sha512; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar.sha512; \
cd /opt/flink/lib && \
sha512sum -c flink-sql-connector-kafka-4.0.1-2.0.jar.sha512 && \
sha512sum -c flink-json-2.2.0.jar.sha512 && \
sha512sum -c flink-sql-avro-confluent-registry-2.2.0.jar.sha512 && \
rm -f \
flink-sql-connector-kafka-4.0.1-2.0.jar.sha512 \
flink-json-2.2.0.jar.sha512 \
flink-sql-avro-confluent-registry-2.2.0.jar.sha512

Copilot uses AI. Check for mistakes.

# Copy configuration
COPY ./conf/* /opt/flink/conf/
COPY ./conf/* /opt/flink/conf/
18 changes: 4 additions & 14 deletions docker/flink/Dockerfile.flink-sql-client-kafka
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,13 @@
FROM flink:1.19.1-scala_2.12-java11
FROM flink:2.2.0-scala_2.12-java17

LABEL io.confluent.docker=true

# Create CLI lib folder
RUN mkdir -p /opt/sql-client/lib

# Download connector libraries
RUN wget -P /opt/sql-client/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/3.2.0-1.19/flink-sql-connector-kafka-3.2.0-1.19.jar; \
wget -P /opt/sql-client/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/1.19.1/flink-json-1.19.1.jar; \
wget -P /opt/sql-client/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/1.19.1/flink-sql-avro-confluent-registry-1.19.1.jar;

# Also copy to Flink lib so that, e.g., other catalog types can be used if desired
RUN cp /opt/sql-client/lib/* /opt/flink/lib/
RUN wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/4.0.1-2.0/flink-sql-connector-kafka-4.0.1-2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar;
Comment on lines +5 to +7
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These images download connector JARs from Maven Central during the Docker build without any integrity verification (checksum/signature). This is a supply-chain risk and can also lead to non-reproducible builds if artifacts are ever replaced. Consider pinning and verifying checksums (or vendoring the artifacts / using a build that resolves deps via Gradle/Maven and copies them into the image).

Suggested change
RUN wget -P /opt/flink/lib https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/4.0.1-2.0/flink-sql-connector-kafka-4.0.1-2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar;
RUN set -eux; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/4.0.1-2.0/flink-sql-connector-kafka-4.0.1-2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka/4.0.1-2.0/flink-sql-connector-kafka-4.0.1-2.0.jar.sha512; \
(cd /opt/flink/lib && sha512sum -c flink-sql-connector-kafka-4.0.1-2.0.jar.sha512); \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar.sha512; \
(cd /opt/flink/lib && sha512sum -c flink-json-2.2.0.jar.sha512); \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar; \
wget -P /opt/flink/lib/ https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/2.2.0/flink-sql-avro-confluent-registry-2.2.0.jar.sha512; \
(cd /opt/flink/lib && sha512sum -c flink-sql-avro-confluent-registry-2.2.0.jar.sha512); \
rm -f /opt/flink/lib/*.sha512

Copilot uses AI. Check for mistakes.

# Copy configuration
COPY ./conf/* /opt/flink/conf/

WORKDIR /opt/sql-client
ENV SQL_CLIENT_HOME=/opt/sql-client

COPY ./docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
2 changes: 1 addition & 1 deletion docker/flink/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FLINK_VERSION ?= 1.19.1-scala_2.12-java17
FLINK_VERSION ?= 2.2.0-scala_2.12-java17
BUILD_PLATFORM ?= linux/amd64
PUSH_PREFIX ?= cnfldemos

Expand Down
30 changes: 30 additions & 0 deletions docker/flink/conf/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
blob:
server:
port: '6124'
taskmanager:
memory:
process:
size: 1728m
bind-host: 0.0.0.0
numberOfTaskSlots: '10'
jobmanager:
execution:
failover-strategy: region
rpc:
address: flink-jobmanager
port: 6123
memory:
process:
size: 1600m
bind-host: 0.0.0.0
query:
server:
port: '6125'
parallelism:
default: 1
rest:
address: flink-jobmanager
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The REST endpoint is exposed on the host via 8081:8081 in docker-compose, but the Flink config here does not set a REST bind address. If Flink defaults to binding REST to localhost inside the container, the Web UI/REST API will not be reachable from outside the container. Consider explicitly setting the REST bind address (and bind port if needed) in this config (e.g., bind to 0.0.0.0).

Suggested change
address: flink-jobmanager
address: flink-jobmanager
bind-address: 0.0.0.0

Copilot uses AI. Check for mistakes.
env:
java:
opts:
all: --add-exports=java.base/sun.net.util=ALL-UNNAMED --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED
Loading