Unable to upgrade from 3.1.0 to 3.2.1.0

[IvanTanJunWen]

Hi Team,

My existing Kong installation is done using the helm chart kong/kong v2.16.5, along with the docker-kong-oidc:3.1.1 image.

I tried to do a helm upgrade with the new 3.2.1 image and I found out that the base image for docker-kong-oidc has been upgraded from kong/kong:3.1.1-alpine to kong/kong-gateway:3.2.1.0-alpine. This has led to my existing Kong deployment to fail. I tried to connect to the admin endpoint and I get the following error:

Error: reading Kong version: making HTTP request: Get "https://kong-kong-admin.<namespace>:8444/": connection error: COMPRESSION_ERROR

Any reason behind this change of base image from kong/kong to kong/kong-gateway?

Thanks!

[cr1cr1]

Sorry about that. Yes, it was an unexpected (and I hope temporary) change because of the CVE-2023-0286 that some don't deem that bad and the official docker image did not yet go to 3.2.x

As soon as it's there will update it back.

[cr1cr1]

But your error should have nothing to do with the change of image, I expect you will get the same with the "official" docker one.

[IvanTanJunWen]

Understood, thanks for the quick update on this issue.

[IvanTanJunWen]

Just to update that kong/kong:3.2.2-alpine is available now, although it's not shown on the Dockerhub page.

[hanlaur]

Note the two different repositories in in Dockerhub, labeled as follows:

• kong: Official images (part of Docker library)
• kong/kong: Nightly images

I have not investigated to say how near these images are of each other in terms of actual code (could be same, I do not know), but definitely the images themselves are not the same. At least packaging is different and some environment variables are set differently. Example from a past version:

$ docker run --entrypoint="" --rm -it kong/kong:3.1.1 env|grep KONG_
KONG_VERSION=2.8.1
$ docker run --entrypoint="" --rm -it kong:3.1.1 env|grep KONG_
KONG_VERSION=3.1.1

$ docker run --entrypoint="" --rm -it kong/kong:3.1.1 kong version         
3.1.1
$ docker run --entrypoint="" --rm -it kong:3.1.1 kong version
3.1.1

$ docker images|grep 3.1.1
kong                                             3.1.1          b76469f8b580   5 weeks ago     164MB
kong                                             3.1.1-alpine   b76469f8b580   5 weeks ago     164MB
kong/kong                                        3.1.1          43f44aed3170   3 months ago    169MB
kong/kong                                        3.1.1-alpine   43f44aed3170   3 months ago    169MB

The official library image for 3.2.2 appears to be in PR review phase at the moment.

[cristichiru]

Bumped kong to 3.2.2 and also pushed to my personal repo as free org repos on docker hub will be gone starting with 14th of April.

If you own a legacy Free Team organization, access to paid features — including private repositories — will be suspended on April 14, 2023 (11:59 pm UTC). Upgrade your subscription before April 14, 2023 to continue accessing your organization.

[IvanTanJunWen]

Thanks for following up on this closely. Would like to highlight that I'm facing a different error, where Kong throws a HTTP 500 error after authenticating with Keycloak:

[lua] openidc.lua:1475: authenticate(): request to the redirect_uri path but there's no session state found, client: <redacted>, server: kong, request: "GET /cb?state=<redacted>&session_state=<redacted>&code=<redacted> HTTP/2.0"

[cristichiru]

Yep, that is an ugly (e.g. not immediately visible) error. It is because you need to set up x_session_secret or env KONG_X_SESSION_SECRET to a base64 encoded value. See README.md#notes

docker-kong-oidc/Dockerfile

Line 41 in 730d5b2

set_decode_base64 \$session_secret \${{X_SESSION_SECRET}};\n" "$TPL" \

[IvanTanJunWen]

We did provide a x_session_secret env variable via values.yaml when performing a helm installation:

env:
    X_SESSION_SECRET:
        valueFrom:
            secretKeyRef:
                name: <redacted>
                key: <redacted>

Could the bump in version for lua-resty-session from 3.10 to 4.0.2 be a factor in this scenario? I read that the session library was basically rewritten.

[cristichiru]

If you refer to the readme or see my comment above, you have to prefix env vars with KONG_, so KONG_X_SESSION_SECRET

[IvanTanJunWen]

Appreciate the prompt response.
Yes I checked that the KONG_X_SESSION_SECRET env variable exists even though I used X_SESSION_SECRET in values.yaml. Perhaps let me try using KONG_X_SESSION_SECRET in values.yaml instead. Thanks for your help.

[hanlaur]

Bumped kong to 3.2.2 and also pushed to my personal repo as free org repos on docker hub will be gone starting with 14th of April.

If you own a legacy Free Team organization, access to paid features — including private repositories — will be suspended on April 14, 2023 (11:59 pm UTC). Upgrade your subscription before April 14, 2023 to continue accessing your organization.

The official release image (https://hub.docker.com/_/kong, kong:3.2.2) is available now.

[cr1cr1]

Already updated :)

[cristichiru]

@IvanTanJunWen you were right, was because of kong using lua-resty-session 4: zmartzone/lua-resty-openidc#463.

Updated plugins (4f6e449), tested the oidc plugin and seems to work as expected now.