[SS-69] user-facing docs for Iceberg sink, now with GCP support by ublubu · Pull Request #36773 · MaterializeInc/materialize

ublubu · 2026-05-28T16:07:58Z

Adding docs for:

GCP connection
GCP Lakehouse/BigLake Iceberg Catalog Connection (BigLake is still the name of the API and everything in the GCP console, but it lives under a Lakehouse umbrella now.)

Small changes to docs for Iceberg sink.

stacked on #36695

kay-kim · 2026-05-29T15:08:06Z


        The name of the user to connect as.

+- name: "syntax-gcp"


Q: How does this affect https://preview.materialize.com/materialize/36773/sql/create-connection/#s3-compatible-object-storage where we mention Google Cloud Storage ... ?

Interesting! That feature is not connected to this one, and they use different auth mechanisms. Maybe one day we'll make a GCS-native "copy from" operation, and that would use the new GCP connection primitive.

Ah .. thanks ... I think we'll want to then be a little more explicit in the connection page. Will add my comments to that page then.

kay-kim · 2026-06-08T19:47:26Z


        The name of the user to connect as.

+- name: "syntax-gcp"


Ah .. thanks ... I think we'll want to then be a little more explicit in the connection page. Will add my comments to that page then.

kay-kim · 2026-06-08T19:51:16Z

 );
 ```

+### GCP


In the above S3 compatible object storage section:

"You can use an AWS connection to perform bulk exports and bulk imports ..." -> "You can use an AWS connection to perform bulk exports (`COPY TO`) and bulk imports (`COPY FROM`) ..."

We could also add a disambiguation sentence that tells them if creating a GCP Iceberg sink to see this section.

kay-kim · 2026-06-08T19:53:21Z

+
+A Google Cloud Platform (GCP) connection gives Materialize
+a [service account](https://docs.cloud.google.com/iam/docs/service-account-overview)
+in your GCP project. You can use a GCP connection to authenticate with


I would break this up and reorder so that the usage (i.e., creating an Iceberg catalog) is more prominent. This might make it easier to reword that service account thing as well.

kay-kim · 2026-06-08T21:37:38Z


+### GCP
+
+A Google Cloud Platform (GCP) connection gives Materialize


"gives MZ a service account" makes it seem like creating a connection creates a service account. it just uses the service account when authenticating, yes?

kay-kim · 2026-06-09T16:05:12Z

+    - name: "`<connection_name>`"
+      description: |
+        A name for the connection.
+    - name: "`<service_account_key>`"


Ah ... I should have been clearer in my comment.

Since it's a syntax for the create connection ... we don't need to explain the syntax of the service account key as part of this syntax. By incorporating ... I meant more SERVICE ACCOUNT KEY description referencing the secret that stores the service account key. and recommend the base64 encoding and using decode ... when creating the secret.

Also, did you mean to remove the example?

kay-kim · 2026-06-09T16:48:09Z

- name: "syntax-iceberg-catalog"
+- name: "syntax-gcp"
+  code: |
+    CREATE SECRET <secret_name> AS decode('<service_account_key>', 'base64');


This already assumes that you have <service_account_key> is base64 encoded.
probably can add a comment that <service_account_key> is base64-encoded. If you want to have an example, we can remove this CREATE SECRET ... decode from here.

kay-kim · 2026-06-09T16:55:33Z

+
+## Prerequisites
+
+Google Cloud [documents the Lakehouse/BigLake setup process here](https://docs.cloud.google.com/lakehouse/docs/lakehouse-iceberg-rest-catalog). The parts you'll need:


Start with the actual prerequisites, the bullet points.

kay-kim · 2026-06-09T16:56:45Z

+- A Google Cloud project with the BigLake API enabled.
+- A Google Cloud Storage bucket to serve as the Iceberg warehouse.
+- A Lakehouse runtime catalog backed by your warehouse bucket.
+  - _NOTE: Materialize uses a service account key, not catalog-vended credentials, to write Iceberg data files._


We generally use {{< note >}} {{< /note >}} blocks for notes.

kay-kim · 2026-06-09T16:59:09Z

+    - `serviceusage.serviceUsageConsumer` (Service Usage Consumer)
+3. Grant the service account this role on your **Iceberg warehouse bucket**:
+    - `storage.objectUser` (Storage Object User)
+4. [Create a service account key.](https://docs.cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-gcloud)


The google docs tell people to use JSON and not p12 ... not sure if we want to just mention create a service account key (JSON format) or something.

Do we need to base64 encode it here? or is it already base64 encoded when you get it from google? That is, the example in step 2 has a comment showing how to base 64 encode the key.json ... but shouldn't we do it explicitly as a step here if people need to do it?

kay-kim · 2026-06-09T17:04:06Z

+
+### Limitations
+
+{{% include-headless "/headless/iceberg-sinks/limitations-list" %}}


This talks about aws

Can we add the compaction here

ublubu · 2026-06-11T14:57:08Z

Uh, this got automatically closed when I merged the previous PR. I will reopen after getting it on the correct base branch again.

ublubu · 2026-06-11T16:22:02Z

Because I accidentally pushed kynan/iceberg-gcp-doc to my fork (instead of to MaterializeInc/materialize), this PR seems to have permanently closed

instead of updating to point at main when kynan/iceberg-gcp got merged and deleted.

def- · 2026-06-13T06:18:06Z

You can just open a new PR from the same branch.

reopening #36773 ---- Adding docs for: - GCP connection - GCP Lakehouse/BigLake Iceberg Catalog Connection _(BigLake is still the name of the API and everything in the GCP console, but it lives under a Lakehouse umbrella now.)_ Small changes to docs for Iceberg sink.

Claude: use updated iceberg-rust fork

3d8b898

ublubu force-pushed the kynan/iceberg-gcp-doc branch 3 times, most recently from 81cac44 to f582bfa Compare May 28, 2026 19:49

ublubu marked this pull request as ready for review May 28, 2026 20:17

ublubu requested a review from a team as a code owner May 28, 2026 20:17

ublubu force-pushed the kynan/iceberg-gcp-doc branch from 1f326ad to 07ae4e1 Compare May 28, 2026 20:58

kay-kim reviewed May 29, 2026

View reviewed changes

ublubu force-pushed the kynan/iceberg-gcp-doc branch from 07ae4e1 to aa79174 Compare June 4, 2026 15:32

ublubu requested review from a team as code owners June 4, 2026 15:32

ublubu force-pushed the kynan/iceberg-gcp branch 3 times, most recently from acc2db7 to 26b818e Compare June 4, 2026 18:28

ublubu force-pushed the kynan/iceberg-gcp-doc branch from aa79174 to 7912fa8 Compare June 4, 2026 18:29

ublubu force-pushed the kynan/iceberg-gcp branch from 26b818e to a3408b2 Compare June 4, 2026 18:35

ublubu force-pushed the kynan/iceberg-gcp-doc branch from 7912fa8 to 0479930 Compare June 4, 2026 18:35

ublubu force-pushed the kynan/iceberg-gcp branch from a3408b2 to 7c400e7 Compare June 4, 2026 21:14

ublubu force-pushed the kynan/iceberg-gcp-doc branch 2 times, most recently from c8b6ed1 to 4f0aa58 Compare June 5, 2026 16:48

ublubu force-pushed the kynan/iceberg-gcp branch from 3bcf327 to 8ec6f5b Compare June 8, 2026 16:30

ublubu force-pushed the kynan/iceberg-gcp-doc branch from a5b9e2b to 1ab3fd1 Compare June 8, 2026 16:30

ublubu force-pushed the kynan/iceberg-gcp branch from 8ec6f5b to e89154b Compare June 8, 2026 16:45

ublubu force-pushed the kynan/iceberg-gcp-doc branch 2 times, most recently from 41dfc25 to 7b1feb5 Compare June 8, 2026 18:48

ublubu force-pushed the kynan/iceberg-gcp branch from e89154b to fdbb619 Compare June 8, 2026 18:48

ublubu added 3 commits June 8, 2026 14:51

move iceberg-rust to mz_v0.9.0

a60c20c

error if header value is not visible ascii

9cddd2a

Body.as_bytes() = None => streaming body, which we can't sign

a318cff

ublubu added 13 commits June 8, 2026 14:51

Dennis token exfiltration repro

45ed9ad

check Iceberg catalog URL for GCP connections at plan time

96d20a7

remove Iceberg Sink's aws_connection from docs

9fae43f

Claude: docs for GCP Iceberg Sink

b444177

Kynan: rewrites for serve-results/

7d255f5

Kynan: rewrites for sql/

2b347c3

Kynan: rewrites for create_connection.yml

7ce7f96

move base64 comment

0c25012

move service account key instructions to examples file

40b29be

delete SCOPE from examples file

2e4939a

use the examples yml for the catalog connection instructions

10e3350

split the catalog connection syntax into AWS and GCP tabs

baf3c86

fix broken syntax-upsert|append embeds

764b7db

ublubu force-pushed the kynan/iceberg-gcp branch from fdbb619 to 96d20a7 Compare June 8, 2026 19:28

ublubu force-pushed the kynan/iceberg-gcp-doc branch from 7b1feb5 to 7fc632d Compare June 8, 2026 19:28

ublubu added 2 commits June 9, 2026 11:27

Claude: split AWS and GCP sink tutorial into separate pages

5716e65

add blurb about removing the sink's AWS CONNECTION

a9a6ed9

ublubu force-pushed the kynan/iceberg-gcp-doc branch from 7fc632d to a9a6ed9 Compare June 9, 2026 15:27

kay-kim reviewed Jun 9, 2026

View reviewed changes

ublubu force-pushed the kynan/iceberg-gcp branch 4 times, most recently from 7428408 to 19a77b5 Compare June 10, 2026 19:26

ublubu deleted the branch MaterializeInc:kynan/iceberg-gcp June 10, 2026 19:48

ublubu closed this Jun 10, 2026

ublubu mentioned this pull request Jun 16, 2026

[SS-69] user-facing docs for Iceberg sink, now with GCP support #37083

Merged

               );
               ```
+              ### GCP


		### GCP

		A Google Cloud Platform (GCP) connection gives Materialize


		## Prerequisites

		Google Cloud [documents the Lakehouse/BigLake setup process here](https://docs.cloud.google.com/lakehouse/docs/lakehouse-iceberg-rest-catalog). The parts you'll need:


		### Limitations

		{{% include-headless "/headless/iceberg-sinks/limitations-list" %}}

Conversation

ublubu commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ublubu commented Jun 11, 2026

Uh oh!

ublubu commented Jun 11, 2026

Uh oh!

def- commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ublubu commented May 28, 2026 •

edited

Loading