Skip to content

Conversation

@HeartLinked
Copy link
Contributor

@HeartLinked HeartLinked commented Nov 6, 2025

This PR introduces the foundational scaffolding for the Iceberg REST Catalog client implementation in C++. It establishes the core infrastructure for communicating with Iceberg REST Catalog servers following the Iceberg REST Catalog Specification.

Key Components Added

1. HTTP Client Infrastructure (http_client.h/cc)

  • Implemented HttpClient class wrapping the CPR library for HTTP operations.
  • Supports GET, POST, POST (form-urlencoded), HEAD, and DELETE methods.
  • Thread-safe session management with mutex protection.
  • HttpResponse wrapper to abstract underlying HTTP library implementation.

2. Configuration Management (catalog_properties.h/cc)

  • RestCatalogProperties class for REST catalog configuration.

3. Resource Path Construction (resource_paths.h/cc)

  • ResourcePaths class for building REST API endpoint URLs.

4. Error Handling Framework (error_handlers.h/cc)

  • Hierarchical error handler design following the REST specification.
  • HTTP status code to ErrorKind mapping.
  • Also, extended result.h with new rest related error kinds.

5. REST Utilities (rest_util.h/cc)

  • URL encoding/decoding (RFC 3986 compliant via libcurl).
  • Namespace encoding/decoding with ASCII Unit Separator (0x1F).
  • Configuration merging with proper precedence (server overrides > client config > server defaults).
  • String utilities (e.g., TrimTrailingSlash).

7. RestCatalog Implementation (rest_catalog.h/cc)

  • RestCatalog class implementing the Catalog interface.
  • Initialization workflow:
    1. Validates client configuration.
    2. Fetches server configuration from /v1/config.
    3. Merges server and client properties.
    4. Updates resource paths based on final configuration.

Testing

  • rest_util_test.cc: Comprehensive tests for URL encoding/decoding, namespace encoding, config merging.
  • rest_catalog_test.cc: These currently introduced tests are merely temporary integration tests that require a local REST server, such as the apache/iceberg-rest-fixture Docker image While this enables local testing, it is incompatible with the GitHub CI process, so they have been marked as DISABLED_. In the future, we aim to follow the example of iceberg-rust by designing comprehensive integration tests to verify the REST catalog client's behavior and integrating them into our GitHub CI pipeline. This work is scheduled for later completion; please refer to issue Implement REST Catalog Integration Tests with Docker in CI Pipeline #333.

@HeartLinked HeartLinked force-pushed the catalog_first_version_1 branch 3 times, most recently from 64e53e3 to 674c8ab Compare November 19, 2025 10:05
Copy link
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just did an initial pass, the main concerns are:

  1. URL encoding & decoding may not match the Java impl.
  2. Namespace encoding is using the deprecated 0x1F.
  3. ErrorResponse and ErrorModel are duplicate. We can improve it by using a single ErrorResponse.

Missing test cases of:

  1. Catalog properties to extract header map.
  2. All functions in the rest_util.h, check TestRestUtil.java.
  3. Resource paths, check TestResourcePaths.java.

Since this is a large PR and we don't have good test fixture for it yet, I'm fine to add them later by adding TODOs.

constexpr std::string_view kHeaderAccept = "Accept";
constexpr std::string_view kHeaderXClientVersion = "X-Client-Version";
constexpr std::string_view kHeaderUserAgent = "User-Agent";
inline const std::string kHeaderContentType = "Content-Type";

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you apply this change? I would prefer constexpr std::string_view instead of const std::string.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reason is that we still need to wrap it with std::string every time to create http headers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So @HeartLinked may need to remove std::string(...) wrapper in some files, such as src/iceberg/catalog/rest/config.cc.

};

TEST_F(RestCatalogTest, MakeCatalogSuccess) {
TEST_F(RestCatalogTest, DISABLED_MakeCatalogSuccess) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you disable the test? Leave a comment with a reason.

Copy link
Contributor Author

@HeartLinked HeartLinked Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a temporary integration test that requires a local REST server, such as the Docker image apache/iceberg-rest-fixture provided by Apache Iceberg. Currently, this method allows testing locally, but it obviously cannot pass the Github CI process, so it is a temporary expedient. In this first version PR, we clearly cannot comprehensively complete such a large amount of work. You can take a look on this issue: #333 :)

@HeartLinked HeartLinked force-pushed the catalog_first_version_1 branch from 6fd456f to 1d924f9 Compare November 25, 2025 02:57
@HeartLinked HeartLinked force-pushed the catalog_first_version_1 branch from 17c0a53 to 1b17be0 Compare November 25, 2025 03:44
constexpr std::string_view kHeaderAccept = "Accept";
constexpr std::string_view kHeaderXClientVersion = "X-Client-Version";
constexpr std::string_view kHeaderUserAgent = "User-Agent";
inline const std::string kHeaderContentType = "Content-Type";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reason is that we still need to wrap it with std::string every time to create http headers.

@wgtmac wgtmac force-pushed the catalog_first_version_1 branch 2 times, most recently from ce9a070 to e71588a Compare November 26, 2025 06:21
@wgtmac wgtmac force-pushed the catalog_first_version_1 branch from e71588a to 7267af5 Compare November 26, 2025 06:31
@wgtmac wgtmac changed the title feat: first version of rest catalog feat: scaffolding work of rest catalog client Nov 26, 2025
@wgtmac wgtmac force-pushed the catalog_first_version_1 branch from ec69dfa to 4efd9d3 Compare November 26, 2025 07:22
@HeartLinked HeartLinked force-pushed the catalog_first_version_1 branch from 4efd9d3 to 5f79c97 Compare November 26, 2025 09:08
@wgtmac
Copy link
Member

wgtmac commented Nov 27, 2025

Thanks all for the review! Great job @HeartLinked. Let me merge this to move forward.

@wgtmac wgtmac merged commit 4a71abe into apache:main Nov 27, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants