Skip to content

docs: add protobuf codegen guide + scenario test example#220

Open
keugenek wants to merge 5 commits intomainfrom
feat/proto-plugin
Open

docs: add protobuf codegen guide + scenario test example#220
keugenek wants to merge 5 commits intomainfrom
feat/proto-plugin

Conversation

@keugenek
Copy link
Copy Markdown
Contributor

@keugenek keugenek commented Mar 30, 2026

Summary

Per review feedback from @JCAlvar and @pkosiec: proto doesn't need a plugin — it's codegen + docs showing how to use generated types with existing plugins.

What changed

Removed: The proto plugin (packages/appkit/src/plugins/proto/) — no resources, no routes, no lifecycle. Not a plugin.

Added:

  • docs/docs/guides/protobuf.md — guide on using buf/ts-proto codegen with AppKit's files and lakebase plugins
  • examples/proto-catalog/ — scenario test demonstrating the pattern (Product Catalog app with Playwright tests)

The pattern

# 1. Define proto
echo 'message Customer { string id = 1; ... }' > proto/myapp/v1/models.proto

# 2. Generate types
npx buf generate proto/

# 3. Use with existing plugins (no proto plugin needed)
import { toJson, fromJson } from "@bufbuild/protobuf";
await app.files("data").upload(path, Buffer.from(JSON.stringify(toJson(schema, msg))));
const row = fromJson(schema, queryResult);

Scenario test (13/13 passing)

The examples/proto-catalog/ app demonstrates proto-style contracts without any proto plugin:

  • 5 public cases: view all, filter, health check, snake_case fields
  • 8 private cases: edge cases, 404, content-type, column headers

Test plan

  • Proto plugin fully removed
  • No @bufbuild/protobuf dependency in package.json
  • Plugin exports reverted to pre-proto state
  • Scenario test runs locally (13/13)
  • Docs reframed as codegen guide
  • pnpm build passes
  • pnpm test passes

New plugin that provides protobuf-based serialization so plugins,
routes, and jobs share a single schema definition. Pairs with the
Files plugin for Volume I/O and Lakebase plugin for database storage.

Plugin API:
- create(schema, init?) — typed message construction
- serialize / deserialize — binary (Uint8Array)
- toJSON / fromJSON — JSON with proto field names

Also includes:
- Proto definitions: appkit/v1/common.proto (Error, Metadata, Value)
  and appkit/v1/pipeline.proto (JobResult, DataBatch, DataRow)
- Generated TypeScript types in packages/shared/src/proto/
- buf configuration for codegen
- Documentation with usage examples at docs/plugins/proto.md
- Unit tests for plugin and serializer

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Product Catalog sample app demonstrating proto plugin usage:
- Proto-style JSON serialization (snake_case field names)
- Binary endpoint with application/x-protobuf content type
- Category filtering with typed product contracts
- Playwright tests parameterized by public/private JSON cases

Runnable locally (npx tsx app/server.ts + playwright test) or
against a deployed app (APP_URL=... playwright test).

5 public cases (developer verification):
- View all products, filter by category, health check,
  API field validation, snake_case convention check

8 private cases (evaluation):
- Single-category filter, in/out-of-stock status, product detail,
  404 handling, proto content type, status text, column headers

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Fixes strict mode violation where "Stock" matched both "Stock" and
"In Stock" columns in Playwright tests. Also applies linter formatting.

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
The proto plugin provides serialization — consumers bring their own
.proto files and generate their own types. Shipping base types
(common.proto, pipeline.proto) in AppKit couples the SDK to specific
domain vocabulary that belongs in consumer projects.

Removed:
- proto/ top-level folder (buf config + .proto source files)
- packages/shared/src/proto/ (generated TypeScript types)
- turbo.json proto:generate task
- shared index proto re-export
- test-results/ artifact

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Per review feedback (Jorge, Pawel): proto doesn't need a plugin —
just codegen (buf/ts-proto) + docs showing how to use generated
types with existing files and lakebase plugins.

- Remove proto plugin (no resources, no routes, no lifecycle needed)
- Keep scenario test (moved to examples/proto-catalog/)
- Reframe docs as codegen guide (docs/guides/protobuf.md)
- Revert plugin exports, package.json dep, pnpm-lock

The proto pattern is: buf generate → import types → use with
existing plugins (files.upload, lakebase.query, express routes).

Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
Co-authored-by: Isaac
Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>
@keugenek keugenek changed the title feat: add proto plugin for typed data contracts docs: add protobuf codegen guide + scenario test example Apr 1, 2026
@keugenek keugenek self-assigned this Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant