Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions docs/docs/guides/protobuf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Using Protobuf with AppKit

Typed data contracts via protobuf codegen. Define data shapes in `.proto` files, generate TypeScript types with `buf`, use them with AppKit's files and lakebase plugins.

Not a plugin — a codegen pattern using `@bufbuild/protobuf` or `ts-proto` directly.

## When to use

- Multiple plugins exchanging data (files + lakebase + jobs)
- Backend jobs produce data that server/frontend consumes
- Python and TypeScript services share data structures
- You want compile-time guarantees on cross-boundary data

## Setup

```bash
pnpm add @bufbuild/protobuf
pnpm add -D @bufbuild/buf @bufbuild/protoc-gen-es
```

```protobuf
// proto/myapp/v1/models.proto
syntax = "proto3";
package myapp.v1;

message Customer {
string id = 1;
string name = 2;
string email = 3;
double lifetime_value = 4;
bool is_active = 5;
}
```

```bash
npx buf generate proto/
```

## With Files plugin

```ts
import { toJson, fromJson } from "@bufbuild/protobuf";
import { CustomerSchema } from "../proto/gen/myapp/v1/models_pb.js";

// Write
const json = toJson(CustomerSchema, customer);
await app.files("data").upload("customers/cust-001.json", Buffer.from(JSON.stringify(json)));

// Read
const data = await app.files("data").read("customers/cust-001.json");
const loaded = fromJson(CustomerSchema, JSON.parse(data.toString()));
```

## With Lakebase plugin

```ts
const json = toJson(CustomerSchema, customer);
await app.lakebase.query(
`INSERT INTO customers (id, name, email, lifetime_value, is_active) VALUES ($1, $2, $3, $4, $5)`,
[json.id, json.name, json.email, json.lifetimeValue, json.isActive],
);

const { rows } = await app.lakebase.query("SELECT * FROM customers WHERE id = $1", [id]);
const customer = fromJson(CustomerSchema, rows[0]);
```

## In API routes

```ts
import { fromJson, toJson } from "@bufbuild/protobuf";

expressApp.post("/api/orders", express.json(), (req, res) => {
const order = fromJson(OrderSchema, req.body); // validates shape
res.json(toJson(OrderSchema, order)); // guarantees output
});
```

## Proto → Lakebase DDL

| Proto type | SQL type | Default |
|-----------|----------|---------|
| `string` | `TEXT` | `''` |
| `bool` | `BOOLEAN` | `false` |
| `int32`/`int64` | `INTEGER`/`BIGINT` | `0` |
| `double` | `DOUBLE PRECISION` | `0.0` |
| `Timestamp` | `TIMESTAMPTZ` | `NOW()` |
| `repeated T` / `map<K,V>` | `JSONB` | `'[]'` / `'{}'` |

## Buf config

```yaml
# proto/buf.yaml
version: v2
lint:
use: [STANDARD]

# proto/buf.gen.yaml
version: v2
plugins:
- local: protoc-gen-es
out: proto/gen
opt: [target=ts]
```

Alternative: [ts-proto](https://github.com/stephenh/ts-proto) if you prefer its codegen style.
47 changes: 47 additions & 0 deletions examples/proto-catalog/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Proto Plugin Scenario Test: Product Catalog

End-to-end scenario test for the proto plugin using a sample Product Catalog app.

## What it tests

- Proto-style JSON serialization (snake_case field names in API responses)
- Proto binary endpoint (content-type `application/x-protobuf`)
- Typed contracts between server and client (same field names, types)
- Category filtering with correct product counts
- All products visible with correct data
- Error handling (404 for non-existent products)

## Run locally

```bash
# Start the app
npx tsx app/server.ts

# Run public test cases
TASK_CASES_PATH=public/cases.json npx playwright test tests/catalog.spec.ts

# Run private test cases (evaluation only)
TASK_CASES_PATH=private/cases.json npx playwright test tests/catalog.spec.ts
```

## Run against a deployment

```bash
APP_URL=https://your-app.databricksapps.com npx playwright test tests/catalog.spec.ts
```

## Structure

```
scenario/
meta.json # Task config (command, URL, timeout)
app/
server.ts # Sample AppKit app with proto-style contracts
catalog.proto # Proto definition (for reference / codegen)
public/
cases.json # 5 basic scenarios (developer verification)
private/
cases.json # 8 comprehensive scenarios (evaluation)
tests/
catalog.spec.ts # Playwright tests parameterized by cases
```
17 changes: 17 additions & 0 deletions examples/proto-catalog/catalog.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
syntax = "proto3";

package catalog.v1;

message Product {
string id = 1;
string name = 2;
string category = 3;
double price = 4;
int32 stock = 5;
bool in_stock = 6;
}

message ProductList {
repeated Product products = 1;
int32 total = 2;
}
7 changes: 7 additions & 0 deletions examples/proto-catalog/meta.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"appCommand": "npx tsx app/server.ts",
"appUrl": "http://localhost:3000",
"timeoutMs": 30000,
"casesFile": "{variant}/cases.json",
"resources": []
}
61 changes: 61 additions & 0 deletions examples/proto-catalog/private/cases.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"cases": [
{
"description": "Filter by Stationery shows single product",
"action": "filter",
"filter": "Stationery",
"expectedCount": 1,
"expectedIds": ["P006"]
},
{
"description": "Out of stock products show correct status",
"action": "filter",
"filter": "Electronics",
"expectedOutOfStock": ["P003"],
"expectedInStock": ["P001", "P002"]
},
{
"description": "Product detail API returns all proto fields",
"action": "api",
"endpoint": "/api/products/P004",
"expectedBody": {
"id": "P004",
"name": "Standing Desk",
"category": "Furniture",
"price": 499.99,
"stock": 12,
"in_stock": true
}
},
{
"description": "Non-existent product returns 404",
"action": "api",
"endpoint": "/api/products/P999",
"expectedStatus": 404
},
{
"description": "Proto binary endpoint sets correct content type",
"action": "api",
"endpoint": "/api/products.bin",
"expectedContentType": "application/x-protobuf"
},
{
"description": "All categories filter returns full catalog",
"action": "filter",
"filter": "all",
"expectedCount": 6
},
{
"description": "UI status text updates after filter",
"action": "filter",
"filter": "Furniture",
"expectedStatusText": "Showing 2 products"
},
{
"description": "Table has correct column headers",
"action": "load",
"filter": "all",
"expectedColumns": ["ID", "Name", "Category", "Price", "Stock", "In Stock"]
}
]
}
37 changes: 37 additions & 0 deletions examples/proto-catalog/public/cases.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"cases": [
{
"description": "View all products",
"action": "load",
"filter": "all",
"expectedCount": 6,
"expectedIds": ["P001", "P002", "P003", "P004", "P005", "P006"]
},
{
"description": "Filter by Electronics",
"action": "filter",
"filter": "Electronics",
"expectedCount": 3,
"expectedIds": ["P001", "P002", "P003"]
},
{
"description": "Filter by Furniture",
"action": "filter",
"filter": "Furniture",
"expectedCount": 2,
"expectedIds": ["P004", "P005"]
},
{
"description": "Health check returns proto plugin status",
"action": "api",
"endpoint": "/api/health",
"expectedBody": { "status": "ok", "plugin": "proto" }
},
{
"description": "API returns snake_case field names (proto convention)",
"action": "api",
"endpoint": "/api/products/P001",
"expectedFields": ["id", "name", "category", "price", "stock", "in_stock"]
}
]
}
Loading
Loading