OTel Event Pipeline

Phase 1 — MVP | Inspired by Booking.com's Events system architecture, rebuilt with open standards.

A Go service that receives OpenTelemetry traces via OTLP gRPC, routes them through Kafka, and forwards to Jaeger for visualization.

Architecture

                        ┌─────────────────────────────────────────────────────┐
                        │                   pipeline (Go)                      │
                        │                                                       │
 OTel SDK               │  ┌────────────┐     ┌──────────┐    ┌────────────┐  │
 (test-app)  ──OTLP──▶  │  │  Receiver  │────▶│ Producer │───▶│  Consumer  │  │
  :4317               │  │  gRPC :4317 │     │ kafka-go │    │ kafka-go   │  │
                        │  └────────────┘     └──────────┘    └────┬───────┘  │
                        │                                           │          │
                        │  ┌────────────┐                          │OTLP      │
                        │  │ HTTP :8080 │     ┌──────────┐         │          │
                        │  │ /healthz   │     │  Kafka   │◀────────┘          │
                        │  │ /readyz    │     │  KRaft   │                    │
                        │  │ /metrics   │     │otel.traces│                   │
                        │  └────────────┘     └──────────┘                    │
                        └─────────────────────────────────────────────────────┘
                                                                    │
                                                                 OTLP gRPC
                                                                    │
                                                                    ▼
                                                          ┌──────────────────┐
                                                          │      Jaeger      │
                                                          │   all-in-one     │
                                                          │  UI: :16686      │
                                                          └──────────────────┘

Data flow:

1. test-app generates traces using the OTel Go SDK → sends to pipeline on port 4317
2. pipeline receives OTLP spans, serializes as protobuf, publishes to Kafka topic otel.traces (keyed by trace ID)
3. pipeline consumer goroutine reads from otel.traces → forwards to Jaeger via OTLP gRPC
4. Jaeger stores and indexes traces — view in the UI at localhost:16686

Quickstart

# Start all services
make up

# Tail logs (Ctrl+C to stop)
make logs

# Verify health
curl localhost:8080/healthz     # → ok
curl localhost:8080/readyz      # → ok (once Kafka is ready)
curl localhost:8080/metrics     # → Prometheus metrics

# Open Jaeger UI
open http://localhost:16686

# Tear down
make down

After make up, wait ~30 seconds for Kafka to initialize. The test-app will start generating traces every 2 seconds. Search for service test-app in the Jaeger UI.

Services

Service	Port(s)	Description
pipeline	4317 (gRPC), 8080	OTLP receiver + Kafka producer/consumer
kafka	9094 (external)	Kafka KRaft single-node broker
jaeger	16686 (UI), 14317	Jaeger all-in-one with OTLP collector
test-app	—	OTel SDK trace generator

Configuration

Variable	Default	Description
GRPC_PORT	4317	OTLP gRPC listen port
HTTP_PORT	8080	Health + metrics HTTP port
KAFKA_BROKERS	kafka:9092	Comma-separated broker list
KAFKA_TOPIC	otel.traces	Kafka topic for span batches
JAEGER_ENDPOINT	jaeger:4317	Jaeger OTLP gRPC endpoint

Development

# Unit tests (no external dependencies)
make test

# Build binaries
make build

# Lint
make lint

Project Structure

otel-event-pipeline/
├── main.go          # Wires receiver + producer/consumer, starts gRPC + HTTP
├── receiver.go      # OTLP TraceService gRPC server
├── producer.go      # Kafka producer (protobuf serialization, trace ID keying)
├── consumer.go      # Kafka consumer → Jaeger OTLP forwarder
├── config.go        # Environment-based configuration
├── config_test.go   # Unit tests for config parsing
├── asyncapi.yaml    # AsyncAPI 3.0 spec — otel.traces channel definition
├── docker-compose.yaml
├── Dockerfile
├── Makefile
├── test-app/        # OTel SDK trace generator
│   ├── main.go
│   ├── Dockerfile
│   ├── go.mod
│   └── go.sum
└── CLAUDE.md

What's next (Phase 2)

• Fan-out to multiple Kafka topics (traces, metrics, logs)
• Metrics receiver (OTLP metrics → otel.metrics)
• Logs receiver (OTLP logs → otel.logs)
• Prometheus + Loki consumers