Don't sample traces for grpc health checks

[jjhuff]

ochttp appears to ignore requests from /healthz, etc (https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/server.go#L96). This would be a really handy feature in ocgrpc

Is your feature request related to a problem? Please describe.
My service gets more health check requests than actual requests, so my trace list is swamped.

Describe the solution you'd like
Filter them out automatically or allow another way to filter them.

Describe alternatives you've considered
Switch to a healthcheck on another port.

[leosunmo]

Would find this feature very useful. The workaround to use a different port for healthchecks is not ideal as you want to healthcheck the actual serving port.

[lunemec]

Agree, the only option I can think of for now is to create local fork of the plugin and hard-code this check.

[lunemec]

I used this wrapper to remove traces from health checks:

import (
	"google.golang.org/grpc/stats"
)

type noTraceKey struct{}

var noTrace = struct{}{}

// statsWrapper wraps grpc stats.Handler and removes healthchecks from tracing.
type statsWrapper struct {
	statsHandler stats.Handler
}

// NewStatsWrapper wraps grpc stats.Handler and removes healthchecks from tracing.
func NewStatsWrapper(statsHandler stats.Handler) stats.Handler {
	return &statsWrapper{statsHandler: statsHandler}
}

// HandleConn exists to satisfy gRPC stats.Handler.
func (s *statsWrapper) HandleConn(ctx context.Context, cs stats.ConnStats) {
	// no-op
}

// TagConn exists to satisfy gRPC stats.Handler.
func (s *statsWrapper) TagConn(ctx context.Context, cti *stats.ConnTagInfo) context.Context {
	// no-op
	return ctx
}

// HandleRPC implements per-RPC tracing and stats instrumentation.
func (s *statsWrapper) HandleRPC(ctx context.Context, rs stats.RPCStats) {
	// Check if the context contains noTraceKey, and trace only when its
	// not present.
	_, ok := ctx.Value(noTraceKey{}).(struct{})
	if !ok {
		s.statsHandler.HandleRPC(ctx, rs)
	}
}

// TagRPC implements per-RPC context management.
func (s *statsWrapper) TagRPC(ctx context.Context, rti *stats.RPCTagInfo) context.Context {
	if rti.FullMethodName == "/grpc.health.v1.Health/Check" {
		// Add to context we don't want to trace this.
		return context.WithValue(ctx, noTraceKey{}, noTrace)
	}
	return s.statsHandler.TagRPC(ctx, rti)
}

Which you must add to your grpc server and client like this:

grpc.StatsHandler(NewStatsWrapper(&ocgrpc.ServerHandler{}))
grpc.WithStatsHandler(NewStatsWrapper(&ocgrpc.ClientHandler{}))