How to add a single retry for a specific scenario in interceptor?

[dshylov]

What version of gRPC are you using?

1.20.0

I am working on auth interceptor which responsible for auth tokens on every single call. Sometimes, when a token expired, I have to change auth metadata on the client side and make a retry for this call.

I have already implemented auth interceptor on start and not I'm trying to find out how to make recall when the interceptor found that token expired on onClose(..){..}

Could anybody give advice or other help with this issue?

Interceptor code:

@Override
  public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(final MethodDescriptor<ReqT, RespT> method,
      final CallOptions callOptions, final Channel next) {
    return new SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {

      // add a token into the headers
      @Override
      public void start(final Listener<RespT> responseListener, final Metadata headers) {
        if (jwtAuthHeader != null) {
          headers.put(ProtoConstants.REQ_HEADER_AUTH, jwtAuthHeader);
        } else if (basicAuthHeader != null) {
          headers.put(ProtoConstants.REQ_HEADER_AUTH, basicAuthHeader);
        }

        super.start(new SimpleForwardingClientCallListener<RespT>(responseListener) {
          // get jwt token
          @Override
          public void onHeaders(final Metadata headers) {
            final String jwtToken = headers.get(ProtoConstants.RSP_HEADER_JWT_TOKEN);
            if (jwtToken != null) {
              jwtAuthHeader = getAuthHeader(AUTH_SCHEME_JWT, jwtToken);
            }
            super.onHeaders(headers);
          }

          @Override
          public void onClose(final Status status, final Metadata trailers) {
            if (!status.isOk()) {
              switch (status.getCode()) {
              case UNAUTHENTICATED:
                if (ProtoConstants.STATUS_TOKEN_EXPIRED.equalsIgnoreCase(status.getDescription())) {
                  // TODO:
                  // here I am trying to do it
                  // TODO
                }
                break;
              default:
                break;
              }
            }
            super.onClose(status, trailers);
          }
        }, headers);
      }
    };
  }

[dapengzhang0]

Using interceptor will be quite difficult, you might need to buffer the initial metadata and request messages and send them in the second attempt.

Would you consider enabling library built-in retry? For the client side, let retryableStatusCodes contain UNAUTHENTICATED; and for the server side, if token expired, send server pushback to tell the client retry immediately and otherwise send server pushback to tell the client don't retry.

• @ejona86

[dshylov]

@dapengzhang0
I tried to use built-in retry but in this case, my client interceptor was called only once. All auth logic built on the client side and the server doesn't renew a JWT token.

Flow example in a case when the client has already had a JWT token:

1. send the existing jwt token
2. if STATUS_TOKEN_EXPIRED happened:
   2.1) erase existing token
   2.2) send base auth token which created on user credentials to get a new one JWT token
3. save new JWT token for further work

[dshylov]

you might need to buffer the initial metadata and request messages and send them in the second attempt.

This is exactly what I want

[dapengzhang0]

If you want to renew the JWT token from the trailing metadata of first failed attempt (Is this your usecase?), then using built-in retry is also difficult - you need to use ClientStreamTracer to do the renew.

So then either buffering the request using interceptor, or renewing token using built-in retry + ClientStreamTracer is difficult. It's up to you to pick suitable approach for your usecase.

[dshylov]

@dapengzhang0

For a clear view.
If a JWT token expired I just want to resend the same request but replace the expired JWT token on user credentials (username-password) and that action will give me a new JWT token.

I would like to try buffering, but unfortunately, I can't find any information about it in grpc-java. Maybe you could give me advice? Thanks

[dapengzhang0]

@dshylov Sure, what type is your RPC, unary, server streaming, client streaming or bidi-streaming? If it's only unary or server streaming the solution will be much easier.

[sanjaypujare]

@dshylov since you have to supply user-credentials (username+password) on STATUS_TOKEN_EXPIRED isn't it better to do the retry logic in the client application rather than an interceptor?

[dshylov]

@dapengzhang0 Sorry for the late response.
Almost all requests are unary-unary and only the one is unary-stream.

[dshylov]

@sanjaypujare
For an unsuccessful request which fails with STATUS_TOKEN_EXPIRED the interceptor erase existing JWT token, after that the interceptor check that JWT token doesn't exist and send basic auth token credentials to obtain a new JWT one. Since we add auth creds in the interceptor, we don't need to duplicate code or some invocation something among all requests. So I truly hope that a buffering solution could help here : )

[sanjaypujare]

@dshylov My point is client-interceptor may not be the best place for this kind of logic given the constraints. Can you check the JWT "exp" field and if the token has expired or about to expire in X milliseconds can you use basic auth in the 1st attempt itself? This logic will be in your start() method.

[dapengzhang0]

@dshylov sorry for not providing a code example for buffering and retry using interceptors. I found that that solution would be quite hacky. It may not work well with other interceptors. So I would like first to see should we really need this. As @sanjaypujare pointed out, checking expiration ahead of time seems more natural, but that could not handle the case that the server revokes the token for other reasons (e.g. no expiration, server thinks that the client is a robot, or server can not cache the token because it crashed and restarted).

[dshylov]

@dapengzhang0 @sanjaypujare
Exactly. This can lead to failure if the client sends a request with a near-expired token (but not enough for renewing it) or fall in retry if the server is UNAVAILABLE (for example, the server may restart) or request will be very slow.

[dapengzhang0]

Hi @dshylov , I'm providing an interceptor that buffers the message and retries, you can add your logic in
needToRetry(status, trailers). Hope this could solve your problem.

  class RetryAuthTokenInterceptor implements ClientInterceptor {

    @Override
    public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
        final MethodDescriptor<ReqT, RespT> method,
        final CallOptions callOptions,
        final Channel next) {

      class RetryingUnaryRequestClientCall<ReqT, RespT> extends ClientCall<ReqT, RespT> {

        Listener listener;
        Metadata metadata;
        ReqT msg;
        int req;
        ClientCall call;

        @Override
        public void start(Listener listener, Metadata metadata) {
          this.listener = listener;
          this.metadata = metadata;
        }

        @Override
        public void sendMessage(ReqT msg) {
          assert this.msg == null;
          this.msg = msg;
        }

        @Override
        public void request(int num) {
          req += num;
          assert this.msg == null;
        }

        @Override
        public boolean isReady() {
          return false;
        }

        @Override
        public void halfClose() {
          startCall(new CheckingListener());
        }

        private void startCall(Listener listener) {
          call = next.newCall(method, callOptions);
          Metadata headers = new Metadata();
          headers.merge(metadata);
          call.start(listener, headers);
          assert this.msg != null;
          call.request(req);
          call.sendMessage(msg);
          call.halfClose();
        }

        @Override
        public void cancel(String s, Throwable t) {
          if (call != null) { // need synchronization
            call.cancel(s, t);
          }
          // technically should use CallOptions.getExecutor() if set
          listener.onClose(Status.CANCELLED.withDescription(s).withCause(t), new Metadata());
        }

        class CheckingListener extends ForwardingClientCallListener {

          Listener<RespT> delegate;

          @Override
          protected Listener delegate() {
            if (delegate == null) {
              throw new IllegalStateException();
            }
            return delegate;
          }

          @Override
          public void onHeaders(Metadata headers) {
            delegate = listener;
            super.onHeaders(headers);
          }

          @Override
          public void onClose(Status status, Metadata trailers) {
            if (delegate != null) {
              super.onClose(status, trailers);
              return;
            }
            if (!needToRetry(status, trailers)) { // YOUR CODE HERE
              delegate = listener;
              super.onClose(status, trailers);
              return;
            }
            startCall(listener); // Only retry once
            // startCall(new CheckingListener()); // to allow multiple retries
          }
        }

      }
      return new RetryingUnaryRequestClientCall<>();
    }
  }