binder: ServerAuthInterceptor might use an Executor that BinderServer has already returned to the pool

[mateusazis]

This is a follow-up to #10566.

(note: I'm describing the issue as I recollect it; Chat ate the discussion thread. @markb74 please correct me if I'm wrong)

#10633 introduced the PendingAuthListener class and the logic to handle AsyncSecurityPolicy without blocking any threads. This required an Executor to be passed to BinderTransportSecurity here. In the same PR, we added a shutdown listener to return that executor to the object pool during server shutdown (here).

@markb74 commented that this was sub-optimal, and we could instead return the executor during termination. One suggestion was to pass an Executor to BinderServer.start and reuse it throught that server's lifetime. I prototyped it in mateusazis@3f262ca, but ended up with a chicken-and-egg issue in BinderServerBuilder.build:

• BinderTransportSecurity.installAuthInterceptor wants an Executor, but it will only be available on BinderServer after it is instantiated and started
• BinderServer is only instantiated after the authorization interceptor gets installed

Any help here is welcome!

[jdcormie]

I think I finally understand the problem Mark was pointing out.

Today BinderServer returns this Executor to the pool in InternalServer#shutdown(). But shutdown() merely "Initiates an orderly shutdown of the server. Existing transports continue ....". And ServerAuthInterceptor is invoked for every call, even on established transports. So we have a bug where ServerAuthInterceptor might use this executor after it has been released. The upshot could be RejectedExecutionExceptions, since the resource pool may call shutdown on a returned Executor.

It seems to me that we can't safely release this executor until the last BinderServerTransport created by this BinderServer has called ServerTransportListener#transportTerminated()

[mateusazis]

Ah, that does ring a bell w.r.t. to the chat I had with him. #11240 is an attempt to address it. PTAL when you have a chance.