Add server-side lookup throttling

[rdhabalia]

Motivation

Sometimes, broker requires a way to throttle newly incoming traffic and requires to restrict number of incoming concurrent lookup requests.

Modifications

• Add broker side throttling to serve max allowed concurrent lookup request
• Update value of maxConcurrentLookupRequest dynamically at broker

Result

Broker can have capability to throttle newly incoming traffic by restricting number of concurrent lookup request.

[merlimat]

To avoid indenting the whole section, what about doing:

if (!service.getLookupRequestSemaphore().tryAcquire()) {
    /// deal with error
    return;
}

// continue normal

[rdhabalia]

Actually, here I have replaced thenAccept/exceptionally with handle(data, ex) so, don't have to keep duplicate service.getLookupRequestSemaphore().release(). So, this change anyway changed entire section so, kept into if/else.

[merlimat]

log at debug level

[merlimat]

Dynamic configuration of the parameter should go in separate PR

[rdhabalia]

is that fine if we remove this configuration-update once other PR is created?

[merlimat]

Either that, or just use the value from config file as it is (like any other config parameter) and then we can open a separate issue to discuss the broader dynamic config approach.

[rdhabalia]

yes, we can read from config-file but part of this PR also requires to test updating semaphore at runtime and if config value gets changed then we are also updating lookupRequestSemaphore so, even with Dynamic-config-change we might need this listener logic and we have to provide a way to inject listener on config value change in dynamic-config-PR.
So, should we keep this PR open until we come up with the approach as anyway it should be part of 1.17 release.

[merlimat]

Sure, that makes sense

[rdhabalia]

@merlimat rebased this one with the #186 changes.

[merlimat]

LGTM

[merlimat]

Minor thing: this should be plural: TooManyRequests