diff --git a/docs/Documentation.md b/docs/Documentation.md index d1bafe46c3aa8..04b97fc5cdf57 100644 --- a/docs/Documentation.md +++ b/docs/Documentation.md @@ -12,6 +12,7 @@ - [Geo-Replication](GeoReplication.md) - [WebSocket API](WebSocket.md) - [Apache Storm adaptor](PulsarStorm.md) + - [Modular Load Manager](ModularLoadManager.md) * Internal Docs - [Binary protocol specification](BinaryProtocol.md) * Other Languages diff --git a/docs/ModularLoadManager.md b/docs/ModularLoadManager.md new file mode 100644 index 0000000000000..873c1a384ca38 --- /dev/null +++ b/docs/ModularLoadManager.md @@ -0,0 +1,112 @@ +# Modular Load Manager +The modular load manager, implemented in `ModularLoadManagerImpl`, is a flexible alternative to the previously +implemented load manager `SimpleLoadManagerImpl` which attempts to simplify how load is manager while also providing +abstractions so that complex load management strategies may be implemented. + +## Usage +To use the modular load manager, change + +`loadManagerClassName=com.yahoo.broker.loadbalance.impl.SimpleLoadManagerImpl` + +in `broker.conf` to + +`loadManagerClassName=com.yahoo.broker.loadbalance.impl.ModularLoadManagerImpl` + +Alternatively, the load manager may also be changed dynamically via the `pulsar-admin` tool as follows: + +`pulsar-admin update-dynamic-config --config loadManagerClassName --value +com.yahoo.broker.loadbalance.impl.ModularLoadManagerImpl` + +The admin tool may also be used to change back to `com.yahoo.broker.loadbalance.impl.SimpleLoadManagerImpl`. In either +case, any mistake in specifying the load manager will cause Pulsar to default to `SimpleLoadManagerImpl`. + +## Implementation + +### Data +The data monitored by the modular load manager is contained in the class `com.yahoo.pulsar.broker.loadbalance.LoadData`. +Here, the available data is subdivided into the bundle data and the broker data. + +#### Broker Data +The broker data is contained in the class `com.yahoo.pulsar.broker.BrokerData`. It is further subdivided into two parts, +one being the local data which every broker individually writes to ZooKeeper, and the other being the historical broker +data which is written to ZooKeeper by the leader broker. + +##### Local Broker Data +The local broker data is contained in the class +`com.yahoo.pulsar.broker.LocalBrokerData` and gives information about the following resources: + +* CPU usage +* JVM heap memory usage +* Direct memory usage +* Bandwidth in/out usage +* Most recent total message rate in/out across all bundles +* Total number of topics, bundles, producers, and consumers +* Names of all bundles assigned to this broker +* Most recent changes in bundle assignments for this broker + +The local broker data is updated periodically according to the service configuration +"loadBalancerReportUpdateMaxIntervalMinutes". After any broker updates their local broker data, the leader broker will +receive the update immediately via a ZooKeeper watch, where the local data is read from the ZooKeeper node +`/loadbalance/brokers/` + +##### Historical Broker Data +The historical broker data is contained in the class `com.yahoo.pulsar.broker.TimeAverageBrokerData`. +In order to reconcile the need to make good decisions in a steady-state scenario and make reactive decisions in a +critical scenario, the historical data is split into two parts: the short-term data for reactive decisions, and the +long-term data for steady-state decisions. Both time frames maintain the following information: + +* Message rate in/out for the entire broker +* Message throughput in/out for the entire broker + +Unlike the bundle data, the broker data does not maintain samples for the global broker message rates and throughputs, +which is not expected to remain steady as new bundles are removed or added. Instead, this data is aggregated over the +short-term and long-term data for the bundles. See the section on bundle data to understand how that data is collected +and maintained. The historical broker data is updated for each broker in memory by the leader broker whenever any broker +writes their local data to ZooKeeper. Then, the historical data is written to ZooKeeper by the leader broker +periodically according to the configuration `loadBalancerResourceQuotaUpdateIntervalMinutes`. + +##### Bundle Data +The bundle data is contained in the class `com.yahoo.pulsar.broker.BundleData`. +Like the historical broker data, the bundle data is split into a short-term and a long-term time frame. +The information maintained in each time frame is + +* Message rate in/out for this bundle +* Message Throughput In/Out for this bundle +* Current number of samples for this bundle + +The time frames are implemented by maintaining the average of these values over a set, limited number of samples, where +the samples are obtained through the message rate and throughput values in the local data. Thus, if the update interval +for the local data is 2 minutes, the number of short samples is 10 and the number of long samples is 1000, the +short-term data is maintained over a period of `10 samples * 2 minutes / sample = 20 minutes`, while the long-term +data is similarly over a period of 2000 minutes. Whenever there are not enough samples to satisfy a given time frame, +the average is taken only over the existing samples. When no samples are available, default values are assumed until +they are overwritten by the first sample. Currently, the default values are + +* Message rate in/out: 50 messages per second both ways +* Message throughput in/out: 50KB per second both ways + +The bundle data is updated in memory on the leader broker whenever any broker writes their local data to ZooKeeper. +Then, the bundle data is written to ZooKeeper by the leader broker periodically at the same time as the historical +broker data, according to the configuration `loadBalancerResourceQuotaUpdateIntervalMinutes`. + +### Traffic Distribution +The modular load manager uses the abstraction provided by +`com.yahoo.pulsar.broker.loadbalance.ModularLoadManagerStrategy` to make decisions about bundle assignment. The +strategy makes a decision by considering the service configuration, the entire load data, and the bundle data for the +bundle to be assigned. Currently, the only supported strategy is +`com.yahoo.pulsar.broker.loadbalance.impl.LeastLongTermMessageRate`, though soon users will have the ability to inject +their own strategies if desired. + +#### Least Long Term Message Rate Strategy +As its name suggests, the least long term message rate strategy attempts to distribute bundles across brokers so that +the message rate in the long-term time window for each broker is roughly the same. However, simply balancing load based +on message rate does not handle the issue of asymmetric resource burden per message on each broker. Thus, the system +resource usages, which are CPU, memory, direct memory, bandwidth in, and bandwidth out, are also considered in the +assignment process. This is done by weighting the final message rate according to +`1 / (overload_threshold - max_usage)`, where `overload_threshold` corresponds to the configuration +`loadBalancerBrokerOverloadedThresholdPercentage` and `max_usage` is the maximum proportion among the system resources +that is being utilized by the candidate broker. This multiplier ensures that machines with are being more heavily taxed +by the same message rates will receive less load. In particular, it tries to ensure that if one machine is overloaded, +then all machines are approximately overloaded. In the case in which a broker's max usage exceeds the overload +threshold, that broker is not considered for bundle assignment. If all brokers are overloaded, the bundle is randomly +assigned. \ No newline at end of file