[SPARK-15353] [CORE] Making peer selection for block replication pluggable#13152

Closed

shubhamchopra wants to merge 25 commits into

apache:masterfrom

shubhamchopra:RackAwareBlockReplication

Commits on Sep 20, 2016

Adding capability to prioritize peer executors based on rack awareness while replicating blocks.
shubhamchopra
committed
Minor modifications to get past the style check errors.
shubhamchopra
committed
Using blockId hashcode as a source of randomness, so we don't keep choosing the same peers for replication.
shubhamchopra
committed
Several changes:
shubhamchopra
committed
Adding null check so a Block Manager can be initiaziled without the master.
shubhamchopra
committed
Renaming classes/variables from rack to a more general topology.
shubhamchopra
committed
Renaming classes/variables from rack to a more general topology.
shubhamchopra
committed
We continue to randomly choose peers, so there is no change in current behavior.
shubhamchopra
committed
Spelling correction and minor changes in comments to use a more general topology instead of rack.
shubhamchopra
committed
Minor change. Changing replication info message to debug level.
shubhamchopra
committed
Providing peersReplicateTo to the prioritizer.
shubhamchopra
committed
Adding developer api annotations to TopologyMapper and BlockReplicationPrioritization
shubhamchopra
committed
Changes recommended by @HyukjinKwon to fix style issues.
shubhamchopra
committed
Updating prioritizer api to use current blockmanager id for self identification.
shubhamchopra
committed
BlockManagerInitialization now only uses a single message to master to get a fully fleshed out id, with topology information, if available.
shubhamchopra
committed
1. Changing tail recursive function in BlockManager to an imperative while loop, as suggested by @ericl
shubhamchopra
committed
Fixing style issues.
shubhamchopra
committed
Fixing style issues.
shubhamchopra
committed
Adding documentation to clarify on topology.
shubhamchopra
committed
Adding a simple file based topology-mapper.
shubhamchopra
committed
Fixing style issues.
shubhamchopra
committed
Inlining call to get topology.
shubhamchopra
committed
converting to not a and not b
shubhamchopra
committed
1. Style modifications as suggested by @rxin. 2. Changing the BlockReplicationPolicy api to take the number of peers needed and adding a sampling algo linear in time and space along with test cases.
shubhamchopra
committed
Incorporating corrections and suggestions by @rxin
shubhamchopra
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-15353] [CORE] Making peer selection for block replication pluggable#13152

[SPARK-15353] [CORE] Making peer selection for block replication pluggable#13152
shubhamchopra wants to merge 25 commits into
apache:masterfrom
shubhamchopra:RackAwareBlockReplication

Commits on Sep 20, 2016

Adding capability to prioritize peer executors based on rack awareness while replicating blocks.

Minor modifications to get past the style check errors.

Using blockId hashcode as a source of randomness, so we don't keep choosing the same peers for replication.

Several changes:

Adding null check so a Block Manager can be initiaziled without the master.

Renaming classes/variables from rack to a more general topology.

Renaming classes/variables from rack to a more general topology.

We continue to randomly choose peers, so there is no change in current behavior.

Spelling correction and minor changes in comments to use a more general topology instead of rack.

Minor change. Changing replication info message to debug level.

Providing peersReplicateTo to the prioritizer.

Adding developer api annotations to TopologyMapper and BlockReplicationPrioritization

Changes recommended by @HyukjinKwon to fix style issues.

Updating prioritizer api to use current blockmanager id for self identification.

BlockManagerInitialization now only uses a single message to master to get a fully fleshed out id, with topology information, if available.

1. Changing tail recursive function in BlockManager to an imperative while loop, as suggested by @ericl

Fixing style issues.

Fixing style issues.

Adding documentation to clarify on topology.

Adding a simple file based topology-mapper.

Fixing style issues.

Inlining call to get topology.

converting to not a and not b

1. Style modifications as suggested by @rxin. 2. Changing the BlockReplicationPolicy api to take the number of peers needed and adding a sampling algo linear in time and space along with test cases.

Incorporating corrections and suggestions by @rxin