Class NodeSorterHostProximity
- java.lang.Object
-
- org.apache.storm.scheduler.resource.strategies.scheduling.sorter.NodeSorterHostProximity
-
- All Implemented Interfaces:
INodeSorter
public class NodeSorterHostProximity extends Object implements INodeSorter
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static interface
NodeSorterHostProximity.ExistingScheduleFunc
interface for calculating the number of existing executors scheduled on a object (rack or node).
-
Field Summary
Fields Modifier and Type Field Description protected Cluster
cluster
protected List<String>
favoredNodeIds
protected List<String>
greyListedSupervisorIds
protected BaseResourceAwareStrategy.NodeSortType
nodeSortType
protected TopologyDetails
topologyDetails
protected List<String>
unFavoredNodeIds
-
Constructor Summary
Constructors Constructor Description NodeSorterHostProximity(Cluster cluster, TopologyDetails topologyDetails)
NodeSorterHostProximity(Cluster cluster, TopologyDetails topologyDetails, BaseResourceAwareStrategy.NodeSortType nodeSortType)
Initialize for the default implementation node sorting.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description Map<String,Set<String>>
getRackIdToHosts()
Map<String,AtomicInteger>
getScheduledExecCntByRackId()
Iterable<ObjectResourcesItem>
getSortedRacks()
Racks are sorted by two criteria.List<RasNode>
hostnameToNodes(String hostname)
hostname to Ids.protected List<String>
makeHostToNodeIds(List<String> hosts)
void
prepare(ExecutorDetails exec)
Prepare for node sorting.Iterable<String>
sortAllNodes()
protected Iterable<ObjectResourcesItem>
sortObjectResources(ObjectResourcesSummary resourcesSummary, ExecutorDetails exec, NodeSorterHostProximity.ExistingScheduleFunc existingScheduleFunc)
Scheduling usessortAllNodes()
which eventually calls this method whose behavior can be altered by settingnodeSortType
.
-
-
-
Field Detail
-
nodeSortType
protected final BaseResourceAwareStrategy.NodeSortType nodeSortType
-
cluster
protected Cluster cluster
-
topologyDetails
protected TopologyDetails topologyDetails
-
-
Constructor Detail
-
NodeSorterHostProximity
public NodeSorterHostProximity(Cluster cluster, TopologyDetails topologyDetails)
-
NodeSorterHostProximity
public NodeSorterHostProximity(Cluster cluster, TopologyDetails topologyDetails, BaseResourceAwareStrategy.NodeSortType nodeSortType)
Initialize for the default implementation node sorting.- Parameters:
cluster
- for which nodes will be sorted.topologyDetails
- the topology to sort for.nodeSortType
- type of sorting to be applied to object resource collectionBaseResourceAwareStrategy.NodeSortType
.
-
-
Method Detail
-
prepare
public void prepare(ExecutorDetails exec)
Description copied from interface:INodeSorter
Prepare for node sorting. This method must be called beforeINodeSorter.getSortedRacks()
andINodeSorter.sortAllNodes()
.- Specified by:
prepare
in interfaceINodeSorter
- Parameters:
exec
- optional, may be null.
-
sortObjectResources
protected Iterable<ObjectResourcesItem> sortObjectResources(ObjectResourcesSummary resourcesSummary, ExecutorDetails exec, NodeSorterHostProximity.ExistingScheduleFunc existingScheduleFunc)
Scheduling usessortAllNodes()
which eventually calls this method whose behavior can be altered by settingnodeSortType
.- Parameters:
resourcesSummary
- contains all individualObjectResourcesItem
as well as cumulative statsexec
- executor for which the sorting is doneexistingScheduleFunc
- a function to get existing executors already scheduled on this object- Returns:
- an
Iterable
of sortedObjectResourcesItem
-
sortAllNodes
public Iterable<String> sortAllNodes()
- Specified by:
sortAllNodes
in interfaceINodeSorter
-
getScheduledExecCntByRackId
public Map<String,AtomicInteger> getScheduledExecCntByRackId()
-
getSortedRacks
public Iterable<ObjectResourcesItem> getSortedRacks()
Racks are sorted by two criteria.1) the number executors of the topology that needs to be scheduled is already on the rack in descending order. The reasoning to sort based on criterion 1 is so we schedule the rest of a topology on the same rack as the existing executors of the topology.
2) the subordinate/subservient resource availability percentage of a rack in descending order We calculate the resource availability percentage by dividing the resource availability on the rack by the resource availability of the entire cluster By doing this calculation, racks that have exhausted or little of one of the resources mentioned above will be ranked after racks that have more balanced resource availability. So we will be less likely to pick a rack that have a lot of one resource but a low amount of another.
- Specified by:
getSortedRacks
in interfaceINodeSorter
- Returns:
- an iterable of sorted racks
-
-