Broadcast and scatter algorithms on meshbased topologies

Lecture outline l Model definition and lower bounds l One-to-All Broadcast algorithm on hypercube l All-to-All Broadcast algorithm on hypercube l One-to-All Scatter algorithm on hypercube l All-to-All Scatter algorithm on hypercube l Broadcast algorithms on torus

Model definition l Ports number (1 -port, k-port, all-port) l Directedness of channels (simplex, l half-duplex, full-duplex) l Switching technique (store-and-forward (SF) or wormhole (WH) ) l Packet manipulation capability (combining model , noncombining model, intermediate reception capability )

Lower bounds for basic communication algorithms on hypercube All-port, full-duplex, SF model comm. operation O-A-B A-A-B O-A-S A-A-S # of rounds # of transmissions

Lower bounds for basic communication algorithms on hypercube O-A-B cannot take less rounds than is the value of the diameter. Since the broadcast packet has to be delivered to distinct nodes, the number of transmissions is at least. l During A-A-B, each node has to receive distinct packets. Node can receive at most d packets in one round (on all its d links). Thus, at least rounds are needed, and the number of transmissions is at least times the number of transmissions of one O-A-B. l During O-A-S, the source has to send the total of distinct packets and it can send at most d packets in one round. There is nodes of the distance k from the source. The number of transmissions cannot be less than. l A-A-S consists of O-A-Ss running concurrently. So, the number of transmissions is. Every round at most packets can be delivered on hypercube. Thus it will take at least rounds. l

Basic communication algorithms on hypercube l The idea – building minimal-weight spanning tree, rooted at broadcasting/scattering node. l Generally, building minimal-weight spanning tree on hypercube is easy task. l In scatter algorithms the tree must be balanced.

One-to-All Broadcast algorithm on hypercube

One-to-All Broadcast algorithm on hypercube l The idea: At every round k communication only on kdimension links. l The Algorithm (for any node i) : ¡ ¡ ¡ to. Send = true; Msg contains value - for broadcasting node, to. Send = false; Msg is null - for all other nodes At round k do: l If to. Send = true then send Msg on link k l Else if received message M` then • Msg M` • to. Send true End of round

One-to-All Broadcast algorithm on hypercube k=0 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

One-to-All Broadcast algorithm on hypercube k=1 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

One-to-All Broadcast algorithm on hypercube k=2 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

One-to-All Broadcast algorithm on hypercube k=3 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

One-to-All Broadcast algorithm on hypercube k=4 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

One-to-All Broadcast algorithm on hypercube k=4 , End of round d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

One-to-All Broadcast algorithm on hypercube The kind of such built tree is called, d-level spanning binomial tree – l Complexity analysis: l # of rounds: d l # of transmissions: l The algorithm is optimal! l Note: The algorithm didn't assume all-port, full -duplex model. So, the algorithm on 1 -port, halfduplex model satisfies lower bound of all-port, fullduplex model. l

All-to-All Broadcast algorithm on hypercube

All-to-All Broadcast algorithm on hypercube l The start: Every node builds its spanning tree. Denote by – spanning tree for node i ( root of is i). The tree consists of sequence of disjoint sets of directed links , where: l q – number of rounds to broadcast packet on l is set of links on which occurs transmission of a packet at round k. l Actually, the sequence defines algorithm for broadcasting packet from node i.

All-to-All Broadcast algorithm on hypercube k=0 (000) d=3 (001) (100) (101) (000) (011) (010) (111)

All-to-All Broadcast algorithm on hypercube k=1 (000) d=3 (001) (100) (001) (101) (000) (011) (010) (111) (100)

All-to-All Broadcast algorithm on hypercube k=2 (000) d=3 (001) (100) (001) (101) (000) (101) (010) (011) (010) (111) (100) (110)

All-to-All Broadcast algorithm on hypercube k=3 (000) d=3 (001) (100) (001) (101) (000) (101) (010) (011) (010) (111) (100) (110)

All-to-All Broadcast algorithm on hypercube d=3 k=3, End of round (000) (001) (101) (000) (101) (010) (011) (010) (111) (100) (110)

All-to-All Broadcast algorithm on hypercube Denote by T spanning tree with root in and accordingly. l Lemma: For every node t, the sequence of sets defines algorithm for broadcasting packet from node t (proof based on fact l that and differ in a particular bit if and only if x and y differ on the same bit, so is a link if and only if (x, y) is a link).

All-to-All Broadcast algorithm on hypercube (001) (000) (101) (111) (010) (011) (100) (110) (011) (111) (010) (101) (000) (001) (110) (100)

All-to-All Broadcast algorithm on hypercube In order to broadcast simultaneously, for every two trees next property should hold: (*) ¡

All-to-All Broadcast algorithm on hypercube Now we show to construct such that will hold property (*). l Definition: A link(x, y) is of type j if x and y differ in the j-th bit. l Observation: If for each k, the links in are of different types, then for each k, the sets , where t ranges over all nodes, are disjoint. (if for , links and were l from the same type, then (since ), and they would be of the same type because and are of the same type as and respectively contradicts the fact that and both belongs to ) l Conclusion: We need to construct such the links in are off different types. that for every i,

All-to-All Broadcast algorithm on hypercube l Step I: Dividing: ¡ Define as a set of d-bit length nodes having k unity bits and d-k zero bits. ¡ For each set define disjoint subsets which are equivalence classes under single bit rotation to the left. ¡ For each k, is the equivalence class of the element.

All-to-All Broadcast algorithm on hypercube Division of space {0, . . , 31}

All-to-All Broadcast algorithm on hypercube l Step ¡ II: Numbering: For each node t associate distinct number n(t) in next order: ¡ Define sequence of numbers. Thus, m(t) that corresponding to sequence n(t) is d, 1, 2, . . , d, 1, 2…

All-to-All Broadcast algorithm on hypercube Numbering of space {0, . . , 31}

All-to-All Broadcast algorithm on hypercube l Step III: Ordering: ¡ Every first element t in must fulfill: the bit in position m(t) from the right should be 1 ¡ Every element t in must fulfill: the bit in position m(t)-1 [m(t)>1] or d [m(t)=1] from the right should be 0

All-to-All Broadcast algorithm on hypercube l Step ¡ IV: Constructing : For , define node sets: ¡ For each set consists of the links, that connect nodes with corresponding nodes in obtained from t by reversing the bit in position m(t). (In particular, nodes in set connected to corresponding nodes in because of ordering)

All-to-All Broadcast algorithm on hypercube l Step V: Building routing table (for node k): In routing table: rows = rounds, columns = links. ¡ Every entry contains only source node. ¡ Every row i constructed from set by creating entry x at column for every link (x, y). ¡ In first row the source is k itself and the destination nodes are k-th neighbors. ¡

All-to-All Broadcast algorithm on hypercube m(0001)=1 m(1001)=4 m(1101)=3 m(0010)=2 m(0011)=1 m(1011)=4 m(1111)=3 m(0100)=3 m(0110)=2 m(0101)=1 m(0111)=1 m(1000)=4 m(1100)=3 m(1010)=2 m(1110)=2 (0000)

All-to-All Broadcast algorithm on hypercube Routing table: roundslinks 1 (0001) 2 (0010) 3 (0100) 4 (1000) 1 0000 2 0010 0100 1000 0001 3 0100 1001 0011 4 0110 1100 1011

All-to-All Broadcast algorithm on hypercube k=0 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

All-to-All Broadcast algorithm on hypercube k=1 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

All-to-All Broadcast algorithm on hypercube k=2 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

All-to-All Broadcast algorithm on hypercube k=3 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

All-to-All Broadcast algorithm on hypercube k=4 d=4 (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

All-to-All Broadcast algorithm on hypercube d=4 k=4, End of round (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

All-to-All Broadcast algorithm on hypercube Algorithm (for any node i): At the initialization: ¡ Build routing table as shown before ¡ Put d packets into queue (packet header include source and destination nodes). l At round k do: ¡ Send all messages in queue that should be sent according to the routing table. ¡ Receive messages and put them into queue ¡ End of round l

All-to-All Broadcast algorithm on hypercube l Complexity analysis: Every one of the sets elements. Also ¡ # of rounds: ¡ # of transmissions: ¡ has exactly d. Therefore l The algorithm is optimal! l Note: We didn't prove correctness of building contains links only of different types (for every k) ¡ every link holds (for every k) – actually, we have to prove only that ¡

One-to-All Scatter algorithm on hypercube

One-to-All Scatter algorithm on hypercube l Observation I: A-A-B and O-A-S problems are similar, but here we need to build for scattering node balanced spanning tree: On first level d sub-trees, with size at most. l Observation II: For any tree T with root t, of size n, scatter problem via single link to root, resolved in n rounds (by giving priority to furthest nodes).

One-to-All Scatter algorithm on hypercube l Definition: class is compatible with class if has d nodes and there exist two nodes. , such that is obtained from. by reversing some unity bit of to zero bit. l Observation III: For every class there exists a compatible class. (Take any node whose rightmost bit is a 1, and leftmost bit is a 0. Let s be string of consecutive zeros with maximal number of bits and let be the node obtained from by reversing the unity bit immediately to the right of s. Then, the equivalence class of is compatible with )

One-to-All Scatter algorithm on hypercube l Algorithm ¡ Steps for scattering node (0, . . , 0) : I-II as in A-A-Broadcast algorithm. ¡ Step IV: Constructing Spanning Tree: l Start with empty tree T (only nodes). l Add to T links, connecting (0, . . , 0) with all the nodes in. The first node in is chosen arbitrarily. l For each class (in ascending order) find its compatible class and add corresponding links to the tree T. l Add to T links, connecting (1, . . , 1) with all the nodes in.

One-to-All Scatter algorithm on hypercube l Observation IV: For any we have where and can be obtained from reversing some unity of to 0. by l Conclusion: Each node x (except (0, . . , 0)) is in the sub -tree. Since there at most nodes x having the same value m(x), each sub-tree contains at most nodes. Therefore tree T is balanced

One-to-All Scatter algorithm on hypercube Algorithm (for scattering node i): l Node i: ¡ At the initialization: l Build spanning tree. l Distribute tree structure (optional). ¡ At round k do: l On every link l send message to node at distance q-k on the sub-tree. (q = l End of round )

One-to-All Scatter algorithm on hypercube Algorithm (for scattering node i): l Any other node j: ¡ At the initialization: l Receive and forward tree structure (optional). ¡ At any round do: l On receiving message: store (if message target = j), or forward according to spanning tree structure. l End of round

One-to-All Scatter algorithm on hypercube m(t)=1 (0001) (0011) (0101) (1011) m(t)=2 (0010) (0110) (1010) (0111) m(t)=3 (0100) (1110) (1111) m(t)=4 (1000) (1001) (1101) (0000)

One-to-All Scatter algorithm on hypercube (0000) (1100) (0101) (1001) (1101) (0001) (0110) (0011) (1010) (1110) (1011) (0111) (1111)

One-to-All Scatter algorithm on hypercube l Complexity analysis: ¡ There are nodes at distance k from the root. ¡ # of rounds: ¡ # of transmissions: ¡ Time + transmission overhead for spanning tree structure distribution. l The algorithm is optimal!

All-to-All Scatter algorithm on hypercube

All-to-All Scatter algorithm on hypercube l The l idea: Divide-and-Conquer Recursive construction. 3 Phases. ¡ Decompose d-cube into two (d-1)-cubes and. ¡ Assume (without loss of generality) that nodes and nodes ¡ I. A-A-S recursive algorithm on and simultaneously. ¡ II. Each node in transmits to it counterpart node all of the packets that are destined to the nodes in , giving priority to the furthest nodes. ¡ III. A-A-S recursive algorithm on and simultaneously. in

All-to-All Scatter algorithm on hypercube d-cube

All-to-All Scatter algorithm on hypercube Phase I

All-to-All Scatter algorithm on hypercube Phase II

All-to-All Scatter algorithm on hypercube Phase III

All-to-All Scatter algorithm on hypercube l# of rounds: l# of transmissions: l The algorithm is not optimal!

All-to-All Scatter algorithm on hypercube l Solution: performing Phase II simultaneously with phases I and III. ¡ During Phases I and III links that algorithm uses in Phase II are not used. ¡ Phases I and III take time every one. l Conclusion: first half of Phase II can be performed simultaneously with Phases I, second half of Phase II performed simultaneously with Phases III.

All-to-All Scatter algorithm on hypercube l Rule to transmit packet at Phase II : Denote by the time unit during which node i transmits own packet that is destined to node j, in this algorithm on d-cube. ¡ Each node transmits packets to node in next order: the packet destined for is transmitted before the packet destined to if. ¡ It is also an order, in which the latter node forwards them in Phases III. ¡ Each node transmits packet destined for node last. ¡

All-to-All Scatter algorithm on hypercube Lemma: Under the above rules Phases III proceed uninterrupted. l Proof: Denote by the number of its own packets that l node i has transmitted up to time n (including n). Particulary, it is also number of packets of node that must be transmitted by node during the first n time units of Phases III. ¡ We observe that is the number of received by node packets, from node after first n time units of Phases III. ¡ Now, by induction on d, we prove that for every d, there exists total exchange algorithm for the d-cube satisfies: and

All-to-All Scatter algorithm on hypercube l Result: ¡# of rounds: ¡# of transmissions (didn't changed): l The algorithm is now optimal!

All-to-All Scatter algorithm on hypercube l Problem: Divide-and-Conquer solution now is not applicable. l Solution: Implementation of iterative algorithm (for any node i)

All-to-All Scatter algorithm on hypercube l Definitions ¡ Write node as binary number ¡ Denote by , node ¡ Denote by reverse of bit ¡ Link between i and . . . called k-th link of i.

All-to-All Scatter algorithm on hypercube l I. Sending its own packets During rounds , node i transmits all its packets for nodes (where - any bits) through its k-th link. ¡ The order of sending packets on k-th link is according to distance from node i. ¡ Packet for node is sent last. ¡ The exact order derived from a table, that can be built iteratively (by combining previous columns). ¡

All-to-All Scatter algorithm on hypercube For node i =(0000) Time Link 1 Link 2 Link 3 Link 1 Time 0001 1 0011 2 0101 1 0001 0011 2 0010 0101 0111 Time Link 1 1 00010010 00110111 2 3 0110 1 0001 3 00100110 0100 4 2 4 0100 5 d=4 Link 4 1001 1010 1101 1111 6 1110 7 1100 8 1000

All-to-All Scatter algorithm on hypercube k=0 (000) (001) d=3 (011) (010) (101) (100) (111)

All-to-All Scatter algorithm on hypercube k=1 (000) (001) d=3 m(001) m(011) (010) m(101) (100) (111)

All-to-All Scatter algorithm on hypercube k=2 (000) (001) m(010) d=3 (011) (010) m(011) m(111) (100) m(101) (110) (101) (111)

All-to-All Scatter algorithm on hypercube k=3 (000) (001) d=3 (011) (010) m(110) (101) (100) m(111) (110) (111)

All-to-All Scatter algorithm on hypercube k=4 (000) (001) d=3 (011) (010) m(100) (101) (100) m(110) m(111)

All-to-All Scatter algorithm on hypercube k=4, End of Round (001) d=3 (000) (011) (010) (101) (100) (111)

All-to-All Scatter algorithm on hypercube l II. ¡ Forwarding arriving packets A packet, destined for node placed in queue which contains packets to be transmitted by i through the -th link, where Packets from different source nodes that placed in same queue, are ordered according to the lexicographic order between and. ¡ At k-th link forwarding packets start at round ¡ is

All-to-All Scatter algorithm on hypercube l Notes: Any node i can build such table by calculating all entries XOR i. Thus any packet route can be calculated locally. ¡ The presented rules for transmitting packets fulfill the properties, demanded earlier in recursive building. ¡

Broadcast algorithms on torus

Broadcast algorithms on torus l Model: All-port, full-duplex, WH l Lower bound: l Intermediate research results: ¡ Recursive tiling approach ¡ Dilated-diagonal-based approach ¡ Both, applicable only on two-dimensional tori.

Broadcast algorithms on torus l Assume l torus Look at the torus as on mosaic of constant patterns

Recursive tiling approach

Recursive tiling approach 5 B-Patterns k=1

Recursive tiling approach A-Pattern k=1

Recursive tiling approach 5 A-Patterns

Recursive tiling approach B-Pattern

Recursive tiling approach 5 B-Patterns k=2

Recursive tiling approach A-Pattern k=2

Recursive tiling approach l Observation: Torus for any k can be represented as a single A-Pattern. (every A-Pattern consists of 5 B-Patterns). l Define for nodes that transmit at any odd round i, for nodes that transmit at any even round i. (the broadcaster has A-sons(B-sons) for every odd(even) i. His A-sons has B-sons for every even i, and Asons for i=3, 5, …k(or k-1). And so on…)

Recursive tiling approach l The Algorithm (for any node t ) : ¡ to. Send = true; Msg contains value - for broadcasting node, to. Send = false; Msg is null ¡ At round i ( l l l ¡ - for all other nodes ) do: If to. Send = true and i is odd then • send Msg to all (5) Else If to. Send = true and i is even then • send Msg to all (5) Else if received message M` then • Msg M` • to. Send true End of round

Recursive tiling approach l Complexity analysis: l # of rounds: 2 k, (reminder: ) l # of transmissions: every node receives message only once, therefore it is l The algorithm is optimal! l Note: The algorithm is applicable only to tori

Dilated-diagonal-based approach l Assume torus. l The algorithm consists out of 3 Phases. l Phase I: Broadcast the packet into all rows so that each row contains exactly one packet. (by recursive splitting the torus into 5 horizontal strips, sending the packet using XY routing). l Phase I takes rounds

Dilated-diagonal-based approach

Dilated-diagonal-based approach l Phase II: Align the packets to the main diagonal in all rows in parallel. l Phase II takes single round.

Dilated-diagonal-based approach

Dilated-diagonal-based approach l Phase III: Decompose the torus into 5 diagonal bands (of width ). The main diagonal is in the middle diagonal of the first one. ¡ Every node on middle diagonal sends 4 packets to the counterpart nodes on the middle diagonals of other 4 bands. ¡ All packets are send on link-disjoint paths. ¡ Recursively divide each band into 5 bands and continue with the algorithm. ¡ l Phase III takes rounds.

Dilated-diagonal-based approach

Dilated-diagonal-based approach Complexity analysis: l # of rounds: 2 k+1 l # of transmissions: every node receives message only once, therefore it is l The algorithm is almost optimal! l Note: The algorithm can be generalized and applied to arbitrary 2 -D torus, in which case it requires at most 5 rounds more than the lower bound. l

Acknowledgements l D. P. BERTSEKAS, C. OZVEREN, G. D. STAMOULIS, P. “Optimal Communication Algorithms for Hypercubes” TSENG, AND J. N. TSITSIKLISt, l PAVEL TVIRDIK, “Topics in Parallel Computing”, CS 838, UNIVERSITY OF WISCONSIN-MADISON

The End