W 3 C-LA Web Mirroring Activity Stéphane Boyera

W 3 C-LA Web Mirroring Activity Stéphane Boyera, Luc Ottavj 25/05/98 Stéphane Boyera, Luc Ottavj 1

Plan Web Mirroring Keywords n Underlying Problems of Mirroring n Experimental Protocol n Hardware n Software n Results n Conclusion n 25/05/98 Stéphane Boyera, Luc Ottavj 2

Web Mirroring Keywords Transparency for the user n Homogeneity between mirrors n Independence n Fault Tolerance and flexibility n 25/05/98 Stéphane Boyera, Luc Ottavj 3

Underlying Problems of Mirroring n Building the package to send n Defining the periodicity of update n Broadcasting the update n Recovering local or global failures 25/05/98 Stéphane Boyera, Luc Ottavj 4

Periodicity of updates n True Real Time : – Initiative of mirroring given to the user n Simulated Real Time : – Initiative to the system – Short periodicity : ~30 mn - 1 hour n Differed mirroring : – Initiative to the system – Periodicity : ~daily 25/05/98 Stéphane Boyera, Luc Ottavj 5

How is an update processed ? n Classical mirroring systems : rdist, ftp, … – A master server contact each mirror one by one – Drawbacks : the more numerous the mirrors the more decreased the perf. of the overall system n Reliable Multicast technology – Advantage : not affected by the number of mirrors – Drawbacks : no standard available but products 25/05/98 Stéphane Boyera, Luc Ottavj 6

The Underlying Network Internet : actually not reliable n Satellite transmission : n – Dedicated Bandwidth – Private Network – Well fitted for the multicast technology – Covering the whole continent 25/05/98 Stéphane Boyera, Luc Ottavj 7

Experimental Protocol Building of update : endless rough process finding files to broadcast (~20 mn) n Periodicity : Simulated Real Time n Broadcast : n – using multicast through Internet – using multicast through Satellite – using unicast/TCP through Internet 25/05/98 Stéphane Boyera, Luc Ottavj 8

Hardware n On the Master Site (Sophia) – One master server housing files and being both multicast sources and unicast server – One Internet Multicast Router – One Satellite Router (Uplink) n On Mirror Sites – One Multicast router for both Internet and Satellite multicast groups – One server receiving the update by the three ways 25/05/98 Stéphane Boyera, Luc Ottavj 9

Software n Unicast / TCP : RSYNC – Free software – Implementing a gdiff algorithm n Multicast : Starburst Communication Mftp product 25/05/98 Stéphane Boyera, Luc Ottavj 10

Results Period : 02/26/1998 to 05/11/1998 n Effective Working Days : 53 n Measures : 3653 n Transfers : 1218 (0, 95 Transfer/hour) n Files : 16284 n Bytes : 0. 54 GBytes (W 3 C site : 1. 6 GBytes) n Average Update Size : 442 KBytes n 25/05/98 Stéphane Boyera, Luc Ottavj 11

Rsync Results Time used : 280975 s n Number of failures : 97 / 4872 n Average time to transmit : 231 s n Average Bandwidth : 15. 3 Kbits/s n 25/05/98 Stéphane Boyera, Luc Ottavj 12

Mftp Internet Results Time Used : 39620 s n Number of Failures : 692 / 4872 n Average Time to transmit : 33 s n Average Bandwidth : 109 Kbits/s n 25/05/98 Stéphane Boyera, Luc Ottavj 13

Mftp Satellite Results Time Used : 22414 s n Number of Failures : 469 / 4872 n Average Time to transmit : 18 s n Average Bandwidth : 196 Kbits/s n 25/05/98 Stéphane Boyera, Luc Ottavj 14

Conclusion Mftp through Internet : Not Usable n Rsync : Surprisingly good n – The most reliable solution – Can fit with the actual need (servers and files change rate) – Not performant – Weak supervision needed – Few Hardware 25/05/98 Stéphane Boyera, Luc Ottavj 15

Conclusion (2) n Mftp + Satellite : not fully operational – High performance – Good Scalability : if number of servers increases n if traffic of the W 3 C site increases n if internet connectivity decreased n – Accurate supervision needed – Low Reliability : Operational Vs Experimental Uplink n Hardware problems with Comatlas card n – Price ? ? 25/05/98 Stéphane Boyera, Luc Ottavj 16