IDGF International Desktop Grid Federation NEW TRENDS IN

IDGF International Desktop Grid Federation NEW TRENDS IN DESKTOP GRID COMPUTING Mikhail Posypkin Centre for Distributed Computing IITP RAS 1

PRESENTATION PLAN § Intro to Desktop Grid Computing § Tools and technologies in DG computing § Russian Chapter of IDGF DEGISCO WP 4 19/03/2018 2 2

WHAT “DESKTOP GRIDS” ARE q Use free CPU cycles q Volunteer computing q. Use master-slave model of distributed computation q Major technologies: § BOINC § XTREMWEB-HEP § OURGRID § CONDOR § X-COM § SARD 3

DESKTOP GRIDS BENEFITS STRONG POINTS WEAK POINTS q. Low-cost: DGs are leaders in terms of Flops/$ q. Ease of setup and maintenance q. Reuse of existing resources q. DGs are “Green” 4 q. Restricted area of application: coarse grain master-slave q. Complex technology q. Low w. r. t. supercomputers efficieincy NOT a REPLACEMENT for SUPECOMPUTERS/SERVICE GRIDS/ CLOUDS

BOINC: Berkeley Open Infrastructure for Network Computing § § § BOINC is an open source middleware system for deploying a desktop “grid” based on volunteer computing. Originally created from the SETI@home project by David Anderson from University of California, Berkeley. Now has over 22 full scale production projects and dozens of alpha projects exist Projects are based on different scientific disciplines BOINC has over 538, 354 active computers (hosts) worldwide processing on average 7, 777. 280 Tera. FLOPS TFLOPS (June 28, 2014) DEGISCO WP 4 19/03/2018 5 5

BOINC 6

BOINC CREDIT SYSTEM The project's server keeps track of how much work your computer has done. This is called credit. To ensure that credit is granted fairly, most BOINC projects work as follows: • Each task may be sent to two computers. • When a computer reports a result, it claims a certain amount of credit, based on how much CPU time was used. • When at least two results have been returned, the server compares them. If the results agree, then users are granted the smaller of the claimed credits. 7

BOINC Client 8

BOINC Server 9

BOINC App Specific Daemons assimilator (handles correct result) work generator (creates stream or batches of jobs) validator (compares replicas, selects “correct” result) BOINC 10

BOINC DEVELOPMENT TOOLS § § § C++ API – BOINC native API – DC-API WRAPPERS – BOINC Wrapper – Gen. Wrapper VIRTUALIZED WRAPPERS – BOINC VBox WRAPPER – GBAC DEGISCO WP 4 19/03/2018 11 11

BOINC Native API (BOINC team) WU Generator Validator Assimilator BOINC API WU Generator: - work unit generation, submission - input and output file handling Validator: - validator frameworks (simple, low-level) Assimilator: - handler function linked with a special program Grid System BOINC API Client Application 12 Initialisation Input and output file identification I/O wrappers Checkpointing Progress, credit reporting Critical sections Atomic file update Time handlers Graphics API (screen savers, icons) Other APIs (status, diagnostics, etc. )

DC-API (SZTAKI team, Hungary) Master Application DC-API Common features: Configuration, logging, Grid System DC-API Client Application 13 Initialisation Work unit generation Input and output file handling Work unit submission Event processing (e. g. results) Result assimilation Work unit deletion Initialisation Input and output file identification Event processing (e. g. checkpointing) Progress reporting

BOINC Wrapper (BOINC team) Without the Wrapper BOINC core client With the Wrapper BOINC core client Executes the legacy application(s) in a specific order BOINC application compiled with the BOINC libraries 14 Wrapper Legacy Application

Gen. Wrapper technology (SZTAKI team) Acts as the BOINC client application 1. Downloads BOINC core client 2. Downloads Can do almost anything that a shell script can (e. g. can start the legacy application) Application 3. Extracts 4. Launches Git. Box GBAC Launcher Work unit Application script A ZIP file (optional) 5. Executes Profile script Input files Legacy application Output files Gen. Wrapper binaries 7. Produces

Gen. Wrapper technology (SZTAKI team) Acts as the BOINC client application 1. Downloads BOINC core client 2. Downloads Can do almost anything that a shell script can (e. g. can start the legacy application) BINARY COMPATIBILITY ISSUE GBAC Application 3. Extracts 4. Launches Git. Box Launcher Work unit Application script A ZIP file (optional) 5. Executes Profile script Input files Legacy application Output files Gen. Wrapper binaries 16 7. Produces

VBox. Wrapper (BOINC team) § Developers work in environment of choice § App consists of – compressed minimal VM image (same across multiple apps) – executable – BOINC “Vbox wrapper” § Volunteers need to install Virtual. Box DEGISCO WP 4 19/03/2018 17 17

VBox. Wrapper BOINC client Vbox wrapper VM instance shared directory: executable input, output files 18

GBAC : Generic BOINC Application Client Developed by SZTAKI ( Hungary ) 19

VBox. Wrapper advantages § Advantages – Portability issues solved (no “mininal” linux is needed) – Safety is almost absolute – No special checkpointing is needed § Drawbacks – Virtual Box should be installed by volunteers – Virtual appliances load network traffic DEGISCO WP 4 19/03/2018 20 20

BOINC on mobile devices

Andro. BOINC: Remote control tool for BOINC clients This light-weighted manager for BOINC clients: - can connect to BOINC core client via Wi. Fi or mobile network - displays status of client: projects, tasks, transfers and messages - provides basic controls for client Pavol Michalec https: //code. google. com/p/androboinc/ 22

Native. BOINC for Android Native. BOINC is Android version of the BOINC client. This software moves BOINC to mobile device's world. Main application is fully-fledged BOINC manager (based on Andro. BOINC), which can controls BOINC client on the Android OS. This version provides manager, BOINC client and especially optimized applications for ARM processors. Mateusz Szpakowski (Poland) http: //nativeboinc. org/site/uncat/start 23

BOINC on Android (by Berkeley team) • BOINC harnesses the unused computing power of your Android device and runs jobs for scientific research projects. • BOINC computes only when your device is plugged in and charged, so it won't run down your battery. • It transfers data over Wi. Fi, so it won't use up your cell phone plan's data limit. 24

Interoperability among Desktop Grids and other HPC resources § System-level interoperability – 3 G-Bridge – Clubo. RUN § Application-level interoperability – g. USE/WS-PGRADE DEGISCO WP 4 19/03/2018 25 25

3 G-BRIDGE BOINC Service grid or Cloud 26

Clubo. RUN: use idle supercomputer cycles § Supercomputers also have ‘idle cycles’ – free rooms in a queue § Periodic script queries the queue and submits BOINC jobs to cluster if there is a room for it § MPI wrapper for BOINC jobs is used to run multiple clients on the cluster IDSTU RAS, IITP RAS, volunteers DEGISCO WP 4 19/03/2018 27 27

WS-PGRADE ARCHITECTURE (SZTAKI) Graphical User Interface: WS-PGRADE Workflow storage Workflow Engine Meta-broker g. USE information system Submitters File storage Application repository Logging Local resources, service grid VOs, Desktop Grid resources, Web services, Databases

Desktop Grid infrastructure on Demand

The activity of Russian Chapter of International Desktop Grid Federation 30

http: //desktopgridfederation. org What is IDGF? DEGISCO WP 4 19/03/2018 31 31

http: //desktopgrid. ru What is Russian Chapter of IDGF? 32

RUSSIAN CHAPTER’S BOARD 33

GOALS § Communication, Dissemination, Community management § Joint Research activity § Collaboration with volunteers’ community § International collaboration DEGISCO WP 4 19/03/2018 34 34

Communication, dissemination, and Community management § Tutorials: GRID’ 2010, IT-EDU’ 2010, DICR’ 2010, PAVT’ 2011, GRID’ 2012, GRID’ 2014 § Special tracks: GRID’ 2010, GRID’ 2012, GRID’ 2014, NSCF’ 2013 § Scientific publications and presentations DEGISCO WP 4 19/03/2018 35 35

PRESS RELEASES § Андреев А. Л. , Манзюк М. М. , Ватутин Э. И. Весь мир как суперкомпьютер // Троицкий вариант наука. № 16 (110) от 14 августа 2012. § Ивашко Е. Е. Выскопроизводительные вычисления в Desktop. Grid // Суперкомпьютеры 2013 § Посыпкин М. А. Как стать компьютерным донором? // Научная Россия 2013 DEGISCO WP 4 19/03/2018 36 36

BOINC-FAST in Karelia 2013 BOINC-FAST in Karelia 2015 is coming! 37

Joint Research § IITP RAS provides support for SAT@home, for new team from Crimea § Clubo. RUN (joint project of IDSTU RAS, IITP RAS and volunteers) § IITP RAS donates unused computing power for volunteer computing DEGISCO WP 4 19/03/2018 38 38

Collaboration with Volunteers’ Community § Promoting and getting resources for scientific projects § Testing scientific projects § Feedback § Joint events § Joint publications DEGISCO WP 4 19/03/2018 39 39

COMMUNITY STUDY Details: Dr. Ilya Kurochkin, Prof. Vladimir Yakimets RESEARCH OF PREFERENCES OF PARTICIPANTS OF THE VOLUNTARY DISTRIBUTED COMPUTING, today, 18. 00 at Desktop Grid section 40

Volunteer’s motivations 27% Спортивный интерес 51% Причастность к научным открытиям Хороший круг общения Хочу помочь науке 10% 93%

International Collaboration § Invited talks to BOINC-FAST (Robert Lovasz, David Anderson) § Invited talks to GRID’ 2010, GRID’ 2012, GRID’ 2014 by Robert Lovasz, Gabor Terstzyanski, Attila Csaba Marosi § Joint projects – FP-7 DEGISCO project (completed) – RSCF project under supervision of Miklos Kozlovzsky submitted DEGISCO WP 4 19/03/2018 42 42

DEGISCO FP 7 PROJECT Main objective: Expand European DCIs into ICPC countries by supporting the creation of new Desktop grids for e-Science in those countries and in Europe and by connecting them using the EDGe. S bridge technology. Support applications on this expanded infrastructure, disseminate, promote and provide training about this expanded infrastructure and its usage. DEGISCO WP 4 19/03/2018 43 43

RSCF Project § Combined distributed computing infrastructure, mathematical models, methods and tools for fundamental and applied resource-consuming research problems § Partners: SZTAKI (Hungary), IITP RAS, KAR RC RAS, IDSTU RAS DEGISCO WP 4 19/03/2018 44 44

Welcome to our section! Today, 14: 30, room 407 DEGISCO WP 4 19/03/2018 45 45

IDGF International Desktop Grid Federation http: // desktopgridfederation. eu http: //edgi-project. eu Globe adapted from http: //upload. wikimedia. org/ wikipedia/commons/f/fa/ Globe. svg 46

DESKTOP GRID ACTIVITY IN RUSSIA q. ISA RAS: OPTIMA@home, DG technology, tools for performance monitoring, dissemination and user support q. ISDCT SB RAS: SAT@home q. KRC RAS: Useful tools (BOINC portal), DG based data mining, dissemination q. BOINC. RU – Russian team of BOINC Volunteers q Other teams not mentioned here… 47

ISA RAS Desktop Grid Infrastructure BOINC Educational Infrastructure boinc-edu. isa. ru BOINC Test Infrastructure boinc-test. isa. ru üBased on SZTAKI Desktop Grid Package DEGISCO WP 4 19/03/2018 48 48 BOINC Production Infrastructure boinc. isa. ru, sat. isa. ru

The Typical Pass Lots of (student) projects DEGISCO WP 4 19/03/2018 Netmax 49 49 OPTIMA@home SAT@home

OPTIMA@home § Project OPTIMA@home - a research project that uses Internet-connected computers to solve challenging large-scale optimization problems http: //boinc. isa. ru/dcsdg/ § Institute for Systems Analysis of RAS, M. Posypkin, N. Khrapov DEGISCO WP 4 19/03/2018 50 50

More than 800 active hosts connected 51

Communication, dissemination, and Community management § § Exhibitions – Softool’ 2010 – Softool’ 2011 – SIMO Network(Spain)’ 2011 Talks 1 st year: IT-EDU’ 2010, DICR’ 2010, PAVT’ 2011 2 nd year: – IT-EDU’ 2011 (regular talk) – PAVT’ 2012 (regular talk) – GRID’ 2012 (regular and plenary + Desktop Grid Track) DEGISCO WP 4 19/03/2018 52 52

Communication, dissemination, and Community management § Courses – Moscow State University – Moscow Institute for Physics and Technology § Localized material – Posters – Flyers – Russian Chapter of IDGF site: desktopgrid. ru DEGISCO WP 4 19/03/2018 53 53

SAT@home: Most Successful BOINC Project in Russia Project SAT@home – research project aimed at solving hard combinatorial problems reduced to SAT problems http: //sat. isa. ru/pdsat/ Institute for System Dynamics and Control Theory of Siberian Branch of Russian Academy of Sciences (ISDCT SB RAS) A. Semenov, O. Zaikin DEGISCO WP 4 19/03/2018 54 54

SAT@home 55

SAT@home: More than 5500 hosts connected 56

SAT@home: Finds Useful Solutions 57

KRC RAS: BOINC BASED DATA MINING Based on Partition algorithm , the special BOINCbased software has been developed in Highperformance Data Center of Karelian Research Centre, RAS (Hp. DC) for discovering associative rules in large data sets. The software performs three stages of data processing with two points of synchronization. The software includes a client's software and the following developed BOINC's services: work generator, scheduler and assimilator. Also it uses the single My. SQL database to store workunits. 58

OPTIMA@home

DEGISCO FP 7 PROJECT Main objective: Expand European DCIs into ICPC countries by supporting the creation of new Desktop grids for e-Science in those countries and in Europe and by connecting them using the EDGe. S bridge technology. Support applications on this expanded infrastructure, disseminate, promote and provide training about this expanded infrastructure and its usage. DEGISCO WP 4 19/03/2018 61 61

International Desktop Grid Federation § § § The International Desktop Grid Federation (IDGF) brings together organisations and persons interested in Desktop grid computing and related technologies. Focuses on application developers and Desktop grid operators. IDGF is a member organisation that provides services for its members (legally: participants). IDGF brings them together, so they can get better information, documentation and training, a platform to exchange experiences. IDGF serves as a marketing channel for its members DEGISCO WP 4 19/03/2018 62 62

The Goals of Russian Chapter § § § Help Russian research and industry teams to develop, deploy and maintain their desktop-grid distributed applications Maintain DCI infrastructures for our projects and for other organizations Provide courses and tutorials on desktop grids and volunteer computing for Russian academic and industrial community Promote Russian BOINC projects Explain the mission of IDGF and benefits of the membership DEGISCO WP 4 19/03/2018 63 63

The Future of Russian Chapter § § Maintain and improve Web-site desktopgrid. ru Dissemination activity: • Tracks • Tutorials • Courses • Papers Create “Umbrella” project for Russian BOINC applications International collaboration DEGISCO WP 4 19/03/2018 64 64

The Future of Russian Chapter § Joint Activity (development) • BOINC Web-portal (easy application deployment and job submission) • Scientific Gateways • Supporting interoperability: Bridges • Messaging service for BOINC • Alternative DG systems • …. DEGISCO WP 4 19/03/2018 65 65

66

WP 2: Service grid infrastructure EGI Grid site: RU-ISA-CGTDC 67 http: //grid. isa. ru

WP 3: Application and User Support § ISA RAS support local developers from – Institute for System Dynamics and Control Theory of Siberian Branch of Russian Academy of Sciences – Moscow Institute for Electronic Technology (MIET) – Joint Institute for Nuclear Research DEGISCO WP 4 19/03/2018 68 68

WP 2: 3 G-Bridge Infrastructure 3 gbridge. isa. ru 69