ARCS Data Fabric Pauline Mak pauline. mak@arcs. org. au ARCS Data Services Pauline Mak
Outline • Introduction to the ARCS Data Fabric • Migration from SRB to i. RODS • Davis • Hermes Pauline Mak
ARCS Data Fabric Pauline Mak
Why SRB? • In June 2008, i. RODS lacked - Full GSI Authentication - Federation • Now migrating to i. RODS 2. 0 - Supports both GSI and federation - Open-sourced - Microservices and Rules Engine Pauline Mak
Migration • Modified migration script handles: - Users - SRB Objects and Collections - Metadata - Groups • Does not migrate - Permissions - User DNs Pauline Mak
Migration • Moving other processes from SRB to i. RODS - Automatic account creation - Usage statistics - User sync Pauline Mak
Automatic Account (SRB) MCAT Checks MCAT for user with the same DN srb. tpac. org. au (patched SRB server) If no account matches, calls external scripts to generate account User attempts to login with a valid certificate Pauline Mak
Automatic Accounts (i. RODS) 5. The server retrieves new username from script output (or query ICAT by DN) and logs in i. RODS Server 1. Client connects using a certificate 2. Fires the rule if the DN is unknown ac. Get. User. By. DN 3. Executes external script (Create. User) Pauline Mak ICAT 4. Create. User generates a new account
Shibboleth & SLCS 2. Forwards user to institution's Shibboleth login page Short Lived Certificate Server (SLCS) TPAC Identify Provider (Id. P) 4. Forwards result back to SLCS server 1. Connects to SLCS server to select Id. P 3. User enters Id. P username and password Pauline Mak 5. (If valid) Creates a certificate and sends it back to the user
Usage Scripts (SRB) srb. tpac. org. au srb. sapac. edu. au Each zone runs a script daily that will query MCAT for usage info Uploads usage XML file to a central location in SRB Central collection zone srb. hpsc. uq. edu. au My. SQL DB Pauline Mak Use Scommands to collect users and resource info, then ingest XML data into DB status. arcs. or g. au
Usage Scripts (i. RODS) srb. tpac. org. au srb. sapac. edu. au Each zone runs a script daily that will query ICAT for usage info Uploads usage XML file to a central location in SRB Central collection zone srb. hpsc. uq. edu. au Ingests XML data into DB My. SQL DB Pauline Mak status. arcs. or g. au
status. arcs. org. au Pauline Mak
Migration – Zone sync • We only sync users • SRB - Szonesync: executed hourly - Federation can lag for up to an hour • i. RODS - Executing rules immediately when a user is added Pauline Mak
Davis • Web. DAV • Supports SRB and i. RODS • Browser and client mode • Supports Shibboleth • Multi-threaded and resumable downloads • Simplifies access Pauline Mak
Davis Architecture SRB/i. RODS Jargon API Browser Mode Handler Web Browser Authentication Processor Web. DAV Handler Web. DAV Client Pauline Mak SLCS Server Id. P
Davis – Modes Pauline Mak
Davis - Permissions Pauline Mak
Davis - Metadata Pauline Mak
Hermes/commons-vfs-grid • Originally from JCU (ARCHER project) – Mathew Wyatt • Commons-vfs - local, SFTP, Web. DAV • Commons-vfs-grid: - grid-related protocols - Grid. FTP contributed by David Meredith Pauline Mak
Hermes Architecture S/FTP Local proxy SLCS + Shib My. Proxy SRBi. RODS Grid. FTP JSCH Jargon Globus Protocol specific libraries GSIProxy. Manager commons-vfs(-grid) GSI Based Credentials Account File. Object Connectors Pauline Mak
Hermes – Browser Pauline Mak
Hermes - Permission Pauline Mak
Hermes - Metadata Pauline Mak
Hermes – metadata search Pauline Mak
Questions? Pauline Mak
Links • http: //www. arcs. org. au (ARCS) • http: //www. tpac. org. au (TPAC) • http: //projects. arcs. org. au/trac/davis/ (Davis) • http: //projects. arcs. org. au/trac/commons-vfs-grid/ (Hermes) • http: //projects. arcs. org. au/trac/systems/wiki/Data. S ervices/SRB • http: //projects. arcs. org. au/trac/systems/wiki/Data. S ervices/i. RODS Pauline Mak
Скачать презентацию ARCS Data Fabric Pauline Mak pauline mak arcs org
902b978aaad9730575038fa40d733f31.ppt