Web FTS File Transfer Web Interface for FTS

Web. FTS File Transfer Web Interface for FTS 3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing IT-SDC : Support for Distributed Computing

Overview § The FTS 3 service § Web. FTS features § Web. FTS cloud integrations § Dropbox § CERNBox § Ongoing development § Dropping X 509 § Data management operations IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 2

What is Web. FTS? § Web based tool to transfer files between grid/cloud storages § Modular protocol support § gsiftp, http(s), xrootd and srm § Cloud extensions: dropbox, CERNBox § Development funded by IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 3

Provide access to leading technology Based on FTS 3 is the service responsible for distributing the majority of LHC data across the WLCG infrastructure Low level data movement service, responsible for moving sets of files from one site to another while allowing participating sites to control the network resource usage Used by LHC VOs + many others VOs part of EGI ~20 PB monthly transfer volume / ~2. 2 M files per day (WLCG) http: //dashb-fts-transfers. cern. ch/ui/ IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 4

Web. FTS architecture BROWSER WEBFTS. js REST API FTS 3 GFAL 2 DAVIX IT-SDC GSIFTP DROP BOX Web. FTS: File Transfer Web Interface for FTS 3 … 17/11/2014 5

Security § Simpler access while keeping the same level of security § X 509 + Oauth for Dropbox § Transparent delegation of credentials § Avoid storing or transferring any sensitive data § Open access to all source code § All sensitive information is used within the browser and forgotten IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 6

Delegation § Delegation is needed to let Web. FTS access the grid on users behalf § Users make private key available to browser § Not available via browser API § VOMS extensions acquired by the service on users behalf § Why it’s important § Gives the users a service which can access the grid for them, from a browser, with full VOMS credentials IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 7

Additional Features § Check-summing and file overwriting § Possibility to resubmit transfer jobs or only-failed files transfers. § Storage Endpoints Auto-completion § For endpoints published on the BDII (EGI and WLCG Information System) § Support for LFC Registration § File catalog developed at CERN and used by EGI and WLCG IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 8

Success Stories § Web. FTS has been successfully tested to transfer from/to: § EUDAT B 2 Stage ( i. RODS DSI) § Any gsiftp/webdav/xrootd aware grid storage ( DPM, d. Cache, Castor, EOS, Storm) § HPC Titan @ Oak Ridge National Lab (ongoing) § https: //www. olcf. ornl. gov/titan/ § Under evaluation by LHCb IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 9

Landing page and Guided-tour IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 10

Credential delegation E IV S T SI N SE ATIO TED O T ER ORM SMI Z F N IN RA T IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 11

Transfer interface IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 12

Job status interface IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 13

Extension for Dropbox § Nice way import/export data from the grid world § Avoid the installation of new software and uses what the user has already installed § Zero development of clients § Multiplatform is given for free § Integration with Oauth § By delegating to FTS the right to interact with dropbox on users behalf § Achieved using web tech § Which requires the interactivity of a browser IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 14

Extension for Dropbox IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 15

Dropbox plugin § Server side the development of a plugin for the metadata management and I/O operations was needed: § FTS REST integrates the plugin to perform metadata management operations § FTS 3 server uses the plugin to perform the transfers: § Grid. FTP <-> dropbox § Http(s) <-> dropbox IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 16

CERNBox integration § While Dropbox has been integrated via the implementation of a plugin for CERNBox we waited for the new version with EOS as backend ( CERNBox 2. 0) § We use EOS access via standard grid protocols ( e. g. xrootd) § We map user credentials to correct EOS namespace § The rest comes for free IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 17

Web. FTS With CERNBox IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 18

Web. FTS With CERNBox IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 19

Web. FTS With CERNBox IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 20

Web. FTS With CERNBox IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 21

Web. FTS With CERNBox IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 22

Ongoing developments: Access without X 509 How can we get rid of the delegation step? 1. An Identity Federation: edu. GAIN § To allow identity providers to authenticate users at their own institute (SSO) 2. A token translation service : STS § To ask the CA for a certificate for the users 3. An “IOTA” Certification Authority § To grant the short lived certificate 4. VOMS § To accept the new cert as a VO member IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 23

EDUGAIN § Built on existing federations and infrastructures § CERN participates in edu. GAIN via SWITCHaai § Many NRENs participate in edu. GAIN too IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 24

Security Token Service (STS) • An EMI service • SAML in, X 509/VOMS out IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 25

“IOTA” CA IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 26

VOMS admin IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 27

Architecture IOTA CA STS Id. P Web. FTS X. 509 VOMS SAML Credentials Attributes Redirect WAYF CERN SSO X. 509 VOMS Grid Storage Element Web Slide adapted from Romain Wartel, GDB Sept 2014 28

Pros/Cons § X 509 -free access to the grid infrastructure § With VOMS support § Without modifying all the services § Federated single sign on § One password to remember § Numerous services potentially accessible § But we need Site acceptance IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 29

Ongoing developments: Data Management § Not only Transfers. . § FTS REST API have been extended to support data management operations § Delete § Create/Remove folders § Rename § Under integration in Web. FTS IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 30

Links § Online service accessible: § https: //webfts. cern. ch try now! § User certificate in your browser § User guide, F. A. Q: § Online guided-tour § http: //fts 3 -service. web. cern. ch/documentation/webfts § Official support & code § fts-support@cern. ch § https: //github. com/cern-it-sdc-id/webfts IT-SDC Web. FTS: File Transfer Web Interface for FTS 3 17/11/2014 31

Questions? fts-support@cern. ch feedback IT-SDC ideas Web. FTS: File Transfer Web Interface for FTS 3 use cases 17/11/2014 32