Grid File System WG GGF 17 Arun

Grid File System WG – GGF 17 Arun Jagatheesan San Diego Supercomputer Center GGF 17 May 11, 2006 Tokyo, Japan

IP & © Intellectual Property Statement The GGF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the GGF Secretariat. The GGF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this recommendation. Please address the information to the GGF Executive Director. Copyright (C) Global Grid Forum (2004). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the GGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the GGF Document process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the GGF or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE GLOBAL GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. " Global Grid Forum-17, Tokyo 2 Grid File System Group

Talk Outline • Our original charter • Status • What we reported in GGF-16 • What we have now at GGF-17 • What we will have for GGF-18 • GFS Architecture • What, why, how… • Summary Global Grid Forum-17, Tokyo 3 Grid File System Group

Grid File System WG • Standard mechanism to describe and organize file-based data in grids • Provide a higher level, human readable logical namespace of data and resources in a data grid • An architecture that would allow vendors and users to distinguish compatible products that are “GGF GFS complaint” Global Grid Forum-17, Tokyo 4 Grid File System Group

Example: GFS Namespace GFS Service /grid ggf aist jp gtrc file 2 file 1 gsiftp: //aist. go. jp/ /a/b/file 3 /b/file 1 file 3 file 2 pub http: //u-tokyo. ac. jp/ ftp: //meti. go. jp/ File Transport Service /a/b/file 2 /d/file 1 /a/b/file 1 /c/file 3 /pub/ Global Grid Forum-17, Tokyo Hierarchical namespace with filesystem metadata Unique path name Access control Write lock Other metadata 5 Grid File System Group

GFS-WG Status reported in GGF-16 • Informational Document : GFS-Architecture • Approved now as a GFD-61 • http: //www. ggf. org/documents/GFD. 61. pdf • That was only version – 1 of the Architecture workbook • RNS Specification • Under discussion, public review (? ) • More discussion on this session too • New Secretary • Christopher Jordon • External Alliances • SNIA, IETF NFSv 4 (? ? ) Global Grid Forum-17, Tokyo 6 Grid File System Group

GFS-WG Status as of GGF-17 (NOW) • Informational Document : GFS-Architecture • Approved now as a GFD-61 (http: //www. ggf. org/documents/GFD. 61. pdf) • Working on next version with end-to-end architecture • Focus will be on “GFS complaint software” for vendors and users • RNS Specification • Re-factored version of RNS document - public discussion • GFSG and community to decide whether incremental progress can be made on this initial draft or wait for community consensus. • External Alliances • SNIA – Currently a group here in GGF. Look forward to working with our SNIA friends. • IETF NFSv 4 : We had initial discussions. But, no follow up was made by our GFS as we still need to know more (and less resources to spread out). Global Grid Forum-17, Tokyo 7 Grid File System Group

What will we have by GGF – 18 ? • Proposed Deliverables • • RNS spec finalized OGSA - GFS use case GGF Architecture Workbook version 2 GGF Architecture Standard document – Skeleton • • • Topic not limited to… Data Grid Composition of services System and User-defined Meta-data in GFS Profile Why and what “GGF GFS complaint” product or service • Work with our Storage friends in GGF and SNIA Global Grid Forum-17, Tokyo 8 Grid File System Group

Talk Outline • Our original charter • Status • What we reported in GGF-16 • What we have now at GGF-17 • What we will have for GGF-18 • GFS Architecture • What, why, how… • Summary Global Grid Forum-17, Tokyo 9 Grid File System Group

GFS Grid Resources • Context (Information) • Information about digital entities (location, size, owners, . . ) • Relationship between digital entities (replicas, collection, . ) • Behavior of the digital entities (services) • Content (Data) • Structured and unstructured • Virtual or derived • Commodity (Producers and consumers) • Physical storage resources and file systems • Logical resources Global Grid Forum-17, Tokyo 10 Grid File System Group

GFS Resource Providers (GRP) providing content and/or storage GRP /txt 3. txt Global Grid Forum-17, Tokyo 11 Grid File System Group

GFS Administrative Domain • GFS Administrative domain with one or more GFS Resource Providers • Could include their data centers • SNIA note: This is at very higher level than block, file/record level. Yet deals with objects Research Lab GRP /txt 3. txt Global Grid Forum-17, Tokyo 12 Grid File System Group

GFS Administrative domains Research Lab data + storage (10) Storage-R-Us Resource Providers data + storage (50) GRP GRP /…/text 1. txt Finance Department data + storage (40) GRP GRP /…//text 2. txt Global Grid Forum-17, Tokyo 13 GRP /txt 3. txt Grid File System Group

Grid File System /home/arun. sdsc/exp 1/text 1. txt /home/arun. sdsc/exp 1/text 2. txt /home/arun. sdsc/exp 1/text 3. txt data + storage (100) Research Lab data + storage (10) Storage-R-Us Resource Providers data + storage (50) GRP GRP /…/text 1. txt Logical Namespace (Need not be same as physical namespace ) Finance Department data + storage (40) GRP GRP /…//text 2. txt Global Grid Forum-17, Tokyo 14 GRP /txt 3. txt Grid File System Group

GFS Client Server Software Stack /home/arun. sdsc/exp 1/text 1. txt /home/arun. sdsc/exp 1/text 2. txt /home/arun. sdsc/exp 1/text 3. txt data + storage (100) Traditional FS clients Users see the same namespace. But, the functionalities and data types supported may differ GFS WS-Client GFS Client Protocol GFS Server (code) /…/text 1. txt Global Grid Forum-17, Tokyo /txt 3. txt 15 Grid File System Group

GFS Architecture Standard Interfaces (? ) Is the user interface to access GFS a standard one? Traditional FS clients GFS WS-Client GFS Server (code) /…/text 1. txt Global Grid Forum-17, Tokyo GFS Client Protocol Is the server interface by each vendor a standard? (This will allow server-toserver peer communication /txt 3. txt 16 Grid File System Group

Questions for GFS Architecture • Logical resource namespace • Logical resources have been popular • Work with existing standards • Without loosing advanced functionalities • Vendor interoperability • Single interface • Composed from others (services) • A single interface of all functionalities (out of the box) • Block level protocols • Bulk operations, data manipulation, … Global Grid Forum-17, Tokyo 17 Grid File System Group

World Wide Datagrid Global Grid Forum-17, Tokyo 18 Grid File System Group

What could be done? • Standards with functionalities or operations used by users in production • Collaborative logical namespace management • Inter/Intra/Multi Organizational namespace • Logical storage resources (no mount points) • Metadata, annotation support for discovery • Bulk operations • User groups and User access control • Automated data integrity checks Global Grid Forum-17, Tokyo 19 Grid File System Group

Suggestions, comments • Self Assessment • Need to restart bi-weekly telecon (at least monthly) • F 2 F discussions (or polycom) • More people to work or just provide input than be passive observers (especially from vendors, we got users) • Suggestions • Focus on what works and what is used in production • Products and standards emerge based on what works and user needs • Flexibility of the WG’s to refocus the charter along with other production groups or users [what is out there and required] Global Grid Forum-17, Tokyo 20 Grid File System Group

Summary • GFS-WG results delivered, has to evolve more • Data Grids : Multiple projects and production users world-wide • Multiple common features and patterns observed in academic usage • Vendor or Industry Interest for similar commercial use cases (work with them also) • GFS-WG more deliverables • Conference calls, Architecture, re-factored documents Global Grid Forum-17, Tokyo 21 Grid File System Group

Acknowledgement RNS, Active Contributors: • Arun Jagatheesan, SDSC • Christopher Jordon, SDSC • Manuel Pereira, IBM Almaden Research Center • Osamu Tatebe, University of Tsukuba Architecture Contributors: • Ted Anderson (IBM Almaden) • Cameron Bahar (Storage Machines) • Leo Luan (IBM Almaden) • Reagan Moore (SDSC) • Ken Wood (Hitachi Data Systems) • Jane Xu (IBM Storage Software) • Alan Yoder (Network Appliance) Global Grid Forum-17, Tokyo 22 Grid File System Group