When DBs met the GRID The Grid Data

When DBs met the GRID The Grid Data Source Engine Dr. Giuliano Taffoni (Ph. D. )

The DMBS Problem Scientific communities use DB for data (Astronomical Archives, Bioinformatics data. . . ) Metadata problem No native access up to GT 3. 9 (OGSA-DAI) GT 4 (WS)

What is a DMBS? A database is simply a bunch of information (data) stored on a computer. DB management system is the service that allow to interact with those information Relational databases consist of tables of data with clearly defined columns.

RDMS Data is presented as a collection of relations Each relation is depicted as a table Columns are attributes Rows ("tuples") represent entities Every table has a set of attributes that taken together as a "key" (technically, a "superkey") uniquely identifies each entity

Software computing machines DBMS <=> software computing machine Memory model Filesystem (table space) data processor Language SQL

SQL Language to access and manipulate DB system Accept logical and math operators You can ask a DB to make simple and complex operations (stats) Language: SELECT, INSERT WHERE, AND, etc.

Meta-computing on the GRID is able to execute binary code! It exists a different type of computing: the virtual machines (ex. Java WM)

Grid Abstract Computing Machine Focus on semantic of the Grid Formal Methods (ASM) Grid assumes a virtual pool of resources: (CPU cycles + Memory) No theoretical limit: Grid can operate with a wide range of resources “A Formal Framework for Defining Grid Systems” Zsolt N. Nemeth & Vaidy Sunderam 2 nd IEEE/ACM (CCGRID'02)

Grid capabilities Extending “job submission process”: dynamic Java class loading functional resolution inference and reasoning query evaluation

Extending the Grid capabilities Theoretical approach Provide a semantic extension of Grid ASM to verify if it is possible to extend Grid capabilities Provide a suitable architectural definition for the Grid meta-computing functions

Extending the Grid capabilities Applicative view Provide an integration Layer within the jobmanager Provide a proper extension of the IS to monitor new resources Security GSI: no need to extend but to use!

A first attempt: DSE

Grid DSE project Architectural analysis of Grid middleware Architectural analysis of DSE Conceptual mapping between DSEs & Grid Resource Framework layer: represent DSE through the Grid resource abstraction.

Grid DSE goal New calculous capabilities to a Grid Node: Inference, Query, Reasoning. . . New fabric element for the Grid: Query Element (analogous of CE) New IS

QE integration

GDSE JM details Grid Data Source Engine GRAM Protocol GASS Globus XIO Job Manager ODBC Manager Local Resource Manager DSE Instance Man ODBC Driver DSE Instance Worker Node ODBC Driver Catalog drv. User DB DSE Internal DB Meta Machine

GDSE & GIS Grid Data Source Engine GRAM Protocol GASS Job Manager Globus XIO Grid info system ODBC Manager Local Resource Manager ODBC Driver DSE Instance ODBC Driver Catalog drv. snmpd DSE Instance Man LDAP snmpd GANGLIA Worker Node User DB MDS DSE Internal DB Meta Machine

QE vs CE

Added values QE is analogous to CE QE is inside the Grid monitored by the IS easy to included in the WMS VOMS

GDSE in practice Globus GT 2. 4. 3 (VDT) + globus patch ODBC & JDBC Perl DBD a DBMS: any one supporting ODBC

GDSE security GSI+VOMS: VO + Groups + Roles; Mapping VOMS cert with site user account; Mapping local user account with DB user User certificate => DB accounting

DB accounting GRANT user: DB access DB managing Table/role access User accounting

GDSE demo usage GDSE accounting GDSE DB administration GDSE in the GRID: BDII usage Load balancing Computing and GDSE

GDSE accounting Site user DMBS (postgresql) inafdbadmin postgres inafdbuser inafdbmanager

Simple Biomedical DB Biomed Private Code Family_name medinfo Name Address Occupation Telephone Code disease Hospital

GDSE usage Uses globus-job-run resource SQL-STATEMENT CONNECTOR DBNAME Ex. globus-job-run grid 006 “select * from uno; ” ODBC TEST SQL

Create table as dbadmin

Insert into table

Select

Drop, vista & transaction

Adding new user

Job submission

GDSE and BDII

Meta computing Made the DB make some calculation for you DB has scientific/statistical functions Sometimes simply extracting some data is useless and time consuming

Astronomical example GSC 22 Star catalogue Huge table: position, magnitude, etc. Luminosity function on a sky area

Luminosity function User LF plot

Luminosity function

Luminosity Function Simple example of Workflow that uses GDSE and WNs

Parallel access to DB select duroc GDSE stop GDSE

GDSE and Metadata Astronomical example FITS files: Metadata + data SIMPLE = BITPIX = NAXIS 1 = NAXIS 2 = T / file does conform to FITS standard 16 / number of bits per data pixel 2 / number of data axes 4 / length of data axis 1 3 / length of data axis 2 123 515 Header == metadata 456 986 345 869 1321 45 84 data

TTYPE 1 = 'signal ' / label for field 1 TFORM 1 = '1024 E ' / data format of field: 4 -byte REAL TUNIT 1 = 'unknown ' / physical unit of field EXTNAME = 'xtension' / name of this binary table extension PIXTYPE = 'HEALPIX ' / HEALPIX pixelisation ORDERING= 'RING ' / Pixel ordering scheme, either RING or NESTED NSIDE = 128 / Resolution parameter for HEALPIX FIRSTPIX= 0 / First pixel # (0 based) LASTPIX = 196607 / Last pixel # (0 based) INDXSCHM= 'IMPLICIT' / Indexing: IMPLICIT or EXPLICIT COMMENT ------------COMMENT POINTING CHARACTERISTICS COMMENT ------------PT_MODEZ= 'gaussian' / the z-axis pointing mode COMMENT ----------PT_SIGZ = 5. 00000 E-01 / [arcmin] mean z-axis pointing error COMMENT Cosmological parameters PT_DZMAX= 2. 00000 E+00 / [arcmin] maximum z-axis pointing error COMMENT ----------PT_MODEP= 'ideal ' / the initial scan phase mode OMEGAB = 0. 05 / Omega in baryons PT_MODER= 'gaussian' / the rotation rate mode OMEGAC = 0. 95 / Omega in dark matter PT_SIGR = 2. 0943951024 E-03 / [rad s**(-1)] mean rotation rate error 0. 00 / Omega in cosmological constant OMEGAV = PT_DRMAX= 2. 0943951024 E-03 / [rad s**(-1)] maximum rotation rate error / Omega in neutrinos OMEGAN = 0. 00 COMMENT ------------HUBBLE = 50. 00 / Hubble constant in km/s/Mpc COMMENT SCAN STRATEGY PARAMETERS NNUNR = 0. 00 / number of massive neutrinos COMMENT ------------NNUR = 3. 04 / number of massless neutrinos SS_MODE = 'cycloidal' / overall type of scan strategy = TCMB 2. 7260 / CMB temperature in Kelvin SS_PMODE= 'followsun' / azimuthal scanning HELFRACT= mode 0. 24 / Helium fraction SS_T 0_I = 1. 1991888000 E+09 / [s since 1970. 0] scan reference time: int 0. 00 / reionisation optical depth OPTDLSS = SS_T 0_F = 0. 00000 E+00 / [s since 1970. 0]IONFRACT= scan reference time: frac / ionisation fraction 0. 20 SS_TH_Z 0= 9. 00000 E+01 / [deg] fiducial scanning colatitide SS_DTH_Z= 7. 00000 E+00 / [deg] fiducial scanning colatitide variation SS_PROT = 6. 00000 E+01 / [s] rotation period of the satellite SS_PPT = 3. 600000 E+03 / [s] pointing period of scan strategy SS_PREPT= 3. 600000 E+03 / [s] repointing period of scan strategy SS_PMOT = 1. 5778800000 E+07 / [s] period of the main scan motion SS_PHASE= 1. 4323944878 E+01 / [deg] initial phase of main scan motion TH_SE_ST= 'warning ' / status of Sun/Earth aspect angle violations TH_S_MAX= 1. 00000 E+01 / [deg] maximum Solar aspect angle TH_E_MAX= 1. 500000 E+01 / [deg] maximum Earth aspect angle SS_N_PT = 8784 / number of pointing periods

Metadata DBMS GUID -> to all metadata Query DB: Globus-job-run Locate files: GUID Get files: lfc Make computation