Enabling Grids for E-scienc E ATLAS DDM Operations

Зарегистрируйтесь, чтобы просмотреть полный документ!

Enabling Grids for E-scienc. E ATLAS DDM Operations - III DPM at T 2’s Jiří Chudoba ATLAS meeting, 25. 9. 2007, CNAF www. eu-egee. org INFSO-RI-508833

DPM at T 2’s Enabling Grids for E-scienc. E • FZK cloud: – FZU srm: //golias 100. farm. particle. cz/dpm/farm. particle. cz/home/atlas/ – CYF srm: //dpm. cyf-kr. edu. pl/dpm/cyf-kr. edu. pl/home/atlas/ • SARA cloud: – NIKHEF: srm: //tbn 18. nikhef. nl/dpm/nikhef. nl/home/atlas/ – IHEP: srm: //dpm 0001. m 45. ihep. su/dpm/m 45. ihep. su/home/atlas/ – PNPI: srm: //cluster. pnpi. nw. ru/dpm/pnpi. nw. ru/home/atlas/dq 2 • CNAF cloud: – Milano: srm: //grid 015. mi. infn. it/dpm/mi. infn. it/home/atlas/ – Roma 1: srm: //grid-cert-03. roma 1. infn. it/dpm/roma 1. infn. it/home/atlas/ – Napoli: srm: //t 2 -dpm-01. na. infn. it/dpm/na. infn. it/home/atlas/ – LNF: srm: //atlasse. lnf. infn. it/dpm/lnf. infn. it/home/atlas/ INFSO-RI-508833 ATLAS DDM Operations 2

DPM Enabling Grids for E-scienc. E • DPM documentation: – https: //twiki. cern. ch/twiki/bin/view/LCG/Data. Management. Documentation • Description of DBs on some wiki pages: http: //www. gridpp. ac. uk/wiki/DPM_My. SQL_database • DBs: cns_db, dpm_db • Tables from cns_db: Cns_file_metadata, Cns_file_replica, Cns_groupinfo, Cns_userinfo INFSO-RI-508833 ATLAS DDM Operations 3

ATLAS groups Enabling Grids for E-scienc. E Examples from DPM at golias 100 select * from Cns_groupinfo where groupname like '%atlas%'; +-------+----------------+ | rowid | groupname | +-------+----------------+ | 1309 | atlassgm | | 3 | 1308 | atlas | | 7 | 1307 | atlas/Role=lcgadmin | | 8 | 1311 | atlasprd | | 10 | 1310 | atlas/Role=production | | 16 | 116 | atlas/lcg 1 | | 18 | 118 | atlas/usatlas | | 21 | 121 | atlas/ca | | 30 | 2695 | atlas/de | | 32 | 2697 | atlas/nl | | 33 | 2698 | atlas/fr | | 35 | 2700 | atlas/usatlas/Role=production | +-------+----------------+ • Until this summer, we mapped all ATLAS users to gid=1307 INFSO-RI-508833 ATLAS DDM Operations 8

ATLAS users Enabling Grids for E-scienc. E select owner_uid, username, sum(filesize)/100000 as size_GB from Cns_file_metadata, Cns_userinfo, Cns_groupinfo where owner_uid=userid and Cns_groupinfo. gid=C ns_file_metadata. gid and Cns_groupinfo. groupname like "atlas%" group by owner_uid; +------------------------------------------------------------+-----+ | owner_uid | username |size_GB | +------------------------------------------------------------+-----+ | 101 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Guido Negri/Email=guido. negri@mi. infn. it | 414. 78 | | 102 | /DC=cz/DC=cesnet-ca/O=Institute of Physics of the Academy of Sciences of the CR/CN=Jiri Chudoba | 56. 85 | | 104 | /C=CH/O=CERN/OU=GRID/CN=Simone Campana 7461 - ATLAS | 21. 24 | | 106 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Silvia Resconi/Email=Silvia. Resconi@mi. infn. it | 95. 37 | | 108 | /C=CH/O=CERN/OU=GRID/CN= Jiri Kosina 6222 | 0. 00 | | 115 | /C=CA/O=Grid/OU=westgrid. ca/CN=Rodney Walker | 22. 54 | | 128 | /C=UK/O=e. Science/OU=Lancaster/L=Physics/CN=james catmore | 0. 00 | | 132 | /O=German. Grid/OU=LMU/CN=Gernot Krobath | 1. 89 | | 146 | /O=German. Grid/OU=LMU/CN=Cedric Serfon | 0. 10 | | 149 | /O=German. Grid/OU=LMU/CN=John Kennedy | 16. 40 | | 154 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Guido Negri 2 | 139. 78 | | 157 | /C=CH/O=CERN/OU=GRID/CN= Miguel De Oliveira Branco 2423 | 217. 00 | | 160 | /C=CA/O=Grid/OU=phys. uvic. ca/CN=Ashok Agarwal | 1. 12 | | 166 | /O=German. Grid/OU=LMU/CN=Johannes Elmsheuser | 0. 00 | | 189 | /C=PL/O=GRID/O=INP/CN= Andrzej Olszewski | 3. 10 | | 191 | /C=TW/O=AS/OU=PHYS/CN= suijian zhou/Email=suijian. zhou@cern. ch | 2. 29 | | 192 | /O=German. Grid/OU=DESY/CN=Andreas Haupt | 1. 11 | | 195 | /C=UK/O=e. Science/OU=Queen. Mary. London/L=Physics/CN=kondo gnanvo | 0. 00 | | 218 | /C=CH/O=CERN/OU=GRID/CN= Piotr Nyczyk 6217 - Atlas | 0. 00 | | 219 | /C=CH/O=CERN/OU=GRID/CN=David Cameron 0841 | 453. 53 | | 247 | /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=dcameron/CN=555105/CN=David Cameron | 0. 00 | | 248 | /C=PL/O=GRID/O=IFJ/CN=Anna Kaczmarska - Atlas | 0. 00 | | 250 | /C=IT/O=INFN/OU=Personal Certificate/L=Milano/CN=Davide Rebatto/Email=David. Rebatto@mi. infn. it | 44. 54 | | 258 | /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=mlassnig/CN=663551/CN=Mario Lassnig | 4741. 78 | | 263 | /DC=org/DC=doegrids/OU=People/CN=Sanjay Padhi 496075 | 155. 73 | | 318 | /C=IT/O=INFN/OU=Personal Certificate/L=Roma 1/CN=Alessandro De Salvo/Email=Alessandro. De. Salvo@roma 1. infn. it | 2. 12 | | 334 | /C=UK/O=e. Science/OU=Glasgow/L=Compserv/CN=caitriana nicholson | 2. 14 | | 358 | /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=sch/CN=664224/CN=Ariel Gustavo Schwartzman | 0. 03 | +------------------------------------------------------------+-----+ INFSO-RI-508833 ATLAS DDM Operations 9

ATLAS production users Enabling Grids for E-scienc. E select owner_uid, username, Cns_groupinfo. groupname, sum(filesize)/100000 as size_GB from Cns_file_metadata, Cns_userinfo, Cns_groupinfo where owner_uid=useri d and Cns_groupinfo. gid=Cns_file_metadata. gid and Cns_groupinfo. gid=1310 group by owner_uid; +------------------------------------------------------+------------+-----+ | owner_uid | username | groupname | size_GB | +------------------------------------------------------+------------+-----+ | 106 | /C=IT/O=INFN/OU= Personal Certificate/L=Milano/CN=Silvia Resconi/Email=Silvia. Resconi@mi. infn. it | atlas/Role= production | 6. 41 | | 115 | /C=CA/O= Grid/OU=westgrid. ca/CN=Rodney Walker | atlas/Role= production | 3. 56 | | 154 | /C=IT/O=INFN/OU= Personal Certificate/L=Milano/CN=Guido Negri 2 | atlas/Role= production | 14. 54 | | 250 | /C=IT/O=INFN/OU= Personal Certificate/L=Milano/CN=Davide Rebatto/Email=David. Rebatto@mi. infn. it | atlas/Role= production | 14. 85 | | 263 | /DC= org/DC=doegrids/OU=People/CN=Sanjay Padhi 496075 | atlas/Role= production | 71. 11 | +------------------------------------------------------+------------+-----+ select owner_uid, Cns_groupinfo. groupname, sum(filesize)/100000 as size_GB from Cns_file_metadata, Cns_userinfo, Cns_groupinfo where owner_uid=userid and Cns_groupinfo. gid=Cns_file_metadata. gid and Cns_groupinfo. gid=1310 group by owner_uid; +-----------------+-----+ | owner_uid | groupname | size_GB | +-----------------+-----+ | 106 | atlas/Role=production | 6. 41 | | 115 | atlas/Role=production | 3. 56 | | 154 | atlas/Role=production | 14. 54 | | 250 | atlas/Role=production | 14. 85 | | 263 | atlas/Role=production | 71. 11 | +-----------------+-----+ INFSO-RI-508833 ATLAS DDM Operations 10

How old are ATLAS files Enabling Grids for E-scienc. E select date_format(from_unixtime(ctime), '%Y-%m') as month, count(*) as nfiles, sum(filesize)/100000 as size_GB from Cns_file_metadata, Cns_groupinfo where Cns_groupinfo. gid=Cns_file_metadata. gid and Cns_groupinfo. groupname like "atlas%" group by month; +-----+---------+ | month | nfiles | size_GB | +-----+---------+ | 2006 -03 | 1920 | 127. 39 | | 2006 -04 | 10718 | 859. 46 | | 2006 -05 | 23333 | 1645. 98 | | 2006 -06 | 5275 | 601. 59 | | 2006 -07 | 506 | 35. 49 | | 2006 -08 | 334 | 35. 53 | | 2006 -09 | 162 | 18. 39 | | 2006 -10 | 5414 | 665. 23 | | 2006 -11 | 39 | 0. 00 | | 2006 -12 | 973 | 0. 00 | | 2007 -01 | 51 | 0. 00 | | 2007 -02 | 159 | 0. 00 | | 2007 -03 | 1922 | 4. 94 | | 2007 -04 | 2336 | 72. 66 | | 2007 -05 | 7966 | 346. 48 | | 2007 -06 | 44197 | 1822. 96 | | 2007 -07 | 845 | 85. 89 | | 2007 -08 | 27705 | 1844. 95 | | 2007 -09 | 22093 | 1020. 92 | +-----+---------+ INFSO-RI-508833 ATLAS DDM Operations 11

Access to files owned by production Enabling Grids for E-scienc. E select nbaccesses, count(*) from Cns_file_replica, Cns_file_metadata where Cns_file_replica. fileid=Cns_file_metadata. fileid and Cns_file_metadata. gi d=1310 group by nbaccesses ; +------+-----+ | nbaccesses | count(*) | +------+-----+ | 1 | 978 | | 2 | 825 | | 3 | 183 | | 4 | 30 | | 5 | 30 | | 6 | 11 | | 7 | 5 | | 8 | 1 | | 10 | 2 | | 11 | | 12 | | 13 | 1 | | 14 | 2 | | 16 | 1 | | 19 | 2 | | 22 | 3 | | 30 | 1 | | 31 | 3 | | 32 | 5 | | 33 | 1 | | 67 | 1 | +------+-----+ 21 rows in set (1. 77 sec) • 1 access = file was only stored – no copy on T 1 INFSO-RI-508833 ATLAS DDM Operations 12

Production files not yet replicated to T 1 Enabling Grids for E-scienc. E select md. name, date_format(from_unixtime(md. ctime), '%Y-%m-%d') as day from Cns_file_replica r, Cns_file_metadata md where r. fileid=md. fileid and md. gid=1310 and r. nbaccesses=1 order by day asc limit 10 ; +-----------------+------+ | name | day | +-----------------+------+ | log. 012663. _00127. job. log. tgz. 1 | 2007 -08 -23 | | log. 010685. _80590. job. log. tgz. 1 | 2007 -08 -23 | | ESD. 011212. _03771. pool. root. 4 | 2007 -08 -29 | | EVNT. 011675. _00528. pool. root. 2 | 2007 -08 -29 | | log. 011680. _00605. job. log. tgz. 2 | 2007 -08 -29 | | log. 013304. _00016. job. log. tgz. 3 | 2007 -08 -29 | | RDO. 013306. _00003. pool. root. 1 | 2007 -08 -29 | | log. 013303. _00022. job. log. tgz. 3 | 2007 -08 -29 | | RDO. 013304. _00061. pool. root. 3 | 2007 -08 -29 | | HITS. 013304. _00014. pool. root. 3 | 2007 -08 -29 | +-----------------+------+ INFSO-RI-508833 ATLAS DDM Operations 13

Usage of files replicated by DDM Enabling Grids for E-scienc. E INFSO-RI-508833 ATLAS DDM Operations 14

AOD Usage Enabling Grids for E-scienc. E select count(*) from Cns_file_replica r where owner_uid=258 and name like '%AOD%'); 66436 AOD files owned by DDM (Mario’s proxy) fileid in (select fileid from Cns_file_metadata where select r. nbaccesses, count(*) from Cns_file_replica r where owner_uid=258 and name like '%AOD%') group by n baccesses limit 15; +------+-----+ | nbaccesses | count(*) | +------+-----+ | 1 | 45411 | | 2 | 11384 | | 3324 | | 4 | 1974 | | 5 | 1629 | | 6 | 755 | | 762 | | 8 | 402 | | 9 | 202 | | 10 | 188 | | 11 | 135 | | 12 | 70 | | 13 | 56 | | 14 | 34 | | 15 | 24 | +------+-----+ fileid in (select fileid from Cns_file_metadata • 66 436 AOD files, 45411 never used via DPM, 90 files used more than 15 times INFSO-RI-508833 ATLAS DDM Operations 15

total transfers Enabling Grids for E-scienc. E select sum(actual_size)/100000 as get_GB from dpm_get_filereq ; +-----+ | get_GB | +-----+ | 47163. 37 | +-----+ select sum(actual_size)/100000 as put_GB from dpm_put_filereq ; +-----+ | put_GB | +-----+ | 22372. 59 | +-----+ INFSO-RI-508833 ATLAS DDM Operations 17

put Enabling Grids for E-scienc. E put requests: select date_format(from_unixtime(ctime), '%Y-%m') as month, count(*) from dpm_req where r_token in (select r_token from dpm_put_filereq) group by month; +----------+ | month | count(*) | +----------+ | 2006 -03 | 2549 | | 2006 -04 | 16604 | | 2006 -05 | 28188 | | 2006 -06 | 9132 | | 2006 -07 | 5342 | | 2006 -08 | 5116 | | 2006 -09 | 4370 | | 2006 -10 | 34282 | | 2006 -11 | 10414 | | 2006 -12 | 15870 | | 2007 -01 | 11820 | | 2007 -02 | 10638 | | 2007 -03 | 31518 | | 2007 -04 | 21938 | | 2007 -05 | 60971 | | 2007 -06 | 70478 | | 2007 -07 | 8267 | | 2007 -08 | 61377 | | 2007 -09 | 75111 | +----------+ INFSO-RI-508833 ATLAS DDM Operations 18

get Enabling Grids for E-scienc. E Get requests: select date_format(from_unixtime(ctime), '%Y-%m') as month, count(*) from dpm_req where r_token in (select r_token from dpm_get_filereq) group by month; +----------+ | month | count(*) | +----------+ | 2006 -03 | 1039 | | 2006 -04 | 8063 | | 2006 -05 | 34846 | | 2006 -06 | 20797 | | 2006 -07 | 9131 | | 2006 -08 | 7142 | | 2006 -09 | 18359 | | 2006 -10 | 16221 | | 2006 -11 | 13251 | | 2006 -12 | 8350 | | 2007 -01 | 3842 | | 2007 -02 | 8873 | | 2007 -03 | 13731 | | 2007 -04 | 10503 | | 2007 -05 | 7373 | | 2007 -06 | 7551 | | 2007 -07 | 9663 | | 2007 -08 | 48762 | | 2007 -09 | 41102 | +----------+ INFSO-RI-508833 ATLAS DDM Operations 19

DPM dumps Enabling Grids for E-scienc. E • export DPNS_HOST=t 2 -dpm-01. na. infn. it • export DPM_HOST=t 2 -dpm-01. na. infn. it • dpns-ls -l. R /dpm/na. infn. it/home/atlas > napoli. dpnsls. 20070924. list INFSO-RI-508833 ATLAS DDM Operations 20

Check. Dpm. Consistency Enabling Grids for E-scienc. E • https: //twiki. cern. ch/twiki/bin/view/LCG/Check. Dpm. Consistency • Author: Lana Abadie • The DPM consistency check tool is an admin tool which can be used for the following purposes – behaviour by default: – it allows listing files which are stored in the name server but not physically stored; – it allows listing files which are stored physically and not declared in the DPNS DB; – it allows listing files whose sizes stored in the name server differ from the physical size INFSO-RI-508833 ATLAS DDM Operations 21

dpns-cleanup Enabling Grids for E-scienc. E • By Lana Abadie • I tested it when we lost all files at 1 pool node • Uses these main calls: – – dpns_listreplicax dpns_delreplica dpns_getpath dpns_unlink • Example: . /dpns-cleanup --dpns_host_name golias 100. farm. particle. cz -server se 2. farm. particle. cz started at 11: 06, finished 14: 22 deleted 24414 files. INFSO-RI-508833 ATLAS DDM Operations 22

ATLAS groups Enabling Grids for E-scienc. E INFSO-RI-508833 ATLAS DDM Operations 23

Скачать презентацию Enabling Grids for E-scienc E ATLAS DDM Operations

a177e4068b29360bc06d8ec561c1fccb.ppt

Количество слайдов: 23