Archiver Hypertable Engined (Hyper. Archiver) Loris Giovannini, Mauro Giacchini Epics Collaboration Meeting 03/06, 2010 L. Giovannini -- M. Giacchini
Target Make the EPICS Archiver embedded DB more reliable and safe especially in large application with a large number of PVs to store First Approach: re-write the Rtree and doublelinked-list embedded DB structure Second approach: look for a complete replacement of the embedded DB with new one L. Giovannini -- M. Giacchini
Why not Beauty ? (Best Ever Archiver Utility) Beauty = Classical Archiver over Oracle DB Pros: SQL statements accept Professional Solution Cons: Not so fast Really expensive (licenses) L. Giovannini -- M. Giacchini
Why not My. SQL ? Potential scalability concerns Designed for a single machine (not distributed) What it takes to make it scale Major engineering effort Solutions are usually ad hoc Solutions usually involve horizontal partitioning + replication Solutions involve expensive hardware L. Giovannini -- M. Giacchini
Big. Table ? Proprietary DB used by Google over their proprietary Google File System High Performance Large amount of data Stable Reliable Distributed L. Giovannini -- M. Giacchini
HYPERTABLE…what is it ? Hypertable has been developed as an in-house software at Zvents Inc. In January 2009, Baidu, the leading Chinese language search engine, became a project sponsor. Rediff. com is one of the premier worldwide online providers of news, information, communication, entertainment and shopping service. Hypertable is licensed under the GNU General Public License Version 2. “Our goal is nothing less than that Hypertable become one of the world’s most massively parallel high performance database platforms. ” Not RDB L. Giovannini -- M. Giacchini
Hadoop/Hypertable Solution Objective • • • Handles applications with large datasets Fast fault detection and fail-over High throughput processing Centralized scheduling of tasks and execution of batch processes Can be deployed on low cost, commodity hardware Eliminates or reduces the need for expensive table joins L. Giovannini -- M. Giacchini
Machines used to the tests Machines and time up Insertion Time Retrieval Time (all pv parameters) Virtualbox virtual machine Debian Linux Stable x 86 4 cores (shared among 3 VMs) 4 GB RAM (1) 5. 4 k RPM SATA Disk $4 k (to 4 Vbox) (1/2 h data acq SNS) • 1 min 31 sec • 1 777 600 • 0. 042 sec to extract • 133 K samples of 4 Debian Linux Stable AMD 64 - 8 cores 64 GB RAM (7) 10 k RPM SAS Disks (RAID 5) $11 k (1/2 h data acq SNS) • 1 min 13. 6 sec • 1 777 600 Debian Linux Stable AMD 64 - 8 cores 64 GB RAM (7) 10 k RPM SAS Disks (RAID 5) $11 k (7 dy data acq SNS) • 11. 2 hours to Debian Linux Stable AMD 64 - 8 cores 64 GB RAM (7) 10 k RPM SAS Disks (RAID 5) $11 k (7 dy data acq. L. Giovannini -SNS) sample • 19. 5 K samples/sec samples • 24 K samples/sec insert • 597 M samples • 14. 8 K samples/sec M. Giacchini pv • 3. 17 M samples/sec • 0. 024 sec to extract • 133 K samples of 4 pv • 5. 54 Msamples/sec • 0. 031 sec to extract • 133360 samples of 4 pv • 4. 3 M samples/sec • 19 min 51 sec • 597. 273. 600 all pv • 500 K samples/sec
Oracle Vs Hypertable Oracle insertion time : peak in write test (8 Ksamples/sec) Hypertable insertion time: 1. 777. 600 per 1 min 13 sec (24 Ksamples/sec) Oracle retrieve time: 36. 657 samples in 79. 294 sec (~462 vals/sec) Hypertable retrieve time: 57624 samples in 0. 36 sec ( 160 K vals/sec) in half hour L. Giovannini -- M. Giacchini
Hypertable into CSS THE OUR GOAL storing: 10000 channels per second retrieve: 1000 samples for up to 4 signal in less than 1 second Machines and time up Ubuntu 9. 10 karmic Kola Intel Pentium M 740 1, 73 Ghz 2 GB RAM 5, 6 K RPM sata Disk Insertion Time Hypertable My. Sql • 8, 45 sec • 27, 1 sec • 110200 sample • 110200 • 13 K sample/sec • 4 K sample/sec L. Giovannini -- M. Giacchini
Retrieval outside CSS First Step: Develop a Java Program for only test the performances for retrieval without any elaboration of data to valuate the possibility to respect our goals using Thrift Java client API (Mandatory to interface with Hypertable) Results 1000 sample for 4 Pv: 1000 sample: 22 ms 1000 sample: 15 ms 56 Ksample/sec 1000 sample: 19 ms L. Giovannini -- M. Giacchini
Retrieval L. Giovannini -- M. Giacchini
TIME Retrieval into CSS Second Step: Develope an Eclipse plug-in org. csstudio. archivereader. htrdb for the support of Hypertable like as org. csstudio. archivereader. rdb Working in Progress The extraction time into CSS is about 10 times worse: The reasons are under study, there are two probable causes: initial query latency of Thrift API when connect to Hypertable wrong classes design into CSS architecture L. Giovannini -- M. Giacchini
Acknowledgments Kay Kasemir (ORNL) Doug Judd (Hypertable Core Team) Bob Dalesio, Ralph Lange, Robert Petkus (BNL) More information www. lnl. infn. it/~epics/ L. Giovannini -- M. Giacchini
Скачать презентацию Archiver Hypertable Engined Hyper Archiver Loris Giovannini Mauro
108161b27b8d3113c2bd46417c693c6f.ppt