HPTS Workshop, October 7 -10, 2007, Asilomar, CA Lessons for High Performance Services or Towards Transaction Processing Engineering Dr. Alfred Z. Spector azs@azs-service. com Views herein are my own, and do not represent any other organization. October 10, 2007
Towards Transaction Processing Engineering Abstract Transaction processing as a field of engineering and science has a 50 -year long record: Thus, due consideration of the engineering lessons of systems from Saber on (and on) and the academic work, say, on atomicity or replication, should now be yielding high quality, non-speculative design principles. Perhaps some elements would be marked as understood and codified; some as explored, but not yet fully understood; and some as emerging and still unexplored. But, there would at least be the beginnings of an unambiguous, engineering backbone to the field. But, the reality is that we have many too many unpooled experiences and too many proposed techniques, structures, and algorithms on which no critical consensus has been reached. For example, I have my own opinions on the value of our teams' work on distributed transaction management (done 10 -20 years ago at Carnegie Mellon and Transarc), but I've yet to write (or see) a broad critical analysis that attempts to put the relevant topics conclusively to bed. There are many more examples like this. My motivation for desiring to begin this classification is a feeling that our field (and computer science as a whole) does not sufficiently and clearly document what we know. We are not good at reducing large aspects of software/systems project to a more traditional type of engineering. Due to the breadth of my recent management experiences, I recognize I may lack qualifications to contribute details to this effort, but indeed no one can see the whole elephant. Perhaps, the right approach is to come up with shared publication mechanisms (and needed structural elements) that can leverage the great successes of social networking technologies to harness the collective wisdom of the community. I will present this critique, illustrate it with categorized examples (some of which may generate debate), and propose some conceptual structures that could lead to a better codification of the science and engineering principles of transaction processing. HPTS 2007 Page 2
Towards Transaction Processing Engineering Outline 1. Our field is old, grand, and of the essence 2. Anecdotal problems 3. Deep Issues in TP 4. From “Towards a Software Science of Design 1” 5. A Step for TP: The codification of what we know 6. Conclusions 1 National Science Foundation Science of Design PI Mtg. , Arlington, VA, 3/07, Towards a Software Science of Design HPTS 2007 Page 3
Towards Transaction Processing Engineering The Grand Ole Field § Transaction Processing, or “service-oriented computing”, is ~50 years old: • • § A Strong Future: • • § Highly successful through many waves: • The few and the big (1 st decade) • Growth of the many (2 nd two decades) • Distributed systems (4 rd decade) • Web support, universal access, & massive utilities (5 th decade) Powered by: • Practical engineering by computer industry and users • Substantial researcher interest • A great and balanced community (!) So much more than 6 billion/(Think. Time. In. Secs) TPS Goal: To minimize overhead, support optimization of all systems, and provide universal aggregation and access of all knowledge We envision highly interconnected systems making life better • However, with substantial design and engineering challenges HPTS 2007 Page 4
Towards Transaction Processing Engineering Our Inventory of Technology Supporting the Agenda Source: Table 2. 1, Current-Cost Net Stock of Private, Commercial Fixed Assets, Equipment and Software, National Bureau of Economic Analysis • I/T (particularly S/W) a Growing Share of U. S. Commercial Capital Stock 1 • Note: Global I/T expenditure per year is roughly $1. 5 x 10^12 • My analysis: Growth in I/T value is probably much steeper; why? • In terms of value, I/T has grown much faster due to price deflation • Software depreciation/expensing is demonstrably fast 1 and by implication, the World’s. HPTS 2007 Page 5
Towards Transaction Processing Engineering We Should and We Do Know a Lot, but… § We make huge, reliable systems with great brilliance § We install small systems to run stores with minimal fuss § Our gurus are indeed gurus § But, • • • HPTS 2007 Why are the technical leaders so overwhelming key to major, new systems design and implementation? Why does (or can) the industry develop many system with minimal advances, but significant incompatibilities? That is, why are there so many different ways to do the same thing? Why do we regularly re-invent the wheel? Why is it virtually impossible to get the details right? Where is the regularity in system design? Why are achieving reuse, failure analysis, security, etc. either so difficult or requiring of genius? Page 6
Towards Transaction Processing Engineering Do We Truly Understand the Deep Issues in TP (1)? § The Data Abstraction Layer: • • § The Service Runtime: • • § Replication techniques Problem determination: • § Confidentiality Access control: data layer, service layer, … Availability Architecture: • § Apache and load balancing Heavier-weight transaction a la J 2 EE, or Microsoft. Net Stored procedures … Security and Privacy Architecture: • • § Relational, XML, Text w/arbitrary annotation, Chunks, Posix … Architecture and techniques Service Architecture with embedded services • HPTS 2007 Balancing encapsulation and transparency Page 7
Towards Transaction Processing Engineering Deep Issues in TP (2) § Reuse • • § The limits to atomicity • § Mirroring vs. event transmission/logging Systems management • § § § Where, when, and how much? Disaster recovery • § How to support and reduce reinvention: e. g. , O. O. , vs. Open Source, vs… Huge opportunities for engineering benefits across society Lower cost management Design for evolution Risk analysis Performance Analysis Software Engineering/Project management And many more… HPTS 2007 Page 8
Towards Transaction Processing Engineering Problems in TP Design are part of the Software Design Problem (from 2004 talk I gave at MIT 1) § Score high on many bad metrics: • • • Amount of code # of dependencies # of programmatic interfaces # of layers Administrative interface size & configuration options • • • Non-uniformity Non-orthogonality Defects Documentation # of programmers involved Added: dynamic change More generally, we have a software design problem 1 From The Conudrum of Systems, MIT, 2004, http: //www. csail. mit. edu/events/DLStalks/dlsspector 03. html HPTS 2007 Page 9
Towards Transaction Processing Engineering Stepping Back: Design, Broadly 1 § The process of design refers to the application of synthetic and analytic processes to plan and create new objects. My perspective – great commonality in design across many human endeavors § • • § Protocols (Medical) Differential Diagnoses Bridge/Building Design 2 Proofs Plays, Music, Novels Urban Areas Political Systems Complex Systems While design is central, it is not today a holistic discipline, studied in the same ways as the classical humanities or even science/engineering. Should it be? Cross-fertilization to be expected: § • • • § AASHTO specifications for requirements of highway bridges Design workbenches Project design Software design is often a part of broader designs so design is often inextricably linked 1 National 2 Bridge Science Foundation Science of Design PI Mtg. , Arlington, VA, 3/07, Towards a Software Science of Design and Construction, CACM 29(4): 267 -283, 4/86, A. Z. Spector and D. K. Gifford HPTS 2007 Page 10
Towards Transaction Processing Engineering Difficulty of Software Design § § § § § Diversity of objectives and scale Quantity of artifacts Growing scale even for individual artificats Post facto, dynamic assembly Malleability Longevity Absence of manufacturing cost Changing development possibilities Difficulty of verification HPTS 2007 Page 11
Towards Transaction Processing Engineering A First Step in TP: The Codification of What We Know § Social Networking approaches to knowledge capture, organization, and refinement have been more successful than many thought: • • • § Wikipedia is an obvious example Linux is another Other domain specific social networking sites Grand Question: • • • HPTS 2007 Could we devise a going-in structure for our field that organized requirements, algorithms, etc…along various dimensions (Recursive application of technology)? Could this be debated and refined over time? Could we then provide answers • Facts and Proofs • Designs • Software templates • Code Some tagging for elements: some would be marked as understood and codified; some as explored, but not yet fully understood; and some as emerging and still unexplored Could we use social networking technologies based on our systems and using our community to put Transaction Processing Engineering on a firm foundation? Page 12
Towards Transaction Processing Engineering Do We Need A TP Knowledge Repository (TPKR)? § § Yes, here’s why: As a field, we don’t get to the bottom of things very well: • • • § § § Industry has challenges: • The good (or bad) old thing • The new, new thing Academe has challenges: • Oft. , lack of detailed knowledge on big systems • The new, new thing (or LPU) • The time to really answer hard problems The whole field is not really disciplined towards achieving design principles TPKR could motivate: • • • More discourse and information exchange Challenges and concomitant debate on important design issues A publication vehicle, more in keeping with the times than the Journal • Contributions to the compendium of definitive information would be judged highly • Perhaps, more so than individual journal/conference articles The social phenomena might unleash a lot of energy I note that Jim Gray was very interested in alternative forms of publication (e. g. , as compared to traditional journals), in my last conversation with him HPTS 2007 Page 13
Towards Transaction Processing Engineering Towards a Software-wide Science of Design A Software Science of Design would facilitate software production across the lifecycle by helping to understand, codify and integrate currently understood approaches to design, but also by creating better approaches and tools. § § A broad, integrative mission Key foci: • Cross-fertilization with other design disciplines • Holistic approach • Scientific analysis (not, “here’s a new approach, it’s real cool ”) • Education and standardization of approaches • Interdisciplinary work Elements probably not included • Domain-specific abstractions and their implementations • Elements directly covered in other areas of the field: • Programming Language Design • Verification • Etc. • These are admittedly blurry lines Note: • I don’t think that S/W design, broadly, will admit to as much standardization as exists in, say, road design • Nonetheless, there can still be families of approaches, parameterized (or meta-), approaches & tools… HPTS 2007 Page 14
Towards Transaction Processing Engineering Elements of a Research Agenda § § § § Establishing Convincing Truths Terminology and reference models Metrics Modularity and reuse Workbenches/platforms Development methodologies Automated assembly Design for change Design for security Complexity Education Team culture Impact of accounting and taxation Impact of liability, intellectual property, and anti-trust law HPTS 2007 Page 15
Towards Transaction Processing Engineering Observations and Conclusions § § § § The state of software design is not sound. (1) We don’t know enough (2) we rarely codify (in the broadest sense) what we know (3) we don’t consistently use or educate what we codify. Design is everywhere. While particularly important in software, there should be a cross-discipline design community looking for synergies in innovation, education, and practice. Design in software is multi-faceted. It applies to all aspects of the lifecycle. A Science of Design should be broad enough to address all aspects. Design is of growing importance. Related not only to amount of software, more serious applications of software, greater diversity of it’s development across time and space, and post-facto, dynamic assembly, with a greater base of components, design (but not programming) is oft. needed! Implications are manifold. The research agenda has great opportunity. There are plenty of topics for a broad research agenda. Work should proceed towards categorization and completeness: tops down and bottoms up. Interdisciplinary research is of particular important. The design of software is not purely a problem of computer science. Establishment of objectives. We should establish quantifiable objectives. § And we could do a big thing in our subfield: TPKR HPTS 2007 Page 16
Towards Transaction Processing Engineering Thank you! HPTS 2007 Page 17
Скачать презентацию HPTS Workshop October 7 -10 2007 Asilomar CA
603025635c6aed881c4dc2567721c216.ppt