Language Tools for Distributed Computing and Program Generation

Language Tools for Distributed Computing and Program Generation (III) Morphing: Bringing Discipline to Meta-Programming Yannis Smaragdakis University of Oregon

These Lectures l NRMI: middleware offering a natural programming model for distributed computing l l J-Orchestra: execute unsuspecting programs over a network, using program rewriting l l solves a long standing, well-known open problem! led to key enhancements of a major open-source software project (JBoss) Morphing: a high-level language facility for safe program transformation Yannis Smaragdakis 2

Program Generation l The kinds of techniques used in J-Orchestra, JBoss AOP, etc. are an instance of program generation l l l program generators = programs that generate other programs This is a research area that I have worked on for a long time Next, I’ll give a taste of why the area inspires me and what research problems are being solved Yannis Smaragdakis 3

Why Do Research on Program Generators? l intellectual fascination l l “If you are a Computer Scientist, you probably think computations are interesting. What then can be more interesting than computing about computations? ” practical benefit l many software engineering tasks can be substantially automated Yannis Smaragdakis 4

Sensationalist Program Generation l You know what I mean if you feel anything when you look at a self-generating program l ((lambda (x) (list x (list (quote) x))) (quote (lambda (x) (list x (list (quote) x))))) l main(a){a=“main(a){a=%c%s%c; printf(a, 34, a, 34); }”; printf(a, 34, a, 34); } Yannis Smaragdakis 5

Why Write a Generator? l l Any approach to automating programming tasks may need to generate programs Two main reasons (if we get to the bottom) l l performance (code specialization) conformance (generate code that interfaces with existing code) l e. g. , generating code for J 2 EE protocols in JBoss l widespread pattern of generation today: generators that take programs as input and inspect them Yannis Smaragdakis 6

A (Big) Problem l l Program generation is viewed as an inherently complex, “dirty”, low-level trick Hard to gain the same confidence in a generated program as in a hand-written program l l even for the generator writer: the inputs to the generator are unknown Much of my work is on offering support for ensuring generators work correctly l l necessary, if we want to move program generation to the mainstream make sure generated program free of typical static “semantic” errors (i. e. , it compiles) Yannis Smaragdakis 7

Meta-Programming Introduction l l Tools for writing program generators: programs that generate other programs `[…] (quote) expr = `[7 + i]; l #[…] (unquote) stmt = `[if (i > 0) return #[expr]; ]; stmt <= `[if (i > 0) return 7 + i; ] Yannis Smaragdakis 8

An Unsafe Generator My buggy generator User Input . . . Output if (pred 1()) emit(`[int i; ]); . . . i++; //i undefined!. . . if (pred 2()) emit(`[i++; ]); . . . • Error in the generator: pred 2() does not imply pred 1() under ALL inputs. Yannis Smaragdakis 9

Statically Safe Program Generation l l Statically check the generator to determine the safety of any generated program, under ALL inputs. Specifically, check the generator to ensure that generated programs compile Yannis Smaragdakis 10

Why Catch Errors Statically? l “After all, the generated program will be checked statically before it runs” l l l Errors in generated programs are really errors in the generator. compile-time for the generated program is run-time for the generator! Statically checking the generator is analogous to static typing for regular programs Yannis Smaragdakis 11

The Problem: l l Asking whether generated program is well-formed is equivalent to asking any hard program analysis question (generally undecidable). Control Flow if (pred 1()) emit (`[int i; ]); if (pred 2()) emit (`[i++; ]); l Data Flow emit ( `[ int #[name 1]; int #[name 2]; ); Yannis Smaragdakis 12

Early Approach: Safe. Gen l l l A language + verification system for writing program generators l generator Input/Output: legal Java programs Describe everything in first-order logic l Java well-formedness semantics: axioms l structure of generator/generated code: facts l type property to check: test l conjecture: (axioms ⋀ facts) → test Prove conjecture valid using automatic theorem prover: SPASS l a great way to catch bugs in the generator that only appear under specific inputs Yannis Smaragdakis 13

Safe. Gen Input/Output: legal Java programs. l Controlled language primitives for control flow, iteration, and name generation. l Expressive power of to define Example: for control first order logic and namethe conditions flow, iteration, “Iterate over all the methods from the input class generation. that havewell-formedness of is public and a return type, one argument that generated program for any l Prove such that it has at least one method with an argument input. l that implements java. io. Serializable” l By proving validity of FOL sentences l Leveraging the power of a theorem prover (with good results) Yannis Smaragdakis 14

Generator: Signature #defgen make. Interface (Interface i) { public interface Foo extends #[i. Name] {. . . } } keyword #defgen, name n input: a single entity, or a set or entities. n Yannis Smaragdakis 15

Inside the Generator #defgen make. Interface (Interface i) { public interface Foo extends #[i. Name] {. . . } } l Between the { … } l l Any legal Java syntax “escapes”: l #[…], #foreach, #when, #name[“…”] Yannis Smaragdakis 16

#[…] l Splice a fragment of Java code into the generator. interface Bar {. . . } input #defgen make. Interface ( Interface i ) { public interface Foo extends #[i. Name] {. . . } public interface Foo extends Bar { }. . . output } Yannis Smaragdakis 17

#foreach Takes a set of values, and a code fragment. Generate the code fragment for each value in the set. interface Bar { int A (float i); input float B (int i); } #defgen make. Interface ( Interface i ) { l public interface Foo extends #[i. Name] { #foreach (Method m : Method. Of(m, i) ) { void #[m. Name] (); } public interface Foo extends Bar { } } void A (); void B (); output } Yannis Smaragdakis 18

Cursors l A variable ranging over all entities satisfying a firstorder logic formula. l predicates and functions correspond to Java reflective methods: Public(m), Method. Of(m, i), m. Ret. Type, etc. l FOL keywords: forall, exists, &, |, etc. #foreach (Method m : Method. Of(m, i)) {. . . } Yannis Smaragdakis 19

A Conjecture – In English Given that all legal Java classes have unique method signatures, (axiom) given that we generate a class with method signatures isomorphic to the method signatures of the input class (fact) can we prove that the generated class has unique method signatures? (test) Yannis Smaragdakis 20

Phase I: Gathering Facts #defgen make. Interface ( Interface i ) { public interface Foo extends #[i. Name] { #foreach (Method m : Method. Of(m, i)) { void #[m. Name] (); } } } ∃i ( Interface(i) ⋀ ∃i’ (Interface(i’) ⋀ name(i’) = Foo ⋀ Super. Class(i’) = i ⋀ (∀m (Method. Of(m, i) ↔ (∃m’ (Method. Of(m’, i’) ⋀ Ret. Type(m’) = void ⋀ name (m’) = name(m) ⋀ ¬(∃t Arg. Type. Of(t, m’)))) Yannis Smaragdakis 21

Phase II: Constructing Test #defgen make. Interface ( Interface i ) { public interface Foo extends #[i. Name] { #foreach (Method m : Method. Of(m, i)) { void #[m. Name] (); } } } ∃i (Interface (i) ⋀ ∃i’ ((Interface(i’) ⋀ name(i’) = Foo ⋀ ∀m ( Method. Of(m, i’) → ¬(∃m’ Method. Of(m’, i’) ⋀ ¬(m = m’) ⋀ name(m’) = name(m) ⋀ Arg. Types(m’) = Arg. Types(m))))) Yannis Smaragdakis 22

When Does It Fail? interface Bar { int A (float i); float A (int i); input } #defgen make. Interface ( Interface i ) { public interface Foo extends #[i. Name] { #foreach (Method m : Method. Of(m, i)) { void #[m. Name] (); } public interface Foo extends Bar { } } void A (); output } Yannis Smaragdakis 23

Safe. Gen Safety l Checks the following properties: l l l A declared super class exists A declared super class is not final Method argument types are valid A returned value’s type is compatible with method return type Return statement for a void-returning method has no argument Yannis Smaragdakis 24

Experience w/ Theorem Provers l We tried several theorem provers: l l l Hand-constructed axioms, facts, and tests for common bugs and generation patterns. Criteria: ability to reason without human guidance and terminate. SPASS became the clear choice. Yannis Smaragdakis 25

Overall Experience l We had predefined a set of ~25 program generation tasks l l Safe. Gen reported all errors correctly, found proofs for correct generators l l pre-selected before Safe. Gen was even designed all proofs in under 1 second Safe. Gen terminated 50% of the time with a proof of error, when one existed l it could conceivably fail to prove a true property and issue a false warning Yannis Smaragdakis 26

Do We Really Want Theorem Provers for This? l l l The Safe. Gen approach is effective But the whole point was to offer certainty to the programmer Theorem proving is an incomplete approach, which is not intuitively satisfying l l no clear boundary of incompleteness: just that theorem prover ran out of time Can we get most of the benefit with a type system? Yannis Smaragdakis 27

Morphing: Shaping Classes in the Image of Other Classes The MJ Language WARNING: The examples are important. Keep me honest!

Morphing: MJ l Static reflection over members of type params l class Method. Logger<class X> extends X { <Y*>[meth] for(public int meth (Y) : X. methods) int meth (Y a) { int i = super. meth(a); System. out. println("Returned: " + i); return i; } } l Other extensions (over Java) in this example? Yannis Smaragdakis 29

Real-World Example (JCF) l public class Make. Synchronized<X> { X x; public Make. Synchronized(X x) { this. x = x; } <R, A*>[m] for(public R m(A) : X. methods) public synchronized R m (A a) { return x. m(a); } } l <A*>[m] for(public void m(A) : X. methods) public synchronized void m(A a) { x. m(a); } 600 LOC in class Collections, just to do this Yannis Smaragdakis 30

More Morphing / MJ public class Array. List<E> extends Abstract. List<E>. . . {. . . <F extends Comparable<F>>[f]for(public F f : E. fields) public Array. List<E> sort. By#f () { public void sort. By#f () { Collections. sort(this, new Comparator<E> () { public int compare(E e 1, E e 2) { return e 1. f. compare. To(e 2. f); } } } Yannis Smaragdakis 31

Modular Type Safety l Our theorem of generator safety for all inputs, is modular type safety in MJ l l l the generic class is verified on its own (not when type-instantiated) type error if any type parameter can cause an error can distribute generic code with high confidence Yannis Smaragdakis 32

Type Errors? l class Call. With. Max<class X> extends X { <Y>[meth]for(public int meth(Y) : X. methods) int meth(Y a 1, Y a 2) { if (a 1. compare. To(a 2) > 0) return super. meth(a 1); else return super. meth(a 2); } } l Where is the bug? l where is the other bug? Yannis Smaragdakis 33

Once More. . . l public class Add. Get. Set<class X> extends X { <T>[f] for(T f : X. fields) {| public T get#f () { return f; } public void set#f (T nf) { f = nf; } |} } l Where is the bug? Yannis Smaragdakis 34

Filter Patterns l public class Add. Get. Set 2<class X> extends X { <T>[f] for( T f : X. fields ; no get#f() : X. methods) public T get#f () { return f; } <T>[f] for( T f : X. fields ; no set#f(T) : X. methods) public void set#f (T nf) { f = nf; } } l keywords “some”, “no” Yannis Smaragdakis 35

Type Checking in More Detail Validity and Well-definedness without Filter Patterns

Well-Definedness (Single Range) l class Copy. Methods<X> { <R, A*>[m] for( R m (A) : X. methods) R m (A a) {. . . } } l l Uniqueness implies uniqueness l what if I am mangling signatures? class Change. Arg. Type<X> { <R, A>[m] for ( R m (A) : X. methods) R m ( List<A> a ) {. . . } } l example of problems? Yannis Smaragdakis 37

Validity l class Invalid. Reference<X> { Foo f; . . . // code to set f field [n] for( void n (int) : X. methods ) void n (int a) { f. n(a); } } class Foo { void foo(int a) {. . . } } l Any problems? Yannis Smaragdakis 38

Easy-to-Show Validity l class Easy. Reflection<X> { X x; . . . // code to set x field [n] for( void n (int) : X. methods ) void n (int i) { x. n(i); } } Yannis Smaragdakis 39

Validity in Full Glory l class Reference<X> { Declaration<X> dx; . . . //code to set dx <A*>[n] for( String n (A) : X. methods ) void n (A a) { dx. n(a); } } class Declaration<Y> { <R, B*>[m] for( R m (B) : Y. methods ) void m (B b) {. . . } } l l l type-checking: range subsumption range R 1 subsumes R 2 if patterns unify (one way) what are the patterns above? Yannis Smaragdakis 40

Well-Definedness l class Static. Name<X> { int foo () {. . . } <R, A*>[m]for (R m (A) : X. methods) R m (A a) {. . . } } l Ok? Yannis Smaragdakis 41

Less Clear When Doing Type Manipulation l class Manipulation. Error<X> { <R>[m] for (R m (List<X>) : X. methods) R m (List<X> a) {. . . } <P>[n] for (P n (X) : X. methods) P n (List<X> a) {. . . } } l Any problems? Yannis Smaragdakis 42

Fixing Previous Example l class Manipulation<X> { <R>[m] for (R m (List<X>) : X. methods) R list#m (List<X> a) {. . . } <P>[n] for ( P n (X) : X. methods) P nolist#n (List<X> a) {. . . } } Yannis Smaragdakis 43

Two Way Unification? l class Why. Two. Way<X> { <A 1, R 1> for ( R 1 foo (A 1) : X. methods) void foo (A 1 a, List<R 1> r) {. . . } <A 2, R 2> for ( R 2 foo (A 2) : X. methods) void foo (List<A 2> a, R 2 r) {. . . } } l Any problems? Yannis Smaragdakis 44

Now Add Filters. . .

Positive Filter Patterns l public class Do. Both<X, Y> { <A*>[m] for(static void m(A): X. methods; some static void m(A): Y. methods) public static void m(A args) { X. m(args); Y. m(args); } } Yannis Smaragdakis 46

Rules l l l P 1, +F 1 subsumes P 2, +F 2 if P 1 subsumes P 2, and F 1 subsumes F 2. P 1, −F 1 subsumes P 2, −F 2 if P 1 subsumes P 2, and F 2 subsumes F 1. P 1, ? F 1, G 1 is disjoint from P 2, ? F 2, G 2 if G 1 is disjoint from G 2. P 1, ? F 1, G 1 is disjoint from P 2, −F 2, G 2 if F 2 subsumes P 1, +F 1, G 1 is disjoint from P 2, −F 2, G 2 if F 2 subsumes F 1. Yannis Smaragdakis 47

Comprehensive Example l public class Union. Of. Static<X, Y> { <A*>[m] for(static void m (A): X. methods) static void m(A args) { X. m(args); } <B*>[n] for(static void n (B): Y. methods; no static void n(int, B): X. methods) static void n(int count, B args) { for (int i = 0; i < count; i++) Y. n(args); } } l First unify primary, then substitute, then unify filter Yannis Smaragdakis 48

So What? Lots of power and modular type safety?

Fill in Interface Methods l class Make. Implement<X, interface I> implements I { X x; Make. Implement(X x) { this. x = x; } // for all methods in I, but not in X, provide default impl. <R, A*>[m]for( R m (A) : I. methods; no R m (A) : X. methods) R m (A a) { return null; } // for X methods that correctly override I methods, copy them <R, A*>[m]for ( R m (A) : I. methods; some R m (A) : X. methods) R m (A a) { return x. m(a); } } // for X methods with no conflicting I method, copy them. <R, A*>[m]for(R m (A) : X. methods; no m (A) : I. methods) R m (A a) { return x. m(a); } Yannis Smaragdakis 50

MJ in the Universe l “Write code once, apply it to many program sites” l l so far the privilege of MOPs, AOP, metaprogramming modular type safety only with MJ Yannis Smaragdakis 51

In Summary What did I talk about?

These Lectures l NRMI: middleware offering a natural programming model for distributed computing l l J-Orchestra: execute unsuspecting programs over a network, using program rewriting l l solves a long standing, well-known open problem! led to key enhancements of a major open-source software project (JBoss) Morphing: a high-level language facility for safe program transformation l “bringing discipline to meta-programming” Yannis Smaragdakis 53

Credits: My Students l Christoph Csallner l automatic testing l JCrasher l Check-n-Crash (Cn. C) l DSD-Crasher l tools used at NC State, MIT, MS Research, Utrecht, UWashington about to intern at MS Research l Yannis Smaragdakis 54

Credits: My Students l Shan Huang l program generators and domain-specific languages l Meta-Aspect. J (MAJ) l Safe. Gen l c. J l MJ l Intel Fellowship NSF Graduate Fellowship l Yannis Smaragdakis 55

Credits: My Students l Brian Mc. Namara l multiparadigm programming l l l FC++ LC++ now at Microsoft Yannis Smaragdakis 56

Credits: My Students l Eli Tilevich l language tools for distributed computing l NRMI l J-Orchestra l GOTECH binary refactoring now an Assistant Professor at Virginia Tech Yannis Smaragdakis 57

Credits: My Students l David Zook l program generators and domain-specific languages l l Meta-Aspect. J (MAJ) Safe. Gen Yannis Smaragdakis 58

Credits: My Students l Ranjith Subramanian (M. Sc. ) l Adaptive replacement algorithms l hardware caching Yannis Smaragdakis 59

Credits: My Students l Austin Chau (M. Sc. ) l language tools for distributed computing l J-Orchestra Yannis Smaragdakis 60

Credits: My Students l Marcus Handte (M. Sc. ) l l language tools for distributed computing l J-Orchestra now a Ph. D. student at Stuttgart Yannis Smaragdakis 61

Credits: My Students l Nikitas Liogkas (M. Sc. ) l l language tools for distributed computing l J-Orchestra now a Ph. D. student at UCLA Yannis Smaragdakis 62

Credits: My Students l Stephan Urbanski (M. Sc. ) l language tools for distributed computing l GOTECH l now a Ph. D. student at Stuttgart Yannis Smaragdakis 63

Thank you!