Next Generation Software Systems and Programming Languages Research

Next Generation Software Systems and Programming Languages Research Matthias Felleisen

Our Experience • PLT has built a bunch of software, including PDE, Web server, IMAP clients, and many more. • These products share a number of characteristics with other modern software products. • What does our experience suggest for PL research?

My Thesis, My Goal • Software systems have become extensible collections of components. • What does this mean for programming languages: linguistics, pragmatics, virtual machines/compilers, and runtime environments?

Outline • Two Examples: Dr. Scheme, Web Server • Language Support for COP • Virtual Machine Support • Contracts for Components • Summary, Advice

Disclaimer • PLT’s experience • My thesis and trajectory • these are not universal concerns

Examples: Extensible Systems

Two Examples • Dr. Scheme: a modern, pedagogic programming environment • The PLT Web Server: an experiment in serving dynamic Web content

Dr. Scheme • What does it take to build a PDE from scratch? • How about extensions?

Dr. Scheme scope-sensitive syntax checker syntax-sensitive editor static debugger algebraic stepper interactive evaluator

Thank You to a Fantastic Team Shriram Krishnamurthi, Brown Matthew Flatt, Utah John Clements, NEU Philippe Meunier, NEU Paul Graunke, NEU Robby Findler, UChicago Cormac, SRC Paul Steckler, NEU

Dr. Scheme model view Editor GUI Read-Eval-Print-Loop mutual references

Dr. Scheme: Plugging in Tools Syntax Checker Editor Tools -- plug-ins Read-Eval-Print-Loop Algebraic Stepper

Dr. Scheme: Plugging in Student Programs Syntax Checker Editor Tools -- plug-ins Read-Eval-Print-Loop Student Programs Algebraic Stepper

Dr. Scheme: The Big Picture a single execution environment: the VM Editor Read-Eval-Print-Loop

Dr. Scheme • powerful plug-ins that interact with editor and read-eval-print loop • student programs become dynamically plugged in components • easy communication among all components

A Web Server From Components • What does it take to build a Web server? • What about dynamic content?

A Web Server From Components A server is basically a read-eval-print loop. Upon a TCP connect: read a request: GET /uniquelylara/paintings/index. html HTTP/1. 0 … headers … interpret the request, the path, and the headers: dispatch on file path, check for defaults, … print the file: send the requested file back via a TCP write

A Web Server From Components pattern matching TCP connections URL parsing These components are good old libraries. Dispatcher REPL aka Web Server HTML

A Web Server From Components pattern matching TCP connections Dis: local Dis: remote URL parsing Dis: dynamic REPL aka Web Server code duplication HTML

A Web Server From Components pattern matching local TCP connections Dispatch URL parsing HTML Dispatch remote dynam REPL aka Web Server these components are parameterized and instantiated

Web Server: Dynamic Content • dynamic content generation is becoming more and more important • dynamic content generation is inefficient and slow • dynamic content generation is difficult to manage

Web Server: Plug-in Scripts Dispatcher Scripts Server Loop

Web Server: Scripts as Components dynamically linked Dispatcher Server Loop exchange of values not bit streams

Web Server: Scripts as Components store script as first-class value Dispatcher Server Loop

Web Server: Scripts as Components link in once Dispatcher Server Loop

Web Server: Scripts as Components link in once, link in twice Dispatcher Server Loop

Web Server: Scripts as Components Dispatcher Server Loop make the internals of the server available: stack, environment, store

A Web Server From Components • Our Web server’s performance for dynamic content generation is four times better than that of Apache/Perl and 20 to 40% than mod-Perl. • It is the only Web server whose scripts are automatically safe for back buttons and cloning: control state.

Other Examples • Linux: – kernel modules – applications • Web browsers: – plug-ins – applets • Multimedia players: – drivers – formats • MS Office – component: COM • Java Beans • . NET

From Component Systems to Programming Languages

From Component Systems to PLs • component-oriented programming: – different than OOP – different than modules • VM (virtual machines) for extensible systems of components – more than a virtual machine • contracts for components

Component Software, Szyperski • A component is … – a unit of independent deployment – a unit of third-party composition – a blueprint without state Szyperski, page 30

COP and Languages: Linguistic Support • parameterization over – imported components – exchange of values, including critical pieces • linking – graph-based (mutual references) – hierarchical – dynamic (first-class values in core) • multi-lingual components

COP and Languages: Virtual Machines • a VM for many languages • a VM for dynamic extensions • a VM for “non-cooperative” extensions

COP and Languages: Quality Assurance • components exchange values, including objects and functions • components must protect these values and must promise certain properties – – types (partial) functional specifications concurrency specifications Qo. S guarantees

Linguistic Support for COP with Matthew Flatt, Robby Findler, Shriram Krishnamurthi and a little bit of help from Dan Friedman

Existing Language Support for Components Which language constructs should we use to support programming with components?

Existing Language Support for Components Proposal: Use first-class procedures and arrange them in patterns. Pro: hierarchical linking, dynamic invoking, by-reference passing work Cons: the granularity is too far off. Maintaining “linking patterns” by hand is error prone.

Existing Language Support for Components Proposal: Use classes and objects as components. Pro: larger granularity, contains many functions, protection, dynamic loading Cons: The super-class is hard-wired, i. e. , no parameterization. It is impossible to wire-in the same class many times. Inheritance poses programming and contract checking problems (Szyperski).

Existing Language Support for Components Proposal: Use Modula-style modules, Java-style packages, or C#-style assemblies. Pro: proper granularity Cons: modules are too static (compile-time). Importing is hard-wired. They are not first-class values.

Existing Language Support for Components Proposal: Use ML-style functors. They find good uses in SML and OCAML. Pro: They are parameterized over their importing modules, by definition. Cons: they are not values. It is not possible to link them in a graphical manner. Like plain modules, they are static and not dynamically linkable.

PLT’s Components: Units are first-class values: • Units are parameterized over – imported components – references to values • Units are linked in – graph-based – hierarchical – dynamic manner

Units: A Language Constructs for COP imports definitions exports

Units: A Language Constructs for COP example: type t , val g : int -> t fun f (an-int, a-t) = … val f : int t -> string

Units: Graphical Linking type t fun f (an-int, a-t) = … type t f : int t -> string fun f (an-int, a-t) = … f : int t -> string type t fun f (an-int, a-t) = … f : int t -> string

Units: Hierarchical Linking type t fun f (an-int, a-t) = … type t f : int t -> string fun f (an-int, a-t) = … f : int t -> string type t fun f (an-int, a-t) = … f : int t -> string

Units: Dynamic Linking type t fun f (an-int, a-t) = … type t f : int t -> string fun f (an-int, a-t) = … f : int t -> string type t fun f (an-int, a-t) = … f : int t -> string

Units: Flatt & Felleisen • PLT’s units support COP well. • … linking together components is flexible, with rich control • … widely used in Version 103 • But there are open problems

Units: A Program Design Problem • Given: a component with – a datatype T with N variants – a collection F of M functions on T • Wanted: extend T with variants and F with additional functions without access to the source of the component Krishnamurthi, Felleisen, Friedman

Units: A Program Design Problem • Why does it matter? – abstract reasons – examples • How do we do it? – where units work – where we need more

Units Extensibility - Why? • maintenance: fix once, fix everywhere • extensibility: add functionality once, add it everywhere

Unit Extensibility: Example 1 • Dr. Scheme: Krishnamurthi et alli the language hierarchy and the set of tools • datatype is a grammar • tools: syntax check, type check, stepper, … and more to come

Unit Extensibility: Example 2 • Geometry server: Hudak, Darpa productivity study • datatype describes arrangements of geometric shapes • tools: containment, drawing, moving and many others

Unit Extensibility: Example 3 • San Francisco project: Bohrer, IBM • datatype describes customer, manager, account hierarchies • tools: back office maintenance tools

Unit Extensibility: The Picture The Core Component datatype t f : … t … -> … g : … -> … t … h : … t … -> … t …

Unit Extensibility: The Picture The Variant Extension datatype t f : … t … -> … g : … -> … t … h : … t … -> … t … f, g, h : extended

Unit Extensibility: The Picture The Functionality Extension datatype t f : … t … -> … g : … -> … t … h : … t … -> … t … j : … t … -> … t. . k : … t … -> … t. . f, g, h : extended

Unit Extensibility: The Picture The Full Component datatype t f : … t … -> … g : … -> … t … h : … t … -> … t … self call j : … t … -> … t. . k : … t … -> … t. . self-reference (has-a) f, g, h : extended

Unit Extensibility: The Experience • Units work well for extensibility • … just re-link for each extension • … each layer around a given component is just an adapter unit (Lieberherr & Lorenz ‘ 01)

Unit Extensibility: The Experience • Units alone are not enough • … need lasses to specify datatypes • … and mixins for extensions

Unit Extensibility: The Experience The Full Component Interfaces don’t know superclass -- mixin datatype t f : … t … -> … g : … -> … t … h : … t … -> … t … f, g, h : extended j : … t … -> … t. . k : … t … -> … t. . Findler & Flatt

Unit Extensibility: The Experience • Linking with units is complex • Units do not support multiple languages • PLT also introduced modules.

Component-Oriented Programming • modules for multi-lingual programs • units for complex linking • mixins for specifying and working with data hierarchies plus implementation inheritance

Components in Languages: Research Problems • still complex: can we simplify this world? • practical experience: – units and modules “feel” inside out • type systems: what will a flexible type system for units and modules look like? – mixins (Krishnamurthi) – sharing by specification (Flatt)

Virtual Machines for COP: Resource Administration with Matthew Flatt, Robby Findler, Shriram Krishnamurthi

Resource Administration: The Situation • a dynamic plug-in component is probably third-party code with unknown properties • several plug-ins may run in parallel • plug-ins compete for finite resources: – files, database servers – network connections – GUI items

Resource Administration: Example 1 • a Web server responds to many requests simultaneously; each may spawn a Web program: – sharing (communicating) with server – sharing among “scripts” • a script consumes – – time memory (references to) server stacks ports, files, databases, …

Resource Administration: Example 2 • Dr. Scheme runs student programs as dynamic extensions so that tools can directly communicate with executing programs • a program “consumes” – – time memory GUI windows, … ports, files, databases, …

Resource Administration: Example 3 • Netscape runs Java applets as dynamic extensions so that … • a program “consumes” – – time memory GUI windows, … ports, files, databases, …

Resource Administration: PLs as OSes • an extensible component system must play OS for its plug-ins • the underlying VM must provide mechanisms for resource control • the underlying programming language must “dress” these mechanisms as language constructs

Resource Administration: PLT’s Solution • dynamic components as threads • threads are a resource • all resources are managed in a custodian

Resource Administration: Custodians current custodian: custodian shutdown thread 3 thread 1 thread 2 network port 1 network port 2 custodian thread 4 GUI 1 file port 3

Resource Administration: Experience • custodians work well in Dr. Scheme and the Web server • “special” custodians for GUI because we need to manage GUI events (event spaces)

Resource Administration: Problems • Custodians don’t deal with memory. (Flatt) • Custodian actions are fixed. Can we specify them? • Other OS services? • Update code dynamically? (Krishnamurthi)

Resource Administration • In the past, advanced languages provided memory administration (garbage collection). Java has introduced GC into the mainstream. • In the future, advanced languages will provide other resource administration tools. . NET will provide meta-data with assemblies.

Enforcing Invariants in COP with Robby Findler, Mario Latendresse

Contract for Components: The Problem The System invariants A Plugin

Contract for Components: The Problem The System consists of many components. The System A Plugin invariants If it blows up, who did it? Economics!

Contracts for Components: The Idea • need behavioral contracts in addition to type signatures for external wires of components • must monitor these assertions at run -time (or partially verify them) • assign blame to faulty component

Contracts for Components: Eiffel • design by contract a la – Eiffel, i. Contract, j. Contractor, Hand. Shake, JVMAssert, … • get it right, scale to other constructs • then scale to COP

Contracts for Components: Example partial behavior: double square_root(double x) { … } @pre: x >= 0 @post: fn result -> result >= 0 total behavior: double square_root(double x) { … } @pre: x >= 0 @post: fn x -> fn result -> x *x >= result >= 0

Contracts for Components: Meaning and Blame • test pre-conditions for method call, blame caller • test post-condition for method return, blame callee • deal with invariants for variables similarly

Contracts for Components: Classes and Objects • classes (or mixins) are in an inheritance hierarchy • method contracts must be related • test implications for pre/post conditions at method call & return, blame class extension

Contracts for Components: Functions • testing predicates on functions is impossible • use proxy functions that perform tests for first-order types, blame current co-variant party for bad arguments and contravariant party for bad result • switch blame positions with applications

Contracts for Components: Experience • contracts are as important as types • contracts complement types: – properties of arguments (subset of types) – relating two arguments – relating two functions in a component

Contracts for Components • OOP and contracts • FP and contracts • COP and contracts: – types – linking: graphs, dynamic, hierarchical

Summary and Advice

Software Systems • build large systems from components – in-house – third party – different sizes • plug-in components statically • plug-in components at run-time

Software Systems • systems have complex interactions • systems act as quasi-OS’es for dynamic plug-ins • such systems consist of components in many languages

Programming Languages • provide VMs that support high-level values and value sharing • provide VMs that support resource administration for non-collectible resources • provide VMs that accommodate many languages

Programming Languages • provide constructs for constructing and linking components of all sizes and all shapes • provide constructs for dynamic extensions, support mini-OSes for COP

Programming Languages • provide interfaces with contracts • check contracts in a well-founded manner • assign blame properly

PLT’s Solutions • modules, units, & mixins are a good solution, but not the last word • custodians are a good start toward resource administration, but not the last word • and this slide isn’t the last word

Advice for Academics: PL Research • • • build systems, not (just) compilers: useful and team-sized, e. g. , Dr. Scheme useful and pair-sized, e. g. , Web server useful individual scripts get “customers” • test academic languages with all kinds of system; use feedback for PL design

Advice for Industry • the JVM was a start; it does not accommodate other languages easily, it does not administrate resources • the. NET world is made for other languages; it still lacks resource administration • these machines are not the last word, they are just a start for COP

now go forth and do some more type research