CS 290 C Formal Models for Web Software

CS 290 C: Formal Models for Web Software Lecture 6: Language and Model-Based Solutions to Navigation Errors Instructor: Tevfik Bultan

Eliminating Navigation Errors • There are several approaches that have been proposed for eliminating navigation errors – Model driven development approaches where the application is specified or enhanced using a formal model • For example: statecharts for modeling navigation – Reverse engineering approaches where a formal model is extracted fro the application • For example: Extracting a state machine model for navigation by analyzing the links that are inserted in web pages

Model Driven Development Approach • Model driven development approach enables – Specification of the behavior of the application at a high level of abstraction, making it easier to develop applications. – The actual implementation can be automatically or semiautomatically generated from the high level models – Separation of concerns can be achieved by specifying different concerns about the application (such as the data model or the navigation constraints) using different specification mechanisms • However, model driven development requires the developers to learn and use the modeling languages • There is a concern about the mapping between the actual implementation and model (they have to maintained together)

Reverse Engineering Approach • Reverse engineering approaches does not require developers to learn a new specification language • Since reverse engineering approaches extract a model directly from the code, there is no maintenance issues (when the application changes, we can extract a new model) • However, reverse engineering is hard: – Extracting sound models using static analysis can lead to very approximate models that do not contain much information or can be undecidable for more precise models – Extracting models by observing runtime behavior is not sound and cannot be used to guarantee correctness

Language Based Approaches • Both model driven development and reverse engineering approaches can be considered software engineering approaches • Another approach would be to use a programming language based approach • Can we model the problems that appear in Web applications in programming language terms and possibly suggest solutions using programming language mechanisms (such as type checking)?

Today I will discuss two approaches • A language based approach for modeling and analyzing navigation problems in Web applications where the navigation problems are resolved using language-based constructs (such as types): “Modeling Web Interactions and Errors, ” S. Krishnamurthi, R. B. Findler, P. Graunke, and M. Felleisen. • A model-driven approach where the navigation problems are addressed using a formal model specifying the navigation behavior and analyzing it: "Eliminating Navigation Errors in Web Applications via Model Checking and Runtime Enforcement of Navigation State Machines. ” Sylvain Halle, Taylor Ettema, Chris Bunch and Tevfik Bultan.

Web Applications • A Web program’s execution consists of a series of interactions between a Web browser and a Web server • When a browser submits a http request whose URL points to a Web program, the server invokes the corresponding program • It then waits from the program to terminate and turns the program’s output into a response that the browser can display, i. e. , it returns a Web page. • Each such program a “script” that reads some inputs and writes some output

Challenges of script oriented programming • This simple request-response style programming using scripts makes design of multi-stage Web interactions difficult • A multi-stage interactive Web program consists of many scripts each handling one request – These scripts communicate with each other via external media since they must remember the earlier part of the interaction – Forcing scripts to communicate this way causes problems since they lead to unstated and easily violated invariants

Web Applications • Use of the Web browser creates further complications – A browser is designed to let a user navigate a web of hyperlinked nodes – When a user uses this power to navigate an interaction with an application many unexpected scenarios can happen • User can backtrack to an earlier stage of the interaction • User can duplicate a page and generate parallel interactions

A Language Based Approach • We will first describe a formal model that captures the essence of Web application behavior • Then we will investigate the use of language based techniques to address the navigation problems

A Formal Model • A Web application (W) consists of – a server (S) and – a client (C) • Server consists of – a storage, and – a dispatcher • Dispatcher contains – a table (P) of programs that associates URLs with programs and – an evaluator that applies programs from the table to the submitted form

A Formal Model • Every page is simply a form (F) that contains – the URL to which the form is submitted, and – a set of form fields • A field name is a value that can be edited by the client • The client stores the – the current form and – the sequence of all the forms that have been visited by the client so far (cached pages)

Web Program Behavior • The behavior of the Web program is described using three types of actions: – Fill-form: This corresponds to client editing values of fields in the current form. The modified form becomes the current form and is added to the cache – Switch: Makes a form from the cache the current form – Submit: dispatches on the current form’s URL to find a program in the table P. This program accesses the server state and the current form and updates the server state and generates a new form which becomes the current form

A Simple Web Programming Language • A simple functional programming language can be specified to characterize the basic operations that are required to write a web application: – Extract a field from a form – Construct a new form – Modify fields of a form • To allow stateful programming we can introduce read and write operations that allow read and write access to the server storage

Navigation Problems • Two navigation problems can be characterized formally in this model: – Script communication problem: Where a script accepts a different type of form than what is delivered to it. For example, the script tries to access a field that does not exist in the form. – HTTP observer problem: Since the http protocol does not allow a proper implementation of the observer pattern (which enables independent observers to be notified of state changes) a page received by the client can become outdated when the data model changes in the server.

Script Communication Problem and Types • The main issue in script communication problem is type mismatch between the forms generated and consumed by different scripts • Since these scripts are loosely coupled programs, there is no standard type checking mechanism that can be used to make sure that these type mismatches do not happen • Checking all scripts together is not feasible since they are developed incrementally and may reside on different Web servers and may be written using different programming languages

An Incremental Type System for Web Applications • The proposed solution is the following: – When the Web server receives a request for a URL that is not already in its table, it installs the relevant program – Before installing the relevant program it checks that there is no type mismatch with the input form and the installed program (internal consistency check) – Furthermore it generates type constraints that this new installed program imposes on other programs in the server that it interacts (these become external consistency checks) • If either the internal or external check fails the program is rejected resulting in an error

A Simple Typed Web Programming Language • The simple functional Web programming language can be extended with types by requiring type declarations for function arguments • The type system for this language shows how external type checking can be done – While traversing the program, the type system generates a set of type constraints on external programs – Each constraints state a condition such as: a program associated with a particular URL should consume Web forms of a particular type

Solving Script Communication Problem with Type Checking • Using type checking with this incremental system it can be guaranteed that – scripts do not get stuck when they are processing appropriately typed forms – Server does not apply the scripts to forms with wrong types

Solving the http observer problem with timestamps • Server keeps track of the number of processed submissions (this represent time) • The external storage is changed so that it maps locations to values + timestamp for the last write • The server also maintains the set of all storage locations read or written during the execution of a script (called a carrier set CS) – When sever sends a page to the consumer, it adds the current time stamp and this set of locations as an extra hidden field

Solving the http observer problem with timestamps • A form with carrier set CS and time stamp T submitted to a server is out of date if and only if any of the locations in CS have a timestamp at the server that is greater than T • A runtime error can be generated when out of date forms are submitted preventing execution of scripts with out of date data – This approach solves the Orbitx problem of booking an unintended flight • However, this approach can also generate false positives (for example a page counter value may make the form out of date) – So the programmers must specify which reads or writes are relevant, and an error is generated only when a relevant field is out of date

Modeling web application behavior with continuations • Another language-based approach that has been investigated in web application development is the use of continuations for modeling web application behavior • A “continuation” is an abstract representation of the control state of a program • In the continuation-passing-style of programming the control is passed explicitly using continuations – When invoking a function written using the continuationpassing-style, the caller function passes a continuation that will be invoked with the return value of the callee after the callee terminates

Modeling web application behavior with continuations • Using continuations we do not have to think of a web application as a collection of scripts • Using continuations we can capture the behavior of a web application as a single program that suspends its behavior while interacting with the user • When a web application is invoked by submitting a form, after it performs its task, it outputs the result and a continuation – This continuation then is used to process the next form submission

Modeling web application behavior with continuations • In the continuation-based model, when a page is sent to the user, the current “continuation” is captured and stored in a table. – The form sent to the user contains a URL that contains a reference to that table entry – When user submits the form, the server invokes the corresponding continuation which then continues the execution from the corresponding control location • If you are interested in this topic here is a paper that discusses this view: “The Influence of Browsers on Evaluators or, Continuations to Program Web Servers, ” Christian Queinnec.

Modeling web application behavior with continuations • Using this continuation-based approach, one can investigate the effects of using the Back button, multiple window creation, direct URL entry, etc. in a web application. • If you are interested in this topic here is a paper that discusses this view: “The Influence of Browsers on Evaluators or, Continuations to Program Web Servers, ” Christian Queinnec.

A model-based approach to navigation problems • We have discussed some language-based ideas for dealing with navigation problems in web applications, now we will discuss a model-based approach • One successful approach to web application development has been adoption of design patterns that bring some structure to the scripts that implement the web application • Web application development frameworks that adopt these design patterns have become very successful

Model-View-Controller (MVC) Architecture • MVC is a design structure for separating representation from presentation using a subscribe/notify protocol • The basic idea is to separate – where and how data (or more generally some state) is stored, i. e. , the model – from how it is presented, i. e. , the views • Follows basic software engineering principles: – Separation of concerns – Abstraction

Model-View-Controller (MVC) Architecture • MVC consists of three kinds of objects – Model is the application object – View is its screen presentation – Controller defines the way the user interface reacts to user input a=50% b=30% c=20% model views

Model-View-Controller (MVC) Architecture • MVC decouples views and models by establishing a subscribe/notify protocol between them – whenever model changes it notifies the views that depend on it – in response each view gets an opportunity to update itself • This architecture allows you to attach multiple views to a model – it is possible to create new views for a model without rewriting it

Model-View-Controller (MVC) Architecture • Taken at face value this may be seen as an architecture for user interface design – It is actually addresses a more general problem: • decoupling objects so that changes to one can affect any number of others without requiring the changed object to know the details of the others – This is called Observer pattern in the design patterns catalog • Observer pattern is a design pattern that is used as part of the Model-View-Controller (MVC) architecture to handle notification of multiple views that depend on a single model

A Brief Overview of Design Patterns • Think about the common data structures you learned – Trees, Stacks, Queues, etc. • These data structures provide a set of tools on how to organize data • Probably you implement them slightly differently in different projects

A Brief Overview of Design Patterns • Main concepts about these data structures, such as – how to store them – manipulation algorithms are well understood • You can easily communicate these data structures to another software developer by just stating their name • Knowing them helps you when you are dealing with data organization in your software projects – Better than re-inventing the wheel

A Brief Overview of Design Patterns • This is the question: – Are there common ideas in architectural design of software that we can learn (and give a name to) so that • We can communicate them to other software developers • We can use them in architectural design in a lot of different contexts (rather than re-inventing the wheel) • The answer is yes according to E. Gamma, R. Helm, R. Johnson, J. Vlissides – They developed a catalog of design patterns that are common in object oriented software design

A Brief Overview of Design Patterns • Design patterns provide a mechanism for expressing common design structures • Design patterns identify, name and abstract common themes in software design • Design patterns can be considered micro architectures that contribute to overall system architecture • Design patterns are helpful – In developing a design – In communicating the design – In understanding a design

A Brief Overview of Design Patterns • The origins of design patterns are in architecture (not in software architecture) • Christopher Alexander, a professor of architecture at UC Berkeley, developed a pattern language for expressing common architectural patterns • Work of Christopher Alexander inspired the work of Gamma et al. • In explaining the patterns for architecture, Christopher Alexander says: “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice” • These comments also apply to software design patterns

Resources for Design Patterns • Original paper: – “Design Patterns: Abstraction and Reuse of Object. Oriented Design” by E. Gamma, R. Helm, R. Johnson, J. Vlissides • Later, same authors published a book which contains an extensive catalog of design patterns: – “Design Patterns: Elements of Reusable Object-Oriented Software”, by E. Gamma, R. Helm, R. Johnson, J. Vlissides, Addison-Wesley, ISBN 0 -201 -63361 -2

Cataloging Design Patterns • Gamma et al. present: – A way to describe design patterns – A way to organize design patterns by giving a classification system • More importantly, in their book on design patterns, the authors give a catalog of design patterns – As a typical developer you can use patterns from this catalog – If you are a good developer you can contribute to the catalog by discovering and reporting new patterns • The template for describing design patterns used by Gamma et al. is given in the next slide

Design Pattern Template DESIGN PATTERN NAME the name should convey pattern’s essence succinctly Jurisdiction Characterization used for categorization Intent What particular design issue or problem does the design pattern address? Motivation A scenario in which the pattern is applicable. This will make it easier to understand the more abstract description that follows. Applicability What are the situations the design pattern can be applied? Participants Describe the classes and/or objects participating in the design pattern and their responsibilities. Collaborations Describe how the participants collaborate to carry out their responsibilities. Diagram A class diagram representation of the pattern (extended with pseudo-code). Consequences What are the trade-offs and results of using the pattern? Implementation What pitfalls, hints, or techniques should one be aware of when implementing the pattern? Examples of applications of the pattern in real systems. See Also What are the related patterns and what are their differences?

Observer Pattern • Observer pattern is a design pattern based on Model-View. Controller (MVC) architecture • In the next slides, I will give the design pattern catalog entry for the Observer pattern.

Observer Behavioral Intent The Observer pattern defines an one-to-many dependency between a subject object and any number of observer objects so that when the subject object changes state, all its observer objects are notified and updated automatically. Motivation The Observer design pattern has two parts and they are subject and observer. The relationship between subject and observer is one-to-many. In order to reuse subject and observer independently, their relationship has to be decoupled. An example of using the observer pattern is the graphical interface toolkit which separates the presentational aspect with application data. The presentation aspect is the observer part and the application data aspect is the subject part. For example, in a spreadsheet program, the Observer pattern can be applied to separate the spreadsheet data from its different views. In one view spreadsheet data can be presented as a bar graph and in another view it can be represented as a pie chart. The spread sheet data object notifies the observers whenever a there is a data change that can make its state inconsistent with its observers.

Class Diagram for the Observer Pattern Subject observers Attach(Observer) Detach(Observer) Notify() Observer Update() for all o in observers { o->Update(); } Concrete. Observer Concrete. Subject observer. State subject. State Update() Get. State() Set. State() return subject. State; observer. State = subject->Get. State();

: Conrete. Subject a: Concrete. Observer b: Concrete. Observer Set. State() Notify() Update() Get. State()

Applicability Use the observer pattern in any of the following situations: • When the abstraction has two aspects with one dependent on the other. Encapsulating these aspects in separate objects will increase the chance to reuse them independently. • When the subject object doesn't know exactly how many observer objects it has. • When the subject object should be able to notify it's observer objects without knowing who these objects are. Participants • Subject • Knows it observers • Has any number of observer • Provides an interface to attach and detaching observer object at run time • Concrete. Subject • Store subject state interested by observer • Send notification to it's observer • Observer • Provides an update interface to receive signal from subject • Concrete. Observer • Maintain reference to a Concrete. Subject object • Maintain observer state • Implement update operation

Consequences Further benefit and drawback of Observe pattern include: • Abstract coupling between subject and observer, each can be extended and reused individually. • Dynamic relationship between subject and observer, such relationship can be established at run time. This gives a lot more programming flexibility. • Support for broadcast communication. The notification is broadcast automatically to all interested objects that subscribed to it. • Unexpected updates. Observes have no knowledge of each other and blind to the cost of changing in subject. With the dynamic relationship between subject and observers, the update dependency can be hard to track down. Known Uses • Smalltalk Model/View/Controller (MVC). User interface framework while Model is subject and View is observer.

Back to MVC How do the MVC architecture and the Observer pattern relate to Web applications? • The reason we are discussing the MVC architecture is that many Web applications nowadays are built based on the MVC architecture • The reason we are discussing the Observer pattern is that the MVC-based Web applications doe not properly use the Observer pattern, causing problems

MVC Architecture in Web Applications • Many web frameworks support web application development based on the MVC architecture – Ruby on Rails, Zend Framework for PHP, Cake. PHP, Spring Framework for Java, Struts Framework for Java, Django for Python, … • MVC architecture has become the standard way to structure web applications

MVC Framework for Web Applications • Use of MVC architecture in Web applications – Model: This is the data model which is an abstract representation of the data stored in the backend database. Typically uses an object-relational mapping to map the class structure for the data model to the tables in the back-send database – Views: These are responsible for rendering of the web pages, i. e. , how is the data presented in user’s browser – Controllers: Controllers are basically event handlers that process incoming user requests. Based on a user request, they can update the data model, and create a new view to be presented to the user

MVC Framework for Web Applications • Note that use of MVC in web applications does not fit the Observer pattern – typically it is not possible to refresh a browser window directly when the data model changes (i. e. , it is not possible to actively notify the observers when the state of the subject has changed) • This can create navigation problems – when there are multiple windows open, they may represent stale views – this was the problem in the orbitz example we discussed earlier

Abstraction in MVC Frameworks • MVC framework provides separation of concerns and abstraction, which can be exploited for analysis – For example, for analyzing properties of the data model we can focus on the data model and ignore the views – We can focus on the behaviors of the controllers to eliminate navigation errors

Achieving Navigation Correctness in MVC • I will discuss some work we have done recently on analyzing navigation behavior in web applications developed using MVC frameworks • The idea is – to exploit the abstraction provided by the MVC architecture by enforcing navigation constraints at the controller – use model driven development to provide a navigation model and analyze it statically – enforce the navigation model synamically at runtime to prevent navigation errors

Request processing in a Web application

A formal model We can formally model an MVC application as • M: is a set of data model states, where the data model can include any stateful representation of application data, such as a database, • V: is a set of session variables, i. e. , data stored on the server on a per-client basis, • I: is a set of sessions used by the server to associate clients with session variables, • A: is a set of controller actions, i. e. , the program segments that are invoked based on the HTTP requests sent by the user, • P: is a set of request parameters, i. e. , input data from the user received as part of the HTTP requests via GET or POST.

A formal model • A web application is a tuple A = (Q, B, T) where: – Q is the set of states, which is the Cartesian product of the model states and the domains of the session variables for each session – B is the set of initial states – T is the transition relation mapping a state, an action, a set of request parameters and a session to a next state • The transition relation must guarantee that each session can only modify its own session variables • The model can change even when there is no action executed (i. e. , the backend database contents can change without a request from a user)

Session traces • Given the formal model, we can define global execution traces of a web application – Each trace starts from an initial state – Each element in the trace is a tuple: (state, action, request parameters, session index) – Any two consecutive tuples in the trace must be consistent with the transition relation of the application • We can project each global trace to a session i (by deleting the tuples which do not contain i) and obtain a session trace

Navigation state machines (NSMs) • A navigation state machine (NSM) is a state machine – that specifies acceptable sequences of actions and request parameters that can appear in a session trace • Given a navigation state machine, and a session trace, – the session trace conforms to the navigation state machine • if the sequence of actions and request parameters for that session trace is accepted by the navigation state machine

Navigation state machines • The states of a navigation state machine is defined by – the values of the session variables, – the last action executed by the application – And the request parameters that were sent with the last action • We assume that this information is enough to figure out what are the next actions that can be executed by the application

Navigation state machines • We developed a simple language to specify navigation state machines • It is a state machine that shows the allowable sequences of controller action executions in a web application • MVC frameworks typically use a hierarchical structure where actions are combined in the controllers and controllers are grouped into modules – We exploit this hierarch to specify the navigation state machines as hierarchical state machines (statecharts)

Navigation state machines • In addition to identifying which action can be executed after which other action, – navigation state machines also identify constraints among the request parameters between two consecutive requests – This can be used to make sure that the values stored in cookies are not changes for example

Navigation state machine example • A portion of the navigation state machine for the Digitalus system (an open source content management system)

What can we do with NSMs • If we can check that the web application conforms to the NSM, – then we can verify navigation properties on the NSM and conclude that the navigation properties hold for the application – We can use model checking to check properties of NSMs – This way we can eliminate the navigation errors • Big problem: How do we ensure that the application conforms to the NSM?

Runtime Enforcement • Statically verifying that a web application conforms to a navigation state machine is a very difficult problem (in general undecidable) • So, instead, we use runtime enforcement – We have a plugin that can be easily added to an MVC web application that takes a NSM as input and makes sure that every incoming request conforms to the NSM – If the incoming request does not obey the NSM, then the plugin either ignores the request and refreshes the previous page or generates an appropriate error message – This way non-compliant user requests can be handled uniformly without generating strange error messages

Model checking NSMs • We translate NSM models to SMV • We write navigation constraints in temporal logic: G (login => (!login U logout)) • We check the properties using a model checker

Model checking and Runtime Enforcement • Our approach combines the following ideas: – Using model driven development for specification of navigation constraints – Using model checking to verify properties of formal navigation models – Using runtime enforcement to ensure that the navigation behavior at runtime obeys the navigation model • We show that when all these are combined, navigation errors can be eliminated