Auto Sim OA A Framework for Automated Analysis

Auto. Sim. OA: A Framework for Automated Analysis of Simulation Output Stewart Robinson (stewart. robinson@warwick. ac. uk), Katy Hoad, Ruth Davies Funded by EPSRC and SIMUL 8 Corporation

The Warwick Simulation Research Group 7 members of staff 2 research fellows 4 Ph. D students Focus on the practice and application of simulation methods DES SD ABS

The Warwick Simulation Research Group Recent/current projects: • • Comparison of DES and SD model building Agent based modelling of social networks Effect of model reuse on learning Conceptual modelling for DES Agent based modelling for service systems Human interactions in supply chains Simulation and lean in the health service …

The Problem • Prevalence of simulation software: ‘easy-to -develop’ models and use by non-experts. • Simulation software generally have very limited facilities for directing/advising user how to run the model to get accurate estimates of performance. • With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

Aim • To develop an automated output analysis system that can be implemented in commercial simulation software with a view to improving the use of simulation, particularly by non-expert simulation users.

More formally… • To develop an automated procedure that obtains unbiased estimators (of specified precision) for the population mean and variance (μ and σ2 respectively) for one or more simulation output statistics.

Transient Simulation Output

Steady-State Simulation Output

3 Main Decisions • How long a warm-up is needed? • How many replications should be run? • How long a run length is needed?

Work Carried Out for Auto. Sim. OA Project • Classification of different model types and output data properties. • Extensive testing of replications algorithm. • Literature review of (44) warm-up methods. • Tested MSER-5 to destruction using over 3000 data sets. • Literature review of batch means methods. • Development of Auto. Sim. OA.

Enter Analyser Auto. Sim. OA Replications or a single run? Replications Single run Warm-up? No Replications Calculator Warm-up? Yes Warm-up Analyser EXIT Analyser No Single Run Analyser

Replications Calculator

Confidence Interval Method with ‘Look-ahead’ Precision > 5% Precision ≤ 5% 95% confidence limits Precision ≤ 5% Cumulative mean, f(k. Limit) Nsol 1 Nsol 2 + f(k. Limit)

Warm-up Analyser • MSER-5 most promising method for automation – Performs robustly and effectively for the majority of data sets tested. – Not model or data type specific. – No estimation of parameters needed. – Can function without user intervention. – Quick to run. – Fairly simple to understand.

Dealing with Initialisation Bias Warm-up Period: MSER-5 Heuristic Minimises mean squared error of output data. Performs analysis on batch mean data – batch size of 5. MSER-5 value calculated as follows:

Dealing with Initialisation Bias Warm-up Period: MSER-5 Heuristic

Heuristic framework around MSER-5 Adaptation in to a sequential procedure: • Iterative procedure for procuring more data when required. • ‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated. • Graphical feedback to user.

Single Run Analyser There are 3 possibilites: 1. User wants a mean estimate with a CI of a specific precision. 2. User has a specific run-length in mind & wants a mean estimate with a valid CI at end of run (i. e. no precision requirement). 3. User neither requires a specific precision nor has a set run length in mind.

SINGLE RUN ANALYSER Use set runlength? NO YES Batch Means Calculator Run-length Calculator ASAP 3 (Steiger et al, 2005) Abort LABATCH 2 (Fishman, 1998)

Example Implementation of Auto. Sim. OA Data: • ‘user support model’ - simulates calls received, processed and actioned at an IT support help desk (Robinson, 2001). • Output of interest = average time calls spend in the system. • Steady-state output with a substantial initial bias. • True steady-state mean estimated as 2, 269 mins (using a long run with 54, 000 data points).

Implementation Issues • Output data type – What should and should not be analysed? – Cumulative values, extreme values – Time or entity data • Multiple outputs – Analyse all outputs of interest to user. • Multiple scenarios – Run for all scenarios? Run for just the base case? – Issues regarding run length with ASAP 3.

Automation Issues • Generation of more data when required. – Run simulation from present termination point. • Single run vs replications. User involvement: • User decision of ‘what to do’- based on knowledge of nature of model & output. – Warm-up needed? Multiple replications? – One run? Length of run for replications? • Determining if recommendations are reasonable – Graphical aids.

Limitations of Auto. Sim. OA • Not directly able to handle cyclic data. • Unable to analyse warm-up for transient output data subject to initialisation bias. • Only performs an analysis on the mean and variance of the output statistics of interest. – Median, mode, quantiles, … • Provides no facilities for scenario analysis. – Ranking and selection, optimisation, …

ACKNOWLEDGMENTS This work is part of the Automating Simulation Output Analysis (Auto. Sim. OA) project (http: //www. wbs. ac. uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D 033640/1). The work is being carried out in collaboration with SIMUL 8 Corporation, who are also providing sponsorship for the project. Stewart Robinson Warwick Business School Brunel DISC Seminar December 2009