- Количество слайдов: 22
Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman for his support
Motivation Calibration Manual Calibration Automated Calibration (e. g. Vensim, Powersim)
Motivation • Once model parameters are estimated with automated calibration, next step: Estimate confidence intervals! • Questions: -Are there available tools at software packages? -Do these methods have any limitations? -Are there alternative methods?
Why are confidence intervals important We reject the claim that the parameter value is equal to 0 (with 95% probability) We can’t reject the claim that the parameter value is equal to 0 (with 95% probability) 95% Confidence Interval 0 Parameter Estimate θ
How can we estimate confidence intervals? Used in the System Dynamics Software (Vensim) /Literature Likelihood Ratio Method The method we suggest for System Dynamics models Bootstrapping Both methods yield approximate confidence intervals!
Likelihood Ratio Method • The likelihood ratio method is used in system dynamics software packages (Vensim) and literature (Oliva and Sterman, 2001). • It relies on asymptotic theory (large sample assumption).
However Likelihood Ratio Method At system dynamics (as it is used at software models: packages) assumes: -Large Sample -It is not always possible to have large sample -No feedback (autocorrelation) -There are many feedback loops -Normally distributed error terms -Error terms are not always normally distributed
Bootstrapping • Introduced by Efron (1979) and based on resampling. Extensive survey in Li and Maddala (1996). • It seems more appropriate for system dynamics models because - It doesn’t require large sample - It is applicable when there is feedback (autocorrelation) - It doesn’t assume normally distributed error terms
Drawbacks of bootstrapping • The software packages do not implement it. • It is time consuming.
Bootstrapping Fit the model and estimate parameters Compute the Error Terms
Bootstrapping uses resampling Nonparametric: Reshuffle Them and Generate many new error term sets using the autocorrelation information Parametric: Fit a distribution and Generate many new error term sets using the autocorrelation and distribution information
Resampling the Error Terms • If we know that: - The error terms are autocorrelated - Their variance is not constant (heteroskedasticity) - They are not normally distributed => We can use this information while resampling the error terms • Flexibility of bootstrapping stems from this stage
FABRICATED ERROR TERMS + FABRICATED “HISTORICAL” DATA =. . .
FABRICATED “HISTORICAL” DATA Fit the model and estimate parameters Parameter Estimate . . . Parameter Estimate 500 Parameter Estimates Fit the model and estimate parameters
Distribution of a model parameter
Experiments • We had experimental time series data from 240 subjects. • Subjects were beer game players. • For each subject we had 48 data points, so we estimated parameters and confidence intervals using 48 data points.
Model (Same as Sterman 1989) • Ot = Max[0, θLRt + (1–θ)ELt + α(S' – St –βSLt) + error termt] Parameters to be estimated are θ, α, β, S‘
Individual Results 95% Confidence Intervals for θ Likelihood Ratio Method 0 95% CI 1 0. 77 θ=0. 95 Bootstrapping 0 0. 01 95% CI 1 θ=0. 95
Individual Results 95% Confidence Intervals for β Likelihood Ratio Method 0 95% CI 0. 2 β =0. 01 Bootstrapping 95% CI 0 β =0. 01 0. 2 Significantly Different From 0!!!
Overall Results Average 95% Confidence Interval Length Theta Alpha Beta S-Prime Likelihood Ratio Method 0. 19 0. 11 13. 20 Bootstrapping 0. 67 0. 30 0. 52 973. 59 Median of 95% Confidence Interval Length Theta Alpha Beta S-Prime Likelihood Ratio Method 0. 10 0. 08 0. 06 2. 32 Bootstrapping 0. 84 0. 24 0. 48 10. 10
Overall Results Percentage of Subjects for whom the bootstrapping confidence interval is wider than the likelihood ratio method confidence interval Theta Bootstrapping CI wider than Likelihood Ratio Method CI Alpha Beta S-Prime 97. 76% 98. 81% 100% 98. 56%
Likelihood Ratio Method vs Bootstrapping • Likelihood Ratio Method: • Bootstrapping: § Is easy to compute § Very fast § BUT depends on assumptions that are usually violated by system dynamics models § Yields very tight confidence intervals § Is NOT easy to compute § Takes longer time § DOES NOT depend on assumptions that are usually violated by system dynamics models § Yields larger confidence intervals. Usually more conservative.