NEW CHALLENGES IN HYDROLOGIC MODELING Prof Vadim Kuzmin

NEW CHALLENGES IN HYDROLOGIC MODELING Prof. Vadim Kuzmin Head of Department of Hydrogeology and Geodesy, Russian State Hydrometeorological University St Petersburg, Russia www. rshu. ru E-mail: vknoaa@hotmail. com Hanoi, Vietnam, 2012

MY CV • 1992: MSc in Hydrology • 1992 -1996: Ph. D student at RSHU • 1996: MSc in Statistics and Probability • 1997 -1998: Head of Lab of Land Hydrology • 1998 -2002: Associate Professor at RSHU • 2000, 2001: Visiting scientist at TU Delft • 2002 -2006: NOAA National Weather Service • 2006 -2008: Australian Bureau of Meteorology • 2008 -2010: Associate Professor at RSHU, project manager • 2010 -2012: Professor, Head of Dept of Hydrogeology and Geodesy at RSHU; project manager; WMO instructor and lecturer in Hydrology.

WHAT I DID ABROAD • Delft, Holland: enhanced streamflow forecasting, hydrological response to the global change and human impact • USA: automatic calibration of the Sacramento Soil Moisture Accounting model • Australia: improving streamflow forecasts by integrating RS, NWP and in situ data and model-data assimilation

WHEN I CAME BACK HOME… • RUSSIA: ~2, 600, 000 streams • Up to 80% of them are poorly gauged or ungauged (1/3 of population, 94% of gas and oil extraction) • Global change: “old-fashioned”/classic forecasting methods are inefficient

WHAT WE DO AT RSHU • Automatic streamflow/flood forecasting systems • Improving stream flow forecasting by integrating satellite observations, radar data, NWP model output data, in situ data and catchment models using model-data assimilation methods • Systems of customer-oriented hydrometeorological support (Russian Railways, hydropower industry etc) • Automatic systems of environmental monitoring

OUR SPONSORS • Ministry of Education and Science • Foreign foundations V

OUR CUSTOMERS • Russian Railways • Hydropower industry • Emergency Ministry • Oil and Gas exporters • Sea ports, etc.

NEW CHALLENGES IN HYDROLOGIC MODELING Lecture 1. AUTOMATED STREAMFLOW FORECASTING: BASIC PRINCIPLES Hanoi, Vietnam, 2012

BASIC PRINCIPLES OF FORECASTING 1. Forecasting should be profitable! 2. Forecasting should be feasible and reliable! 3. Forecasts should be realistic and informative under any conditions! 4. Decision makers should be able to understand forecasts!

1. FORECASTING SHOULD BE PROFITABLE Forecasting should be profitable for: • Customers • Forecasters • Society

2. FEASIBILITY AND RELIABILITY Forecasting should be feasible and reliable for customers! Otherwise, they will not be able to use them for any actions!

2. FEASIBILITY AND RELIABILITY Forecasting should be feasible and reliable for customers! Otherwise, they will not be able to use them for any actions! — Hey, what about forecasts uncertainty?

2. FEASIBILITY AND RELIABILITY — Well, have you been thinking of probabilistic forecasting? • Ensemble forecasting (Poor Man’s forecasts) • Forecasting PDF • Mean/median/mode; variance/range

2. FEASIBILITY AND RELIABILITY • Reliability of input data • Feasibility and reliability of a model • Feasibility, reliability and stability of the model parameters • Feasibility of the model output

3. FORECASTS SHOULD BE REALISTIC AND INFORMATIVE Forecasts should be realistic and informative under any conditions! Special attention should be paid to extreme events and initial conditions before extreme events!

3. FORECASTS SHOULD BE REALISTIC AND INFORMATIVE

4. FORECASTS SHOULD BE UNDERSTANDABLE Decision makers should be able to understand forecasts! In most cases, they don’t really need to see your forecasts. They want to know what they should do!

4. FORECASTS SHOULD BE UNDERSTANDABLE Decision makers should be able to understand forecasts! In most cases, they don’t really need to see your forecasts. They want to know what they should do!

4. FORECASTS SHOULD BE UNDERSTANDABLE 3 Q=3250 m/sec V 1) Build a 2 m dam 2) Evacuate people from A, B, C… V 3) Run 4) Pray

FORECASTS SHOULD BE CUSTOMER-ORIENTED!

FORECASTS SHOULD BE CUSTOMER-ORIENTED!

FORECASTS SHOULD BE CUSTOMER-ORIENTED!

FORECASTS SHOULD BE CUSTOMER-ORIENTED! 1) Hydropower Station: without forecasts 2) Hydropower Station: with forecasts

SCHEME OF AUTOMATED FORECASTING SYSTEMS FORCING DATA Satellite data, radar data, NWP model output and in situ observation STREAMFLOW MODELING & FORECASTING DECISION-MAKING SUPPORT Automatic model selection Predefined Decision Automatic calibration Automatic data acquisition and processing Streamflow observations Automatic validation Automatic forecasting Forecast postprocessing Visualization Dissemination of information Feedback Customer service

FORECASTING STRATEGY • Two-level forecasting: 1) Background streamflow/flood forecasting in large basins (exceeding thresholds → risk assessment) 2) Enhanced flood forecasting in catchments where risk was increased • Automated decision support (PREDEFINED DECISION©)

AUTOMATIC FLOOD FORECASTING SYSTEMS: BACKGROUND FORECASTING • Quite qualitative… • Large areas are covered • Saves resources • Simple and robust , yet reliable models are used

Statistical models Conceptual models Physically based models Increasing complexity Increasing sensitivity to data quality Increasing sensitivity to the process stability Increasing role of the process understanding Increasing sensitivity to the calibration quality

NEW CHALLENGES IN HYDROLOGIC MODELING Lecture 2. AUTOMATED STREAMFLOW FORECASTING: AUTOMATED DATA PROCESSING Hanoi, Vietnam, 2012

DATA SOURSES The available data sources for hydrologic modeldata assimilation are divided here into three categories: • REMOTE SENSING: satellite & radar data • NWP model output • IN SITU: various surface observations

REMOTE SENSING DATA

NWP PRODUCTS • WRF • GASP • LAPS • etc

IN SITU DATA • Surface observational networks • Expeditions • Volunteers • Customer’s networks

BASIC STRATEGY • To integrate data obtained from different sources • To compose complete data arrays regardless availability surface observations • To force a model…

WHAT WE MEAN BY DATA PROCESSING • Quality analysis and quality control (QA/QC) • Filling gaps • Editing (including correcting typos and errors) • Rescaling (upscaling, downscaling) • Data assimilation • Rewriting to a proper format • Archiving (original and edited data) • Documenting…

QUALITY ANALYSIS AND QUALITY CONTROL • Automatic determination of data properties and sources • Detection of typos • Identification of outliers • Computation of stochastic characteristics (statistics / moments etc)

FILLING GAPS • Using interpolation • Using a model • Using statistical relationships with neighboring gauges or stations

DATA EDITING Fully automated data editing still remains a HUGE problem, while this procedure is not efficient enough. In most cases, an expert’s action is required.

DATA ASSIMILATION Model-data assimilation describes a set of mathematical algorithms used to optimally combine the information contained in models and observations. The underlying principle is to minimise the mismatch between model predictions and observations through direct adjustment of model states (‘state estimation’) or through adjustment of model parameters (‘parameter estimation’). The output from an assimilation scheme is estimates of ‘target variables’ of the model (state variables, parameters and/or fluxes) obtained when the mismatch between model and observations is minimal. While simple in principle, there are numerous difficulties to overcome. However, the outcome of implementing a successful assimilation scheme is a quantifiable improvement in predictive capability even in systems dominated by chaotic processes.

DATA RESCALING Transformation of data spatial and temporal resolution into required size.

PROPER FORMAT Data rewriting into a proper format is a simple technical procedure aimed to keep all available data in the same format, so that they could be directly read by a model.

DATA ARCHIVING Both original and edited data should be archived and kept at least in THREE different places!

DOCUMENTING All corrections made both manually and automatically should by recorded in a special text file (a protocol).

DATA ASSIMILATION Model-data assimilation describes a set of mathematical algorithms used to optimally combine the information contained in models and observations. The underlying principle is to minimise the mismatch between model predictions and observations through direct adjustment of model states (‘state estimation’) or through adjustment of model parameters (‘parameter estimation’). The output from an assimilation scheme is estimates of ‘target variables’ of the model (state variables, parameters and/or fluxes) obtained when the mismatch between model and observations is minimal. While simple in principle, there are numerous difficulties to overcome. However, the outcome of implementing a successful assimilation scheme is a quantifiable improvement in predictive capability even in systems dominated by chaotic processes.

BASIC LITERATURE • WMO Guide to Hydrological Practices, No. 168 • Barrett D. J. , Kuzmin V. A. , Walker J. P. , Mc. Vicar T. R. and Draper C. 2008. Improving stream flow forecasting by integrating satellite observations, in situ data and catchment models using model-data assimilation methods. e. Water Technical Report. e. Water CRC, Canberra. http: //ewatercrc. com. au/reports/Barrett_et_al 2008 -Flow_Forecasting. pdf

NEW CHALLENGES IN HYDROLOGIC MODELING Lecture 3. TOOLS FOR AUTOMATED STREAMFLOW FORECASTING Hanoi, Vietnam, 2012

POTENTIALLY SUITABLE MODELS All models are wrong, but some of them are useful!

WHAT IS THE BEST MODEL? M(Parameters, Time, Place, Event) → max [Efficiency]

POTENTIALLY SUITABLE MODELS

THE SACRAMENTO SOIL MOISTURE ACCOUNTING MODEL (USA)

THE SACRAMENTO SOIL MOISTURE ACCOUNTING MODEL (USA) R. J. C. Burnash, R. L. Ferral, Mc. Guire R. A. A generalized streamflow simulation system – conceptual modeling for digital computers [Text] / // Joint Federal and State River Forecast Center. US NWS and California DWR. – Technical Report. – 1973. – Sacramento, CA. – 204 p.

AUSTRALIAN WATER BALANCE MODEL (AWBM)

‘SIMPLIFIED HYDROGRAPH’ MODEL (SIMHYD)

The Soil Moisture Accounting Runoff model (SMAR)

VARIABLE INFILTRATION CAPACITY MACROSCALE HYDROLOGIC MODEL (VIC)

PROBABILITY DISTRIBUTED MODEL (PDM)

MODELING TOOLKITS An example: E 2 modeling toolkit http: //www. toolkit. net. au/cgibin/Web. Objects/toolkit HEC

SECTION SUMMARY (1) In this section, we have noted that the hydrological models used for stream flow forecasting must be capable of simulating the dynamics of infiltration, runoff, flows, and evapotranspiration with minimal complexity, using forcing data and parameters acquired at scales appropriate to model application.

SECTION SUMMARY (2) Note that all the models included into E 2 modelling toolkit can be useful for a single basin or training purposes, but they cannot be used for the background forecasting, because the toolkit does not provide any tools of the model automatic calibration.

BACKGROUND FORECASTING: HYDROLOGICAL MODELS • The SACRAMENTO soil moisture accounting model (Burnash, 1973 (USA)): 11— 16 parameters • The Multi-Layer Conceptual Model (MLCM) (Kuzmin, 2010 (Russia)) – 2 N+4 parameters (0— 20…)

MULTI-LAYER CONCEPTUAL MODEL (MLCM) 2 N+4 parameters

MULTI-LAYER CONCEPTUAL MODEL (MLCM) 1) Liquid precipitation on the catchment surface, which then comes into soil. (In case if the snowmelting process affects the runoff generation, an equivalent amount of liquid water can be computed, for instance, by using SNOW-17 program, usually used together with SAC-SMA). Please note that, for background forecasting flash floods, precipitation layer can be set by using the WRF model output, while for improved forecasting radar data and data collected in the local network of automatic weather stations are required.

MULTI-LAYER CONCEPTUAL MODEL (MLCM) 2) Surface detention of water in micro- and mesodepressions and direct surface runoff are reflected through optimizing the rate of the surface runoff α 0; 3)Water losses associated with evapotranspiration, including direct evaporation from the surface of streams and water bodies, and water consumption by aquatic plants (at the present time, total evaporation is defined by using the WRF model output; besides, it can be estimated from the remote sensing observations);

MULTI-LAYER CONCEPTUAL MODEL (MLCM) 4) Water percolation and further propagation into channel through N soil layers with thicknesses Zi (at present, percolation rates αi withing each layer are constant; the rate of vertical percolation equals to the module of percolation rate within the layer); propagation of ground water within the layers is described with Darcy equation; 5) Flood wave propagation along the river channel described by the kinematic wave or Muskingum-Cunge schemes.

MULTI-LAYER CONCEPTUAL MODEL (MLCM) A total number of parameters depends on a number of layers: 2 N+4 N=0 — 4 parameters N=1 — 6 parameters N=2 — 8 parameters N=3 — 10 parameters N=4 — 12 parameters N=5 — 14 parameters, etc.

MULTI-LAYER CONCEPTUAL MODEL: ADVANTAGES • Simplicity • Flexible structure • Easy to calibrate • A state variable for the top soil layer can be set • Can be integrated with other models

MLCM: SOFTWARE FOR ONLINE MODELING

MLCM: SOFTWARE FOR ONLINE MODELING • ‘Cloud’ technologies • Available in the Internet • Suitable forecasting any types of floods • Can be used for operational or training purposes • Can be incorporated in any automatic forecasting systems • Can be supplied with “Predefined Decision” © (automatic decision making support)

MLCM SOFTWARE FOR ONLINE MODELING AND FORECASTING WILL BE AVAILABLE AT WWW. RSHU. RU

NEW CHALLENGES IN HYDROLOGIC MODELING Lecture 4. AUTOMATIC MODEL CALIBRATION: OBJECTIVE FUNCTIONS, RESPONSE SURFACE Hanoi, Vietnam, 2012

MODEL CALIBRATION / PARAMETERS F-transformation of OPTIMIZATION the response surface J denotes an objective function (a criterion of the model efficiency) J Pmin a A Pmax

EXAMPLES OF AN OBJECTIVE FUNCTION • Average error • Root mean square error • Nash-Sutcliff • Average absolute error • Absolute cubic error

AUTOMATIC MODEL CALIBRATION: SAC-SMA • Quasi-global algorithm SCE (Shuffled Complex Evolution) is computationally “expensive” and sensitive to a number of parameters, needs well defined boundaries • Quasi-local algorithm SLS (Stepwise Line Search) is computationally efficient, but quite sensitive to an initial point for pattern search

Disadvantages of “global” calibration

Simultaneous random generation of the model input ensembles and quasi-local model calibration • Low sensitivity to setting of the random generator • Possibility to calibrate models in data scarce regions • Use of the Stepwise Line Search algorithm (modified pattern search) • Possibility to obtain unbiased parameters

AUTOMATIC MODEL CALIBRATION: MOST CRITICAL ISSUES • Parameter stability in time → their reliability • Parameter stability in space → possibility to be used in ungauged basins

Need to identify many interdependent parameters Irregularity of the multi-dimensional response surface Low stability of the response surface Selection of appropriate objective function Possibility to calibrate models in realistic time

PARAMETER STABILITY IN TIME J Behavior of the response surface when new data are added → P

PARAMETER STABILITY IN SPACE J Parameter transferability: ability to be used in a different catchment → P

WHAT WE MEAN BY PARAMETER CONTINUITY GENERAL PATTERNS OF THE RESPONSE SURFACES • for the same basin and different time intervals → parameter/forecast reliability • for different basins → parameter transferability

HOW TO ASSESS PARAMETER CONTINUITY? Temporal stability slightly change the initial point for pattern search Spatial stability calibrate a (semi)distributed model using various cells or subbasins of the same basin

STABILITY IN TIME P T

STABILITY IN TIME

STABILITY IN SPACE… F-transformation of the response surface J Pmin a A Pmax …AND TIME!

INTERCOMPARISON OF OPTIMA F-transformation of the response surface J Pmin a A Pmax

― HUH… COULD YOU SIMPLIFY IT? . . J F-index: the average J in an optimum neighborhood F → P ― SURE!

DO YOU WANT EXTRA PARAMETERS? New parameters should decrease F-index!

“NATURAL” SMOOTHING RESPONSE SURFACE • Smoothing based on penalty functions x • Smoothing through F-transformation v • Smoothing through using special objective functions: - Multi-Scale Objective Function (MSOF) - Most Informative Scales Objective Function (MISOF) - All Scales Objective Function (ASOF) v

MULTI-SCALE OBJECTIVE FUNCTION Kuzmin, Seo, Koren, “Fast and efficient…”, Journal of Hydrology, 2008

MULTI-SCALE OBJECTIVE FUNCTIONS • MSOF • MISOF • ASOF

SLS BASED CALIBRATION ALGORITHMS • Basic Stepwise Line Search (SLS) algorithm (needs a priori parameter set) • SLS-F (F-indices + SLS) • SLS-2 L (two loops) • SLS-E (ensembles of input + SLS) • SLS-EF, SLS-2 LE, SLS-2 LF, SLS-2 LEF

Catchment area, км 2 INITIAL MSOF FINAL MSOF Onion Creek –Austin ATIT 2 844 23. 21 19. 36 20. 84 20. 85 24. 01 16. 24 Denton Creek – Justin DCJT 2 1039 18. 47 16. 13 16. 57 18. 88 14. 99 Greens Bayou – Houston GBHT 2 137 13. 82 11. 35 11. 66 14. 12 9. 51 Catchments SCE SLS-F SLS 2 L SLS-E South Fork – GETT 2 334 17. 39 16. 22 16. 54 17. 32 16. 03 Cowleech Creek – Greenville GNVT 2 212 16. 89 14. 39 14. 60 17. 90 11. 72 Brays Bayou – Houston HBMT 2 246 35. 69 27. 02 28. 53 42. 48 24. 18 Guadalupe River – Hunt HNTT 2 769 39. 50 30. 99 31. 01 31. 02 37. 00 28. 12 Double Mount Fork – Justiceburg JTBT 2 945 13. 73 12. 19 12. 86 12. 89 15. 97 10. 67 Sandy Creek – Kingsland KNLT 2 904 18. 38 11. 55 13. 67 13. 88 9. 66 Davidson Creek – Lyons LYNT 2 508 10. 51 10. 22 10. 37 10. 41 9. 10 East Fork Trinity – Mc. Kinney MCKT 2 427 16. 84 13. 87 14. 18 14. 19 15. 31 12. 42 Bedias Creek – Madisonville MDST 2 870 33. 92 25. 79 28. 56 32. 50 23. 42 Midfield – Tres Palacios MTPT 2 435 35. 43 33. 92 33. 83 34. 00 29. 45 Cowhouse Creek –Pidcoke PICT 2 1178 38. 99 38. 00 37. 68 37. 70 38. 12 22. 90 Navidad River – Sublime SBMT 2 896 56. 66 53. 92 54. 57 55. 73 48. 83

VALIDATION 1 2 CALIBRATION 3 4 INITIAL MSOF 3. 55 3. 31 2. 99 1. 90 Kyeabma Creek – Book 1. 472 1. 320 1. 659 1. 118 Kyeamba Creek – Lady Smith 1. 475 1. 272 1. 734 1. 238 Hillis Creek – Mount Adrah 1. 801 1. 423 1. 045 0. 999 Billabong Creek – Aberfeldy 1. 656 1. 512 1. 307 0. 800

AUTOMATIC MODEL CALIBRATION: MLCM A special calibration algorithm is implemented: • For 0, 1 or 2 layers: all possible parameter sets are examined (step size: 5 or 10%) • For 3 and more layers: SCE

INTERESTED IN COLLABORATION? WELCOME!!! vknoaa@hotmail. com

Thank you!