Taming Development and Data Chaos in Military Logistics

Taming Development and Data Chaos in Military Logistics: The ICIS Experience With Lightweight Languages Gary Kratkiewicz Technologies Geoffrey Knauth BAE Systems LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth BBN

ICIS Background • • Web-based software system Analyzes future combat and humanitarian operations Measures ability of Defense Logistics Agency to satisfy wartime requirements for following items: • • • Fuel Food Replacement items (repair parts, clothing, medical items, etc. ) Identifies potential problems • • • When will I run out of stock? What should I buy today for tomorrows needs? Do I have adequate sources & transportation? LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 2

Accomplishments • • • Early Web-based DOD system Saved over $100 million • By computing reduced inventory needs for fuel and food Adopted as part of DOD doctrine Months of calculations reduced to hours • • What-if capability leads to better decisions Short analyses expanded role to crisis action Used in military operations to: • • Assess readiness Compute operational requirements LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 3

Development Notes • • • Started in 1994 Grew from small prototype to large system in 4 phases User interface moved to Web in stages from 1996 to 1999 Approximately a third of code is in Lightweight Languages Significant data issues LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 4

Data, Data • • Large amounts of data • 800 -1000 input files; 200 different formats • Gigabytes of input and output per analysis • Hundreds of analyses “active” Data management issues • Received from many different agencies • Regular and irregular updates • Meandering formats • Duplication between analyses: • • Computations Intermediate & output files LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 5

1. Prototype ICIS – 1994/1995 • • Original prototypes completely coded in Tcl/Tk Vague, rapidly changing, and conflicting requirements Less than 1 MB of data Ease of developing/tailoring lightweight prototypes greatly contributed to initial success LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 6

2. Production ICIS – 1995/1996 • • Lots of data: approaching 1 GB Addition of (mostly) well-specified computational modules Real use with classified data in mid-1996 Mix of languages: • • • Tcl/Tk for GUI and glue for module execution Perl for data preprocessing C/C++ for computational modules LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 7

3. Web ICIS – 1996 -1999 • • Address remote access issues Moved to Web in stages On server: • • Tcl/Tk – glue for module execution Perl – data preprocessing C/C++ and Java – computational modules Perl and C++ – dynamic generation of HTML and Java. Script In browser: • • • HTML – static GUI elements Java. Script – form-based GUI elements Java AWT applets – interactive GUI elements LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 8

4. Redesigned Web ICIS – 2001/2002 • Improve GUI and better manage data • • • Data approaching 1 TB On server: • • • JScheme – managing data & module execution (sophisticated “glue”) Perl – data preprocessing C/C++ and Java – computational modules In browser: • • • HTML – static GUI elements Java. Script (much less) – simple GUI elements Java AWT applets – more complex GUI elements Java Swing applets – most complex GUI elements JScheme for understanding scenario LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 9

Source Code Evolution LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 10

Summary of Lightweight Language Use in ICIS • Rapid prototyping • LL replaced by HL: completely for computations, • • • partially for GUI Highly iterative, innovative design & development • LL used in Data Set Manager, patent pending Processing of meandering data Gluing together heavyweight modules • LL replaced with more sophisticated LL Building little languages Test code LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 11

JScheme — Application Flexibility • Interpreter for debugging • Scripts for component functionality • Can be a server for execution management of multiple tasks • Example implementation: BBN’s Data Set Manager LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 12

User Asks ICIS Questions Example: “How much Fuel do I need? ” • • • User interaction with ICIS is web-based User defines composite scenario User specifies results of interest ICIS runs the assessment • Numerous modules might be run • It might take a while… Results appear in browser LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 13

An Example Small Assessment LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 14

Problems • Obvious approach of using shell • • scripts and files doesn’t cut it here Too much data Many computations recomputed needlessly LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 15

A Full ICIS Assessment Does Many Computations The readable version of this graph takes up a large wall. LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 16

Data Set Manager • • • Data is described by features, not file names A module spec describes • Inputs and outputs of a module • Process relationships with other modules Modules can be implemented in any language A module spec is applied to a composite scenario to produce an execution script The script is executed All computations are memoized LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 17

Execution Management Needs Dynamic Capabilities Data Set Manager takes full advantage of Scheme LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 18

Implementing the Data Set Manager in JScheme was fairly easy because: • Quasiquote was used to go from • • module spec to execution script A full language was required to write module specs Executing the script was just walking a tree structure LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 19

Murphy Strikes Start Goal Mistrial! LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 20

MVC: JScheme = M+C, Java = V Draw graph from knowledge of assessment state Generate User Options Show Details Try Again LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 21

Data Set Manager — Conclusion • • • Core implementation is 3804 lines of JScheme code Supported by 287 lines of Perl, 102 lines of SQL ICIS Module specs add another 1853 lines of JScheme Current APIs for other languages are: • 688 lines of C • 496 lines of Perl • 356 lines of Java (1 class) • 120 lines of JScheme BBN has applied for two patents LL 2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth 22