Infopipes and the Infopipe Stub Generator Galen Swint

Infopipes and the Infopipe Stub Generator Galen Swint, Calton Pu, Younggyun Koh, Wenchang Yan

Overview The Infosphere project Goals of the ISG Implementation Current results Infopipes and OEP Future Work

The Infosphere Project Distributed computing with RPC is hard RPC semantics do not fit streaming applications Need services to be more composable Qo. S concerns • “expanded” Qo. S • eliminate redundant Qo. S coding Enhance portability • Same specification can be made for several different comm. machines

RPC Motivation • Procedure call abstraction hides network Application must address Qo. S Does not understand application packets Request/Response is an unnatural fit for streaming media (movies/sound) • Composable? • Basis for CORBA, RMI, Sun RPC, SOAP We need an information-centric abstraction that is network aware to complement RPC

Goals for ISL/ISG Form the core of an Infopipe toolkit ISL • • Simple description for flows Support datatypes Support “basic” composition Qo. S requirements ISG • Generate datatypes, communication stubs • Support multiple communication layers • Support multiple languages

Implementation ISG becomes part of a process XSLT + C++ Brand new XIP (Young) New comm. added more quickly (but needs more typing!) XSLT allows easy modularity in templates (xsl: include) Must re-implement the aspects Buzzword compliant

Implementation v 0. 2

μ-benchmarks Round trip time (ping-pong) • One number sent and received on separate simplex channels • 100 experiments of 1000 ping-pongs Mean Time Java RMI ECho/Infopipe TCP socket Std. Dev. 1. 5 sec 0. 1 sec 0. 35 sec 0. 04 sec 0. 152 sec 0. 001 sec

Experiment 1 - UAV demo Do. D project ECho Remote camera sends data via wireless link • Demonstrates code uploading • ISG generates replacement comm. code from SIP/XIP • • •

Experiment 1 - UAV - Results Initialization • 101 runs, discard first • No statisical difference Original Infopipe Frame transfer • 101 runs, discard first • No statistical difference Original Infopipe Mean Time 19. 5 ms 19. 6 ms Std. Dev. 0. 6 ms 0. 7 ms Mean Time Std. Dev. 598 ms 33 ms 606 ms 45 ms Same performance with 36% fewer LOC!

Conclusions No performance hit Substantial LOC savings Time savings for development Potential portability LOC savings probably varies with App

OEP Integration Recap Participation in BBN OEP • Source-based filters for adaptive constrained resource management in the Multi-UAV demo • New filters as Infopipes (compression, encryption) • Better DRE application programs and better Qo. S support (more platforms and dimensions) Infopipe software tools • Infopipe Stub Generator, program viewer, other tools • Demo apps (BBN OEP, Boeing OEP, etc)

New Infopipe Functionality Three kinds of filters (and more) • Image filters (lower/higher resolution, B&W) • Compression (gzip, lzo, JPEG) • Encryption (Rijndael) Two underlying platforms • BSD sockets/TCP and TAO AVStreams/UDP • And more: event channels, publish/subscribe Automated Qo. S management • Adaptation by choosing appropriate combination of filters (also controlled manually at runtime)

Abstract Infopipe View Defragmenter Assembler PBIO/XML Stub Middle Stub PBIO/XML generator method parser UDP Web. Cam ML PBIO/X tor ra Middle Stub gene /XML Stub method PBIO r parse Wireless Link-TCP BBN Video Distributor DVDview MPEG/PPM ATR (PPM) Qu. O Contract Source Filter Control GUI

Single Filter Experiments Application Infopipe Spec. (evolving) Lang. (ISL) GUI for Infopipe Specification XIP ISL 2 XIP (fixed, extensible) Infopipe Stub Generator XML parser Stub Middle method Stub XML generator XML

Multiple-Filter Experiment (1) Refining the information flow by combining Infopipes through connectors Source Image Filter Compress Encrypt DVDview MPEG/PPM Decrypt Decompress ATR

Multiple-Filter Experiment (2) Self-configured Infopipes (when data stream is unencrypted, system bypasses the decryption stage) Source Image Filter Compress DVDview MPEG/PPM Decompress ATR

Multiple-Filter Experiment (3) Adaptive and flexible distribution of Infopipe stages at runtime Source Intermediate Processing UAV Compress Image Filter Distributor Decrypt Encrypt DVDview MPEG/PPM Decompress ATR

Connector Specialization Different machines XML Middle XML Stub Socket Connector parser method generator Same machine Same process/CPU IPC Connector Function Call Conn. XML Middle Stub parser method Function Call Conn. Stub Middle Stub XML method generator

Future Work Stabilize comm. layers Some basic type checking Use aspects for Qo. S (w/Lenin S. ) Wrap in specialization (w/Young K. ) Use aspects for performance adaptation

Planned Experiments Infopipe (filter) development and integration • Development cost/time, variety of filter code • Code quantity (ISL/XIP module size), quality (runtime overhead), portability (variety of underlying platforms) • Integration into OEP, making Qo. S work with new filters (measure Qo. S dimensions below) Qo. S dimensions and trade-offs • Performance (e. g. , latency, bandwidth, image resolution) • Security (e. g. , encryption level) • Other platforms in OEP that support Qo. S

Credits Initial stub generator – Morimori ECho – Greg Eisenhauer, Karsten Schwan Java socket template – Younggyun Koh C socket template – Volkan Altuntas Initial Qo. S – Wei Han Demo harnesses, demo integration – Wenchang Yan Initial UAV demo – Fabian Bustamante, P@rick Widener

XIP Example Creating a data type • From the UAV demo, this holds a frame <datatype name="Raw_data"> <arg type="integer" name="tag" /> <arg type="char" name="ppm 1" /> <arg type="char" name="ppm 2" /> <arg type="integer" name="size" /> <arg type="integer" name="width" /> <arg type="integer" name="height" /> <arg type="integer" name="maxval" /> <array. Arg type="byte" name="buff" size="155000" /> </datatype>

XIP Example Creating a filter • From the UAV example, this crops an image to half size <filter name="crop. Image 1 Cto 2 C" uri="crop. Image. ecl" in. Type="Raw_data 1 C" out. Type="Raw_data 2 C" takes. Params="1" param. Type="Params_rectangle" />

XIP Example Declaring a simple, singular pipe • From UAV example, the sender and receivers • Note each is only a half-pipe <pipe name="uav. Sender" abs. Machine="ECho 2"> <outport name="send 1" type="Raw_data" /> </pipe> <pipe name="uav. Receiver. Normal" abs. Machine="ECho 2"> <inport name="receive 1" type="Raw_data" /> </pipe>

XIP Example Connecting pipes together <composed. Pipe name="multi. UAV"> <declarations> <pipe name="sender" class="uav. Sender" /> <pipe name="receiver" class="uav. Receiver. Normal" /> </declarations> <ports /> <connections> <connection> <from pipe="sender" port="send 1" /> <to pipe="receiver" port="receive 1" /> </connection> </connections> </composed. Pipe>

ISG Template Snippet for current version of ISG • Submit function for sending data • for-each command generates for multiple outputs • “node. Pipe. Out. Connect” captures some semantic data about the template <xsl: for-each select="/xip/pipes/pipe[@name=$this. Pipe. Class]/outport"> <xsl: variable name="port. Name" select="@name"/> <node. Pipe. Out. Connect mark="true"> int <xsl: value-of select="$this. Pipe. Name"/>_<conn. Type/>_<xsl: value -of select="$port. Name"/><conn. Name/>Submit( ); </node. Pipe. Out. Connect> </xsl: for-each>

Et voila! The code produced //int sender__send 1 Submit( Carrier. Struct * ); int sender__send 1 Submit( );

μ-benchmarks Single Integer • Loop 100, 000 times sending 1 integer (4 bytes) ECho/Infopipe TCP socket Mean Time 2. 6 sec 0. 122 sec Std. Dev. 0. 015 sec 0. 003 sec 1000 Integers • Loop 10, 000 times sending 1, 000 integers (4000 bytes) ECho/Infopipe TCP socket Mean Time 3. 463 sec 3. 393 sec Std. Dev. 0. 004 sec 0. 003 sec